Honestly, all you actually need (at least to play) is found in the SRD; if you've got the core three, then you've got everything you need and a little more already. Everything else is a question of how many options you want to have available for your character. If you're in an 'any source' type of game, then you'll probably find that other players have items, feats, spells, races, classes etc that you don't know about; as a player, that generally doesn't matter.
If you want to run, rather than play, you'll either have to run a 'core only' game (or which there are a reasonable number around anyway) or accept not knowing the full details of everything your players can do (which can be a little dicey, as you don't really know in advance how powerful they are). That's easily solved by recruiting someone with lots of books to look over sheets, as they can pull out anything you need to know about.