But why can't the recursive resolvers directly query the TLD servers? From my understanding, there's only approximately 1500 TLD servers, which could easily be distributed to all resolver servers.
Because when you have a list of 1500 servers, you need to distribute updates considerably more often – and avoiding the need to distribute such a list to all users (even if it's just those running a resolver) is kind of the reason DNS was built in the first place.
It would be possible to do this, yes (and in some situations it's actually done) but you'd still need to have those X root servers for distributing the list anyway, so it's easier to just use the existing mechanisms – most likely they are more efficient than retrieving the entire list on every startup.
Something like a root server would be needed to distribute all the TLD servers to resolvers. But they should only be polled once to initialise the TLD list. From my understanding, every DNS request will ask the root server e.g "where is the TLD server for .com?" That seems rather inefficient?
Yes, every DNS request will need such a query, but most of the time it will be fulfilled directly from the resolver's cache, for as long as its TTL allows. There's nothing special about root queries in that regard – they're cached like all other queries in the chain (as well as the final query, of course).
As each TLD is cached individually, this is likely to be more efficient than retrieving the entire list – for a resolver whose users mainly need 10 TLDs, it only has to make 10 queries to the root servers approximately once a day. With a large list of 1500 servers, refreshing the entire list at the same frequency would consume a lot of resources on the root servers, and even if you used incremental transfer you'd still be receiving many changes that you don't need.