This is (partially) answered in this question: Git: Access Control? How to do in practice:
This (always downloading the entire repo) is a fundamental design decision in git, no way around this.
If you just want to provide read access, you could use some different system, e.g. have some regular job that mirrors file to e.g. a web server, where they can be viewed & downloaded.
If you want to provide write access, but still restrict read access to other parts, there's probably no way around splitting the repo (or implementing some complex synchronization scheme involving multiple repos and pushing & pulling).