I too noticed that browsing the "Offline Files Folder" in Windows 10 can be extremely slow. After playing with the feature a bit, my conclusion is that you don't need it most of the time.
First, to make clear what we're talking about, if you go to the Windows 10 Sync Center, click "Manage offline files", and then click "View your offline files", it opens a folder called "Offline Files Folder". It's the browsing of this folder that can be extremely slow, and I also found that Explorer would crash quite often while browsing the folder.
The good news is that you don't really need the "Offline Files Folder" to work with offline files. Normally, you'll interact with all of your offline files at their normal online location, be that a UNC path or a mapped drive letter. That's where your applications will see the files, and that's where you browse in Explorer to select which files to make "Always available offline" (by right clicking, or in the "Easy access" menu). That's also where you can see which files have been made always available offline, by setting your Explorer view to Details and then adding the "Availability" column. Each file will be shown with an availability of "Online-only" or "Available offline". When working offline, the Online-only files will either be totally missing from the filesystem, or will be shown with a blank page icon that has a grey X at the lower left-hand corner. Access to offline files is speedy, just like with any other file on the system drive.
The "Offline Files Folder" is really only good for some specific maintenance tasks. If you right click a server, file, or folder there and then click "Delete Offline Copy", it will delete the offline copy of all sub-folders and files, and change the availability back to Online-only. If I was having a problem with offline files, the first thing I would try would be to delete everything in the Offline Files Folder and start over.
After using offline files for a bit, I've found the feature to be more trouble than it's worth. It often complains of sync conflicts when it shouldn't, and sometimes forgets when I have told it to work offline. I gave up on the feature and now I just copy files manually. Offline files hasn't seen any new features since Windows 8, and probably doesn't have much of a future. Microsoft is more interested in Work Folders and OneDrive these days.