Why is this happening?
Because you're storing a lot of tiny files.
Filesystems have a minimum file size that they can store. For NTFS filesystems, it's usually 4KB. For exFAT, it can be much larger. That's called the block or cluster size. Files that are smaller than this size will still use up the minimum size, so a 1KB file might use 4KB of disk space. A 3KB file would also use 4KB of disk space. If you have a 5KB file, it'll use 8KB of disk space.
You can imagine it like a grid of holes. Each hole can hold a certain amount of data. Files are spread across as many holes as necessary to hold all the file's data, but holes can't have data from more than one file. So, if a file's data doesn't completely fill a hole, some of that space is wasted. No other file can use it that hole so the unused space is unavailable.
What can you do about it?
In your case, you have a lot of files that don't fill the holes, so there's lots of wasted space. If you were to put all the files into a ZIP file, then all that data would be contained in a single file and it would use a lot less space on the drive.
Some USB drives are formatted as exFAT by default, so alternatively, if you're just using this drive to copy files between Windows computers (or just for storage), you could try reformatting the drive as NTFS (but copy all the files off first, obviously!) to try to get a smaller cluster size.