The solution is to add the network drive to trusted locations.
To do this, open Excel (or any office desktop app), then go to File -> Options -> Trust Center -> Trust Center Settings... -> Trusted Locations.
At the bottom, make sure to check "Allow Trusted Locations on my network"
Then click Add new location...
Click on Browse... and browse to the location of the network file where the preview pane isn't working. Hit OK, and if you are selecting a parent directory of the file, make sure to check "Subfolders of this location are also trusted," and click OK.
Make sure to click OK on the trust center window, and you should be good to go! You may need to restart explorer and/or Excel, but it should work now.
I'm adding this on here because I've had multiple users have this problem, and there are a LOT of unresolved microsoft forum posts discussing this exact issue.