After years struggling with safely removing USB external hard drives, and getting the famous "Windows can't stop your device because a program is still using it. Close any programs that might be using the device, and then try again later", I've finally found out what was holding it.
I've read issues and tried every possible solution suggested all over superuser and the web, nothing worked. (such as all answers in this: How to safely remove a USB drive when "Windows can't stop your 'Generic volume' device because a program is still using it."?)