I actually figured it out...
The reason is the Downloads folder is "locked". You can see whether is locked or not by going -> Get Info in the macOS GUI.
ls -l shows that the user priviledge was lrwx------+ meaning that is locked.
Once it's unlocked, the priviledge will become drwx------+ as usual.