So I downloaded and installed "MiddleClick" application.
If you're talking about this MiddleClick.app, you didn't really install it. It's just an application that runs whenever you double-click it and stops when you quit it from the menu bar. Nothing is installed.
First, quit the application if it's running. Its icon sits in the menubar.
Then, try to delete the MiddleClick.app file again.
If it still doesn't delete, or if you couldn't find the menubar icon, it might still have a process running. Open Applications/Utilities/Activity Monitor.app and search for "MiddleClick":
Select it and click the big red Quit Process icon. Then, try to delete the file MiddleClick.app again.