The easiest would be to download and install RCDefaultApp. Go to System Preferences » Default Apps, then the MIME Types tab.
Enter your new MIME type in the text box below, press the + button, and select your handler.
Via command line, you need to edit the
~/Library/Preferences/com.apple.LaunchServices.plist preference list. In its array of
LSHandlers, you need to insert a
Dictionary containing your new MIME type as:
LSHandlerContentTag – the MIME type
LSHandlerContentTagClass, which is
LSHandlerRoleAll, which specifies the Bundle ID of the application that handles it, e.g.
org.videolan.vlc. You find out the bundle ID by inspecting the
.app folder and the
Info.plist preference list therein.