If you'd rather not use Java Web Start as Andrew suggested and would like to learn how Windows specifically handles default applications, here's a quick guide:
When you "Right click -> Open With..." and set something as a default application, a registry value is created in
HKEY_CLASSES_ROOT with the name "somefileextension_auto_file" and a key "/shell/open/command" (so,
HKEY_CLASSES_ROOT\somefileextension_auto_file\shell\Open\command). The default string value of this key is the shell command to be executed when a file of
somefileextension type is double clicked. The default value created looks something like
"c:\program files\somepath\test.exe" %1 where %1 will be a variable containing the full path to the file that was double clicked.
If you wanted to programmatically make your jar file the default application for a certain file extension, you would need to create the registry key at
HKEY_CLASSES_ROOT\somefileextension_auto_file\shell\Open\command with a default value similar to the following:
java -jar c:\somewhere\yourJarFile.jar %1
and your program would need to detect when a command line argument has been given to it using
public static void main(String args) where
args would be the path to the file that was clicked. If a command line argument was given, your program should automatically attempt to use it as an input file.
Note: This really shouldn't be done manually as Java was designed to be cross platform and as such there are cross platform solutions ( https://stackoverflow.com/tags/java-web-start/info ) but it seems the question author was interested in how Windows actually deals with default applications. Maybe i'll help another reader. Who knows.