Generally yes, you can use whatever extension you want, also there's no need to, say, register file extensions through some international body or anything.
However, some programs expect their files to have a certain extension - this usually matters more in a graphical environment - i.e. Open File dialogs filter their contents by extension, or Save File dialog can guess in what format to stave the file according to which extension user provided.
In Windows using arbitrary extensions may be e bit more troublesome because of the way Open and Save dialogs work, but still you can have a text file with extension .kjhkhkj
Regarding "how the system decides what program to use": you tell it. In the terminal you call the program and give it a file name:
gedit message.kjhkjhkj
and it'll be open in gedit
In Nautilus, you right-click on the file and choose "Open with...", where you can specify a program. It's also possible to make this association permanent so .kjhkhkj
files are always opened with gedit.
In Windows the procedure is similar, though Windows tend to rely more on known file extensions (i.e. video files will have preview thumbnails in Ubuntu regardless of their extension, Windows need them to have one of known extensions (.avi, .mov etc.)