I've installed [insert favourite text editor here] and associated it with all file extensions which by default open with notepad.exe. Still, there is a context menu entry Edit e.g. on *.txt files which still brings up Notepad.

For *.txt files, the default value of HKEY_CLASSES_ROOT\.txt points to HKEY_CLASSES_ROOT\txtfile, which defines the context menu entries
*.txt files have a perceived type text (see PerceivedType property in key HKEY_CLASSES_ROOT\.txt), and that type defines its context menu entries at HKEY_CLASSES_ROOT\SystemFileAssociations\text
If you also change the commands for the perceived type text, you should always get your favourite editor for all text files.