In Windows XP I'm using a little gadget called 'Thumbnails'.
Xentient Thumbnails is a utility
for replacing the generic icons of
image files with thumbnail icons of
the actual image. These thumbnail
icons are much easier to recognize and
they work everywhere icons are
displayed, including on the Desktop,
in Open and Save dialogs, and in
Windows Explorer. The utility
automatically generates thumbnail
icons for most popular image types
including JPG, JPE, JPEG, JP2,
J2K, GIF, PNG, BMP, WBMP, EMF, WMF, PCX, PBM, PGM, PNM, PPM, RAS,
TIF, TIFF, and TGA. Xentient
Thumbnails is licensed as freeware,
full size of the download is 0.71 MB.
Xentient doesn't offer the program for download anymore, but here's a working link from Brothersoft.
Thumbnails works with Windows 7. Voilá, JPEG2000 preview in Windows 7 Explorer :)

However it doesn't stop there: see the 'thumbnailed' desktop shortcut of the same JP2 file? sweet! :)