I found HFSExplorer, which can handle encrypted DMG files, but it only supports HFS, and the DMG I want to open is NTFS. I can't just use any image mounter since the DMG file is encrypted, and I don't want to need to copy it over to my main hard drive first. (The DMG is on an external hard drive.) If copying it is the only way, however, I'd be willing to settle for that, but I'd rather just leave it on the external hard drive in an encrypted form.
The encrypted layer was reverse engineered in an implementation called VileFault (A spoonerism of FileVault), and dmg2img and DMGExtractor have since implemented support for encrypted images.
Use dmg2img or DMGExtractor to convert the encrypted dmg into a regular one. This removes the outer encryption layer which is independent of the actual filesystem stored inside the
dmg. Then, if it's NTFS, use an app that can mount a raw disk image in Windows (this thread suggests that you can use ImDisk to do that).
Using HFSExplorer will probably not work if it only works with HFS volumes.