According to djvu.org
DjVu (pronounced "déjà vu") is a digital document format with advanced compression technology and high performance value. DjVu allows for the distribution on the Internet and on DVD of very high resolution images of scanned documents
So we have
- advanced compression technology
- high resolution bitmap images
PDF is optimised for vector based drawings and text not for compression of bitmapped images.
(update)
PDF supports several types of compression for raster images so choosing the right one can help get size down. The CCIT Fax4 compression is better than some of the others I tried but I believe it is nowhere near as efficient as the compression used in DJVU. I converted a 236 KB DJVU file to a 916 KB TIFF with FAX4 compression. By comparison, using JPEG compression (default settings) produced a 28080 KB TIFF file. I expect similar results after converting to PDF with same compression algorithms.
(end update)
Running the DJVU through OCR may help get the size down but may be a process that needs a lot of human interaction to get content and layout correct.