Based on what what little information was provided in the question, here are a few guesses:
That is CSS. You need a descrambling/video-cracking program to remove it. The most popular programs for that are DVDFab and AnyDVD (the free DVDShrink could do it as well, but was discontinued a long time ago, so it won’t handle newer protections).
The distorted video and audio is due to the CSS (Content Scrambling System), so you need to decrypt before you send it through the compressor/transcoder (ie, before you shrink the video). If you open a VOB file in a hex-editor directly, you will see different bytes than if you open the same file after it is decrypted (no surprise). Play the video with and without encryption and you will see magenta blocks, hear screeches, and so on.
If you already tried some programs (DVDFab, RipIt4Me) without success, it is likely because you opened the VOB files (eg
VTS_01_0.VOB) directly instead of opening the disc (ie
VIDEO_TS\VIDEO_TS.IFO), or at least the VOB’s
If you use a real-time decrypter, ,ake sure to rip the disc to the hard-drive, and then compress it instead of just sending the decrypted stream to the compressor because they don’t usually handle structural protections.
Decoding CSS is usually not enough. DVDs use all kinds of protections both old and new, so you will need a proper, up-to-date DVD decrypting program.