Utility to normalize text files

Question

Does there exist a command line utility to check for and ideally fix grunge in text files? e.g.

• Inconsistent or non-native line endings (lf vs cr/lf)

• Extra whitespace at end of lines

• Spurious byte order markings (Visual Studio sometimes inserts these, even though they are neither necessary nor desirable in UTF-8 files)

And possibly other things I hadn't thought of? (Closest I can find so far is something called tab2space, but the primary function of that is as it says on the tin, to expand tabs, and that's something I don't want.)

Answer 1

To convert to/from various line endings there are programs called unix2dos/dos2unix or todos/fromdos.

To remove extra whitespace at the end of the line you can use sed s/[ \t]*$// (do this step after correcting line endings).