How to unwrap 80 character text

Question

Many text filess around Internet are word split before line reaches 80th character. I understand it is desirable by those using 80 character terminals. However I like sending longer articles to my kindle, but those files look very bad on it. See this picture:

80-charater splits are clearly visible when text ends in middle of line.

My question is: How to unwrap/uncut those lines? "By hand" method is, of course, out of discussion. EDIT: Lets clarify: I'm asking for script or other method that would be possible to run from Linux terminal. bash/ruby/python/perl/awk are ok.

Also, assume I have just plaintext, LWN is just example.

Answer 1

The answer using fmt seems to be to wrap text rather than unwrap it.

In general, this can be a difficult problem. For example, distinguishing between adjacent lines of text which are deliberately finished early (e.g bullet points) and adjacent lines of free flowing text can require some context. Distinguishing between hyphenated words split across lines and split up words is also hard.

A common form for prose, however, is adjacent wrapped lines of text forming a paragraph separated by a single empty newline.

This can be unwrapped using the following, rather involved, sed one liner:

sed -n '/.+/ H; /^$/ { x; s/\n/ /g; s/$/\n/ ; p}'

Alternatively you might prefer a tiny python script, particularly if you are going to handle some special cases:

import sys
paragraph = []

for line in sys.stdin:
    line = line.strip()
    if line:
        paragraph.append(line)
    else:
        print ' '.join(paragraph).replace('  ', ' ')
        paragraph = []
if paragraph:
    print ' '.join(paragraph).replace(' ', ' ')

If you find yourself adding special casing then you like to find the origin of your line-wrapped text and obtain it in a non line-wrapped form.

Answer 2

Special cases, as Att Righ said…

I found this question because I wanted to "unwrap" output from the fortune program, which annoyingly isn't even standardized — some fortune cookies are wrapped at 78 character, others at 77, 76, or even 75.
My script tries to determine if a newline has been inserted on purpose or because of the length limit by determining if the line would violate the length limit if it hadn't been broken at this exact length (i.e. if it would be too long if it also included the first word from the next line). As a useful side effect, if the next line starts with whitespace, the first word (as separated by whitespace) is the empty string, so indented paragraphs are never merged onto the line above them.

#!/usr/bin/python3

import sys
import fileinput

lines = list(fileinput.input())
lines = [l.strip('\r\n') for l in lines]

for i, l in enumerate(lines):
    # We need to account for 8-char-wide tabulators when calculating our line
    # length, but still want to print the original \t characters verbatim
    sanitized_line = l.replace('\t', ' '*8)

    # Is there a next line?
    if i+1 < len(lines):
        sanitized_next_line = lines[i+1].replace('\t', ' '*8)
    else:
        sanitized_next_line = ''

    next_line_first_word = sanitized_next_line.split(' ', 1)[0]

    if next_line_first_word != '':
        extended_line = sanitized_line + ' ' + next_line_first_word
    else:
        extended_line = sanitized_line

    if len(sanitized_line) <= 78 and len(extended_line) > 74:
        # This line was wrapped due to 78-char limit => unwrap it!
        sys.stdout.write(l + ' ')
    else:
        sys.stdout.write(l + '\n')

Answer 3

I just came across this post while looking for way to do this with sed. Here's my shell script I have been using for most of my unwrapping:

# unwrap text with linebreaks in paragraphs and blank lines between paragraphs
# This script reads stdin and writes stdout
# cat is used to read stdin into a here-string so that lnum may be printed at the end, outside of the loop

oline=""
lnum=0

while read aline
do
    lnum=$((++lnum))
    # check for probable blank line by its length, end of paragraph
    if [ ${#aline} -lt 3 ]
        then
        # output concatenated line
        echo "$oline"
        # output blank line
        echo ""
        # prepare for the next paragraph
        oline=""
    else
        # otherwise add it to the unwrapped line
        oline=$(echo $oline|tr -d '\n'|tr -d '\r')
        oline="$oline $aline"
    fi
done <<< "$(cat)"

echo "$lnum"
exit 0

Answer 4

I had the same problem -- emailing plain text files containing hard line feeds within the body of a paragraph at 80 characters ... and those files look terrible on devices that are (a) doing their own line wrap, and (b) wrapping at a width less than 80 ... that is, the 80-charater splits are clearly visible where the text ends in the middle of line.

How to unwrap those lines?

Use the unix command line tool 'fmt', as suggested by Raúl Salinas-Monteagudo. Change the line length (-w ##) to a useful number. You could try 65 or 55, or, try 1111 or larger.

Example:

This file from the weather office (https://forecast.weather.gov/product.php?site=NWS&issuedby=AT&product=TWO&format=txt&version=1&glossary=0) contains hard line feed embedded within the body of a paragraph. I want line feeds only at end of paragraph. Piping the output to 'fmt -w 1111' (for paragraphs less than 1111 characters in length) takes care of the problem:

/usr/local/bin/w3m -dump 'https://forecast.weather.gov/product.php?site=NWS&issuedby=AT&product=TWO&format=txt&version=1&glossary=0' | sed -n '/TWOAT/,/Forecaster/p' | fmt -w 1111

.

Changes appearance from this:

The National Hurricane Center is issuing advisories on
Tropical Storm Beryl, located a couple of hundred miles
east of the Lesser Antilles, and on Tropical Storm
Chris, located more than one hundred miles south of the
North Carolina Outer Banks.

Tropical cyclone formation is not expected during the
next 5 days.

to this:

The National Hurricane Center is issuing advisories on Tropical Storm Beryl, located a couple of hundred miles east of the Lesser Antilles, and on Tropical Storm Chris, located more than one hundred miles south of the North Carolina Outer Banks.

Tropical cyclone formation is not expected during the next 5 days.

Answer 5

Use perl -00 to slurp in paragraph mode

This makes a basic unwrapping command very easy:

perl -00 -lpe 's/\n/ /g'

Adding compression of multiple spaces:

perl -00 -lpe 's/\n/ /g, s/ +/ /g'

Answer 6

I think that exactly what you want is performed by a simple:

fmt -w 80 < input > output