5

Many text filess around Internet are word split before line reaches 80th character. I understand it is desirable by those using 80 character terminals. However I like sending longer articles to my kindle, but those files look very bad on it. See this picture:This is  LWN old issue that is automatically fetched and assembled into .mobi

80-charater splits are clearly visible when text ends in middle of line.

My question is: How to unwrap/uncut those lines? "By hand" method is, of course, out of discussion. EDIT: Lets clarify: I'm asking for script or other method that would be possible to run from Linux terminal. bash/ruby/python/perl/awk are ok.

Also, assume I have just plaintext, LWN is just example.

  • 1
    part of the question is, how is the wrapping done in the first place, and how can you distinguish it from natural line breaks? the text you are receiving must have been modified on the source side, since it can't be re-flowed by your client. if you can answer that question, its a trivial matter to write a shell script to remove the wrapped line breaks. if you can't distinguish between the two, the job gets much harder though. – Frank Thomas Jun 22 '13 at 23:14
  • Most of the text I'd like to reflow have two \n`'s when natural break is (and it happens only when new paragraph is created, so there happends to be second newline created). So one null` would make no new line, two null's - two new lines – Szymon Szydełko Jun 22 '13 at 23:20
  • 3
    what editor are you using? Can it "Show CR LF"??? I think NotePad++ can.... most decent text editors can, and then you can do a find/replace on the chars. – Logman Jun 22 '13 at 23:27
  • 1
    @Logman you can also use the TextFX plugin (used to come with Notepad++, I think you have to DL it now from the Plugin Manager) to wrap/unwrap text of an arbitrary width (it will detect and handle the whitespace automagically). – Breakthrough Jun 22 '13 at 23:50
  • CR? At the end of line there are only LF's... Just like there I also use vim/sublimetext/kate (depending on mood) – Szymon Szydełko Jun 22 '13 at 23:54
5

The answer using fmt seems to be to wrap text rather than unwrap it.

In general, this can be a difficult problem. For example, distinguishing between adjacent lines of text which are deliberately finished early (e.g bullet points) and adjacent lines of free flowing text can require some context. Distinguishing between hyphenated words split across lines and split up words is also hard.

A common form for prose, however, is adjacent wrapped lines of text forming a paragraph separated by a single empty newline.

This can be unwrapped using the following, rather involved, sed one liner:

sed -n '/.+/ H; /^$/ { x; s/\n/ /g; s/$/\n/ ; p}'

Alternatively you might prefer a tiny python script, particularly if you are going to handle some special cases:

import sys
paragraph = []

for line in sys.stdin:
    line = line.strip()
    if line:
        paragraph.append(line)
    else:
        print ' '.join(paragraph).replace('  ', ' ')
        paragraph = []
if paragraph:
    print ' '.join(paragraph).replace(' ', ' ')

If you find yourself adding special casing then you like to find the origin of your line-wrapped text and obtain it in a non line-wrapped form.

|improve this answer|||||
  • Is there a BSD sed equivalent? – Alec Jacobson Oct 25 '16 at 2:13
  • Also, the python command should probably have a print ' '.join(paragraph).replace(' ', ' ') after the for loop if paragraph isn't empty to flush out the last paragraph. – Alec Jacobson Oct 25 '16 at 2:16
  • Yep, mostly likely. I didn't actually run this code. I just thought it behoved me to provide a non-sed solution. – Att Righ Oct 27 '16 at 13:35
  • Answer edited for include Alec Jacobson's suggestions. – Att Righ Jan 29 '17 at 21:55
1

Special cases, as Att Righ said

I found this question because I wanted to "unwrap" output from the fortune program, which annoyingly isn't even standardized — some fortune cookies are wrapped at 78 character, others at 77, 76, or even 75.
My script tries to determine if a newline has been inserted on purpose or because of the length limit by determining if the line would violate the length limit if it hadn't been broken at this exact length (i.e. if it would be too long if it also included the first word from the next line). As a useful side effect, if the next line starts with whitespace, the first word (as separated by whitespace) is the empty string, so indented paragraphs are never merged onto the line above them.

#!/usr/bin/python3

import sys
import fileinput

lines = list(fileinput.input())
lines = [l.strip('\r\n') for l in lines]

for i, l in enumerate(lines):
    # We need to account for 8-char-wide tabulators when calculating our line
    # length, but still want to print the original \t characters verbatim
    sanitized_line = l.replace('\t', ' '*8)

    # Is there a next line?
    if i+1 < len(lines):
        sanitized_next_line = lines[i+1].replace('\t', ' '*8)
    else:
        sanitized_next_line = ''

    next_line_first_word = sanitized_next_line.split(' ', 1)[0]

    if next_line_first_word != '':
        extended_line = sanitized_line + ' ' + next_line_first_word
    else:
        extended_line = sanitized_line

    if len(sanitized_line) <= 78 and len(extended_line) > 74:
        # This line was wrapped due to 78-char limit => unwrap it!
        sys.stdout.write(l + ' ')
    else:
        sys.stdout.write(l + '\n')
|improve this answer|||||
1

I just came across this post while looking for way to do this with sed. Here's my shell script I have been using for most of my unwrapping:

# unwrap text with linebreaks in paragraphs and blank lines between paragraphs
# This script reads stdin and writes stdout
# cat is used to read stdin into a here-string so that lnum may be printed at the end, outside of the loop

oline=""
lnum=0

while read aline
do
    lnum=$((++lnum))
    # check for probable blank line by its length, end of paragraph
    if [ ${#aline} -lt 3 ]
        then
        # output concatenated line
        echo "$oline"
        # output blank line
        echo ""
        # prepare for the next paragraph
        oline=""
    else
        # otherwise add it to the unwrapped line
        oline=$(echo $oline|tr -d '\n'|tr -d '\r')
        oline="$oline $aline"
    fi
done <<< "$(cat)"

echo "$lnum"
exit 0
|improve this answer|||||
1

I had the same problem -- emailing plain text files containing hard line feeds within the body of a paragraph at 80 characters ... and those files look terrible on devices that are (a) doing their own line wrap, and (b) wrapping at a width less than 80 ... that is, the 80-charater splits are clearly visible where the text ends in the middle of line.

How to unwrap those lines?

Use the unix command line tool 'fmt', as suggested by Raúl Salinas-Monteagudo. Change the line length (-w ##) to a useful number. You could try 65 or 55, or, try 1111 or larger.

Example:

This file from the weather office (https://forecast.weather.gov/product.php?site=NWS&issuedby=AT&product=TWO&format=txt&version=1&glossary=0) contains hard line feed embedded within the body of a paragraph. I want line feeds only at end of paragraph. Piping the output to 'fmt -w 1111' (for paragraphs less than 1111 characters in length) takes care of the problem:

/usr/local/bin/w3m -dump 'https://forecast.weather.gov/product.php?site=NWS&issuedby=AT&product=TWO&format=txt&version=1&glossary=0' | sed -n '/TWOAT/,/Forecaster/p' | fmt -w 1111

.

Changes appearance from this:

The National Hurricane Center is issuing advisories on
Tropical Storm Beryl, located a couple of hundred miles
east of the Lesser Antilles, and on Tropical Storm
Chris, located more than one hundred miles south of the
North Carolina Outer Banks.

Tropical cyclone formation is not expected during the
next 5 days.

to this:

The National Hurricane Center is issuing advisories on Tropical Storm Beryl, located a couple of hundred miles east of the Lesser Antilles, and on Tropical Storm Chris, located more than one hundred miles south of the North Carolina Outer Banks.

Tropical cyclone formation is not expected during the next 5 days.
|improve this answer|||||
-2

I think that exactly what you want is performed by a simple:

fmt -w 80 < input > output
|improve this answer|||||
  • 3
    This seems to be the opposite of what was asked. – Alec Jacobson Oct 25 '16 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.