0

I want to automate selecting data out of a pdf file for later processing in a spreadsheet. 1: Copy all text into a .txt file, using Notepad - ok. 2: Remove all newlines, CRLF, to create a single line file,using Powershell - ok

3: Sed edits to cut out unwanted text don't work (I have successfully used sed for 2 years).

powershell -command ""$(Get-Content T1.txt)" > t2.txt"

sed -i -r "s/Limited/XX/g" t2.txt - does not work.

Yet removing all newlines manually using Notepad++ (using regex \r\n) to create the txt file, the same sed command works ok. Any ideas? Maybe alternative to removing all newlines in a .bat file? Using Win7 Prof 64-bit.

1 Answer 1

0

sed is a POSIX tool designed to handle text. Possibly your "single line file" is not a text file in terms of POSIX:

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. […] many utilities only produce predictable or meaningful output when operating on text files.

Unless empty, it will contain an incomplete line only:

A sequence of one or more non-<newline> characters at the end of the file.

Compare to the definition of line:

A sequence of zero or more non-<newline> characters plus a terminating <newline> character.

LF is the <newline> character (CR is not; in this context CR is not special).

I don't know Powershell and I cannot tell if your code is sane, but if it removes all newlines then it probably creates an incomplete line. I guess Notepad++ is smart enough to complete the line anyway.

Some implementations of sed don't process incomplete lines (example) and some do (at least GNU sed does). Apparently your sed doesn't. Find sed that does or make sure the "single line" is properly terminated by a <newline> character. E.g. in Unix this command:

echo >> T1.txt

appends exactly one <newline> character to T1.txt. I don't know a Powershell way, sorry.

Note if the fixed T1.txt exceeded {LINE_MAX} bytes in its single line, then formally it would still not be a text file, so text processing utilities might still not work with it. My point is your sed may have a line length limit and therefore it may not work in some cases, even after you fix the file by adding a terminating <newline> character.

See this answer to get an idea for workaround.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .