I have a large set of ASCII text files that I need to modify with content found in the file then concatenate them all into one summary file. I've had moderate success using a step-by-step approach so far. But, I'd like to do this in one fell swoop and haven't been able to figure it out. I've written a script in PowerShell which will do what I want for one file but I can't seem to figure out how to modify it for all files in the folder. I've been struggling with the looping and the final file concatenation sections.

Here is my basic file structure:


Line1 - abc123 - header (I want to strip characters 2,3, & 4 from here and add to the data lines)
Line2 - data1
Line3 - data2
LineN - abc123 - footer (a duplicate of the header)


Line1 - efg456
Line2 - data3
Line3 - data4
LineN - efg456


Line1 - hij789
Line2 - data5
Line3 - data6
LineN - hij789

I would like the final output file to have the following structure with the header and footers removed and the content extracted from the header in a new column at the beginning of the file (these are fixed width files):


Line1 - bc1data1
Line2 - bc1data2
Line3 - fg4data3
Line4 - fg4data4
Line5 - ij7data5
Line6 - ij7data6

I've been able to solve this for one file and have put the code below:

cd "C:\Data\Files\"
$S1 = Get-Content File1.txt -First 1
$S2 = $S1.Substring(2,3)
Get-Content File1.txt | ForEach-Object {Add-Content a.txt "$S2$_"}
Get-Content a.txt | Select -Skip 1 | Select -SkipLast 1 | Set-Content out.txt
# extract header line
# limit to characters of interest
# append to the beginning of each line in file
# append all files together w/o first & last line

Ideally I could do this without the junk file a.txt too but I've only been able to get it to work by writing to a separate file so far. Any suggestions/tips would be appreciated as I'm fairly new to PowerShell (few hours of practice).

## Q:\Test\2017\08\25\SU_1244148.ps1

Set-Location "C:\Data\Files\"
$Files = Get-ChildItem File*.txt
$Final = ForEach ($File in $Files){
    $Content = Get-Content $File
    $Prefix = $Content[0].Substring(1,3)
    For ($i = 1;$i -lt $Content.Length-1;$i++){
         "{0}{1}" -f $Prefix,$Content[$i]
$Final | Set-Content Final.txt

> gc .\Final.txt

Version 2 handles large files

## Q:\Test\2017\08\25\SU_1244148_2.ps1
#Set-Location "C:\Data\Files\"

$Files = (Get-ChildItem File*.txt|Sort)
$Final = '.\Final.txt'

## As we append to $Final initially clear
If (Test-Path $Final){Remove-Item $Final}

ForEach ($File in $Files){
    $Reader = [IO.File]::OpenText($File)
    $Header = $Reader.ReadLine()
    $Prefix = $Header.SubString(1,3)
    $Line = $Reader.ReadLine()
    while ($Reader.Peek() -ge 0) {
        $Prefix+$Line| Out-File $Final -Append
        $Line = $Reader.ReadLine()

Optimized code to not have to test for trailing header.
The while loop prints the previous line and reads the next this way dropping the very last line.

  • Thank you @LotPings this worked great on small test files but as soon as I ran it on large ones it threw an out of memory exception. – Karl Aug 25 '17 at 22:29
  • In this Q&A on SO are some concepts to deal with large files. See appended 2nd script version. Worth an upvote ;-) – LotPings Aug 26 '17 at 9:14
  • I tabled this for a bit so circling back now. Version 2 worked well for me but was really slow when I ran it on large (>= 0.5 GB) files. So I swapped the Out-File line for StreamWriter.WriteLine(). – Karl Oct 13 '17 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.