4

I have rescued some files (Python, Shell scripts etc., so just text files) from a Linux SD card using FTK Imager Lite. The files are ok, but they now contain a lot of 0x00 bytes at the end.

How could I bulk-remove these trailing 0x00 bytes with my Windows machine?

I am familiar with the for command, so a suggestion on how to do it for one file would be sufficient. I can then apply it to all files.

3 Answers 3

2

Since you say that your files are simple text files, I'll assume all null bytes can be removed, regardless of location.

You can use the MORE command to convert each \0x00 into \0x0D\0x0A

more yourFile.ext >yourFile.ext.new
move /y yourFile.ext.new yourFile.ext >nul

Caveats:

  • Each file must have fewer than ~64k lines else the command will hang (inserts a prompt and waits for a keypress after 64k lines)
  • Tabs will be converted into a string of spaces

A more robust alternative is to use JREPL.BAT - a regular expression text processing command line utility for Windows. It is pure script (JScript/batch) that runs natively on any Windows machine from XP onward - no 3rd party exe file required.

As long as no file is too large (approaching 1GB?), then you can use the /M option to read the entire file into memory and perform the search/replace operation on one big string:

jrepl \x00 "" /m /f yourFile.ext /o -

For large files you can process the file line by line, but the null bytes require specification of a character set for the input which forces the use of ADO. The standard JScript file input routines choke on lines containing null bytes. ADO solves that problem. It does not matter what character set is chosen as long as it is a single byte character set.

jrepl \x00 "" /f "yourFile.ext|windows-1252" /o -

Full documentation is built into JREPL.BAT and available via jrepl /? or jrepl /?? for paged help. A summary of all options is available via jrepl /?options, and a description of all help options is gotten via jrepl /?help.

JREPL.BAT is a lot of code for just this one task. It wouldn't be hard to whip up a very small dedicated JScript or VBS script for this task. But once you have JREPL in your arsenal, you will likely find many uses - it is a powerful tool.

4
  • In my tests, both approaches remove all null characters and not just those at the end.
    – Andrew
    Aug 29, 2020 at 5:35
  • @Andrew - Yes, that is true. The OP did say he had null bytes at the end to trim, but he also said the files were simple text files, which don't normally contain any null bytes. So I assumed all null bytes could be removed, and the fact that all his null bytes are at the end is inconsequential. I'll edit my answer to make my intent clear.
    – dbenham
    Aug 29, 2020 at 19:20
  • Great, thanks. I commented that because I tried these on some binary files but no luck. Keith's solution worked at least.
    – Andrew
    Aug 29, 2020 at 21:13
  • 1
    @Andrew - You can get your desired result easily with JREPL - jrepl "\x00+(?![\s\S])" "" /m /f input.bin /o -. That looks for a string of null bytes without any byte following, and replaces with empty string.
    – dbenham
    Aug 29, 2020 at 22:07
2

The core of a PowerShell solution would look like this:

$file = "C:\Users\keith\SomeFile.txt"

$Bytes = [IO.File]::ReadAllBytes($file)

$i = $Bytes.Length - 1

While ($Bytes[$i] -eq 0) { $i-- }

$Bytes = $Bytes[0..$i]

[IO.File]::writeAllBytes($file, $bytes)

Update: Before I tackle your code, here's the above as Pipelline-enabled Function that can process multiple files. Got rid of the intermediate $Bytes = $Bytes[0..$i] & just specified the truncated array in the [IO.File]::writeAllBytes() method. That should help somewhat with processing time.

Function Strip-Null {
Param(
  [Parameter(Mandatory,
             ValueFromPipeline,
             ValueFromPipelineByPropertyName,
             Position=0)][SupportsWildcards()]
  [String[]]$Path,
  [Parameter()]
  [Switch]$NoKeepDate=$False
)
  Process{
    ForEach ( $file in $Path ) {
        If ( Test-Path $file ) {
            Resolve-Path $file | ForEach {
                Write-Host "Processing '$file'..." -NoNewline
                $Bytes = [IO.File]::ReadAllBytes($_)
                $i = $Bytes.Length - 1
                While ($Bytes[$i] -eq 0) { $i-- }
                $Dest = $_ -Replace '(\.[^\.]+)$', '_fixed$1'
                [IO.File]::writeAllBytes($dest, $Bytes[0..$i])
                If ( ! ( $NoKeepDate.IsPresent )) {
                    ( Get-Item $dest ).LastWriteTime = ( Get-Item $_ ).LastWriteTime
                }
                Write-Host ( "'{0}' created." -f ( Split-Path $dest -Leaf ))
            }
        } Else { Write-Warning "'$file' is not a valid path." }
    }
  }
}
4
  • This one works, but it's very slow if the file is big. So I designed my own application. Here is the code.
    – Andrew
    Aug 30, 2020 at 7:16
  • 1
    Cool. Mind if I play with wrapping that in an Add-Type statement so it's available in PowerShell? Aug 30, 2020 at 16:26
  • Please, go ahead. It's not like I copyrighted it, LOL. I first thought of doing that in PowerShell, including keeping the file date, but I don't have that much experience in PS so I went with C#. ;)
    – Andrew
    Aug 31, 2020 at 2:12
  • BTW, I tried your script into a .ps1 file, and added a few lines: # Before calling: Set-ExecutionPolicy RemoteSigned -Scope CurrentUser $file = $args[0] $fileData = Get-Item $file $output = $fileData.DirectoryName + $fileData.BaseName + "fixed" + $fileData.Extension echo "Procesing $($file) into $($output)"
    – Andrew
    Aug 31, 2020 at 2:44
0

If you know where (what byte) the file is supposed to end (before the null bytes start trailing), truncate (set file EOF) after that byte eg.

FSUTIL file seteof <file name> <file size without null trailer>

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .