1

I'm looking for any method that would allow to : read X bytes at offset Y from each input file in a folder, then add it at the end of a single output file. For instance, there's a folder with many fragments of video files, the idea would be to read for instance 16 bytes at offset 1024 of the first file, write it to a new file, then add a line break, then process the whole folder like that. That file would then be used as a list of keywords for a simultaneous search in WinHex, so as to hopefully solve that issue.

I've done something similar once with ddrescue. Here I need a method working on Windows. I found out that ddrescue can work on Windows as a Cygwin port, but I can't get it to work with individual files. Then there's a dd port for Windows, it can read a chunk of data from an input file but I couldn't find a way to get it to write that chunk at the end of the output file. I know two command line tools called dsfo and dsfi (included in dsfok), which can each do one half of the intended task (dsfo can read any defined chunk of data and extract it as a new file, it can't write it into an existing file ; dsfi can write an input file to any location within an output file, but it doesn't allow to define a specific part of the input file), I tried to make them work together in the same script but it failed.

Could this be done with PowerShell ? How ?


EDIT (as per Keith Miller's suggestion) :

This command works as intended :

foreach ($file in gci *.mts, *.vob) {
$16Bytes = [System.Text.Encoding]::Default.GetString([System.IO.File]::ReadAllBytes("$file"), 1024, 16)
Add-Content -path "G:\PowerShell search terms MTS-VOB.txt" -value $16Bytes
}

It reads 16 bytes at offset 1024 from each input file and writes it to the TXT file, and it automatically adds a line break (0D 0A) after each ASCII encoded string (if that's the correct terminology), so there's no need to add a specific command for that.
But it is too slow, as if each file were being read in its entirety (it's much quicker if I run the command again with different offset parameters, probably because the files were copied to RAM during the first run). Is there a way to speed up the process, to ensure that only the relevant portion of each file is actually parsed ?

Another issue is that WinHex seems to import / export lists of seach terms as “Unicode BOM” only, whereas this PS script produces an ANSI file. If importing the ANSI file directly, the list stays empty. I can copy the output from that file to WinHex, but it gets truncated if there's a “null” (00) character. It also gets truncated if I first convert the ANSI file to “Unicode BOM” with Notepad2 and then import it in WinHex's search window. But there's an option called “direct byte-wise translation for GREP”, which allows to perform a search of any sequence of bytes regardless of the code page.

Excerpt from WinHex Help document :

You can search the same search terms simultaneously in up to 6 code pages. The default code page, that is active in your Windows system, is marked with an asterisk and initially preselected. E.g. on computers in the US and in Western Europe, the usual default code page is 1252 ANSI Latin I. The code pages named "ANSI" are used in Microsoft Windows. "MAC" indicates an Apple Macintosh code page. "OEM" indicates a code page used in MS-DOS and Windows command prompts. If a search term cannot be converted to the specified code page because of characters unknown in that code page, a warning is issued. Code page independent GREP searches for exact byte values are possible when searching in a "non" code page called "Direct byte-wise translation for GREP", which translates byte values without any mapping for certain code pages or case matching. X-Ways Forensics also allows to search in both little-endian and big-endian UTF-16, and in any regional Windows code page plus UTF16 with the MS Outlook cipher (compressible encryption) applied.

If, in WinHex, I copy a small block as “GREP Hex”, it gets pasted in this form :

\xFC\x70\x28\x4C\x00\xB5\x47\x00\x52\x30\x96\xA3\x17\x51\x4A\x44

And doing a search with a list formatted like this, with “GREP syntax” activated, and searching as “direct byte-wise translation for GREP”, seems to work reliably, even with “00” bytes as the example above. So, based on an article linked in a comment below, I tried this :

foreach ($file in gci *.mts, *.vob) {
$16Bytes = [System.Text.Encoding]::Default.GetString([System.IO.File]::ReadAllBytes("$file"), 100000, 32)
[System.BitConverter]::ToString($16Bytes)
Add-Content -path "G:\HGST_recherche_fragments_PowerShell_test_hex.txt" -value $16Bytes
}

But got multiple errors like this :

Impossible de convertir l'argument « 0 » (valeur « Qe▬?'QhNèJ↕?÷ß}÷ûï« → * l ?♠ @ ») de « ToString » en type « System.Byte[] » : « Impossible de con
vertir la valeur « Qe▬?'QhNèJ↕?÷ß}÷ûï« → * l ?♠ @ » en type « System.Byte[] ». Erreur : « Impossible de convertir la valeur « Qe▬?'QhNèJ↕?÷ß}÷ûï« →
* l ?♠ @ » en type « System.Byte ». Erreur : « Le format de la chaîne d'entrée est incorrect. » » »
Au niveau de ligne : 3 Caractère : 32
+ [System.BitConverter]::ToString <<<< ($16Bytes)
    + CategoryInfo          : NotSpecified: (:) [], MethodException
    + FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument

Obviously there's something wrong, but it's getting closer to a method that actually works... So how can I get PS to read X bytes at offset Y and write them as a sequence of hexadecimal values ?


Now, an extra step to make this as quick and painless as possible would be to perform automated checksum comparisons, in order to avoid doing manual comparisons once I have a list of potentially matching files. I found out that WinHex can do a “logical search” within a whole volume, meaning that for each search hit it can report the absolute offset (relative to the start of the partition) as well as the file offset (where the searched string was found within a file identified through that partition's filesystem, even if it is fragmented or NTFS-compressed). So, once I have the list of search hits, with the path / names of the files, what I would like is to :
– compute the MD5 checksum of file “A” (the one from which the search term was copied) ;
– compute the MD5 checksum of a block in file “B” (the one where a hit for the search term was found) which supposedly coincides with file “A” ;
– print the result to a report file ;
– if the MD5 checksums match, it means that file “A” is totally and exactly included in file “B”, and can therefore be deleted ; if not, either it's a false positive (the search term wasn't specific enough, or the original file was fragmented so the recovered file may contain foreign data), in which case it has to be manually checked.
To do that I would have to define, for each pair of files, in a loop, a block within file B starting at [offset where the hit was found in file B] - [offset where the search term was copied from file A], and ending at [starting offset] + [size of file A]. Then calculate the MD5 checksum of this block, the MD5 of file A, and report if both values match or not.
Does it seem like this can be done with a simple PowerShell script ?

21
  • To get "X bytes at Y offset" the following could be used: $16Bytes = [System.Text.Encoding]::Default.GetString([System.IO.File]::ReadAllBytes("<FilePath>"),1023, 1039) by changing <filepath> to what is required, I have set this to 16 bytes from offset of1024 as you have specified this as an example. the rest of what you describe should be pretty trivial reading up on powershell loops and reading/writing file contents.
    – CraftyB
    May 1, 2020 at 11:38
  • Thanks for this. What does the "System.Text.Encoding" part mean ? I want it to read and write the bytes as-is, with no text encoding, as the binary content of those strings has to match exactly for WinHex to find hits where there should be. Also, where would be a good place to read about reading/writing file contents in Powershell ? Most websites related to PS assume a prior knowledge which I don't have, or are organized in such a way that each small part of a task is treated in a separate page and one has to know how it's called before reading about it, which is a bit overwhelming.
    – GabrielB
    May 1, 2020 at 19:08
  • Tried this but got immediately stuck : cjoint.com/c/JEbtK0zpSfA (and with the error messages in french it's even more difficult to search for an explanation / solution as most quality sources are in english).
    – GabrielB
    May 1, 2020 at 19:39
  • "...16 bytes at offset 1024 of the first file, write it to a new file, then add a line break" -- there's some conceptual confusion here. Binary files don't have line breaks --- that's a concept specific to text files. May 3, 2020 at 5:18

0

You must log in to answer this question.

Browse other questions tagged .