4

I have this script:

for /f "delims=" %%a in (data_A.txt) DO (
    set "flag="
    for /f "delims=" %%b in (data_B.txt) do (
        if "%%a"=="%%b" CALL :flagit
    )
    if not defined flag >>notmatch.txt echo %%a
)

:flagit
set flag=1
goto :eof

I am comparing each line between two text files to determine entries that don't match. The files contain checksum hash values and relative path information.

It works, but it just is quite slow. A file with 100,000 entries takes over two and a half hours, and I have to do it twice because I have to compare both ways to capture any entries on A that might not be on B but also B that might not be on A. The data isn't necessarily in any particular sort order either.

I'm looking to capture:

  • Files on set A not on set B
  • Files on set B not on set A
  • Files where checksums don't match

The first two I'm comfortable handling (files not on A/B) because once I get the the "not matched" entries, they should be few and comparing if the relative path exists on both should be straight forward (I hope). But that last one is the real time suck.

An example of the data:

dd2da0dcb5a54989dd4d2312013ddb12345c0593ed59a6d307461d925d57226d89d24c2e5a95c0d4082b14118cb8766d89ae69e40c4dac1ab5bd718bd7c58d9a \Personal\Pictures\Camera 2019\2019-02-17 15.02.34.jpg 
509ebfd1e2c180ccd6bd679204b7c255f3c7abcdefg7660e219fa9eb58658d96a3ef8cec179221acb78be81f8dd78bd3a8b1a3cdaef0cd691725d3402a495b0b \Personal\Pictures\Camera 2019\2019-02-17 15.03.59.jpg 
a3180dce7675aeb161f8fe25fcbd39ff2678faf2326d3e2a39fchfasff90a714134bdd22f91103026c494e6ffcfd62d5cb3d46992de9dfff71b49f9a734c0ab9 \Personal\Pictures\Camera 2019\2019-02-17 17.11.41.jpg 
b5262c6ce5c4425a4ed737a7a8fdbc040c68003785d67177a25c86d9fb531ce42f74648783aed4bbb3aff7304b00d44b14eaa2a6c728b8802cafd22059570212 \Personal\Pictures\Camera 2019\2019-02-18 18.06.14.jpg 
da7e1eb7ec147628a59e702c55159bc32d66f3c540dfb4be436f136137af913a7139640701eba84f34796da4f35c9fasdffae35542f56b1dccf009d1cec30d20 \Personal\Pictures\Camera 2019\2019-02-22 06.18.15.jpg 
72c99a6f4394b4f65d4b66b00071de1d40cb717f525863875c36b2bc79dd0a8491ee8854b8b4437bfcfe4aa8379861aa43a7850dfac144d5db5b2c6b75dcf292 \Personal\Pictures\Camera 2019\2019-02-22 06.18.23.jpg 
4a8a39e68379b2c671d83935b13dc82dd60d5e8b36a32a8677698a9306876zcvaffaaa4af292d53a8f52df4ee1c7bc701068064f4d28009566e8825abf2ab077 \Personal\Pictures\Camera 2019\2019-02-22 06.20.10.jpg 
074103664be0c91664bd4e2e51d0e051c9cf8f27c26511d3a691d0asdfadfa134234808a16bf0679a8500910b09cf24d9e9c88788b4a749a81ec2d15f78cacfd \Personal\Pictures\Camera 2019\2019-02-22 06.27.14.jpg 
28dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701dd7 \Personal\Pictures\Camera 2019\2019-02-23 11.54.34.jpg 

The way the for loops work it scans the other file until it finds a non match, outputs it to the log file, then goes to the next file. It seems like there should be a way that once a file has been matched or not matched to eliminate it from the search.

EDIT - OK, I'm messing with PowerShell a bit. The Compare-Object works well, but just have to figure out how to show only files that are different that exist on both A and B, and log to separate files those ONLY on A and those ONLY on B...

3
  • In PowerShell /Update:

To do the same in PowerShell, you can use the @JoseZ method mentioned in this answer, and with minimal editing, you get the strings that differ in each file saved in the NoMatch.txt file:

  • To obtain content that differs in both, that is, to obtain the lines/content of data_A.txt which is not in data_B.txt and also of data_B.txt which is not in data_A.txt
Set-Location -Path "D:\Your\Folder"; Clear-Content -path ".\NoMatch.txt"; $filebefore=".\data_A.txt"; $file_after=".\data_B.txt"

### Compare-Object way
$array = Compare-Object $(Get-Content $filebefore) $(Get-Content $file_after)
$array | where {$_.SideIndicator -eq "<="} | Format-Table -Property InputObject -AutoSize -HideTableHeaders

### -NotIn operator way
$(Get-Content $filebefore) | Where-Object {$_ -notIn $(Get-Content $file_after)} | Out-File ".\NoMatch.txt" -Append
$(Get-Content $file_after) | Where-Object {$_ -notIn $(Get-Content $filebefore)} | Out-File ".\NoMatch.txt" -Append
  • Output is the last line of each file saved in NoMatch.txt
>Get-Content ".\NoMatch.txt"

18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
28dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701


  • To get only the lines/content from data_B.txt that are not in data_A.txt
Set-Location -Path "D:\Your\Folder"; Clear-Content -path ".\NoMatch.txt"; $filebefore=".\data_A.txt"; $file_after=".\data_B.txt"

### Compare-Object way
$array = Compare-Object $(Get-Content $filebefore) $(Get-Content $file_after)
$array | where {$_.SideIndicator -eq "<="} | Format-Table -Property InputObject -AutoSize -HideTableHeaders

### -NotIn operator way
$(Get-Content $filebefore) | Where-Object {$_ -notIn $(Get-Content $file_after)} | Out-File ".\NoMatch.txt" -Append
  • Output is the last line in file data_B.txt saved in file NoMatch.txt

>Get-Content ".\NoMatch.txt"

18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701


  • In bat/cmd file:

For shorter strings, you can try using Findstr /vixg:data_A.txt data_B.txt

  /I         Specifies that the search is not to be case-sensitive.
  /X         Prints lines that match exactly.

  /V         Prints only lines that do not contain a match.

  /G:file    Gets search strings from the specified file(/ stands for console).

Obs.: In order for Findstr to make this comparison/find the same/different strings, there is a limitation of a maximum limit of 250 characters in total length:

(String_A).Length + (String_B).Length <= 250 characters

- Which is why I reduced your strings to a maximum of 125 characters in the example below:

Only on the last line, and on the first character, in the file data_A.txt will not match the file data_B.txt and will be saved in the file NoMatch.txt


  • To get only the lines/content from data_B.txt that are not in data_A.txt
@echo off 

cd /d "%~dp0" && if not exist data_B.txt call :^) 

cd. >nul >.\NoMatch.txt && for /f tokens^=* %%i in (
'findstr /vixg:data_A.txt data_B.txt')do >>.\NoMatch.txt echo=%%~i

goto :EOF

:^)
>data_A.txt ^
    (
     echo=dd2da0dcb5a54989dd4d2312013ddb12345c0593ed59a6d307461d925d57226d89d24c2e5a95c0d4082b14118cb8766d89ae69e40c4dac1ab5bd718bd7c58
     echo=509ebfd1e2c180ccd6bd679204b7c255f3c7abcdefg7660e219fa9eb58658d96a3ef8cec179221acb78be81f8dd78bd3a8b1a3cdaef0cd691725d3402a495
     echo=a3180dce7675aeb161f8fe25fcbd39ff2678faf2326d3e2a39fchfasff90a714134bdd22f91103026c494e6ffcfd62d5cb3d46992de9dfff71b49f9a734c0
     echo=b5262c6ce5c4425a4ed737a7a8fdbc040c68003785d67177a25c86d9fb531ce42f74648783aed4bbb3aff7304b00d44b14eaa2a6c728b8802cafd22059570
     echo=da7e1eb7ec147628a59e702c55159bc32d66f3c540dfb4be436f136137af913a7139640701eba84f34796da4f35c9fasdffae35542f56b1dccf009d1cec30
     echo=72c99a6f4394b4f65d4b66b00071de1d40cb717f525863875c36b2bc79dd0a8491ee8854b8b4437bfcfe4aa8379861aa43a7850dfac144d5db5b2c6b75dcf
     echo=4a8a39e68379b2c671d83935b13dc82dd60d5e8b36a32a8677698a9306876zcvaffaaa4af292d53a8f52df4ee1c7bc701068064f4d28009566e8825abf2ab
     echo=074103664be0c91664bd4e2e51d0e051c9cf8f27c26511d3a691d0asdfadfa134234808a16bf0679a8500910b09cf24d9e9c88788b4a749a81ec2d15f78ca
     echo=28dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
    ) && (
>data_B.txt ^
    (
     echo=dd2da0dcb5a54989dd4d2312013ddb12345c0593ed59a6d307461d925d57226d89d24c2e5a95c0d4082b14118cb8766d89ae69e40c4dac1ab5bd718bd7c58
     echo=509ebfd1e2c180ccd6bd679204b7c255f3c7abcdefg7660e219fa9eb58658d96a3ef8cec179221acb78be81f8dd78bd3a8b1a3cdaef0cd691725d3402a495
     echo=a3180dce7675aeb161f8fe25fcbd39ff2678faf2326d3e2a39fchfasff90a714134bdd22f91103026c494e6ffcfd62d5cb3d46992de9dfff71b49f9a734c0
     echo=b5262c6ce5c4425a4ed737a7a8fdbc040c68003785d67177a25c86d9fb531ce42f74648783aed4bbb3aff7304b00d44b14eaa2a6c728b8802cafd22059570
     echo=da7e1eb7ec147628a59e702c55159bc32d66f3c540dfb4be436f136137af913a7139640701eba84f34796da4f35c9fasdffae35542f56b1dccf009d1cec30
     echo=72c99a6f4394b4f65d4b66b00071de1d40cb717f525863875c36b2bc79dd0a8491ee8854b8b4437bfcfe4aa8379861aa43a7850dfac144d5db5b2c6b75dcf
     echo=4a8a39e68379b2c671d83935b13dc82dd60d5e8b36a32a8677698a9306876zcvaffaaa4af292d53a8f52df4ee1c7bc701068064f4d28009566e8825abf2ab
     echo=074103664be0c91664bd4e2e51d0e051c9cf8f27c26511d3a691d0asdfadfa134234808a16bf0679a8500910b09cf24d9e9c88788b4a749a81ec2d15f78ca
     echo=18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
    ) ) && exit /b 
  • Output is the last line in file data_B.txt saved in file .\NoMatch.txt
18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701


  • To obtain content that differs in both, that is, to obtain the lines/content of data_A.txt which is not in data_B.txt and also of data_B.txt which is not in data_A.txt
@echo off 

cd /d "%~dp0" && if not exist data_B.txt call :^) 

cd. >nul >notmatch.txt
for /f tokens^=* %%i in ('findstr /vixg:data_A.txt data_B.txt')do >>notmatch.txt echo=%%~i
for /f tokens^=* %%i in ('findstr /vixg:data_B.txt data_A.txt')do >>notmatch.txt echo=%%~i

goto :EOF

:^)
>data_A.txt ^
    (
     echo=dd2da0dcb5a54989dd4d2312013ddb12345c0593ed59a6d307461d925d57226d89d24c2e5a95c0d4082b14118cb8766d89ae69e40c4dac1ab5bd718bd7c58
     echo=509ebfd1e2c180ccd6bd679204b7c255f3c7abcdefg7660e219fa9eb58658d96a3ef8cec179221acb78be81f8dd78bd3a8b1a3cdaef0cd691725d3402a495
     echo=a3180dce7675aeb161f8fe25fcbd39ff2678faf2326d3e2a39fchfasff90a714134bdd22f91103026c494e6ffcfd62d5cb3d46992de9dfff71b49f9a734c0
     echo=b5262c6ce5c4425a4ed737a7a8fdbc040c68003785d67177a25c86d9fb531ce42f74648783aed4bbb3aff7304b00d44b14eaa2a6c728b8802cafd22059570
     echo=da7e1eb7ec147628a59e702c55159bc32d66f3c540dfb4be436f136137af913a7139640701eba84f34796da4f35c9fasdffae35542f56b1dccf009d1cec30
     echo=72c99a6f4394b4f65d4b66b00071de1d40cb717f525863875c36b2bc79dd0a8491ee8854b8b4437bfcfe4aa8379861aa43a7850dfac144d5db5b2c6b75dcf
     echo=4a8a39e68379b2c671d83935b13dc82dd60d5e8b36a32a8677698a9306876zcvaffaaa4af292d53a8f52df4ee1c7bc701068064f4d28009566e8825abf2ab
     echo=074103664be0c91664bd4e2e51d0e051c9cf8f27c26511d3a691d0asdfadfa134234808a16bf0679a8500910b09cf24d9e9c88788b4a749a81ec2d15f78ca
     echo=28dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
    ) && (
>data_B.txt ^
    (
     echo=dd2da0dcb5a54989dd4d2312013ddb12345c0593ed59a6d307461d925d57226d89d24c2e5a95c0d4082b14118cb8766d89ae69e40c4dac1ab5bd718bd7c58
     echo=509ebfd1e2c180ccd6bd679204b7c255f3c7abcdefg7660e219fa9eb58658d96a3ef8cec179221acb78be81f8dd78bd3a8b1a3cdaef0cd691725d3402a495
     echo=a3180dce7675aeb161f8fe25fcbd39ff2678faf2326d3e2a39fchfasff90a714134bdd22f91103026c494e6ffcfd62d5cb3d46992de9dfff71b49f9a734c0
     echo=b5262c6ce5c4425a4ed737a7a8fdbc040c68003785d67177a25c86d9fb531ce42f74648783aed4bbb3aff7304b00d44b14eaa2a6c728b8802cafd22059570
     echo=da7e1eb7ec147628a59e702c55159bc32d66f3c540dfb4be436f136137af913a7139640701eba84f34796da4f35c9fasdffae35542f56b1dccf009d1cec30
     echo=72c99a6f4394b4f65d4b66b00071de1d40cb717f525863875c36b2bc79dd0a8491ee8854b8b4437bfcfe4aa8379861aa43a7850dfac144d5db5b2c6b75dcf
     echo=4a8a39e68379b2c671d83935b13dc82dd60d5e8b36a32a8677698a9306876zcvaffaaa4af292d53a8f52df4ee1c7bc701068064f4d28009566e8825abf2ab
     echo=074103664be0c91664bd4e2e51d0e051c9cf8f27c26511d3a691d0asdfadfa134234808a16bf0679a8500910b09cf24d9e9c88788b4a749a81ec2d15f78ca
     echo=18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
    ) ) && exit /b 
  • Output is the last line of each file saved in NoMatch.txt
18dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701
28dc03a7722b0781caa4dfasdf664w666777068c79456941a159ffefa1d9c34fed83b98858394c1aa471396a0b1a448d8dd89e361c564e6b27e451b2dd701

| improve this answer | |
  • 1
    Thanks very much for your prompt response! Just FYI, I threw in some random hashes in my post, lol only to show representation. Anyhow, your script works, and one CPU core is pegged at 100%, but unfortunately it didn't seem to save much time (using my ~10k line test set). I just ran a basic Compare-Object 'data_A.txt' 'data_B.txt' in Powershell (I don't know much about it) and it spit out a difference in like five seconds... I guess I better learn to code in Powershell... – HTWingNut May 31 at 4:58
  • Thank you! You have gone above and beyond. I appreciate your effort. This looks like it should work! – HTWingNut May 31 at 17:36
  • 1
    @HTWingNut This is good question, I learned some things with it, thank you too! – It Wasn't Me May 31 at 17:38
  • One more question, if you don't mind. If I only wanted to compare FILE NAMES instead of the full line? I know in batch, you can set delims and split it that way, could you possibly point me in the right direction for powershell? I know you can use .split and set it up into an array, but not sure how to integrate just that path portion to compare... – HTWingNut May 31 at 18:32
  • 1
    @HMy English makes me answer 4x the same question sometimes, because every time I read the question, I understand it differently ;-), Your doubts it is becoming complicated for me to understand. Why not ask another question, where you will have more space to also explain what you want ... it also allows other OPs to answer better than me, see, I am not denying myself to answer you, the case is I not get understand your achieve target... Sorry.. – It Wasn't Me May 31 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.