Windows command line ImageMagick percent escapes

Question

I'm trying to use ImageMagick from the command line in Windows 7 to convert many images at once. I would like to use the -set option interface, but I'm getting unexpected results. A short example of some of the weirdness I'm seeing is this:

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world "%[filename:foo]_%[filename:bar].png"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.125u 0:00.135

Notice that the underscore character is mysteriously deleted from the output filename. I can use a caret (^) to escape the second percent sign in order to retain the underscore:

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world "%[filename:foo]_^%[filename:bar].png"
example.jpg=>hello_world.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.125u 0:00.126

What's going on here? Why does inserting the caret cause the underscore to be retained? Is the caret (^) even the correct escaper to be trying? Is this behavior due to command shell environment variable expansion or does the weirdness result from the way ImageMagick handles the percent escapes? Either way, are there general principles regarding use of percent symbols in command line args that can be followed to cause predictable behavior?

What I actually want to do is a version of the above command with many more properties set, that can handle very filenames with special characters like spaces and percent symbols, but it seems foolish to try that if I can't understand this much simpler example. Ultimately, I'll need to execute the command from a .NET program.

I've looked at http://www.robvanderwoude.com/escapechars.php and Escaping %’s in file-/folder-names at the command-line but could not figure out how to apply the advice to my problem.

Update

Adding some more examples to answer questions in comments from @dbenham and @Synetech.

Moving the underscore from before the second percent symbol, as in the latter example above, to before the underscore, produces a file named hello^world.png:

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world "%[filename:foo]^_%[filename:bar].png"
example.jpg=>hello^world.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.141u 0:00.132

Caret-escaping the surrounding quotation marks always always results in helloworld.png, regardless of where carets are placed within:

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world ^"^%[filename:foo]_^%[filename:bar].png^"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.109u 0:00.115

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world ^"%[filename:foo]_^%[filename:bar].png^"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.141u 0:00.143

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world ^"^%[filename:foo]_%[filename:bar].png^"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.125u 0:00.126

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world ^"%[filename:foo]_%[filename:bar].png^"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.125u 0:00.131

C:\>convert example.jpg -verbose -set filename:foo hello -set filename:bar world ^"%[filename:foo]^_%[filename:bar].png^"
example.jpg=>helloworld.png JPEG 352x264 352x264+0+0 8-bit DirectClass 131KB 0.125u 0:00.111

Answer 1

I can't quite explain the weird behavior you are seeing. It almost seems like it is a result of cmd.exe percent processing, but it does not quite match the cmd.exe behavior as I understand it.

Escaping percents within a batch file is easy - just double them up. But that does not work from the command line. Technically, there is no way to escape a percent on the Windows cmd.exe command line.

The command line can expand percents in one of two ways:

1) Expand Environment variables.

If text between two percents matches the name of an environment variable, then the percent construct is replaced by the value.

C:\test>set test=hello

C:\test>echo %test%
hello

If the text between the percents does not match an environment variable name, then the full original text is retained.

C:\test>echo %notDefined%
%notDefined%

Microsoft's own documentation claims you can escape the percent by putting a caret in front.

C:\test>echo ^%test^%
%test%

But that is not really escaping the percent. Instead, it is looking for a variable named test^, and not finding it. So the original text is undisturbed. Then the carets are removed during the normal escape processing. This can be proven by defining a variable with the caret in the name.

C:\test>set "test^=goodbye"

C:\test>echo ^%test^%
goodbye

The caret before the first percent does nothing. You can place a caret anywhere between the percents and it will prevent expansion of the variable as long as no variable exists with the caret in the name.

C:\test>echo %te^st%
%test%

If the text is quoted, then the caret will not be dropped unless the quotes are also escaped.

C:\test>echo "%te^st%"
"%te^st%"

C:\test>echo ^"te^st%^"
"%test%"

The situation is a bit more complicated when there are colons. The colon is used by variable expansion for substitution and substring operations. The variable name is the text between the first percent and the colon.

C:\test>echo %test:el=a%
halo

C:\test>echo %test:~1,2%
el

If the text between the first percent and the colon does not match a variable name, or if the text between the colon and the last percent is not a valid substitution or substring construct, then the original text is preserved without substitution.

C:\test>echo %test:1,2%
%test:1,2%

This last example most resembles your situation. The text between your first percent and the colon does not match a variable, and the text between the colon and the last percent is not a substitution or substring construct, so the entire construct should be preserved, including the underscore. For this reason, I don't see how cmd.exe could be stripping your underscore. But the fact that adding a caret before the second percent preserves the underscore has me suspicious. I'd really like to know what happens if you move the caret before the underscore instead.

2) Expand FOR variables

Not an issue in your case, but percents can also cause issues if a FOR loop is in effect. There is no way to protect the percent by using a caret in this case, because the caret is stripped prior to the FOR variable expansion. In this example, I want the output to read %A=1, but it does not work.

C:\test>for %A in (1 2) do @echo ^%^A=%A
1=1
2=2

The most convenient way to protect the percent is to introduce another FOR variable

C:\test>for %C in (%) do @for %A in (1 2) do @echo %CA=%A
%A=1
%A=2

It works just as well to put the A in another variable

C:\test>for %C in (A) do @for %A in (1 2) do @echo %%C=%A
%A=1
%A=2

---

Update

One good way to test where the underscore is getting stripped is to put ECHO before your command. If the original command is preserved with the underscore, then you know that the convert utility must be the culprit.

C:\>echo convert example.jpg -verbose -set filename:foo hello -set filename:bar world "%[filename:foo]_%[filename:bar].png"

The underscore is preserved on my machine :-)

---

If you have further interest in how the command parser works, then see https://stackoverflow.com/a/4095133/1012053. That answer mostly concerns itself with batch parsing, but the rules are very similar, and a section at the end briefly describes the major differences.