Filter curl header output in verbose mode

Question

Using curl -v ..., i.e. the verbose mode that prints out the input and output headers. However, this info is not piped and I can not grep-out lines which I dont need.

How can you filter curl header output?

Answer 1

The header output from curl is gets printed to standard error. So you have to use redirection, for example grepping out the Content-Length header:

curl -v google.com 2>&1 | grep -vi content-length

Answer 2

Instead of using curl -v (verbose mode), a better option is to tell curl to output the headers to STDOUT without all of the extra debugging stuff and without printing the response body. This way, you can grep through just the response headers, avoid the need for shell redirection, and know that your filtering will not have to understand and parse out curl’s internal debugging messages (which are probably not considered part of its API).

You can do this by combining the -I (do not print body), -X«METHOD» (override method), and -s (silent, do not show progress bars) parameters.

curl -sIXGET https://www.google.com/ | grep -vi content-length

• -s instructs curl to hide the progress bars. If omitted and curl detects that it is running in a terminal, it will bypass the pipe and directly talk to the terminal to show progress bars which is often unhelpful and clutters up the terminal.
• -I instructs curl to disconnect from the server after receiving the headers instead of waiting for a body. This is useful if you actually, as stated in your question, are interested in grepping through only the HTTP response headers. It also has the side-effect of defaulting the request method to HEAD as if you had specified -XHEAD.
• -X«method» (e.g., -XGET) overrides the method. In many cases, a HEAD request is what you want because the HTTP standard says that HEAD must return the exact same headers as the same GET request. However, when debugging a server or working with one which does not adhere to the standard, it can be useful to instruct curl to make a GET request but disconnect before reading the response body. This way, you can see the actual headers the server would send in response to GET instead of hoping that the server is implemented correctly.