19

I have endpoint which returns JSON (response body). I need get by curl the response body and process it (for example using jq). It works:

response=$(curl -s https://swapi.dev/api/people/1?format=json)
name=$(echo $response tmpFile | jq '.name') # irrelevant command, but I need here response body
echo "name:"$name

But I also need show the HTTP Code (to show if the request is succeed):

curl -s -w "%{http_code}\n" -o /dev/null https://swapi.dev/api/people/1?format=json

How get the response body to variable and show HTTP code at the same time (one request)?


I find out solution witch temporary file:

touch tmpFile
curl -s -w "%{http_code}\n" -o tmpFile https://swapi.dev/api/people/1?format=json
name=$(cat tmpFile | jq '.name') # irrelevant command, but I need here only body response
echo "name: "$name
rm tmpFile

How to do without creating file?

I try with named pipe (but it still need to creating file on disk...):

mkfifo tmpFifo
curl -s -w "%{http_code}\n" -o tmpFifo https://swapi.dev/api/people/1?format=json
name=$(cat tmpFifo | jq '.name') # irrelevant command, but I need here only body response
echo "name: "$name
rm tmpFifo

But the named pipe is not removing.

There is solution without creating any file, for example only witch variables or streams?

5 Answers 5

17

It looks like the content of the response is a single line. You could use two read calls to read two lines:

curl -s -w "\n%{http_code}" 'https://swapi.dev/api/people/1?format=json' | {
    read body
    read code
    echo $code
    jq .name <<< "$body"
}
5
  • I can not get it to work. could you please add some explanations? Apr 2, 2021 at 13:06
  • @janos what is this braces block called in Bash spec? Sep 19, 2023 at 10:37
  • @FlorianBachmann It's called a "group command".
    – janos
    Sep 19, 2023 at 15:23
  • 1
    @PeymanMohamadpour if you give more precise details about what is not working, somebody might be able to help out.
    – janos
    Sep 19, 2023 at 15:24
  • @janos Thank you very much :) Sep 21, 2023 at 6:26
7

Solution with return body and HTTP code at last line:

response=$(curl -s -w "\n%{http_code}" https://swapi.co/api/people/1/?format=json)
response=(${response[@]}) # convert to array
code=${response[-1]} # get last element (last line)
body=${response[@]::${#response[@]}-1} # get all elements except last
name=$(echo $body | jq '.name')
echo $code
echo "name: "$name

But still I would rather do this with two separate variables/streams instead of concatenate response body and HTTP code in one variable.

2
  • I would add IFS=$'\n' to the first line to ensure that the value in the code variable doesn't truncate on its last space character. May 29, 2020 at 14:47
  • I had to add the IFS=$'\n' to the array conversion line, not literally the first line. Also before Bash 4.2 (I think) I had to use code=${response[*]: -1} to get the last element
    – Sebastian
    May 13, 2022 at 10:12
3

Here's a simple bash two-liner that will return the status code and content, and work with single or multiple line document bodies.

If we were to run the script below, curl would write the document body followed by the HTTP status code to stdout.

URL="https://superuser.com/questions/1320674/get-response-body-and-show-http-code-by-curl" 
curl -s -w "%{http_code}\n" $URL

If we reorder these so the HTTP status code is output first, followed by the document body, we can utilize a feature of the bash builtin read to stuff all the body into a single shell variable.

man builtins

...
read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
      One line is read from the standard input, or from the file descriptor fd supplied as an argument to the -u option, split into words as  described  in  bash(1)  under  Word
      Splitting,  and  the  first word is assigned to the first name, the second word to the second name, and so on.  If there are more words than names, the remaining words and
      their intervening delimiters are assigned to the last name.  If there are fewer words read from the input stream than names, the remaining names are assigned empty values.
      The  characters  in IFS are used to split the line into words using the same rules the shell uses for expansion (described in bash(1) under Word Splitting).  The backslash
      character (\) may be used to remove any special meaning for the next character read and for line continuation.  Options, if supplied, have the following meanings:
...

Reordering the curl output can be achieved by writing the output to a file. curl will send the information requested by the -w, --write-out option to stdout first. The file the body is being written to is substituted for a sub process using process substitution >(), with the body sent to this sub process's stdin. Within this sub process, the read command is used to grab the document body from stdin by using the -u0 option to specify the input stream. IFS= disables word splitting so whitespace is passed through unchanged. The body is saved to a variable named stdin, named purely for ease of reference. printf is used to write the value of this varible to stdout.

IFS=$'\n' read -r -d '' http_status body < <(curl -s -w "%{http_code}\n" -o \
  >(IFS= read -r -d '' -u0 stdin; printf "%s"  "$stdin") $URL)

Now the data is correctly ordered inside stdout, we use a subsequent process substitution to send it to another read command in the parent shell process that will place it into shell variables. IFS=$'\n' sets word splitting to only occur on new lines. The first word is the HTTP status code so this is placed into the http_status variable. All subsequent words are placed into the body variable. The -r option to read is used to disable backslash escape processing of the document body data. The -d '' option sets the delimiter to null to ensure read consumes everything.

echo $http_status
200
echo $body | head -8                                                                                                                                          
<!DOCTYPE html>


    <html itemscope itemtype="https://schema.org/QAPage" class="html__responsive " lang="en">

    <head>

        <title>bash - Get response body and show HTTP code by curl - Super User</title>
1

Tested on bash4 on Ubuntu 18.04LTS:

IFS=$'\n' read -d "" body code  < <(curl -s -w "\n%{http_code}\n" "${uri}")

This forces the output of curl to be on two lines. The change to IFS makes the linefeed the only field separator and the -d "" forces read to read beyond the line feed, treating the two lines as though they are one. Not the most elegant solution, but a one-liner.

1

Even though we are appending http_code as a newline the http_code is showing up as the last line. I think this is because as the man page describes: "-w Makes curl display information on stdout after a completed transfer".

Solution using tail + sed to grab the correct variables with text manipulation:

URL="https://swapi.dev/api/people/1?format=json";
response="$(curl -s -w "\n%{http_code}" $URL)";

http_body="$(echo "$response" | sed '$d' | jq .name)";
http_status_code="$(echo "$response" | tail -n 1)";

echo "$http_body";
echo "$http_status_code";

Command breakdown:
sed '$d' - delete the last line and return the rest, this is useful for getting the html body.
tail -n 1 - give all except the last line, this is useful for getting the status code.

2
  • 1
    Instead of sed '$d' we can head -n -1 for getting all except last line. It looks similar to tail -n 1 for getting last line.
    – mkczyk
    Sep 27, 2023 at 8:49
  • 1
    good point, but I think negative values were not supported in bash 3.2 which I was using to solve this problem. Sep 27, 2023 at 15:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .