Handling JSON with jq
Handling JSON with the jq command line tool.
Overview
jq is a command tool for handling JSON. You will more likely come across it in a Linux environment. In Debian based distros you would install it like this for example:
1
2sudo apt update
3sudo apt install -y jq
However, checkout the jq GitHub Releases and you will see that it is supported on Windows, Mac and Linux. In what follows we will be using jq in an Ubuntu 20.04 environment.
In this post I am not going to cover learning all aspects of jq and JSON as there is already a jq Tutorial and plenty of other online content in terms of using jq and JSON. What I will cover is a couple of online sites that can help you and also a couple of recent use cases where I have used it.
Also bear in mind that when dealing with JSON it might be better considering a programming or scripting language that has built-in support for JSON. For example, Python or PowerShell might make your goal easier to achieve.
In my examples below I was working with an existing bash script that I wanted to extend and also make more DRY by separating script from config, with the result being that the same script did not need to be copied to multiple repositories.
Processing JSON data from an Azure App Service SCM site
It doesn't really matter what is outputting the JSON data but in my first use case it is an Azure App Service. My requirement was to check some content of the docker logs of an Azure App Service SCM site.
If I wanted to get the available docker logs from the seventh instance of my sensitive website agent instances I would do something like the below steps.
Firstly, get the credentials to (with an identity that has the appropriate permissions) be able to access the logs from the Kudo REST API from the command line:
1
2# use the Azure CLI to get the App Service publishing credentials
3req=$(az webapp deployment list-publishing-credentials --name "secret-agent-007" --resource-group "rg-bonding" --output json)
4
5username=$(echo "$req" | jq -r '.publishingUserName')
6password=$(echo "$req" | jq -r '.publishingPassword')
7app_cred="$username:$password"
Note in the above we are using jq to return the values of a couple of fields from the JSON output from the az command (there are other ways to achieve this by using a query with the az CLI but as our topic is jq...)
I can then get the available docker logs like this:
1
2curl -s -X GET https://secret-agent-007.scm.azurewebsites.net/api/logs/docker -u "$app_cred"
Which would produce an output something like this:
1
2[{"machineName":"ln0sdlwk000198_default","lastUpdated":"2022-09-01T10:15:42.0162231Z","size":67867,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_default_docker.log","path":"/home/LogFiles/2022_09_01_ln0sdlwk000198_default_docker.log"},{"machineName":"ln0sdlwk000198","lastUpdated":"2022-09-01T10:14:11.9543359Z","size":12466,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_docker.log","path":"/home/LogFiles/2022_09_01_ln0sdlwk000198_docker.log"},{"machineName":"ln0sdlwk000198_easyauth","lastUpdated":"2022-09-01T09:31:56.3263685Z","size":1010,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_easyauth_docker.log","path":"/home/LogFiles/2022_09_01_ln0sdlwk000198_easyauth_docker.log"},{"machineName":"ln0sdlwk000198_msi","lastUpdated":"2022-09-01T10:15:42.0318298Z","size":6353,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_msi_docker.log","path":"/home/LogFiles/2022_09_01_ln0sdlwk000198_msi_docker.log"},
3{"machineName":"ln1sdlwk0001G3_default","lastUpdated":"2022-08-27T06:12:17.1310669Z","size":0,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_default_docker.log","path":"/home/LogFiles/2022_08_27_ln1sdlwk0001G3_default_docker.log"},{"machineName":"ln1sdlwk0001G3","lastUpdated":"2022-08-27T06:06:14.680061Z","size":0,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_docker.log","path":"/home/LogFiles/2022_08_27_ln1sdlwk0001G3_docker.log"},{"machineName":"ln1sdlwk0001G3_easyauth","lastUpdated":"2022-08-27T06:12:17.1466914Z","size":0,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_easyauth_docker.log","path":"/home/LogFiles/2022_08_27_ln1sdlwk0001G3_easyauth_docker.log"},{"machineName":"ln1sdlwk0001G3_msi","lastUpdated":"2022-08-27T06:12:17.1154409Z","size":0,"href":"https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_msi_docker.log","path":"/home/LogFiles/2022_08_27_ln1sdlwk0001G3_msi_docker.log"}]
Now that is a lot of compressed JSON output, which can be tricky to read. So, the first tip I will give you regarding JSON, whether wanting to check that the data is valid or just wanting to make it easier to read, is to use JSONLint.
Note: always be mindful of pasting anything that might be classed as confidential into any online tool.
If we paste in our compressed JSON and click Validate JSON the tool will validate that it is indeed valid and output it in pretty format:
1
2[{
3 "machineName": "ln0sdlwk000198_default",
4 "lastUpdated": "2022-09-01T10:15:42.0162231Z",
5 "size": 67867,
6 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_default_docker.log",
7 "path": "/home/LogFiles/2022_09_01_ln0sdlwk000198_default_docker.log"
8 }, {
9 "machineName": "ln0sdlwk000198",
10 "lastUpdated": "2022-09-01T10:14:11.9543359Z",
11 "size": 12466,
12 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_docker.log",
13 "path": "/home/LogFiles/2022_09_01_ln0sdlwk000198_docker.log"
14 }, {
15 "machineName": "ln0sdlwk000198_easyauth",
16 "lastUpdated": "2022-09-01T09:31:56.3263685Z",
17 "size": 1010,
18 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_easyauth_docker.log",
19 "path": "/home/LogFiles/2022_09_01_ln0sdlwk000198_easyauth_docker.log"
20 }, {
21 "machineName": "ln0sdlwk000198_msi",
22 "lastUpdated": "2022-09-01T10:15:42.0318298Z",
23 "size": 6353,
24 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_msi_docker.log",
25 "path": "/home/LogFiles/2022_09_01_ln0sdlwk000198_msi_docker.log"
26 },
27 {
28 "machineName": "ln1sdlwk0001G3_default",
29 "lastUpdated": "2022-08-27T06:12:17.1310669Z",
30 "size": 0,
31 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_default_docker.log",
32 "path": "/home/LogFiles/2022_08_27_ln1sdlwk0001G3_default_docker.log"
33 }, {
34 "machineName": "ln1sdlwk0001G3",
35 "lastUpdated": "2022-08-27T06:06:14.680061Z",
36 "size": 0,
37 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_docker.log",
38 "path": "/home/LogFiles/2022_08_27_ln1sdlwk0001G3_docker.log"
39 }, {
40 "machineName": "ln1sdlwk0001G3_easyauth",
41 "lastUpdated": "2022-08-27T06:12:17.1466914Z",
42 "size": 0,
43 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_easyauth_docker.log",
44 "path": "/home/LogFiles/2022_08_27_ln1sdlwk0001G3_easyauth_docker.log"
45 }, {
46 "machineName": "ln1sdlwk0001G3_msi",
47 "lastUpdated": "2022-08-27T06:12:17.1154409Z",
48 "size": 0,
49 "href": "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_08_27_ln1sdlwk0001G3_msi_docker.log",
50 "path": "/home/LogFiles/2022_08_27_ln1sdlwk0001G3_msi_docker.log"
51 }
52]
That does indeed make it much easier to read!
Now that I have my list of maps of available logs how do I determine the log file I need to query to get the data I want to check? I need the latest available log file and I know that the log file that contains the data I want doesn't end with msi_docker.log, default_docker.log or easyauth_docker.log.
Here is the second tip, to test a jq query you can use jqplay.
We can take our JSON data from above (compressed or pretty) and paste it into the JSON pane of jqplay. Once we have done that we need to turn the above requirement into a query that will return just the data that we want. We can use the jqplay Filter pane to start experimenting with the syntax we need to build the required query.
To achieve the desired results we can take the following query and paste it into the Filter pane:
1
2. | map(select(.path|test("^.*[^msi|^default|^easyauth]_docker.log$"))) | sort_by(.lastUpdated) | last | .href
We can break that down as follows:
- select the
pathfield - test that
pathdoesn't match any of our unwanted patterns - sort_by the
.lastUpdatedfield - Use last to find the latest
.hrefvalue i.e. the newest docker log file matching our requirements.
This will produce "https://secret-agent-007.scm.azuresecrets.net/api/vfs/LogFiles/2022_09_01_ln0sdlwk000198_docker.log" in the Result pane of jqplay from the JSON and Filter we provided.
You can try taking away some functions of the final syntax to see how it affects the results.
We can now construct our final bash commands to find the latest log and save to a file for further processing using all of the above content.
1
2# set variables
3app_name="secret-agent-007"
4resource_group="rg-bonding"
5
6# use az cli to get credentials
7req=$(az webapp deployment list-publishing-credentials --name "$app_name" --resource-group "$resource_group" --output json)
8username=$(echo "$req" | jq -r '.publishingUserName')
9password=$(echo "$req" | jq -r '.publishingPassword')
10app_cred="$username:$password"
11
12# get the available logs from the Kudu/SCM site
13logs_docker=$(curl -s -X GET https://$app_name.scm.azurewebsites.net/api/logs/docker -u "$app_cred")
14
15# determine the latest log file that we are interested in
16log_latest=$(echo "$logs_docker" | jq -r '. | map(select(.path|test("^.*[^msi|^default|^easyauth]_docker.log$"))) | sort_by(.lastUpdated) | last | .href')
17
18# save to file for latest processing
19curl -s -X GET "$log_latest" -u "$app_cred" -o docker_log
Reading variables from a JSON list of maps
Our second use case for jq is where we mentioned in the overview the separating of a script from configuration, allowing us to use a centralized script controlled by a config.json. This is a slightly silly, contrived and simplified version of the original script and configuration just to show our method.
To follow along with the example below will need cowsay installed on your Linux environment e.g.:
1
2sudo apt install cowsay
and then create the config.json and the run_script.sh files in the same directory.
The config.json file content is shown below.
1
2{
3 "name": "message_config",
4 "message": "Rm9yIFlvdXIgRXllcyBPbmx5IC0gSlNPTiBCb3VybmUhCg==",
5 "app_params": [
6 { "-f": "eyes" }
7 ]
8}
In the example below we use jq in various places, most notably using the to_entries function to load the app_params list of maps into a key/value associative bash array. We later loop through that associative array to create another array for passing parameters to our command. In the config.json above there is just a single map, but further entries could be added into the list. In the original script we used a similar mechanism to set a number of exported variables that are referenced in a separately sourced script.
The run_script.sh script that loads and processes the JSON configuration is shown below, with commented code.
1
2#!/bin/bash
3
4# load in config.json
5json_config=$(<config.json)
6
7# get root level keys and assign values to variables using jq
8message=$(jq -r '.message' <<< "${json_config}")
9name=$(jq -r '.name' <<< "${json_config}")
10
11# print out the name from the config
12printf "Config name:%s\n" "$name"
13
14# check that app_params key exists using jq before attempting to process
15if jq -r 'has("app_params")' <<< "${json_config}" | grep -q "true"
16then
17 # declare a bash associative array
18 declare -A app_params
19
20 # loop through the list of maps and create key/value pairs using to_entries to add the entries to associative array
21 while IFS="=" read -r key value; do
22 app_params["$key"]="$value"
23 done < <(jq -r '.app_params[] | to_entries | .[] | .key + "=" + .value' <<< "${json_config}")
24
25 # create a new array to store the parameters
26 declare -a app_params_arr
27
28 # loop through the array and variables key=value
29 for key in "${!app_params[@]}"; do
30 # set variables and add to the parameters array
31 app_params_arr+=("$key" "${app_params[$key]}")
32
33 # this could also be used to take the JSON list of maps to export variables for another sourced script e.g.
34 # export "$key"="${app_params[$key]}"
35 done
36
37else
38 echo "No app parameters found"
39fi
40
41# combine our app_params array and the decoded version of the message (as an array)
42params=( "${app_params_arr[@]}" "($(echo "$message" | base64 -d))" )
43
44# run our application with parameters supplied as the elements of the above array
45cowsay "${params[@]}"
Now to run the scripts that you created run the commands below.
1
2chmod +x run_script.sh
3./run_script.sh
Hopefully there was some useful information in this blog post on JSON and jq.