Using jq for JSON Parsing and Manipulation in Shell Scripts

In today’s DevOps and automation landscape, working with JSON data in shell scripts is unavoidable. APIs return JSON, configuration files use JSON, and cloud services communicate in JSON. The jq command-line tool transforms JSON manipulation from a painful chore into an elegant, powerful operation that integrates seamlessly with shell scripting.

What is jq?

jq is a lightweight and flexible command-line JSON processor that works like sed for JSON data. It allows you to slice, filter, map, and transform structured JSON data with the same ease that Unix tools like sed, awk, and grep provide for text. Written in C with zero runtime dependencies, jq is fast, portable, and available on virtually every platform.

Installation

Installing jq is straightforward across different platforms:

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install jq

CentOS/RHEL:

sudo yum install jq

macOS (using Homebrew):

brew install jq

Windows (using Chocolatey):

choco install jq

Verify installation:

jq --version

Basic Syntax and Concepts

The fundamental jq syntax is:

jq [options] 'filter' [file]

If no file is specified, jq reads from standard input, making it perfect for pipelines. The filter expression tells jq what to extract or how to transform the data.

The Identity Filter

The simplest jq filter is the dot . which represents the entire input:

echo '{"name":"John","age":30}' | jq '.'

This pretty-prints the JSON with syntax highlighting and proper indentation.

Extracting Data

Accessing Object Fields

Use dot notation to access object fields:

echo '{"name":"John","age":30}' | jq '.name'
# Output: "John"

Access nested fields with chained dots:

echo '{"user":{"name":"John","email":"[email protected]"}}' | jq '.user.email'
# Output: "[email protected]"

Working with Arrays

Access array elements by index:

echo '{"users":["Alice","Bob","Charlie"]}' | jq '.users[0]'
# Output: "Alice"

Iterate over all array elements:

echo '{"users":["Alice","Bob","Charlie"]}' | jq '.users[]'
# Output:
# "Alice"
# "Bob"
# "Charlie"

Multiple Fields

Extract multiple fields at once:

echo '{"name":"John","age":30,"city":"NYC"}' | jq '.name, .age'
# Output:
# "John"
# 30

Filtering Data

Select Function

Filter array elements based on conditions:

echo '[{"name":"Alice","age":25},{"name":"Bob","age":35}]' | jq '.[] | select(.age > 30)'
# Output: {"name":"Bob","age":35}

Testing for Existence

Check if a field exists:

echo '{"name":"Alice","email":"[email protected]"}' | jq 'has("email")'
# Output: true

Transforming Data

Creating New Objects

Construct new JSON objects from existing data:

echo '{"firstName":"John","lastName":"Doe","age":30}' | jq '{name: (.firstName + " " + .lastName), age: .age}'
# Output: {"name":"John Doe","age":30}

Mapping Over Arrays

Transform each element in an array:

echo '[{"price":10},{"price":20},{"price":30}]' | jq 'map(.price * 1.1)'
# Output: [11,22,33]

Adding Fields

Add new fields to existing objects:

echo '{"name":"Alice","age":25}' | jq '. + {country: "USA"}'
# Output: {"name":"Alice","age":25,"country":"USA"}

Practical Shell Script Examples

Example 1: Parsing API Response

#!/bin/bash

# Fetch user data from API and extract specific fields
response=$(curl -s https://api.example.com/users)

# Extract all user names
echo "$response" | jq -r '.users[].name'

# Get users older than 25
echo "$response" | jq '.users[] | select(.age > 25)'

Example 2: Processing Configuration Files

#!/bin/bash

config_file="config.json"

# Read database credentials
db_host=$(jq -r '.database.host' "$config_file")
db_port=$(jq -r '.database.port' "$config_file")
db_name=$(jq -r '.database.name' "$config_file")

echo "Connecting to $db_name at $db_host:$db_port"

Example 3: Updating JSON Files

#!/bin/bash

# Update version number in package.json
jq '.version = "2.0.0"' package.json > temp.json && mv temp.json package.json

# Add a new dependency
jq '.dependencies.express = "^4.18.0"' package.json > temp.json && mv temp.json package.json

Example 4: Converting Between Formats

#!/bin/bash

# Convert JSON array to CSV
echo '[{"name":"Alice","age":25},{"name":"Bob","age":30}]' | \
  jq -r '.[] | [.name, .age] | @csv'

# Output:
# "Alice",25
# "Bob",30

Example 5: Aggregating Data

#!/bin/bash

# Calculate total from array of prices
echo '[{"item":"apple","price":1.5},{"item":"banana","price":0.8}]' | \
  jq '[.[].price] | add'
# Output: 2.3

# Count items
echo '[{"status":"active"},{"status":"inactive"},{"status":"active"}]' | \
  jq '[.[] | select(.status == "active")] | length'
# Output: 2

Advanced Techniques

Raw Output Mode

Use -r flag to get raw strings without quotes (essential for shell scripts):

username=$(echo '{"user":"alice"}' | jq -r '.user')
echo "Hello, $username"  # Output: Hello, alice

Compact Output

Remove whitespace with -c flag:

echo '{"name": "John", "age": 30}' | jq -c '.'
# Output: {"name":"John","age":30}

Slurp Mode

Read multiple JSON objects into an array with -s:

echo '{"id":1}
{"id":2}
{"id":3}' | jq -s '.'
# Output: [{"id":1},{"id":2},{"id":3}]

Conditional Logic

Use if-then-else expressions:

echo '{"score":85}' | jq 'if .score >= 90 then "A" elif .score >= 80 then "B" else "C" end'
# Output: "B"

Variables

Pass variables to jq from shell:

threshold=30
echo '[{"age":25},{"age":35}]' | jq --arg t "$threshold" '.[] | select(.age > ($t | tonumber))'

Group By

Group array elements by a field:

echo '[{"type":"fruit","name":"apple"},{"type":"vegetable","name":"carrot"}]' | \
  jq 'group_by(.type)'

Error Handling in Scripts

Always handle potential JSON parsing errors in production scripts:

#!/bin/bash

response=$(curl -s https://api.example.com/data)

if echo "$response" | jq empty 2>/dev/null; then
    # Valid JSON, proceed with parsing
    value=$(echo "$response" | jq -r '.data.value')
    echo "Value: $value"
else
    echo "Error: Invalid JSON response"
    exit 1
fi

Performance Considerations

When working with large JSON files or in performance-critical scripts:

• Stream processing: Use --stream for very large files to avoid loading everything into memory
• Minimize calls: Extract multiple values in one jq invocation rather than multiple calls
• Use filters efficiently: Place selective filters early in the pipeline to reduce data processing
• Compile filters: For repeated operations, consider using jq as a library in a compiled language

Common Use Cases

DevOps engineers and system administrators frequently use jq for:

• Parsing responses from REST APIs and webhooks
• Extracting specific data from cloud service outputs (AWS CLI, kubectl, etc.)
• Transforming JSON configuration files during deployments
• Processing log files in JSON format
• Building CI/CD pipeline scripts that interact with JSON-based tools
• Validating and testing JSON data structures
• Creating reports from JSON metrics and monitoring data

Debugging jq Expressions

When developing complex jq filters:

• Use jq '.' to verify your input JSON is valid
• Build filters incrementally, testing each step
• Use the debug filter to inspect intermediate values
• Try the interactive jq playground at jqplay.org for testing
• Add | type to check data types at any point in your filter

Best Practices

To write maintainable and robust shell scripts with jq:

• Always use raw output (-r) in scripts: This removes quotes and makes the output suitable for variable assignment
• Quote your filters: Use single quotes around jq filters to prevent shell interpretation
• Handle null values: Use the // operator to provide defaults for null values
• Validate JSON input: Check that input is valid JSON before processing
• Use meaningful variable names: When using --arg, choose descriptive names
• Comment complex filters: Add comments in your shell scripts explaining what complex jq expressions do
• Test with edge cases: Verify your filters work with empty arrays, null values, and missing fields

Conclusion

Mastering jq transforms JSON from an obstacle into an asset in shell scripting. Its powerful filtering and transformation capabilities make it an essential tool for anyone working with APIs, cloud services, or modern DevOps workflows. The investment in learning jq pays immediate dividends in cleaner, more maintainable scripts and faster development cycles.

Start by using jq for simple field extraction, then gradually explore its more advanced features as your needs grow. With jq in your toolkit, JSON data becomes just another text format to process elegantly in the Unix tradition.