How to Use the curl Command in Bash

The curl command is a powerful command-line tool for transferring data over various protocols, including HTTP, HTTPS, FTP, and more. It allows users to interact with web servers and APIs, making it essential for developers, system administrators, and anyone who needs to automate internet data transfer. In this guide, you will learn how to effectively use the curl command in Bash, covering its syntax, options, practical applications, advanced features, and best practices to help you harness its full potential.

Understanding curl

The name curl stands for “Client URL,” and it is widely used for downloading and uploading data from the command line or from scripts. By allowing users to specify a variety of options, curl enables complex operations to be performed with ease.

Here are some of the key functionalities curl offers:

• Downloading files from the internet or servers.
• Uploading files to a server.
• Sending HTTP requests to APIs.
• Interacting with RESTful services.

Mastering the curl command can significantly enhance your ability to work with web data and automate tasks efficiently.

Basic Syntax

The basic syntax of the curl command is as follows:

curl [options] [URL]

In this syntax:

• [options]: Various command line options to customize the behavior of the curl request.
• [URL]: The URL of the resource you want to access (e.g., a file, webpage, or API endpoint).

Commonly Used Options

The curl command comes with a wealth of options, allowing you to tailor your requests to fit your needs. Here are some commonly used options:

• -O: Save the downloaded file with the same name as on the server.

  curl -O https://example.com/file.zip

• -o filename: Save the downloaded content to a specified filename.

  curl -o savedfile.zip https://example.com/file.zip

• -L: Follow redirects. If the URL redirects somewhere else, curl will follow that link automatically.

  curl -L https://example.com/redirect

• -u username:password: Include basic authentication with the request.

  curl -u user:password https://example.com/protected

• -H “Header: Value”: Include a custom header in the request.

  curl -H "Authorization: Bearer token" https://api.example.com

• -X METHOD: Specify an HTTP method (GET, POST, PUT, DELETE, etc.).

  curl -X POST https://example.com/api

• -d data: Send data with a POST request.

  curl -d "param1=value1&param2=value2" -X POST https://example.com/api

• -s: Silent mode. Will not show progress or error messages.

  curl -s https://example.com

• -v: Verbose output. Will show detailed information about the request and response.

  curl -v https://example.com

Practical Examples of Using curl

1. Downloading Files from a URL

One of the simplest tasks you can perform with curl is downloading files. For example, to download a PDF file:

curl -O https://example.com/somefile.pdf

This will save the file to your current working directory with the same name as it has on the server.

2. Saving Output to a Specific File

If you want to download a file but save it under a different name:

curl -o my_document.pdf https://example.com/somefile.pdf

This command saves the file as my_document.pdf.

3. Retrieving Web Page Content

You can use curl to fetch the HTML content of a webpage:

curl https://www.example.com

This will output the HTML code of the specified webpage to the terminal.

4. Making API Requests

When interacting with APIs, you often need to send GET or POST requests. To retrieve JSON data from an API:

curl -H "Accept: application/json" https://api.example.com/data

This fetches the JSON data from the specified API endpoint.

5. Sending POST Data

To send data to a server using a POST request, use the -d option:

curl -d "username=user&password=1234" -X POST https://example.com/login

This command sends login data to the server.

6. Authenticating with APIs

Many APIs require authentication. To include your credentials with requests:

curl -u user:password https://api.example.com/protected

This example shows how to pass your credentials directly in the request.

7. Following Redirects

If a URL redirects to another URL, using the -L option will ensure that curl follows that redirect:

curl -L https://example.com/redirect

8. Downloading Multiple Files

To download multiple files simultaneously, create a text file with the list of URLs:

wget -i urls.txt

In this setup, the curl command reads each URL from urls.txt and downloads them one by one.

Advanced Features of curl

1. Limiting Download Speed

If you’re concerned about consuming too much bandwidth while downloading files, you can control the download speed with the following:

curl --limit-rate 200K https://example.com/file.zip

This speeds up the download process to 200 kilobytes per second.

2. Background Downloading

To download files in the background, you can use the -b option:

curl -b https://example.com/largefile.zip &

This command allows you to continue using the terminal while the file downloads.

3. Downloading over FTP

You can also use curl to download files from FTP servers:

curl -u username:password ftp://example.com/largefile.zip

Ensure you provide the proper credentials and specify the FTP URL appropriately.

4. Handling Cookies

Cookies can be handled using curl, allowing you to store and send cookies in your requests:

• -c cookie.txt: Saves cookies to a file named cookie.txt.
• -b cookie.txt: Reads cookies from cookie.txt and sends them with your request.

Example:

curl -c cookie.txt https://example.com/login

5. Downloading with Authentication

For resources requiring user authentication, you may pass your credentials using:

curl -u username:password https://example.com/private

This command accesses protected files using the provided username and password. However, be careful with sensitive information.

6. Verbose and Silent Options

During operations, it’s helpful to see what curl is doing. Use:

• -v for verbose output: Displays detailed request and response information.
• -s for silent mode: Disables progress and error messages, useful for scripting where you don’t want any extra output.

Example of a verbose command:

curl -v https://example.com

Common Mistakes When Using curl

1. Omitting Protocols

When using curl, always start your URL with a protocol (e.g., http:// or https://). Forgetting this can lead to connection errors:

curl example.com  // This will fail because the protocol is missing!

2. Case Sensitivity in URLs

URLs are case-sensitive. Ensure that the paths are typed correctly to avoid 404 errors:

curl https://example.com/Page 

3. Not Escaping Special Characters

When URLs contain special characters (like spaces, &, ?, and =), ensure you escape them properly to avoid syntax errors:

curl "https://example.com/search?query=hello world"

Using quotes around the URL can help here.

Best Practices for Using curl

• Use Quotes for URLs: When dealing with URLs that include special characters or spaces, always wrap them in quotes to prevent errors.
• Use Verbose for Debugging: When things go wrong, using -v can help you troubleshoot issues quickly.
• Regularly Review Documentation: The man page for curl is extensive. Review options to discover new features you might find useful.
• Customize Your Environment: Set up aliases or functions in your shell profile for frequent curl commands to streamline your workflow.

Conclusion

The curl command is an essential tool for anyone working in the Linux environment or dealing with data transfer via HTTP, FTP, and other protocols. Once you familiarize yourself with its various capabilities and syntax, you’ll be able to leverage it for a wide range of tasks, including file downloads, API interactions, and data management tasks.

By practicing and experimenting with the provided examples and tips, you can master the curl command and enhance your command-line skills. Whether you’re delivering data to an API or retrieving datasets for analysis, curl will prove to be an invaluable resource.

Start integrating curl into your day-to-day scripting activities, and watch how it dramatically simplifies your interactions with web servers. Mastering this command-line tool will empower you to automate tasks efficiently and improve your productivity as a developer or system administrator.