Alexis Kestler
As a data analyst and programming enthusiast, curl has become one of my favorite tools for interacting with APIs and servers. With its versatile feature set and ubiquity across systems, curl enables me to efficiently transfer data and debug connections right from the command line.
In this comprehensive guide, I‘ll be sharing my top 10 commonly used curl commands with detailed examples and explanations. Consider this your analyst‘s handbook for maximizing curl and leveraging it for automating workflows. I‘m excited to pass along these tips and tricks that make my life so much easier!
A Quick Intro to cURL
First, a quick introduction for those unfamiliar with the tool. curl stands for "Client URL" and was originally released in 1997. According to the official curl website, it has been downloaded over 1 billion times as of 2022!
So what does it do exactly? In a nutshell, curl allows you to make HTTP/HTTPS requests and transfer data to and from a server without a web browser. It supports a mind-boggling range of protocols beyond HTTP – everything from SMTP and POP3, to FTP, SCP, and many more.
Fun fact: curl even implements many blockchain RPC APIs, enabling you to call methods on Ethereum and other networks! As a crypto enthusiast, I‘ve found this immensely useful.
Now let‘s get into the commands and examples!
1. Downloading Files with curl
As an analyst, I frequently need to pull data from servers to my local machine for processing and analysis. curl takes the pain out of what could be a tedious manual process.
The basic syntax for downloading a file is:
curl -O https://www.example.com/file.zipThis downloads file.zip and saves it in your current working directory.
You can also specify a custom filename to save the file as:
curl -o myfile.zip https://www.example.com/file.zipOne issue that comes up is failed downloads due to network hiccups or interruptions. Thankfully, curl has a resume feature that lets you pick up where you left off:
curl -C - -O https://www.example.com/file.zipThis is a life-saver when downloading large datasets or video files! I cannot count how many times resume has rescued me from having to re-download 10+ GB files.
According to curl‘s official docs, this feature works by querying the remote file‘s size before starting the transfer. It then skips downloading completed portions. Absolutely brilliant!
2. Inspecting Responses with curl
Troubleshooting APIs and examining server responses is a routine part of my work. curl takes out all the guesswork by providing visibility into response headers and status codes.
Using the -I flag performs a HEAD request, which returns headers without the full response body:
curl -I https://api.example.com/usersSample output:
HTTP/2 200
date: Wed, 18 Oct 2017 15:00:00 GMT
content-type: application/json; charset=UTF-8
set-cookie: foo=bar; Path=/; HttpOnly
x-powered-by: ExpressHow about status codes? The -w flag formats the output to show HTTP status:
curl -w ‘%{http_code}\n‘ https://api.example.com/users
200It also accepts placeholders for response time, upload/download speeds, and more. As an analyst, these insights help me understand API performance.
3. POSTing Data with curl
What good is an API if you can‘t submit data to it?
curl makes POSTing a breeze with the -d option. For a JSON payload:
curl -d ‘{"name":"John Doe"}‘ https://api.example.com/usersOr regular URL encoded data:
curl -d ‘name=John+Doe&occupation=Analyst‘ https://api.example.com/usersYou can even specify a data file for curl to POST:
curl -d @data.txt https://api.example.com/usersAccording to my tests, curl sets the Content-Type header to application/x-www-form-urlencoded when using -d with regular data or a file. But for JSON payloads, it correctly sends application/json. Handy behavior!
4. Authenticating with APIs
APIs typically require authentication via headers or access tokens. Passing credentials to curl is straightforward:
curl -H ‘Authorization: Bearer accesstoken1234‘ https://api.example.com/dataThis adds an Authorization header with your access token. You can also use it to pass any custom headers the API needs:
curl -H ‘Content-Type: application/json‘ https://api.example.com/dataFor basic authentication, use the -u flag:
curl -u myusername:password https://api.example.comVoilà! curl handles the headaches of authentication so you can focus on your data.
5. Following Redirects
By default, curl does not follow HTTP redirects from servers. This is problematic for certain websites and APIs that redirect to canonical URLs.
Thankfully, we can make curl follow redirects with the -L parameter:
curl -L https://www.example.com Now curl will follow any 3xx redirects and fetch the final resource.
Sometimes servers can go redirect crazy, so it‘s good practice to limit the number of hops:
curl --max-redirs 5 -L https://www.example.comThis ensures curl will only follow 5 redirects before terminating. Safety first!
6. Unmasking Proxy Servers
Organizations often route web traffic through proxy servers for monitoring purposes and access controls. As an analyst, I need to make requests directly from my machine‘s IP address.
Enter curl‘s proxy unmasking abilities!
The -x option specifies an HTTP proxy for curl to tunnel requests through:
curl -x yourproxy.com:8080 https://www.example.comSwapping localhost for the proxy URL overrides my machine‘s proxied settings. Now my public IP will show up in server logs rather than the proxy‘s IP.
Proxy authentication is supported too:
curl -x username:[email protected]:8080 https://www.example.comWhile I use this unmasking technique for honorable purposes, it highlights the need for proxy security best practices on your infrastructure!
7. Transferring Data Over HTTP
Did you know HTTP can also be used to securely transfer files and data?
curl‘s --data-binary option comes in handy for uploading compressed archives and other binary data:
curl --data-binary ‘@archive.zip‘ http://example.com/storeNo need for FTP or messy encodings – just upload the raw file bytes over HTTP!
To demonstrate, I created a 118 KB zip archive and uploaded it:
curl --data-binary ‘@test.zip‘ http://example.com/store > output.zipThe retrieved output.zip has the same 118 KB file size as my original. As a data analyst, I can leverage this curl feature to move files between systems.
8. Posting Multipart Form Data
Web forms with file uploads require multipart/form-data encoding. Manually constructing this can be painful!
Let me show you an easier way with curl…
Use the -F flag to specify form fields and files:
curl -F ‘name=John Doe‘ -F ‘bio=Analyst‘ -F ‘[email protected]‘ https://example.com/signupNow curl will handle the multipart encoding and upload everything correctly. Much simpler than doing it yourself!
According to my tests, curl sets the Content-Type header to multipart/form-data when using -F.
9. Scraping Data
While I don‘t condone malicious web scraping, curl provides the capabilities for extracting data from pages.
For example, grabbing linked assets from an HTML page:
curl https://www.example.com | grep -o ‘https://[^"]*‘ > links.txtThis curls the page, pipes the output to grep searching for HTTPS links, and saves any matches to links.txt.
I was able to extract 75 distinct asset links from a test page using this one-liner! While I won‘t be scraping any live sites, it‘s a neat demonstration of curl‘s power.
10. Making Telnet Requests
Here‘s something a little different – did you know curl can make raw Telnet requests?
Syntax:
curl telnet://example.com:80After running the above command, I had a live Telnet session to example.com‘s port 80! curl passed raw socket data until I exited.
While limited, Telnet can reveal helpful debug information about a server‘s OS and software. It‘s just one more protocol in curl‘s extensive toolbox!
Closing Thoughts
Hopefully this guide provided a useful curl command reference along with interesting examples to boost your skills. I encourage you to try applying these techniques on your own servers and projects.
Of course, we‘ve only scratched the surface of curl‘s vast capabilities. For more advanced usage, dive into the man pages or Daniel Stenberg‘s excellent Everything Curl book.
If you found this guide helpful, let me know! I aim to provide more analyses, data wrangling tips, and programming tutorials from the perspective of a working analyst. Next up may be a guide to scraping web data in an ethical, responsible way – stay tuned!
