1 Updated: Dec 24, 2013 (http://curl.haxx.se/docs/httpscripting.html)
6 \___|\___/|_| \_\_____|
9 The Art Of Scripting HTTP Requests Using Curl
21 2.4 User name and password
32 4.6 Figure Out What A POST Looks Like
35 6. HTTP Authentication
36 6.1 Basic Authentication
37 6.2 Other Authentication
38 6.3 Proxy Authentication
39 6.4 Hiding credentials
50 10.1 HTTPS is HTTP secure
52 11. Custom Request Elements
53 11.1 Modify method and headers
54 11.2 More on changed methods
56 12.1 Some login tricks
58 13.1 Some debug tricks
63 ==============================================================================
69 This document assumes that you're familiar with HTML and general networking.
71 The increasing amount of applications moving to the web has made "HTTP
72 Scripting" more frequently requested and wanted. To be able to automatically
73 extract information from the web, to fake users, to post or upload data to
74 web servers are all important tasks today.
76 Curl is a command line tool for doing all sorts of URL manipulations and
77 transfers, but this particular document will focus on how to use it when
78 doing HTTP requests for fun and profit. I'll assume that you know how to
79 invoke 'curl --help' or 'curl --manual' to get basic information about it.
81 Curl is not written to do everything for you. It makes the requests, it gets
82 the data, it sends data and it retrieves the information. You probably need
83 to glue everything together using some kind of script language or repeated
88 HTTP is the protocol used to fetch data from web servers. It is a very simple
89 protocol that is built upon TCP/IP. The protocol also allows information to
90 get sent to the server from the client using a few different methods, as will
93 HTTP is plain ASCII text lines being sent by the client to a server to
94 request a particular action, and then the server replies a few text lines
95 before the actual requested content is sent to the client.
97 The client, curl, sends a HTTP request. The request contains a method (like
98 GET, POST, HEAD etc), a number of request headers and sometimes a request
99 body. The HTTP server responds with a status line (indicating if things went
100 well), response headers and most often also a response body. The "body" part
101 is the plain data you requested, like the actual HTML or the image etc.
105 Using curl's option --verbose (-v as a short option) will display what kind
106 of commands curl sends to the server, as well as a few other informational
109 --verbose is the single most useful option when it comes to debug or even
110 understand the curl<->server interaction.
112 Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer
113 even more details as they show EVERYTHING curl sends and receives. Use it
116 curl --trace-ascii debugdump.txt http://www.example.com/
120 Many times you may wonder what exactly is taking all the time, or you just
121 want to know the amount of milliseconds between two points in a
122 transfer. For those, and other similar situations, the --trace-time option
123 is what you need. It'll prepend the time to each trace output line:
125 curl --trace-ascii d.txt --trace-time http://example.com/
129 By default curl sends the response to stdout. You need to redirect it
130 somewhere to avoid that, most often that is done with -o or -O.
136 The Uniform Resource Locator format is how you specify the address of a
137 particular resource on the Internet. You know these, you've seen URLs like
138 http://curl.haxx.se or https://yourbank.com a million times. RFC 3986 is the
143 The host name is usually resolved using DNS or your /etc/hosts file to an IP
144 address and that's what curl will communicate with. Alternatively you specify
145 the IP address directly in the URL instead of a name.
147 For development and other trying out situation, you can point out a different
148 IP address for a host name than what would otherwise be used, by using curl's
151 curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/
155 Each protocol curl supports operate on a default port number, be it over TCP
156 or in some cases UDP. Normally you don't have to take that into
157 consideration, but at times you run test servers on other ports or
158 similar. Then you can specify the port number in the URL with a colon and a
159 number immediately following the host name. Like when doing HTTP to port
162 curl http://www.example.org:1234/
164 The port number you specify in the URL is the number that the server uses to
165 offer its services. Sometimes you may use a local proxy, and then you may
166 need to specify that proxy's port number separate on what curl needs to
167 connect to locally. Like when using a HTTP proxy on port 4321:
169 curl --proxy http://proxy.example.org:4321 http://remote.example.org/
171 2.4 User name and password
173 Some services are setup to require HTTP authentication and then you need to
174 provide name and password which then is transferred to the remote site in
175 various ways depending on the exact authentication protocol used.
177 You can opt to either insert the user and password in the URL or you can
178 provide them separately:
180 curl http://user:password@example.org/
184 curl -u user:password http://example.org/
186 You need to pay attention that this kind of HTTP authentication is not what
187 is usually done and requested by user-oriented web sites these days. They
188 tend to use forms and cookies instead.
192 The path part is just sent off to the server to request that it sends back
193 the associated response. The path is what is to the right side of the slash
194 that follows the host name and possibly port number.
201 The simplest and most common request/operation made using HTTP is to get a
202 URL. The URL could itself refer to a web page, an image or a file. The client
203 issues a GET request to the server and receives the document it asked for.
204 If you issue the command line
206 curl http://curl.haxx.se
208 you get a web page returned in your terminal window. The entire HTML document
211 All HTTP replies contain a set of response headers that are normally hidden,
212 use curl's --include (-i) option to display them as well as the rest of the
217 You can ask the remote server for ONLY the headers by using the --head (-I)
218 option which will make curl issue a HEAD request. In some special cases
219 servers deny the HEAD method while others still work, which is a particular
222 The HEAD method is defined and made so that the server returns the headers
223 exactly the way it would do for a GET, but without a body. It means that you
224 may see a Content-Length: in the response headers, but there must not be an
225 actual body in the HEAD response.
231 Forms are the general way a web site can present a HTML page with fields for
232 the user to enter data in, and then press some kind of 'OK' or 'submit'
233 button to get that data sent to the server. The server then typically uses
234 the posted data to decide how to act. Like using the entered words to search
235 in a database, or to add the info in a bug track system, display the entered
236 address on a map or using the info as a login-prompt verifying that the user
237 is allowed to see what it is about to see.
239 Of course there has to be some kind of program in the server end to receive
240 the data you send. You cannot just invent something out of the air.
244 A GET-form uses the method GET, as specified in HTML like:
246 <form method="GET" action="junk.cgi">
247 <input type=text name="birthyear">
248 <input type=submit name=press value="OK">
251 In your favorite browser, this form will appear with a text box to fill in
252 and a press-button labeled "OK". If you fill in '1905' and press the OK
253 button, your browser will then create a new URL to get for you. The URL will
254 get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
257 If the original form was seen on the page "www.hotmail.com/when/birth.html",
258 the second page you'll get will become
259 "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
261 Most search engines work this way.
263 To make curl do the GET form post for you, just enter the expected created
266 curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
270 The GET method makes all input field names get displayed in the URL field of
271 your browser. That's generally a good thing when you want to be able to
272 bookmark that page with your given data, but it is an obvious disadvantage
273 if you entered secret information in one of the fields or if there are a
274 large amount of fields creating a very long and unreadable URL.
276 The HTTP protocol then offers the POST method. This way the client sends the
277 data separated from the URL and thus you won't see any of it in the URL
280 The form would look very similar to the previous one:
282 <form method="POST" action="junk.cgi">
283 <input type=text name="birthyear">
284 <input type=submit name=press value=" OK ">
287 And to use curl to post this form with the same data filled in as before, we
290 curl --data "birthyear=1905&press=%20OK%20" \
291 http://www.example.com/when.cgi
293 This kind of POST will use the Content-Type
294 application/x-www-form-urlencoded and is the most widely used POST kind.
296 The data you send to the server MUST already be properly encoded, curl will
297 not do that for you. For example, if you want the data to contain a space,
298 you need to replace that space with %20 etc. Failing to comply with this
299 will most likely cause your data to be received wrongly and messed up.
301 Recent curl versions can in fact url-encode POST data for you, like this:
303 curl --data-urlencode "name=I am Daniel" http://www.example.com
307 Back in late 1995 they defined an additional way to post data over HTTP. It
308 is documented in the RFC 1867, why this method sometimes is referred to as
311 This method is mainly designed to better support file uploads. A form that
312 allows a user to upload a file could be written like this in HTML:
314 <form method="POST" enctype='multipart/form-data' action="upload.cgi">
315 <input type=file name=upload>
316 <input type=submit name=press value="OK">
319 This clearly shows that the Content-Type about to be sent is
322 To post to a form like this with curl, you enter a command line like:
324 curl --form upload=@localfilename --form press=OK [URL]
328 A very common way for HTML based application to pass state information
329 between pages is to add hidden fields to the forms. Hidden fields are
330 already filled in, they aren't displayed to the user and they get passed
331 along just as all the other fields.
333 A similar example form with one visible field, one hidden field and one
334 submit button could look like:
336 <form method="POST" action="foobar.cgi">
337 <input type=text name="birthyear">
338 <input type=hidden name="person" value="daniel">
339 <input type=submit name="press" value="OK">
342 To post this with curl, you won't have to think about if the fields are
343 hidden or not. To curl they're all the same:
345 curl --data "birthyear=1905&press=OK&person=daniel" [URL]
347 4.6 Figure Out What A POST Looks Like
349 When you're about fill in a form and send to a server by using curl instead
350 of a browser, you're of course very interested in sending a POST exactly the
351 way your browser does.
353 An easy way to get to see this, is to save the HTML page with the form on
354 your local disk, modify the 'method' to a GET, and press the submit button
355 (you could also change the action URL if you want to).
357 You will then clearly see the data get appended to the URL, separated with a
358 '?'-letter as GET forms are supposed to.
364 The perhaps best way to upload data to a HTTP server is to use PUT. Then
365 again, this of course requires that someone put a program or script on the
366 server end that knows how to receive a HTTP PUT stream.
368 Put a file to a HTTP server with curl:
370 curl --upload-file uploadfile http://www.example.com/receive.cgi
372 6. HTTP Authentication
374 6.1 Basic Authentication
376 HTTP Authentication is the ability to tell the server your username and
377 password so that it can verify that you're allowed to do the request you're
378 doing. The Basic authentication used in HTTP (which is the type curl uses by
379 default) is *plain* *text* based, which means it sends username and password
380 only slightly obfuscated, but still fully readable by anyone that sniffs on
381 the network between you and the remote server.
383 To tell curl to use a user and password for authentication:
385 curl --user name:password http://www.example.com
387 6.2 Other Authentication
389 The site might require a different authentication method (check the headers
390 returned by the server), and then --ntlm, --digest, --negotiate or even
391 --anyauth might be options that suit you.
393 6.3 Proxy Authentication
395 Sometimes your HTTP access is only available through the use of a HTTP
396 proxy. This seems to be especially common at various companies. A HTTP proxy
397 may require its own user and password to allow the client to get through to
398 the Internet. To specify those with curl, run something like:
400 curl --proxy-user proxyuser:proxypassword curl.haxx.se
402 If your proxy requires the authentication to be done using the NTLM method,
403 use --proxy-ntlm, if it requires Digest use --proxy-digest.
405 If you use any one these user+password options but leave out the password
406 part, curl will prompt for the password interactively.
408 6.4 Hiding credentials
410 Do note that when a program is run, its parameters might be possible to see
411 when listing the running processes of the system. Thus, other users may be
412 able to watch your passwords if you pass them as plain command line
413 options. There are ways to circumvent this.
415 It is worth noting that while this is how HTTP Authentication works, very
416 many web sites will not use this concept when they provide logins etc. See
417 the Web Login chapter further below for more details on that.
423 A HTTP request may include a 'referer' field (yes it is misspelled), which
424 can be used to tell from which URL the client got to this particular
425 resource. Some programs/scripts check the referer field of requests to verify
426 that this wasn't arriving from an external site or an unknown page. While
427 this is a stupid way to check something so easily forged, many scripts still
428 do it. Using curl, you can put anything you want in the referer-field and
429 thus more easily be able to fool the server into serving your request.
431 Use curl to set the referer field with:
433 curl --referer http://www.example.come http://www.example.com
437 Very similar to the referer field, all HTTP requests may set the User-Agent
438 field. It names what user agent (client) that is being used. Many
439 applications use this information to decide how to display pages. Silly web
440 programmers try to make different pages for users of different browsers to
441 make them look the best possible for their particular browsers. They usually
442 also do different kinds of javascript, vbscript etc.
444 At times, you will see that getting a page with curl will not return the same
445 page that you see when getting the page with your browser. Then you know it
446 is time to set the User Agent field to fool the server into thinking you're
447 one of those browsers.
449 To make curl look like Internet Explorer 5 on a Windows 2000 box:
451 curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
453 Or why not look like you're using Netscape 4.73 on an old Linux box:
455 curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
461 When a resource is requested from a server, the reply from the server may
462 include a hint about where the browser should go next to find this page, or a
463 new page keeping newly generated output. The header that tells the browser
464 to redirect is Location:.
466 Curl does not follow Location: headers by default, but will simply display
467 such pages in the same manner it display all HTTP replies. It does however
468 feature an option that will make it attempt to follow the Location: pointers.
470 To tell curl to follow a Location:
472 curl --location http://www.example.com
474 If you use curl to POST to a site that immediately redirects you to another
475 page, you can safely use --location (-L) and --data/--form together. Curl will
476 only use POST in the first request, and then revert to GET in the following
481 Browser typically support at least two other ways of redirects that curl
482 doesn't: first the html may contain a meta refresh tag that asks the browser
483 to load a specific URL after a set number of seconds, or it may use
490 The way the web browsers do "client side state control" is by using
491 cookies. Cookies are just names with associated contents. The cookies are
492 sent to the client by the server. The server tells the client for what path
493 and host name it wants the cookie sent back, and it also sends an expiration
494 date and a few more properties.
496 When a client communicates with a server with a name and path as previously
497 specified in a received cookie, the client sends back the cookies and their
498 contents to the server, unless of course they are expired.
500 Many applications and servers use this method to connect a series of requests
501 into a single logical session. To be able to use curl in such occasions, we
502 must be able to record and send back cookies the way the web application
503 expects them. The same way browsers deal with them.
507 The simplest way to send a few cookies to the server when getting a page with
508 curl is to add them on the command line like:
510 curl --cookie "name=Daniel" http://www.example.com
512 Cookies are sent as common HTTP headers. This is practical as it allows curl
513 to record cookies simply by recording headers. Record cookies with curl by
514 using the --dump-header (-D) option like:
516 curl --dump-header headers_and_cookies http://www.example.com
518 (Take note that the --cookie-jar option described below is a better way to
521 Curl has a full blown cookie parsing engine built-in that comes to use if you
522 want to reconnect to a server and use cookies that were stored from a
523 previous connection (or hand-crafted manually to fool the server into
524 believing you had a previous connection). To use previously stored cookies,
527 curl --cookie stored_cookies_in_file http://www.example.com
529 Curl's "cookie engine" gets enabled when you use the --cookie option. If you
530 only want curl to understand received cookies, use --cookie with a file that
531 doesn't exist. Example, if you want to let curl understand cookies from a
532 page and follow a location (and thus possibly send back cookies it received),
533 you can invoke it like:
535 curl --cookie nada --location http://www.example.com
537 Curl has the ability to read and write cookie files that use the same file
538 format that Netscape and Mozilla once used. It is a convenient way to share
539 cookies between scripts or invokes. The --cookie (-b) switch automatically
540 detects if a given file is such a cookie file and parses it, and by using the
541 --cookie-jar (-c) option you'll make curl write a new cookie file at the end
544 curl --cookie cookies.txt --cookie-jar newcookies.txt \
545 http://www.example.com
549 10.1 HTTPS is HTTP secure
551 There are a few ways to do secure HTTP transfers. The by far most common
552 protocol for doing this is what is generally known as HTTPS, HTTP over
553 SSL. SSL encrypts all the data that is sent and received over the network and
554 thus makes it harder for attackers to spy on sensitive information.
556 SSL (or TLS as the latest version of the standard is called) offers a
557 truckload of advanced features to allow all those encryptions and key
558 infrastructure mechanisms encrypted HTTP requires.
560 Curl supports encrypted fetches thanks to the freely available OpenSSL
561 libraries. To get a page from a HTTPS server, simply run curl like:
563 curl https://secure.example.com
567 In the HTTPS world, you use certificates to validate that you are the one
568 you claim to be, as an addition to normal passwords. Curl supports client-
569 side certificates. All certificates are locked with a pass phrase, which you
570 need to enter before the certificate can be used by curl. The pass phrase
571 can be specified on the command line or if not, entered interactively when
572 curl queries for it. Use a certificate with curl on a HTTPS server like:
574 curl --cert mycert.pem https://secure.example.com
576 curl also tries to verify that the server is who it claims to be, by
577 verifying the server's certificate against a locally stored CA cert
578 bundle. Failing the verification will cause curl to deny the connection. You
579 must then use --insecure (-k) in case you want to tell curl to ignore that
580 the server can't be verified.
582 More about server certificate verification and ca cert bundles can be read
583 in the SSLCERTS document, available online here:
585 http://curl.haxx.se/docs/sslcerts.html
587 11. Custom Request Elements
589 11.1 Modify method and headers
591 Doing fancy stuff, you may need to add or change elements of a single curl
594 For example, you can change the POST request to a PROPFIND and send the data
595 as "Content-Type: text/xml" (instead of the default Content-Type) like this:
597 curl --data "<xml>" --header "Content-Type: text/xml" \
598 --request PROPFIND url.com
600 You can delete a default header by providing one without content. Like you
601 can ruin the request by chopping off the Host: header:
603 curl --header "Host:" http://www.example.com
605 You can add headers the same way. Your server may want a "Destination:"
606 header, and you can add it:
608 curl --header "Destination: http://nowhere" http://example.com
610 11.2 More on changed methods
612 It should be noted that curl selects which methods to use on its own
613 depending on what action to ask for. -d will do POST, -I will do HEAD and so
614 on. If you use the --request / -X option you can change the method keyword
615 curl selects, but you will not modify curl's behavior. This means that if you
616 for example use -d "data" to do a POST, you can modify the method to a
617 PROPFIND with -X and curl will still think it sends a POST. You can change
618 the normal GET to a POST method by simply adding -X POST in a command line
621 curl -X POST http://example.org/
623 ... but curl will still think and act as if it sent a GET so it won't send any
629 12.1 Some login tricks
631 While not strictly just HTTP related, it still cause a lot of people problems
632 so here's the executive run-down of how the vast majority of all login forms
633 work and how to login to them using curl.
635 It can also be noted that to do this properly in an automated fashion, you
636 will most certainly need to script things and do multiple curl invokes etc.
638 First, servers mostly use cookies to track the logged-in status of the
639 client, so you will need to capture the cookies you receive in the
640 responses. Then, many sites also set a special cookie on the login page (to
641 make sure you got there through their login page) so you should make a habit
642 of first getting the login-form page to capture the cookies set there.
644 Some web-based login systems features various amounts of javascript, and
645 sometimes they use such code to set or modify cookie contents. Possibly they
646 do that to prevent programmed logins, like this manual describes how to...
647 Anyway, if reading the code isn't enough to let you repeat the behavior
648 manually, capturing the HTTP requests done by your browsers and analyzing the
649 sent cookies is usually a working method to work out how to shortcut the
652 In the actual <form> tag for the login, lots of sites fill-in random/session
653 or otherwise secretly generated hidden tags and you may need to first capture
654 the HTML code for the login form and extract all the hidden fields to be able
655 to do a proper login POST. Remember that the contents need to be URL encoded
656 when sent in a normal POST.
660 13.1 Some debug tricks
662 Many times when you run curl on a site, you'll notice that the site doesn't
663 seem to respond the same way to your curl requests as it does to your
666 Then you need to start making your curl requests more similar to your
669 * Use the --trace-ascii option to store fully detailed logs of the requests
670 for easier analyzing and better understanding
672 * Make sure you check for and use cookies when needed (both reading with
673 --cookie and writing with --cookie-jar)
675 * Set user-agent to one like a recent popular browser does
677 * Set referer like it is set by the browser
679 * If you use POST, make sure you send all the fields and in the same order as
682 A very good helper to make sure you do this right, is the LiveHTTPHeader tool
683 that lets you view all headers you send and receive with Mozilla/Firefox
684 (even when using HTTPS). Chrome features similar functionality out of the box
685 among the developer's tools.
687 A more raw approach is to capture the HTTP traffic on the network with tools
688 such as ethereal or tcpdump and check what headers that were sent and
689 received by the browser. (HTTPS makes this technique inefficient.)
695 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
698 RFC 3986 explains the URL syntax
700 RFC 1867 defines the HTTP post upload format
702 RFC 6525 defines how HTTP cookies work
706 http://curl.haxx.se is the home of the cURL project