-Updated: Dec 24, 2013 (http://curl.haxx.se/docs/httpscripting.html)
_ _ ____ _
___| | | | _ \| |
/ __| | | | |_) | |
3. Fetch a page
3.1 GET
3.2 HEAD
+ 3.3 Multiple URLs in a single command line
+ 3.4 Multiple HTTP methods in a single command line
4. HTML forms
4.1 Forms explained
4.2 GET
The Uniform Resource Locator format is how you specify the address of a
particular resource on the Internet. You know these, you've seen URLs like
- http://curl.haxx.se or https://yourbank.com a million times. RFC 3986 is the
- canonical spec.
+ https://curl.haxx.se or https://yourbank.com a million times. RFC 3986 is the
+ canonical spec. And yeah, the formal name is not URL, it is URI.
2.2 Host
address and that's what curl will communicate with. Alternatively you specify
the IP address directly in the URL instead of a name.
- For development and other trying out situation, you can point out a different
+ For development and other trying out situations, you can point to a different
IP address for a host name than what would otherwise be used, by using curl's
--resolve option:
2.3 Port number
- Each protocol curl supports operate on a default port number, be it over TCP
+ Each protocol curl supports operates on a default port number, be it over TCP
or in some cases UDP. Normally you don't have to take that into
consideration, but at times you run test servers on other ports or
similar. Then you can specify the port number in the URL with a colon and a
The port number you specify in the URL is the number that the server uses to
offer its services. Sometimes you may use a local proxy, and then you may
- need to specify that proxy's port number separate on what curl needs to
+ need to specify that proxy's port number separately for what curl needs to
connect to locally. Like when using a HTTP proxy on port 4321:
curl --proxy http://proxy.example.org:4321 http://remote.example.org/
2.4 User name and password
Some services are setup to require HTTP authentication and then you need to
- provide name and password which then is transferred to the remote site in
+ provide name and password which is then transferred to the remote site in
various ways depending on the exact authentication protocol used.
You can opt to either insert the user and password in the URL or you can
the associated response. The path is what is to the right side of the slash
that follows the host name and possibly port number.
-
3. Fetch a page
3.1 GET
- The simplest and most common request/operation made using HTTP is to get a
+ The simplest and most common request/operation made using HTTP is to GET a
URL. The URL could itself refer to a web page, an image or a file. The client
issues a GET request to the server and receives the document it asked for.
If you issue the command line
- curl http://curl.haxx.se
+ curl https://curl.haxx.se
you get a web page returned in your terminal window. The entire HTML document
that that URL holds.
may see a Content-Length: in the response headers, but there must not be an
actual body in the HEAD response.
+ 3.3 Multiple URLs in a single command line
+
+ A single curl command line may involve one or many URLs. The most common case
+ is probably to just use one, but you can specify any amount of URLs. Yes
+ any. No limits. You'll then get requests repeated over and over for all the
+ given URLs.
+
+ Example, send two GETs:
+
+ curl http://url1.example.com http://url2.example.com
+
+ If you use --data to POST to the URL, using multiple URLs means that you send
+ that same POST to all the given URLs.
+
+ Example, send two POSTs:
+
+ curl --data name=curl http://url1.example.com http://url2.example.com
+
+
+ 3.4 Multiple HTTP methods in a single command line
+
+ Sometimes you need to operate on several URLs in a single command line and do
+ different HTTP methods on each. For this, you'll enjoy the --next option. It
+ is basically a separator that separates a bunch of options from the next. All
+ the URLs before --next will get the same method and will get all the POST
+ data merged into one.
+
+ When curl reaches the --next on the command line, it'll sort of reset the
+ method and the POST data and allow a new set.
+
+ Perhaps this is best shown with a few examples. To send first a HEAD and then
+ a GET:
+
+ curl -I http://example.com --next http://example.com
+
+ To first send a POST and then a GET:
+
+ curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html
+
+
4. HTML forms
4.1 Forms explained
Forms are the general way a web site can present a HTML page with fields for
- the user to enter data in, and then press some kind of 'OK' or 'submit'
+ the user to enter data in, and then press some kind of 'OK' or 'Submit'
button to get that data sent to the server. The server then typically uses
the posted data to decide how to act. Like using the entered words to search
- in a database, or to add the info in a bug track system, display the entered
+ in a database, or to add the info in a bug tracking system, display the entered
address on a map or using the info as a login-prompt verifying that the user
is allowed to see what it is about to see.
- Of course there has to be some kind of program in the server end to receive
+ Of course there has to be some kind of program on the server end to receive
the data you send. You cannot just invent something out of the air.
4.2 GET
curl --data-urlencode "name=I am Daniel" http://www.example.com
+ If you repeat --data several times on the command line, curl will
+ concatenate all the given data pieces - and put a '&' symbol between each
+ data segment.
+
4.4 File Upload POST
Back in late 1995 they defined an additional way to post data over HTTP. It
4.5 Hidden Fields
- A very common way for HTML based application to pass state information
+ A very common way for HTML based applications to pass state information
between pages is to add hidden fields to the forms. Hidden fields are
already filled in, they aren't displayed to the user and they get passed
along just as all the other fields.
<input type=submit name="press" value="OK">
</form>
- To post this with curl, you won't have to think about if the fields are
+ To POST this with curl, you won't have to think about if the fields are
hidden or not. To curl they're all the same:
curl --data "birthyear=1905&press=OK&person=daniel" [URL]
5.1 PUT
- The perhaps best way to upload data to a HTTP server is to use PUT. Then
+ Perhaps the best way to upload data to a HTTP server is to use PUT. Then
again, this of course requires that someone put a program or script on the
server end that knows how to receive a HTTP PUT stream.
If your proxy requires the authentication to be done using the NTLM method,
use --proxy-ntlm, if it requires Digest use --proxy-digest.
- If you use any one these user+password options but leave out the password
+ If you use any one of these user+password options but leave out the password
part, curl will prompt for the password interactively.
6.4 Hiding credentials
to redirect is Location:.
Curl does not follow Location: headers by default, but will simply display
- such pages in the same manner it display all HTTP replies. It does however
+ such pages in the same manner it displays all HTTP replies. It does however
feature an option that will make it attempt to follow the Location: pointers.
To tell curl to follow a Location:
(Take note that the --cookie-jar option described below is a better way to
store cookies.)
- Curl has a full blown cookie parsing engine built-in that comes to use if you
+ Curl has a full blown cookie parsing engine built-in that comes in use if you
want to reconnect to a server and use cookies that were stored from a
previous connection (or hand-crafted manually to fool the server into
believing you had a previous connection). To use previously stored cookies,
10.1 HTTPS is HTTP secure
- There are a few ways to do secure HTTP transfers. The by far most common
+ There are a few ways to do secure HTTP transfers. By far the most common
protocol for doing this is what is generally known as HTTPS, HTTP over
SSL. SSL encrypts all the data that is sent and received over the network and
thus makes it harder for attackers to spy on sensitive information.
truckload of advanced features to allow all those encryptions and key
infrastructure mechanisms encrypted HTTP requires.
- Curl supports encrypted fetches thanks to the freely available OpenSSL
- libraries. To get a page from a HTTPS server, simply run curl like:
+ Curl supports encrypted fetches when built to use a TLS library and it can be
+ built to use one out of a fairly large set of libraries - "curl -V" will show
+ which one your curl was built to use (if any!). To get a page from a HTTPS
+ server, simply run curl like:
curl https://secure.example.com
More about server certificate verification and ca cert bundles can be read
in the SSLCERTS document, available online here:
- http://curl.haxx.se/docs/sslcerts.html
+ https://curl.haxx.se/docs/sslcerts.html
+
+ At times you may end up with your own CA cert store and then you can tell
+ curl to use that to verify the server's certificate:
+
+ curl --cacert ca-bundle.pem https://example.com/
+
11. Custom Request Elements
12.1 Some login tricks
- While not strictly just HTTP related, it still cause a lot of people problems
+ While not strictly just HTTP related, it still causes a lot of people problems
so here's the executive run-down of how the vast majority of all login forms
work and how to login to them using curl.
make sure you got there through their login page) so you should make a habit
of first getting the login-form page to capture the cookies set there.
- Some web-based login systems features various amounts of javascript, and
+ Some web-based login systems feature various amounts of javascript, and
sometimes they use such code to set or modify cookie contents. Possibly they
do that to prevent programmed logins, like this manual describes how to...
Anyway, if reading the code isn't enough to let you repeat the behavior
14.1 Standards
- RFC 2616 is a must to read if you want in-depth understanding of the HTTP
+ RFC 7230 is a must to read if you want in-depth understanding of the HTTP
protocol
RFC 3986 explains the URL syntax
14.2 Sites
- http://curl.haxx.se is the home of the cURL project
+ https://curl.haxx.se is the home of the curl project
projects / platform / upstream / curl.git / blobdiff