1 .\" **************************************************************************
3 .\" * Project ___| | | | _ \| |
4 .\" * / __| | | | |_) | |
5 .\" * | (__| |_| | _ <| |___
6 .\" * \___|\___/|_| \_\_____|
8 .\" * Copyright (C) 1998 - 2014, Daniel Stenberg, <daniel@haxx.se>, et al.
10 .\" * This software is licensed as described in the file COPYING, which
11 .\" * you should have received as part of this distribution. The terms
12 .\" * are also available at http://curl.haxx.se/docs/copyright.html.
14 .\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell
15 .\" * copies of the Software, and permit persons to whom the Software is
16 .\" * furnished to do so, under the terms of the COPYING file.
18 .\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
19 .\" * KIND, either express or implied.
21 .\" **************************************************************************
23 .TH libcurl-tutorial 3 "19 Sep 2014" "libcurl" "libcurl programming"
25 libcurl-tutorial \- libcurl programming tutorial
27 This document attempts to describe the general principles and some basic
28 approaches to consider when programming with libcurl. The text will focus
29 mainly on the C interface but might apply fairly well on other interfaces as
30 well as they usually follow the C one pretty closely.
32 This document will refer to 'the user' as the person writing the source code
33 that uses libcurl. That would probably be you or someone in your position.
34 What will be generally referred to as 'the program' will be the collected
35 source code that you write that is using libcurl for transfers. The program
36 is outside libcurl and libcurl is outside of the program.
38 To get more details on all options and functions described herein, please
39 refer to their respective man pages.
42 There are many different ways to build C programs. This chapter will assume a
43 Unix style build process. If you use a different build system, you can still
44 read this to get general information that may apply to your environment as
46 .IP "Compiling the Program"
47 Your compiler needs to know where the libcurl headers are located. Therefore
48 you must set your compiler's include path to point to the directory where you
49 installed them. The 'curl-config'[3] tool can be used to get this information:
51 $ curl-config --cflags
53 .IP "Linking the Program with libcurl"
54 When having compiled the program, you need to link your object files to create
55 a single executable. For that to succeed, you need to link with libcurl and
56 possibly also with other libraries that libcurl itself depends on. Like the
57 OpenSSL libraries, but even some standard OS libraries may be needed on the
58 command line. To figure out which flags to use, once again the 'curl-config'
59 tool comes to the rescue:
64 libcurl can be built and customized in many ways. One of the things that
65 varies from different libraries and builds is the support for SSL-based
66 transfers, like HTTPS and FTPS. If a supported SSL library was detected
67 properly at build-time, libcurl will be built with SSL support. To figure out
68 if an installed libcurl has been built with SSL support enabled, use
69 \&'curl-config' like this:
71 $ curl-config --feature
73 And if SSL is supported, the keyword 'SSL' will be written to stdout,
74 possibly together with a few other features that could be either on or off on
75 for different libcurls.
77 See also the "Features libcurl Provides" further down.
79 When you write your configure script to detect libcurl and setup variables
80 accordingly, we offer a prewritten macro that probably does everything you
81 need in this area. See docs/libcurl/libcurl.m4 file - it includes docs on how
84 .SH "Portable Code in a Portable World"
85 The people behind libcurl have put a considerable effort to make libcurl work
86 on a large amount of different operating systems and environments.
88 You program libcurl the same way on all platforms that libcurl runs on. There
89 are only very few minor considerations that differ. If you just make sure to
90 write your code portable enough, you may very well create yourself a very
91 portable program. libcurl shouldn't stop you from that.
93 .SH "Global Preparation"
94 The program must initialize some of the libcurl functionality globally. That
95 means it should be done exactly once, no matter how many times you intend to
96 use the library. Once for your program's entire life time. This is done using
100 and it takes one parameter which is a bit pattern that tells libcurl what to
101 initialize. Using \fICURL_GLOBAL_ALL\fP will make it initialize all known
102 internal sub modules, and might be a good default option. The current two bits
103 that are specified are:
105 .IP "CURL_GLOBAL_WIN32"
106 which only does anything on Windows machines. When used on
107 a Windows machine, it'll make libcurl initialize the win32 socket
108 stuff. Without having that initialized properly, your program cannot use
109 sockets properly. You should only do this once for each application, so if
110 your program already does this or of another library in use does it, you
111 should not tell libcurl to do this as well.
113 which only does anything on libcurls compiled and built SSL-enabled. On these
114 systems, this will make libcurl initialize the SSL library properly for this
115 application. This only needs to be done once for each application so if your
116 program or another library already does this, this bit should not be needed.
119 libcurl has a default protection mechanism that detects if
120 \fIcurl_global_init(3)\fP hasn't been called by the time
121 \fIcurl_easy_perform(3)\fP is called and if that is the case, libcurl runs the
122 function itself with a guessed bit pattern. Please note that depending solely
123 on this is not considered nice nor very good.
125 When the program no longer uses libcurl, it should call
126 \fIcurl_global_cleanup(3)\fP, which is the opposite of the init call. It will
127 then do the reversed operations to cleanup the resources the
128 \fIcurl_global_init(3)\fP call initialized.
130 Repeated calls to \fIcurl_global_init(3)\fP and \fIcurl_global_cleanup(3)\fP
131 should be avoided. They should only be called once each.
133 .SH "Features libcurl Provides"
134 It is considered best-practice to determine libcurl features at run-time
135 rather than at build-time (if possible of course). By calling
136 \fIcurl_version_info(3)\fP and checking out the details of the returned
137 struct, your program can figure out exactly what the currently running libcurl
141 libcurl first introduced the so called easy interface. All operations in the
142 easy interface are prefixed with 'curl_easy'. The easy interface lets you do
143 single transfers with a synchronous and blocking function call.
145 libcurl also offers another interface that allows multiple simultaneous
146 transfers in a single thread, the so called multi interface. More about that
147 interface is detailed in a separate chapter further down. You still need to
148 understand the easy interface first, so please continue reading for better
150 .SH "Handle the Easy libcurl"
151 To use the easy interface, you must first create yourself an easy handle. You
152 need one handle for each easy session you want to perform. Basically, you
153 should use one handle for every thread you plan to use for transferring. You
154 must never share the same handle in multiple threads.
156 Get an easy handle with
158 easyhandle = curl_easy_init();
160 It returns an easy handle. Using that you proceed to the next step: setting
161 up your preferred actions. A handle is just a logic entity for the upcoming
162 transfer or series of transfers.
164 You set properties and options for this handle using
165 \fIcurl_easy_setopt(3)\fP. They control how the subsequent transfer or
166 transfers will be made. Options remain set in the handle until set again to
167 something different. They are sticky. Multiple requests using the same handle
168 will use the same options.
170 If you at any point would like to blank all previously set options for a
171 single easy handle, you can call \fIcurl_easy_reset(3)\fP and you can also
172 make a clone of an easy handle (with all its set options) using
173 \fIcurl_easy_duphandle(3)\fP.
175 Many of the options you set in libcurl are "strings", pointers to data
176 terminated with a zero byte. When you set strings with
177 \fIcurl_easy_setopt(3)\fP, libcurl makes its own copy so that they don't need
178 to be kept around in your application after being set[4].
180 One of the most basic properties to set in the handle is the URL. You set your
181 preferred URL to transfer with \fICURLOPT_URL(3)\fP in a manner similar to:
184 curl_easy_setopt(handle, CURLOPT_URL, "http://domain.com/");
187 Let's assume for a while that you want to receive data as the URL identifies a
188 remote resource you want to get here. Since you write a sort of application
189 that needs this transfer, I assume that you would like to get the data passed
190 to you directly instead of simply getting it passed to stdout. So, you write
191 your own function that matches this prototype:
193 size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp);
195 You tell libcurl to pass all data to this function by issuing a function
198 curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data);
200 You can control what data your callback function gets in the fourth argument
201 by setting another property:
203 curl_easy_setopt(easyhandle, CURLOPT_WRITEDATA, &internal_struct);
205 Using that property, you can easily pass local data between your application
206 and the function that gets invoked by libcurl. libcurl itself won't touch the
207 data you pass with \fICURLOPT_WRITEDATA(3)\fP.
209 libcurl offers its own default internal callback that will take care of the
210 data if you don't set the callback with \fICURLOPT_WRITEFUNCTION(3)\fP. It
211 will then simply output the received data to stdout. You can have the default
212 callback write the data to a different file handle by passing a 'FILE *' to a
213 file opened for writing with the \fICURLOPT_WRITEDATA(3)\fP option.
215 Now, we need to take a step back and have a deep breath. Here's one of those
216 rare platform-dependent nitpicks. Did you spot it? On some platforms[2],
217 libcurl won't be able to operate on files opened by the program. Thus, if you
218 use the default callback and pass in an open file with
219 \fICURLOPT_WRITEDATA(3)\fP, it will crash. You should therefore avoid this to
220 make your program run fine virtually everywhere.
222 (\fICURLOPT_WRITEDATA(3)\fP was formerly known as \fICURLOPT_FILE\fP. Both
223 names still work and do the same thing).
225 If you're using libcurl as a win32 DLL, you MUST use the
226 \fICURLOPT_WRITEFUNCTION(3)\fP if you set \fICURLOPT_WRITEDATA(3)\fP - or you
227 will experience crashes.
229 There are of course many more options you can set, and we'll get back to a few
230 of them later. Let's instead continue to the actual transfer:
232 success = curl_easy_perform(easyhandle);
234 \fIcurl_easy_perform(3)\fP will connect to the remote site, do the necessary
235 commands and receive the transfer. Whenever it receives data, it calls the
236 callback function we previously set. The function may get one byte at a time,
237 or it may get many kilobytes at once. libcurl delivers as much as possible as
238 often as possible. Your callback function should return the number of bytes it
239 \&"took care of". If that is not the exact same amount of bytes that was
240 passed to it, libcurl will abort the operation and return with an error code.
242 When the transfer is complete, the function returns a return code that informs
243 you if it succeeded in its mission or not. If a return code isn't enough for
244 you, you can use the \fICURLOPT_ERRORBUFFER(3)\fP to point libcurl to a buffer
245 of yours where it'll store a human readable error message as well.
247 If you then want to transfer another file, the handle is ready to be used
248 again. Mind you, it is even preferred that you re-use an existing handle if
249 you intend to make another transfer. libcurl will then attempt to re-use the
252 For some protocols, downloading a file can involve a complicated process of
253 logging in, setting the transfer mode, changing the current directory and
254 finally transferring the file data. libcurl takes care of all that
255 complication for you. Given simply the URL to a file, libcurl will take care
256 of all the details needed to get the file moved from one machine to another.
258 .SH "Multi-threading Issues"
259 The first basic rule is that you must \fBnever\fP simultaneously share a
260 libcurl handle (be it easy or multi or whatever) between multiple
261 threads. Only use one handle in one thread at any time. You can pass the
262 handles around among threads, but you must never use a single handle from more
263 than one thread at any given time.
265 libcurl is completely thread safe, except for two issues: signals and SSL/TLS
266 handlers. Signals are used for timing out name resolves (during DNS lookup) -
267 when built without using either the c-ares or threaded resolver backends.
269 If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are
270 then of course using the underlying SSL library multi-threaded and those libs
271 might have their own requirements on this issue. Basically, you need to
272 provide one or two functions to allow it to function properly. For all
277 http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION
281 http://gnutls.org/manual/html_node/Thread-safety.html
285 is claimed to be thread-safe already without anything required.
289 Required actions unknown.
293 Required actions unknown.
297 Required actions unknown.
301 The engine is fully thread-safe, and no additional steps are required.
303 When using multiple threads you should set the \fICURLOPT_NOSIGNAL(3)\fP
304 option to 1 for all handles. Everything will or might work fine except that
305 timeouts are not honored during the DNS lookup - which you can work around by
306 building libcurl with c-ares support. c-ares is a library that provides
307 asynchronous name resolves. On some platforms, libcurl simply will not
308 function properly multi-threaded unless this option is set.
310 Also, note that \fICURLOPT_DNS_USE_GLOBAL_CACHE(3)\fP is not thread-safe.
312 .SH "When It Doesn't Work"
313 There will always be times when the transfer fails for some reason. You might
314 have set the wrong libcurl option or misunderstood what the libcurl option
315 actually does, or the remote server might return non-standard replies that
316 confuse the library which then confuses your program.
318 There's one golden rule when these things occur: set the
319 \fICURLOPT_VERBOSE(3)\fP option to 1. It'll cause the library to spew out the
320 entire protocol details it sends, some internal info and some received
321 protocol data as well (especially when using FTP). If you're using HTTP,
322 adding the headers in the received output to study is also a clever way to get
323 a better understanding why the server behaves the way it does. Include headers
324 in the normal body output with \fICURLOPT_HEADER(3)\fP set 1.
326 Of course, there are bugs left. We need to know about them to be able to fix
327 them, so we're quite dependent on your bug reports! When you do report
328 suspected bugs in libcurl, please include as many details as you possibly can:
329 a protocol dump that \fICURLOPT_VERBOSE(3)\fP produces, library version, as
330 much as possible of your code that uses libcurl, operating system name and
331 version, compiler name and version etc.
333 If \fICURLOPT_VERBOSE(3)\fP is not enough, you increase the level of debug
334 data your application receive by using the \fICURLOPT_DEBUGFUNCTION(3)\fP.
336 Getting some in-depth knowledge about the protocols involved is never wrong,
337 and if you're trying to do funny things, you might very well understand
338 libcurl and how to use it better if you study the appropriate RFC documents
341 .SH "Upload Data to a Remote Site"
342 libcurl tries to keep a protocol independent approach to most transfers, thus
343 uploading to a remote FTP site is very similar to uploading data to a HTTP
344 server with a PUT request.
346 Of course, first you either create an easy handle or you re-use one existing
347 one. Then you set the URL to operate on just like before. This is the remote
348 URL, that we now will upload.
350 Since we write an application, we most likely want libcurl to get the upload
351 data by asking us for it. To make it do that, we set the read callback and
352 the custom pointer libcurl will pass to our read callback. The read callback
353 should have a prototype similar to:
355 size_t function(char *bufptr, size_t size, size_t nitems, void *userp);
357 Where bufptr is the pointer to a buffer we fill in with data to upload and
358 size*nitems is the size of the buffer and therefore also the maximum amount
359 of data we can return to libcurl in this call. The 'userp' pointer is the
360 custom pointer we set to point to a struct of ours to pass private data
361 between the application and the callback.
363 curl_easy_setopt(easyhandle, CURLOPT_READFUNCTION, read_function);
365 curl_easy_setopt(easyhandle, CURLOPT_READDATA, &filedata);
367 Tell libcurl that we want to upload:
369 curl_easy_setopt(easyhandle, CURLOPT_UPLOAD, 1L);
371 A few protocols won't behave properly when uploads are done without any prior
372 knowledge of the expected file size. So, set the upload file size using the
373 \fICURLOPT_INFILESIZE_LARGE(3)\fP for all known file sizes like this[1]:
376 /* in this example, file_size must be an curl_off_t variable */
377 curl_easy_setopt(easyhandle, CURLOPT_INFILESIZE_LARGE, file_size);
380 When you call \fIcurl_easy_perform(3)\fP this time, it'll perform all the
381 necessary operations and when it has invoked the upload it'll call your
382 supplied callback to get the data to upload. The program should return as much
383 data as possible in every invoke, as that is likely to make the upload perform
384 as fast as possible. The callback should return the number of bytes it wrote
385 in the buffer. Returning 0 will signal the end of the upload.
388 Many protocols use or even require that user name and password are provided
389 to be able to download or upload the data of your choice. libcurl offers
390 several ways to specify them.
392 Most protocols support that you specify the name and password in the URL
393 itself. libcurl will detect this and use them accordingly. This is written
396 protocol://user:password@example.com/path/
398 If you need any odd letters in your user name or password, you should enter
399 them URL encoded, as %XX where XX is a two-digit hexadecimal number.
401 libcurl also provides options to set various passwords. The user name and
402 password as shown embedded in the URL can instead get set with the
403 \fICURLOPT_USERPWD(3)\fP option. The argument passed to libcurl should be a
404 char * to a string in the format "user:password". In a manner like this:
406 curl_easy_setopt(easyhandle, CURLOPT_USERPWD, "myname:thesecret");
408 Another case where name and password might be needed at times, is for those
409 users who need to authenticate themselves to a proxy they use. libcurl offers
410 another option for this, the \fICURLOPT_PROXYUSERPWD(3)\fP. It is used quite
411 similar to the \fICURLOPT_USERPWD(3)\fP option like this:
413 curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "myname:thesecret");
415 There's a long time Unix "standard" way of storing FTP user names and
416 passwords, namely in the $HOME/.netrc file. The file should be made private
417 so that only the user may read it (see also the "Security Considerations"
418 chapter), as it might contain the password in plain text. libcurl has the
419 ability to use this file to figure out what set of user name and password to
420 use for a particular host. As an extension to the normal functionality,
421 libcurl also supports this file for non-FTP protocols such as HTTP. To make
422 curl use this file, use the \fICURLOPT_NETRC(3)\fP option:
424 curl_easy_setopt(easyhandle, CURLOPT_NETRC, 1L);
426 And a very basic example of how such a .netrc file may look like:
429 machine myhost.mydomain.com
434 All these examples have been cases where the password has been optional, or
435 at least you could leave it out and have libcurl attempt to do its job
436 without it. There are times when the password isn't optional, like when
437 you're using an SSL private key for secure transfers.
439 To pass the known private key password to libcurl:
441 curl_easy_setopt(easyhandle, CURLOPT_KEYPASSWD, "keypassword");
443 .SH "HTTP Authentication"
444 The previous chapter showed how to set user name and password for getting
445 URLs that require authentication. When using the HTTP protocol, there are
446 many different ways a client can provide those credentials to the server and
447 you can control which way libcurl will (attempt to) use them. The default HTTP
448 authentication method is called 'Basic', which is sending the name and
449 password in clear-text in the HTTP request, base64-encoded. This is insecure.
451 At the time of this writing, libcurl can be built to use: Basic, Digest, NTLM,
452 Negotiate (SPNEGO). You can tell libcurl which one to use
453 with \fICURLOPT_HTTPAUTH(3)\fP as in:
455 curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST);
457 And when you send authentication to a proxy, you can also set authentication
458 type the same way but instead with \fICURLOPT_PROXYAUTH(3)\fP:
460 curl_easy_setopt(easyhandle, CURLOPT_PROXYAUTH, CURLAUTH_NTLM);
462 Both these options allow you to set multiple types (by ORing them together),
463 to make libcurl pick the most secure one out of the types the server/proxy
464 claims to support. This method does however add a round-trip since libcurl
465 must first ask the server what it supports:
467 curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH,
468 CURLAUTH_DIGEST|CURLAUTH_BASIC);
470 For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list
471 with specific types) which allows libcurl to use whatever method it wants.
473 When asking for multiple types, libcurl will pick the available one it
474 considers "best" in its own internal order of preference.
477 We get many questions regarding how to issue HTTP POSTs with libcurl the
478 proper way. This chapter will thus include examples using both different
479 versions of HTTP POST that libcurl supports.
481 The first version is the simple POST, the most common version, that most HTML
482 pages using the <form> tag uses. We provide a pointer to the data and tell
483 libcurl to post it all to the remote site:
486 char *data="name=daniel&project=curl";
487 curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, data);
488 curl_easy_setopt(easyhandle, CURLOPT_URL, "http://posthere.com/");
490 curl_easy_perform(easyhandle); /* post away! */
493 Simple enough, huh? Since you set the POST options with the
494 \fICURLOPT_POSTFIELDS(3)\fP, this automatically switches the handle to use
495 POST in the upcoming request.
497 Ok, so what if you want to post binary data that also requires you to set the
498 Content-Type: header of the post? Well, binary posts prevent libcurl from
499 being able to do strlen() on the data to figure out the size, so therefore we
500 must tell libcurl the size of the post data. Setting headers in libcurl
501 requests are done in a generic way, by building a list of our own headers and
502 then passing that list to libcurl.
505 struct curl_slist *headers=NULL;
506 headers = curl_slist_append(headers, "Content-Type: text/xml");
508 /* post binary data */
509 curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, binaryptr);
511 /* set the size of the postfields data */
512 curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDSIZE, 23L);
514 /* pass our list of custom made headers */
515 curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers);
517 curl_easy_perform(easyhandle); /* post away! */
519 curl_slist_free_all(headers); /* free the header list */
522 While the simple examples above cover the majority of all cases where HTTP
523 POST operations are required, they don't do multi-part formposts. Multi-part
524 formposts were introduced as a better way to post (possibly large) binary data
525 and were first documented in the RFC1867 (updated in RFC2388). They're called
526 multi-part because they're built by a chain of parts, each part being a single
527 unit of data. Each part has its own name and contents. You can in fact create
528 and post a multi-part formpost with the regular libcurl POST support described
529 above, but that would require that you build a formpost yourself and provide
530 to libcurl. To make that easier, libcurl provides \fIcurl_formadd(3)\fP. Using
531 this function, you add parts to the form. When you're done adding parts, you
534 The following example sets two simple text parts with plain textual contents,
535 and then a file with binary contents and uploads the whole thing.
538 struct curl_httppost *post=NULL;
539 struct curl_httppost *last=NULL;
540 curl_formadd(&post, &last,
541 CURLFORM_COPYNAME, "name",
542 CURLFORM_COPYCONTENTS, "daniel", CURLFORM_END);
543 curl_formadd(&post, &last,
544 CURLFORM_COPYNAME, "project",
545 CURLFORM_COPYCONTENTS, "curl", CURLFORM_END);
546 curl_formadd(&post, &last,
547 CURLFORM_COPYNAME, "logotype-image",
548 CURLFORM_FILECONTENT, "curl.png", CURLFORM_END);
550 /* Set the form info */
551 curl_easy_setopt(easyhandle, CURLOPT_HTTPPOST, post);
553 curl_easy_perform(easyhandle); /* post away! */
555 /* free the post data again */
559 Multipart formposts are chains of parts using MIME-style separators and
560 headers. It means that each one of these separate parts get a few headers set
561 that describe the individual content-type, size etc. To enable your
562 application to handicraft this formpost even more, libcurl allows you to
563 supply your own set of custom headers to such an individual form part. You can
564 of course supply headers to as many parts as you like, but this little example
565 will show how you set headers to one specific part when you add that to the
569 struct curl_slist *headers=NULL;
570 headers = curl_slist_append(headers, "Content-Type: text/xml");
572 curl_formadd(&post, &last,
573 CURLFORM_COPYNAME, "logotype-image",
574 CURLFORM_FILECONTENT, "curl.xml",
575 CURLFORM_CONTENTHEADER, headers,
578 curl_easy_perform(easyhandle); /* post away! */
580 curl_formfree(post); /* free post */
581 curl_slist_free_all(headers); /* free custom header list */
584 Since all options on an easyhandle are "sticky", they remain the same until
585 changed even if you do call \fIcurl_easy_perform(3)\fP, you may need to tell
586 curl to go back to a plain GET request if you intend to do one as your next
587 request. You force an easyhandle to go back to GET by using the
588 \fICURLOPT_HTTPGET(3)\fP option:
590 curl_easy_setopt(easyhandle, CURLOPT_HTTPGET, 1L);
592 Just setting \fICURLOPT_POSTFIELDS(3)\fP to "" or NULL will *not* stop libcurl
593 from doing a POST. It will just make it POST without any data to send!
595 .SH "Showing Progress"
597 For historical and traditional reasons, libcurl has a built-in progress meter
598 that can be switched on and then makes it present a progress meter in your
601 Switch on the progress meter by, oddly enough, setting
602 \fICURLOPT_NOPROGRESS(3)\fP to zero. This option is set to 1 by default.
604 For most applications however, the built-in progress meter is useless and
605 what instead is interesting is the ability to specify a progress
606 callback. The function pointer you pass to libcurl will then be called on
607 irregular intervals with information about the current transfer.
609 Set the progress callback by using \fICURLOPT_PROGRESSFUNCTION(3)\fP. And pass
610 a pointer to a function that matches this prototype:
613 int progress_callback(void *clientp,
620 If any of the input arguments is unknown, a 0 will be passed. The first
621 argument, the 'clientp' is the pointer you pass to libcurl with
622 \fICURLOPT_PROGRESSDATA(3)\fP. libcurl won't touch it.
624 .SH "libcurl with C++"
626 There's basically only one thing to keep in mind when using C++ instead of C
627 when interfacing libcurl:
629 The callbacks CANNOT be non-static class member functions
635 static size_t write_data(void *ptr, size_t size, size_t nmemb,
638 /* do what you want with the data */
645 What "proxy" means according to Merriam-Webster: "a person authorized to act
646 for another" but also "the agency, function, or office of a deputy who acts as
647 a substitute for another".
649 Proxies are exceedingly common these days. Companies often only offer Internet
650 access to employees through their proxies. Network clients or user-agents ask
651 the proxy for documents, the proxy does the actual request and then it returns
654 libcurl supports SOCKS and HTTP proxies. When a given URL is wanted, libcurl
655 will ask the proxy for it instead of trying to connect to the actual host
656 identified in the URL.
658 If you're using a SOCKS proxy, you may find that libcurl doesn't quite support
659 all operations through it.
661 For HTTP proxies: the fact that the proxy is a HTTP proxy puts certain
662 restrictions on what can actually happen. A requested URL that might not be a
663 HTTP URL will be still be passed to the HTTP proxy to deliver back to
664 libcurl. This happens transparently, and an application may not need to
665 know. I say "may", because at times it is very important to understand that
666 all operations over a HTTP proxy use the HTTP protocol. For example, you
667 can't invoke your own custom FTP commands or even proper FTP directory
672 To tell libcurl to use a proxy at a given port number:
674 curl_easy_setopt(easyhandle, CURLOPT_PROXY, "proxy-host.com:8080");
676 Some proxies require user authentication before allowing a request, and you
677 pass that information similar to this:
679 curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "user:password");
681 If you want to, you can specify the host name only in the
682 \fICURLOPT_PROXY(3)\fP option, and set the port number separately with
683 \fICURLOPT_PROXYPORT(3)\fP.
685 Tell libcurl what kind of proxy it is with \fICURLOPT_PROXYTYPE(3)\fP (if not,
686 it will default to assume a HTTP proxy):
688 curl_easy_setopt(easyhandle, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS4);
690 .IP "Environment Variables"
692 libcurl automatically checks and uses a set of environment variables to know
693 what proxies to use for certain protocols. The names of the variables are
694 following an ancient de facto standard and are built up as "[protocol]_proxy"
695 (note the lower casing). Which makes the variable \&'http_proxy' checked for a
696 name of a proxy to use when the input URL is HTTP. Following the same rule,
697 the variable named 'ftp_proxy' is checked for FTP URLs. Again, the proxies are
698 always HTTP proxies, the different names of the variables simply allows
699 different HTTP proxies to be used.
701 The proxy environment variable contents should be in the format
702 \&"[protocol://][user:password@]machine[:port]". Where the protocol:// part is
703 simply ignored if present (so http://proxy and bluerk://proxy will do the
704 same) and the optional port number specifies on which port the proxy operates
705 on the host. If not specified, the internal default port number will be used
706 and that is most likely *not* the one you would like it to be.
708 There are two special environment variables. 'all_proxy' is what sets proxy
709 for any URL in case the protocol specific variable wasn't set, and
710 \&'no_proxy' defines a list of hosts that should not use a proxy even though a
711 variable may say so. If 'no_proxy' is a plain asterisk ("*") it matches all
714 To explicitly disable libcurl's checking for and using the proxy environment
715 variables, set the proxy name to "" - an empty string - with
716 \fICURLOPT_PROXY(3)\fP.
717 .IP "SSL and Proxies"
719 SSL is for secure point-to-point connections. This involves strong encryption
720 and similar things, which effectively makes it impossible for a proxy to
721 operate as a "man in between" which the proxy's task is, as previously
722 discussed. Instead, the only way to have SSL work over a HTTP proxy is to ask
723 the proxy to tunnel trough everything without being able to check or fiddle
726 Opening an SSL connection over a HTTP proxy is therefor a matter of asking the
727 proxy for a straight connection to the target host on a specified port. This
728 is made with the HTTP request CONNECT. ("please mr proxy, connect me to that
731 Because of the nature of this operation, where the proxy has no idea what kind
732 of data that is passed in and out through this tunnel, this breaks some of the
733 very few advantages that come from using a proxy, such as caching. Many
734 organizations prevent this kind of tunneling to other destination port numbers
735 than 443 (which is the default HTTPS port number).
737 .IP "Tunneling Through Proxy"
738 As explained above, tunneling is required for SSL to work and often even
739 restricted to the operation intended for SSL; HTTPS.
741 This is however not the only time proxy-tunneling might offer benefits to
742 you or your application.
744 As tunneling opens a direct connection from your application to the remote
745 machine, it suddenly also re-introduces the ability to do non-HTTP
746 operations over a HTTP proxy. You can in fact use things such as FTP
747 upload or FTP custom commands this way.
749 Again, this is often prevented by the administrators of proxies and is
752 Tell libcurl to use proxy tunneling like this:
754 curl_easy_setopt(easyhandle, CURLOPT_HTTPPROXYTUNNEL, 1L);
756 In fact, there might even be times when you want to do plain HTTP
757 operations using a tunnel like this, as it then enables you to operate on
758 the remote server instead of asking the proxy to do so. libcurl will not
759 stand in the way for such innovative actions either!
761 .IP "Proxy Auto-Config"
763 Netscape first came up with this. It is basically a web page (usually using a
764 \&.pac extension) with a Javascript that when executed by the browser with the
765 requested URL as input, returns information to the browser on how to connect
766 to the URL. The returned information might be "DIRECT" (which means no proxy
767 should be used), "PROXY host:port" (to tell the browser where the proxy for
768 this particular URL is) or "SOCKS host:port" (to direct the browser to a SOCKS
771 libcurl has no means to interpret or evaluate Javascript and thus it doesn't
772 support this. If you get yourself in a position where you face this nasty
773 invention, the following advice have been mentioned and used in the past:
775 - Depending on the Javascript complexity, write up a script that translates it
776 to another language and execute that.
778 - Read the Javascript code and rewrite the same logic in another language.
780 - Implement a Javascript interpreter; people have successfully used the
781 Mozilla Javascript engine in the past.
783 - Ask your admins to stop this, for a static proxy setup or similar.
785 .SH "Persistence Is The Way to Happiness"
787 Re-cycling the same easy handle several times when doing multiple requests is
790 After each single \fIcurl_easy_perform(3)\fP operation, libcurl will keep the
791 connection alive and open. A subsequent request using the same easy handle to
792 the same host might just be able to use the already open connection! This
793 reduces network impact a lot.
795 Even if the connection is dropped, all connections involving SSL to the same
796 host again, will benefit from libcurl's session ID cache that drastically
797 reduces re-connection time.
799 FTP connections that are kept alive save a lot of time, as the command-
800 response round-trips are skipped, and also you don't risk getting blocked
801 without permission to login again like on many FTP servers only allowing N
802 persons to be logged in at the same time.
804 libcurl caches DNS name resolving results, to make lookups of a previously
805 looked up name a lot faster.
807 Other interesting details that improve performance for subsequent requests
808 may also be added in the future.
810 Each easy handle will attempt to keep the last few connections alive for a
811 while in case they are to be used again. You can set the size of this "cache"
812 with the \fICURLOPT_MAXCONNECTS(3)\fP option. Default is 5. There is very
813 seldom any point in changing this value, and if you think of changing this it
814 is often just a matter of thinking again.
816 To force your upcoming request to not use an already existing connection (it
817 will even close one first if there happens to be one alive to the same host
818 you're about to operate on), you can do that by setting
819 \fICURLOPT_FRESH_CONNECT(3)\fP to 1. In a similar spirit, you can also forbid
820 the upcoming request to be "lying" around and possibly get re-used after the
821 request by setting \fICURLOPT_FORBID_REUSE(3)\fP to 1.
823 .SH "HTTP Headers Used by libcurl"
824 When you use libcurl to do HTTP requests, it'll pass along a series of headers
825 automatically. It might be good for you to know and understand these. You
826 can replace or remove them by using the \fICURLOPT_HTTPHEADER(3)\fP option.
829 This header is required by HTTP 1.1 and even many 1.0 servers and should be
830 the name of the server we want to talk to. This includes the port number if
831 anything but default.
837 When doing POST requests, libcurl sets this header to \&"100-continue" to ask
838 the server for an "OK" message before it proceeds with sending the data part
839 of the post. If the POSTed data amount is deemed "small", libcurl will not use
842 .SH "Customizing Operations"
843 There is an ongoing development today where more and more protocols are built
844 upon HTTP for transport. This has obvious benefits as HTTP is a tested and
845 reliable protocol that is widely deployed and has excellent proxy-support.
847 When you use one of these protocols, and even when doing other kinds of
848 programming you may need to change the traditional HTTP (or FTP or...)
849 manners. You may need to change words, headers or various data.
851 libcurl is your friend here too.
854 If just changing the actual HTTP request keyword is what you want, like when
855 GET, HEAD or POST is not good enough for you, \fICURLOPT_CUSTOMREQUEST(3)\fP
856 is there for you. It is very simple to use:
858 curl_easy_setopt(easyhandle, CURLOPT_CUSTOMREQUEST, "MYOWNREQUEST");
860 When using the custom request, you change the request keyword of the actual
861 request you are performing. Thus, by default you make a GET request but you can
862 also make a POST operation (as described before) and then replace the POST
863 keyword if you want to. You're the boss.
866 HTTP-like protocols pass a series of headers to the server when doing the
867 request, and you're free to pass any amount of extra headers that you
868 think fit. Adding headers is this easy:
871 struct curl_slist *headers=NULL; /* init to NULL is important */
873 headers = curl_slist_append(headers, "Hey-server-hey: how are you?");
874 headers = curl_slist_append(headers, "X-silly-content: yes");
876 /* pass our list of custom made headers */
877 curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers);
879 curl_easy_perform(easyhandle); /* transfer http */
881 curl_slist_free_all(headers); /* free the header list */
884 \&... and if you think some of the internally generated headers, such as
885 Accept: or Host: don't contain the data you want them to contain, you can
886 replace them by simply setting them too:
889 headers = curl_slist_append(headers, "Accept: Agent-007");
890 headers = curl_slist_append(headers, "Host: munged.host.line");
894 If you replace an existing header with one with no contents, you will prevent
895 the header from being sent. For instance, if you want to completely prevent the
896 \&"Accept:" header from being sent, you can disable it with code similar to this:
898 headers = curl_slist_append(headers, "Accept:");
900 Both replacing and canceling internal headers should be done with careful
901 consideration and you should be aware that you may violate the HTTP protocol
904 .IP "Enforcing chunked transfer-encoding"
906 By making sure a request uses the custom header "Transfer-Encoding: chunked"
907 when doing a non-GET HTTP operation, libcurl will switch over to "chunked"
908 upload, even though the size of the data to upload might be known. By default,
909 libcurl usually switches over to chunked upload automatically if the upload
910 data size is unknown.
914 All HTTP requests includes the version number to tell the server which version
915 we support. libcurl speaks HTTP 1.1 by default. Some very old servers don't
916 like getting 1.1-requests and when dealing with stubborn old things like that,
917 you can tell libcurl to use 1.0 instead by doing something like this:
919 curl_easy_setopt(easyhandle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
921 .IP "FTP Custom Commands"
923 Not all protocols are HTTP-like, and thus the above may not help you when
924 you want to make, for example, your FTP transfers to behave differently.
926 Sending custom commands to a FTP server means that you need to send the
927 commands exactly as the FTP server expects them (RFC959 is a good guide
928 here), and you can only use commands that work on the control-connection
929 alone. All kinds of commands that require data interchange and thus need
930 a data-connection must be left to libcurl's own judgement. Also be aware
931 that libcurl will do its very best to change directory to the target
932 directory before doing any transfer, so if you change directory (with CWD
933 or similar) you might confuse libcurl and then it might not attempt to
934 transfer the file in the correct remote directory.
936 A little example that deletes a given file before an operation:
939 headers = curl_slist_append(headers, "DELE file-to-remove");
941 /* pass the list of custom commands to the handle */
942 curl_easy_setopt(easyhandle, CURLOPT_QUOTE, headers);
944 curl_easy_perform(easyhandle); /* transfer ftp data! */
946 curl_slist_free_all(headers); /* free the header list */
949 If you would instead want this operation (or chain of operations) to happen
950 _after_ the data transfer took place the option to \fIcurl_easy_setopt(3)\fP
951 would instead be called \fICURLOPT_POSTQUOTE(3)\fP and used the exact same
954 The custom FTP command will be issued to the server in the same order they are
955 added to the list, and if a command gets an error code returned back from the
956 server, no more commands will be issued and libcurl will bail out with an
957 error code (CURLE_QUOTE_ERROR). Note that if you use \fICURLOPT_QUOTE(3)\fP to
958 send commands before a transfer, no transfer will actually take place when a
959 quote command has failed.
961 If you set the \fICURLOPT_HEADER(3)\fP to 1, you will tell libcurl to get
962 information about the target file and output "headers" about it. The headers
963 will be in "HTTP-style", looking like they do in HTTP.
965 The option to enable headers or to run custom FTP commands may be useful to
966 combine with \fICURLOPT_NOBODY(3)\fP. If this option is set, no actual file
967 content transfer will be performed.
969 .IP "FTP Custom CUSTOMREQUEST"
970 If you do want to list the contents of a FTP directory using your own defined
971 FTP command, \fICURLOPT_CUSTOMREQUEST(3)\fP will do just that. "NLST" is the
972 default one for listing directories but you're free to pass in your idea of a
975 .SH "Cookies Without Chocolate Chips"
976 In the HTTP sense, a cookie is a name with an associated value. A server sends
977 the name and value to the client, and expects it to get sent back on every
978 subsequent request to the server that matches the particular conditions
979 set. The conditions include that the domain name and path match and that the
980 cookie hasn't become too old.
982 In real-world cases, servers send new cookies to replace existing ones to
983 update them. Server use cookies to "track" users and to keep "sessions".
985 Cookies are sent from server to clients with the header Set-Cookie: and
986 they're sent from clients to servers with the Cookie: header.
988 To just send whatever cookie you want to a server, you can use
989 \fICURLOPT_COOKIE(3)\fP to set a cookie string like this:
991 curl_easy_setopt(easyhandle, CURLOPT_COOKIE, "name1=var1; name2=var2;");
993 In many cases, that is not enough. You might want to dynamically save
994 whatever cookies the remote server passes to you, and make sure those cookies
995 are then used accordingly on later requests.
997 One way to do this, is to save all headers you receive in a plain file and
998 when you make a request, you tell libcurl to read the previous headers to
999 figure out which cookies to use. Set the header file to read cookies from with
1000 \fICURLOPT_COOKIEFILE(3)\fP.
1002 The \fICURLOPT_COOKIEFILE(3)\fP option also automatically enables the cookie
1003 parser in libcurl. Until the cookie parser is enabled, libcurl will not parse
1004 or understand incoming cookies and they will just be ignored. However, when
1005 the parser is enabled the cookies will be understood and the cookies will be
1006 kept in memory and used properly in subsequent requests when the same handle
1007 is used. Many times this is enough, and you may not have to save the cookies
1008 to disk at all. Note that the file you specify to \ICURLOPT_COOKIEFILE(3)\fP
1009 doesn't have to exist to enable the parser, so a common way to just enable the
1010 parser and not read any cookies is to use the name of a file you know doesn't
1013 If you would rather use existing cookies that you've previously received with
1014 your Netscape or Mozilla browsers, you can make libcurl use that cookie file
1015 as input. The \fICURLOPT_COOKIEFILE(3)\fP is used for that too, as libcurl
1016 will automatically find out what kind of file it is and act accordingly.
1018 Perhaps the most advanced cookie operation libcurl offers, is saving the
1019 entire internal cookie state back into a Netscape/Mozilla formatted cookie
1020 file. We call that the cookie-jar. When you set a file name with
1021 \fICURLOPT_COOKIEJAR(3)\fP, that file name will be created and all received
1022 cookies will be stored in it when \fIcurl_easy_cleanup(3)\fP is called. This
1023 enables cookies to get passed on properly between multiple handles without any
1024 information getting lost.
1026 .SH "FTP Peculiarities We Need"
1028 FTP transfers use a second TCP/IP connection for the data transfer. This is
1029 usually a fact you can forget and ignore but at times this fact will come
1030 back to haunt you. libcurl offers several different ways to customize how the
1031 second connection is being made.
1033 libcurl can either connect to the server a second time or tell the server to
1034 connect back to it. The first option is the default and it is also what works
1035 best for all the people behind firewalls, NATs or IP-masquerading setups.
1036 libcurl then tells the server to open up a new port and wait for a second
1037 connection. This is by default attempted with EPSV first, and if that doesn't
1038 work it tries PASV instead. (EPSV is an extension to the original FTP spec
1039 and does not exist nor work on all FTP servers.)
1041 You can prevent libcurl from first trying the EPSV command by setting
1042 \fICURLOPT_FTP_USE_EPSV(3)\fP to zero.
1044 In some cases, you will prefer to have the server connect back to you for the
1045 second connection. This might be when the server is perhaps behind a firewall
1046 or something and only allows connections on a single port. libcurl then
1047 informs the remote server which IP address and port number to connect to.
1048 This is made with the \fICURLOPT_FTPPORT(3)\fP option. If you set it to "-",
1049 libcurl will use your system's "default IP address". If you want to use a
1050 particular IP, you can set the full IP address, a host name to resolve to an
1051 IP address or even a local network interface name that libcurl will get the IP
1054 When doing the "PORT" approach, libcurl will attempt to use the EPRT and the
1055 LPRT before trying PORT, as they work with more protocols. You can disable
1056 this behavior by setting \fICURLOPT_FTP_USE_EPRT(3)\fP to zero.
1058 .SH "Headers Equal Fun"
1060 Some protocols provide "headers", meta-data separated from the normal
1061 data. These headers are by default not included in the normal data stream, but
1062 you can make them appear in the data stream by setting \fICURLOPT_HEADER(3)\fP
1065 What might be even more useful, is libcurl's ability to separate the headers
1066 from the data and thus make the callbacks differ. You can for example set a
1067 different pointer to pass to the ordinary write callback by setting
1068 \fICURLOPT_HEADERDATA(3)\fP.
1070 Or, you can set an entirely separate function to receive the headers, by using
1071 \fICURLOPT_HEADERFUNCTION(3)\fP.
1073 The headers are passed to the callback function one by one, and you can
1074 depend on that fact. It makes it easier for you to add custom header parsers
1077 \&"Headers" for FTP transfers equal all the FTP server responses. They aren't
1078 actually true headers, but in this case we pretend they are! ;-)
1080 .SH "Post Transfer Information"
1082 [ curl_easy_getinfo ]
1084 .SH "Security Considerations"
1086 The libcurl project takes security seriously. The library is written with
1087 caution and precautions are taken to mitigate many kinds of risks encountered
1088 while operating with potentially malicious servers on the Internet. It is a
1089 powerful library, however, which allows application writers to make trade offs
1090 between ease of writing and exposure to potential risky operations. If
1091 used the right way, you can use libcurl to transfer data pretty safely.
1093 Many applications are used in closed networks where users and servers
1094 can be trusted, but many others are used on arbitrary servers and are fed
1095 input from potentially untrusted users. Following is a discussion about
1096 some risks in the ways in which applications commonly use libcurl and
1097 potential mitigations of those risks. It is by no means comprehensive, but
1098 shows classes of attacks that robust applications should consider. The
1099 Common Weakness Enumeration project at http://cwe.mitre.org/ is a good
1100 reference for many of these and similar types of weaknesses of which
1101 application writers should be aware.
1104 If you use a command line tool (such as curl) that uses libcurl, and you give
1105 options to the tool on the command line those options can very likely get read
1106 by other users of your system when they use 'ps' or other tools to list
1107 currently running processes.
1109 To avoid this problem, never feed sensitive things to programs using command
1110 line options. Write them to a protected file and use the \-K option to
1114 \&.netrc is a pretty handy file/feature that allows you to login quickly and
1115 automatically to frequently visited sites. The file contains passwords in
1116 clear text and is a real security risk. In some cases, your .netrc is also
1117 stored in a home directory that is NFS mounted or used on another network
1118 based file system, so the clear text password will fly through your network
1119 every time anyone reads that file!
1121 To avoid this problem, don't use .netrc files and never store passwords in
1122 plain text anywhere.
1124 .IP "Clear Text Passwords"
1125 Many of the protocols libcurl supports send name and password unencrypted as
1126 clear text (HTTP Basic authentication, FTP, TELNET etc). It is very easy for
1127 anyone on your network or a network nearby yours to just fire up a network
1128 analyzer tool and eavesdrop on your passwords. Don't let the fact that HTTP
1129 Basic uses base64 encoded passwords fool you. They may not look readable at a
1130 first glance, but they very easily "deciphered" by anyone within seconds.
1132 To avoid this problem, use an authentication mechanism or other protocol that
1133 doesn't let snoopers see your password: Digest, CRAM-MD5, Kerberos, SPNEGO or
1134 NTLM authentication, HTTPS, FTPS, SCP and SFTP are a few examples.
1137 The \fICURLOPT_FOLLOWLOCATION(3)\fP option automatically follows HTTP
1138 redirects sent by a remote server. These redirects can refer to any kind of
1139 URL, not just HTTP. A redirect to a file: URL would cause the libcurl to read
1140 (or write) arbitrary files from the local filesystem. If the application
1141 returns the data back to the user (as would happen in some kinds of CGI
1142 scripts), an attacker could leverage this to read otherwise forbidden data
1143 (e.g. file://localhost/etc/passwd).
1145 If authentication credentials are stored in the ~/.netrc file, or Kerberos
1146 is in use, any other URL type (not just file:) that requires
1147 authentication is also at risk. A redirect such as
1148 ftp://some-internal-server/private-file would then return data even when
1149 the server is password protected.
1151 In the same way, if an unencrypted SSH private key has been configured for
1152 the user running the libcurl application, SCP: or SFTP: URLs could access
1153 password or private-key protected resources,
1154 e.g. sftp://user@some-internal-server/etc/passwd
1156 The \fICURLOPT_REDIR_PROTOCOLS(3)\fP and \fICURLOPT_NETRC(3)\fP options can be
1157 used to mitigate against this kind of attack.
1159 A redirect can also specify a location available only on the machine running
1160 libcurl, including servers hidden behind a firewall from the attacker.
1161 e.g. http://127.0.0.1/ or http://intranet/delete-stuff.cgi?delete=all or
1162 tftp://bootp-server/pc-config-data
1164 Apps can mitigate against this by disabling \fICURLOPT_FOLLOWLOCATION(3)\fP
1165 and handling redirects itself, sanitizing URLs as necessary. Alternately, an
1166 app could leave \fICURLOPT_FOLLOWLOCATION(3)\fP enabled but set
1167 \fICURLOPT_REDIR_PROTOCOLS(3)\fP and install a
1168 \fICURLOPT_OPENSOCKETFUNCTION(3)\fP callback function in which addresses are
1169 sanitized before use.
1171 .IP "Private Resources"
1172 A user who can control the DNS server of a domain being passed in within a URL
1173 can change the address of the host to a local, private address which a
1174 server-side libcurl-using application could then use. e.g. the innocuous URL
1175 http://fuzzybunnies.example.com/ could actually resolve to the IP address of a
1176 server behind a firewall, such as 127.0.0.1 or 10.1.2.3. Apps can mitigate
1177 against this by setting a \fICURLOPT_OPENSOCKETFUNCTION(3)\fP and checking the
1178 address before a connection.
1180 All the malicious scenarios regarding redirected URLs apply just as well to
1181 non-redirected URLs, if the user is allowed to specify an arbitrary URL that
1182 could point to a private resource. For example, a web app providing a
1183 translation service might happily translate file://localhost/etc/passwd and
1184 display the result. Apps can mitigate against this with the
1185 \fICURLOPT_PROTOCOLS(3)\fP option as well as by similar mitigation techniques
1188 A malicious FTP server could in response to the PASV command return an IP
1189 address and port number for a server local to the app running libcurl but
1190 behind a firewall. Apps can mitigate against this by using the
1191 \fICURLOPT_FTP_SKIP_PASV_IP(3)\fP option or \fICURLOPT_FTPPORT(3)\fP.
1193 .IP "IPv6 Addresses"
1194 libcurl will normally handle IPv6 addresses transparently and just as easily
1195 as IPv4 addresses. That means that a sanitizing function that filters out
1196 addressses like 127.0.0.1 isn't sufficient--the equivalent IPv6 addresses ::1,
1197 ::, 0:00::0:1, ::127.0.0.1 and ::ffff:7f00:1 supplied somehow by an attacker
1198 would all bypass a naive filter and could allow access to undesired local
1199 resources. IPv6 also has special address blocks like link-local and site-local
1200 that generally shouldn't be accessed by a server-side libcurl-using
1201 application. A poorly-configured firewall installed in a data center,
1202 organization or server may also be configured to limit IPv4 connections but
1203 leave IPv6 connections wide open. In some cases, the CURL_IPRESOLVE_V4 option
1204 can be used to limit resolved addresses to IPv4 only and bypass these issues.
1207 When uploading, a redirect can cause a local (or remote) file to be
1208 overwritten. Apps must not allow any unsanitized URL to be passed in for
1209 uploads. Also, \fICURLOPT_FOLLOWLOCATION(3)\fP should not be used on uploads.
1210 Instead, the app should handle redirects itself, sanitizing each URL first.
1213 Use of \fICURLOPT_UNRESTRICTED_AUTH(3)\fP could cause authentication
1214 information to be sent to an unknown second server. Apps can mitigate against
1215 this by disabling \fICURLOPT_FOLLOWLOCATION(3)\fP and handling redirects
1216 itself, sanitizing where necessary.
1218 Use of the CURLAUTH_ANY option to \fICURLOPT_HTTPAUTH(3)\fP could result in
1219 user name and password being sent in clear text to an HTTP server. Instead,
1220 use CURLAUTH_ANYSAFE which ensures that the password is encrypted over the
1221 network, or else fail the request.
1223 Use of the CURLUSESSL_TRY option to \fICURLOPT_USE_SSL(3)\fP could result in
1224 user name and password being sent in clear text to an FTP server. Instead,
1225 use CURLUSESSL_CONTROL to ensure that an encrypted connection is used or else
1229 If cookies are enabled and cached, then a user could craft a URL which
1230 performs some malicious action to a site whose authentication is already
1231 stored in a cookie. e.g. http://mail.example.com/delete-stuff.cgi?delete=all
1232 Apps can mitigate against this by disabling cookies or clearing them
1235 .IP "Dangerous URLs"
1236 SCP URLs can contain raw commands within the scp: URL, which is a side effect
1237 of how the SCP protocol is designed. e.g.
1238 scp://user:pass@host/a;date >/tmp/test;
1239 Apps must not allow unsanitized SCP: URLs to be passed in for downloads.
1241 .IP "Denial of Service"
1242 A malicious server could cause libcurl to effectively hang by sending a
1243 trickle of data through, or even no data at all but just keeping the TCP
1244 connection open. This could result in a denial-of-service attack. The
1245 \fICURLOPT_TIMEOUT(3)\fP and/or \fICURLOPT_LOW_SPEED_LIMIT(3)\fP options can
1246 be used to mitigate against this.
1248 A malicious server could cause libcurl to effectively hang by starting to send
1249 data, then severing the connection without cleanly closing the TCP connection.
1250 The app could install a \fICURLOPT_SOCKOPTFUNCTION(3)\fP callback function and
1251 set the TCP SO_KEEPALIVE option to mitigate against this. Setting one of the
1252 timeout options would also work against this attack.
1254 A malicious server could cause libcurl to download an infinite amount of data,
1255 potentially causing all of memory or disk to be filled. Setting the
1256 \fICURLOPT_MAXFILESIZE_LARGE(3)\fP option is not sufficient to guard against
1257 this. Instead, the app should monitor the amount of data received within the
1258 write or progress callback and abort once the limit is reached.
1260 A malicious HTTP server could cause an infinite redirection loop, causing a
1261 denial-of-service. This can be mitigated by using the
1262 \fICURLOPT_MAXREDIRS(3)\fP option.
1264 .IP "Arbitrary Headers"
1265 User-supplied data must be sanitized when used in options like
1266 \fICURLOPT_USERAGENT(3)\fP, \fICURLOPT_HTTPHEADER(3)\fP,
1267 \fICURLOPT_POSTFIELDS(3)\fP and others that are used to generate structured
1268 data. Characters like embedded carriage returns or ampersands could allow the
1269 user to create additional headers or fields that could cause malicious
1272 .IP "Server-supplied Names"
1273 A server can supply data which the application may, in some cases, use as
1274 a file name. The curl command-line tool does this with --remote-header-name,
1275 using the Content-disposition: header to generate a file name. An application
1276 could also use CURLINFO_EFFECTIVE_URL to generate a file name from a
1277 server-supplied redirect URL. Special care must be taken to sanitize such
1278 names to avoid the possibility of a malicious server supplying one like
1279 "/etc/passwd", "\\autoexec.bat", "prn:" or even ".bashrc".
1281 .IP "Server Certificates"
1282 A secure application should never use the \fICURLOPT_SSL_VERIFYPEER(3)\fP
1283 option to disable certificate validation. There are numerous attacks that are
1284 enabled by apps that fail to properly validate server TLS/SSL certificates,
1285 thus enabling a malicious server to spoof a legitimate one. HTTPS without
1286 validated certificates is potentially as insecure as a plain HTTP connection.
1288 .IP "Showing What You Do"
1289 On a related issue, be aware that even in situations like when you have
1290 problems with libcurl and ask someone for help, everything you reveal in order
1291 to get best possible help might also impose certain security related
1292 risks. Host names, user names, paths, operating system specifics, etc. (not to
1293 mention passwords of course) may in fact be used by intruders to gain
1294 additional information of a potential target.
1296 Be sure to limit access to application logs if they could hold private or
1297 security-related data. Besides the obvious candidates like user names and
1298 passwords, things like URLs, cookies or even file names could also hold
1301 To avoid this problem, you must of course use your common sense. Often, you
1302 can just edit out the sensitive data or just search/replace your true
1303 information with faked data.
1305 .SH "The multi Interface"
1306 The easy interface as described in detail in this document is a synchronous
1307 interface that transfers one file at a time and doesn't return until it is
1310 The multi interface, on the other hand, allows your program to transfer
1311 multiple files in both directions at the same time, without forcing you to use
1312 multiple threads. The name might make it seem that the multi interface is for
1313 multi-threaded programs, but the truth is almost the reverse. The multi
1314 interface allows a single-threaded application to perform the same kinds of
1315 multiple, simultaneous transfers that multi-threaded programs can perform. It
1316 allows many of the benefits of multi-threaded transfers without the complexity
1317 of managing and synchronizing many threads.
1319 To complicate matters somewhat more, there are even two versions of the multi
1320 interface. The event based one, also called multi_socket and the "normal one"
1321 designed for using with select(). See the libcurl-multi.3 man page for details
1322 on the multi_socket event based API, this description here is for the select()
1325 To use this interface, you are better off if you first understand the basics
1326 of how to use the easy interface. The multi interface is simply a way to make
1327 multiple transfers at the same time by adding up multiple easy handles into
1330 You create the easy handles you want, one for each concurrent transfer, and
1331 you set all the options just like you learned above, and then you create a
1332 multi handle with \fIcurl_multi_init(3)\fP and add all those easy handles to
1333 that multi handle with \fIcurl_multi_add_handle(3)\fP.
1335 When you've added the handles you have for the moment (you can still add new
1336 ones at any time), you start the transfers by calling
1337 \fIcurl_multi_perform(3)\fP.
1339 \fIcurl_multi_perform(3)\fP is asynchronous. It will only perform what can be
1340 done now and then return back control to your program. It is designed to never
1341 block. You need to keep calling the function until all transfers are
1344 The best usage of this interface is when you do a select() on all possible
1345 file descriptors or sockets to know when to call libcurl again. This also
1346 makes it easy for you to wait and respond to actions on your own application's
1347 sockets/handles. You figure out what to select() for by using
1348 \fIcurl_multi_fdset(3)\fP, that fills in a set of fd_set variables for you
1349 with the particular file descriptors libcurl uses for the moment.
1351 When you then call select(), it'll return when one of the file handles signal
1352 action and you then call \fIcurl_multi_perform(3)\fP to allow libcurl to do
1353 what it wants to do. Take note that libcurl does also feature some time-out
1354 code so we advise you to never use very long timeouts on select() before you
1355 call \fIcurl_multi_perform(3)\fP again. \fIcurl_multi_timeout(3)\fP is
1356 provided to help you get a suitable timeout period.
1358 Another precaution you should use: always call \fIcurl_multi_fdset(3)\fP
1359 immediately before the select() call since the current set of file descriptors
1360 may change in any curl function invoke.
1362 If you want to stop the transfer of one of the easy handles in the stack, you
1363 can use \fIcurl_multi_remove_handle(3)\fP to remove individual easy
1364 handles. Remember that easy handles should be \fIcurl_easy_cleanup(3)\fPed.
1366 When a transfer within the multi stack has finished, the counter of running
1367 transfers (as filled in by \fIcurl_multi_perform(3)\fP) will decrease. When
1368 the number reaches zero, all transfers are done.
1370 \fIcurl_multi_info_read(3)\fP can be used to get information about completed
1371 transfers. It then returns the CURLcode for each easy transfer, to allow you
1372 to figure out success on each individual transfer.
1374 .SH "SSL, Certificates and Other Tricks"
1376 [ seeding, passwords, keys, certificates, ENGINE, ca certs ]
1378 .SH "Sharing Data Between Easy Handles"
1379 You can share some data between easy handles when the easy interface is used,
1380 and some data is share automatically when you use the multi interface.
1382 When you add easy handles to a multi handle, these easy handles will
1383 automatically share a lot of the data that otherwise would be kept on a
1384 per-easy handle basis when the easy interface is used.
1386 The DNS cache is shared between handles within a multi handle, making
1387 subsequent name resolving faster, and the connection pool that is kept to
1388 better allow persistent connections and connection re-use is also shared. If
1389 you're using the easy interface, you can still share these between specific
1390 easy handles by using the share interface, see \fIlibcurl-share(3)\fP.
1392 Some things are never shared automatically, not within multi handles, like for
1393 example cookies so the only way to share that is with the share interface.
1397 libcurl 7.10.3 and later have the ability to switch over to chunked
1398 Transfer-Encoding in cases where HTTP uploads are done with data of an unknown
1401 This happens on Windows machines when libcurl is built and used as a
1402 DLL. However, you can still do this on Windows if you link with a static
1405 The curl-config tool is generated at build-time (on Unix-like systems) and
1406 should be installed with the 'make install' or similar instruction that
1407 installs the library, header files, man pages etc.
1409 This behavior was different in versions before 7.17.0, where strings had to
1410 remain valid past the end of the \fIcurl_easy_setopt(3)\fP call.
1412 .BR libcurl-errors "(3), " libcurl-multi "(3), " libcurl-easy "(3) "