5 \___|\___/|_| \_\_____|
9 The project is split in two. The library and the client. The client part uses
10 the library, but the library is designed to allow other applications to use
13 The largest amount of code and complexity is in the library part.
17 All changes to the sources are committed to the git repository as soon as
18 they're somewhat verified to work. Changes shall be committed as independently
19 as possible so that individual changes can be easier spotted and tracked
22 Tagging shall be used extensively, and by the time we release new archives we
23 should tag the sources with a name similar to the released version number.
28 We write curl and libcurl to compile with C89 compilers. On 32bit and up
29 machines. Most of libcurl assumes more or less POSIX compliance but that's
32 We write libcurl to build and work with lots of third party tools, and we
33 want it to remain functional and buildable with these and later versions
34 (older versions may still work but is not what we work hard to maintain):
50 On systems where configure runs, we aim at working on them all - if they have
51 a suitable C compiler. On systems that don't run configure, we strive to keep
60 When writing code (mostly for generating stuff included in release tarballs)
61 we use a few "build tools" and we make sure that we remain functional with
66 GNU Automake 1.7 (we currently avoid 1.10 due to Solaris-related bugs)
70 groff ? (any version that supports "groff -Tps -man [in] [out]")
76 There are a few differences in how to program curl the unix way compared to
77 the Windows way. The four perhaps most notable details are:
79 1. Different function names for socket operations.
81 In curl, this is solved with defines and macros, so that the source looks
82 the same at all places except for the header file that defines them. The
83 macros in use are sclose(), sread() and swrite().
85 2. Windows requires a couple of init calls for the socket stuff.
87 That's taken care of by the curl_global_init() call, but if other libs also
88 do it etc there might be reasons for applications to alter that behaviour.
90 3. The file descriptors for network communication and file operations are
91 not easily interchangeable as in unix.
93 We avoid this by not trying any funny tricks on file descriptors.
95 4. When writing data to stdout, Windows makes end-of-lines the DOS way, thus
96 destroying binary data, although you do want that conversion if it is
97 text coming through... (sigh)
99 We set stdout to binary under windows
101 Inside the source code, We make an effort to avoid '#ifdef [Your OS]'. All
102 conditionals that deal with features *should* instead be in the format
103 '#ifdef HAVE_THAT_WEIRD_FUNCTION'. Since Windows can't run configure scripts,
104 we maintain a curl_config-win32.h file in lib directory that is supposed to
105 look exactly as a curl_config.h file would have looked like on a Windows
108 Generally speaking: always remember that this will be compiled on dozens of
109 operating systems. Don't walk on the edge.
114 There are plenty of entry points to the library, namely each publicly defined
115 function that libcurl offers to applications. All of those functions are
116 rather small and easy-to-follow. All the ones prefixed with 'curl_easy' are
117 put in the lib/curl_easy.c file.
119 curl_global_init_() and curl_global_cleanup() should be called by the
120 application to initialize and clean up global stuff in the library. As of
121 today, it can handle the global SSL initing if SSL is enabled and it can init
122 the socket layer on windows machines. libcurl itself has no "global" scope.
124 All printf()-style functions use the supplied clones in lib/curl_mprintf.c.
125 This makes sure we stay absolutely platform independent.
127 curl_easy_init() allocates an internal struct and makes some initializations.
128 The returned handle does not reveal internals. This is the 'SessionHandle'
129 struct which works as an "anchor" struct for all curl_easy functions. All
130 connections performed will get connect-specific data allocated that should be
131 used for things related to particular connections/requests.
133 curl_easy_setopt() takes three arguments, where the option stuff must be
134 passed in pairs: the parameter-ID and the parameter-value. The list of
135 options is documented in the man page. This function mainly sets things in
136 the 'SessionHandle' struct.
138 curl_easy_perform() does a whole lot of things:
140 It starts off in the lib/curl_easy.c file by calling Curl_perform() and the
141 main work then continues in lib/curl_url.c. The flow continues with a call to
142 Curl_connect() to connect to the remote site.
146 ... analyzes the URL, it separates the different components and connects to
147 the remote host. This may involve using a proxy and/or using SSL. The
148 Curl_resolv() function in lib/curl_hostip.c is used for looking up host
149 names (it does then use the proper underlying method, which may vary
150 between platforms and builds).
152 When Curl_connect is done, we are connected to the remote site. Then it is
153 time to tell the server to get a document/file. Curl_do() arranges this.
155 This function makes sure there's an allocated and initiated 'connectdata'
156 struct that is used for this particular connection only (although there may
157 be several requests performed on the same connect). A bunch of things are
158 inited/inherited from the SessionHandle struct.
162 Curl_do() makes sure the proper protocol-specific function is called. The
163 functions are named after the protocols they handle. Curl_ftp(),
164 Curl_http(), Curl_dict(), etc. They all reside in their respective files
165 (curl_ftp.c, curl_http.c and curl_dict.c). HTTPS is handled by Curl_http()
166 and FTPS by Curl_ftp().
168 The protocol-specific functions of course deal with protocol-specific
169 negotiations and setup. They have access to the Curl_sendf() (from
170 lib/curl_sendf.c) function to send printf-style formatted data to the
171 remote host and when they're ready to make the actual file transfer they
172 call the Curl_Transfer() function (in lib/curl_transfer.c) to setup the
173 transfer and returns.
175 If this DO function fails and the connection is being re-used, libcurl will
176 then close this connection, setup a new connection and re-issue the DO
177 request on that. This is because there is no way to be perfectly sure that
178 we have discovered a dead connection before the DO function and thus we
179 might wrongly be re-using a connection that was closed by the remote peer.
181 Some time during the DO function, the Curl_setup_transfer() function must
182 be called with some basic info about the upcoming transfer: what socket(s)
183 to read/write and the expected file transfer sizes (if known).
187 Curl_perform() then calls Transfer() in lib/curl_transfer.c that performs
188 the entire file transfer.
190 During transfer, the progress functions in lib/curl_progress.c are called
191 at a frequent interval (or at the user's choice, a specified callback
192 might get called). The speedcheck functions in lib/curl_speedcheck.c are
193 also used to verify that the transfer is as fast as required.
197 Called after a transfer is done. This function takes care of everything
198 that has to be done after a transfer. This function attempts to leave
199 matters in a state so that Curl_do() should be possible to call again on
200 the same connection (in a persistent connection case). It might also soon
201 be closed with Curl_disconnect().
205 When doing normal connections and transfers, no one ever tries to close any
206 connections so this is not normally called when curl_easy_perform() is
207 used. This function is only used when we are certain that no more transfers
208 is going to be made on the connection. It can be also closed by force, or
209 it can be called to make sure that libcurl doesn't keep too many
210 connections alive at the same time (there's a default amount of 5 but that
211 can be changed with the CURLOPT_MAXCONNECTS option).
213 This function cleans up all resources that are associated with a single
216 Curl_perform() is the function that does the main "connect - do - transfer -
217 done" loop. It loops if there's a Location: to follow.
219 When completed, the curl_easy_cleanup() should be called to free up used
220 resources. It runs Curl_disconnect() on all open connections.
222 A quick roundup on internal function sequences (many of these call
223 protocol-specific function-pointers):
225 Curl_connect - connects to a remote site and does initial connect fluff
226 This also checks for an existing connection to the requested site and uses
227 that one if it is possible.
229 Curl_do - starts a transfer
230 Curl_handler::do_it() - transfers data
231 Curl_done - ends a transfer
233 Curl_disconnect - disconnects from a remote site. This is called when the
234 disconnect is really requested, which doesn't necessarily have to be
235 exactly after curl_done in case we want to keep the connection open for
240 HTTP offers a lot and is the protocol in curl that uses the most lines of
241 code. There is a special file (lib/curl_formdata.c) that offers all the
242 multipart post functions.
244 base64-functions for user+password stuff (and more) is in (lib/curl_base64.c)
245 and all functions for parsing and sending cookies in (lib/curl_cookie.c).
247 HTTPS uses in almost every means the same procedure as HTTP, with only two
248 exceptions: the connect procedure is different and the function used to read
249 or write from the socket is different, although the latter fact is hidden in
250 the source by the use of Curl_read() for reading and Curl_write() for writing
251 data to the remote server.
253 curl_http_chunks.c contains functions that understands HTTP 1.1 chunked
256 An interesting detail with the HTTP(S) request, is the Curl_add_buffer()
257 series of functions we use. They append data to one single buffer, and when
258 the building is done the entire request is sent off in one single write. This
259 is done this way to overcome problems with flawed firewalls and lame servers.
263 The Curl_if2ip() function can be used for getting the IP number of a
264 specified network interface, and it resides in lib/curl_if2ip.c.
266 Curl_ftpsendf() is used for sending FTP commands to the remote server. It was
267 made a separate function to prevent us programmers from forgetting that they
268 must be CRLF terminated. They must also be sent in one single write() to make
269 firewalls and similar happy.
273 The kerberos support is mainly in lib/curl_krb4.c and lib/curl_security.c.
277 Telnet is implemented in lib/curl_telnet.c.
281 The file:// protocol is dealt with in lib/curl_file.c.
285 Everything LDAP is in lib/curl_ldap.c and lib/curl_openldap.c
289 URL encoding and decoding, called escaping and unescaping in the source code,
290 is found in lib/curl_escape.c.
292 While transferring data in Transfer() a few functions might get used.
293 curl_getdate() in lib/curl_parsedate.c is for HTTP date comparisons (and
296 lib/curl_getenv.c offers curl_getenv() which is for reading environment
297 variables in a neat platform independent way. That's used in the client,
298 but also in lib/curl_url.c when checking the proxy environment variables.
299 Note that contrary to the normal unix getenv(), this returns an allocated
300 buffer that must be free()ed after use.
302 lib/curl_netrc.c holds the .netrc parser
304 lib/curl_timeval.c features replacement functions for systems that don't have
305 gettimeofday() and a few support functions for timeval conversions.
307 A function named curl_version() that returns the full curl version string is
308 found in lib/curl_version.c.
310 Persistent Connections
311 ======================
313 The persistent connection support in libcurl requires some considerations on
314 how to do things inside of the library.
316 o The 'SessionHandle' struct returned in the curl_easy_init() call must never
317 hold connection-oriented data. It is meant to hold the root data as well as
318 all the options etc that the library-user may choose.
319 o The 'SessionHandle' struct holds the "connection cache" (an array of
320 pointers to 'connectdata' structs). There's one connectdata struct
321 allocated for each connection that libcurl knows about. Note that when you
322 use the multi interface, the multi handle will hold the connection cache
323 and not the particular easy handle. This of course to allow all easy handles
324 in a multi stack to be able to share and re-use connections.
325 o This enables the 'curl handle' to be reused on subsequent transfers.
326 o When we are about to perform a transfer with curl_easy_perform(), we first
327 check for an already existing connection in the cache that we can use,
328 otherwise we create a new one and add to the cache. If the cache is full
329 already when we add a new connection, we close one of the present ones. We
330 select which one to close dependent on the close policy that may have been
332 o When the transfer operation is complete, we try to leave the connection
333 open. Particular options may tell us not to, and protocols may signal
334 closure on connections and then we don't keep it open of course.
335 o When curl_easy_cleanup() is called, we close all still opened connections,
336 unless of course the multi interface "owns" the connections.
338 You do realize that the curl handle must be re-used in order for the
339 persistent connections to work.
341 multi interface/non-blocking
342 ============================
344 We make an effort to provide a non-blocking interface to the library, the
345 multi interface. To make that interface work as good as possible, no
346 low-level functions within libcurl must be written to work in a blocking
349 One of the primary reasons we introduced c-ares support was to allow the name
350 resolve phase to be perfectly non-blocking as well.
352 The ultimate goal is to provide the easy interface simply by wrapping the
353 multi interface functions and thus treat everything internally as the multi
354 interface is the single interface we have.
356 The FTP and the SFTP/SCP protocols are thus perfect examples of how we adapt
357 and adjust the code to allow non-blocking operations even on multi-stage
358 protocols. They are built around state machines that return when they could
359 block waiting for data. The DICT, LDAP and TELNET protocols are crappy
360 examples and they are subject for rewrite in the future to better fit the
361 libcurl protocol family.
366 Originally libcurl supported SSLeay for SSL/TLS transports, but that was then
367 extended to its successor OpenSSL but has since also been extended to several
368 other SSL/TLS libraries and we expect and hope to further extend the support
369 in future libcurl versions.
371 To deal with this internally in the best way possible, we have a generic SSL
372 function API as provided by the sslgen.[ch] system, and they are the only SSL
373 functions we must use from within libcurl. sslgen is then crafted to use the
374 appropriate lower-level function calls to whatever SSL library that is in
380 All symbols used internally in libcurl must use a 'Curl_' prefix if they're
381 used in more than a single file. Single-file symbols must be made static.
382 Public ("exported") symbols must use a 'curl_' prefix. (There are exceptions,
383 but they are to be changed to follow this pattern in future versions.) Public
384 API functions are marked with CURL_EXTERN in the public header files so that
385 all others can be hidden on platforms where this is possible.
387 Return Codes and Informationals
388 ===============================
390 I've made things simple. Almost every function in libcurl returns a CURLcode,
391 that must be CURLE_OK if everything is OK or otherwise a suitable error code
392 as the curl/curl.h include file defines. The very spot that detects an error
393 must use the Curl_failf() function to set the human-readable error
396 In aiding the user to understand what's happening and to debug curl usage, we
397 must supply a fair amount of informational messages by using the Curl_infof()
398 function. Those messages are only displayed when the user explicitly asks for
399 them. They are best used when revealing information that isn't otherwise
405 We make an effort to not export or show internals or how internals work, as
406 that makes it easier to keep a solid API/ABI over time. See docs/libcurl/ABI
407 for our promise to users.
412 main() resides in src/tool_main.c together with most of the client code.
414 src/tool_hugehelp.c is automatically generated by the mkhelp.pl perl script
415 to display the complete "manual" and the src/tool_urlglob.c file holds the
416 functions used for the URL-"globbing" support. Globbing in the sense that
417 the {} and [] expansion stuff is there.
419 The client mostly messes around to setup its 'config' struct properly, then
420 it calls the curl_easy_*() functions of the library and when it gets back
421 control after the curl_easy_perform() it cleans up the library, checks status
424 When the operation is done, the ourWriteOut() function in
425 src/tool_writeout.c may be called to report about the operation. That
426 function is using the curl_easy_getinfo() function to extract useful
427 information from the curl session.
429 Recent versions may loop and do all this several times if many URLs were
430 specified on the command line or config file.
435 The file lib/curl_memdebug.c contains debug-versions of a few functions.
436 Functions such as malloc, free, fopen, fclose, etc that somehow deal with
437 resources that might give us problems if we "leak" them. The functions in
438 the memory tracking system do nothing fancy, they do their normal function
439 and then log information about what they just did. The logged data can then
440 be analyzed after a complete session,
442 memanalyze.pl is the perl script present in tests/ that analyzes a log file
443 generated by the memory tracking system. It detects if resources are
444 allocated but never freed and other kinds of errors related to resource
447 Internally, definition of preprocessor symbol DEBUGBUILD restricts code which
448 is only compiled for debug enabled builds. And symbol CURLDEBUG is used to
449 differentiate code which is _only_ used for memory tracking/debugging.
451 Use -DCURLDEBUG when compiling to enable memory debugging, this is also
452 switched on by running configure with --enable-curldebug. Use -DDEBUGBUILD
453 when compiling to enable a debug build or run configure with --enable-debug.
455 curl --version will list 'Debug' feature for debug enabled builds, and
456 will list 'TrackMemory' feature for curl debug memory tracking capable
457 builds. These features are independent and can be controlled when running
458 the configure script. When --enable-debug is given both features will be
459 enabled, unless some restriction prevents memory tracking from being used.
464 The test suite is placed in its own subdirectory directly off the root in the
465 curl archive tree, and it contains a bunch of scripts and a lot of test case
468 The main test script is runtests.pl that will invoke test servers like
469 httpserver.pl and ftpserver.pl before all the test cases are performed. The
470 test suite currently only runs on unix-like platforms.
472 You'll find a description of the test suite in the tests/README file, and the
473 test case data files in the tests/FILEFORMAT file.
475 The test suite automatically detects if curl was built with the memory
476 debugging enabled, and if it was it will detect memory leaks, too.
481 There's no magic to this. When you consider everything stable enough to be
484 1. Tag the source code accordingly.
486 2. run the 'maketgz' script (using 'make distcheck' will give you a pretty
487 good view on the status of the current sources). maketgz requires a
488 version number and creates the release archive. maketgz uses 'make dist'
489 for the actual archive building, why you need to fill in the Makefile.am
490 files properly for which files that should be included in the release
493 3. When that's complete, sign the output files.
497 5. Update web site and changelog on site
499 6. Send announcement to the mailing lists
501 NOTE: you must have curl checked out from git to be able to do a proper
502 release build. The release tarballs do not have everything setup in order to
503 do releases properly.
projects / platform / upstream / curl.git / blob