Things that could be nice to do in the future
- Things to do in project cURL. Please tell us what you think, contribute and
+ Things to do in project curl. Please tell us what you think, contribute and
send us patches that improve things!
Be aware that these are things that we could do, or have once been considered
1.5 get rid of PATH_MAX
1.6 Modified buffer size approach
1.7 Detect when called from within callbacks
- 1.8 Allow SSL (HTTPS) to proxy
+ 1.8 CURLOPT_RESOLVE for any port number
1.9 Cache negative name resolves
- 1.10 Support IDNA2008
1.11 minimize dependencies with dynamicly loaded modules
1.12 have form functions use CURL handle argument
1.14 Typesafe curl_easy_setopt()
1.20 SRV and URI DNS records
1.21 API for URL parsing/splitting
1.23 Offer API to flush the connection pool
+ 1.24 TCP Fast Open for windows
+ 1.25 Remove the generated include file
2. libcurl - multi interface
2.1 More non-blocking
5.1 Better persistency for HTTP 1.0
5.2 support FF3 sqlite cookie files
5.3 Rearrange request header order
- 5.4 Use huge HTTP/2 windows
+ 5.4 HTTP Digest using SHA-256
5.5 auth= in URLs
5.6 Refuse "downgrade" redirects
5.7 Brotli compression
5.8 QUIC
+ 5.9 Improve formpost API
+ 5.10 Leave secure cookies alone
+ 5.11 Chunked transfer multipart formpost
+ 5.12 OPTIONS *
6. TELNET
6.1 ditch stdin
13.1 Disable specific versions
13.2 Provide mutex locking API
13.3 Evaluate SSL patches
- 13.4 Cache OpenSSL contexts
+ 13.4 Cache/share OpenSSL contexts
13.5 Export session ids
13.6 Provide callback for cert verification
13.7 improve configure --with-ssl
13.8 Support DANE
- 13.9 Support TLS v1.3
+ 13.10 Support SSLKEYLOGFILE
+ 13.11 Support intermediate & root pinning for PINNEDPUBLICKEY
+ 13.12 Support HSTS
+ 13.13 Support HPKP
14. GnuTLS
14.1 SSL engine stuff
17. SSH protocols
17.1 Multiplexing
17.2 SFTP performance
+ 17.3 Support better than MD5 hostkey hash
18. Command line tool
18.1 sync
18.12 keep running, read instructions from pipe/socket
18.13 support metalink in http headers
18.14 --fail without --location should treat 3xx as a failure
+ 18.15 --retry should resume
+ 18.16 send only part of --data
+ 18.17 consider file name from the redirected URL with -O ?
19. Build
19.1 roffit
+ 19.2 Enable PIE and RELRO by default
20. Test suite
20.1 SSL tunnel
subsequently can add code within libcurl that returns error if called within
callbacks for when that's not supported.
-1.8 Allow SSL (HTTPS) to proxy
+1.8 CURLOPT_RESOLVE for any port number
- To prevent local users from snooping on your traffic to the proxy. Supported
- by Firefox and Chrome already:
- https://www.chromium.org/developers/design-documents/secure-web-proxy
+ This option allows applications to set a replacement IP address for a given
+ host + port pair. Consider making support for providing a replacement address
+ for the host name on all port numbers.
- See this stale work in progress branch:
- https://github.com/curl/curl/tree/HTTPS-proxy based on this PR:
- https://github.com/curl/curl/pull/305
+ See https://github.com/curl/curl/issues/1264
1.9 Cache negative name resolves
A name resolve that has failed is likely to fail when made again within a
short period of time. Currently we only cache positive responses.
-1.10 Support IDNA2008
-
- International Domain Names are supported in libcurl since years back, powered
- by libidn. libidn implements IDNA2003 which has been superseded by IDNA2008.
- libidn2 is an existing library offering support for IDNA2008.
-
1.11 minimize dependencies with dynamicly loaded modules
We can create a system with loadable modules/plug-ins, where these modules
An API could allow a forced flush or just a forced loop that would properly
close all connections that have been closed by the server already.
+1.24 TCP Fast Open for windows
+
+ libcurl supports the CURLOPT_TCP_FASTOPEN option since 7.49.0 for Linux and
+ Mac OS. Windows supports TCP Fast Open starting with Windows 10, version 1607
+ and we should add support for it.
+
+1.25 Remove the generated include file
+
+ When curl and libcurl are built, one of the public include files are
+ generated and is populated with a set of defines that are derevid from sizes
+ and constants for the particular target architecture that build is made. For
+ platforms that can select between 32 bit and 64 bit at build time, this
+ approach makes the libcurl build only create a set of public headers suitable
+ for one of the architectures and not both. If you build libcurl for such a
+ platform and you want to allow applications to get built using either 32/64
+ version, you must generate the libcurl headers once for each setup and you
+ must then add a replacement curl header that would itself select the correct
+ 32 or 64 bit specific header as necessary.
+
+ Your curl/curl.h alternative could then look like (replace with suitable CPP
+ variable to check):
+
+ #ifdef ARCH_32bit
+ #include <curl32/curl.h>
+ #else /* ARCH_64bit */
+ #include <curl64/curl.h>
+ #endif
+
+ A fix would either (A) fix the 32/64 setup automatically or even better (B)
+ work away the architecture specific defines from the headers so that they can
+ be used for all architectures independently of what libcurl was built for.
+
2. libcurl - multi interface
Make sure we don't ever loop because of non-blocking sockets returning
EWOULDBLOCK or similar. Blocking cases include:
- - Name resolves on non-windows unless c-ares is used
- - NSS SSL connections
+ - Name resolves on non-windows unless c-ares or the threaded resolver is used
- HTTP proxy CONNECT operations
- SOCKS proxy handshakes
- file:// transfers
headers use a default value so only headers that need to be moved have to be
specified.
-5.4 Use huge HTTP/2 windows
+5.4 HTTP Digest using SHA-256
+
+ RFC 7616 introduces an update to the HTTP Digest authentication
+ specification, which amongst other thing defines how new digest algorithms
+ can be used instead of MD5 which is considered old and not recommanded.
- We're currently using nghttp2's default window size which is terribly small
- (64K). This becomes a bottle neck over high bandwidth networks. We should
- instead make the window size to be very big (512MB?) as we really don't do
- much flow control anyway.
+ See https://tools.ietf.org/html/rfc7616 and
+ https://github.com/curl/curl/issues/1018
5.5 auth= in URLs
5.7 Brotli compression
- Compression algorithms that perform better than gzip are being considered for
- use and inclusion in existing browsers. For example 'brotli'. If servers
- follow along it is a good reason for us to also allow users to take advantage
- of this. The algorithm: https://github.com/google/brotli The Firefox bug:
- https://bugzilla.mozilla.org/show_bug.cgi?id=366559
+ Brotli compression performs better than gzip and is being implemented by
+ browsers and servers widely. The algorithm: https://github.com/google/brotli
+ The Firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=366559
5.8 QUIC
implemented. This, to allow other projects to benefit from the work and to
thus broaden the interest and chance of others to participate.
+5.9 Improve formpost API
+
+ Revamp the formpost API and making something that is easier to use and
+ understand:
+
+ https://github.com/curl/curl/wiki/formpost-API-redesigned
+
+5.10 Leave secure cookies alone
+
+ Non-secure origins (HTTP sites) should not be allowed to set or modify
+ cookies with the 'secure' property:
+
+ https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-01
+
+5.11 Chunked transfer multipart formpost
+
+ For a case where the file is being made during the upload is progressing
+ (like passed on stdin to the curl tool), we cannot know the size before-hand
+ and we rather not read the entire thing into memory before it can start the
+ upload.
+
+ https://github.com/curl/curl/issues/1139
+
+5.12 OPTIONS *
+
+ HTTP defines an OPTIONS method that can be sent with an asterisk option like
+ "OPTIONS *" to ask about options from the server and not a specific URL
+ resource. https://tools.ietf.org/html/rfc7230#section-5.3.4
+
+ libcurl as it currently works will always sent HTTP methods with a path that
+ starts with a slash so there's no way for an application to send a proper
+ "OPTIONS *" using libcurl. This should be fixed.
+
+ I can't think of any other non-slash paths we should support so it will
+ probably make sense to add a new boolean option for issuign an "OPTIONS *"
+ request. CURLOPT_OPTIONSASTERISK perhaps (and a corresponding command line
+ option)?
+
+ See https://github.com/curl/curl/issues/1280
+
6. TELNET
Evaluate/apply Gertjan van Wingerde's SSL patches:
https://curl.haxx.se/mail/lib-2004-03/0087.html
-13.4 Cache OpenSSL contexts
+13.4 Cache/share OpenSSL contexts
"Look at SSL cafile - quick traces look to me like these are done on every
request as well, when they should only be necessary once per SSL context (or
style connections are re-used. It will make us use slightly more memory but
it will libcurl do less creations and deletions of SSL contexts.
+ Technically, the "caching" is probably best implemented by getting added to
+ the share interface so that easy handles who want to and can reuse the
+ context specify that by sharing with the right properties set.
+
+ https://github.com/curl/curl/issues/1110
+
13.5 Export session ids
Add an interface to libcurl that enables "session IDs" to get
Björn Stenberg wrote a separate initial take on DANE that was never
completed.
-13.9 Support TLS v1.3
+13.10 Support SSLKEYLOGFILE
- TLS version 1.3 is about to ship and is getting implemented by TLS libraries
- as we speak. We should start to support the symbol and make sure all backends
- handle it accordingly, then gradually add support as the TLS libraries add
- the corresponding support. There may be a need to add some additional options
- to allow libcurl to take advantage of the new features in 1.3.
+ When used, Firefox and Chrome dumps their master TLS keys to the file name
+ this environment variable specifies. This allows tools like for example
+ Wireshark to capture and decipher TLS traffic to/from those clients. libcurl
+ could be made to support this more widely (presumably this already works when
+ built with NSS). Peter Wu made a OpenSSL preload to make possible that can be
+ used as inspiration and guidance
+ https://git.lekensteyn.nl/peter/wireshark-notes/tree/src/sslkeylog.c
+13.11 Support intermediate & root pinning for PINNEDPUBLICKEY
+
+ CURLOPT_PINNEDPUBLICKEY does not consider the hashes of intermediate & root
+ certificates when comparing the pinned keys. Therefore it is not compatible
+ with "HTTP Public Key Pinning" as there also intermediate and root certificates
+ can be pinned. This is very useful as it prevents webadmins from "locking
+ themself out of their servers".
+
+ Adding this feature would make curls pinning 100% compatible to HPKP and allow
+ more flexible pinning.
+
+13.12 Support HSTS
+
+ "HTTP Strict Transport Security" is TOFU (trust on first use), time-based
+ features indicated by a HTTP header send by the webserver. It is widely used
+ in browsers and it's purpose is to prevent insecure HTTP connections after
+ a previous HTTPS connection. It protects against SSLStripping attacks.
+
+ Doc: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
+ RFC 6797: https://tools.ietf.org/html/rfc6797
+
+13.13 Support HPKP
+
+ "HTTP Public Key Pinning" is TOFU (trust on first use), time-based
+ features indicated by a HTTP header send by the webserver. It's purpose is
+ to prevent Man-in-the-middle attacks by trusted CAs by allowing webadmins
+ to specify which CAs/certificates/public keys to trust when connection to
+ their websites.
+
+ It can be build based on PINNEDPUBLICKEY.
+
+ Wikipedia: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
+ OWASP: https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
+ Doc: https://developer.mozilla.org/de/docs/Web/Security/Public_Key_Pinning
+ RFC: https://tools.ietf.org/html/draft-ietf-websec-key-pinning-21
14. GnuTLS
libcurl's SFTP transfer performance is sub par and can be improved, mostly by
the approach mentioned in "1.6 Modified buffer size approach".
+17.3 Support better than MD5 hostkey hash
+
+ libcurl offers the CURLOPT_SSH_HOST_PUBLIC_KEY_MD5 option for verifying the
+ server's key. MD5 is generally being deprecated so we should implement
+ support for stronger hashing algorithms. libssh2 itself is what provides this
+ underlying functionality and it supports at least SHA-1 as an alternative.
+ SHA-1 is also being deprecated these days so we should consider workign with
+ libssh2 to instead offer support for SHA-256 or similar.
+
+
18. Command line tool
18.1 sync
18.3 prevent file overwriting
- Add an option that prevents cURL from overwriting existing local files. When
+ Add an option that prevents curl from overwriting existing local files. When
used, and there already is an existing file with the target file name
(either -O or -o), a number should be appended (and increased if already
existing). So that index.html becomes first index.html.1 and then
way to implement this is probably to add that new logic in the command line
tool only and not in the underlying CURLOPT_FAILONERROR logic.
+18.15 --retry should resume
+
+ When --retry is used and curl actually retries transfer, it should use the
+ already transfered data and do a resumed transfer for the rest (when
+ possible) so that it doesn't have to transfer the same data again that was
+ already tranfered before the retry.
+
+ See https://github.com/curl/curl/issues/1084
+
+18.16 send only part of --data
+
+ When the user only wants to send a small piece of the data provided with
+ --data or --data-binary, like when that data is a huge file, consider a way
+ to specify that curl should only send a piece of that. One suggested syntax
+ would be: "--data-binary @largefile.zip!1073741823-2147483647".
+
+ See https://github.com/curl/curl/issues/1200
+
+18.17 consider file name from the redirected URL with -O ?
+
+ When a user gives a URL and uses -O, and curl follows a redirect to a new
+ URL, the file name is not extracted and used from the newly redirected-to URL
+ even if the new URL may have a much more sensible file name.
+
+ This is clearly documented and helps for security since there's no surprise
+ to users which file name that might get overwritten. But maybe a new option
+ could allow for this or maybe -J should imply such a treatment as well as -J
+ already allows for the server to decide what file name to use so it already
+ provides the "may overwrite any file" risk.
+
+ This is extra tricky if the original URL has no file name part at all since
+ then the current code path will error out with an error message, and we can't
+ *know* already at that point if curl will be redirected to a URL that has a
+ file name...
+
+ See https://github.com/curl/curl/issues/1241
19. Build
Consider extending 'roffit' to produce decent ASCII output, and use that
instead of (g)nroff when building src/tool_hugehelp.c
+19.2 Enable PIE and RELRO by default
+
+ Especially when having programs that execute curl via the command line, PIE
+ renders the exploitation of memory corruption vulnerabilities a lot more
+ difficult. This can be attributed to the additional information leaks being
+ required to conduct a successful attack. RELRO, on the other hand, masks
+ different binary sections like the GOT as read-only and thus kills a handful
+ of techniques that come in handy when attackers are able to arbitrarily
+ overwrite memory. A few tests showed that enabling these features had close
+ to no impact, neither on the performance nor on the general functionality of
+ curl.
+
+
20. Test suite
20.1 SSL tunnel
projects / platform / upstream / curl.git / blobdiff