Things that could be nice to do in the future
- Things to do in project cURL. Please tell us what you think, contribute and
+ Things to do in project curl. Please tell us what you think, contribute and
send us patches that improve things!
Be aware that these are things that we could do, or have once been considered
All bugs documented in the KNOWN_BUGS document are subject for fixing!
1. libcurl
+ 1.1 Option to refuse usernames in URLs
1.2 More data sharing
1.3 struct lifreq
1.4 signal-based resolver timeouts
1.5 get rid of PATH_MAX
1.6 Modified buffer size approach
- 1.7 Detect when called from within callbacks
- 1.8 Allow SSL (HTTPS) to proxy
+ 1.7 Support HTTP/2 for HTTP(S) proxies
+ 1.8 CURLOPT_RESOLVE for any port number
1.9 Cache negative name resolves
- 1.10 Support IDNA2008
- 1.11 minimize dependencies with dynamicly loaded modules
- 1.12 have form functions use CURL handle argument
+ 1.10 auto-detect proxy
+ 1.11 minimize dependencies with dynamically loaded modules
+ 1.12 updated DNS server while running
+ 1.13 DNS-over-HTTPS
1.14 Typesafe curl_easy_setopt()
1.15 Monitor connections in the connection pool
1.16 Try to URL encode given URL
1.20 SRV and URI DNS records
1.21 API for URL parsing/splitting
1.23 Offer API to flush the connection pool
+ 1.24 TCP Fast Open for windows
+ 1.25 Expose tried IP addresses that failed
+ 1.26 CURL_REFUSE_CLEARTEXT
+ 1.27 hardcode the "localhost" addresses
2. libcurl - multi interface
2.1 More non-blocking
2.5 Edge-triggered sockets should work
3. Documentation
- 3.1 Update date and version in man pages
3.2 Provide cmake config-file
4. FTP
4.5 ASCII support
4.6 GSSAPI via Windows SSPI
4.7 STAT for LIST without data connection
+ 4.8 Option to ignore private IP addresses in PASV response
5. HTTP
5.1 Better persistency for HTTP 1.0
5.2 support FF3 sqlite cookie files
5.3 Rearrange request header order
- 5.4 Use huge HTTP/2 windows
5.5 auth= in URLs
5.6 Refuse "downgrade" redirects
- 5.7 Brotli compression
- 5.8 QUIC
+ 5.7 QUIC
+ 5.8 Leave secure cookies alone
6. TELNET
6.1 ditch stdin
6.2 ditch telnet-specific select
6.3 feature negotiation debug data
- 6.4 send data in chunks
7. SMTP
7.1 Pipelining
13. SSL
13.1 Disable specific versions
13.2 Provide mutex locking API
- 13.3 Evaluate SSL patches
- 13.4 Cache OpenSSL contexts
+ 13.3 Support in-memory certs/ca certs/keys
+ 13.4 Cache/share OpenSSL contexts
13.5 Export session ids
13.6 Provide callback for cert verification
13.7 improve configure --with-ssl
13.8 Support DANE
- 13.9 Support TLS v1.3
+ 13.11 Support intermediate & root pinning for PINNEDPUBLICKEY
+ 13.12 Support HSTS
+ 13.13 Support HPKP
14. GnuTLS
14.1 SSL engine stuff
17. SSH protocols
17.1 Multiplexing
17.2 SFTP performance
+ 17.3 Support better than MD5 hostkey hash
+ 17.4 Support CURLOPT_PREQUOTE
18. Command line tool
18.1 sync
18.2 glob posts
18.3 prevent file overwriting
18.4 simultaneous parallel transfers
- 18.5 provide formpost headers
+ 18.5 UTF-8 filenames in Content-Disposition
18.6 warning when setting an option
- 18.7 warning when sending binary output to terminal
+ 18.7 warning if curl version is not in sync with libcurl version
18.8 offer color-coded HTTP header output
18.9 Choose the name of file in braces for complex URLs
18.10 improve how curl works in a windows console window
18.12 keep running, read instructions from pipe/socket
18.13 support metalink in http headers
18.14 --fail without --location should treat 3xx as a failure
+ 18.15 --retry should resume
+ 18.16 send only part of --data
+ 18.17 consider file name from the redirected URL with -O ?
+ 18.18 retry on network is unreachable
19. Build
19.1 roffit
+ 19.2 Enable PIE and RELRO by default
20. Test suite
20.1 SSL tunnel
1. libcurl
+1.1 Option to refuse usernames in URLs
+
+ There's a certain risk for application in allowing user names in URLs. For
+ example: if the wrong person gets to set the URL and manages to set a user
+ name in there when .netrc is used, the application may send along a password
+ that otherwise the person couldn't provide.
+
+ A new libcurl option could be added to allow applications to switch off this
+ feature and thus avoid a potential risk.
+
1.2 More data sharing
curl_share_* functions already exist and work, and they can be extended to
1.5 get rid of PATH_MAX
Having code use and rely on PATH_MAX is not nice:
- http://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html
+ https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html
Currently the SSH based code uses it a bit, but to remove PATH_MAX from there
we need libssh2 to properly tell us when we pass in a too small buffer and
Dynamically allocate buffer size depending on protocol in use in combination
with freeing it after each individual transfer? Other suggestions?
-1.7 Detect when called from within callbacks
+1.7 Support HTTP/2 for HTTP(S) proxies
- We should set a state variable before calling callbacks, so that we
- subsequently can add code within libcurl that returns error if called within
- callbacks for when that's not supported.
+ Support for doing HTTP/2 to HTTP and HTTPS proxies is still missing.
-1.8 Allow SSL (HTTPS) to proxy
+1.8 CURLOPT_RESOLVE for any port number
- To prevent local users from snooping on your traffic to the proxy. Supported
- by Firefox and Chrome already:
- https://www.chromium.org/developers/design-documents/secure-web-proxy
+ This option allows applications to set a replacement IP address for a given
+ host + port pair. Consider making support for providing a replacement address
+ for the host name on all port numbers.
- See this stale work in progress branch:
- https://github.com/curl/curl/tree/HTTPS-proxy based on this PR:
- https://github.com/curl/curl/pull/305
+ See https://github.com/curl/curl/issues/1264
1.9 Cache negative name resolves
A name resolve that has failed is likely to fail when made again within a
short period of time. Currently we only cache positive responses.
-1.10 Support IDNA2008
+1.10 auto-detect proxy
+
+ libcurl could be made to detect the system proxy setup automatically and use
+ that. On Windows, macOS and Linux desktops for example.
- International Domain Names are supported in libcurl since years back, powered
- by libidn. libidn implements IDNA2003 which has been superseded by IDNA2008.
- libidn2 is an existing library offering support for IDNA2008.
+ The pull-request to use libproxy for this was deferred due to doubts on the
+ reliability of the dependency and how to use it:
+ https://github.com/curl/curl/pull/977
-1.11 minimize dependencies with dynamicly loaded modules
+ libdetectproxy is a (C++) library for detecting the proxy on Windows
+ https://github.com/paulharris/libdetectproxy
+
+1.11 minimize dependencies with dynamically loaded modules
We can create a system with loadable modules/plug-ins, where these modules
would be the ones that link to 3rd party libs. That would allow us to avoid
app/invoke/used protocols would be necessary to load. See
https://github.com/curl/curl/issues/349
-1.12 have form functions use CURL handle argument
+1.12 updated DNS server while running
+
+ If /etc/resolv.conf gets updated while a program using libcurl is running, it
+ is may cause name resolves to fail unless res_init() is called. We should
+ consider calling res_init() + retry once unconditionally on all name resolve
+ failures to mitigate against this. Firefox works like that. Note that Windows
+ doesn't have res_init() or an alternative.
+
+ https://github.com/curl/curl/issues/2251
+
+1.13 DNS-over-HTTPS
- curl_formadd() and curl_formget() both currently have no CURL handle
- argument, but both can use a callback that is set in the easy handle, and
- thus curl_formget() with callback cannot function without first having
- curl_easy_perform() (or similar) called - which is hard to grasp and a design
- mistake.
+ By adding support for DNS-over-HTTPS curl could resolve host names using a
+ totally separate name server than the standard system resolver, while at the
+ same time doing so over a communication channel that enhances privacy and
+ security.
- The curl_formadd() design can probably also be reconsidered to make it easier
- to use and less error-prone. Probably easiest by splitting it into several
- function calls.
+ https://github.com/curl/curl/wiki/DNS-over-HTTPS
1.14 Typesafe curl_easy_setopt()
An API could allow a forced flush or just a forced loop that would properly
close all connections that have been closed by the server already.
+1.24 TCP Fast Open for windows
+
+ libcurl supports the CURLOPT_TCP_FASTOPEN option since 7.49.0 for Linux and
+ Mac OS. Windows supports TCP Fast Open starting with Windows 10, version 1607
+ and we should add support for it.
+
+1.25 Expose tried IP addresses that failed
+
+ When libcurl fails to connect to a host, it should be able to offer the
+ application the list of IP addresses that were used in the attempt.
+
+ https://github.com/curl/curl/issues/2126
+
+1.26 CURL_REFUSE_CLEARTEXT
+
+ An environment variable that when set will make libcurl refuse to use any
+ cleartext network protocol. That's all non-encrypted ones (FTP, HTTP, Gopher,
+ etc). By adding the check to libcurl and not just curl, this environment
+ variable can then help users to block all libcurl-using programs from
+ accessing the network using unsafe protocols.
+
+ The variable could be given some sort of syntax or different levels and be
+ used to also allow for example users to refuse libcurl to do transfers with
+ HTTPS certificate checks disabled.
+
+ It could also offer to refuse usernames in URLs (see TODO 1.1)
+
+1.27 hardcode the "localhost" addresses
+
+ There's this new spec getting adopted that says "localhost" should always and
+ unconditionally be a local address and not get resolved by a DNS server. A
+ fine way for curl to fix this would be to simply hard-code the response to
+ 127.0.0.1 and/or ::1 (depending on what IP versions that are requested). This
+ is what the browsers probably will do with this hostname.
+
+ https://bugzilla.mozilla.org/show_bug.cgi?id=1220810
+
+ https://tools.ietf.org/html/draft-ietf-dnsop-let-localhost-be-localhost-02
2. libcurl - multi interface
Make sure we don't ever loop because of non-blocking sockets returning
EWOULDBLOCK or similar. Blocking cases include:
- - Name resolves on non-windows unless c-ares is used
- - NSS SSL connections
- - HTTP proxy CONNECT operations
+ - Name resolves on non-windows unless c-ares or the threaded resolver is used
- SOCKS proxy handshakes
- file:// transfers
- TELNET transfers
3. Documentation
-3.1 Update date and version in man pages
-
- 'maketgz' or another suitable script could update the .TH sections of the man
- pages at release time to use the current date and curl/libcurl version
- number.
-
3.2 Provide cmake config-file
A config-file package is a set of files provided by us to allow applications
4.6 GSSAPI via Windows SSPI
-In addition to currently supporting the SASL GSSAPI mechanism (Kerberos V5)
-via third-party GSS-API libraries, such as Heimdal or MIT Kerberos, also add
-support for GSSAPI authentication via Windows SSPI.
+ In addition to currently supporting the SASL GSSAPI mechanism (Kerberos V5)
+ via third-party GSS-API libraries, such as Heimdal or MIT Kerberos, also add
+ support for GSSAPI authentication via Windows SSPI.
4.7 STAT for LIST without data connection
-Some FTP servers allow STAT for listing directories instead of using LIST, and
-the response is then sent over the control connection instead of as the
-otherwise usedw data connection: http://www.nsftools.com/tips/RawFTP.htm#STAT
+ Some FTP servers allow STAT for listing directories instead of using LIST,
+ and the response is then sent over the control connection instead of as the
+ otherwise usedw data connection: http://www.nsftools.com/tips/RawFTP.htm#STAT
+
+ This is not detailed in any FTP specification.
+
+4.8 Option to ignore private IP addresses in PASV response
-This is not detailed in any FTP specification.
+ Some servers respond with and some other FTP client implementations can
+ ignore private (RFC 1918 style) IP addresses when received in PASV responses.
+ To consider for libcurl as well. See https://github.com/curl/curl/issues/1455
5. HTTP
headers use a default value so only headers that need to be moved have to be
specified.
-5.4 Use huge HTTP/2 windows
-
- We're currently using nghttp2's default window size which is terribly small
- (64K). This becomes a bottle neck over high bandwidth networks. We should
- instead make the window size to be very big (512MB?) as we really don't do
- much flow control anyway.
-
5.5 auth= in URLs
Add the ability to specify the preferred authentication mechanism to use by
Consider a way to tell curl to refuse to "downgrade" protocol with a redirect
and/or possibly a bit that refuses redirect to change protocol completely.
-5.7 Brotli compression
-
- Compression algorithms that perform better than gzip are being considered for
- use and inclusion in existing browsers. For example 'brotli'. If servers
- follow along it is a good reason for us to also allow users to take advantage
- of this. The algorithm: https://github.com/google/brotli The Firefox bug:
- https://bugzilla.mozilla.org/show_bug.cgi?id=366559
-
-5.8 QUIC
+5.7 QUIC
The standardization process of QUIC has been taken to the IETF and can be
followed on the [IETF QUIC Mailing
implemented. This, to allow other projects to benefit from the work and to
thus broaden the interest and chance of others to participate.
+5.8 Leave secure cookies alone
+
+ Non-secure origins (HTTP sites) should not be allowed to set or modify
+ cookies with the 'secure' property:
+
+ https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-01
+
6. TELNET
Add telnet feature negotiation data to the debug callback as header data.
-6.4 send data in chunks
-
- Currently, telnet sends data one byte at a time. This is fine for interactive
- use, but inefficient for any other. Sent data should be sent in larger
- chunks.
7. SMTP
library, so that the same application code can use mutex-locking
independently of OpenSSL or GnutTLS being used.
-13.3 Evaluate SSL patches
+13.3 Support in-memory certs/ca certs/keys
- Evaluate/apply Gertjan van Wingerde's SSL patches:
- https://curl.haxx.se/mail/lib-2004-03/0087.html
+ You can specify the private and public keys for SSH/SSL as file paths. Some
+ programs want to avoid using files and instead just pass them as in-memory
+ data blobs. There's probably a challenge to make this work across the
+ plethory of different TLS and SSH backends that curl suppports.
+ https://github.com/curl/curl/issues/2310
-13.4 Cache OpenSSL contexts
+13.4 Cache/share OpenSSL contexts
"Look at SSL cafile - quick traces look to me like these are done on every
request as well, when they should only be necessary once per SSL context (or
style connections are re-used. It will make us use slightly more memory but
it will libcurl do less creations and deletions of SSL contexts.
+ Technically, the "caching" is probably best implemented by getting added to
+ the share interface so that easy handles who want to and can reuse the
+ context specify that by sharing with the right properties set.
+
+ https://github.com/curl/curl/issues/1110
+
13.5 Export session ids
Add an interface to libcurl that enables "session IDs" to get
Björn Stenberg wrote a separate initial take on DANE that was never
completed.
-13.9 Support TLS v1.3
+13.11 Support intermediate & root pinning for PINNEDPUBLICKEY
+
+ CURLOPT_PINNEDPUBLICKEY does not consider the hashes of intermediate & root
+ certificates when comparing the pinned keys. Therefore it is not compatible
+ with "HTTP Public Key Pinning" as there also intermediate and root certificates
+ can be pinned. This is very useful as it prevents webadmins from "locking
+ themself out of their servers".
+
+ Adding this feature would make curls pinning 100% compatible to HPKP and allow
+ more flexible pinning.
+
+13.12 Support HSTS
+
+ "HTTP Strict Transport Security" is TOFU (trust on first use), time-based
+ features indicated by a HTTP header send by the webserver. It is widely used
+ in browsers and it's purpose is to prevent insecure HTTP connections after
+ a previous HTTPS connection. It protects against SSLStripping attacks.
+
+ Doc: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
+ RFC 6797: https://tools.ietf.org/html/rfc6797
+
+13.13 Support HPKP
- TLS version 1.3 is about to ship and is getting implemented by TLS libraries
- as we speak. We should start to support the symbol and make sure all backends
- handle it accordingly, then gradually add support as the TLS libraries add
- the corresponding support. There may be a need to add some additional options
- to allow libcurl to take advantage of the new features in 1.3.
+ "HTTP Public Key Pinning" is TOFU (trust on first use), time-based
+ features indicated by a HTTP header send by the webserver. It's purpose is
+ to prevent Man-in-the-middle attacks by trusted CAs by allowing webadmins
+ to specify which CAs/certificates/public keys to trust when connection to
+ their websites.
+ It can be build based on PINNEDPUBLICKEY.
+
+ Wikipedia: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
+ OWASP: https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
+ Doc: https://developer.mozilla.org/de/docs/Web/Security/Public_Key_Pinning
+ RFC: https://tools.ietf.org/html/draft-ietf-websec-key-pinning-21
14. GnuTLS
libcurl's SFTP transfer performance is sub par and can be improved, mostly by
the approach mentioned in "1.6 Modified buffer size approach".
+17.3 Support better than MD5 hostkey hash
+
+ libcurl offers the CURLOPT_SSH_HOST_PUBLIC_KEY_MD5 option for verifying the
+ server's key. MD5 is generally being deprecated so we should implement
+ support for stronger hashing algorithms. libssh2 itself is what provides this
+ underlying functionality and it supports at least SHA-1 as an alternative.
+ SHA-1 is also being deprecated these days so we should consider workign with
+ libssh2 to instead offer support for SHA-256 or similar.
+
+17.4 Support CURLOPT_PREQUOTE
+
+ The two other QUOTE options are supported for SFTP, but this was left out for
+ unknown reasons!
+
18. Command line tool
18.1 sync
18.3 prevent file overwriting
- Add an option that prevents cURL from overwriting existing local files. When
+ Add an option that prevents curl from overwriting existing local files. When
used, and there already is an existing file with the target file name
(either -O or -o), a number should be appended (and increased if already
existing). So that index.html becomes first index.html.1 and then
Using the multi interface would also allow properly using parallel transfers
with HTTP/2 and supporting HTTP/2 server push from the command line.
-18.5 provide formpost headers
-
- Extending the capabilities of the multipart formposting. How about leaving
- the ';type=foo' syntax as it is and adding an extra tag (headers) which
- works like this: curl -F "coolfiles=@fil1.txt;headers=@fil1.hdr" where
- fil1.hdr contains extra headers like
+18.5 UTF-8 filenames in Content-Disposition
- Content-Type: text/plain; charset=KOI8-R"
- Content-Transfer-Encoding: base64
- X-User-Comment: Please don't use browser specific HTML code
+ RFC 6266 documents how UTF-8 names can be passed to a client in the
+ Content-Disposition header, and curl does not support this.
- which should overwrite the program reasonable defaults (plain/text,
- 8bit...)
+ https://github.com/curl/curl/issues/1888
18.6 warning when setting an option
This can be useful to tell when support for a particular feature hasn't been
compiled into the library.
-18.7 warning when sending binary output to terminal
+18.7 warning if curl version is not in sync with libcurl version
- Provide a way that prompts the user for confirmation before binary data is
- sent to the terminal, much in the style 'less' does it.
+ This is usually a sign of a funny, weird or unexpected install situations
+ that aren't always quickly nor easily detected by users. curl and libcurl are
+ always released in sync and should use the same version numbers unless very
+ special situations.
18.8 offer color-coded HTTP header output
way to implement this is probably to add that new logic in the command line
tool only and not in the underlying CURLOPT_FAILONERROR logic.
+18.15 --retry should resume
+
+ When --retry is used and curl actually retries transfer, it should use the
+ already transferred data and do a resumed transfer for the rest (when
+ possible) so that it doesn't have to transfer the same data again that was
+ already transferred before the retry.
+
+ See https://github.com/curl/curl/issues/1084
+
+18.16 send only part of --data
+
+ When the user only wants to send a small piece of the data provided with
+ --data or --data-binary, like when that data is a huge file, consider a way
+ to specify that curl should only send a piece of that. One suggested syntax
+ would be: "--data-binary @largefile.zip!1073741823-2147483647".
+
+ See https://github.com/curl/curl/issues/1200
+
+18.17 consider file name from the redirected URL with -O ?
+
+ When a user gives a URL and uses -O, and curl follows a redirect to a new
+ URL, the file name is not extracted and used from the newly redirected-to URL
+ even if the new URL may have a much more sensible file name.
+
+ This is clearly documented and helps for security since there's no surprise
+ to users which file name that might get overwritten. But maybe a new option
+ could allow for this or maybe -J should imply such a treatment as well as -J
+ already allows for the server to decide what file name to use so it already
+ provides the "may overwrite any file" risk.
+
+ This is extra tricky if the original URL has no file name part at all since
+ then the current code path will error out with an error message, and we can't
+ *know* already at that point if curl will be redirected to a URL that has a
+ file name...
+
+ See https://github.com/curl/curl/issues/1241
+
+18.18 retry on network is unreachable
+
+ The --retry option retries transfers on "transient failures". We later added
+ --retry-connrefused to also retry for "connection refused" errors.
+
+ Suggestions have been brought to also allow retry on "network is unreachable"
+ errors and while totally reasonable, maybe we should consider a way to make
+ this more configurable than to add a new option for every new error people
+ want to retry for?
+
+ https://github.com/curl/curl/issues/1603
19. Build
Consider extending 'roffit' to produce decent ASCII output, and use that
instead of (g)nroff when building src/tool_hugehelp.c
+19.2 Enable PIE and RELRO by default
+
+ Especially when having programs that execute curl via the command line, PIE
+ renders the exploitation of memory corruption vulnerabilities a lot more
+ difficult. This can be attributed to the additional information leaks being
+ required to conduct a successful attack. RELRO, on the other hand, masks
+ different binary sections like the GOT as read-only and thus kills a handful
+ of techniques that come in handy when attackers are able to arbitrarily
+ overwrite memory. A few tests showed that enabling these features had close
+ to no impact, neither on the performance nor on the general functionality of
+ curl.
+
+
20. Test suite
20.1 SSL tunnel
projects / platform / upstream / curl.git / blobdiff