1 Tutorial: HTTP/2 server
2 =========================
4 In this tutorial, we are going to write single-threaded, event-based
5 HTTP/2 web server, which supports HTTPS only. It can handle
6 concurrent multiple requests, but only the GET method is supported. The
7 complete source code, `libevent-server.c`_, is attached at the end of
8 this page. It also resides in examples directory in the archive or
11 This simple server takes 3 arguments, a port number to listen to, a path to
12 your SSL/TLS private key file and a path to your certificate file. Its
13 synopsis is like this::
15 $ libevent-server PORT /path/to/server.key /path/to/server.crt
17 We use libevent in this tutorial to handle networking I/O. Please
18 note that nghttp2 itself does not depend on libevent.
20 First we create a setup routine for libevent and OpenSSL in the functions
21 ``main()`` and ``run()``. One thing in there you should look at, is the setup
22 of the NPN callback. The NPN callback is used for the server to advertise
23 which application protocols the server supports to a client. In this example
24 program, when creating ``SSL_CTX`` object, we store the application protocol
25 name in the wire format of NPN in a statically allocated buffer. This is safe
26 because we only create one ``SSL_CTX`` object in the program's entire life
29 static unsigned char next_proto_list[256];
30 static size_t next_proto_list_len;
32 static int next_proto_cb(SSL *s _U_, const unsigned char **data,
33 unsigned int *len, void *arg _U_) {
34 *data = next_proto_list;
35 *len = (unsigned int)next_proto_list_len;
36 return SSL_TLSEXT_ERR_OK;
39 static SSL_CTX *create_ssl_ctx(const char *key_file, const char *cert_file) {
43 ssl_ctx = SSL_CTX_new(SSLv23_server_method());
47 next_proto_list[0] = NGHTTP2_PROTO_VERSION_ID_LEN;
48 memcpy(&next_proto_list[1], NGHTTP2_PROTO_VERSION_ID,
49 NGHTTP2_PROTO_VERSION_ID_LEN);
50 next_proto_list_len = 1 + NGHTTP2_PROTO_VERSION_ID_LEN;
52 SSL_CTX_set_next_protos_advertised_cb(ssl_ctx, next_proto_cb, NULL);
56 The wire format of NPN is a sequence of length prefixed string. Exactly one
57 byte is used to specify the length of each protocol identifier. In this
58 tutorial, we advertise the specific HTTP/2 protocol version the current
59 nghttp2 library supports. The nghttp2 library exports its identifier in
60 :macro:`NGHTTP2_PROTO_VERSION_ID`. The ``next_proto_cb()`` function is the
61 server-side NPN callback. In the OpenSSL implementation, we just assign the
62 pointer to the NPN buffers we filled in earlier. The NPN callback function is
63 set to the ``SSL_CTX`` object using
64 ``SSL_CTX_set_next_protos_advertised_cb()``.
66 We use the ``app_content`` structure to store application-wide data::
70 struct event_base *evbase;
73 We use the ``http2_session_data`` structure to store session-level
74 (which corresponds to one HTTP/2 connection) data::
76 typedef struct http2_session_data {
77 struct http2_stream_data root;
78 struct bufferevent *bev;
80 nghttp2_session *session;
84 We use the ``http2_stream_data`` structure to store stream-level data::
86 typedef struct http2_stream_data {
87 struct http2_stream_data *prev, *next;
93 A single HTTP/2 session can have multiple streams. We manage these
94 multiple streams with a doubly linked list. The first element of this
95 list is pointed to by the ``root->next`` in ``http2_session_data``.
96 Initially, ``root->next`` is ``NULL``. We use libevent's bufferevent
97 structure to perform network I/O. Note that the bufferevent object is
98 kept in ``http2_session_data`` and not in ``http2_stream_data``. This
99 is because ``http2_stream_data`` is just a logical stream multiplexed
100 over the single connection managed by bufferevent in
101 ``http2_session_data``.
103 We first create a listener object to accept incoming connections. We use
104 libevent's ``struct evconnlistener`` for this purpose::
106 static void start_listen(struct event_base *evbase, const char *service,
107 app_context *app_ctx) {
109 struct addrinfo hints;
110 struct addrinfo *res, *rp;
112 memset(&hints, 0, sizeof(hints));
113 hints.ai_family = AF_UNSPEC;
114 hints.ai_socktype = SOCK_STREAM;
115 hints.ai_flags = AI_PASSIVE;
117 hints.ai_flags |= AI_ADDRCONFIG;
118 #endif /* AI_ADDRCONFIG */
120 rv = getaddrinfo(NULL, service, &hints, &res);
124 for (rp = res; rp; rp = rp->ai_next) {
125 struct evconnlistener *listener;
126 listener = evconnlistener_new_bind(
127 evbase, acceptcb, app_ctx, LEV_OPT_CLOSE_ON_FREE | LEV_OPT_REUSEABLE,
128 16, rp->ai_addr, rp->ai_addrlen);
135 errx(1, "Could not start listener");
138 We specify the ``acceptcb`` callback which is called when a new connection is
141 static void acceptcb(struct evconnlistener *listener _U_, int fd,
142 struct sockaddr *addr, int addrlen, void *arg) {
143 app_context *app_ctx = (app_context *)arg;
144 http2_session_data *session_data;
146 session_data = create_http2_session_data(app_ctx, fd, addr, addrlen);
148 bufferevent_setcb(session_data->bev, readcb, writecb, eventcb, session_data);
151 Here we create the ``http2_session_data`` object. The bufferevent for
152 this connection is also initialized at this time. We specify three
153 callbacks for the bufferevent: ``readcb``, ``writecb`` and
156 The ``eventcb()`` callback is invoked by the libevent event loop when an event
157 (e.g., connection has been established, timeout, etc) happens on the
158 underlying network socket::
160 static void eventcb(struct bufferevent *bev _U_, short events, void *ptr) {
161 http2_session_data *session_data = (http2_session_data *)ptr;
162 if (events & BEV_EVENT_CONNECTED) {
163 fprintf(stderr, "%s connected\n", session_data->client_addr);
165 initialize_nghttp2_session(session_data);
167 if (send_server_connection_header(session_data) != 0) {
168 delete_http2_session_data(session_data);
174 if (events & BEV_EVENT_EOF) {
175 fprintf(stderr, "%s EOF\n", session_data->client_addr);
176 } else if (events & BEV_EVENT_ERROR) {
177 fprintf(stderr, "%s network error\n", session_data->client_addr);
178 } else if (events & BEV_EVENT_TIMEOUT) {
179 fprintf(stderr, "%s timeout\n", session_data->client_addr);
181 delete_http2_session_data(session_data);
184 For the ``BEV_EVENT_EOF``, ``BEV_EVENT_ERROR`` and
185 ``BEV_EVENT_TIMEOUT`` events, we just simply tear down the connection.
186 The ``delete_http2_session_data()`` function destroys the
187 ``http2_session_data`` object and thus also its bufferevent member.
188 As a result, the underlying connection is closed. The
189 ``BEV_EVENT_CONNECTED`` event is invoked when SSL/TLS handshake is
190 finished successfully. Now we are ready to start the HTTP/2
193 We initialize a nghttp2 session object which is done in
194 ``initialize_nghttp2_session()``::
196 static void initialize_nghttp2_session(http2_session_data *session_data) {
197 nghttp2_option *option;
198 nghttp2_session_callbacks *callbacks;
200 nghttp2_option_new(&option);
202 /* Tells nghttp2_session object that it handles client connection
204 nghttp2_option_set_recv_client_preface(option, 1);
206 nghttp2_session_callbacks_new(&callbacks);
208 nghttp2_session_callbacks_set_send_callback(callbacks, send_callback);
210 nghttp2_session_callbacks_set_on_frame_recv_callback(callbacks,
211 on_frame_recv_callback);
213 nghttp2_session_callbacks_set_on_stream_close_callback(
214 callbacks, on_stream_close_callback);
216 nghttp2_session_callbacks_set_on_header_callback(callbacks,
219 nghttp2_session_callbacks_set_on_begin_headers_callback(
220 callbacks, on_begin_headers_callback);
222 nghttp2_session_server_new2(&session_data->session, callbacks, session_data,
225 nghttp2_session_callbacks_del(callbacks);
226 nghttp2_option_del(option);
229 Since we are creating a server and uses options, the nghttp2 session
230 object is created using `nghttp2_session_server_new2()` function. We
231 registers five callbacks for nghttp2 session object. We'll talk about
232 these callbacks later. Our server only speaks HTTP/2. In this case,
233 we use `nghttp2_option_set_recv_client_preface()` to make
234 :type:`nghttp2_session` object handle client connection preface, which
235 saves some lines of application code.
237 After initialization of the nghttp2 session object, we are going to send
238 a server connection header in ``send_server_connection_header()``::
240 static int send_server_connection_header(http2_session_data *session_data) {
241 nghttp2_settings_entry iv[1] = {
242 {NGHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, 100}};
245 rv = nghttp2_submit_settings(session_data->session, NGHTTP2_FLAG_NONE, iv,
248 warnx("Fatal error: %s", nghttp2_strerror(rv));
254 The server connection header is a SETTINGS frame. We specify
255 SETTINGS_MAX_CONCURRENT_STREAMS to 100 in the SETTINGS frame. To queue
256 the SETTINGS frame for the transmission, we use
257 `nghttp2_submit_settings()`. Note that `nghttp2_submit_settings()`
258 function only queues the frame and it does not actually send it. All
259 functions in the ``nghttp2_submit_*()`` family have this property. To
260 actually send the frame, `nghttp2_session_send()` should be used, as
263 Since bufferevent may buffer more than the first 24 bytes from the client, we
264 have to process them here since libevent won't invoke callback functions for
265 this pending data. To process the received data, we call the
266 ``session_recv()`` function::
268 static int session_recv(http2_session_data *session_data) {
270 struct evbuffer *input = bufferevent_get_input(session_data->bev);
271 size_t datalen = evbuffer_get_length(input);
272 unsigned char *data = evbuffer_pullup(input, -1);
274 readlen = nghttp2_session_mem_recv(session_data->session, data, datalen);
276 warnx("Fatal error: %s", nghttp2_strerror((int)readlen));
279 if (evbuffer_drain(input, readlen) != 0) {
280 warnx("Fatal error: evbuffer_drain failed");
283 if (session_send(session_data) != 0) {
289 In this function, we feed all unprocessed but already received data to the
290 nghttp2 session object using the `nghttp2_session_mem_recv()` function. The
291 `nghttp2_session_mem_recv()` function processes the data and may invoke the
292 nghttp2 callbacks and also queue outgoing frames. Since there may be pending
293 outgoing frames, we call ``session_send()`` function to send off those
294 frames. The ``session_send()`` function is defined as follows::
296 static int session_send(http2_session_data *session_data) {
298 rv = nghttp2_session_send(session_data->session);
300 warnx("Fatal error: %s", nghttp2_strerror(rv));
306 The `nghttp2_session_send()` function serializes the frame into wire
307 format and calls ``send_callback()`` of type
308 :type:`nghttp2_send_callback`. The ``send_callback()`` is defined as
311 static ssize_t send_callback(nghttp2_session *session _U_, const uint8_t *data,
312 size_t length, int flags _U_, void *user_data) {
313 http2_session_data *session_data = (http2_session_data *)user_data;
314 struct bufferevent *bev = session_data->bev;
315 /* Avoid excessive buffering in server side. */
316 if (evbuffer_get_length(bufferevent_get_output(session_data->bev)) >=
317 OUTPUT_WOULDBLOCK_THRESHOLD) {
318 return NGHTTP2_ERR_WOULDBLOCK;
320 bufferevent_write(bev, data, length);
324 Since we use bufferevent to abstract network I/O, we just write the
325 data to the bufferevent object. Note that `nghttp2_session_send()`
326 continues to write all frames queued so far. If we were writing the
327 data to a non-blocking socket directly using ``write()`` system call
328 in the ``send_callback()``, we would surely get ``EAGAIN`` or
329 ``EWOULDBLOCK`` back since the socket has limited send buffer. If that
330 happens, we can return :macro:`NGHTTP2_ERR_WOULDBLOCK` to signal the
331 nghttp2 library to stop sending further data. But when writing to the
332 bufferevent, we have to regulate the amount data to get buffered
333 ourselves to avoid using huge amounts of memory. To achieve this, we
334 check the size of the output buffer and if it reaches more than or
335 equal to ``OUTPUT_WOULDBLOCK_THRESHOLD`` bytes, we stop writing data
336 and return :macro:`NGHTTP2_ERR_WOULDBLOCK` to tell the library to stop
337 calling send_callback.
339 The next bufferevent callback is ``readcb()``, which is invoked when
340 data is available to read in the bufferevent input buffer::
342 static void readcb(struct bufferevent *bev _U_, void *ptr) {
343 http2_session_data *session_data = (http2_session_data *)ptr;
344 if (session_recv(session_data) != 0) {
345 delete_http2_session_data(session_data);
350 In this function, we just call ``session_recv()`` to process incoming
353 The third bufferevent callback is ``writecb()``, which is invoked when all
354 data in the bufferevent output buffer has been sent::
356 static void writecb(struct bufferevent *bev, void *ptr) {
357 http2_session_data *session_data = (http2_session_data *)ptr;
358 if (evbuffer_get_length(bufferevent_get_output(bev)) > 0) {
361 if (nghttp2_session_want_read(session_data->session) == 0 &&
362 nghttp2_session_want_write(session_data->session) == 0) {
363 delete_http2_session_data(session_data);
366 if (session_send(session_data) != 0) {
367 delete_http2_session_data(session_data);
372 First we check whether we should drop the connection or not. The nghttp2
373 session object keeps track of reception and transmission of GOAWAY frames and
374 other error conditions as well. Using this information, the nghttp2 session
375 object will tell whether the connection should be dropped or not. More
376 specifically, if both `nghttp2_session_want_read()` and
377 `nghttp2_session_want_write()` return 0, we have no business left in the
378 connection. But since we are using bufferevent and its deferred callback
379 option, the bufferevent output buffer may contain pending data when the
380 ``writecb()`` is called. To handle this, we check whether the output buffer is
381 empty or not. If all these conditions are met, we drop connection.
383 Otherwise, we call ``session_send()`` to process the pending output
384 data. Remember that in ``send_callback()``, we must not write all data to
385 bufferevent to avoid excessive buffering. We continue processing pending data
386 when the output buffer becomes empty.
388 We have already described the nghttp2 callback ``send_callback()``. Let's
389 learn about the remaining nghttp2 callbacks we setup in
390 ``initialize_nghttp2_setup()`` function.
392 The ``on_begin_headers_callback()`` function is invoked when the reception of
393 a header block in HEADERS or PUSH_PROMISE frame is started::
395 static int on_begin_headers_callback(nghttp2_session *session,
396 const nghttp2_frame *frame,
398 http2_session_data *session_data = (http2_session_data *)user_data;
399 http2_stream_data *stream_data;
401 if (frame->hd.type != NGHTTP2_HEADERS ||
402 frame->headers.cat != NGHTTP2_HCAT_REQUEST) {
405 stream_data = create_http2_stream_data(session_data, frame->hd.stream_id);
406 nghttp2_session_set_stream_user_data(session, frame->hd.stream_id,
411 We are only interested in the HEADERS frame in this function. Since the
412 HEADERS frame has several roles in the HTTP/2 protocol, we check that it is a
413 request HEADERS, which opens new stream. If the frame is a request HEADERS, we
414 create a ``http2_stream_data`` object to store the stream related data. We
415 associate the created ``http2_stream_data`` object with the stream in the
416 nghttp2 session object using `nghttp2_set_stream_user_data()` to get the
417 object without searching through the doubly linked list.
419 In this example server, we want to serve files relative to the current working
420 directory in which the program was invoked. Each header name/value pair is
421 emitted via ``on_header_callback`` function, which is called after
422 ``on_begin_headers_callback()``::
424 static int on_header_callback(nghttp2_session *session,
425 const nghttp2_frame *frame, const uint8_t *name,
426 size_t namelen, const uint8_t *value,
427 size_t valuelen, uint8_t flags _U_,
428 void *user_data _U_) {
429 http2_stream_data *stream_data;
430 const char PATH[] = ":path";
431 switch (frame->hd.type) {
432 case NGHTTP2_HEADERS:
433 if (frame->headers.cat != NGHTTP2_HCAT_REQUEST) {
437 nghttp2_session_get_stream_user_data(session, frame->hd.stream_id);
438 if (!stream_data || stream_data->request_path) {
441 if (namelen == sizeof(PATH) - 1 && memcmp(PATH, name, namelen) == 0) {
443 for (j = 0; j < valuelen && value[j] != '?'; ++j)
445 stream_data->request_path = percent_decode(value, j);
452 We search for the ``:path`` header field among the request headers and store
453 the requested path in the ``http2_stream_data`` object. In this example
454 program, we ignore ``:method`` header field and always treat the request as a
457 The ``on_frame_recv_callback()`` function is invoked when a frame is
460 static int on_frame_recv_callback(nghttp2_session *session,
461 const nghttp2_frame *frame, void *user_data) {
462 http2_session_data *session_data = (http2_session_data *)user_data;
463 http2_stream_data *stream_data;
464 switch (frame->hd.type) {
466 case NGHTTP2_HEADERS:
467 /* Check that the client request has finished */
468 if (frame->hd.flags & NGHTTP2_FLAG_END_STREAM) {
470 nghttp2_session_get_stream_user_data(session, frame->hd.stream_id);
471 /* For DATA and HEADERS frame, this callback may be called after
472 on_stream_close_callback. Check that stream still alive. */
476 return on_request_recv(session, session_data, stream_data);
485 First we retrieve the ``http2_stream_data`` object associated with the stream
486 in ``on_begin_headers_callback()``. It is done using
487 `nghttp2_session_get_stream_user_data()`. If the requested path cannot be
488 served for some reason (e.g., file is not found), we send a 404 response,
489 which is done in ``error_reply()``. Otherwise, we open the requested file and
490 send its content. We send the header field ``:status`` as a single response
493 Sending the content of the file is done in ``send_response()`` function::
495 static int send_response(nghttp2_session *session, int32_t stream_id,
496 nghttp2_nv *nva, size_t nvlen, int fd) {
498 nghttp2_data_provider data_prd;
499 data_prd.source.fd = fd;
500 data_prd.read_callback = file_read_callback;
502 rv = nghttp2_submit_response(session, stream_id, nva, nvlen, &data_prd);
504 warnx("Fatal error: %s", nghttp2_strerror(rv));
510 The nghttp2 library uses the :type:`nghttp2_data_provider` structure to
511 send entity body to the remote peer. The ``source`` member of this
512 structure is a union and it can be either void pointer or int which is
513 intended to be used as file descriptor. In this example server, we use
514 the file descriptor. We also set the ``file_read_callback()`` callback
515 function to read the contents of the file::
517 static ssize_t file_read_callback(nghttp2_session *session _U_,
518 int32_t stream_id _U_, uint8_t *buf,
519 size_t length, uint32_t *data_flags,
520 nghttp2_data_source *source,
521 void *user_data _U_) {
524 while ((r = read(fd, buf, length)) == -1 && errno == EINTR)
527 return NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE;
530 *data_flags |= NGHTTP2_DATA_FLAG_EOF;
535 If an error happens while reading the file, we return
536 :macro:`NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE`. This tells the
537 library to send RST_STREAM to the stream. When all data has been read, set
538 the :macro:`NGHTTP2_DATA_FLAG_EOF` flag to ``*data_flags`` to tell the
539 nghttp2 library that we have finished reading the file.
541 The `nghttp2_submit_response()` function is used to send the response to the
544 The ``on_stream_close_callback()`` function is invoked when the stream
547 static int on_stream_close_callback(nghttp2_session *session, int32_t stream_id,
548 uint32_t error_code _U_, void *user_data) {
549 http2_session_data *session_data = (http2_session_data *)user_data;
550 http2_stream_data *stream_data;
552 stream_data = nghttp2_session_get_stream_user_data(session, stream_id);
556 remove_stream(session_data, stream_data);
557 delete_http2_stream_data(stream_data);
561 We destroy the ``http2_stream_data`` object in this function since the stream
562 is about to close and we no longer use that object.