5 A stream is an abstract interface implemented by various objects in
6 Node.js. For example a [request to an HTTP
7 server](http.html#http_http_incomingmessage) is a stream, as is
8 [stdout][]. Streams are readable, writable, or both. All streams are
9 instances of [EventEmitter][]
11 You can load the Stream base classes by doing `require('stream')`.
12 There are base classes provided for [Readable][] streams, [Writable][]
13 streams, [Duplex][] streams, and [Transform][] streams.
15 This document is split up into 3 sections. The first explains the
16 parts of the API that you need to be aware of to use streams in your
17 programs. If you never implement a streaming API yourself, you can
20 The second section explains the parts of the API that you need to use
21 if you implement your own custom streams yourself. The API is
22 designed to make this easy for you to do.
24 The third section goes into more depth about how streams work,
25 including some of the internal mechanisms and functions that you
26 should probably not modify unless you definitely know what you are
30 ## API for Stream Consumers
34 Streams can be either [Readable][], [Writable][], or both ([Duplex][]).
36 All streams are EventEmitters, but they also have other custom methods
37 and properties depending on whether they are Readable, Writable, or
40 If a stream is both Readable and Writable, then it implements all of
41 the methods and events below. So, a [Duplex][] or [Transform][] stream is
42 fully described by this API, though their implementation may be
45 It is not necessary to implement Stream interfaces in order to consume
46 streams in your programs. If you **are** implementing streaming
47 interfaces in your own program, please also refer to
48 [API for Stream Implementors][] below.
50 Almost all Node.js programs, no matter how simple, use Streams in some
51 way. Here is an example of using Streams in an Node.js program:
54 var http = require('http');
56 var server = http.createServer(function (req, res) {
57 // req is an http.IncomingMessage, which is a Readable Stream
58 // res is an http.ServerResponse, which is a Writable Stream
61 // we want to get the data as utf8 strings
62 // If you don't set an encoding, then you'll get Buffer objects
63 req.setEncoding('utf8');
65 // Readable streams emit 'data' events once a listener is added
66 req.on('data', function (chunk) {
70 // the end event tells you that you have entire body
71 req.on('end', function () {
73 var data = JSON.parse(body);
77 return res.end('error: ' + er.message);
80 // write back something interesting to the user:
81 res.write(typeof data);
88 // $ curl localhost:1337 -d '{}'
90 // $ curl localhost:1337 -d '"foo"'
92 // $ curl localhost:1337 -d 'not json'
93 // error: Unexpected token o
96 ### Class: stream.Duplex
98 Duplex streams are streams that implement both the [Readable][] and
99 [Writable][] interfaces. See above for usage.
101 Examples of Duplex streams include:
107 ### Class: stream.Readable
111 The Readable stream interface is the abstraction for a *source* of
112 data that you are reading from. In other words, data comes *out* of a
115 A Readable stream will not start emitting data until you indicate that
116 you are ready to receive it.
118 Readable streams have two "modes": a **flowing mode** and a **paused
119 mode**. When in flowing mode, data is read from the underlying system
120 and provided to your program as fast as possible. In paused mode, you
121 must explicitly call `stream.read()` to get chunks of data out.
122 Streams start out in paused mode.
124 **Note**: If no data event handlers are attached, and there are no
125 [`pipe()`][] destinations, and the stream is switched into flowing
126 mode, then data will be lost.
128 You can switch to flowing mode by doing any of the following:
130 * Adding a [`'data'` event][] handler to listen for data.
131 * Calling the [`resume()`][] method to explicitly open the flow.
132 * Calling the [`pipe()`][] method to send the data to a [Writable][].
134 You can switch back to paused mode by doing either of the following:
136 * If there are no pipe destinations, by calling the [`pause()`][]
138 * If there are pipe destinations, by removing any [`'data'` event][]
139 handlers, and removing all pipe destinations by calling the
140 [`unpipe()`][] method.
142 Note that, for backwards compatibility reasons, removing `'data'`
143 event handlers will **not** automatically pause the stream. Also, if
144 there are piped destinations, then calling `pause()` will not
145 guarantee that the stream will *remain* paused once those
146 destinations drain and ask for more data.
148 Examples of readable streams include:
150 * [http responses, on the client](http.html#http_http_incomingmessage)
151 * [http requests, on the server](http.html#http_http_incomingmessage)
152 * [fs read streams](fs.html#fs_class_fs_readstream)
156 * [child process stdout and stderr][]
161 Emitted when the stream and any of its underlying resources (a file
162 descriptor, for example) have been closed. The event indicates that
163 no more events will be emitted, and no further computation will occur.
165 Not all streams will emit the 'close' event.
169 * `chunk` {Buffer | String} The chunk of data.
171 Attaching a `data` event listener to a stream that has not been
172 explicitly paused will switch the stream into flowing mode. Data will
173 then be passed as soon as it is available.
175 If you just want to get all the data out of the stream as fast as
176 possible, this is the best way to do so.
179 var readable = getReadableStreamSomehow();
180 readable.on('data', function(chunk) {
181 console.log('got %d bytes of data', chunk.length);
187 This event fires when there will be no more data to read.
189 Note that the `end` event **will not fire** unless the data is
190 completely consumed. This can be done by switching into flowing mode,
191 or by calling `read()` repeatedly until you get to the end.
194 var readable = getReadableStreamSomehow();
195 readable.on('data', function(chunk) {
196 console.log('got %d bytes of data', chunk.length);
198 readable.on('end', function() {
199 console.log('there will be no more data.');
207 Emitted if there was an error receiving data.
209 #### Event: 'readable'
211 When a chunk of data can be read from the stream, it will emit a
214 In some cases, listening for a `'readable'` event will cause some data
215 to be read into the internal buffer from the underlying system, if it
219 var readable = getReadableStreamSomehow();
220 readable.on('readable', function() {
221 // there is some data to read now
225 Once the internal buffer is drained, a `readable` event will fire
226 again when more data is available.
228 The `readable` event is not emitted in the "flowing" mode with the
229 sole exception of the last one, on end-of-stream.
231 The 'readable' event indicates that the stream has new information:
232 either new data is available or the end of the stream has been reached.
233 In the former case, `.read()` will return that data. In the latter case,
234 `.read()` will return null. For instance, in the following example, `foo.txt`
238 var fs = require('fs');
239 var rr = fs.createReadStream('foo.txt');
240 rr.on('readable', function() {
241 console.log('readable:', rr.read());
243 rr.on('end', function() {
248 The output of running this script is:
251 bash-3.2$ node test.js
256 #### readable.isPaused()
260 This method returns whether or not the `readable` has been **explicitly**
261 paused by client code (using `readable.pause()` without a corresponding
262 `readable.resume()`).
265 var readable = new stream.Readable
267 readable.isPaused() // === false
269 readable.isPaused() // === true
271 readable.isPaused() // === false
274 #### readable.pause()
278 This method will cause a stream in flowing mode to stop emitting
279 `data` events, switching out of flowing mode. Any data that becomes
280 available will remain in the internal buffer.
283 var readable = getReadableStreamSomehow();
284 readable.on('data', function(chunk) {
285 console.log('got %d bytes of data', chunk.length);
287 console.log('there will be no more data for 1 second');
288 setTimeout(function() {
289 console.log('now data will start flowing again');
295 #### readable.pipe(destination[, options])
297 * `destination` {[Writable][] Stream} The destination for writing data
298 * `options` {Object} Pipe options
299 * `end` {Boolean} End the writer when the reader ends. Default = `true`
301 This method pulls all the data out of a readable stream, and writes it
302 to the supplied destination, automatically managing the flow so that
303 the destination is not overwhelmed by a fast readable stream.
305 Multiple destinations can be piped to safely.
308 var readable = getReadableStreamSomehow();
309 var writable = fs.createWriteStream('file.txt');
310 // All the data from readable goes into 'file.txt'
311 readable.pipe(writable);
314 This function returns the destination stream, so you can set up pipe
318 var r = fs.createReadStream('file.txt');
319 var z = zlib.createGzip();
320 var w = fs.createWriteStream('file.txt.gz');
324 For example, emulating the Unix `cat` command:
327 process.stdin.pipe(process.stdout);
330 By default [`end()`][] is called on the destination when the source stream
331 emits `end`, so that `destination` is no longer writable. Pass `{ end:
332 false }` as `options` to keep the destination stream open.
334 This keeps `writer` open so that "Goodbye" can be written at the
338 reader.pipe(writer, { end: false });
339 reader.on('end', function() {
340 writer.end('Goodbye\n');
344 Note that `process.stderr` and `process.stdout` are never closed until
345 the process exits, regardless of the specified options.
347 #### readable.read([size])
349 * `size` {Number} Optional argument to specify how much data to read.
350 * Return {String | Buffer | null}
352 The `read()` method pulls some data out of the internal buffer and
353 returns it. If there is no data available, then it will return
356 If you pass in a `size` argument, then it will return that many
357 bytes. If `size` bytes are not available, then it will return `null`,
358 unless we've ended, in which case it will return the data remaining
361 If you do not specify a `size` argument, then it will return all the
362 data in the internal buffer.
364 This method should only be called in paused mode. In flowing mode,
365 this method is called automatically until the internal buffer is
369 var readable = getReadableStreamSomehow();
370 readable.on('readable', function() {
372 while (null !== (chunk = readable.read())) {
373 console.log('got %d bytes of data', chunk.length);
378 If this method returns a data chunk, then it will also trigger the
379 emission of a [`'data'` event][].
381 Note that calling `readable.read([size])` after the `end` event has been
382 triggered will return `null`. No runtime error will be raised.
384 #### readable.resume()
388 This method will cause the readable stream to resume emitting `data`
391 This method will switch the stream into flowing mode. If you do *not*
392 want to consume the data from a stream, but you *do* want to get to
393 its `end` event, you can call [`readable.resume()`][] to open the flow of
397 var readable = getReadableStreamSomehow();
399 readable.on('end', function() {
400 console.log('got to the end, but did not read anything');
404 #### readable.setEncoding(encoding)
406 * `encoding` {String} The encoding to use.
409 Call this function to cause the stream to return strings of the
410 specified encoding instead of Buffer objects. For example, if you do
411 `readable.setEncoding('utf8')`, then the output data will be
412 interpreted as UTF-8 data, and returned as strings. If you do
413 `readable.setEncoding('hex')`, then the data will be encoded in
414 hexadecimal string format.
416 This properly handles multi-byte characters that would otherwise be
417 potentially mangled if you simply pulled the Buffers directly and
418 called `buf.toString(encoding)` on them. If you want to read the data
419 as strings, always use this method.
422 var readable = getReadableStreamSomehow();
423 readable.setEncoding('utf8');
424 readable.on('data', function(chunk) {
425 assert.equal(typeof chunk, 'string');
426 console.log('got %d characters of string data', chunk.length);
430 #### readable.unpipe([destination])
432 * `destination` {[Writable][] Stream} Optional specific stream to unpipe
434 This method will remove the hooks set up for a previous `pipe()` call.
436 If the destination is not specified, then all pipes are removed.
438 If the destination is specified, but no pipe is set up for it, then
442 var readable = getReadableStreamSomehow();
443 var writable = fs.createWriteStream('file.txt');
444 // All the data from readable goes into 'file.txt',
445 // but only for the first second
446 readable.pipe(writable);
447 setTimeout(function() {
448 console.log('stop writing to file.txt');
449 readable.unpipe(writable);
450 console.log('manually close the file stream');
455 #### readable.unshift(chunk)
457 * `chunk` {Buffer | String} Chunk of data to unshift onto the read queue
459 This is useful in certain cases where a stream is being consumed by a
460 parser, which needs to "un-consume" some data that it has
461 optimistically pulled out of the source, so that the stream can be
462 passed on to some other party.
464 Note that `stream.unshift(chunk)` cannot be called after the `end` event
465 has been triggered; a runtime error will be raised.
467 If you find that you must often call `stream.unshift(chunk)` in your
468 programs, consider implementing a [Transform][] stream instead. (See API
469 for Stream Implementors, below.)
472 // Pull off a header delimited by \n\n
473 // use unshift() if we get too much
474 // Call the callback with (error, header, stream)
475 var StringDecoder = require('string_decoder').StringDecoder;
476 function parseHeader(stream, callback) {
477 stream.on('error', callback);
478 stream.on('readable', onReadable);
479 var decoder = new StringDecoder('utf8');
481 function onReadable() {
483 while (null !== (chunk = stream.read())) {
484 var str = decoder.write(chunk);
485 if (str.match(/\n\n/)) {
486 // found the header boundary
487 var split = str.split(/\n\n/);
488 header += split.shift();
489 var remaining = split.join('\n\n');
490 var buf = new Buffer(remaining, 'utf8');
493 stream.removeListener('error', callback);
494 stream.removeListener('readable', onReadable);
495 // now the body of the message can be read from the stream.
496 callback(null, header, stream);
498 // still reading the header.
505 Note that, unlike `stream.push(chunk)`, `stream.unshift(chunk)` will not
506 end the reading process by resetting the internal reading state of the
507 stream. This can cause unexpected results if `unshift` is called during a
508 read (i.e. from within a `_read` implementation on a custom stream). Following
509 the call to `unshift` with an immediate `stream.push('')` will reset the
510 reading state appropriately, however it is best to simply avoid calling
511 `unshift` while in the process of performing a read.
513 #### readable.wrap(stream)
515 * `stream` {Stream} An "old style" readable stream
517 Versions of Node.js prior to v0.10 had streams that did not implement the
518 entire Streams API as it is today. (See "Compatibility" below for
521 If you are using an older Node.js library that emits `'data'` events and
522 has a [`pause()`][] method that is advisory only, then you can use the
523 `wrap()` method to create a [Readable][] stream that uses the old stream
526 You will very rarely ever need to call this function, but it exists
527 as a convenience for interacting with old Node.js programs and libraries.
532 var OldReader = require('./old-api-module.js').OldReader;
533 var oreader = new OldReader;
534 var Readable = require('stream').Readable;
535 var myReader = new Readable().wrap(oreader);
537 myReader.on('readable', function() {
538 myReader.read(); // etc.
542 ### Class: stream.Transform
544 Transform streams are [Duplex][] streams where the output is in some way
545 computed from the input. They implement both the [Readable][] and
546 [Writable][] interfaces. See above for usage.
548 Examples of Transform streams include:
553 ### Class: stream.Writable
557 The Writable stream interface is an abstraction for a *destination*
558 that you are writing data *to*.
560 Examples of writable streams include:
562 * [http requests, on the client](http.html#http_class_http_clientrequest)
563 * [http responses, on the server](http.html#http_class_http_serverresponse)
564 * [fs write streams](fs.html#fs_class_fs_writestream)
568 * [child process stdin](child_process.html#child_process_child_stdin)
569 * [process.stdout][], [process.stderr][]
573 If a [`writable.write(chunk)`][] call returns false, then the `drain`
574 event will indicate when it is appropriate to begin writing more data
578 // Write the data to the supplied writable stream one million times.
579 // Be attentive to back-pressure.
580 function writeOneMillionTimes(writer, data, encoding, callback) {
589 writer.write(data, encoding, callback);
591 // see if we should continue, or wait
592 // don't pass the callback, because we're not done yet.
593 ok = writer.write(data, encoding);
595 } while (i > 0 && ok);
597 // had to stop early!
598 // write some more once it drains
599 writer.once('drain', write);
609 Emitted if there was an error when writing or piping data.
613 When the [`end()`][] method has been called, and all data has been flushed
614 to the underlying system, this event is emitted.
617 var writer = getWritableStreamSomehow();
618 for (var i = 0; i < 100; i ++) {
619 writer.write('hello, #' + i + '!\n');
621 writer.end('this is the end\n');
622 writer.on('finish', function() {
623 console.error('all writes are now complete.');
629 * `src` {[Readable][] Stream} source stream that is piping to this writable
631 This is emitted whenever the `pipe()` method is called on a readable
632 stream, adding this writable to its set of destinations.
635 var writer = getWritableStreamSomehow();
636 var reader = getReadableStreamSomehow();
637 writer.on('pipe', function(src) {
638 console.error('something is piping into the writer');
639 assert.equal(src, reader);
646 * `src` {[Readable][] Stream} The source stream that [unpiped][] this writable
648 This is emitted whenever the [`unpipe()`][] method is called on a
649 readable stream, removing this writable from its set of destinations.
652 var writer = getWritableStreamSomehow();
653 var reader = getReadableStreamSomehow();
654 writer.on('unpipe', function(src) {
655 console.error('something has stopped piping into the writer');
656 assert.equal(src, reader);
659 reader.unpipe(writer);
664 Forces buffering of all writes.
666 Buffered data will be flushed either at `.uncork()` or at `.end()` call.
668 #### writable.end([chunk][, encoding][, callback])
670 * `chunk` {String | Buffer} Optional data to write
671 * `encoding` {String} The encoding, if `chunk` is a String
672 * `callback` {Function} Optional callback for when the stream is finished
674 Call this method when no more data will be written to the stream. If
675 supplied, the callback is attached as a listener on the `finish` event.
677 Calling [`write()`][] after calling [`end()`][] will raise an error.
680 // write 'hello, ' and then end with 'world!'
681 var file = fs.createWriteStream('example.txt');
682 file.write('hello, ');
684 // writing more now is not allowed!
687 #### writable.setDefaultEncoding(encoding)
689 * `encoding` {String} The new default encoding
691 Sets the default encoding for a writable stream.
693 #### writable.uncork()
695 Flush all data, buffered since `.cork()` call.
697 #### writable.write(chunk[, encoding][, callback])
699 * `chunk` {String | Buffer} The data to write
700 * `encoding` {String} The encoding, if `chunk` is a String
701 * `callback` {Function} Callback for when this chunk of data is flushed
702 * Returns: {Boolean} True if the data was handled completely.
704 This method writes some data to the underlying system, and calls the
705 supplied callback once the data has been fully handled.
707 The return value indicates if you should continue writing right now.
708 If the data had to be buffered internally, then it will return
709 `false`. Otherwise, it will return `true`.
711 This return value is strictly advisory. You MAY continue to write,
712 even if it returns `false`. However, writes will be buffered in
713 memory, so it is best not to do this excessively. Instead, wait for
714 the `drain` event before writing more data.
717 ## API for Stream Implementors
721 To implement any sort of stream, the pattern is the same:
723 1. Extend the appropriate parent class in your own subclass. (The
724 [`util.inherits`][] method is particularly helpful for this.)
725 2. Call the appropriate parent class constructor in your constructor,
726 to be sure that the internal mechanisms are set up properly.
727 2. Implement one or more specific methods, as detailed below.
729 The class to extend and the method(s) to implement depend on the sort
730 of stream class you are writing:
742 <p>Method(s) to implement</p>
751 <p>[Readable](#stream_class_stream_readable_1)</p>
754 <p><code>[_read][]</code></p>
762 <p>[Writable](#stream_class_stream_writable_1)</p>
765 <p><code>[_write][]</code>, <code>_writev</code></p>
770 <p>Reading and writing</p>
773 <p>[Duplex](#stream_class_stream_duplex_1)</p>
776 <p><code>[_read][]</code>, <code>[_write][]</code>, <code>_writev</code></p>
781 <p>Operate on written data, then read the result</p>
784 <p>[Transform](#stream_class_stream_transform_1)</p>
787 <p><code>_transform</code>, <code>_flush</code></p>
792 In your implementation code, it is very important to never call the
793 methods described in [API for Stream Consumers][] above. Otherwise, you
794 can potentially cause adverse side effects in programs that consume
795 your streaming interfaces.
797 ### Class: stream.Duplex
801 A "duplex" stream is one that is both Readable and Writable, such as a
802 TCP socket connection.
804 Note that `stream.Duplex` is an abstract class designed to be extended
805 with an underlying implementation of the `_read(size)` and
806 [`_write(chunk, encoding, callback)`][] methods as you would with a
807 Readable or Writable stream class.
809 Since JavaScript doesn't have multiple prototypal inheritance, this
810 class prototypally inherits from Readable, and then parasitically from
811 Writable. It is thus up to the user to implement both the lowlevel
812 `_read(n)` method as well as the lowlevel
813 [`_write(chunk, encoding, callback)`][] method on extension duplex classes.
815 #### new stream.Duplex(options)
817 * `options` {Object} Passed to both Writable and Readable
818 constructors. Also has the following fields:
819 * `allowHalfOpen` {Boolean} Default=true. If set to `false`, then
820 the stream will automatically end the readable side when the
821 writable side ends and vice versa.
822 * `readableObjectMode` {Boolean} Default=false. Sets `objectMode`
823 for readable side of the stream. Has no effect if `objectMode`
825 * `writableObjectMode` {Boolean} Default=false. Sets `objectMode`
826 for writable side of the stream. Has no effect if `objectMode`
829 In classes that extend the Duplex class, make sure to call the
830 constructor so that the buffering settings can be properly
833 ### Class: stream.PassThrough
835 This is a trivial implementation of a [Transform][] stream that simply
836 passes the input bytes across to the output. Its purpose is mainly
837 for examples and testing, but there are occasionally use cases where
838 it can come in handy as a building block for novel sorts of streams.
840 ### Class: stream.Readable
844 `stream.Readable` is an abstract class designed to be extended with an
845 underlying implementation of the [`_read(size)`][] method.
847 Please see above under [API for Stream Consumers][] for how to consume
848 streams in your programs. What follows is an explanation of how to
849 implement Readable streams in your programs.
851 #### new stream.Readable([options])
854 * `highWaterMark` {Number} The maximum number of bytes to store in
855 the internal buffer before ceasing to read from the underlying
856 resource. Default=16kb, or 16 for `objectMode` streams
857 * `encoding` {String} If specified, then buffers will be decoded to
858 strings using the specified encoding. Default=null
859 * `objectMode` {Boolean} Whether this stream should behave
860 as a stream of objects. Meaning that stream.read(n) returns
861 a single value instead of a Buffer of size n. Default=false
863 In classes that extend the Readable class, make sure to call the
864 Readable constructor so that the buffering settings can be properly
867 #### readable.\_read(size)
869 * `size` {Number} Number of bytes to read asynchronously
871 Note: **Implement this method, but do NOT call it directly.**
873 This method is prefixed with an underscore because it is internal to the
874 class that defines it and should only be called by the internal Readable
875 class methods. All Readable stream implementations must provide a _read
876 method to fetch data from the underlying resource.
878 When _read is called, if data is available from the resource, `_read` should
879 start pushing that data into the read queue by calling `this.push(dataChunk)`.
880 `_read` should continue reading from the resource and pushing data until push
881 returns false, at which point it should stop reading from the resource. Only
882 when _read is called again after it has stopped should it start reading
883 more data from the resource and pushing that data onto the queue.
885 Note: once the `_read()` method is called, it will not be called again until
886 the `push` method is called.
888 The `size` argument is advisory. Implementations where a "read" is a
889 single call that returns data can use this to know how much data to
890 fetch. Implementations where that is not relevant, such as TCP or
891 TLS, may ignore this argument, and simply provide data whenever it
892 becomes available. There is no need, for example to "wait" until
893 `size` bytes are available before calling [`stream.push(chunk)`][].
895 #### readable.push(chunk[, encoding])
897 * `chunk` {Buffer | null | String} Chunk of data to push into the read queue
898 * `encoding` {String} Encoding of String chunks. Must be a valid
899 Buffer encoding, such as `'utf8'` or `'ascii'`
900 * return {Boolean} Whether or not more pushes should be performed
902 Note: **This method should be called by Readable implementors, NOT
903 by consumers of Readable streams.**
905 If a value other than null is passed, The `push()` method adds a chunk of data
906 into the queue for subsequent stream processors to consume. If `null` is
907 passed, it signals the end of the stream (EOF), after which no more data
910 The data added with `push` can be pulled out by calling the `read()` method
911 when the `'readable'`event fires.
913 This API is designed to be as flexible as possible. For example,
914 you may be wrapping a lower-level source which has some sort of
915 pause/resume mechanism, and a data callback. In those cases, you
916 could wrap the low-level source object by doing something like this:
919 // source is an object with readStop() and readStart() methods,
920 // and an `ondata` member that gets called when it has data, and
921 // an `onend` member that gets called when the data is over.
923 util.inherits(SourceWrapper, Readable);
925 function SourceWrapper(options) {
926 Readable.call(this, options);
928 this._source = getLowlevelSourceObject();
931 // Every time there's data, we push it into the internal buffer.
932 this._source.ondata = function(chunk) {
933 // if push() returns false, then we need to stop reading from source
934 if (!self.push(chunk))
935 self._source.readStop();
938 // When the source ends, we push the EOF-signaling `null` chunk
939 this._source.onend = function() {
944 // _read will be called when the stream wants to pull more data in
945 // the advisory size argument is ignored in this case.
946 SourceWrapper.prototype._read = function(size) {
947 this._source.readStart();
951 #### Example: A Counting Stream
955 This is a basic example of a Readable stream. It emits the numerals
956 from 1 to 1,000,000 in ascending order, and then ends.
959 var Readable = require('stream').Readable;
960 var util = require('util');
961 util.inherits(Counter, Readable);
963 function Counter(opt) {
964 Readable.call(this, opt);
969 Counter.prototype._read = function() {
970 var i = this._index++;
975 var buf = new Buffer(str, 'ascii');
981 #### Example: SimpleProtocol v1 (Sub-optimal)
983 This is similar to the `parseHeader` function described above, but
984 implemented as a custom stream. Also, note that this implementation
985 does not convert the incoming data to a string.
987 However, this would be better implemented as a [Transform][] stream. See
988 below for a better implementation.
991 // A parser for a simple data protocol.
992 // The "header" is a JSON object, followed by 2 \n characters, and
993 // then a message body.
995 // NOTE: This can be done more simply as a Transform stream!
996 // Using Readable directly for this is sub-optimal. See the
997 // alternative example below under the Transform section.
999 var Readable = require('stream').Readable;
1000 var util = require('util');
1002 util.inherits(SimpleProtocol, Readable);
1004 function SimpleProtocol(source, options) {
1005 if (!(this instanceof SimpleProtocol))
1006 return new SimpleProtocol(source, options);
1008 Readable.call(this, options);
1009 this._inBody = false;
1010 this._sawFirstCr = false;
1012 // source is a readable stream, such as a socket or file
1013 this._source = source;
1016 source.on('end', function() {
1020 // give it a kick whenever the source is readable
1021 // read(0) will not consume any bytes
1022 source.on('readable', function() {
1026 this._rawHeader = [];
1030 SimpleProtocol.prototype._read = function(n) {
1031 if (!this._inBody) {
1032 var chunk = this._source.read();
1034 // if the source doesn't have data, we don't have data yet.
1036 return this.push('');
1038 // check if the chunk has a \n\n
1040 for (var i = 0; i < chunk.length; i++) {
1041 if (chunk[i] === 10) { // '\n'
1042 if (this._sawFirstCr) {
1046 this._sawFirstCr = true;
1049 this._sawFirstCr = false;
1054 // still waiting for the \n\n
1055 // stash the chunk, and try again.
1056 this._rawHeader.push(chunk);
1059 this._inBody = true;
1060 var h = chunk.slice(0, split);
1061 this._rawHeader.push(h);
1062 var header = Buffer.concat(this._rawHeader).toString();
1064 this.header = JSON.parse(header);
1066 this.emit('error', new Error('invalid simple protocol data'));
1069 // now, because we got some extra data, unshift the rest
1070 // back into the read queue so that our consumer will see it.
1071 var b = chunk.slice(split);
1073 // calling unshift by itself does not reset the reading state
1074 // of the stream; since we're inside _read, doing an additional
1075 // push('') will reset the state appropriately.
1078 // and let them know that we are done parsing the header.
1079 this.emit('header', this.header);
1082 // from there on, just provide the data to our consumer.
1083 // careful not to push(null), since that would indicate EOF.
1084 var chunk = this._source.read();
1085 if (chunk) this.push(chunk);
1090 // var parser = new SimpleProtocol(source);
1091 // Now parser is a readable stream that will emit 'header'
1092 // with the parsed header data.
1095 ### Class: stream.Transform
1097 A "transform" stream is a duplex stream where the output is causally
1098 connected in some way to the input, such as a [zlib][] stream or a
1101 There is no requirement that the output be the same size as the input,
1102 the same number of chunks, or arrive at the same time. For example, a
1103 Hash stream will only ever have a single chunk of output which is
1104 provided when the input is ended. A zlib stream will produce output
1105 that is either much smaller or much larger than its input.
1107 Rather than implement the [`_read()`][] and [`_write()`][] methods, Transform
1108 classes must implement the `_transform()` method, and may optionally
1109 also implement the `_flush()` method. (See below.)
1111 #### new stream.Transform([options])
1113 * `options` {Object} Passed to both Writable and Readable
1116 In classes that extend the Transform class, make sure to call the
1117 constructor so that the buffering settings can be properly
1120 #### Events: 'finish' and 'end'
1122 The [`finish`][] and [`end`][] events are from the parent Writable
1123 and Readable classes respectively. The `finish` event is fired after
1124 `.end()` is called and all chunks have been processed by `_transform`,
1125 `end` is fired after all data has been output which is after the callback
1126 in `_flush` has been called.
1128 #### transform.\_flush(callback)
1130 * `callback` {Function} Call this function (optionally with an error
1131 argument) when you are done flushing any remaining data.
1133 Note: **This function MUST NOT be called directly.** It MAY be implemented
1134 by child classes, and if so, will be called by the internal Transform
1137 In some cases, your transform operation may need to emit a bit more
1138 data at the end of the stream. For example, a `Zlib` compression
1139 stream will store up some internal state so that it can optimally
1140 compress the output. At the end, however, it needs to do the best it
1141 can with what is left, so that the data will be complete.
1143 In those cases, you can implement a `_flush` method, which will be
1144 called at the very end, after all the written data is consumed, but
1145 before emitting `end` to signal the end of the readable side. Just
1146 like with `_transform`, call `transform.push(chunk)` zero or more
1147 times, as appropriate, and call `callback` when the flush operation is
1150 This method is prefixed with an underscore because it is internal to
1151 the class that defines it, and should not be called directly by user
1152 programs. However, you **are** expected to override this method in
1153 your own extension classes.
1155 #### transform.\_transform(chunk, encoding, callback)
1157 * `chunk` {Buffer | String} The chunk to be transformed. Will **always**
1158 be a buffer unless the `decodeStrings` option was set to `false`.
1159 * `encoding` {String} If the chunk is a string, then this is the
1160 encoding type. If chunk is a buffer, then this is the special
1161 value - 'buffer', ignore it in this case.
1162 * `callback` {Function} Call this function (optionally with an error
1163 argument and data) when you are done processing the supplied chunk.
1165 Note: **This function MUST NOT be called directly.** It should be
1166 implemented by child classes, and called by the internal Transform
1169 All Transform stream implementations must provide a `_transform`
1170 method to accept input and produce output.
1172 `_transform` should do whatever has to be done in this specific
1173 Transform class, to handle the bytes being written, and pass them off
1174 to the readable portion of the interface. Do asynchronous I/O,
1175 process things, and so on.
1177 Call `transform.push(outputChunk)` 0 or more times to generate output
1178 from this input chunk, depending on how much data you want to output
1179 as a result of this chunk.
1181 Call the callback function only when the current chunk is completely
1182 consumed. Note that there may or may not be output as a result of any
1183 particular input chunk. If you supply a second argument to the callback
1184 it will be passed to the push method. In other words the following are
1188 transform.prototype._transform = function (data, encoding, callback) {
1193 transform.prototype._transform = function (data, encoding, callback) {
1194 callback(null, data);
1198 This method is prefixed with an underscore because it is internal to
1199 the class that defines it, and should not be called directly by user
1200 programs. However, you **are** expected to override this method in
1201 your own extension classes.
1203 #### Example: `SimpleProtocol` parser v2
1205 The example above of a simple protocol parser can be implemented
1206 simply by using the higher level [Transform][] stream class, similar to
1207 the `parseHeader` and `SimpleProtocol v1` examples above.
1209 In this example, rather than providing the input as an argument, it
1210 would be piped into the parser, which is a more idiomatic Node.js stream
1214 var util = require('util');
1215 var Transform = require('stream').Transform;
1216 util.inherits(SimpleProtocol, Transform);
1218 function SimpleProtocol(options) {
1219 if (!(this instanceof SimpleProtocol))
1220 return new SimpleProtocol(options);
1222 Transform.call(this, options);
1223 this._inBody = false;
1224 this._sawFirstCr = false;
1225 this._rawHeader = [];
1229 SimpleProtocol.prototype._transform = function(chunk, encoding, done) {
1230 if (!this._inBody) {
1231 // check if the chunk has a \n\n
1233 for (var i = 0; i < chunk.length; i++) {
1234 if (chunk[i] === 10) { // '\n'
1235 if (this._sawFirstCr) {
1239 this._sawFirstCr = true;
1242 this._sawFirstCr = false;
1247 // still waiting for the \n\n
1248 // stash the chunk, and try again.
1249 this._rawHeader.push(chunk);
1251 this._inBody = true;
1252 var h = chunk.slice(0, split);
1253 this._rawHeader.push(h);
1254 var header = Buffer.concat(this._rawHeader).toString();
1256 this.header = JSON.parse(header);
1258 this.emit('error', new Error('invalid simple protocol data'));
1261 // and let them know that we are done parsing the header.
1262 this.emit('header', this.header);
1264 // now, because we got some extra data, emit this first.
1265 this.push(chunk.slice(split));
1268 // from there on, just provide the data to our consumer as-is.
1275 // var parser = new SimpleProtocol();
1276 // source.pipe(parser)
1277 // Now parser is a readable stream that will emit 'header'
1278 // with the parsed header data.
1281 ### Class: stream.Writable
1285 `stream.Writable` is an abstract class designed to be extended with an
1286 underlying implementation of the [`_write(chunk, encoding, callback)`][] method.
1288 Please see above under [API for Stream Consumers][] for how to consume
1289 writable streams in your programs. What follows is an explanation of
1290 how to implement Writable streams in your programs.
1292 #### new stream.Writable([options])
1294 * `options` {Object}
1295 * `highWaterMark` {Number} Buffer level when [`write()`][] starts
1296 returning false. Default=16kb, or 16 for `objectMode` streams
1297 * `decodeStrings` {Boolean} Whether or not to decode strings into
1298 Buffers before passing them to [`_write()`][]. Default=true
1299 * `objectMode` {Boolean} Whether or not the `write(anyObj)` is
1300 a valid operation. If set you can write arbitrary data instead
1301 of only `Buffer` / `String` data. Default=false
1303 In classes that extend the Writable class, make sure to call the
1304 constructor so that the buffering settings can be properly
1307 #### writable.\_write(chunk, encoding, callback)
1309 * `chunk` {Buffer | String} The chunk to be written. Will **always**
1310 be a buffer unless the `decodeStrings` option was set to `false`.
1311 * `encoding` {String} If the chunk is a string, then this is the
1312 encoding type. If chunk is a buffer, then this is the special
1313 value - 'buffer', ignore it in this case.
1314 * `callback` {Function} Call this function (optionally with an error
1315 argument) when you are done processing the supplied chunk.
1317 All Writable stream implementations must provide a [`_write()`][]
1318 method to send data to the underlying resource.
1320 Note: **This function MUST NOT be called directly.** It should be
1321 implemented by child classes, and called by the internal Writable
1324 Call the callback using the standard `callback(error)` pattern to
1325 signal that the write completed successfully or with an error.
1327 If the `decodeStrings` flag is set in the constructor options, then
1328 `chunk` may be a string rather than a Buffer, and `encoding` will
1329 indicate the sort of string that it is. This is to support
1330 implementations that have an optimized handling for certain string
1331 data encodings. If you do not explicitly set the `decodeStrings`
1332 option to `false`, then you can safely ignore the `encoding` argument,
1333 and assume that `chunk` will always be a Buffer.
1335 This method is prefixed with an underscore because it is internal to
1336 the class that defines it, and should not be called directly by user
1337 programs. However, you **are** expected to override this method in
1338 your own extension classes.
1340 #### writable.\_writev(chunks, callback)
1342 * `chunks` {Array} The chunks to be written. Each chunk has following
1343 format: `{ chunk: ..., encoding: ... }`.
1344 * `callback` {Function} Call this function (optionally with an error
1345 argument) when you are done processing the supplied chunks.
1347 Note: **This function MUST NOT be called directly.** It may be
1348 implemented by child classes, and called by the internal Writable
1351 This function is completely optional to implement. In most cases it is
1352 unnecessary. If implemented, it will be called with all the chunks
1353 that are buffered in the write queue.
1356 ## Simplified Constructor API
1360 In simple cases there is now the added benefit of being able to construct a stream without inheritance.
1362 This can be done by passing the appropriate methods as constructor options:
1368 var duplex = new stream.Duplex({
1370 // sets this._read under the hood
1372 // push data onto the read queue, passing null
1373 // will signal the end of the stream (EOF)
1376 write: function(chunk, encoding, next) {
1377 // sets this._write under the hood
1379 // An optional error can be passed as the first argument
1386 var duplex = new stream.Duplex({
1388 // sets this._read under the hood
1390 // push data onto the read queue, passing null
1391 // will signal the end of the stream (EOF)
1394 writev: function(chunks, next) {
1395 // sets this._writev under the hood
1397 // An optional error can be passed as the first argument
1405 var readable = new stream.Readable({
1407 // sets this._read under the hood
1409 // push data onto the read queue, passing null
1410 // will signal the end of the stream (EOF)
1418 var transform = new stream.Transform({
1419 transform: function(chunk, encoding, next) {
1420 // sets this._transform under the hood
1422 // generate output as many times as needed
1423 // this.push(chunk);
1425 // call when the current chunk is consumed
1428 flush: function(done) {
1429 // sets this._flush under the hood
1431 // generate output as many times as needed
1432 // this.push(chunk);
1441 var writable = new stream.Writable({
1442 write: function(chunk, encoding, next) {
1443 // sets this._write under the hood
1445 // An optional error can be passed as the first argument
1452 var writable = new stream.Writable({
1453 writev: function(chunks, next) {
1454 // sets this._writev under the hood
1456 // An optional error can be passed as the first argument
1462 ## Streams: Under the Hood
1470 Both Writable and Readable streams will buffer data on an internal
1471 object which can be retrieved from `_writableState.getBuffer()` or
1472 `_readableState.buffer`, respectively.
1474 The amount of data that will potentially be buffered depends on the
1475 `highWaterMark` option which is passed into the constructor.
1477 Buffering in Readable streams happens when the implementation calls
1478 [`stream.push(chunk)`][]. If the consumer of the Stream does not call
1479 `stream.read()`, then the data will sit in the internal queue until it
1482 Buffering in Writable streams happens when the user calls
1483 [`stream.write(chunk)`][] repeatedly, even when `write()` returns `false`.
1485 The purpose of streams, especially with the `pipe()` method, is to
1486 limit the buffering of data to acceptable levels, so that sources and
1487 destinations of varying speed will not overwhelm the available memory.
1489 ### Compatibility with Older Node.js Versions
1493 In versions of Node.js prior to v0.10, the Readable stream interface was
1494 simpler, but also less powerful and less useful.
1496 * Rather than waiting for you to call the `read()` method, `'data'`
1497 events would start emitting immediately. If you needed to do some
1498 I/O to decide how to handle data, then you had to store the chunks
1499 in some kind of buffer so that they would not be lost.
1500 * The [`pause()`][] method was advisory, rather than guaranteed. This
1501 meant that you still had to be prepared to receive `'data'` events
1502 even when the stream was in a paused state.
1504 In Node.js v0.10, the Readable class described below was added.
1505 For backwards compatibility with older Node.js programs, Readable streams
1506 switch into "flowing mode" when a `'data'` event handler is added, or
1507 when the [`resume()`][] method is called. The effect is that, even if
1508 you are not using the new `read()` method and `'readable'` event, you
1509 no longer have to worry about losing `'data'` chunks.
1511 Most programs will continue to function normally. However, this
1512 introduces an edge case in the following conditions:
1514 * No [`'data'` event][] handler is added.
1515 * The [`resume()`][] method is never called.
1516 * The stream is not piped to any writable destination.
1518 For example, consider the following code:
1522 net.createServer(function(socket) {
1524 // we add an 'end' method, but never consume the data
1525 socket.on('end', function() {
1526 // It will never get here.
1527 socket.end('I got your message (but didnt read it)\n');
1533 In versions of Node.js prior to v0.10, the incoming message data would be
1534 simply discarded. However, in Node.js v0.10 and beyond,
1535 the socket will remain paused forever.
1537 The workaround in this situation is to call the `resume()` method to
1538 start the flow of data:
1542 net.createServer(function(socket) {
1544 socket.on('end', function() {
1545 socket.end('I got your message (but didnt read it)\n');
1548 // start the flow of data, discarding it.
1554 In addition to new Readable streams switching into flowing mode,
1555 pre-v0.10 style streams can be wrapped in a Readable class using the
1563 Normally, Streams operate on Strings and Buffers exclusively.
1565 Streams that are in **object mode** can emit generic JavaScript values
1566 other than Buffers and Strings.
1568 A Readable stream in object mode will always return a single item from
1569 a call to `stream.read(size)`, regardless of what the size argument
1572 A Writable stream in object mode will always ignore the `encoding`
1573 argument to `stream.write(data, encoding)`.
1575 The special value `null` still retains its special value for object
1576 mode streams. That is, for object mode readable streams, `null` as a
1577 return value from `stream.read()` indicates that there is no more
1578 data, and [`stream.push(null)`][] will signal the end of stream data
1581 No streams in Node.js core are object mode streams. This pattern is only
1582 used by userland streaming libraries.
1584 You should set `objectMode` in your stream child class constructor on
1585 the options object. Setting `objectMode` mid-stream is not safe.
1587 For Duplex streams `objectMode` can be set exclusively for readable or
1588 writable side with `readableObjectMode` and `writableObjectMode`
1589 respectively. These options can be used to implement parsers and
1590 serializers with Transform streams.
1593 var util = require('util');
1594 var StringDecoder = require('string_decoder').StringDecoder;
1595 var Transform = require('stream').Transform;
1596 util.inherits(JSONParseStream, Transform);
1598 // Gets \n-delimited JSON string data, and emits the parsed objects
1599 function JSONParseStream() {
1600 if (!(this instanceof JSONParseStream))
1601 return new JSONParseStream();
1603 Transform.call(this, { readableObjectMode : true });
1606 this._decoder = new StringDecoder('utf8');
1609 JSONParseStream.prototype._transform = function(chunk, encoding, cb) {
1610 this._buffer += this._decoder.write(chunk);
1611 // split on newlines
1612 var lines = this._buffer.split(/\r?\n/);
1613 // keep the last partial line buffered
1614 this._buffer = lines.pop();
1615 for (var l = 0; l < lines.length; l++) {
1616 var line = lines[l];
1618 var obj = JSON.parse(line);
1620 this.emit('error', er);
1623 // push the parsed object out to the readable consumer
1629 JSONParseStream.prototype._flush = function(cb) {
1630 // Just handle any leftover
1631 var rem = this._buffer.trim();
1634 var obj = JSON.parse(rem);
1636 this.emit('error', er);
1639 // push the parsed object out to the readable consumer
1646 ### `stream.read(0)`
1648 There are some cases where you want to trigger a refresh of the
1649 underlying readable stream mechanisms, without actually consuming any
1650 data. In that case, you can call `stream.read(0)`, which will always
1653 If the internal read buffer is below the `highWaterMark`, and the
1654 stream is not currently reading, then calling `read(0)` will trigger
1655 a low-level `_read` call.
1657 There is almost never a need to do this. However, you will see some
1658 cases in Node.js's internals where this is done, particularly in the
1659 Readable stream class internals.
1661 ### `stream.push('')`
1663 Pushing a zero-byte string or Buffer (when not in [Object mode][]) has an
1664 interesting side effect. Because it *is* a call to
1665 [`stream.push()`][], it will end the `reading` process. However, it
1666 does *not* add any data to the readable buffer, so there's nothing for
1669 Very rarely, there are cases where you have no data to provide now,
1670 but the consumer of your stream (or, perhaps, another bit of your own
1671 code) will know when to check again, by calling `stream.read(0)`. In
1672 those cases, you *may* call `stream.push('')`.
1674 So far, the only use case for this functionality is in the
1675 [tls.CryptoStream][] class, which is deprecated in Node.js/io.js v1.0. If you
1676 find that you have to use `stream.push('')`, please consider another
1677 approach, because it almost certainly indicates that something is
1680 [EventEmitter]: events.html#events_class_events_eventemitter
1681 [Object mode]: #stream_object_mode
1682 [`stream.push(chunk)`]: #stream_readable_push_chunk_encoding
1683 [`stream.push(null)`]: #stream_readable_push_chunk_encoding
1684 [`stream.push()`]: #stream_readable_push_chunk_encoding
1685 [`unpipe()`]: #stream_readable_unpipe_destination
1686 [unpiped]: #stream_readable_unpipe_destination
1687 [tcp sockets]: net.html#net_class_net_socket
1688 [zlib streams]: zlib.html
1690 [crypto streams]: crypto.html
1691 [crypto]: crypto.html
1692 [tls.CryptoStream]: tls.html#tls_class_cryptostream
1693 [process.stdin]: process.html#process_process_stdin
1694 [stdout]: process.html#process_process_stdout
1695 [process.stdout]: process.html#process_process_stdout
1696 [process.stderr]: process.html#process_process_stderr
1697 [child process stdout and stderr]: child_process.html#child_process_child_stdout
1698 [API for Stream Consumers]: #stream_api_for_stream_consumers
1699 [API for Stream Implementors]: #stream_api_for_stream_implementors
1700 [Readable]: #stream_class_stream_readable
1701 [Writable]: #stream_class_stream_writable
1702 [Duplex]: #stream_class_stream_duplex
1703 [Transform]: #stream_class_stream_transform
1704 [`end`]: #stream_event_end
1705 [`finish`]: #stream_event_finish
1706 [`_read(size)`]: #stream_readable_read_size_1
1707 [`_read()`]: #stream_readable_read_size_1
1708 [_read]: #stream_readable_read_size_1
1709 [`writable.write(chunk)`]: #stream_writable_write_chunk_encoding_callback
1710 [`write(chunk, encoding, callback)`]: #stream_writable_write_chunk_encoding_callback
1711 [`write()`]: #stream_writable_write_chunk_encoding_callback
1712 [`stream.write(chunk)`]: #stream_writable_write_chunk_encoding_callback
1713 [`_write(chunk, encoding, callback)`]: #stream_writable_write_chunk_encoding_callback_1
1714 [`_write()`]: #stream_writable_write_chunk_encoding_callback_1
1715 [_write]: #stream_writable_write_chunk_encoding_callback_1
1716 [`util.inherits`]: util.html#util_util_inherits_constructor_superconstructor
1717 [`end()`]: #stream_writable_end_chunk_encoding_callback
1718 [`'data'` event]: #stream_event_data
1719 [`resume()`]: #stream_readable_resume
1720 [`readable.resume()`]: #stream_readable_resume
1721 [`pause()`]: #stream_readable_pause
1722 [`unpipe()`]: #stream_readable_unpipe_destination
1723 [`pipe()`]: #stream_readable_pipe_destination_options