3 Welcome to Chromium Media! This directory primarily contains a collection of
4 components related to media capture and playback. Feel free to reach out to the
5 media-dev@chromium.org mailing list with questions.
7 As a top level component this may be depended on by almost every other Chromium
8 component except base/. Certain components may not work properly in sandboxed
15 * audio/ - Code for audio input and output. Includes platform specific output
16 and input implementations. Due to use of platform APIs, can not normally be used
17 from within a sandboxed process.
19 * base/ - Contains miscellaneous enums, utility classes, and shuttling
20 primitives used throughout `media/` and beyond; i.e. `AudioBus`, `AudioCodec`, and
21 `VideoFrame` just to name a few. Can be used in any process.
23 * blink/ - Code for interfacing with the Blink rendering engine for `MediaStreams`
24 as well as `<video>` and `<audio>` playback. Used only in the same process as Blink;
25 typically the render process.
27 * capture/ - Contains content (as in the content layer) capturing and platform
28 specific video capture implementations.
30 * cast/ - Contains the tab casting implementation; not to be confused with the
31 Chromecast code which lives in the top-level cast/ directory.
33 * cdm/ - Contains code related to the Content Decryption Module (CDM) used for
34 playback of content via Encrypted Media Extensions (EME).
36 * device_monitors/ - Contains code for monitoring device changes; e.g. webcam
37 and microphone plugin and unplug events.
39 * ffmpeg/ - Contains binding code and helper methods necessary to use the ffmpeg
40 library located in //third_party/ffmpeg.
42 * filters/ - Contains data sources, decoders, demuxers, parsers, and rendering
43 algorithms used for media playback.
45 * formats/ - Contains parsers used by Media Source Extensions (MSE).
47 * gpu/ - Contains the platform hardware encoder and decoder implementations.
49 * midi/ - Contains the WebMIDI API implementation.
51 * mojo/ - Contains mojo services for media. Typically used for providing out of
52 process media functionality to a sandboxed process.
54 * muxers/ - Code for muxing content for the Media Recorder API.
56 * remoting/ - Code for transmitting muxed packets to a remote endpoint for
59 * renderers/ - Code for rendering audio and video to an output sink.
61 * test/ - Code and data for testing the media playback pipeline.
63 * tools/ - Standalone media test tools.
65 * video/ - Abstract hardware video decoder interfaces and tooling.
71 TODO(miu, chfemer): Fill in this section.
77 See [media/mojo documentation](/media/mojo).
83 TODO(toyoshim): Fill in this section.
89 Media playback encompasses a large swatch of technologies, so by necessity this
90 will provide only a brief outline. Inside this directory you'll find components
91 for media demuxing, software and hardware video decode, audio output, as well as
92 audio and video rendering.
94 Specifically under the playback heading, media/ contains the implementations of
95 components required for HTML media elements and extensions:
97 * [HTML5 Audio & Video](https://www.w3.org/html/wg/spec/video.html)
98 * [Media Source Extensions](https://www.w3.org/TR/media-source/)
99 * [Encrypted Media Extensions](https://www.w3.org/TR/encrypted-media/)
101 The following diagram provides a simplified overview of the media playback
104 ![Media Pipeline Overview](/docs/media/media_pipeline_overview.png)
106 As a case study we'll consider the playback of a video through the `<video>` tag.
108 `<video>` (and `<audio>`) starts in `blink::HTMLMediaElement` in
109 third_party/blink/ and reaches third_party/blink/public/platform/media/ in
110 `media::WebMediaPlayerImpl` after a brief hop through `content::MediaFactory`.
111 Each `blink::HTMLMediaElement` owns a `media::WebMediaPlayerImpl` for handling
112 things like play, pause, seeks, and volume changes (among other things).
114 `media::WebMediaPlayerImpl` handles or delegates media loading over the network
115 as well as demuxer and pipeline initialization. `media::WebMediaPlayerImpl`
116 owns a `media::PipelineController` which manages the coordination of a
117 `media::DataSource`, `media::Demuxer`, and `media::Renderer` during playback.
119 During a normal playback, the `media::Demuxer` owned by WebMediaPlayerImpl may
120 be either `media::FFmpegDemuxer` or `media::ChunkDemuxer`. The ffmpeg variant
121 is used for standard src= playback where WebMediaPlayerImpl is responsible for
122 loading bytes over the network. `media::ChunkDemuxer` is used with Media Source
123 Extensions (MSE), where JavaScript code provides the muxed bytes.
125 The media::Renderer is typically `media::RendererImpl` which owns and
126 coordinates `media::AudioRenderer` and `media::VideoRenderer` instances. Each
127 of these in turn own a set of `media::AudioDecoder` and `media::VideoDecoder`
128 implementations. Each issues an async read to a `media::DemuxerStream` exposed
129 by the `media::Demuxer` which is routed to the right decoder by
130 `media::DecoderStream`. Decoding is again async, so decoded frames are
131 delivered at some later time to each renderer.
133 The media/ library contains hardware decoder implementations in media/gpu for
134 all supported Chromium platforms, as well as software decoding implementations
135 in media/filters backed by FFmpeg and libvpx. Decoders are attempted in the
136 order provided via the `media::RendererFactory`; the first one which reports
137 success will be used for playback (typically the hardware decoder for video).
139 Each renderer manages timing and rendering of audio and video via the event-
140 driven `media::AudioRendererSink` and `media::VideoRendererSink` interfaces
141 respectively. These interfaces both accept a callback that they will issue
142 periodically when new audio or video frames are required.
144 On the audio side, again in the normal case, the `media::AudioRendererSink` is
145 driven via a `base::SyncSocket` and shared memory segment owned by the browser
146 process. This socket is ticked periodically by a platform level implementation
147 of `media::AudioOutputStream` within media/audio.
149 On the video side, the `media::VideoRendererSink` is driven by async callbacks
150 issued by the compositor to `media::VideoFrameCompositor`. The
151 `media::VideoRenderer` will talk to the `media::AudioRenderer` through a
152 `media::TimeSource` for coordinating audio and video sync.
154 With that we've covered the basic flow of a typical playback. When debugging
155 issues, it's helpful to review the internal logs at chrome://media-internals.
156 The internals page contains information about active
157 `media::WebMediaPlayerImpl`, `media::AudioInputController`,
158 `media::AudioOutputController`, and `media::AudioOutputStream` instances.
164 Media playback typically involves multiple threads, in many cases even multiple
165 processes. Media operations are often asynchronous running in a sandbox. These
166 make attaching a debugger (e.g. GDB) sometimes less efficient than other
167 mechanisms like logging.
171 In media we use DVLOG() a lot. It makes filename-based filtering super easy.
172 Within one file, not all logs are created equal. To make log filtering
173 more convenient, use appropriate log levels. Here are some general
176 * DVLOG(1): Once per playback events or other important events, e.g.
177 construction/destruction, initialization, playback start/end, suspend/resume,
178 any error conditions.
179 * DVLOG(2): Recurring events per playback, e.g. seek/reset/flush, config change.
180 * DVLOG(3): Frequent events, e.g. demuxer read, audio/video buffer decrypt or
181 decode, audio/video frame rendering.
185 MediaLog will send logs to `about://media-internals`, which is easily accessible
186 by developers (including web developes), testers and even users to get detailed
187 information about a playback instance. For guidance on how to use MediaLog, see
188 `media/base/media_log.h`.
190 MediaLog messages should be concise and free of implementation details. Error
191 messages should provide clues as to how to fix them, usually by precisely
192 describing the circumstances that led to the error. Use properties, rather
193 than messages, to record metadata and state changes.
197 When adding logs, it's often helpful to log the function name, e.g.
199 DVLOG(?) << __func__;
202 When adding logs with values, prefer the following format for consistency and
205 DVLOG(?) << __func__ << ": param1=" << param1 << ", param2=" << param2;