Name: ffmpeg URL: http://ffmpeg.org/ License: LGPL 2.1 License File: LICENSE.md Upstream Git: git://source.ffmpeg.org/ffmpeg.git Last Upstream Merge: afd62b3f184569492230e7f34ad55581c6b2d2c3, Aug 1 2014 This file documents the layout of the Chromium copy of FFmpeg git, some common tasks, how to produce the FFmpeg include directory, and how to create the ffmpeg.gyp file and related configurations. FFmpeg Git Layout: ================== Chromium's copy of FFmpeg is a fork of the upstream git repository, see the tags above for up-to-date cut information. All of the chromium specific bits, minus the gyp files, are located under the chromium/ folder off the root. ffmpeg.gyp: See the section on ffmpeg.gyp below. ffmpeg_generated.gypi: Pregenerated gyp listing of files necessary to build every platform. See chromium/scripts/generate_gyp.py for more details. chromium/binaries/c99conv.exe: Prebuilt binary for converting FFmpeg sources from C99 to C89 for compilation with Visual C++. chromium/scripts: Utilities for building the gyp and config files. chromium/config/...: Pregenerated FFmpeg config options for each platform and architecture; e.g., ChromeOS: ARM-Neon, Linux: X64, etc. chromium/patches/...: Chromium specific changes which haven't yet made it upstream. See chromium/patches/README for more details on each patch. Historically, the .patch files were staged on top of a source tarball instead of the Git repository we have now. The .patch files are kept for tracking purposes. The new system only requires that you add an entry to the README file with a link to the tracking issue and code review. Help w/ Common Tasks: ===================== -- Submitting changes to Chromium's FFmpeg git repository. The goal of Chromium's FFmpeg repository is to just be a mirror of the upstream Git repository. Which means every change made must be upstreamed. If you make a change, please add an entry to chromium/patches/README with a link to the tracking issue and code review for your change. Unfortunately the normal Chromium CL submission process through Rietveld does not work with Git DEPS like FFmpeg, so you must use Gerrit to upload your change for review: git push ssh://gerrit.chromium.org:29418/chromium/third_party/ffmpeg \ HEAD:refs/for/master This will create a gerrit chanage. Make sure to "git commit --amend" and add "Change-Id: XXX" as the last line so that future pushes do not create new changes. You can get the "XXX" from the gerrit web interface. Note: You'll need a Gerrit account, see: http://www.chromium.org/chromium-os/developer-guide/gerrit-guide If your change adds new files to the repository, you'll need to regenerate the GYP defines by following the directions in chromium/scripts/generate_gyp.py. After all that's done and your change has landed in the Git repository, you'll need to roll DEPS before the change will show up in Chrome. -- Performing an upstream merge. The upstream merge process follows the normal Git merge process: # First, modify the origin url to enable push of merge result later. # e.g., change remote.origin.url from: # https://chromium.googlesource.com/chromium/third_party/ffmpeg.git # to: # ssh://gerrit.chromium.org:29418/chromium/third_party/ffmpeg.git # One way is to use custom_deps in .gclient. Another might be to run # git config -e within this folder and edit the URLs if already synced, # though this alternative could have danger. git remote add upstream git://source.ffmpeg.org/ffmpeg.git git fetch upstream git merge upstream/master # If conflicts: git mergetool # Use git push for submission so commit history is kept. Might require merge # approval on your Gerrit account. Do not commit any other way. git push origin master Once complete you'll need to regenerate the configuration files for every platform (see "Short Directions" in the "Building" section below). You'll also need to regenerate the gyp files (see chromium/scripts/generate_gyp.py). After all that is done, you can update Chromium's root DEPS file to point to the tip of tree commit. -- Upstreaming a patch. git checkout -b my_patch git apply git rebase origin/master < Follow FFmpeg guide: http://ffmpeg.org/developer.html#Submitting-patches-1 > Once your change has been upstreamed, please update the chromium/patches/README file with the status. The next time an upstream merge is done, the committer will clear out all the patches which have been upstreamed. -- Cherry-picking a patch from upstream. # Tell our repo about the upstream one. git remote add upstream git://source.ffmpeg.org/ffmpeg.git git fetch upstream # Create a new branch based off of master for committing our patch. git checkout master -b my_new_branch # Pull the patch out of upstream. git cherry-pick Building: ========= -- FFmpeg headers in the 'chromium/include' directory. The include directory contains FFmpeg's public header files from the output of a "make install" command. The header files are from Chromium's copy of FFmpeg. Steps to reproduce: 1) If on Windows, refer to our MinGW/MSYS environment setup: http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/mingw/ 2) Chromium's copy of FFmpeg, should already have it if you're reading this. 3) Follow the instructions to build and install. 4) Go to your install location and copy the following into the Chromium tree: /path/to/install/include/libavcodec /path/to/install/include/libavformat /path/to/install/include/libavutil On Windows, the libraries are linked in using /DELAYLOAD to avoid having the DLLs present at run-time. On POSIX systems, dlopen() is used to achieve a similar effect. We don't use the import libraries generated from building FFmpeg because they export every method by ordinal, which makes binary compatibility with different builds of FFmpeg difficult if not impossible. Furthermore, it is much easier to update a DEF file instead of rebuilding FFmpeg to generate new import libraries. -- Recreating the ffmpeg.gyp file and populating the config directory. The ffmpeg.gyp file is meant to be used in place of FFmpeg's ./configure && make steps. The file was created by inspecting the build log from above. The FFmpeg build is relatively straightforward. All files are built with the same CFLAGS. The config.h and version.h files are the only files generated by ./configure that are included elsewhere. They require a small bit of post-processing. Other than the configure step, FFmpeg just compiles its .c files, assembles a few more using yasm, and that's it. Exact instructions for reproducing ffmpeg.gyp are in the "Detailed Directions" section. Here is a list of gotchas that have shown up. 1) FFmpeg requires special configure (--disable-optimizations) in order to be built with -O0 successfully due to some of the hand-written assembler using ebp. -O0 implies -fno-omit-frame-pointer which breaks this. This will produce compiler errors like: libavcodec/cabac.h:527: error: can't find a register in class 'GENERAL_REGS' while reloading 'asm' cabac.h:527: error: 'asm' operand has impossible constraints 2) Sometimes, with -O0, invalid code will be exposed because dead-branch pruning is disabled in gcc. This can manifest itself as strange link issues or compile issues. Be careful to read all warnings in this case. 3) Since config.h is generated via ./configure, the generated file will be sensitive to the configuration of the machine it was produced on. In particular, yasm does not seem to always be detected if cross-compiling for 32-bit on a 64-bit machine. Since yasm is built in tree, make sure to force things with --enable-yasm. 4) yasm needs to be installed on mac and windows if not already there. 5) Similar to issue #3, ./configure may detect the presence of SDL and adjust config.h accordingly. This is harmless because all the SDL related code has been disabled in our configuration. 6) On Mac ia32, we want to be able to compile WITHOUT -fomit-frame-pointer (so breakpad can function). To do this, we need to disable the use of the EBP register, otherwise some of FFmpeg's inline assembly will cause compilation errors similar to gotcha #1. For more details, see the file comment in the munge_config_optimizations.sh. This script will fix up the generated config.h to be building without -fomit-frame-pointer. 7) On Windows, FFmpeg must first be run through a preprocessor to be compiled via MSVC++ because it doesn't support the C99 syntax used by FFmpeg. The converter is open source and built on libclang. The c99conv.exe in the chromium/scripts directory is a statically linked version of the converter from: https://github.com/rbultje/c99-to-c89 . Compiling the dynamic linked library is relatively easy, generating a statically linked one is not. To build the dynamic version you first need libclang: git clone http://llvm.org/git/llvm.git cd tools; git clone http://llvm.org/git/clang.git Now you need to run cmake, see http://llvm.org/docs/GettingStartedVS.html, step #5. From there you can start the Visual Studio IDE with the .vcproj or .sln files generated by cmake. Then select "Release" from the build drop down menu. You're ready to build. Right click on the libclang target and select build. Once complete you should have a libclang.dll in \out\bin\Release. Now you can build the c99-to-c89 converter using the packed makefile. If you're building the dynamic version you'll need to ensure c99conv.exe and libclang.dll remain in the same directory. Note: Building the statically linked version is painful and requires changes to both the llvm project and the converter. Proceed at your own risk. First, you need to modify the libclang project and change every target's "Project Defaults" from "Dynamic Library .DLL" to "Static Library (.lib)" and under "Code Generation" change every targets "Runtime Library" from "Multi-threaded DLL (/MD)" to "Multi-threaded (/MT)" then rebuild the libclang project. Afterward, you will have a lot of lib files in \out\lib\Release and will need to specify all of them under LIBS in the c99-to-c89 makefile as well as Advapi32.lib and Shell32.lib. Before compiling c99conv.exe you will need to modify libclang a bit, open "tools\clang\include\clang-c\Platform.h" and remove the "__declspec(dllimport)" entry. You can now compile as normal and will only need to package the c99conv.exe. It's not necessary to build the statically linked version, but it's nice to only have a single binary checked in. 8) On various platforms, ffmpeg configure may detect external iconv library and include it by default. FFmpeg used by chromium does not depend on this library, and the lib may not exist on target despite existing in the build environment. Hence, we need to change CONFIG_ICONV to 0 in the config.h (and config.asm where appropriate.) build_ffmpeg.sh does this with configure parameter --disable-iconv. See http://crbug.com/223295 Short Directions: ================= 1) Create config.h and config.asm as needed. On Linux (with Chromium's yasm build output in $PATH) run ./chromium/scripts/build_ffmpeg.py linux ia32 ./chromium/scripts/build_ffmpeg.py linux x64 ./chromium/scripts/build_ffmpeg.py linux-noasm x64 On Linux with MIPS cross-toolchain in $PATH ./chromium/scripts/build_ffmpeg.py linux mipsel On Linux chroot run ./chromium/scripts/build_ffmpeg.py linux arm ./chromium/scripts/build_ffmpeg.py linux arm-neon On Mac run ./chromium/scripts/build_ffmpeg.py mac ia32 ./chromium/scripts/build_ffmpeg.py mac x64 On Windows run ./chromium/scripts/build_ffmpeg.py win ia32 ./chromium/scripts/build_ffmpeg.py win x64 2) Finally, collect all these directories and copy all config files into the source tree using ./chromium/scripts/copy_config.sh Detailed Directions: ==================== 1) Run the configure in a directory out of the tree with the arguments you want. To see what was used before, find the config.h for the platform of interest in: src/third_party/ffmpeg/source/config/[branding]/[platform]/[variant] The value of the FFMPEG_CONFIGURATION macro should have the configure commandline that generated the file. Note that if you are trying to build a 32-bit FFmpeg for linux on a 64-bit box, the extra flags you want to pass to ./configure are --arch=i686 --extra-cflags=-m32 --extra-ldflags=-m32 Also, as noted in gotcha #4, explicitly setting --enable-yasm is a good idea. (These flags have been added to build_ffmpeg.sh.) 2) Copy the newly generated config.h and version.h into the correct platform location: src/third_party/ffmpeg/chromium/config/[branding]/[platform]/[variant] Make sure to double-check that config.h and version.h are the only files of interest. By that, I mean check that the other generated files are makefiles, documentation, .pc files, or something else that is not relevant to our build. TODO(ajwong): Check if we can modify version.h to tag our builds. 3) If on ia32, handle gotcha #6 by munging the geneated config.h file to disable use of EBP. Call the munge_config_optimizations.sh script on the config.h for each ia32 variant. (This has been implemented in build_ffmpeg.sh.) ** This script is not idempotent. Don't run it twice ** Remember, this is only necessary for ia32 config.h files. Running this on config.h files for other platforms (in particular, for x64) will likely result in unecessarily slow code, or compile failures. 4) Handle gotcha #8 by using --disable-iconv configure option. This is implemented in build_ffmpeg.sh. 5) Next, capture all the output from a build of libavcodec.so and libavformat.so. We will use the build log as a reference for making the ffmpeg.gyp file. make libavcodec/libavcodec.so libavformat/libavformat.so \ > ffmpeg_build_log 2> ffmpeg_build_err For Mac, replace the ".so" in the files above with ".dylib". To get detailed output you might have to comment in common.mak #$(foreach VAR,$(BRIEF), \ # $(eval override $(VAR) = $($(VAR)))) 6) Check ffmpeg_build_err to see if there are any significant anomalies. FFmpeg source generates a lot of compiler warnings; it is safe to ignore those. 7) Examine all non-gcc commands to see if we're missing anything interesting: grep -v '^gcc' ffmpeg_build_log There should be yasm commands for assembling two yasm files, but nothing else. Include those yasm files in the sources list for gyp. That means grep -v '^gcc\|^yasm' should generate nothing beyond "cd" and "ln" commands. 8) Verify that the all the gcc commands have the same compiler flags. Do that with the following "one-liner": grep - '^gcc' ffmpeg_build_log | grep -v ' -MM ' | grep -v ' -shared ' | sed -e 's/ -MF .*$//' | sort | uniq -c This should find all gcc commands, exclude the dependency generation lines, the link lines, and strip the output/input file names leaving just the compiler flags + invocation. You should only see one "line" of output. If there is more than one, figure out if the differences in compiler flags are significant, and then use your best judgment. 9) Examine the output from step 7 and update the compiler flags in ffmpeg.gyp. For easier cut/paste, append the following to the previous command line to isolate each flag on its own line and add single-quotes: tr -s ' ' | tr ' ' '\n' | sed -e "s/\(.*\)/'\1',/" | sort -u 10) Next, examine the link flags to see if anything interesting appears. grep ' -shared ' ffmpeg_build_log | tr ' ' '\n' | grep -Ev '^[^-].*' | grep -v rpath | grep -Ev '^-L' | sort -u This should find all link lines, move each flag to its own line, remove any argument that isn't a flag, remove all the rpaths (not useful for us anyways), and remove all the -L lines (also not useful for us). The most interesting will likely be the -Wl,.* lines. Update the ldflags section in ffmpeg.gyp accordingly. 11) Lastly, Find all the build .c files and update the sources line (this is very similar to step 7): grep -E '^gcc' ffmpeg_build_log | grep -v ' -MM ' | grep -v ' -shared ' | sed -e "s|.* -o .* \(.*\)$|'source/patched-ffmpeg/\1',|" | sort 12) Attempt to build. :) *13) Update the the sources! clause to exclude files that should only be built for Chromium. For this, you basically need to do the steps above once with the configure options for Chrome, then once with the options for Chromium and diff the list of .c and .asm source files.