`PyTorch.org Tutorial Contribution
Guide <https://github.com/pytorch/tutorials/#contributing>`__
-
-Code Style
-~~~~~~~~~~
-
-**Python style**
-
-**C++ style**
-
-Submitting a Pull Request
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-PyTorch development happens publicly on our Github repo.
-
-To have your feature or fix added to PyTorch, please submit a Pull
-Request.
-
-Running Tests
-~~~~~~~~~~~~~
-
-Show examples for running all tests, just one individual...
-
-Technical Process
------------------
-
-Developing PyTorch
-~~~~~~~~~~~~~~~~~~
-
-To develop PyTorch on your machine, here are some tips:
-
-1. Uninstall all existing PyTorch installs:
-
-::
-
- conda uninstall pytorch
- pip uninstall torch
- pip uninstall torch # run this command twice
-
-2. Clone a copy of PyTorch from source:
-
-::
-
- git clone https://github.com/pytorch/pytorch
- cd pytorch
-
-3. Install PyTorch in ``build develop`` mode:
-
-A full set of instructions on installing PyTorch from source is here:
-https://github.com/pytorch/pytorch#from-source
-
-The change you have to make is to replace
-
-::
-
- python setup.py install
-
-with
-
-::
-
- python setup.py build develop
-
-This is especially useful if you are only changing Python files.
-
-This mode will symlink the Python files from the current local source
-tree into the Python install.
-
-Hence, if you modify a Python file, you do not need to reinstall PyTorch
-again and again.
-
-For example:
-
-- Install local PyTorch in ``build develop`` mode
-- modify your Python file ``torch/__init__.py`` (for example)
-- test functionality
-- modify your Python file ``torch/__init__.py``
-- test functionality
-- modify your Python file ``torch/__init__.py``
-- test functionality
-
-You do not need to repeatedly install after modifying Python files.
-
-In case you want to reinstall, make sure that you uninstall PyTorch
-first by running ``pip uninstall torch`` and ``python setup.py clean``.
-Then you can install in ``build develop`` mode again.
-
-Codebase structure
-------------------
-
-- `c10 <https://github.com/pytorch/pytorch/blob/master/c10>`__ - Core
- library files that work everywhere, both server and mobile. We are
- slowly moving pieces from
- `ATen/core <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core>`__
- here. This library is intended only to contain essential
- functionality, and appropriate to use in settings where binary size
- matters. (But you'll have a lot of missing functionality if you try
- to use it directly.)
-- `aten <https://github.com/pytorch/pytorch/blob/master/aten>`__ - C++
- tensor library for PyTorch (no autograd support)
-
- - `src <https://github.com/pytorch/pytorch/blob/master/aten/src>`__
-
- - `TH <https://github.com/pytorch/pytorch/blob/master/aten/src/TH>`__
- `THC <https://github.com/pytorch/pytorch/blob/master/aten/src/THC>`__
- `THNN <https://github.com/pytorch/pytorch/blob/master/aten/src/THNN>`__
- `THCUNN <https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN>`__
- - Legacy library code from the original Torch. Try not to add
- things here; we're slowly porting these to
- `native <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native>`__.
-
- - generic - Contains actual implementations of operators,
- parametrized over ``scalar_t``. Files here get compiled N
- times per supported scalar type in PyTorch.
-
- - `ATen <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen>`__
-
- - `core <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core>`__
- - Core functionality of ATen. This is migrating to top-level
- c10 folder.
- - `native <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native>`__
- - Modern implementations of operators. If you want to write
- a new operator, here is where it should go. Most CPU
- operators go in the top level directory, except for
- operators which need to be compiled specially; see cpu
- below.
-
- - `cpu <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu>`__
- - Not actually CPU implementations of operators, but
- specifically implementations which are compiled with
- processor-specific instructions, like AVX. See the
- `README <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/README.md>`__
- for more details.
- - `cuda <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda>`__
- - CUDA implementations of operators.
- - `sparse <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/sparse>`__
- - CPU and CUDA implementations of COO sparse tensor
- operations
- - `mkl <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mkl>`__
- `mkldnn <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mkldnn>`__
- `miopen <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/miopen>`__
- `cudnn <https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn>`__
-
- - implementations of operators which simply bind to some
- backend library.
-
-- `torch <https://github.com/pytorch/pytorch/blob/master/torch>`__ -
- The actual PyTorch library. Everything that is not in
- `csrc <https://github.com/pytorch/pytorch/blob/master/torch/csrc>`__
- is a Python module, following the PyTorch Python frontend module
- structure.
-
- - `csrc <https://github.com/pytorch/pytorch/blob/master/torch/csrc>`__
- - C++ files composing the PyTorch library. Files in this directory
- tree are a mix of Python binding code, and C++ heavy lifting.
- Consult ``setup.py`` for the canonical list of Python binding
- files; conventionally, they are often prefixed with ``python_``.
-
- - `jit <https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit>`__
- - Compiler and frontend for TorchScript JIT frontend.
- - `autograd <https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd>`__
- - Implementation of reverse-mode automatic differentiation.
- - `api <https://github.com/pytorch/pytorch/blob/master/torch/csrc/api>`__
- - The PyTorch C++ frontend.
- - `distributed <https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed>`__
- - Distributed training support for PyTorch.
-
-- `tools <https://github.com/pytorch/pytorch/blob/master/tools>`__ -
- Code generation scripts for the PyTorch library. See
- `README <https://github.com/pytorch/pytorch/blob/master/tools/README.md>`__
- of this directory for more details.
-- `test <https://github.com/pytorch/pytorch/blob/master/tests>`__ -
- Python unit tests for PyTorch Python frontend.
-
- - `test\_torch.py <https://github.com/pytorch/pytorch/blob/master/test/test_torch.py>`__
- - Basic tests for PyTorch functionality.
- - `test\_autograd.py <https://github.com/pytorch/pytorch/blob/master/test/test_autograd.py>`__
- - Tests for non-NN automatic differentiation support.
- - `test\_nn.py <https://github.com/pytorch/pytorch/blob/master/test/test_nn.py>`__
- - Tests for NN operators and their automatic differentiation.
- - `test\_jit.py <https://github.com/pytorch/pytorch/blob/master/test/test_jit.py>`__
- - Tests for the JIT compiler and TorchScript.
- - ...
- - `cpp <https://github.com/pytorch/pytorch/blob/master/test/cpp>`__
- - C++ unit tests for PyTorch C++ frontend.
- - `expect <https://github.com/pytorch/pytorch/blob/master/test/expect>`__
- - Automatically generated "expect" files which are used to compare
- against expected output.
- - `onnx <https://github.com/pytorch/pytorch/blob/master/test/onnx>`__
- - Tests for ONNX export functionality, using both PyTorch and
- Caffe2.
-
-- `caffe2 <https://github.com/pytorch/pytorch/blob/master/caffe2>`__ -
- The Caffe2 library.
-
- - `core <https://github.com/pytorch/pytorch/blob/master/caffe2/core>`__
- - Core files of Caffe2, e.g., tensor, workspace, blobs, etc.
- - `operators <https://github.com/pytorch/pytorch/blob/master/caffe2/operators>`__
- - Operators of Caffe2.
- - `python <https://github.com/pytorch/pytorch/blob/master/caffe2/python>`__
- - Python bindings to Caffe2.
- - ...
-
-Unit Testing
-------------
-
-PyTorch's testing is located under ``test/``. Run the entire test suite
-with
-
-::
-
- python test/run_test.py
-
-or run individual test files, like ``python test/test_nn.py``, for
-individual test suites.
-
-Better local unit tests with pytest
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-We don't officially support ``pytest``, but it works well with our
-``unittest`` tests and offers a number of useful features for local
-developing. Install it via ``pip install pytest``.
-
-If you want to just run tests that contain a specific substring, you can
-use the ``-k`` flag:
-
-::
-
- pytest test/test_nn.py -k Loss -v
-
-The above is an example of testing a change to Loss functions: this
-command runs tests such as ``TestNN.test_BCELoss``\ and
-``TestNN.test_MSELoss`` and can be useful to save keystrokes.
-
-Writing documentation
----------------------
-
-PyTorch uses `Google
-style <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`__
-for formatting docstrings. Length of line inside docstrings block must
-be limited to 80 characters to fit into Jupyter documentation popups.
-
-For C++ documentation (https://pytorch.org/cppdocs), we use
-`Doxygen <http://www.doxygen.nl/>`__ and then convert it to
-`Sphinx <http://www.sphinx-doc.org/>`__ via
-`Breathe <https://github.com/michaeljones/breathe>`__
-and\ `Exhale <https://github.com/svenevs/exhale>`__. Check the `Doxygen
-reference <http://www.stack.nl/~dimitri/doxygen/manual/index.html>`__
-for more information on the documentation syntax. To build the
-documentation locally, ``cd`` into ``docs/cpp`` and then ``make html``.
-
-We run Doxygen in CI (Travis) to verify that you do not use invalid
-Doxygen commands. To run this check locally, run ``./check-doxygen.sh``
-from inside ``docs/cpp``.
-
-Managing multiple build trees
------------------------------
-
-One downside to using ``python setup.py develop`` is that your
-development version of PyTorch will be installed globally on your
-account (e.g., if you run ``import torch`` anywhere else, the
-development version will be used.
-
-If you want to manage multiple builds of PyTorch, you can make use of
-`conda environments <https://conda.io/docs/using/envs.html>`__ to
-maintain separate Python package environments, each of which can be tied
-to a specific build of PyTorch. To set one up:
-
-::
-
- conda create -n pytorch-myfeaturesource activate pytorch-myfeature# if you run python now, torch will NOT be installed
- python setup.py build develop
-
-C++ Development tips
---------------------
-
-If you are working on the C++ code, there are a few important things
-that you will want to keep in mind:
-
-1. How to rebuild only the code you are working on.
-2. How to make rebuilds in the absence of changes go faster.
-
-Build only what you need.
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-``python setup.py build`` will build everything, but since our build
-system is not very optimized for incremental rebuilds, this will
-actually be very slow. Far better is to only request rebuilds of the
-parts of the project you are working on:
-
-- Working on the Python bindings? Run ``python setup.py develop`` to
- rebuild (NB: no ``build`` here!)
-- Working on ``torch/csrc`` or ``aten``? Run
- ``python setup.py rebuild_libtorch`` to rebuild and avoid having to
- rebuild other dependent libraries we depend on.
-- Working on one of the other dependent libraries? The other valid
- targets are listed in ``dep_libs`` in ``setup.py``. prepend
- ``build_`` to get a target, and run as e.g.
- ``python setup.py build_gloo``.
-- Working on a test binary? Run
- ``(cd build && ninja bin/test_binary_name)`` to rebuild only that
- test binary (without rerunning cmake). (Replace ``ninja`` with
- ``make`` if you don't have ninja installed).
-
-On the initial build, you can also speed things up with the environment
-variables ``DEBUG`` and ``NO_CUDA``.
-
-- ``DEBUG=1`` will enable debug builds (-g -O0)
-- ``REL_WITH_DEB_INFO=1`` will enable debug symbols with optimizations
- (-g -O3)
-- ``NO_CUDA=1`` will disable compiling CUDA (in case you are developing
- on something not CUDA related), to save compile time.
-
-For example:
-
-::
-
- NO_CUDA=1 DEBUG=1 python setup.py build develop
-
-Make sure you continue to pass these flags on subsequent builds.
-
-Code completion and IDE support
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When using ``python setup.py develop``, PyTorch will generate a
-``compile_commands.json`` file that can be used by many editors to
-provide command completion and error highlighting for PyTorch's C++
-code. You need to ``pip install ninja`` to generate accurate information
-for the code in ``torch/csrc``. More information at:
-
-- https://sarcasm.github.io/notes/dev/compilation-database.html
-
-Make no-op build fast.
-~~~~~~~~~~~~~~~~~~~~~~
-
-Use Ninja
-~~~~~~~~~
-
-Python ``setuptools`` is pretty dumb, and always rebuilds every C file
-in a project. If you install the ninja build system with
-``pip install ninja``, then PyTorch will use it to track dependencies
-correctly. If PyTorch was already built, you will need to run
-``python setup.py clean`` once after installing ninja for builds to
-succeed.
-
-Use CCache
-~~~~~~~~~~
-
-Even when dependencies are tracked with file modification, there are
-many situations where files get rebuilt when a previous compilation was
-exactly the same.
-
-Using ccache in a situation like this is a real time-saver. However, by
-default, ccache does not properly support CUDA stuff, so here are the
-instructions for installing a custom ccache fork that has CUDA support:
-
-::
-
- # install and export ccacheif ! ls ~/ccache/bin/ccachethen
- sudo apt-get update
- sudo apt-get install -y automake autoconf
- sudo apt-get install -y asciidoc
- mkdir -p ~/ccache
- pushd /tmp
- rm -rf ccache
- git clone https://github.com/colesbury/ccache -b ccbin
- pushd ccache
- ./autogen.sh
- ./configure
- make install prefix=~/ccache
- popdpopd
-
- mkdir -p ~/ccache/lib
- mkdir -p ~/ccache/cuda
- ln -s ~/ccache/bin/ccache ~/ccache/lib/cc
- ln -s ~/ccache/bin/ccache ~/ccache/lib/c++
- ln -s ~/ccache/bin/ccache ~/ccache/lib/gcc
- ln -s ~/ccache/bin/ccache ~/ccache/lib/g++
- ln -s ~/ccache/bin/ccache ~/ccache/cuda/nvcc
-
- ~/ccache/bin/ccache -M 25Gifiexport PATH=~/ccache/lib:$PATHexport CUDA_NVCC_EXECUTABLE=~/ccache/cuda/nvcc
-
-CUDA Development tips
----------------------
-
-If you are working on the CUDA code, here are some useful CUDA debugging
-tips:
-
-1. ``CUDA_DEVICE_DEBUG=1`` will enable CUDA device function debug
- symbols (``-g -G``). This will be particularly helpful in debugging
- device code. However, it will slow down the build process for about
- 50% (compared to only ``DEBUG=1``), so use wisely.
-2. ``cuda-gdb`` and ``cuda-memcheck`` are your best CUDA debugging
- friends. Unlike\ ``gdb``, ``cuda-gdb`` can display actual values in a
- CUDA tensor (rather than all zeros).
-
-Hope this helps, and thanks for considering to contribute.
-
-Windows development tips
-------------------------
-
-Occasionally, you will write a patch which works on Linux, but fails CI
-on Windows. There are a few aspects in which MSVC (the Windows compiler
-toolchain we use) is stricter than Linux, which are worth keeping in
-mind when fixing these problems.
-
-1. Symbols are NOT exported by default on Windows; instead, you have to
- explicitly mark a symbol as exported/imported in a header file with
- ``__declspec(dllexport)`` / ``__declspec(dllimport)``. We have
- codified this pattern into a set of macros which follow the
- convention ``*_API``, e.g., ``CAFFE2_API`` inside Caffe2 and ATen.
- (Every separate shared library needs a unique macro name, because
- symbol visibility is on a per shared library basis. See
- c10/macros/Macros.h for more details.) The upshot is if you see an
- "unresolved external" error in your Windows build, this is probably
- because you forgot to mark a function with ``*_API``. However, there
- is one important counterexample to this principle: if you want a
- *templated* function to be instantiated at the call site, do NOT mark
- it with ``*_API`` (if you do mark it, you'll have to explicitly
- instantiate all of the specializations used by the call sites.)
-2. If you link against a library, this does not make its dependencies
- transitively visible. You must explicitly specify a link dependency
- against every library whose symbols you use. (This is different from
- Linux where in most environments, transitive dependencies can be used
- to fulfill unresolved symbols.)
-3. If you have a Windows box (we have a few on EC2 which you can request
- access to) and you want to run the build, the easiest way is to just
- run ``.jenkins/pytorch/win-build.sh``. If you need to rebuild, run
- ``REBUILD=1 .jenkins/pytorch/win-build.sh`` (this will avoid blowing
- away your Conda environment.)
-
-Even if you don't know anything about MSVC, you can use cmake to build
-simple programs on Windows; this can be helpful if you want to learn
-more about some peculiar linking behavior by reproducing it on a small
-example. Here's a simple example cmake file that defines two dynamic
-libraries, one linking with the other:
-
-::
-
- project(myproject CXX)set(CMAKE_CXX_STANDARD 11)add_library(foo SHARED foo.cpp)add_library(bar SHARED bar.cpp)# NB: don't forget to __declspec(dllexport) at least one symbol from foo,# otherwise foo.lib will not be created.target_link_libraries(bar PUBLIC foo)
-
-You can build it with:
-
-::
-
- mkdir buildcd build
- cmake ..
- cmake --build .
-
-Known MSVC (and MSVC with NVCC) bugs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The PyTorch codebase sometimes likes to use exciting C++ features, and
-these exciting features lead to exciting bugs in Windows compilers. To
-add insult to injury, the error messages will often not tell you which
-line of code actually induced the erroring template instantiation. We've
-found the most effective way to debug these problems is to carefully
-read over diffs, keeping in mind known bugs in MSVC/NVCC. Here are a few
-well known pitfalls and workarounds:
-
-- This is not actually a bug per se, but in general, code generated by
- MSVC is more sensitive to memory errors; you may have written some
- code that does a use-after-free or stack overflows; on Linux the code
- might work, but on Windows your program will crash. ASAN may not
- catch all of these problems: stay vigilant to the possibility that
- your crash is due to a real memory problem.
-- (NVCC) ``c10::optional`` does not work when used from device code.
- Don't use it from kernels. Upstream issue:
- https://github.com/akrzemi1/Optional/issues/58 and our local issue
- #10329.
-- ``constexpr`` generally works less well on MSVC.
-
- - The idiom ``static_assert(f() == f())`` to test if ``f`` is
- constexpr does not work; you'll get "error C2131: expression did
- not evaluate to a constant". Don't use these asserts on Windows.
- (Example: ``c10/util/intrusive_ptr.h``)
-
-- (NVCC) Code you access inside a ``static_assert`` will eagerly be
- evaluated as if it were device code, and so you might get an error
- that the code is "not accessible".
-
-::
-
- class A {
- static A singleton_;
- static constexpr inline A* singleton() {
- return &singleton_;
- }
- };static_assert(std::is_same(A*, decltype(A::singleton()))::value, "hmm");
-
-- The compiler will run out of heap space if you attempt to compile
- files that are too large. Splitting such files into separate files
- helps. (Example: ``THTensorMath``, ``THTensorMoreMath``,
- ``THTensorEvenMoreMath``.)
-- MSVC's preprocessor (but not the standard compiler) has a bug where
- it incorrectly tokenizes raw string literals, ending when it sees a
- ``"``. This causes preprocessor tokens inside the literal like
- an\ ``#endif`` to be incorrectly treated as preprocessor directives.
- See https://godbolt.org/z/eVTIJq as an example.
-
-Running Clang-Tidy
-~~~~~~~~~~~~~~~~~~
-
-`Clang-Tidy <https://clang.llvm.org/extra/clang-tidy/index.html>`__ is a
-C++ linter and static analysis tool based on the clang compiler. We run
-clang-tidy in our CI to make sure that new C++ code is safe, sane and
-efficient. See our
-`.travis.yml <https://github.com/pytorch/pytorch/blob/master/.travis.yml>`__
-file for the simple commands we use for this. To run clang-tidy locally,
-follow these steps:
-
-1. Install clang-tidy. First, check if you already have clang-tidy by
- simply writing ``clang-tidy`` in your terminal. If you don't yet have
- clang-tidy, you should be able to install it easily with your package
- manager, e.g. by writing ``apt-get install clang-tidy`` on Ubuntu.
- See `https://apt.llvm.org <https://apt.llvm.org/>`__ for details on
- how to install the latest version. Note that newer versions of
- clang-tidy will have more checks than older versions. In our CI, we
- run clang-tidy-6.0.
-2. Use our driver script to run clang-tidy over any changes relative to
- some git revision (you may want to replace ``HEAD~1`` with ``HEAD``
- to pick up uncommitted changes). Changes are picked up based on a
- ``git diff`` with the given revision:
-
-::
-
- python tools/clang_tidy.py -d build -p torch/csrc --diff 'HEAD~1'
-
-Above, it is assumed you are in the PyTorch root folder.
-``path/to/build`` should be the path to where you built PyTorch from
-source, e.g. ``build`` in the PyTorch root folder if you used
-``setup.py build``. You can use ``-c <clang-tidy-binary>``\ to change
-the clang-tidy this script uses. Make sure you have PyYaml installed,
-which is in PyTorch's ``requirements.txt``.
-
-Pre-commit Tidy/Linting Hook
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-We use clang-tidy and flake8 to perform additional formatting and
-semantic checking of code. We provide a pre-commit git hook for
-performing these checks, before a commit is created:
-
-::
-
- ln -s ../../tools/git-pre-commit .git/hooks/pre-commit
-
-Caffe2 notes
-------------
-
-In 2018, we merged Caffe2 into the PyTorch source repository. While the
-steady state aspiration is that Caffe2 and PyTorch share code freely, in
-the meantime there will be some separation. If you submit a PR to only
-PyTorch or only Caffe2 code, CI will only run for the project you
-edited. The logic for this is implemented in
-``.jenkins/pytorch/dirty.sh`` and ``.jenkins/caffe2/dirty.sh``; you can
-look at this to see what path prefixes constitute changes. This also
-means if you ADD a new top-level path, or you start sharing code between
-projects, you need to modify these files. There are a few "unusual"
-directories which, for historical reasons, are Caffe2/PyTorch specific.
-Here they are:
-
-- ``CMakeLists.txt``, ``Makefile``, ``binaries``, ``cmake``, ``conda``,
- ``modules``, ``scripts`` are Caffe2-specific. Don't put PyTorch code
- in them without extra coordination.
-- ``mypy*``, ``requirements.txt``, ``setup.py``, ``test``, ``tools``
- are PyTorch-specific. Don't put Caffe2 code in them without extra
- coordination.
projects / platform / upstream / pytorch.git / commitdiff