1 # How to Cross-Compile ArmNN on x86_64 for arm64
3 * [Introduction](#introduction)
4 * [Build and install Google's Protobuf library](#buildProtobuf)
5 * [Build Caffe for x86_64](#buildCaffe)
6 * [Cross-compiling ToolChain](#installCCT)
7 * [Build Boost library for arm64](#installBaarch)
8 * [Build Compute Library](#buildCL)
9 * [Build Compute Library](#buildCL)
10 * [Build ArmNN](#buildANN)
11 * [Run Unit Tests](#unittests)
12 * [Troubleshooting and Errors](#troubleshooting)
15 #### <a name="introduction">Introduction</a>
16 These are the step by step instructions on Cross-Compiling ArmNN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
17 The instructions show how to build the ArmNN core library and the Boost, Protobuf, Caffe and Compute Libraries necessary for compilation.
19 #### <a name="buildProtobuf">Build and install Google's Protobuf library</a>
21 * Get protobuf-all-3.5.1.tar.gz from here: https://github.com/protocolbuffers/protobuf/releases/tag/v3.5.1
24 tar -zxvf protobuf-all-3.5.1.tar.gz
27 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
28 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
32 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
36 * Build the arm64 version of the protobuf libraries:
40 CC=aarch64-linux-gnu-gcc \
41 CXX=aarch64-linux-gnu-g++ \
42 ../configure --host=aarch64-linux \
43 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
44 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
49 #### <a name="buildCaffe">Build Caffe for x86_64</a>
50 * Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html
51 * Install dependencies:
53 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev
54 sudo apt-get install --no-install-recommends libboost-all-dev
55 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
56 sudo apt-get install libopenblas-dev
57 sudo apt-get install libatlas-base-dev
59 * Download Caffe-Master from: https://github.com/BVLC/caffe
61 git clone https://github.com/BVLC/caffe.git
63 cp Makefile.config.example Makefile.config
65 * Adjust Makefile.config as necessary for your environment, for example:
70 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir):
71 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/include/
72 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/lib/
74 #g++ needs to be version 5:
79 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH
80 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH
82 * Compilation with Make:
88 These should all run without errors
89 * caffe.pb.h and caffe.pb.cc will be needed when building ArmNN's Caffe Parser
91 #### <a name="installCCT">Cross-compiling ToolChain</a>
92 * Install the standard cross-compilation libraries for arm64:
94 sudo apt install crossbuild-essential-arm64
96 #### <a name="installBaarch">Build Boost library for arm64</a>
97 * Build Boost library for arm64
98 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
99 Version 1.66 is not supported.
101 tar -zxvf boost_1_64_0.tar.gz
103 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
104 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
105 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-filesystem --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
108 #### <a name="buildCL">Build Compute Library</a>
109 * Building the Arm Compute Library:
111 git clone https://github.com/ARM-software/ComputeLibrary.git
113 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0
116 #### <a name="buildANN">Build ArmNN</a>
117 * Compile ArmNN for arm64:
119 git clone https://github.com/ARM-software/armnn.git
125 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the armNN build:
128 CXX=aarch64-linux-gnu-g++ \
129 CC=aarch64-linux-gnu-gcc \
131 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
132 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
133 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
134 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 \
135 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \
136 -DBUILD_CAFFE_PARSER=1 \
137 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
138 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.15.0.1 \
139 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.15.0.1
146 #### <a name="unittests">Run Unit Tests</a>
147 * Copy the build folder to an arm64 linux machine
148 * Copy the libprotobuf.so.15.0.1 library file to the build folder
149 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
152 export LD_LIBRARY_PATH=`pwd`
154 * Create a symbolic link to libprotobuf.so.15.0.1:
156 ln -s libprotobuf.so.15.0.1 ./libprotobuf.so.15
161 Running 567 test cases...
163 *** No errors detected
165 #### <a name="troubleshooting">Troubleshooting and Errors:</a>
166 #### Error adding symbols: File in wrong format
167 * When building armNN:
169 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
170 collect2: error: ld returned 1 exit status
171 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
172 make[2]: *** [libarmnn.so] Error 1
173 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
174 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
175 Makefile:127: recipe for target 'all' failed
176 make: *** [all] Error 2
178 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
180 #### Virtual memory exhausted
181 * When compiling the boost libraries:
183 virtual memory exhausted: Cannot allocate memory
185 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
188 #### Unrecognized command line option '-m64'
189 * When compiling the boost libraries:
191 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
193 * Clean the boost library directory before trying to build with a different architecture:
197 * It should show the following for arm64:
205 #### Missing libz.so.1
206 * When compiling armNN:
208 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.15.0.0, not found (try using -rpath or -rpath-link)
211 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
213 sudo dpkg --add-architecture arm64
214 sudo apt-get install zlib1g:arm64
218 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
219 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
220 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
222 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
225 #### Unable to install arm64 packages after adding arm64 architecture
226 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
227 * From stackoverflow:
228 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
229 * Open /etc/apt/sources.list with your preferred text editor.
231 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
233 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
235 * Then add the following:
237 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
238 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
239 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
240 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
241 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
242 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
243 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
245 * Update and install again:
247 sudo apt-get install zlib1g:arm64
252 #### Undefined references to google::protobuf:: functions
253 * When compiling armNN there are multiple errors of the following type:
255 libarmnnCaffeParser.so: undefined reference to `google::protobuf:*
257 * Missing or out of date protobuf compilation libraries.
258 Use the command 'protoc --version' to check which version of protobuf is available (version 3.5.1 is required).
259 Follow the instructions above to install protobuf 3.5.1
260 Note this will require you to recompile Caffe for x86_64
263 #### Errors on strict-aliasing rules when compiling the Compute Library
264 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
266 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
268 * Add Werror=0 to the scons command:
270 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0