1 # How to Cross-Compile ArmNN on x86_64 for arm64
3 * [Introduction](#introduction)
4 * [Cross-compiling ToolChain](#installCCT)
5 * [Build and install Google's Protobuf library](#buildProtobuf)
6 * [Build Caffe for x86_64](#buildCaffe)
7 * [Build Boost library for arm64](#installBaarch)
8 * [Build Compute Library](#buildCL)
9 * [Build ArmNN](#buildANN)
10 * [Run Unit Tests](#unittests)
11 * [Troubleshooting and Errors](#troubleshooting)
14 #### <a name="introduction">Introduction</a>
15 These are the step by step instructions on Cross-Compiling ArmNN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
16 The instructions show how to build the ArmNN core library and the Boost, Protobuf, Caffe, Tensorflow, Tflite, Flatbuffer and Compute Libraries necessary for compilation.
18 #### <a name="installCCT">Cross-compiling ToolChain</a>
19 * Install the standard cross-compilation libraries for arm64:
21 sudo apt install crossbuild-essential-arm64
24 #### <a name="buildProtobuf">Build and install Google's Protobuf library</a>
26 ## We support protobuf version 3.5.2
27 * Get protobuf from here: https://github.com/protocolbuffers/protobuf :
29 git clone -b v3.5.2 https://github.com/google/protobuf.git protobuf
31 git submodule update --init --recursive
34 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
35 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
39 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
43 * Build the arm64 version of the protobuf libraries:
47 export CC=aarch64-linux-gnu-gcc \
48 export CXX=aarch64-linux-gnu-g++ \
49 ../configure --host=aarch64-linux \
50 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
51 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
56 #### <a name="buildCaffe">Build Caffe for x86_64</a>
57 * Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html
58 * Install dependencies:
60 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev
61 sudo apt-get install --no-install-recommends libboost-all-dev
62 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
63 sudo apt-get install libopenblas-dev
64 sudo apt-get install libatlas-base-dev
66 * Download Caffe-Master from: https://github.com/BVLC/caffe
68 git clone https://github.com/BVLC/caffe.git
70 cp Makefile.config.example Makefile.config
72 * Adjust Makefile.config as necessary for your environment, for example:
77 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir):
78 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/include/
79 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/lib/
83 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH
84 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH
86 * Compilation with Make:
92 These should all run without errors
93 * caffe.pb.h and caffe.pb.cc will be needed when building ArmNN's Caffe Parser
95 #### <a name="installBaarch">Build Boost library for arm64</a>
96 * Build Boost library for arm64
97 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
98 Using any version of Boost greater than 1.64 will fail to build ArmNN, due to different dependency issues.
100 tar -zxvf boost_1_64_0.tar.gz
102 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
103 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
104 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
107 #### <a name="buildCL">Build Compute Library</a>
108 * Building the Arm Compute Library:
110 git clone https://github.com/ARM-software/ComputeLibrary.git
112 git checkout <branch_name>
114 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0
117 For example, if you want to checkout release branch of 20.02:
119 git checkout branches/arm_compute_20_02
123 #### <a name="buildtf">Build Tensorflow</a>
124 * Building Tensorflow version 1.15:
126 git clone https://github.com/tensorflow/tensorflow.git
128 git checkout 590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b
129 ../armnn/scripts/generate_tensorflow_protobuf.sh ../tensorflow-protobuf ../google/x86_64_pb_install
132 #### <a name="buildflatbuffer">Build Flatbuffer</a>
133 * Building Flatbuffer version 1.12.0
135 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
136 tar xf flatbuffers-1.12.0.tar.gz
137 cd flatbuffers-1.12.0
141 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
142 -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \
143 -DFLATBUFFERS_BUILD_TESTS=0
147 * Build arm64 version of flatbuffer
151 # Add -fPIC to allow us to use the libraries in shared objects.
152 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
153 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
154 -DFLATBUFFERS_BUILD_FLATC=1 \
155 -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \
156 -DFLATBUFFERS_BUILD_TESTS=0
160 #### <a name="buildingONNX">Build Onnx</a>
163 git clone https://github.com/onnx/onnx.git
165 git fetch https://github.com/onnx/onnx.git f612532843bd8e24efeab2815e45b436479cc9ab && git checkout FETCH_HEAD
166 export LD_LIBRARY_PATH=$<DIRECTORY_PATH>/protobuf-host/lib:$LD_LIBRARY_PATH
167 ../google/x86_64_pb_install/bin/protoc onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out ../onnx
170 #### <a name="buildingtflite">Build TfLite</a>
175 cp ../tensorflow/tensorflow/lite/schema/schema.fbs .
176 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
179 #### <a name="buildANN">Build ArmNN</a>
180 * Compile ArmNN for arm64:
182 git clone https://github.com/ARM-software/armnn.git
184 git checkout <branch_name>
190 For example, if you want to checkout release branch of 20.02:
192 git checkout branches/armnn_20_02
196 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the armNN build:
199 export CXX=aarch64-linux-gnu-g++ \
200 export CC=aarch64-linux-gnu-gcc \
202 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
203 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
204 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
205 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
206 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \
207 -DBUILD_CAFFE_PARSER=1 \
208 -DONNX_GENERATED_SOURCES=$HOME/onnx \
209 -DBUILD_ONNX_PARSER=1 \
210 -DTF_GENERATED_SOURCES=$HOME/tensorflow-protobuf \
211 -DBUILD_TF_PARSER=1 \
212 -DBUILD_TF_LITE_PARSER=1 \
213 -DTF_LITE_GENERATED_PATH=$HOME/tflite \
214 -DFLATBUFFERS_ROOT=$HOME/flatbuffers-arm64 \
215 -DFLATC_DIR=$HOME/flatbuffers-1.12.0/build \
216 -DPROTOBUF_ROOT=$HOME/google/x86_64_pb_install \
217 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
218 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.15.0.1 \
219 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.15.0.1
222 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
224 -DSAMPLE_DYNAMIC_BACKEND=1 \
225 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
232 #### <a name="buildStandaloneBackend">Build Standalone Sample Dynamic Backend</a>
233 * The sample dynamic backend is located in armnn/src/dynamic/sample
239 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the armNN build:
242 export CXX=aarch64-linux-gnu-g++ \
243 export CC=aarch64-linux-gnu-gcc \
245 -DCMAKE_CXX_FLAGS=--std=c++14 \
246 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
247 -DBoost_SYSTEM_LIBRARY=$HOME/armnn-devenv/boost_arm64_install/lib/libboost_system.a \
248 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
256 #### <a name="unittests">Run Unit Tests</a>
257 * Copy the build folder to an arm64 linux machine
258 * Copy the libprotobuf.so.15.0.1 library file to the build folder
259 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build ArmNN
260 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
263 export LD_LIBRARY_PATH=`pwd`
265 * Create a symbolic link to libprotobuf.so.15.0.1:
267 ln -s libprotobuf.so.15.0.1 ./libprotobuf.so.15
272 Running 567 test cases...
274 *** No errors detected
276 #### <a name="troubleshooting">Troubleshooting and Errors:</a>
277 #### Error adding symbols: File in wrong format
278 * When building armNN:
280 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
281 collect2: error: ld returned 1 exit status
282 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
283 make[2]: *** [libarmnn.so] Error 1
284 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
285 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
286 Makefile:127: recipe for target 'all' failed
287 make: *** [all] Error 2
289 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
291 #### Virtual memory exhausted
292 * When compiling the boost libraries:
294 virtual memory exhausted: Cannot allocate memory
296 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
299 #### Unrecognized command line option '-m64'
300 * When compiling the boost libraries:
302 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
304 * Clean the boost library directory before trying to build with a different architecture:
308 * It should show the following for arm64:
316 #### Missing libz.so.1
317 * When compiling armNN:
319 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.15.0.0, not found (try using -rpath or -rpath-link)
322 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
324 sudo dpkg --add-architecture arm64
325 sudo apt-get install zlib1g:arm64
329 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
330 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
331 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
333 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
336 #### Unable to install arm64 packages after adding arm64 architecture
337 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
338 * From stackoverflow:
339 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
340 * Open /etc/apt/sources.list with your preferred text editor.
342 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
344 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
346 * Then add the following:
348 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
349 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
350 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
351 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
352 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
353 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
354 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
356 * Update and install again:
358 sudo apt-get install zlib1g:arm64
363 #### Undefined references to google::protobuf:: functions
364 * When compiling armNN there are multiple errors of the following type:
366 libarmnnCaffeParser.so: undefined reference to `google::protobuf:*
368 * Missing or out of date protobuf compilation libraries.
369 Use the command 'protoc --version' to check which version of protobuf is available (version 3.5.2 is required).
370 Follow the instructions above to install protobuf 3.5.2
371 Note this will require you to recompile Caffe for x86_64
374 #### Errors on strict-aliasing rules when compiling the Compute Library
375 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
377 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
379 * Add Werror=0 to the scons command:
381 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0