BuildGuideCrossCompilation.md

   1 # How to Cross-Compile ArmNN on x86_64 for arm64
   2
   3 - [Introduction](#introduction)
   4 - [Cross-compiling ToolChain](#cross-compiling-toolchain)
   5 - [Build and install Google's Protobuf library](#build-and-install-google-s-protobuf-library)
   6 - [Build Caffe for x86_64](#build-caffe-for-x86-64)
   7 - [Build Boost library for arm64](#build-boost-library-for-arm64)
   8 - [Build Compute Library](#build-compute-library)
   9 - [Build Tensorflow](#build-tensorflow)
  10 - [Build Flatbuffer](#build-flatbuffer)
  11 - [Build Onnx](#build-onnx)
  12 - [Build TfLite](#build-tflite)
  13 - [Build ArmNN](#build-armnn)
  14 - [Build Standalone Sample Dynamic Backend](#build-standalone-sample-dynamic-backend)
  15 - [Run Unit Tests](#run-unit-tests)
  16 - [Troubleshooting and Errors:](#troubleshooting-and-errors-)
  17
  18
  19 ## Introduction
  20 These are the step by step instructions on Cross-Compiling ArmNN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
  21 The instructions show how to build the ArmNN core library and the Boost, Protobuf, Caffe, Tensorflow, Tflite, Flatbuffer and Compute Libraries necessary for compilation.
  22
  23 ## Cross-compiling ToolChain
  24 * Install the standard cross-compilation libraries for arm64:
  25 ```
  26 sudo apt install crossbuild-essential-arm64
  27 ```
  28
  29 ## Build and install Google's Protobuf library
  30
  31 We support protobuf version 3.12.0
  32 * Get protobuf from here: https://github.com/protocolbuffers/protobuf :
  33 ```bash
  34 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf
  35 cd protobuf
  36 git submodule update --init --recursive
  37 ./autogen
  38 ```
  39 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
  40   (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
  41 ```
  42 mkdir x86_64_build
  43 cd x86_64_build
  44 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
  45 make install -j16
  46 cd ..
  47 ```
  48 * Build the arm64 version of the protobuf libraries:
  49 ```
  50 mkdir arm64_build
  51 cd arm64_build
  52 export CC=aarch64-linux-gnu-gcc \
  53 export CXX=aarch64-linux-gnu-g++ \
  54 ../configure --host=aarch64-linux \
  55 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
  56 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
  57 make install -j16
  58 cd ..
  59 ```
  60
  61 ## Build Caffe for x86_64
  62 * Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html
  63 * Install dependencies:
  64 ```bash
  65 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev
  66 sudo apt-get install --no-install-recommends libboost-all-dev
  67 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
  68 sudo apt-get install libopenblas-dev
  69 sudo apt-get install libatlas-base-dev
  70 ```
  71 * Download Caffe from: https://github.com/BVLC/caffe. We have tested using tag 1.0
  72 ```bash
  73 git clone https://github.com/BVLC/caffe.git
  74 cd caffe
  75 git checkout eeebdab16155d34ff8f5f42137da7df4d1c7eab0
  76 cp Makefile.config.example Makefile.config
  77 ```
  78 * Adjust Makefile.config as necessary for your environment, for example:
  79 ```
  80 #CPU only version:
  81 CPU_ONLY := 1
  82
  83 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir):
  84 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/include/
  85 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/lib/
  86 ```
  87 * Setup environment:
  88 ```bash
  89 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH
  90 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH
  91 ```
  92 * Compilation with Make:
  93 ```bash
  94 make all
  95 make test
  96 make runtest
  97 # These should all run without errors
  98 ```
  99
 100 * caffe.pb.h and caffe.pb.cc will be needed when building ArmNN's Caffe Parser
 101
 102 ## Build Boost library for arm64
 103 * Build Boost library for arm64
 104     Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
 105     Using any version of Boost greater than 1.64 will fail to build ArmNN, due to different dependency issues.
 106 ```bash
 107 tar -zxvf boost_1_64_0.tar.gz
 108 cd boost_1_64_0
 109 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
 110 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
 111 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
 112 ```
 113
 114 ## Build Compute Library
 115 * Building the Arm Compute Library:
 116 ```bash
 117 git clone https://github.com/ARM-software/ComputeLibrary.git
 118 cd ComputeLibrary/
 119 git checkout <branch_name>
 120 git pull
 121 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0
 122 ```
 123
 124 For example, if you want to checkout release branch of 20.02:
 125 ```bash
 126 git checkout branches/arm_compute_20_02
 127 git pull
 128 ```
 129
 130 ## Build Tensorflow
 131 * Building Tensorflow version 2.3.1:
 132 ```bash
 133 git clone https://github.com/tensorflow/tensorflow.git
 134 cd tensorflow/
 135 git checkout fcc4b966f1265f466e82617020af93670141b009
 136 ../armnn/scripts/generate_tensorflow_protobuf.sh ../tensorflow-protobuf ../google/x86_64_pb_install
 137 ```
 138
 139 ## Build Flatbuffer
 140 * Building Flatbuffer version 1.12.0
 141 ```bash
 142 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
 143 tar xf flatbuffers-1.12.0.tar.gz
 144 cd flatbuffers-1.12.0
 145 rm -f CMakeCache.txt
 146 mkdir build
 147 cd build
 148 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
 149      -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \
 150      -DFLATBUFFERS_BUILD_TESTS=0
 151 make all install
 152 ```
 153
 154 * Build arm64 version of flatbuffer
 155 ```bash
 156 mkdir build-arm64
 157 cd build-arm64
 158 # Add -fPIC to allow us to use the libraries in shared objects.
 159 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
 160      -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
 161      -DFLATBUFFERS_BUILD_FLATC=1 \
 162      -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \
 163      -DFLATBUFFERS_BUILD_TESTS=0
 164 make all install
 165 ```
 166
 167 ## Build Onnx
 168 * Building Onnx
 169 ```bash
 170 git clone https://github.com/onnx/onnx.git
 171 cd onnx
 172 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD
 173 export LD_LIBRARY_PATH=$<DIRECTORY_PATH>/protobuf-host/lib:$LD_LIBRARY_PATH
 174 ../google/x86_64_pb_install/bin/protoc onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out ../onnx
 175 ```
 176
 177 ## Build TfLite
 178 * Building TfLite
 179 ```bash
 180 mkdir tflite
 181 cd tflite
 182 cp ../tensorflow/tensorflow/lite/schema/schema.fbs .
 183 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
 184 ```
 185
 186 ## Build ArmNN
 187 * Compile ArmNN for arm64:
 188 ```bash
 189 git clone https://github.com/ARM-software/armnn.git
 190 cd armnn
 191 git checkout <branch_name>
 192 git pull
 193 mkdir build
 194 cd build
 195 ```
 196
 197 For example, if you want to checkout release branch of 20.02:
 198 ```bash
 199 git checkout branches/armnn_20_02
 200 git pull
 201 ```
 202
 203 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the armNN build:
 204 ```bash
 205 #!/bin/bash
 206 export CXX=aarch64-linux-gnu-g++ \
 207 export CC=aarch64-linux-gnu-gcc \
 208 cmake .. \
 209 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
 210 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
 211 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
 212 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
 213 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \
 214 -DBUILD_CAFFE_PARSER=1 \
 215 -DONNX_GENERATED_SOURCES=$HOME/onnx \
 216 -DBUILD_ONNX_PARSER=1 \
 217 -DTF_GENERATED_SOURCES=$HOME/tensorflow-protobuf \
 218 -DBUILD_TF_PARSER=1 \
 219 -DBUILD_TF_LITE_PARSER=1 \
 220 -DTF_LITE_GENERATED_PATH=$HOME/tflite \
 221 -DFLATBUFFERS_ROOT=$HOME/flatbuffers-arm64 \
 222 -DFLATC_DIR=$HOME/flatbuffers-1.12.0/build \
 223 -DPROTOBUF_ROOT=$HOME/google/x86_64_pb_install \
 224 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
 225 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 \
 226 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0
 227 ```
 228
 229 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
 230 ```bash
 231 -DSAMPLE_DYNAMIC_BACKEND=1 \
 232 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
 233 ```
 234 * Run the build
 235 ```bash
 236 make -j32
 237 ```
 238
 239 ## Build Standalone Sample Dynamic Backend
 240 * The sample dynamic backend is located in armnn/src/dynamic/sample
 241 ```bash
 242 mkdir build
 243 cd build
 244 ```
 245
 246 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the armNN build:
 247 ```bash
 248 #!/bin/bash
 249 export CXX=aarch64-linux-gnu-g++ \
 250 export CC=aarch64-linux-gnu-gcc \
 251 cmake .. \
 252 -DCMAKE_CXX_FLAGS=--std=c++14 \
 253 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
 254 -DBoost_SYSTEM_LIBRARY=$HOME/armnn-devenv/boost_arm64_install/lib/libboost_system.a \
 255 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
 256 ```
 257
 258 * Run the build
 259 ```bash
 260 make
 261 ```
 262
 263 ## Run Unit Tests
 264 * Copy the build folder to an arm64 linux machine
 265 * Copy the libprotobuf.so.15.0.1 library file to the build folder
 266 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build ArmNN
 267 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
 268
 269 ```bash
 270 cd build/
 271 export LD_LIBRARY_PATH=<current_working_directory>
 272 ```
 273
 274 * Create a symbolic link to libprotobuf.so.23.0.0:
 275
 276 ```bash
 277 ln -s libprotobuf.so.23.0.0 ./libprotobuf.so.23
 278 ```
 279
 280 * Run the UnitTests:
 281
 282 ```bash
 283 ./UnitTests
 284 Running 567 test cases...
 285
 286 *** No errors detected
 287 ```
 288
 289 ## Troubleshooting and Errors:
 290 ### Error adding symbols: File in wrong format
 291 * When building armNN:
 292 ```bash
 293 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
 294 collect2: error: ld returned 1 exit status
 295 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
 296 make[2]: *** [libarmnn.so] Error 1
 297 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
 298 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
 299 Makefile:127: recipe for target 'all' failed
 300 make: *** [all] Error 2
 301 ```
 302 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
 303 <br><br>
 304
 305 ### Virtual memory exhausted
 306 * When compiling the boost libraries:
 307 ```bash
 308 virtual memory exhausted: Cannot allocate memory
 309 ```
 310 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
 311 <br><br>
 312
 313 ### Unrecognized command line option '-m64'
 314 * When compiling the boost libraries:
 315 ```bash
 316 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
 317 ```
 318 * Clean the boost library directory before trying to build with a different architecture:
 319 ```bash
 320 sudo ./b2 clean
 321 ```
 322 * It should show the following for arm64:
 323 ```bash
 324 - 32-bit                   : no
 325 - 64-bit                   : yes
 326 - arm                      : yes
 327 ```
 328 <br><br>
 329
 330 ### Missing libz.so.1
 331 * When compiling armNN:
 332 ```bash
 333 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.23.0.0, not found (try using -rpath or -rpath-link)
 334 ```
 335
 336 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
 337 ```bash
 338 sudo dpkg --add-architecture arm64
 339 sudo apt-get install zlib1g:arm64
 340 sudo apt-get update
 341 sudo ldconfig
 342 ```
 343 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
 344 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
 345       https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
 346 ```bash
 347 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
 348 ```
 349 <br><br>
 350
 351 ### Unable to install arm64 packages after adding arm64 architecture
 352 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
 353 * From stackoverflow:
 354 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
 355 * Open /etc/apt/sources.list with your preferred text editor.
 356
 357 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
 358 ```bash
 359 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
 360 ```
 361 * Then add the following:
 362 ```bash
 363 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
 364 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
 365 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
 366 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
 367 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
 368 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
 369 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
 370 ```
 371 * Update and install again:
 372 ```bash
 373 sudo apt-get install zlib1g:arm64
 374 sudo apt-get update
 375 sudo ldconfig
 376 ```
 377 <br><br>
 378
 379 ### Undefined references to google::protobuf:: functions
 380 * When compiling armNN there are multiple errors of the following type:
 381 ```
 382 libarmnnCaffeParser.so: undefined reference to `google::protobuf:*
 383 ```
 384 * Missing or out of date protobuf compilation libraries.
 385     Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required).
 386     Follow the instructions above to install protobuf 3.12.0
 387     Note this will require you to recompile Caffe for x86_64
 388 <br><br>
 389
 390 ### Errors on strict-aliasing rules when compiling the Compute Library
 391 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
 392  ```
 393 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
 394  ```
 395 * Add Werror=0 to the scons command:
 396 ```
 397 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0
 398 ```