Follow the steps [here](https://www.tensorflow.org/mobile/tflite/demo_android) to install Tensorflow, Bazel, and the Android NDK and SDK.
-## To test the benchmarker:
+## Test the benchmarker:
The testing utilities helps the developers (you) to make sure that your submissions in TfLite format will be processed as expected in the competition's benchmarking system.
Now you can run the bazel tests to catch any runtime issues with the submission.
Note: Please make sure that your submission passes the test. If a submission fails to pass the test it will not be processed by the submission server.
+
+## Measure on-device latency
+
+We provide two ways to measure the on-device latency of your submission. The first is through our competition server, which is reliable and repeatable, but is limited to a few trials per day. The second is through the benchmarker Apk, which requires a device and may not be as accurate as the server, but has a fast turn-around and no access limitations. We recommend that the participants use the benchmarker apk for early development, and reserve the competition server for evaluating promising submissions.
+
+### Running the benchmarker app
+
+Make sure that you have followed instructions in [Test your submissions](#test-your-submissions) to add your model to the testdata folder and to the corresponding build rules.
+
+Modify `tensorflow/contrib/lite/java/ovic/demo/app/OvicBenchmarkerActivity.java`:
+
+* Add your model to the benchmarker apk by changing `MODEL_PATH` and `TEST_IMAGE_PATH` below to your submission and test image.
+
+```
+ private static final String TEST_IMAGE_PATH = "my_test_image.jpg";
+ private static final String MODEL_PATH = "my_model.lite";
+```
+
+* Adjust the benchmark parameters when needed:
+
+You can chnage the length of each experiment, and the processor affinity below. `BIG_CORE_MASK` is an integer whose binary encoding represents the set of used cores. This number is phone-specific. For example, Pixel 2 has 8 cores: the 4 little cores are represented by the 4 less significant bits, and the 4 big cores by the 4 more significant bits. Therefore a mask value of 16, or in binary `00010000`, represents using only the first big core. The mask 32, or in binary `00100000` uses the second big core and should deliver identical results as the mask 16 because the big cores are interchangeable.
+
+```
+ /** Wall time for each benchmarking experiment. */
+ private static final double WALL_TIME = 3000;
+ /** Maximum number of iterations in each benchmarking experiment. */
+ private static final int MAX_ITERATIONS = 100;
+ /** Mask for binding to a single big core. Pixel 1 (4), Pixel 2 (16). */
+ private static final int BIG_CORE_MASK = 16;
+```
+
+Note: You'll need ROOT access to the phone to change processor affinity.
+
+* Build and install the app.
+
+```
+bazel build -c opt --cxxopt=--std=c++11 --cxxopt=-Wno-all //tensorflow/contrib/lite/java/ovic/demo/app:ovic_benchmarker_binary
+adb install -r bazel-bin/tensorflow/contrib/lite/java/ovic/demo/app/ovic_benchmarker_binary.apk
+```
+
+Start the app and click the `Start` button in dark green. The button should turn bright green, signaling that the experiment is running. The benchmarking results will be displayed after about the `WALL_TIME` you specified above. For example:
+
+```
+my_model.lite: Average latency=158.6ms after 20 runs.
+```
+
+### Sample latencies
+
+Note: the benchmarking results can be quite different depending on the background processes running on the phone. A few things that help stabilize the app's readings are placing the phone on a cooling plate, restarting the phone, and shutting down internet access.
+
+| Model | Pixel 1 latency (ms) | Pixel 2 latency (ms) |
+| -------------------- |:---------------------:| --------------------:|
+| float_model.lite | 120 | 155 |
+| quantized_model.lite | 85 | 74 |
+| low_res_model.lite | 4.2 | 4.0 |
+
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.Bundle;
+import android.os.Process;
+import android.os.SystemClock;
import android.util.Log;
import android.view.View;
import android.widget.TextView;
+import java.io.BufferedReader;
+import java.io.File;
import java.io.FileInputStream;
+import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.nio.MappedByteBuffer;
private static final double WALL_TIME = 3000;
/** Maximum number of iterations in each benchmarking experiment. */
private static final int MAX_ITERATIONS = 100;
+ /** Mask for binding to a single big core. Pixel 1 (4), Pixel 2 (16). */
+ private static final int BIG_CORE_MASK = 16;
+ /** Amount of time in milliseconds to wait for affinity to set. */
+ private static final int WAIT_TIME_FOR_AFFINITY = 1000;
/* The model to be benchmarked. */
private MappedByteBuffer model = null;
Log.e(TAG, "Can't initialize benchmarker.", e);
throw e;
}
+ String displayText = "";
+ try {
+ setProcessorAffinity(BIG_CORE_MASK);
+ } catch (IOException e) {
+ Log.e(TAG, e.getMessage());
+ displayText = e.getMessage() + "\n";
+ }
Log.i(TAG, "Successfully initialized benchmarker.");
int testIter = 0;
Boolean iterSuccess = false;
if (textView != null) {
if (testIter > 0) {
- textView
- .setText(
- MODEL_PATH
- + ": Average latency="
- + df2.format(totalLatency / testIter)
- + "ms after "
- + testIter
- + " runs.");
+ textView.setText(
+ displayText
+ + MODEL_PATH
+ + ": Average latency="
+ + df2.format(totalLatency / testIter)
+ + "ms after "
+ + testIter
+ + " runs.");
} else {
textView.setText("Benchmarker failed to run on more than one images.");
}
}
}
+
+ private static void setProcessorAffinity(int mask) throws IOException {
+ int myPid = Process.myPid();
+ Log.i(TAG, String.format("Setting processor affinity to 0x%02x", mask));
+
+ String command = String.format("taskset -a -p %x %d", mask, myPid);
+ try {
+ Runtime.getRuntime().exec(command).waitFor();
+ } catch (InterruptedException e) {
+ throw new IOException("Interrupted: " + e);
+ }
+
+ // Make sure set took effect - try for a second to confirm the change took. If not then fail.
+ long startTimeMs = SystemClock.elapsedRealtime();
+ while (true) {
+ int readBackMask = readCpusAllowedMask();
+ if (readBackMask == mask) {
+ Log.i(TAG, String.format("Successfully set affinity to 0x%02x", mask));
+ return;
+ }
+ if (SystemClock.elapsedRealtime() > startTimeMs + WAIT_TIME_FOR_AFFINITY) {
+ throw new IOException(
+ String.format(
+ "Core-binding failed: affinity set to 0x%02x but read back as 0x%02x\n"
+ + "please root device.",
+ mask, readBackMask));
+ }
+
+ try {
+ Thread.sleep(50);
+ } catch (InterruptedException e) {
+ // Ignore sleep interrupted, will sleep again and compare is final cross-check.
+ }
+ }
+ }
+
+ public static int readCpusAllowedMask() throws IOException {
+ // Determine how many CPUs there are total
+ final String pathname = "/proc/self/status";
+ final String resultPrefix = "Cpus_allowed:";
+ File file = new File(pathname);
+ String line = "<NO LINE READ>";
+ String allowedCPU = "";
+ Integer allowedMask = null;
+ BufferedReader bufReader = null;
+ try {
+ bufReader = new BufferedReader(new FileReader(file));
+ while ((line = bufReader.readLine()) != null) {
+ if (line.startsWith(resultPrefix)) {
+ allowedMask = Integer.valueOf(line.substring(resultPrefix.length()).trim(), 16);
+ allowedCPU = bufReader.readLine();
+ break;
+ }
+ }
+ } catch (RuntimeException e) {
+ throw new IOException(
+ "Invalid number in " + pathname + " line: \"" + line + "\": " + e.getMessage());
+ } finally {
+ if (bufReader != null) {
+ bufReader.close();
+ }
+ }
+ if (allowedMask == null) {
+ throw new IOException(pathname + " missing " + resultPrefix + " line");
+ }
+ Log.i(TAG, allowedCPU);
+ return allowedMask;
+ }
}
projects / platform / upstream / tensorflow.git / commitdiff