[platform/upstream/hailort.git] /

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Python inference tutorial\n",
    "\n",
    "This tutorial will walk you through the inference process.\n",
    "\n",
    "**Requirements:**\n",
    "\n",
    "* Run the notebook inside the Python virtual environment: ```source hailo_virtualenv/bin/activate```\n",
    "\n",
    "It is recommended to use the command ``hailo tutorial`` (when inside the virtualenv) to open a Jupyter server that contains the tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standalone hardware deployment\n",
    "\n",
    "The standalone flow allows direct access to the HW, developing applications directly on top of Hailo\n",
    "core HW, using HailoRT. This way we can use the Hailo hardware without Tensorflow, and\n",
    "even without the Hailo SDK (after the HEF is built).\n",
    "\n",
    "An HEF is Hailo's binary format for neural networks. The HEF files contain:\n",
    "\n",
    "* Target HW configuration\n",
    "* Weights\n",
    "* Metadata for HailoRT (e.g. input/output scaling)\n",
    "\n",
    "First create the desired target object. In our example we use the Hailo-8 PCIe interface:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from multiprocessing import Process\n",
    "from hailo_platform import (HEF, VDevice, HailoStreamInterface, InferVStreams, ConfigureParams,\n",
    "    InputVStreamParams, OutputVStreamParams, InputVStreams, OutputVStreams, FormatType)\n",
    "\n",
    "# The target can be used as a context manager (\"with\" statement) to ensure it's released on time.\n",
    "# Here it's avoided for the sake of simplicity\n",
    "target = VDevice()\n",
    "\n",
    "# Loading compiled HEFs to device:\n",
    "model_name = 'resnet_v1_18'\n",
    "hef_path = '../hefs/{}.hef'.format(model_name) \n",
    "hef = HEF(hef_path)\n",
    "    \n",
    "# Configure network groups\n",
    "configure_params = ConfigureParams.create_from_hef(hef=hef, interface=HailoStreamInterface.PCIe)\n",
    "network_groups = target.configure(hef, configure_params)\n",
    "network_group = network_groups[0]\n",
    "network_group_params = network_group.create_params()\n",
    "\n",
    "# Create input and output virtual streams params\n",
    "input_vstreams_params = InputVStreamParams.make(network_group, format_type=FormatType.FLOAT32)\n",
    "output_vstreams_params = OutputVStreamParams.make(network_group, format_type=FormatType.UINT8)\n",
    "\n",
    "# Define dataset params\n",
    "input_vstream_info = hef.get_input_vstream_infos()[0]\n",
    "output_vstream_info = hef.get_output_vstream_infos()[0]\n",
    "image_height, image_width, channels = input_vstream_info.shape\n",
    "num_of_images = 10\n",
    "low, high = 2, 20\n",
    "\n",
    "# Generate random dataset\n",
    "dataset = np.random.randint(low, high, (num_of_images, image_height, image_width, channels)).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running hardware inference\n",
    "Infer the model and then display the output shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Infer \n",
    "with InferVStreams(network_group, input_vstreams_params, output_vstreams_params) as infer_pipeline:\n",
    "    input_data = {input_vstream_info.name: dataset}\n",
    "    with network_group.activate(network_group_params):\n",
    "        infer_results = infer_pipeline.infer(input_data)\n",
    "        print('Stream output shape is {}'.format(infer_results[output_vstream_info.name].shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Streaming inference\n",
    "\n",
    "This section shows how to run streaming inference using multiple processes in Python.\n",
    "\n",
    "Note: This flow is not supported on Windows.\n",
    "\n",
    "We will not use infer. Instead we will use a send and receive model.\n",
    "The send function and the receive function will run in different processes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the send and receive functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def send(configured_network, num_frames):\n",
    "    configured_network.wait_for_activation(1000)\n",
    "    vstreams_params = InputVStreamParams.make(configured_network)\n",
    "    with InputVStreams(configured_network, vstreams_params) as vstreams:\n",
    "        vstream_to_buffer = {vstream: np.ndarray([1] + list(vstream.shape), dtype=vstream.dtype) for vstream in vstreams}\n",
    "        for _ in range(num_frames):\n",
    "            for vstream, buff in vstream_to_buffer.items():\n",
    "                vstream.send(buff)\n",
    "\n",
    "def recv(configured_network, vstreams_params, num_frames):\n",
    "    configured_network.wait_for_activation(1000)\n",
    "    with OutputVStreams(configured_network, vstreams_params) as vstreams:\n",
    "        for _ in range(num_frames):\n",
    "            for vstream in vstreams:\n",
    "                data = vstream.recv()\n",
    "\n",
    "def recv_all(configured_network, num_frames):\n",
    "    vstreams_params_groups = OutputVStreamParams.make_groups(configured_network)\n",
    "    recv_procs = []\n",
    "    for vstreams_params in vstreams_params_groups:\n",
    "        proc = Process(target=recv, args=(configured_network, vstreams_params, num_frames))\n",
    "        proc.start()\n",
    "        recv_procs.append(proc)\n",
    "    for proc in recv_procs:\n",
    "        proc.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the amount of frames to stream, define the processes, create the target and run processes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the amount of frames to stream\n",
    "num_of_frames = 1000\n",
    "\n",
    "send_process = Process(target=send, args=(network_group, num_of_frames))\n",
    "recv_process = Process(target=recv_all, args=(network_group, num_of_frames))\n",
    "recv_process.start()\n",
    "send_process.start()\n",
    "print('Starting streaming (hef=\\'{}\\', num_of_frames={})'.format(model_name, num_of_frames))\n",
    "with network_group.activate(network_group_params):\n",
    "    send_process.join()\n",
    "    recv_process.join()\n",
    "print('Done')\n",
    "\n",
    "target.release()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}