review.tizen.org Git - platform/upstream/hailort.git/blob

   1 {
   2  "cells": [
   3   {
   4    "cell_type": "markdown",
   5    "metadata": {},
   6    "source": [
   7     "\n",
   8     "# Python Inference Tutorial - Multi Process Service and Model Scheduler\n",
   9     "\n",
  10     "This tutorial will walk you through the inference process using The Model Scheduler.\n",
  11     "\n",
  12     "**Requirements:**\n",
  13     "\n",
  14     "* Enable HailoRT Multi-Process Service before running inference\n",
  15     "* Run the notebook inside the Python virtual environment: ```source hailo_virtualenv/bin/activate```\n",
  16     "\n",
  17     "It is recommended to use the command ``hailo tutorial`` (when inside the virtualenv) to open a Jupyter server that contains the tutorials."
  18    ]
  19   },
  20   {
  21    "cell_type": "markdown",
  22    "metadata": {},
  23    "source": [
  24     "## Running Inference using HailoRT\n",
  25     "\n",
  26     "In this example we will use the Model Scheduler to run inference on multiple models.\n",
  27     "Each model is represented by an HEF which is built using the Hailo Dataflow Compiler.\n",
  28     "An HEF is Hailo's binary format for neural networks. The HEF files contain:\n",
  29     "\n",
  30     "* Target HW configuration\n",
  31     "* Weights\n",
  32     "* Metadata for HailoRT (e.g. input/output scaling)\n",
  33     "\n",
  34     "The Model Scheduler is an HailoRT component that comes to enhance and simplify the usage\n",
  35     "of the same Hailo device by multiple networks. The responsibility for activating/deactivating the network\n",
  36     "groups is now under HailoRT, and done **automatically** without user application intervention.\n",
  37     "In order to use the Model Scheduler, create the VDevice with scheduler enabled, configure all models to the device, and start inference on all models:"
  38    ]
  39   },
  40   {
  41    "cell_type": "code",
  42    "execution_count": null,
  43    "metadata": {},
  44    "outputs": [],
  45    "source": [
  46     "import numpy as np\n",
  47     "from multiprocessing import Process\n",
  48     "from hailo_platform import (HEF, VDevice, HailoStreamInterface, InferVStreams, ConfigureParams,\n",
  49     "    InputVStreamParams, OutputVStreamParams, InputVStreams, OutputVStreams, FormatType, HailoSchedulingAlgorithm)\n",
  50     "\n",
  51     "\n",
  52     "# Define the function to run inference on the model\n",
  53     "def infer(network_group, input_vstreams_params, output_vstreams_params, input_data):\n",
  54     "    rep_count = 100\n",
  55     "    with InferVStreams(network_group, input_vstreams_params, output_vstreams_params) as infer_pipeline:\n",
  56     "        for i in range(rep_count):\n",
  57     "            infer_results = infer_pipeline.infer(input_data)\n",
  58     "\n",
  59     "\n",
  60     "# Loading compiled HEFs:\n",
  61     "first_hef_path = '../hefs/resnet_v1_18.hef'\n",
  62     "second_hef_path = '../hefs/shortcut_net.hef'\n",
  63     "first_hef = HEF(first_hef_path)\n",
  64     "second_hef = HEF(second_hef_path)\n",
  65     "hefs = [first_hef, second_hef]\n",
  66     "\n",
  67     "# Creating the VDevice target with scheduler enabled\n",
  68     "params = VDevice.create_params()\n",
  69     "params.scheduling_algorithm = HailoSchedulingAlgorithm.ROUND_ROBIN\n",
  70     "with VDevice(params) as target:\n",
  71     "    infer_processes = []\n",
  72     "\n",
  73     "    # Configure network groups\n",
  74     "    for hef in hefs:\n",
  75     "        configure_params = ConfigureParams.create_from_hef(hef=hef, interface=HailoStreamInterface.PCIe)\n",
  76     "        network_groups = target.configure(hef, configure_params)\n",
  77     "        network_group = network_groups[0]\n",
  78     "\n",
  79     "        # Create input and output virtual streams params\n",
  80     "        input_vstreams_params = InputVStreamParams.make(network_group, format_type=FormatType.FLOAT32)\n",
  81     "        output_vstreams_params = OutputVStreamParams.make(network_group, format_type=FormatType.UINT8)\n",
  82     "\n",
  83     "        # Define dataset params\n",
  84     "        input_vstream_info = hef.get_input_vstream_infos()[0]\n",
  85     "        image_height, image_width, channels = input_vstream_info.shape\n",
  86     "        num_of_frames = 10\n",
  87     "        low, high = 2, 20\n",
  88     "\n",
  89     "        # Generate random dataset\n",
  90     "        dataset = np.random.randint(low, high, (num_of_frames, image_height, image_width, channels)).astype(np.float32)\n",
  91     "        input_data = {input_vstream_info.name: dataset}\n",
  92     "\n",
  93     "        # Create infer process\n",
  94     "        infer_process = Process(target=infer, args=(network_group, input_vstreams_params, output_vstreams_params, input_data))\n",
  95     "        infer_processes.append(infer_process)\n",
  96     "\n",
  97     "    print(f'Starting streaming on multiple models using scheduler')\n",
  98     "    for infer_process in infer_processes:\n",
  99     "        infer_process.start()\n",
 100     "    for infer_process in infer_processes:\n",
 101     "        infer_process.join()\n",
 102     "\n",
 103     "    print('Done inference')"
 104    ]
 105   }
 106  ],
 107  "metadata": {
 108   "kernelspec": {
 109    "display_name": "Python 3 (ipykernel)",
 110    "language": "python",
 111    "name": "python3"
 112   },
 113   "language_info": {
 114    "codemirror_mode": {
 115     "name": "ipython",
 116     "version": 3
 117    },
 118    "file_extension": ".py",
 119    "mimetype": "text/x-python",
 120    "name": "python",
 121    "nbconvert_exporter": "python",
 122    "pygments_lexer": "ipython3",
 123    "version": "3.8.10"
 124   }
 125  },
 126  "nbformat": 4,
 127  "nbformat_minor": 2
 128 }