Monday, October 30, 2023
HomeBig DataAI Optimization and Deployment with Intel's OpenVINO Toolkit

AI Optimization and Deployment with Intel’s OpenVINO Toolkit


We speak about AI nearly every day as a consequence of its rising affect in changing people’ handbook work. Constructing AI-enabled software program has quickly grown in a quick time. Enterprises and companies consider in integrating dependable and accountable AI of their utility to generate extra income. Essentially the most difficult a part of integrating AI into an utility is the mannequin inference and computation assets utilized in coaching the mannequin. Many methods exist already that enhance the efficiency by optimizing the mannequin throughout inference with fewer computation assets. With this drawback assertion, Intel launched the OpenVINO Toolkit, an absolute game-changer. OpenVINO is an open-source toolkit for optimizing and deploying AI inference.

Studying Goals

On this article, we’ll:

  • Perceive what the OpenVINO Toolkit is and its goal in optimizing and deploying AI inference fashions.
  • Discover the sensible use circumstances of OpenVINO, particularly its significance in the way forward for AI on the edge.
  • Learn to implement a textual content detection challenge in a picture utilizing OpenVINO in Google Colab.
  • Uncover the important thing options and benefits of utilizing OpenVINO, together with its mannequin compatibility and assist for {hardware} accelerators and the way it can affect numerous industries and functions.

This text was revealed as part of the Information Science Blogathon.

What’s OpenVINO?

OpenVINO | Intel's OpenVINO Toolkit

OpenVINO, which stands for Open Visible Inference and Neural Community Optimization, is an open-source toolkit developed by the Intel workforce to facilitate the optimization of deep studying fashions. The imaginative and prescient of the OpenVINO toolkit is to spice up your AI deep-learning fashions and deploy the appliance on-premise, on-device, or within the cloud with extra effectivity and effectiveness.

OpenVINO Toolkit is especially beneficial as a result of it helps many deep studying frameworks, together with in style ones like TensorFlow, PyTorch, Onnx, and Caffe. You may prepare your fashions utilizing your most popular framework after which use OpenVINO to transform and optimize them for deployment on Intel’s {hardware} accelerators, like CPUs, GPUs, FPGAs, and VPUs.

Regarding inference, OpenVINO Toolkit provides numerous instruments for mannequin quantization and compression, which may considerably cut back the scale of deep studying fashions with out shedding inference accuracy.

Why Use OpenVINO?

The craze of AI is at the moment in no temper to decelerate. With this recognition, it’s evident that increasingly more functions might be developed to run AI functions on-premise and on-device. Just a few of the difficult areas the place OpenVINO excels make it a perfect alternative why it’s essential to make use of OpenVINO:

OpenVINO Mannequin Zoo

OpenVINO offers a mannequin zoo with pre-trained deep-learning fashions for duties like Secure Diffusion, Speech, Object detection, and extra. These fashions can function a place to begin to your initiatives, saving you time and assets.

Mannequin Compatibility

OpenVINO helps many deep studying frameworks, together with TensorFlow, PyTorch, ONNx, and Caffe. This implies you should use your most popular framework to coach your fashions after which convert and optimize them for deployment utilizing the OpenVINO Toolkit.

Excessive Efficiency

OpenVINO is optimized for quick inference, making it appropriate for real-time functions like laptop imaginative and prescient, robotics, and IoT gadgets. It leverages {hardware} acceleration corresponding to FPGA, GPU, and TPU to attain excessive throughput and low latency.

AI in Edge Future Utilizing Intel OpenVINO

AI in Edge Future Using Intel OpenVINO

AI in Edge is probably the most difficult space to deal with. Constructing an optimized resolution to resolve {hardware} constraints is now not unimaginable with the assistance of OpenVINO. The way forward for AI in Edge with this Toolkit has the potential to revolutionize numerous industries and functions.

Let’s learn the way OpenVINO works to make it appropriate for AI in Edge.

  • The first step is to construct a mannequin utilizing your favourite deep-learning frameworks and convert it into an OpenVINO core mannequin. Each other different is to make use of pre-trained fashions utilizing the OpenVINO mannequin zoo.
  • As soon as the mannequin is been educated, the following step is compression. OpenVINO Toolkit offers a Neural Community compression framework (NNCF).
  • Mannequin Optimizer converts the pre-trained mannequin into an acceptable format. The optimizer consists of IR information. IR information refers back to the Intermediate Illustration of a deep studying mannequin, which is already optimized and reworked for deployment with OpenVINO. The mannequin weights are in .XML and .bin file format.
  • At mannequin deployment, the OpenVINO Inference Engine can load and use the IR information on the goal {hardware}, enabling quick and environment friendly inference for numerous functions.

With this strategy, OpenVINO can play a significant position in AI in Edge. Let’s soiled our palms with a code challenge to implement Textual content detection in a picture utilizing the OpenVINO Toolkit.

Textual content Detection in an Picture Utilizing OpenVINO Toolkit

On this challenge implementation, we’ll use Google Colab as a medium to run the appliance efficiently. On this challenge, we’ll use the horizontal-text-detection-0001 mannequin from the OpenVINO mannequin Zoo. This pre-trained mannequin detects horizontal textual content in enter photographs and returns a blob of knowledge within the form (100,5). This response seems like (x_min, y_min, x_max, y_max, conf) format.

Step-by-Step Code Implementation

Set up

!pip set up openvino

Import Required Libraries

Let’s import the required modules to run this utility. OpenVINO helps a utils helper perform to obtain pre-trained weights from the supplied supply code URL.

import urllib.request

base = ""
utils_file = "/principal/notebooks/utils/"

    url= base + utils_file,

from notebook_utils import download_file

You may confirm, that notebook_utils is now efficiently downloaded, let’s rapidly import the remaining modules.

import openvino

import cv2
import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path

Obtain Weights

Initialize the Path to obtain IR information mannequin weight recordsdata of horizontal textual content detection in .xml and .bin format.

base_model_dir = Path("./mannequin").expanduser()

model_name = "horizontal-text-detection-0001"model_xml_name = f'{model_name}.xml'
model_bin_name = f'{model_name}.bin'

model_xml_path = base_model_dir / model_xml_name
model_bin_path = base_model_dir / model_bin_name

Within the following code snippet, we use three variables to simplify the trail the place the pre-trained mannequin weights exist.

model_zoo = ""
algo = "horizontal-text-detection-0001/FP32/"
xml_url = "horizontal-text-detection-0001.xml"
bin_url = "horizontal-text-detection-0001.bin"

model_xml_url = model_zoo+algo+xml_url
model_bin_url =  model_zoo+algo+bin_url

download_file(model_xml_url, model_xml_name, base_model_dir)
download_file(model_bin_url, model_bin_name, base_model_dir)
download the file | Intel's OpenVINO Toolkit

Load Mannequin

OpenVINO offers a Core class to work together with the OpenVINO toolkit. The Core class offers numerous strategies and capabilities for working with fashions and performing inference. Use read_model and go the model_xml_path. After studying the mannequin, compile the mannequin for a selected goal machine.

core = Core()

mannequin = core.read_model(mannequin=model_xml_path)
compiled_model = core.compile_model(mannequin=mannequin, device_name="CPU")

input_layer_ir = compiled_model.enter(0)
output_layer_ir = compiled_model.output("bins")

Within the above code snippet, the complied mannequin returns the enter picture form (704,704,3), an RGB picture however in PyTorch format (1,3,704,704) the place 1 is the batch measurement, 3 is the variety of channels, 704 is peak and weight. Output returns (x_min, y_min, x_max, y_max, conf). Let’s load an enter picture now.

Load model

Load Picture

The mannequin weight is [1,3,704,704]. Consequently, you need to resize the enter picture accordingly to match this form. In Google Colab or your code editor, you possibly can add your enter picture, and in our case, the picture file is known as sample_image.jpg.

picture = cv2.imread("sample_image.jpg")

# N,C,H,W = batch measurement, variety of channels, peak, width.
N, C, H, W = input_layer_ir.form

# Resize the picture to satisfy community anticipated enter sizes.
resized_image = cv2.resize(picture, (W, H))

# Reshape to the community enter form.
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)

print("Mannequin weights form:")
print("Picture after resize:")
Input image | Intel's OpenVINO Toolkit

Show the enter picture.

plt.imshow(cv2.cvtColor(picture, cv2.COLOR_BGR2RGB))

Inference Engine

Beforehand, we used mannequin weights to compile the mannequin. Use compile the mannequin in context to the enter picture.

# Create an inference request.
bins = compiled_model([input_image])[output_layer_ir]

# Take away zero solely bins.
bins = bins[~np.all(boxes == 0, axis=1)]


The compiled_model returns bins with the bounding field coordinates. We use the cv2 module to create a rectangle and putText so as to add the boldness rating above the detected textual content.

def detect_text(bgr_image, resized_image, bins, threshold=0.3, conf_labels=True):
    # Fetch the picture shapes to calculate a ratio.
    (real_y, real_x), (resized_y, resized_x) = bgr_image.form[:2], resized_image.form[:2]
    ratio_x, ratio_y = real_x / resized_x, real_y / resized_y

    # Convert picture from BGR to RGB format.
    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)

    # Iterate by non-zero bins.
    for field in bins:
        # Decide a confidence issue from the final place in an array.
        conf = field[-1]
        if conf > threshold:
            (x_min, y_min, x_max, y_max) = [
                int(max(corner_position * ratio_y, 10)) if idx % 2
                else int(corner_position * ratio_x)
                for idx, corner_position in enumerate(box[:-1])

            # Draw a field based mostly on the place, parameters in rectangle perform are: 
            # picture, start_point, end_point, coloration, thickness.
            rgb_image = cv2.rectangle(rgb_image, (x_min, y_min), (x_max, y_max),(0,255, 0), 10)

            # Add textual content to the picture based mostly on place and confidence.
            if conf_labels:
                rgb_image = cv2.putText(
                    (x_min, y_min - 10),
                    (255, 0, 0),

    return rgb_image

Show the output picture

plt.imshow(detect_text(picture, resized_image, bins));


To conclude, we efficiently constructed Textual content detection in a picture challenge utilizing the OpenVINO Toolkit. Intel workforce repeatedly improves the Toolkit. OpenVINO additionally helps pre-trained Generative AI fashions corresponding to Secure Diffusion, ControlNet, Speech-to-text, and extra.

Key Takeaways

  • OpenVINO is a game-changer open supply instrument to spice up your AI deep-learning fashions and deploy the appliance on-premise, on-device, or within the cloud.
  • The first aim of OpenVINO is to optimize the deep fashions with numerous mannequin quantization and compression, which may considerably cut back the scale of deep studying fashions with out shedding inference accuracy.
  • This Toolkit additionally helps deploying AI functions on {hardware} accelerators corresponding to GPUs, FPGAs, ASIC, TPUs, and extra.
  • Varied industries can undertake OpenVINO and leverage its potential to make an affect on AI on the edge.
  • Utilization of the mannequin zoo pre-trained mannequin is easy as we applied textual content detection in photographs with just some traces of code.

Often Requested Questions

Q1. What’s Intel OpenVINO used for?

A. Intel OpenVINO offers a mannequin zoo with pre-trained deep-learning fashions for duties like Secure Diffusion, Speech, and extra. OpenVINO runs mannequin zoo pre-trained fashions on-premise, on-device, and within the cloud extra effectively and successfully.

Q2. What’s the distinction between OpenVINO and TensorFlow?

A. Each OpenVINO and TensorFlow are free and open-source. Builders use TensorFlow, a deep-learning framework, for mannequin improvement, whereas OpenVINO, a Toolkit, optimizes deep-learning fashions and deploys them on Intel {hardware} accelerators.

Q3. The place is OpenVINO used?

A. OpenVINO’s versatility and talent to optimize deep studying fashions for Intel {hardware} make it a beneficial instrument for AI and laptop imaginative and prescient functions throughout numerous industries corresponding to Army protection, Healthcare, Good cities, and lots of extra.

This fall. Is the Intel’s OpenVINO Toolkit free to make use of?

A. Sure, Intel’s OpenVINO toolkit is free to make use of. The Intel workforce developed this open-source toolkit to facilitate the optimization of deep studying fashions.

The media proven on this article shouldn’t be owned by Analytics Vidhya and is used on the Writer’s discretion.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments