Azure IoT Edge can make your IoT solution more efficient by moving workloads out of the cloud and to the edge. This capability lends itself well to services that process a lot of data, like computer vision models.
The Custom Vision Service lets you build custom image classifiers and deploy them to devices as containers.
Together, these two services enable you to find insights from images or video streams without having to transfer all of the data off site first. Custom Vision provides a classifier that compares an image against a trained model to generate insights.
For example, Custom Vision on an IoT Edge device could determine whether a highway is experiencing higher or lower traffic than normal, or whether a parking garage has available parking spots in a row. These insights can be shared with another service to take action.
In this tutorial, you will learn how to
- Build an image classifier with Custom Vision.
- Create an IoT Edge solution.
- Build and deploy your IoT Edge solution.
- Evaluate the result on UWP.
- Send the results of the image classifier to IoT Hub.
An Azure IoT Edge device:
| Operation System | Support |
|---|---|
| Windows 10 Desktop 1809 or later | Yes |
| Windows Server 2016 | Yes |
| Windows Server 2019 | Yes |
A device installed above OS and Azure IoT Edge runtime.
Cloud resources:
- A standard-tier IoT Hub in Azure.
- A container registry. This tutorial uses Azure Container Registry.
- Know the credentials for your container registry admin account.
Development resources:
- Python
- Git
- Visual Studio Code
- Azure IoT Edge extension for Visual Studio Code
- Python extension for Visual Studio Code
- Docker CE
To build an image classifier, you need to create a Custom Vision project and provide training images. For more information about the steps that you take in this section, see How to build a classifier with Custom Vision.
Once your image classifier is built and trained, you can export it as a Docker container and deploy it to an IoT Edge device.
1. In your web browser, navigate to the Custom Vision web page.
2. Select # and # with the same account that you use to access Azure resources.
3. Select New project.
4. Create your project with the following values:
5. Select Create project.
Creating an image classifier requires a set of training images, as well as test images.
1. Prepare sample images for training and testing your classifier.
2. Return to your Custom Vision project and select Add images.
3. Select all images in the folder and then Open.
4. Select Upload files. (notes: Object Detection model needs at least 15 images per class)
5. Tag the images with dragging bounds and labelling.
6. When all of images are tagged, select Train to train the classifier.
1. After training your classifier, select Export on the Performance page of the classifier.
2. Select DockerFile for Windows OS version.
3. Select Export and then Download the .zip file.
Now you have the files for a container version of your image classifier on your development machine. In this section, you configure the image classifier container to run as an IoT Edge module.
A solution is a logical way of developing and organizing multiple modules for a single IoT Edge deployment. A solution contains code for one or more modules as well as the deployment manifest that declares how to configure them on an IoT Edge device.
-
In Visual Studio Code, select View > Terminal to open the VS Code integrated terminal.
-
In the integrated terminal, enter the following command to install (or update) cookiecutter, which you use to create the IoT Edge python module template in VS Code:
pip install --upgrade --user cookiecutter(Notes: Ensure the directory where cookiecutter will be installed is in your environment’s Path in order to make it possible to invoke it from a command prompt.)
-
Select View > Command Palette to open the VS Code command palette.
-
In the command palette, enter and run the command Azure IoT Edge: New IoT Edge solution. In the command palette, provide the following information to create your solution:
| Steps | Description |
|---|---|
| Select folder | Choose the location on your development machine for VS Code to create the solution files. |
| Provide a solution name | Enter a descriptive name for your solution, like CustomVisionSolution, or accept the default. |
| Select module template | Choose Python Module. |
| Provide a module name | Name your module classifier. It's important that this module name be lowercase. IoT Edge is case-sensitive when referring to modules, and this solution uses a library that formats all requests in lowercase. |
| Provide Docker image repository for the module | An image repository includes the name of your container registry and the name of your container image. Your container image is prepopulated from the last step. Replace localhost:5000 with the login server value from your Azure container registry. You can retrieve the login server from the Overview page of your container registry in the Azure portal. The final string looks like .azurecr.io/classifier. |
The Python module template in Visual Studio code contains some sample code that you can run to test IoT Edge. You won't use that code in this scenario. Instead, use the steps in this section to replace the sample code with the image classifier container that you exported previously.
- In your file explorer, browse to the Custom Vision package that you downloaded and extracted. Copy all the contents from the extracted package. It should be two folders, app and azureml, and two files, Dockerfile and README.
- In your file explorer, browse to the directory where you told Visual Studio Code to create your IoT Edge solution.
- Open the classifier module folder. If you used the suggested names in the previous section, the folder structure looks like CustomVisionSolution / modules / classifier.
- Paste the files into the classifier folder.
- Open the module.json file in the classifier folder.
- Update the platforms parameter to point to the new Dockerfile that you added, and remove the ARM32 architecture and AMD64.debug options. Save the changes.
"platforms": {
"amd64": "./Dockerfile"
}So far in this tutorial you've trained a Custom Vision model to classify images of trees, and packaged that model up as an IoT Edge module. Now, you're ready to create the deployment manifest that will tell an IoT Edge device how to start and run the module.
The IoT Edge extension for Visual Studio Code provides a template in each IoT Edge solution to help you create a deployment manifest.
1. Open the deployment.template.json file in the solution folder.
2. Find the modules section, which should contain two modules: the one that you created, classifier, and a second that's included by default, tempSensor.
3. Delete the tempSensor module with all of its parameters. This module is included to provide sample data for test scenarios, but we don't need it in this deployment.
4. If you named the image classification module something other than classifier, check the name now and ensure that it's all lowercase. The cameraCapture module calls the classifier module using a requests library that formats all requests in lowercase, and IoT Edge is case-sensitive.
5. Update the createOptions parameter for the classifier module with the following JSON.
"createOptions": {
"HostConfig": {
"PortBindings": {
"80/tcp": [
{
"HostPort": "23114"
}
]
}
}
}
6. Save the changes.
The prerequisites for this tutorial listed a container registry, which is necessary to store the container images for the modules that you created. You need to provide access credentials for your registry in two places: in Visual Studio Code so that you can build and push your images to the registry, and in the deployment manifest so that your IoT Edge device can pull and deploy the images.
If you're using Azure Container Registry, make sure you know the username, login server, and password for the admin account.
- In Visual Studio Code, open the integrated terminal by selecting View > Terminal.
- Enter the following command in the integrated terminal:
docker login -u <registry username> <registry login server>- When prompted, provide your registry password and press Enter.
- Open the .env file in the solution folder. This file is git ignored and stores your registry credentials so that you don't have to hardcode them into the deployment template file.
CONTAINER_REGISTRY_USERNAME=<registry username>
CONTAINER_REGISTRY_PASSWORD=<registry password>- Provide the username and password for your container registry, without quotation marks around the values. Save the .env file.
With the module created and the deployment manifest template configured, you're ready to build the container images and push it to your container registry.
Once the images are in your registry, you can deploy the solution to an IoT Edge device. You can set modules on a device through the IoT Hub, but you can also access your IoT Hub and devices through Visual Studio Code. In this section, you set up access to your IoT Hub then use VS Code to deploy your solution to your IoT Edge device.
First, build and push your solution to your container registry.
- In the VS Code explorer, right-click the deployment.template.json file and select Build and push IoT Edge solution. You can watch the progress of this operation in the integrated terminal in VS Code.
- Notice that a new folder was added to your solution, config. Expand this folder and open the deployment.json file inside.
- Review the information in the deployment.json file. The deployment.json file is created (or updated) automatically based on the deployment template file that you configured and information from the solution, including the .env file and the module.json files.
Next, set up access to your IoT Hub from within Visual Studio Code.
- In the VS Code command palette, select Azure IoT Hub: Select IoT Hub.
- Follow the prompts to # to your Azure account.
- In the command palette, select your Azure subscription then select your IoT Hub.
Finally, select your device and deploy your solution.
- In the VS Code explorer, expand the Azure IoT Hub Devices section.
- Right-click on the device that you want to target with your deployment and select Create deployment for single device.
- In the file explorer, navigate to the config folder inside your solution and choose deployment.json. Click Select Edge deployment manifest.
If the deployment is successful, a confirmation message is printed in the VS Code output. In the VS Code explorer, expand the details about the IoT Edge device that you used for this deployment. Hover your cursor on the Azure IoT Hub Devices header to enable the refresh button if the modules don't show up right away. It may take a few moments for the modules to start and report back to IoT Hub.
You can also check to see that all the modules are up and running on your device itself. On your IoT Edge device, run the following command to see the status of the modules. It may take a few moments for the modules to start.
iotedge list
The repo provide the sample UWP code to connect DJI drone with the Azure Custom Vision model on IoT Edge Device. Now you can connect DJI drone to the IoT Edge Device than Run the sample code.
- Open the DJIWindowsSDKSample.sln with Visual Studio 2017.
- Make sure the develop mode is debug and platform is x86.
- Run the sample
- Press Activate button.
- Select FPV on the left navigator.
You can see the logs of iotedge module by:
iotedge logs <module name>
This section is optional for testing Custom Vision model. In a real Custom Vision deployment, you would have a camera providing live images or video streams. For this scenario, you simulate the camera by building a module that sends a test image to the image classifier.
Follow this guide.