PULP-Detector

Lorenzo Lamberti, Luca Bompani, Victor Javier Kartsch, Manuele Rusci, Daniele Palossi, Luca Benini.

Copyright (C) 2023 University of Bologna, ETH Zürich. All rights reserved.

Video Demo · IEEExplore · arXiv

Video: YouTube

Citing: "Bio-inspired Autonomous Exploration Policies with CNN-based Object Detection on Nano-drones" (IEEExplore, arXiv)

@INPROCEEDINGS{pulp_detector,
  author={Lamberti, Lorenzo and Bompani, Luca and Kartsch, Victor Javier and Rusci, Manuele and Palossi, Daniele and Benini, Luca},
  booktitle={2023 Design, Automation \& Test in Europe Conference \& Exhibition (DATE)},
  title={{{Bio-inspired Autonomous Exploration Policies with CNN-based Object Detection on Nano-drones}}},
  year={2023},
  volume={},
  number={},
  pages={1-6},
  doi={10.23919/DATE56975.2023.10137154}}

1. Introduction

What is PULP-Detector ?

PULP-Detector is a nano-drone system that strives for both maximizing the exploration of a room while performing visual object detection. The Exploration policies as implemented as lightweight and bio-inpired state machines. The object detection CNN is based on the MobilenetV2-SSD network. The drone performs obstacle avoidance thanks to Time-of-flight sensors. The drone is completely autonomous -- no human operator, no ad-hoc external signals, and no remote laptop!

• Software component: Object detection CNN: is a shallow convolutional neural network (CNN) composed of Mobilenet-v2 backbone plus the SSD (single-shot detector) heads. It runs at 1.6-4.3 FPS onboard.

• Hardware components: The hardware soul of PULP-Detector is an ultra-low power visual navigation module embodied by a pluggable PCB (called shield or deck) for the Crazyflie 2.0/2.1 nano-drone. The shield features a Parallel Ultra-Low-Power (PULP) GAP8 System-on-Chip (SoC) from GreenWaves Technologies (GWT), an ultra-low power HiMax HBM01 camera, and off-chip Flash/DRAM memory; This pluggable PCB has evolved over time, from the PULP-Shield , the first custom-made prototype version developed at ETH Zürich, and its commercial off-the-shelf evolution, the AI-deck.

Summary of characteristics:

• Hardware: AI-deck

• Deep learning framework: Tensorflow 1.15 (Tensorflow Object detection API)

• Quantization: fixed-point 8 bits, fully automated with NNTool

• Deployment: fully automated with AutoTiler

We release here, as open source, all our code, hardware designs, datasets, and trained networks.

Setup

Clone recursively to download all submodules

git clone git@github.com:pulp-platform/pulp-detector.git --recursive

PULP-Platforms refs

PULP Platform Youtube channel (subscribe it!)

PULP Platform Website.

Licenses

All files under:

• ./crazyflie_app/random-following-spiral
• ./crazyflie_app/rotate
• ./gap8_app/SSD_tin_can_bottle.c

are original and licensed under Apache-2.0, see LICENSE.Apache.md.

The images used for the training and testing need to be downloaded and copied into the following folder:

• dataset/

all the files can be downloaded from this link and are under the Creative Commons Attribution Non Commercial No Derivatives 4.0 International see LICENSE.CC.md

All files under:

• ./training/

Are from Tensorflow, released under Apache-2.0 License, see LICENSE.Apache.md.

All files under:

• ./gap8_app/ (except for ./gap8_app/SSD_tin_can_bottle.c)

Are from GreenWaves Technologies, released under a BSD License, see LICENSE.BSD.md

The external modules under:

• ./viewer-pulp-detector/
• ./crazyflie_app/crazyflie-firmware
• ./crazyflie_app/crazyflie-firmware-modified

Are from Bitcraze, released under a GPL-3.0 license.