Custom Image Classifier

If you want a custom image classifier, but don’t have the right data or the know-how to build it, you’ve come to the right place. With this Flutter app (runs on both iOS and Android) you can create datasets, collaborate on the collection of training data, and then trigger the training of the custom image classifier from the device. Once the classifier is ready, you can use it from the same app, and run inferences on device.

             

        Create Datasets                           Gather training data                      Infer on-device

See a live demo of the app presented at Google I/O '19.

Status

This sample is no longer actively maintained and is left here for reference only.

Full list of features

• Create datasets and corresponding labels (e.g. flowers -> rose, tulip, sunflower, etc).
• Collect images for a dataset via the phone’s camera. Images can be collected as individual pictures or short videos that are converted to multiple individual images.
• Invite other users to contribute to your dataset.
• Export your dataset.
• Trigger cloud-based training of a model based on your dataset.
• Once the model is trained, run on-device inferences from within the same app.
• Export the trained TFLite model to use in any other apps using the ML Kit SDK.

Google Services used in the sample app

• AutoML Vision Edge in ML Kit
• Cloud Storage for Firebase
• Cloud Functions for Firebase
• Cloud Firestore
• Firebase Authentication for Google Sign-In
• Firebase Cloud Messaging

AutoML Vision Edge in ML Kit generates models meant to run on-device. If you are interested in cloud-hosted models for more fine-grained image recognition or higher accuracy, visit the Cloud AutoML documentation to learn more.

Please note:

• Treat this project as sample code. You’ll need to create your own Firebase project and include its configuration files in the app in order to use it.
• Given that the app requires access to several Google Cloud APIs, your Firebase project will need to be on the Blaze plan.
• This app makes use of several Firebase and Google Cloud resources. You may be charged for any resources consumed. Learn more about # here.
• You are responsible for your app and its distribution, and complying with applicable terms and usage policies for Firebase and Google Cloud services.
• You are responsible for taking any necessary steps before distributing your app to third-party users, such as adding a privacy policy.

Getting Started

Note: Before proceeding, make sure you have Flutter & Node.js/NPM setup for your system.

1. Clone this repo locally

   git clone https://github.com/firebase/mlkit-custom-image-classifier
   

2. Create a new Firebase project from the console.

3. Take a note of the Project ID from Settings page as we'll use it later. Also add a support email - this is required for sign-in to work.

4. Follow the steps to add Firebase to the Flutter app in this repo. For the Android version, use com.example.classifier as bundle ID and for iOS use com.example.imageclassifier. If you want to provide your own bundleIDs, remember to replace the pre-filled bundleIDs from the code in flutter/

   NOTE: For the Android app, make sure to add a Debug signing certificate SHA-1 for Google Sign-in to work.

Enable Services

1. Enable Firestore by going to Database tab in Firebase Console, click on Create Database, select "Start in locked mode" & click on Enable.
2. Next, head to Sign-In Method tab under Authentication tab in Firebase console and enable the Google sign-in provider.
3. Next up, we'd need to enable ML Kit by going to ML Kit and click on getting started.
4. Follow that by clicking on the AutoML tab and click on getting started. This should take a few minutes to provision AutoML.
5. Once that's done, we'd need to head to Storage and enable Storage. Finally, we'll need to add the pre-generated AutoML bucket (named: ${PROJECT_ID}-vcm) by doing the following:
   • Click on Add bucket in the overflow menu
   • In the "Set up bucket" dialog, choose Import existing Google Cloud Storage buckets and here you'll see a bucket name ${PROJECT_ID}-vcm. Select this bucket and click on continue.
   • Finally, head to the rules tab and copy the rules from storage.rules & automlstorage.rules to the ${PROJECT_ID}.appspot.com & ${PROJECT_ID}-vcm buckets respectively.

Firebase setup

1. You must have the Firebase CLI installed. If you don't have it install it with npm install -g firebase-tools and then configure it with firebase login.
2. Configure the CLI locally by using firebase use --add and select your project in the list.
3. Install dependencies locally by running: cd functions; npm install; cd -
4. For sending emails via your firebase functions, you'd need to set your api_key for the service you're using. In this case, we use Sendgrid. Once you have an API key, run the following command to configure it

$ firebase functions:config:set sendgrid.key="<API_KEY>"

5. Next, set the FROM_EMAIL constant in this file to be email ID from which emails will be sent.
6. Follow that up with setting the APP_NAME that you've decided for your app.
7. Go to the Firebase Console, select the gear image > project settings > Service Accounts and click Generate New Private Key to download a Service Account Key JSON document.
8. Add the JSON service account credentials file to the Functions directory as functions/lib/service-account-key.json.
9. Now run, npm run deploy to deploy the functions, firestore & storage configuration.

Flutter app setup

1. Set the PROJECT_ID constant in the flutter app in this file.
2. If you're using iOS, you'd need to follow these steps for Google Sign-In to work.
3. You can now run the app locally!
4. If you want to deploy the app, make sure to read the Android and iOS instructions to finalize the app name, icon etc.

Using the app

1. To create your own datasets, manage datasets or add data to existing datasets, log into the app.
2. Use the main screen to add new datasets, or browse available datasets. Use the camera button to take a picture and run an inference with the latest model available for a dataset.
3. Tap on a dataset to view the labels. If you are the owner, you can add more labels to it. If you've been invited to collaborate on a dataset, you can add data to labels in dataset.
4. Once a dataset has enough data, the owner can initiate training of a new model on AutoML. Note: For best results, make sure each label has a diverse set of images and at least 100 images per label.
5. If you want to view a dataset and start model training from the console, click "Export to Firebase" from the overflow menu, then go to ML Kit -> AutoML in the Firebase console. Otherwise you can train directly from the app by clicking "Train Model" under a dataset. You will be notified when the model training is complete.