This document provides guidance and background for a hands-on lab (in Python) leveraging Watson Studio and Watson AI services.
This lab is broken into five notebooks which provide an end-to-end solution for a very common and powerful use case, the Call Center Instrumentation and Analytics (CCIA) pattern. These notebooks leverage several Watson AI services to help organizations extract useful information from the unstructured data (audio files) to better understand the unstructured "dark data" that arises from phone calls to call centers.
Enterprises spend more than $1 trillion on 250 billion customer service calls each year. By using multiple IBM Watson "signal services" to extract signal from raw audio data; perform data analytics, clustering, unsupervised and machine learning, and visualizations, technical teams can use data understand patterns in call centers. KPI and ROI positive.
This lab consists of five notebooks (each step below is implemented as a separate notebook) detailed as follows:
- Step 1 - Speech to Text (STT) – Converts Raw Audio to Transcripts
- Step 2 - Natural Language Understanding (NLU) - extracts features concepts, entities, keywords, categories/topics, sentiment and emotion
- Step 3 - Natural Language Classifier (NLC) - is a user trained classification service, with user defined “ground truth” that classifies text chunks
- Step 4 - Tone Analyzer – uses linguistic analysis to detect emotional and language tones in written text
- Step 5 - Call Center Analytics – analyzes and visualizes the data signal to allow for interpretation of data and in cases, actionable insights
- This is a beginner lab intended to educate on the fundamentals of getting from data to insights with IBM Watson and open source tools
- Audience may include IT and operations teams curious about enriching unstructured data – the lab is NOT intended for sophisticated call center technologists
- Lab/code does NOT purport to compete with expensive and sophisticated solutions already in market
- The lab and code cover the basics – to educate on the fundamental plumbing and steps, to provide base for instrumentation
This lab requires the following services which you can create on IBM Cloud.
- Watson Studio: Watson Studio provides a suite of tools and a collaborative environment for data scientists, developers and domain experts for end-to-end data exploration, analysis and insights using the most popular open source machine learning packages and frameworks.
- Cloud Object Storage: IBM Cloud Object Storage is a highly scalable cloud storage service, designed for high durability, resiliency and security.
- Apache Spark: Apache Spark service on IBM Cloud is a hosted offering for Apache Spark which is an open source cluster computing framework optimized for extremely fast and large scale data processing, which you can access via the newly integrated notebook interface IBM Analytics for Apache Spark. You can connect to your existing data sources or take advantage of the on-demand big data optimization of Object Storage.
- Speech to Text (STT): The Watson Speech to Text service converts human voice to text. It supports multiple languages and it offers the unique capability of being able to train custom language and acoustic models, a much needed feature for enterprise applications.
- Natural Language Understanding (NLU): The Watson Natural Language Understanding service extracts several useful features from text such as concepts, entities, keywords, relations, sentiment, semantic roles, categories, and emotion.
- Natural Language Classifier (NLC): The Watson Natural Language Classifier service applies cognitive computing techniques to return the best matching classes for a sentence or a phrase. NLC basically informs what the text is about.
- Tone Analyzer (TA): The Watson Tone Analyzer service leverages cognitive linguistic analysis to identify a variety of tones at both the sentence and document level. To create these services, you need to have an IBM Cloud account. You can # for an IBM Cloud account if you don't already have one.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Storage category in the left navigation column.
- Select Object Storage service, provide a name for the service (or use default name), choose the Lite plan, and click Create.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Data & Analytics category in the left navigation column.
- Select Apache Spark service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Watson category in the left navigation column.
- Select Speech to Text service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Once the service is created, its info page is loaded. Click on Service Credentials in the left navigation column. If no credentials are created, click on New Credentials to create a new set of credentials.
{
"url": "https://stream.watsonplatform.net/speech-to-text/api",
"username": "**********",
"password": "**********"
}
- Copy the credentials as we'll need them in the notebook.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Watson category in the left navigation column.
- Select Natural Language Understanding service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Once the service is created, its info page is loaded. Click on Service Credentials in the left navigation column. If no credentials are created, click on New Credentials to create a new set of credentials.
{
"url": "https://gateway.watsonplatform.net/natural-language-understanding/api",
"username": "**********",
"password": "**********"
}
- Copy the credentials as we'll need them in the notebook.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Watson category in the left navigation column.
- Select Natural Language Classifier service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Once the service is created, its info page is loaded. Click on Service Credentials in the left navigation column. If no credentials are created, click on New Credentials to create a new set of credentials.
{
"url": "https://gateway.watsonplatform.net/natural-language-classifier/api",
"username": "**********",
"password": "**********"
}
- Copy the credentials as we'll need them in the notebook.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Watson category in the left navigation column.
- Select Tone Analyzer service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Once the service is created, its info page is loaded. Click on Service Credentials in the left navigation column. If no credentials are created, click on New Credentials to create a new set of credentials.
{
"url": "https://gateway.watsonplatform.net/tone-analyzer/api",
"username": "**********",
"password": "**********"
}
- Copy the credentials as we'll need them in the notebook.
- Log into your IBM Cloud account
- Navigate to Catalog.
- Click on Watson category in the left navigation column.
- Select Watson Studio service, provide a name for the service (or use default name), choose the Lite plan (only use Lite plan for evaluation), and click Create.
- Once the service is created, its info page is loaded. Click on Get Started to launch Watson studio.
To execute these notebooks:
- In Watson Studio, create a new Project and select the Complete option.
- Provide a project name and under Define Storage, select the cloud object storage instance to associate with your project. If you haven't created any cloud object storage instances, click Add to associate a Cloud Object Storage instance with your project. If you already have a Cloud Object Storage created, you can select it. If not, you can step through the screens to create a Cloud Object Storage service and associate it with your project.
- Associate an Apache Spark service with your project:
- In your project, navigate to the Settings tab and scroll down to the Associated Services section.
- Click on the Add Service drop down and select Spark ==> this will load the Apache Spark service page on IBM Cloud.
- If you have already created the Apache Spark service, select the Existing tab and select the Apache Spark service you want to associate with your project. If not, then selec the New tab to create a new Apache Spark service. Select the Lite plan and click Create.
- Provide a name for the service (or use the auto-generated name) and select the Lite plan and the Space (any of your IBM Cloud spaces; you may only have one). Click Confirm. ==> This would associate an Apache Spark service with your project which we need later on since we'll use SparkML machine learning code in the notebook.
- In your project, under the Assets tab, click on New Notebook.
- Provide a name for the notebook, and click on the From URL tab and provide the URL link for the notebook: https://github.com/mamoonraja/call-center-think18/blob/master/notebooks/Step1-speech-to-text.ipynb
- Click Create Notebook This would load the notebook in Watson Studio
- Execute the steps of the notebook. Make sure the steps of each notebook are run to completion before proceeding to the sub-sequent notebooks.
- Repeat steps 4-7 for the the notebooks corresponding to steps 2, 3, and 4:
- In your project, under Assets tab, click on New Notebook.
- Provide a name for the notebook, and click on the From URL tab and provide the URL link for the notebook: https://github.com/mamoonraja/call-center-think18/blob/master/notebooks/Step5-call-center-analytics.ipynb
- For the runtime, select the Apache Spark service instance you had associated with your project.
- Click Create Notebook This would load the notebook in Watson Studio.
- Make sure your kernel says Python 3.5 with Spark 2.1 If not, then go to the Kernel menu drop down, select Change Kernel and choose the Python 3.5 with Spark 2.1 kernel.
- Execute the steps of the notebook.
If successful – the lab participants or notebook users will
- Gain experience in using an IPython/Jupyter notebooks
- Connect to four Watson ‘signal service’ APIs
- Connect to IBM Cloud Object storage for data read and write
- Understand how the tools and methods might benefit org