notebooks/datasetsComparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b59d369",
   "metadata": {},
   "source": [
    "# Kazakh-Russian Code-Switching: dataset samples\n",
    "* For the term project, I explored two dataset samples to analyze how these resources would help to address the research question related to code-switching.\n",
    "* The preliminary EDA of two samples are given below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9eded878",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libs\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import glob\n",
    "import seaborn as sns\n",
    "import re\n",
    "from nltk.corpus import PlaintextCorpusReader "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc6810d",
   "metadata": {},
   "source": [
    "## Exploring dataset No.1: Kazakh Speech Corpus\n",
    "* Source: https://issai.nu.edu.kz/kz-speech-corpus/?version=1.1\n",
    "* Topics: Parliament and Podcasts\n",
    "\n",
    "#### Overview of code snippets:\n",
    "- Read .txt files from the KSC (Parliament and Podcasts folders) using nltk's PlaintextCorpusReader\n",
    "- Build dataframe from those files and add Source column\n",
    "- Merge dataframes\n",
    "- Simple stats: token counts, value counts by Source "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f67a609d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read all available .txt files with different topics and build a dataframe.\n",
    "# Files from Parliament - US Congress\n",
    "\n",
    "corpus_root = \"/Users/aidyn/Downloads/ISSAI_KSC2/Test/parliament\"\n",
    "ksc_mp = PlaintextCorpusReader(corpus_root, '.*txt')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "09fb14a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of files: 773\n",
      "# of words: 9980\n"
     ]
    }
   ],
   "source": [
    "print('# of files:', len(ksc_mp.fileids()))\n",
    "print('# of words:',len(ksc_mp.words()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ef1bec68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Text</th>\n",
       "      <th>Source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mazhilis_01_000.txt</td>\n",
       "      <td>қайырлы күн құрметті әріптестер құрметті депут...</td>\n",
       "      <td>Parliament</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mazhilis_01_001.txt</td>\n",
       "      <td>баршаңызды тіркеуден өтулеріңізді сұраймын</td>\n",
       "      <td>Parliament</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mazhilis_01_002.txt</td>\n",
       "      <td>тіркеу режимі қосылсын</td>\n",
       "      <td>Parliament</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mazhilis_01_003.txt</td>\n",
       "      <td>жұмысымызды бастау үшін қажетті кворум бар</td>\n",
       "      <td>Parliament</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mazhilis_01_004.txt</td>\n",
       "      <td>мәжілістің жалпы отырысын ашық деп жариялаймын</td>\n",
       "      <td>Parliament</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Filename                                               Text  \\\n",
       "0  mazhilis_01_000.txt  қайырлы күн құрметті әріптестер құрметті депут...   \n",
       "1  mazhilis_01_001.txt         баршаңызды тіркеуден өтулеріңізді сұраймын   \n",
       "2  mazhilis_01_002.txt                             тіркеу режимі қосылсын   \n",
       "3  mazhilis_01_003.txt         жұмысымызды бастау үшін қажетті кворум бар   \n",
       "4  mazhilis_01_004.txt     мәжілістің жалпы отырысын ашық деп жариялаймын   \n",
       "\n",
       "       Source  \n",
       "0  Parliament  \n",
       "1  Parliament  \n",
       "2  Parliament  \n",
       "3  Parliament  \n",
       "4  Parliament  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ksc_list = []\n",
    "for f in ksc_mp.fileids():\n",
    "    Filename = f\n",
    "    Text = ksc_mp.raw(f)\n",
    "    ksc_list.append((Filename, Text))\n",
    "# ets_list\n",
    "mp_df = pd.DataFrame(ksc_list)\n",
    "mp_df.rename(columns={0: \"Filename\", 1: \"Text\"}, inplace = True)\n",
    "mp_df['Source'] = 'Parliament'\n",
    "mp_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "664f6427",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Text</th>\n",
       "      <th>Source</th>\n",
       "      <th>Tokens</th>\n",
       "      <th>Token_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mazhilis_01_000.txt</td>\n",
       "      <td>қайырлы күн құрметті әріптестер құрметті депут...</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[қайырлы, күн, құрметті, әріптестер, құрметті,...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mazhilis_01_001.txt</td>\n",
       "      <td>баршаңызды тіркеуден өтулеріңізді сұраймын</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[баршаңызды, тіркеуден, өтулеріңізді, сұраймын]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mazhilis_01_002.txt</td>\n",
       "      <td>тіркеу режимі қосылсын</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[тіркеу, режимі, қосылсын]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mazhilis_01_003.txt</td>\n",
       "      <td>жұмысымызды бастау үшін қажетті кворум бар</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[жұмысымызды, бастау, үшін, қажетті, кворум, бар]</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mazhilis_01_004.txt</td>\n",
       "      <td>мәжілістің жалпы отырысын ашық деп жариялаймын</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[мәжілістің, жалпы, отырысын, ашық, деп, жария...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Filename                                               Text  \\\n",
       "0  mazhilis_01_000.txt  қайырлы күн құрметті әріптестер құрметті депут...   \n",
       "1  mazhilis_01_001.txt         баршаңызды тіркеуден өтулеріңізді сұраймын   \n",
       "2  mazhilis_01_002.txt                             тіркеу режимі қосылсын   \n",
       "3  mazhilis_01_003.txt         жұмысымызды бастау үшін қажетті кворум бар   \n",
       "4  mazhilis_01_004.txt     мәжілістің жалпы отырысын ашық деп жариялаймын   \n",
       "\n",
       "       Source                                             Tokens  Token_count  \n",
       "0  Parliament  [қайырлы, күн, құрметті, әріптестер, құрметті,...            6  \n",
       "1  Parliament    [баршаңызды, тіркеуден, өтулеріңізді, сұраймын]            4  \n",
       "2  Parliament                         [тіркеу, режимі, қосылсын]            3  \n",
       "3  Parliament  [жұмысымызды, бастау, үшін, қажетті, кворум, бар]            6  \n",
       "4  Parliament  [мәжілістің, жалпы, отырысын, ашық, деп, жария...            6  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add additional columns\n",
    "Tokens = mp_df.Text.map(nltk.word_tokenize)\n",
    "# tokenized essays\n",
    "mp_df['Tokens'] = Tokens\n",
    "# length of essays\n",
    "mp_df['Token_count'] = mp_df.Tokens.map(len)\n",
    "mp_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7cfa4988",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Text</th>\n",
       "      <th>Source</th>\n",
       "      <th>Tokens</th>\n",
       "      <th>Token_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>768</th>\n",
       "      <td>senate_01_397.txt</td>\n",
       "      <td>иностранной валюты</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[иностранной, валюты]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769</th>\n",
       "      <td>senate_01_398.txt</td>\n",
       "      <td>депутат перепечина</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[депутат, перепечина]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>770</th>\n",
       "      <td>senate_01_399.txt</td>\n",
       "      <td>спасибо құрметті әріптестер басқа сауалдар жоқ</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[спасибо, құрметті, әріптестер, басқа, сауалда...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>771</th>\n",
       "      <td>senate_01_400.txt</td>\n",
       "      <td>біз бүгін еліміздің қоғамдық саяси өмірін және...</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[біз, бүгін, еліміздің, қоғамдық, саяси, өмірі...</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>772</th>\n",
       "      <td>senate_01_401.txt</td>\n",
       "      <td>осымен сенаттың отырысын жабық деп жариялаймын...</td>\n",
       "      <td>Parliament</td>\n",
       "      <td>[осымен, сенаттың, отырысын, жабық, деп, жария...</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Filename                                               Text  \\\n",
       "768  senate_01_397.txt                                 иностранной валюты   \n",
       "769  senate_01_398.txt                                 депутат перепечина   \n",
       "770  senate_01_399.txt     спасибо құрметті әріптестер басқа сауалдар жоқ   \n",
       "771  senate_01_400.txt  біз бүгін еліміздің қоғамдық саяси өмірін және...   \n",
       "772  senate_01_401.txt  осымен сенаттың отырысын жабық деп жариялаймын...   \n",
       "\n",
       "         Source                                             Tokens  \\\n",
       "768  Parliament                              [иностранной, валюты]   \n",
       "769  Parliament                              [депутат, перепечина]   \n",
       "770  Parliament  [спасибо, құрметті, әріптестер, басқа, сауалда...   \n",
       "771  Parliament  [біз, бүгін, еліміздің, қоғамдық, саяси, өмірі...   \n",
       "772  Parliament  [осымен, сенаттың, отырысын, жабық, деп, жария...   \n",
       "\n",
       "     Token_count  \n",
       "768            2  \n",
       "769            2  \n",
       "770            6  \n",
       "771           15  \n",
       "772            8  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mp_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e719b69e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12.910737386804657"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mp_df.Token_count.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cad21159",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_root = \"/Users/aidyn/Downloads/ISSAI_KSC2/Test/podcasts\"\n",
    "ksc_pc = PlaintextCorpusReader(corpus_root, '.*txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ca99fa6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of files: 1547\n",
      "# of words: 21870\n"
     ]
    }
   ],
   "source": [
    "print('# of files:',len(ksc_pc.fileids()))\n",
    "print('# of words:',len(ksc_pc.words()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6909948f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Text</th>\n",
       "      <th>Source</th>\n",
       "      <th>Tokens</th>\n",
       "      <th>Token_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>09_00_000.txt</td>\n",
       "      <td>қайырлы күн құрметті не әңгіме подкастын тыңда...</td>\n",
       "      <td>Podcast</td>\n",
       "      <td>[қайырлы, күн, құрметті, не, әңгіме, подкастын...</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>09_00_001.txt</td>\n",
       "      <td>бүгін сіздермен тағы да қауышқалы отырмыз</td>\n",
       "      <td>Podcast</td>\n",
       "      <td>[бүгін, сіздермен, тағы, да, қауышқалы, отырмыз]</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>09_00_002.txt</td>\n",
       "      <td>ыыы не әңгіме подкастының бүгінгі қонағы ммм н...</td>\n",
       "      <td>Podcast</td>\n",
       "      <td>[ыыы, не, әңгіме, подкастының, бүгінгі, қонағы...</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>09_00_003.txt</td>\n",
       "      <td>жәке қош келдіңіз</td>\n",
       "      <td>Podcast</td>\n",
       "      <td>[жәке, қош, келдіңіз]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>09_00_005.txt</td>\n",
       "      <td>бүгін ыыы біз сізбен ммм еліміздегі ііі дін мә...</td>\n",
       "      <td>Podcast</td>\n",
       "      <td>[бүгін, ыыы, біз, сізбен, ммм, еліміздегі, ііі...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Filename                                               Text   Source  \\\n",
       "0  09_00_000.txt  қайырлы күн құрметті не әңгіме подкастын тыңда...  Podcast   \n",
       "1  09_00_001.txt          бүгін сіздермен тағы да қауышқалы отырмыз  Podcast   \n",
       "2  09_00_002.txt  ыыы не әңгіме подкастының бүгінгі қонағы ммм н...  Podcast   \n",
       "3  09_00_003.txt                                  жәке қош келдіңіз  Podcast   \n",
       "4  09_00_005.txt  бүгін ыыы біз сізбен ммм еліміздегі ііі дін мә...  Podcast   \n",
       "\n",
       "                                              Tokens  Token_count  \n",
       "0  [қайырлы, күн, құрметті, не, әңгіме, подкастын...            7  \n",
       "1   [бүгін, сіздермен, тағы, да, қауышқалы, отырмыз]            6  \n",
       "2  [ыыы, не, әңгіме, подкастының, бүгінгі, қонағы...           23  \n",
       "3                              [жәке, қош, келдіңіз]            3  \n",
       "4  [бүгін, ыыы, біз, сізбен, ммм, еліміздегі, ііі...           20  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pc_list = []\n",
    "for f in ksc_pc.fileids():\n",
    "    Filename = f\n",
    "    Text = ksc_pc.raw(f)\n",
    "    pc_list.append((Filename, Text))\n",
    "# ets_list\n",
    "pc_df = pd.DataFrame(pc_list)\n",
    "pc_df.rename(columns={0: \"Filename\", 1: \"Text\"}, inplace = True)\n",
    "pc_df['Source'] = 'Podcast'\n",
    "\n",
    "# Add additional columns\n",
    "Tokens = pc_df.Text.map(nltk.word_tokenize)\n",
    "# tokenized essays\n",
    "pc_df['Tokens'] = Tokens\n",
    "# length of essays\n",
    "pc_df['Token_count'] = pc_df.Tokens.map(len)\n",
    "pc_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fdccad02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.137039431157078"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pc_df.Token_count.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d763d1",
   "metadata": {},
   "source": [
    "### Merging dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9bbc2f27",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2320"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frames = [mp_df, pc_df]\n",
    "ksc_1 = pd.concat(frames)\n",
    "len(ksc_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b8225b8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Source\n",
       "Parliament     9980\n",
       "Podcast       21870\n",
       "Name: Token_count, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ksc_1.groupby(\"Source\")[\"Token_count\"].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cd65d3fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Podcast       1547\n",
       "Parliament     773\n",
       "Name: Source, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ksc_1.Source.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "aa731db2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Source</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Parliament</th>\n",
       "      <td>773.0</td>\n",
       "      <td>12.910737</td>\n",
       "      <td>7.436157</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Podcast</th>\n",
       "      <td>1547.0</td>\n",
       "      <td>14.137039</td>\n",
       "      <td>6.642164</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             count       mean       std  min  25%   50%   75%   max\n",
       "Source                                                             \n",
       "Parliament   773.0  12.910737  7.436157  2.0  6.0  12.0  18.0  35.0\n",
       "Podcast     1547.0  14.137039  6.642164  2.0  9.0  13.0  19.0  34.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ksc_1.groupby(\"Source\")[\"Token_count\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "eadeb5aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       [қайырлы, күн, құрметті, әріптестер, құрметті,...\n",
       "1         [баршаңызды, тіркеуден, өтулеріңізді, сұраймын]\n",
       "2                              [тіркеу, режимі, қосылсын]\n",
       "3       [жұмысымызды, бастау, үшін, қажетті, кворум, бар]\n",
       "4       [мәжілістің, жалпы, отырысын, ашық, деп, жария...\n",
       "                              ...                        \n",
       "1542       [бәрекелді, олай, болса, келгеніңізге, рахмет]\n",
       "1543                  [ы, пікір, білдіргеніңізге, рахмет]\n",
       "1544    [ы, қазақ, айтады, адасқанның, айыбы, жоқ, қай...\n",
       "1545    [сондықтан, адасқан, бауырымыз, бауырларымыз, ...\n",
       "1546    [жаңағы, келгендеріне, қуанып, жатыр, сол, қуа...\n",
       "Name: Tokens, Length: 2320, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ksc_1.Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4c78d969",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7fbc96fe9190>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAHqCAYAAADLbQ06AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoaElEQVR4nO3deXRUZZ7/8U9BQiWQhUUgxElYJiBhlU0FBIIEkMGwiMp2EBAcVoNEBSMiKEKAbmgycMQmjiRKg9GxTTOOC3SaVdGGYMAWBgRiJ4NAEDGBYBdL7u8PflRbJEAKa3kg79c5dZq6devWN5wu3zy3KlU2y7IsAQAAI1Xx9wAAAODaCDUAAAYj1AAAGIxQAwBgMEINAIDBCDUAAAYj1AAAGIxQAwBgsNs+1JZlqbi4WHyuCwDgVnTbh/rMmTMKDw/XmTNn/D0KAABuu+1DDQDArYxQAwBgMEINAIDBCDUAAAYj1AAAGIxQAwBgMEINAIDBCDUAAAYj1AAAGIxQAwBgMEINAIDBCDUAAAYj1AAAGIxQAwBgMEINAIDBCDUAAAYj1AAAGCzA3wMAADxj2rRpOnnypCSpbt26Sk1N9fNE8ARCDQC3iZMnT+rEiRP+HgMexqlvAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAM5tdQb926VQkJCYqMjJTNZlNWVtY1950wYYJsNpuWLVvms/kAAPA3v36EaElJidq2bauxY8dqyJAh19wvKytLX375pSIjI304HQBfyX+ltb9HuC1c/KmOpKr//8/f8/fqQdEvfe23x/ZrqPv166d+/fpdd5+jR49q6tSp+vTTT9W/f38fTQYAgBmMfo26tLRUo0aN0nPPPaeWLVv6exwAAHzO6G/PWrRokQICApSYmFjh+zgcDjkcDuf14uJib4wGAIBPGLuizsnJUWpqqtLT02Wz2Sp8v5SUFIWHhzsvUVFRXpwSAADvMjbU27ZtU2FhoaKjoxUQEKCAgAD9/e9/1zPPPKNGjRpd837JyckqKipyXgoKCnw3NAAAHmbsqe9Ro0YpPj7eZVvfvn01atQojR079pr3s9vtstvt3h4PAACf8Guoz549q0OHDjmv5+XlKTc3V7Vr11Z0dLTq1Knjsn9gYKAiIiJ01113+XpUAAD8wq+h3rVrl3r27Om8npSUJEkaPXq00tPT/TQVAADm8Guo4+LiZFlWhff/7rvvvDcMANziatsvlftn3NqMfY0aAOCeF9r95O8R4AXGvusbAAAQagAAjEaoAQAwGKEGAMBghBoAAIMRagAADMavZ8Hvpk2bppMnT0qS6tatq9TUVD9PBADmINTwu5MnT+rEiRP+HgMAjMSpbwAADEaoAQAwGKEGAMBghBoAAIMRagAADEaoAQAwGKEGAMBghBoAAIMRagAADEaoAQAwGKEGAMBghBoAAIMRagAADEaoAQAwGKEGAMBgfB/1Terw3Fv+HuG2EXb6rPNfjMdOn+Xv1kNyfvO4v0cA4AGsqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACD8X3U8LvSwBrl/hkAQKhhgLN39fP3CABgLE59AwBgMEINAIDBCDUAAAbza6i3bt2qhIQERUZGymazKSsry3nbhQsXNHPmTLVu3Vo1atRQZGSkHn/8cX3//ff+GxgAAB/za6hLSkrUtm1brVixosxt586d0+7duzV79mzt3r1bf/zjH3Xw4EENGDDAD5MCAOAffn3Xd79+/dSvX/nv+A0PD9fGjRtdti1fvlz33HOP8vPzFR0d7YsRAQDwq1vqNeqioiLZbDbVrFnT36MAAOATt8zvUf/jH//Q888/rxEjRigsLOya+zkcDjkcDuf14uJiX4wHAIBX3BIr6gsXLmjYsGEqLS3Va6+9dt19U1JSFB4e7rxERUX5aEoAADzP+FBfuHBBjz32mPLy8rRx48brrqYlKTk5WUVFRc5LQUGBjyYFAMDzjD71fSXS3377rTZt2qQ6derc8D52u112u90H0wEA4H1+DfXZs2d16NAh5/W8vDzl5uaqdu3aioyM1COPPKLdu3frww8/1KVLl3T8+HFJUu3atVWtWjV/jQ0AgM/4NdS7du1Sz549ndeTkpIkSaNHj9bcuXO1fv16SdLdd9/tcr9NmzYpLi7OV2MCAOA3fg11XFycLMu65u3Xuw0AgMrA+DeTAQBQmRFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAg/k11Fu3blVCQoIiIyNls9mUlZXlcrtlWZo7d64iIyMVHBysuLg4ffPNN/4ZFgAAP/BrqEtKStS2bVutWLGi3NsXL16spUuXasWKFdq5c6ciIiLUu3dvnTlzxseTAgDgHwH+fPB+/fqpX79+5d5mWZaWLVumWbNm6eGHH5YkZWRkqH79+lq7dq0mTJjgy1EBAPALY1+jzsvL0/Hjx9WnTx/nNrvdrh49eujzzz/342QAAPiOX1fU13P8+HFJUv369V22169fX3//+9+veT+HwyGHw+G8Xlxc7J0BAQDwAWNX1FfYbDaX65Zlldn2SykpKQoPD3deoqKivD0iAABeY2yoIyIiJP1zZX1FYWFhmVX2LyUnJ6uoqMh5KSgo8OqcAAB4k7Ghbty4sSIiIrRx40bntvPnz2vLli3q0qXLNe9nt9sVFhbmcgEA4Fbl19eoz549q0OHDjmv5+XlKTc3V7Vr11Z0dLSefvppLViwQE2bNlXTpk21YMECVa9eXSNGjPDj1AAA+I5fQ71r1y717NnTeT0pKUmSNHr0aKWnp2vGjBn6+eefNXnyZJ0+fVr33nuvNmzYoNDQUH+NDACAT/k11HFxcbIs65q322w2zZ07V3PnzvXdUAAAGMTY16gBAAChBgDAaIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDuR3qt956Sw6Ho8z28+fP66233vLIUAAA4DK3Qz127FgVFRWV2X7mzBmNHTvWI0MBAIDL3A61ZVmy2Wxltv/f//2fwsPDPTIUAAC4LKCiO7Zr1042m002m029evVSQMA/73rp0iXl5eXpwQcf9MqQAABUVhUO9aBBgyRJubm56tu3r0JCQpy3VatWTY0aNdKQIUM8PiAAAJVZhUM9Z84cSVKjRo00dOhQBQUFeW0oAABwWYVDfcXo0aMlXX6Xd2FhoUpLS11uj46O9sxkAADA/VB/++23euKJJ/T555+7bL/yJrNLly55bDgAACo7t0M9ZswYBQQE6MMPP1SDBg3KfQc4AADwDLdDnZubq5ycHDVv3twb8wAAgF9w+/eoW7RooR9++MEbswAAgKu4HepFixZpxowZ2rx5s06dOqXi4mKXCwAA8By3T33Hx8dLknr16uWynTeTAQDgeW6HetOmTd6YAwAAlMPtUPfo0cMbcwAAgHK4HeqtW7de9/bu3bvf9DAAAMCV26GOi4srs+2Xv0vNa9QAAHiO2+/6Pn36tMulsLBQn3zyiTp16qQNGzZ4Y0YAACott1fU5X3ndO/evWW32zV9+nTl5OR4ZDAAAHATK+prqVu3rg4cOOCpwwEAAN3Einrv3r0u1y3L0rFjx7Rw4UK1bdvWY4MBAICbCPXdd98tm80my7Jctt9333168803PTYYAAC4iVDn5eW5XK9SpYrq1q2roKAgjw0FAAAuczvUDRs29MYcAACgHDf1ZrItW7YoISFBMTExatq0qQYMGKBt27Z5ejYAACo9t0O9Zs0axcfHq3r16kpMTNTUqVMVHBysXr16ae3atd6YEQCASsvtU9/z58/X4sWLNX36dOe2adOmaenSpZo3b55GjBjh0QEBAKjM3F5RHzlyRAkJCWW2DxgwoMwbzTzh4sWLevHFF9W4cWMFBwerSZMmeuWVV1RaWurxxwIAwDRur6ijoqKUnZ2tmJgYl+3Z2dmKiory2GBXLFq0SK+//royMjLUsmVL7dq1S2PHjlV4eLimTZvm8ccDAMAkbof6mWeeUWJionJzc9WlSxfZbDZt375d6enpSk1N9fiAO3bs0MCBA9W/f39JUqNGjbRu3Trt2rXL448FAIBp3A71pEmTFBERoSVLlujdd9+VJMXGxiozM1MDBw70+ID333+/Xn/9dR08eFDNmjXTnj17tH37di1btqzc/R0OhxwOh/N6cXGxx2cCAMBX3A61JA0ePFiDBw/29CzlmjlzpoqKitS8eXNVrVpVly5d0vz58zV8+PBy909JSdHLL7/sk9kAAPA2t99MtnPnTn355Zdltn/55ZdeOR2dmZmpNWvWaO3atdq9e7cyMjL029/+VhkZGeXun5ycrKKiIueloKDA4zMBAOArbod6ypQp5cbv6NGjmjJlikeG+qXnnntOzz//vIYNG6bWrVtr1KhRmj59ulJSUsrd3263KywszOUCAMCtyu1Q79u3T+3bty+zvV27dtq3b59Hhvqlc+fOqUoV1zGrVq3Kr2cBACoFt1+jttvtOnHihJo0aeKy/dixYwoIuKmXvK8rISFB8+fPV3R0tFq2bKmvvvpKS5cu1RNPPOHxxwIAwDRur6h79+7tfB34ip9++kkvvPCCevfu7dHhJGn58uV65JFHNHnyZMXGxurZZ5/VhAkTNG/ePI8/FgAApnF7CbxkyRJ1795dDRs2VLt27SRJubm5ql+/vt5++22PDxgaGqply5Zd89exAAC4nbkd6jvvvFN79+7VH/7wB+3Zs0fBwcEaO3ashg8frsDAQG/MCABApXVTLyrXqFFD//7v/37dffr376833nhDDRo0uKnBAADATX4fdUVs3bpVP//8s7cODwBApeC1UAMAgF+PUAMAYDBCDQCAwQg1AAAGI9QAABjMa6F+4YUXVLt2bW8dHgCASuGmfo/64MGD2rx5swoLC8t8OcZLL70k6fLXTQIAgF/H7VCnpaVp0qRJuuOOOxQRESGbzea8zWazOUMNAAB+PbdD/eqrr2r+/PmaOXOmN+YBAAC/4PZr1KdPn9ajjz7qjVkAAMBV3A71o48+qg0bNnhjFgAAcBW3T33HxMRo9uzZ+uKLL9S6desy35iVmJjoseEAAKjs3A71qlWrFBISoi1btmjLli0ut9lsNkINAIAHuR3qvLw8b8wBAADKcdMfeHL+/HkdOHBAFy9e9OQ8AADgF9wO9blz5zRu3DhVr15dLVu2VH5+vqTLr00vXLjQ4wMCAFCZuR3q5ORk7dmzR5s3b1ZQUJBze3x8vDIzMz06HAAAlZ3br1FnZWUpMzNT9913n8unkrVo0UKHDx/26HAAAFR2bq+oT548qXr16pXZXlJS4hJuAADw67kd6k6dOul//ud/nNevxDktLU2dO3f23GQAAMD9U98pKSl68MEHtW/fPl28eFGpqan65ptvtGPHjjK/Vw0AAH4dt1fUXbp00WeffaZz587pX//1X7VhwwbVr19fO3bsUIcOHbwxIwAAlZbbK+q9e/eqTZs2ysjIKHNbVlaWBg0a5Im5AACAbmJF3bdvXx05cqTM9vfff18jR470yFAAAOAyt0M9adIk9erVS8eOHXNuy8zM1OOPP6709HRPzgYAQKXn9qnvl156SadOnVJ8fLy2bdumTz75ROPHj9fbb7+tIUOGeGNGAAAqLbdDLUmpqakaNWqU7rvvPh09elTr1q3TwIEDPT0bAACVXoVCvX79+jLbBg0apC1btmj48OGy2WzOfQYMGODZCQEAqMQqFOrrvZP7zTff1Jtvvinp8oefXLp0ySODAQCACoa6tLTU23MAAIBy3PT3UQMAAO+7qVBv2bJFCQkJiomJUdOmTTVgwABt27bN07MBAFDpuR3qNWvWKD4+XtWrV1diYqKmTp2q4OBg9erVS2vXrvXGjAAAVFpu/3rW/PnztXjxYk2fPt25bdq0aVq6dKnmzZunESNGeHRAAAAqM7dX1EeOHFFCQkKZ7QMGDFBeXp5HhgIAAJe5HeqoqChlZ2eX2Z6dna2oqCiPDAUAAC6r8KnvJ554QqmpqXrmmWeUmJio3NxcdenSRTabTdu3b1d6erpSU1O9OSsAAJVOhUOdkZGhhQsXatKkSYqIiNCSJUv07rvvSpJiY2OVmZnJx4gCAOBhFQ61ZVnOPw8ePFiDBw/2ykAAAOCf3HqN2mazeWsOAABQDrd+PatZs2Y3jPWPP/74qwYCAAD/5FaoX375ZYWHh3trFgAAcBW3Qj1s2DDVq1fPW7Nc09GjRzVz5kx9/PHH+vnnn9WsWTP953/+pzp06ODzWQAA8KUKh9pfr0+fPn1aXbt2Vc+ePfXxxx+rXr16Onz4sGrWrOmXeQAA8KWbete3Ly1atEhRUVFavXq1c1ujRo38MgsAAL5W4Xd9l5aW+uW09/r169WxY0c9+uijqlevntq1a6e0tLRr7u9wOFRcXOxyAQDgVmX891EfOXJEK1euVNOmTfXpp59q4sSJSkxM1FtvvVXu/ikpKQoPD3de+FhTAMCtzPhQl5aWqn379lqwYIHatWunCRMm6Mknn9TKlSvL3T85OVlFRUXOS0FBgY8nBgDAc4wPdYMGDdSiRQuXbbGxscrPzy93f7vdrrCwMJcLAAC3KuND3bVrVx04cMBl28GDB9WwYUM/TQQAgO8YH+rp06friy++0IIFC3To0CGtXbtWq1at0pQpU/w9GgAAXmd8qDt16qQPPvhA69atU6tWrTRv3jwtW7ZMI0eO9PdoAAB4nVufTOYvDz30kB566CF/jwEAgM8Zv6IGAKAyI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGI9QAABiMUAMAYDBCDQCAwQg1AAAGu6VCnZKSIpvNpqefftrfowAA4BO3TKh37typVatWqU2bNv4eBQAAn7klQn327FmNHDlSaWlpqlWrlr/HAQDAZ26JUE+ZMkX9+/dXfHz8Dfd1OBwqLi52uQAAcKsK8PcAN/LOO+9o9+7d2rlzZ4X2T0lJ0csvv+zlqQAA8A2jV9QFBQWaNm2a1qxZo6CgoArdJzk5WUVFRc5LQUGBl6cEAMB7jF5R5+TkqLCwUB06dHBuu3TpkrZu3aoVK1bI4XCoatWqLvex2+2y2+2+HhUAAK8wOtS9evXS119/7bJt7Nixat68uWbOnFkm0gAA3G6MDnVoaKhatWrlsq1GjRqqU6dOme0AANyOjH6NGgCAys7oFXV5Nm/e7O8RAADwGVbUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYjFADAGAwQg0AgMEINQAABiPUAAAYzPhQp6SkqFOnTgoNDVW9evU0aNAgHThwwN9jAQDgE8aHesuWLZoyZYq++OILbdy4URcvXlSfPn1UUlLi79EAAPC6AH8PcCOffPKJy/XVq1erXr16ysnJUffu3f00FQAAvmH8ivpqRUVFkqTatWv7eRIAALzP+BX1L1mWpaSkJN1///1q1apVufs4HA45HA7n9eLiYl+NBwCAx91SK+qpU6dq7969Wrdu3TX3SUlJUXh4uPMSFRXlwwkBAPCsWybUTz31lNavX69NmzbpX/7lX665X3JysoqKipyXgoICH04JAIBnGX/q27IsPfXUU/rggw+0efNmNW7c+Lr72+122e12H00HAIB3GR/qKVOmaO3atfrTn/6k0NBQHT9+XJIUHh6u4OBgP08HAIB3GX/qe+XKlSoqKlJcXJwaNGjgvGRmZvp7NAAAvM74FbVlWf4eAQAAvzF+RQ0AQGVGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAxGqAEAMBihBgDAYIQaAACDEWoAAAx2S4T6tddeU+PGjRUUFKQOHTpo27Zt/h4JAACfMD7UmZmZevrppzVr1ix99dVX6tatm/r166f8/Hx/jwYAgNcZH+qlS5dq3LhxGj9+vGJjY7Vs2TJFRUVp5cqV/h4NAACvMzrU58+fV05Ojvr06eOyvU+fPvr888/9NBUAAL4T4O8BrueHH37QpUuXVL9+fZft9evX1/Hjx8u9j8PhkMPhcF4vKiqSJBUXF3t0tkuOnz16PMDTPP3/eW86849L/h4BuC5vPZ9CQ0Nls9muu4/Rob7i6h/Csqxr/mApKSl6+eWXy2yPioryymyAqcKXT/T3CMDtIyXcK4ctKipSWFjYdfcxOtR33HGHqlatWmb1XFhYWGaVfUVycrKSkpKc10tLS/Xjjz+qTp06N/xXC/ynuLhYUVFRKigouOH/aQFcG8+lW0toaOgN9zE61NWqVVOHDh20ceNGDR482Ll948aNGjhwYLn3sdvtstvtLttq1qzpzTHhQWFhYfzHBfAAnku3D6NDLUlJSUkaNWqUOnbsqM6dO2vVqlXKz8/XxImc1gMA3P6MD/XQoUN16tQpvfLKKzp27JhatWqljz76SA0bNvT3aAAAeJ3xoZakyZMna/Lkyf4eA15kt9s1Z86cMi9bAHAPz6Xbj82yLMvfQwAAgPIZ/YEnAABUdoQaAACDEWp41Ny5c3X33Xc7r48ZM0aDBg3y2zzA7W7z5s2y2Wz66aef/D0KvIRQV2JjxoyRzWaTzWZTYGCgmjRpomeffVYlJSUee4zU1FSlp6d77HjelJ6ezu/cw+t88bzzJpvNpqysLH+PUancEu/6hvc8+OCDWr16tS5cuKBt27Zp/PjxKikpcfvbySzL0qVLZT+vOTzcOx+7B9zKPPW8Q+XAirqSs9vtioiIUFRUlEaMGKGRI0cqKytLa9asUceOHRUaGqqIiAiNGDFChYWFzvtdOd326aefqmPHjrLb7dq2bVuZ41996vuTTz7R/fffr5o1a6pOnTp66KGHdPjwYeft3333nWw2m959911169ZNwcHB6tSpkw4ePKidO3eqY8eOCgkJ0YMPPqiTJ0+6PNbq1asVGxuroKAgNW/eXK+99lqZ4/7xj39Uz549Vb16dbVt21Y7duxw/jxjx45VUVGRc7Uzd+5cD/0tA66u9bxzOBxKTExUvXr1FBQUpPvvv187d+50ue9HH32kZs2aKTg4WD179tR3331X5vifffaZevTooerVq6tWrVrq27evTp8+LenGz8Hz589r6tSpatCggYKCgtSoUSOlpKRIkho1aiRJGjx4sGw2m/M6vItQw0VwcLAuXLig8+fPa968edqzZ4+ysrKUl5enMWPGlNl/xowZSklJ0f79+9WmTZsbHr+kpERJSUnauXOnsrOzVaVKFQ0ePFilpaUu+82ZM0cvvviidu/erYCAAA0fPlwzZsxQamqqtm3bpsOHD+ull15y7p+WlqZZs2Zp/vz52r9/vxYsWKDZs2crIyPD5bizZs3Ss88+q9zcXDVr1kzDhw/XxYsX1aVLFy1btkxhYWE6duyYjh07pmefffbm/hIBN1153s2YMUPvv/++MjIytHv3bsXExKhv37768ccfJUkFBQV6+OGH9W//9m/Kzc3V+PHj9fzzz7scKzc3V7169VLLli21Y8cObd++XQkJCc4zXjd6Dv7Hf/yH1q9fr3fffVcHDhzQmjVrnEG+8o+G1atX69ixY2X+EQEvsVBpjR492ho4cKDz+pdffmnVqVPHeuyxx8rs+9e//tWSZJ05c8ayLMvatGmTJcnKyspy2W/OnDlW27Ztr/kYVyssLLQkWV9//bVlWZaVl5dnSbLeeOMN5z7r1q2zJFnZ2dnObSkpKdZdd93lvB4VFWWtXbvW5djz5s2zOnfufM3jfvPNN5Yka//+/ZZlWdbq1aut8PDwa84KeMK1nnePPPKIFRgYaP3hD39w3nb+/HkrMjLSWrx4sWVZlpWcnGzFxsZapaWlzn1mzpxpSbJOnz5tWZZlDR8+3OratWuF57n6OfjUU09ZDzzwgMtj/JIk64MPPqjw8fHrsaKu5D788EOFhIQoKChInTt3Vvfu3bV8+XJ99dVXGjhwoBo2bKjQ0FDFxcVJkvLz813u37FjR7ce7/DhwxoxYoSaNGmisLAwNW7cuNzj/nJ1fuWb0lq3bu2y7cqp+JMnT6qgoEDjxo1TSEiI8/Lqq6+6nNK7+rgNGjSQJJdT+oAvlPe8e+qpp3ThwgV17drVuV9gYKDuuece7d+/X5K0f/9+3XfffS7fBNi5c2eXY19ZUV/LjZ6DY8aMUW5uru666y4lJiZqw4YNHvu5cXN4M1kl17NnT61cuVKBgYGKjIxUYGCgSkpK1KdPH/Xp00dr1qxR3bp1lZ+fr759++r8+fMu969Ro4Zbj5eQkKCoqCilpaUpMjJSpaWlatWqVZnjBgYGOv985T9KV2+7cqruyv+mpaXp3nvvdTlO1apVb3jcq0+7A95W3vNuz549klTm63gty3JusyrwQZLBwcHXvf1Gz8H27dsrLy9PH3/8sf785z/rscceU3x8vP7rv/7rZn5UeAAr6kquRo0aiomJUcOGDZ0R+9///V/98MMPWrhwobp166bmzZt7ZNV56tQp7d+/Xy+++KJ69eql2NhY5xtcfo369evrzjvv1JEjRxQTE+NyubJaqIhq1aqV+851wNPKe97FxMSoWrVq2r59u3O/CxcuaNeuXYqNjZUktWjRQl988YXLsa6+3qZNG2VnZ5f7uBV9DoaFhWno0KFKS0tTZmam3n//fefr5IGBgTxPfIwVNcqIjo5WtWrVtHz5ck2cOFF/+9vfNG/evF993Fq1aqlOnTpatWqVGjRooPz8/DJvhLlZc+fOVWJiosLCwtSvXz85HA7t2rVLp0+fVlJSUoWO0ahRI509e1bZ2dlq27atqlevrurVq3tkPuBGatSooUmTJum5555T7dq1FR0drcWLF+vcuXMaN26cJGnixIlasmSJkpKSNGHCBOXk5JT5nILk5GS1bt1akydP1sSJE1WtWjVt2rRJjz76qGrXrn3D5+Dvfvc7NWjQQHfffbeqVKmi9957TxEREc7PGGjUqJGys7PVtWtX2e121apVyxd/PZUaK2qUUbduXaWnp+u9995TixYttHDhQv32t7/91cetUqWK3nnnHeXk5KhVq1aaPn26fvOb33hgYmn8+PF64403lJ6ertatW6tHjx5KT093a0XdpUsXTZw4UUOHDlXdunW1ePFij8wGVNTChQs1ZMgQjRo1Su3bt9ehQ4f06aefOmMYHR2t999/X//93/+ttm3b6vXXX9eCBQtcjtGsWTNt2LBBe/bs0T333KPOnTvrT3/6kwICAir0HAwJCdGiRYvUsWNHderUSd99950++ugjValyORdLlizRxo0bFRUVpXbt2vnmL6aS49uzAAAwGCtqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqiB21RhYaEmTJig6Oho2e12RUREqG/fvtqxY4e/RwPgBr6UA7hNDRkyRBcuXFBGRoaaNGmiEydOKDs72/ktSN5w/vx5VatWzWvHByojVtTAbeinn37S9u3btWjRIvXs2VMNGzbUPffco+TkZPXv31+SlJ+fr4EDByokJERhYWF67LHHdOLECecxxowZo0GDBrkc9+mnn1ZcXJzzelxcnKZOnaqkpCTdcccd6t27tyTpm2++Uf/+/RUWFqbQ0FB169ZNhw8fdt5v9erVio2NVVBQkJo3b67XXnvNe38ZwC2OUAO3oZCQEIWEhCgrK0sOh6PM7ZZladCgQfrxxx+1ZcsWbdy4UYcPH9bQoUPdfqyMjAwFBATos88+0+9//3sdPXpU3bt3V1BQkP7yl78oJydHTzzxhC5evChJSktL06xZszR//nzt379fCxYs0OzZs5WRkfGrf27gdsSpb+A2FBAQoPT0dD355JN6/fXX1b59e/Xo0UPDhg1TmzZt9Oc//1l79+5VXl6eoqKiJElvv/22WrZsqZ07d6pTp04VfqyYmBiXrwR94YUXFB4ernfeeUeBgYGSLn/14hXz5s3TkiVL9PDDD0uSGjdurH379un3v/+9Ro8e7YkfH7itsKIGblNDhgzR999/r/Xr16tv377avHmz2rdvr/T0dO3fv19RUVHOSEtSixYtVLNmTe3fv9+tx+nYsaPL9dzcXHXr1s0Z6V86efKkCgoKNG7cOOeqPyQkRK+++qrLqXEA/8SKGriNBQUFqXfv3urdu7deeukljR8/XnPmzFFSUpJsNluZ/S3Lcm6vUqWKrv66+gsXLpS5T40aNVyuBwcHX3Oe0tJSSZdPf997770ut1WtWrViPxRQybCiBiqRFi1aqKSkRC1atFB+fr4KCgqct+3bt09FRUWKjY2VJNWtW1fHjh1zuX9ubu4NH6NNmzbatm1buVGvX7++7rzzTh05ckQxMTEul8aNG/+6Hw64TRFq4DZ06tQpPfDAA1qzZo3ztej33ntPixcv1sCBAxUfH682bdpo5MiR2r17t/7617/q8ccfV48ePZynsh944AHt2rVLb731lr799lvNmTNHf/vb32742FOnTlVxcbGGDRumXbt26dtvv9Xbb7+tAwcOSJLmzp2rlJQUpaam6uDBg/r666+1evVqLV261Kt/J8CtilADt6GQkBDde++9+t3vfqfu3burVatWmj17tp588kmtWLFCNptNWVlZqlWrlrp37674+Hg1adJEmZmZzmP07dtXs2fP1owZM9SpUyedOXNGjz/++A0fu06dOvrLX/6is2fPqkePHurQoYPS0tKcr1mPHz9eb7zxhtLT09W6dWv16NFD6enprKiBa7BZV78IBQAAjMGKGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGCEGgAAgxFqAAAMRqgBADAYoQYAwGD/D/B7emynn5NEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(data = ksc_1, x = 'Source', y = 'Token_count', kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6bacf05",
   "metadata": {},
   "source": [
    "## Summary of KSC dataset\n",
    "\n",
    "- **Advantages**: \n",
    "    - Publicly available for research and can be redistributed under *Creative Commons Attribution 4.0 International License*. \n",
    "    - Utterances related to multiple topics such as Government, Podcast, Talkshow, Radio, TV_news;\n",
    "    - Relatively easy to compile all .txt files into the dataframe;\n",
    "\n",
    "- **Disadvantages**:\n",
    "    - Utterances are too short and not connected to each other even though it is spoken data;\n",
    "    - Samples were manually selected by the researchers from books, news outlets, etc. to ensure the high quality of utterances. For more details: https://doi.org/10.48550/arXiv.2009.10334\n",
    "    - Authors cleaned out the Russian sentences or purely Russian words for the target Kazakh ASR purposes;\n",
    "    - The downloaded file lacks metadata: age, sex, region, SES of speakers. The dataset description says  it has that data but I was able to download the dataset only partially (10 GB out of 75 GB).\n",
    "    - After manually screening two folders with .txt files, I found only 26 utterances with Russian borrowings or morphologically adapted words;\n",
    "    - So, there is no clear evidence for Kazakh-Russian code-switching within the conversation or sentence. Just having Russian borrowings in Kazakh sentences might be hard to count as pure code-switching.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7027f9b",
   "metadata": {},
   "source": [
    "******"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53c84e4",
   "metadata": {},
   "source": [
    "## Exploring dataset No.2: IARPA Babel Kazakh Language Pack\n",
    "- Source: https://catalog.ldc.upenn.edu/LDC2018S13\n",
    "- Brief description: It contains approximately 203 hours of Kazakh conversational and scripted telephone speech collected in 2013 and 2014 along with corresponding transcripts.\n",
    "\n",
    "#### Overview of code snippets:\n",
    "- Read dev and sub-train folders with demographics\n",
    "- Read .txt files (conversation transcriptions) from the same folders \n",
    "- Reorganize filenames to merge metadata and txt files\n",
    "- Merge metadata and txt files\n",
    "- Clean Speech_raw column since it contains timestamps, remarks in English and other unnecessary tokens\n",
    "- Add Sents and Tokens columns\n",
    "- The inal dataframe contains 400 conversations with metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "95666f24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>outputFn</th>\n",
       "      <th>sessID</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>spkrCode</th>\n",
       "      <th>lineType</th>\n",
       "      <th>dialect</th>\n",
       "      <th>gen</th>\n",
       "      <th>envType</th>\n",
       "      <th>age</th>\n",
       "      <th>network</th>\n",
       "      <th>phoneModel</th>\n",
       "      <th>sampleCount</th>\n",
       "      <th>sampleRate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BABEL_OP2_302_10002_20140316_215637_outLine.sph</td>\n",
       "      <td>10002</td>\n",
       "      <td>20140316</td>\n",
       "      <td>215637</td>\n",
       "      <td>10002</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>42</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4823040</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BABEL_OP2_302_10002_20140316_215637_inLine.sph</td>\n",
       "      <td>10002</td>\n",
       "      <td>20140316</td>\n",
       "      <td>215637</td>\n",
       "      <td>10002</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>PUBLIC_PLACE</td>\n",
       "      <td>36</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800960</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BABEL_OP2_302_10188_20131030_194100_outLine.sph</td>\n",
       "      <td>10188</td>\n",
       "      <td>20131030</td>\n",
       "      <td>194100</td>\n",
       "      <td>10188</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>F</td>\n",
       "      <td>VEHICLE</td>\n",
       "      <td>17</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>SAMSUNG</td>\n",
       "      <td>4801600</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BABEL_OP2_302_10188_20131030_194100_inLine.wav</td>\n",
       "      <td>10188</td>\n",
       "      <td>20131030</td>\n",
       "      <td>194100</td>\n",
       "      <td>10188</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>F</td>\n",
       "      <td>MICROPHONE</td>\n",
       "      <td>18</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>SAMSUNG</td>\n",
       "      <td>30855552</td>\n",
       "      <td>48000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BABEL_OP2_302_11673_20131104_223908_inLine.sph</td>\n",
       "      <td>11673</td>\n",
       "      <td>20131104</td>\n",
       "      <td>223908</td>\n",
       "      <td>11673</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>19</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800320</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          outputFn  sessID      date    time  \\\n",
       "0  BABEL_OP2_302_10002_20140316_215637_outLine.sph   10002  20140316  215637   \n",
       "1   BABEL_OP2_302_10002_20140316_215637_inLine.sph   10002  20140316  215637   \n",
       "2  BABEL_OP2_302_10188_20131030_194100_outLine.sph   10188  20131030  194100   \n",
       "3   BABEL_OP2_302_10188_20131030_194100_inLine.wav   10188  20131030  194100   \n",
       "4   BABEL_OP2_302_11673_20131104_223908_inLine.sph   11673  20131104  223908   \n",
       "\n",
       "   spkrCode lineType   dialect gen             envType  age       network  \\\n",
       "0     10002  outLine  SOUTHERN   M  HOME_OFFICE_MOBILE   42  ACTIVE_KCELL   \n",
       "1     10002   inLine  SOUTHERN   M        PUBLIC_PLACE   36  ACTIVE_KCELL   \n",
       "2     10188  outLine  SOUTHERN   F             VEHICLE   17         TELE2   \n",
       "3     10188   inLine  SOUTHERN   F          MICROPHONE   18         TELE2   \n",
       "4     11673   inLine  SOUTHERN   M  HOME_OFFICE_MOBILE   19         TELE2   \n",
       "\n",
       "  phoneModel  sampleCount  sampleRate  \n",
       "0      NOKIA      4823040        8000  \n",
       "1      NOKIA      4800960        8000  \n",
       "2    SAMSUNG      4801600        8000  \n",
       "3    SAMSUNG     30855552       48000  \n",
       "4      NOKIA      4800320        8000  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read tsv file\n",
    "metadata1 = pd.read_table(\"/Users/aidyn/Downloads/IARPA_BABEL_OP2_302/conversational/reference_materials/demographics.tsv\")\n",
    "metadata1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4e34ff45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(668, 14)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "77ef2f3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sessID</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>spkrCode</th>\n",
       "      <th>age</th>\n",
       "      <th>sampleCount</th>\n",
       "      <th>sampleRate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>668.000000</td>\n",
       "      <td>6.680000e+02</td>\n",
       "      <td>668.000000</td>\n",
       "      <td>668.000000</td>\n",
       "      <td>668.000000</td>\n",
       "      <td>6.680000e+02</td>\n",
       "      <td>668.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>49998.931138</td>\n",
       "      <td>2.013500e+07</td>\n",
       "      <td>170583.000000</td>\n",
       "      <td>49998.931138</td>\n",
       "      <td>33.005988</td>\n",
       "      <td>7.217622e+06</td>\n",
       "      <td>12431.137725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>27708.568322</td>\n",
       "      <td>4.470589e+03</td>\n",
       "      <td>67210.898592</td>\n",
       "      <td>27708.568322</td>\n",
       "      <td>13.760098</td>\n",
       "      <td>8.163342e+06</td>\n",
       "      <td>12563.710642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10002.000000</td>\n",
       "      <td>2.013103e+07</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>10002.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>6.036800e+05</td>\n",
       "      <td>8000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>24241.000000</td>\n",
       "      <td>2.013112e+07</td>\n",
       "      <td>164054.000000</td>\n",
       "      <td>24241.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>4.800160e+06</td>\n",
       "      <td>8000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>45706.000000</td>\n",
       "      <td>2.013122e+07</td>\n",
       "      <td>192920.500000</td>\n",
       "      <td>45706.000000</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>4.800800e+06</td>\n",
       "      <td>8000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>77033.000000</td>\n",
       "      <td>2.014013e+07</td>\n",
       "      <td>212825.000000</td>\n",
       "      <td>77033.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>4.822240e+06</td>\n",
       "      <td>8000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>99887.000000</td>\n",
       "      <td>2.014033e+07</td>\n",
       "      <td>235807.000000</td>\n",
       "      <td>99887.000000</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>3.234547e+07</td>\n",
       "      <td>48000.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             sessID          date           time      spkrCode         age  \\\n",
       "count    668.000000  6.680000e+02     668.000000    668.000000  668.000000   \n",
       "mean   49998.931138  2.013500e+07  170583.000000  49998.931138   33.005988   \n",
       "std    27708.568322  4.470589e+03   67210.898592  27708.568322   13.760098   \n",
       "min    10002.000000  2.013103e+07      22.000000  10002.000000   16.000000   \n",
       "25%    24241.000000  2.013112e+07  164054.000000  24241.000000   20.000000   \n",
       "50%    45706.000000  2.013122e+07  192920.500000  45706.000000   30.000000   \n",
       "75%    77033.000000  2.014013e+07  212825.000000  77033.000000   46.000000   \n",
       "max    99887.000000  2.014033e+07  235807.000000  99887.000000   64.000000   \n",
       "\n",
       "        sampleCount    sampleRate  \n",
       "count  6.680000e+02    668.000000  \n",
       "mean   7.217622e+06  12431.137725  \n",
       "std    8.163342e+06  12563.710642  \n",
       "min    6.036800e+05   8000.000000  \n",
       "25%    4.800160e+06   8000.000000  \n",
       "50%    4.800800e+06   8000.000000  \n",
       "75%    4.822240e+06   8000.000000  \n",
       "max    3.234547e+07  48000.000000  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "33c4930a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(130, 14)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata2 = pd.read_table(\"/Users/aidyn/Downloads/IARPA_BABEL_OP2_302/conversational/reference_materials/demographics.sub-train.tsv\")\n",
    "metadata2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f4a8601a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(798, 14)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata = pd.concat([metadata1, metadata2])\n",
    "metadata.rename(columns={'outputFn': 'fileName'}, inplace=True)\n",
    "# metadata.info()\n",
    "# metadata.head()\n",
    "metadata.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5d1ba16f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fileName</th>\n",
       "      <th>sessID</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>spkrCode</th>\n",
       "      <th>lineType</th>\n",
       "      <th>dialect</th>\n",
       "      <th>gen</th>\n",
       "      <th>envType</th>\n",
       "      <th>age</th>\n",
       "      <th>network</th>\n",
       "      <th>phoneModel</th>\n",
       "      <th>sampleCount</th>\n",
       "      <th>sampleRate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>_OP2_302_10002_20140316_215637_outLine.txt</td>\n",
       "      <td>10002</td>\n",
       "      <td>20140316</td>\n",
       "      <td>215637</td>\n",
       "      <td>10002</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>42</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4823040</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>_OP2_302_10002_20140316_215637_inLine.txt</td>\n",
       "      <td>10002</td>\n",
       "      <td>20140316</td>\n",
       "      <td>215637</td>\n",
       "      <td>10002</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>PUBLIC_PLACE</td>\n",
       "      <td>36</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800960</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>_OP2_302_10188_20131030_194100_outLine.txt</td>\n",
       "      <td>10188</td>\n",
       "      <td>20131030</td>\n",
       "      <td>194100</td>\n",
       "      <td>10188</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>F</td>\n",
       "      <td>VEHICLE</td>\n",
       "      <td>17</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>SAMSUNG</td>\n",
       "      <td>4801600</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>_OP2_302_10188_20131030_194100_inLine.txt</td>\n",
       "      <td>10188</td>\n",
       "      <td>20131030</td>\n",
       "      <td>194100</td>\n",
       "      <td>10188</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>F</td>\n",
       "      <td>MICROPHONE</td>\n",
       "      <td>18</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>SAMSUNG</td>\n",
       "      <td>30855552</td>\n",
       "      <td>48000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>_OP2_302_11673_20131104_223908_inLine.txt</td>\n",
       "      <td>11673</td>\n",
       "      <td>20131104</td>\n",
       "      <td>223908</td>\n",
       "      <td>11673</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>19</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800320</td>\n",
       "      <td>8000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     fileName  sessID      date    time  \\\n",
       "0  _OP2_302_10002_20140316_215637_outLine.txt   10002  20140316  215637   \n",
       "1   _OP2_302_10002_20140316_215637_inLine.txt   10002  20140316  215637   \n",
       "2  _OP2_302_10188_20131030_194100_outLine.txt   10188  20131030  194100   \n",
       "3   _OP2_302_10188_20131030_194100_inLine.txt   10188  20131030  194100   \n",
       "4   _OP2_302_11673_20131104_223908_inLine.txt   11673  20131104  223908   \n",
       "\n",
       "   spkrCode lineType   dialect gen             envType  age       network  \\\n",
       "0     10002  outLine  SOUTHERN   M  HOME_OFFICE_MOBILE   42  ACTIVE_KCELL   \n",
       "1     10002   inLine  SOUTHERN   M        PUBLIC_PLACE   36  ACTIVE_KCELL   \n",
       "2     10188  outLine  SOUTHERN   F             VEHICLE   17         TELE2   \n",
       "3     10188   inLine  SOUTHERN   F          MICROPHONE   18         TELE2   \n",
       "4     11673   inLine  SOUTHERN   M  HOME_OFFICE_MOBILE   19         TELE2   \n",
       "\n",
       "  phoneModel  sampleCount  sampleRate  \n",
       "0      NOKIA      4823040        8000  \n",
       "1      NOKIA      4800960        8000  \n",
       "2    SAMSUNG      4801600        8000  \n",
       "3    SAMSUNG     30855552       48000  \n",
       "4      NOKIA      4800320        8000  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata['fileName'] = metadata['fileName'].astype(str).str.replace('BABEL', '')\n",
    "metadata['fileName'] = metadata['fileName'].astype(str).str.replace('sph', 'txt')\n",
    "metadata['fileName'] = metadata['fileName'].astype(str).str.replace('wav', 'txt')\n",
    "metadata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0170e417",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/aidyn/Downloads/IARPA_BABEL_OP2_302/conversational/dev/transcription/BABEL_OP2_302_85322_20131108_161437_outLine.txt'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# txt files \n",
    "corpus_dir = \"/Users/aidyn/Downloads/IARPA_BABEL_OP2_302/conversational\"\n",
    "\n",
    "speech_data1 = glob.glob(corpus_dir + '/dev/transcription/*.txt')\n",
    "speech_data1[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c798f1a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "140"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(speech_data1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b07359bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "pathlen = len(corpus_dir + '/dev/transcription/*.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "bd255659",
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_dict1 = {}\n",
    "\n",
    "for fname in speech_data1:\n",
    "    fname_short = fname[pathlen:] # Using Option 2\n",
    "    # print(fname_short)      # print short name for checking\n",
    "    txt = open(fname).read()  # open file and read text content! \n",
    "    speech_dict1[fname_short] = txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9e17b17e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.000]\n",
      "<no-speech>\n",
      "[0.095]\n",
      "<no-speech>\n",
      "[3.005]\n",
      "<no-speech>\n",
      "[5.675]\n",
      "алло иә иә\n",
      "[6.675]\n",
      "<no-speech>\n",
      "[\n"
     ]
    }
   ],
   "source": [
    "print(speech_dict1['_OP2_302_85322_20131108_161437_outLine.txt'][:100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "48a2a513",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fileName</th>\n",
       "      <th>Speech_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>_OP2_302_85322_20131108_161437_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[0.095]\\n&lt;no-speech&gt;\\n[3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>_OP2_302_44868_20131217_205716_inLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[8.375]\\nалло\\n[8.905]\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>_OP2_302_19703_20131202_234704_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[2.375]\\n&lt;no-speech&gt;\\n[2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>_OP2_302_60830_20131205_225122_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[6.645]\\nалло\\n[7.335]\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>_OP2_302_47156_20140313_011009_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[0.505]\\n&lt;no-speech&gt;\\n[4...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     fileName  \\\n",
       "0  _OP2_302_85322_20131108_161437_outLine.txt   \n",
       "1   _OP2_302_44868_20131217_205716_inLine.txt   \n",
       "2  _OP2_302_19703_20131202_234704_outLine.txt   \n",
       "3  _OP2_302_60830_20131205_225122_outLine.txt   \n",
       "4  _OP2_302_47156_20140313_011009_outLine.txt   \n",
       "\n",
       "                                          Speech_raw  \n",
       "0  [0.000]\\n<no-speech>\\n[0.095]\\n<no-speech>\\n[3...  \n",
       "1  [0.000]\\n<no-speech>\\n[8.375]\\nалло\\n[8.905]\\n...  \n",
       "2  [0.000]\\n<no-speech>\\n[2.375]\\n<no-speech>\\n[2...  \n",
       "3  [0.000]\\n<no-speech>\\n[6.645]\\nалло\\n[7.335]\\n...  \n",
       "4  [0.000]\\n<no-speech>\\n[0.505]\\n<no-speech>\\n[4...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speech_df = pd.DataFrame({'fileName':speech_dict1.keys(), 'Speech_raw':speech_dict1.values()})\n",
    "speech_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "5d48978d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/aidyn/Downloads/IARPA_BABEL_OP2_302/conversational/sub-train/transcription/BABEL_OP2_302_85010_20140316_222754_outLine.txt'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speech_data2 = glob.glob(corpus_dir + '/sub-train/transcription/*.txt')\n",
    "speech_data2[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "495b5a01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "130"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(speech_data2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5faea776",
   "metadata": {},
   "outputs": [],
   "source": [
    "pathlen = len(corpus_dir + '/sub-train/transcription/*.txt')\n",
    "speech_dict2 = {}\n",
    "\n",
    "for fname in speech_data2:\n",
    "    fname_short = fname[pathlen:] # Using Option 2\n",
    "    # print(fname_short)      # print short name for checking\n",
    "    txt = open(fname).read()  # open file and read text content! \n",
    "    speech_dict2[fname_short] = txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a9f9f53b",
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_df2 = pd.DataFrame({'fileName':speech_dict2.keys(), 'Speech_raw':speech_dict2.values()})\n",
    "# speech_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "184268f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fileName</th>\n",
       "      <th>Speech_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>_OP2_302_85322_20131108_161437_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[0.095]\\n&lt;no-speech&gt;\\n[3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>_OP2_302_44868_20131217_205716_inLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[8.375]\\nалло\\n[8.905]\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>_OP2_302_19703_20131202_234704_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[2.375]\\n&lt;no-speech&gt;\\n[2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>_OP2_302_60830_20131205_225122_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[6.645]\\nалло\\n[7.335]\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>_OP2_302_47156_20140313_011009_outLine.txt</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[0.505]\\n&lt;no-speech&gt;\\n[4...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     fileName  \\\n",
       "0  _OP2_302_85322_20131108_161437_outLine.txt   \n",
       "1   _OP2_302_44868_20131217_205716_inLine.txt   \n",
       "2  _OP2_302_19703_20131202_234704_outLine.txt   \n",
       "3  _OP2_302_60830_20131205_225122_outLine.txt   \n",
       "4  _OP2_302_47156_20140313_011009_outLine.txt   \n",
       "\n",
       "                                          Speech_raw  \n",
       "0  [0.000]\\n<no-speech>\\n[0.095]\\n<no-speech>\\n[3...  \n",
       "1  [0.000]\\n<no-speech>\\n[8.375]\\nалло\\n[8.905]\\n...  \n",
       "2  [0.000]\\n<no-speech>\\n[2.375]\\n<no-speech>\\n[2...  \n",
       "3  [0.000]\\n<no-speech>\\n[6.645]\\nалло\\n[7.335]\\n...  \n",
       "4  [0.000]\\n<no-speech>\\n[0.505]\\n<no-speech>\\n[4...  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speech_data = pd.concat([speech_df, speech_df2])\n",
    "speech_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a775ff42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(270, 2)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speech_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "299a0aeb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fileName</th>\n",
       "      <th>sessID</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>spkrCode</th>\n",
       "      <th>lineType</th>\n",
       "      <th>dialect</th>\n",
       "      <th>gen</th>\n",
       "      <th>envType</th>\n",
       "      <th>age</th>\n",
       "      <th>network</th>\n",
       "      <th>phoneModel</th>\n",
       "      <th>sampleCount</th>\n",
       "      <th>sampleRate</th>\n",
       "      <th>Speech_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>_OP2_302_19703_20131202_234704_outLine.txt</td>\n",
       "      <td>19703</td>\n",
       "      <td>20131202</td>\n",
       "      <td>234704</td>\n",
       "      <td>19703</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>25</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4801760</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[2.375]\\n&lt;no-speech&gt;\\n[2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>_OP2_302_53492_20140124_221354_inLine.txt</td>\n",
       "      <td>53492</td>\n",
       "      <td>20140124</td>\n",
       "      <td>221354</td>\n",
       "      <td>53492</td>\n",
       "      <td>inLine</td>\n",
       "      <td>NORTHEASTERN</td>\n",
       "      <td>M</td>\n",
       "      <td>PUBLIC_PLACE</td>\n",
       "      <td>31</td>\n",
       "      <td>BEELINE</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>2815360</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[1.065]\\n&lt;no-speech&gt;\\n[1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>_OP2_302_50779_20131219_172746_inLine.txt</td>\n",
       "      <td>50779</td>\n",
       "      <td>20131219</td>\n",
       "      <td>172746</td>\n",
       "      <td>50779</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>CAR_KIT</td>\n",
       "      <td>52</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800160</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[8.845]\\n&lt;no-speech&gt;\\n[9...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       fileName  sessID      date    time  \\\n",
       "20   _OP2_302_19703_20131202_234704_outLine.txt   19703  20131202  234704   \n",
       "281   _OP2_302_53492_20140124_221354_inLine.txt   53492  20140124  221354   \n",
       "275   _OP2_302_50779_20131219_172746_inLine.txt   50779  20131219  172746   \n",
       "\n",
       "     spkrCode lineType       dialect gen             envType  age  \\\n",
       "20      19703  outLine      SOUTHERN   M  HOME_OFFICE_MOBILE   25   \n",
       "281     53492   inLine  NORTHEASTERN   M        PUBLIC_PLACE   31   \n",
       "275     50779   inLine      SOUTHERN   M             CAR_KIT   52   \n",
       "\n",
       "          network phoneModel  sampleCount  sampleRate  \\\n",
       "20   ACTIVE_KCELL      NOKIA      4801760        8000   \n",
       "281       BEELINE      NOKIA      2815360        8000   \n",
       "275  ACTIVE_KCELL      NOKIA      4800160        8000   \n",
       "\n",
       "                                            Speech_raw  \n",
       "20   [0.000]\\n<no-speech>\\n[2.375]\\n<no-speech>\\n[2...  \n",
       "281  [0.000]\\n<no-speech>\\n[1.065]\\n<no-speech>\\n[1...  \n",
       "275  [0.000]\\n<no-speech>\\n[8.845]\\n<no-speech>\\n[9...  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge two df on fileName\n",
    "data_kaz = pd.merge(metadata, speech_data)\n",
    "data_kaz.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e5209cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 400 entries, 0 to 399\n",
      "Data columns (total 15 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   fileName     400 non-null    object\n",
      " 1   sessID       400 non-null    int64 \n",
      " 2   date         400 non-null    int64 \n",
      " 3   time         400 non-null    int64 \n",
      " 4   spkrCode     400 non-null    int64 \n",
      " 5   lineType     400 non-null    object\n",
      " 6   dialect      400 non-null    object\n",
      " 7   gen          400 non-null    object\n",
      " 8   envType      400 non-null    object\n",
      " 9   age          400 non-null    int64 \n",
      " 10  network      400 non-null    object\n",
      " 11  phoneModel   400 non-null    object\n",
      " 12  sampleCount  400 non-null    int64 \n",
      " 13  sampleRate   400 non-null    int64 \n",
      " 14  Speech_raw   400 non-null    object\n",
      "dtypes: int64(7), object(8)\n",
      "memory usage: 50.0+ KB\n"
     ]
    }
   ],
   "source": [
    "data_kaz.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "17269853",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_kaz['Speech_clean'] = data_kaz.Speech_raw.str.replace(r'[^\\w\\s]+', '', regex=True)\n",
    "data_kaz['Speech_clean'] = data_kaz.Speech_clean.str.replace(r'\\d+', '', regex=True)\n",
    "data_kaz['Speech_clean'] = data_kaz.Speech_clean.str.replace(r'[a-z]+', '', regex=True)\n",
    "data_kaz['Speech_clean'] = data_kaz.Speech_clean.str.replace(r'\\n+', '.', regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d3b35b10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    .неғыл дейсіз.алло.алло. алейкум ассалаум.алло...\n",
       "1    .ассалаумалейкум ассалаумалейкум  алло.ааа Сәк...\n",
       "2    .нет . да.алло.иә иә.привет жақсы өзің.алло.жа...\n",
       "3    .алло.Айдана  алло Айдана привет ой қалайсың. ...\n",
       "4    .алло. алло.алло ассалаумалейкум .қалай жағдай...\n",
       "5    . алло .алло .қазір . алейкум ассалаум қалай ж...\n",
       "6    . ойбай  ау.алло.ааа привет.аманесен жақсы өзі...\n",
       "7    .алло.алло алло Жұлдыз алло Жұлдыз қалай жағда...\n",
       "8    .жоқ сөйле.жоқ сөйлесе беріңіз менімен бірге  ...\n",
       "9    .мен ке келістім келістім келіспедім деген жау...\n",
       "Name: Speech_clean, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_kaz.Speech_clean[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "72af8a5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " 'ассалаумалейкум ассалаумалейкум  алло',\n",
       " 'ааа Сәкен аға қалайсыз деймін  ей мен ғой мен Жарқынбай',\n",
       " 'алло алло алло',\n",
       " 'жақсы жақсы қалай өзіңіз иә аман аман қалай  ааа сол бір хабарласайын дедім сөйлесіп  ааа']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_kaz['Sents'] = data_kaz.Speech_clean.str.split('.')\n",
    "data_kaz.Sents[1][:5]\n",
    "# data_kaz['Sents']  = data_kaz.Sents.dropna(how='all')\n",
    "\n",
    "# data_kaz['Sents'] = data_kaz.Speech_clean.map(nltk.sent_tokenize)\n",
    "# len(data_kaz.Sents[1])\n",
    "# data_kaz.Sents[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "9e37d74d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fileName</th>\n",
       "      <th>sessID</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>spkrCode</th>\n",
       "      <th>lineType</th>\n",
       "      <th>dialect</th>\n",
       "      <th>gen</th>\n",
       "      <th>envType</th>\n",
       "      <th>age</th>\n",
       "      <th>network</th>\n",
       "      <th>phoneModel</th>\n",
       "      <th>sampleCount</th>\n",
       "      <th>sampleRate</th>\n",
       "      <th>Speech_raw</th>\n",
       "      <th>Speech_clean</th>\n",
       "      <th>Sents</th>\n",
       "      <th>Tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>_OP2_302_23505_20131113_214234_inLine.txt</td>\n",
       "      <td>23505</td>\n",
       "      <td>20131113</td>\n",
       "      <td>214234</td>\n",
       "      <td>23505</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>VEHICLE</td>\n",
       "      <td>18</td>\n",
       "      <td>TELE2</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>2819520</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[6.035]\\nалло Нұреке\\n[8...</td>\n",
       "      <td>.алло Нұреке.алоу.жұмыстасын ба.алейкум ассала...</td>\n",
       "      <td>[, алло Нұреке, алоу, жұмыстасын ба, алейкум а...</td>\n",
       "      <td>[.алло, Нұреке.алоу.жұмыстасын, ба.алейкум, ас...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>_OP2_302_15638_20131227_190456_inLine.txt</td>\n",
       "      <td>15638</td>\n",
       "      <td>20131227</td>\n",
       "      <td>190456</td>\n",
       "      <td>15638</td>\n",
       "      <td>inLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>M</td>\n",
       "      <td>CAR_KIT</td>\n",
       "      <td>18</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4800160</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;no-speech&gt;\\n[0.325]\\n&lt;no-speech&gt;\\n[7...</td>\n",
       "      <td>.алло.алло.алло.Бексұлтан сен бе.алло.қайдасың...</td>\n",
       "      <td>[, алло, алло, алло, Бексұлтан сен бе, алло, қ...</td>\n",
       "      <td>[.алло.алло.алло.Бексұлтан, сен, бе.алло.қайда...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>346</th>\n",
       "      <td>_OP2_302_82145_20140301_225354_outLine.txt</td>\n",
       "      <td>82145</td>\n",
       "      <td>20140301</td>\n",
       "      <td>225354</td>\n",
       "      <td>82145</td>\n",
       "      <td>outLine</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>F</td>\n",
       "      <td>HOME_OFFICE_MOBILE</td>\n",
       "      <td>50</td>\n",
       "      <td>ACTIVE_KCELL</td>\n",
       "      <td>NOKIA</td>\n",
       "      <td>4822400</td>\n",
       "      <td>8000</td>\n",
       "      <td>[0.000]\\n&lt;int&gt; қалай қал &lt;no-speech&gt; алло &lt;no-...</td>\n",
       "      <td>. қалай қал  алло  алло.алло да да ау.жақсы қа...</td>\n",
       "      <td>[,  қалай қал  алло  алло, алло да да ау, жақс...</td>\n",
       "      <td>[., қалай, қал, алло, алло.алло, да, да, ау.жа...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       fileName  sessID      date    time  \\\n",
       "191   _OP2_302_23505_20131113_214234_inLine.txt   23505  20131113  214234   \n",
       "154   _OP2_302_15638_20131227_190456_inLine.txt   15638  20131227  190456   \n",
       "346  _OP2_302_82145_20140301_225354_outLine.txt   82145  20140301  225354   \n",
       "\n",
       "     spkrCode lineType   dialect gen             envType  age       network  \\\n",
       "191     23505   inLine  SOUTHERN   M             VEHICLE   18         TELE2   \n",
       "154     15638   inLine  SOUTHERN   M             CAR_KIT   18  ACTIVE_KCELL   \n",
       "346     82145  outLine  SOUTHERN   F  HOME_OFFICE_MOBILE   50  ACTIVE_KCELL   \n",
       "\n",
       "    phoneModel  sampleCount  sampleRate  \\\n",
       "191      NOKIA      2819520        8000   \n",
       "154      NOKIA      4800160        8000   \n",
       "346      NOKIA      4822400        8000   \n",
       "\n",
       "                                            Speech_raw  \\\n",
       "191  [0.000]\\n<no-speech>\\n[6.035]\\nалло Нұреке\\n[8...   \n",
       "154  [0.000]\\n<no-speech>\\n[0.325]\\n<no-speech>\\n[7...   \n",
       "346  [0.000]\\n<int> қалай қал <no-speech> алло <no-...   \n",
       "\n",
       "                                          Speech_clean  \\\n",
       "191  .алло Нұреке.алоу.жұмыстасын ба.алейкум ассала...   \n",
       "154  .алло.алло.алло.Бексұлтан сен бе.алло.қайдасың...   \n",
       "346  . қалай қал  алло  алло.алло да да ау.жақсы қа...   \n",
       "\n",
       "                                                 Sents  \\\n",
       "191  [, алло Нұреке, алоу, жұмыстасын ба, алейкум а...   \n",
       "154  [, алло, алло, алло, Бексұлтан сен бе, алло, қ...   \n",
       "346  [,  қалай қал  алло  алло, алло да да ау, жақс...   \n",
       "\n",
       "                                                Tokens  \n",
       "191  [.алло, Нұреке.алоу.жұмыстасын, ба.алейкум, ас...  \n",
       "154  [.алло.алло.алло.Бексұлтан, сен, бе.алло.қайда...  \n",
       "346  [., қалай, қал, алло, алло.алло, да, да, ау.жа...  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_kaz['Tokens'] = data_kaz.Speech_clean.map(nltk.word_tokenize)\n",
    "data_kaz.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "580b9f9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 18)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_kaz.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebb2ace",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "- **Advantages**:\n",
    "    - Metadata: age, gender, 'dialect', environment type, etc.\n",
    "    - Each .txt file is a telephone conversation and therefore contains great examples of Kazak-Russian code-switching (phrase level code-switching so far has been observed) as well as adapted Russian words within Kazakh utterance.\n",
    "    - Connected speech;\n",
    "\n",
    "- **Disadvantages**:\n",
    "    - The dataset cannot be fully shared/redistributed;\n",
    "    - I can use this dataset as a Pitt student only for the university related projects/coursework;\n",
    "    \n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4604a3b",
   "metadata": {},
   "source": [
    "### **Future Work**\n",
    "\n",
    "Summing up, the dataset No.2: IARPA Babel Kazakh Language Pack contains more relevant data points to explore Kazakh-Russian code-switching and make reasonable inferences regarding linguistic characteristics of the phenomenon. Therefore, after careful consideration, I will further use this dataset as a main source of my term project. After manually screening the txt files (integrated into Speech_raw column), I have encountered multiple examples of intra-sentential and intra-word code switching instances. My goals are the following:\n",
    "\n",
    "- clean the Speech column to get rid off unnecessary tokens like extra punctuation marks(.,);\n",
    "- EDA analysis of the dataset overall;\n",
    "- annotate instances of code-switching using the annoation scheme;\n",
    "- closely examine Russian code-switching instances:\n",
    "    - what type of code-switching is more prevalent: intra-sentential or intra-word;\n",
    "- one strategy to recognize Russian words within Kazakh utterances is look for specific letters that do not occur in original Kazakh words, such as sequences of consonants at the beginning of the word, consonant clusters typical for Russian, and so on. For more details: https://link.springer.com/chapter/10.1007/978-3-319-23132-7_3 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3727233",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "Khassanov, Y., Mussakhojayeva, S., Mirzakhmetov, A., Adiyev, A., Nurpeiissov, M., & Varol, H. A. (2020). A crowdsourced open-source Kazakh speech corpus and initial speech recognition baseline. arXiv preprint arXiv:2009.10334.\n",
    "\n",
    "Khomitsevich, O., Mendelev, V., Tomashenko, N., Rybin, S., Medennikov, I., Kudubayeva, S. (2015). A Bilingual Kazakh-Russian System for Automatic Speech Recognition and Synthesis. In: Ronzhin, A., Potapova, R., Fakotakis, N. (eds) Speech and Computer. SPECOM 2015. Lecture Notes in Computer Science(), vol 9319. Springer, Cham.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}