{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tN1oNjn9VGdB"
},
"source": [
"# SIT719\n",
"**(Week 01: A Touch of Data Science)**\n",
"\n",
"---\n",
"- Materials in this module include resources collected from various open-source online repositories.\n",
"- You are free to use, change and distribute this package.\n",
"- Thanks to A/Prof Gang Li for sharing the material \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"# Session - NumPy's Structured Arrays"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "2bonTwf8VGdK"
},
"source": [
"While often our data can be well represented by a homogeneous array of values, sometimes this is not the case. This section demonstrates the use of NumPy's *structured arrays* and *record arrays*, which provide efficient storage for compound, heterogeneous data. While the patterns shown here are useful for simple operations, scenarios like this often lend themselves to the use of Pandas ``Dataframe``s, which we'll explore in future sessions."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "WXUqJH5kVGdR"
},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "KTQfLWePVGde"
},
"source": [
"Imagine that we have several categories of data on a number of people (say, name, age, and weight), and we'd like to store these values for use in a Python program.\n",
"It would be possible to store these in three separate arrays:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "m1bo6ntuVGdh"
},
"outputs": [],
"source": [
"name = ['Alice', 'Bob', 'Cathy', 'Doug']\n",
"age = [25, 45, 37, 19]\n",
"weight = [55.0, 85.5, 68.0, 61.5]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "rix5zqmPVGdq"
},
"source": [
"But this is a bit clumsy. There's nothing here that tells us that the three arrays are related; it would be more natural if we could use a single structure to store all of this data.\n",
"NumPy can handle this through structured arrays, which are arrays with compound data types.\n",
"\n",
"Recall that previously we created a simple array using an expression like this:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "AcQ8KY1uVGdv"
},
"outputs": [],
"source": [
"x = np.zeros(4, dtype=int)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Ke0w8f26VGd4"
},
"source": [
"We can similarly create a structured array using a compound data type specification:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "vZLFmnhMVGd7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[('name', '<U10'), ('age', '<i4'), ('weight', '<f8')]\n"
]
}
],
"source": [
"# Use a compound data type for structured arrays\n",
"data = np.zeros(4, dtype={'names':('name', 'age', 'weight'),\n",
" 'formats':('U10', 'i4', 'f8')})\n",
"print(data.dtype)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "heNzAie2VGeF"
},
"source": [
"Here ``'U10'`` translates to \"Unicode string of maximum length 10,\" ``'i4'`` translates to \"4-byte (i.e., 32 bit) integer,\" and ``'f8'`` translates to \"8-byte (i.e., 64 bit) float.\"\n",
"We'll discuss other options for these type codes in the following section.\n",
"\n",
"Now that we've created an empty container array, we can fill the array with our lists of values:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "RHGZKSf6VGeI"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[('Alice', 25, 55. ) ('Bob', 45, 85.5) ('Cathy', 37, 68. )\n",
" ('Doug', 19, 61.5)]\n"
]
}
],
"source": [
"data['name'] = name\n",
"data['age'] = age\n",
"data['weight'] = weight\n",
"print(data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GRR0ZQMcVGeR"
},
"source": [
"As we had hoped, the data is now arranged together in one convenient block of memory.\n",
"\n",
"The handy thing with structured arrays is that you can now refer to values either by index or by name:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "N58jqN5AVGeU"
},
"outputs": [
{
"data": {
"text/plain": [
"array(['Alice', 'Bob', 'Cathy', 'Doug'], dtype='<U10')"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get all names\n",
"data['name']"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "QAQTvJOHVGel"
},
"outputs": [
{
"data": {
"text/plain": [
"('Alice', 25, 55.)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get first row of data\n",
"data[0]"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Le5J2X5GVGey"
},
"outputs": [
{
"data": {
"text/plain": [
"'Doug'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the name from the last row\n",
"data[-1]['name']"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "h7QXxzmpVGe9"
},
"source": [
"Using Boolean masking, this even allows you to do some more sophisticated operations such as filtering on age:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "8XWhfGsQVGfH"
},
"outputs": [
{
"data": {
"text/plain": [
"array(['Alice', 'Doug'], dtype='<U10')"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get names where age is under 30\n",
"data[data['age'] < 30]['name']"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-pwa4tFSVGfP"
},
"source": [
"Note that if you'd like to do any operations that are any more complicated than these, you should probably consider the Pandas package, covered in the next chapter.\n",
"As we'll see, Pandas provides a ``Dataframe`` object, which is a structure built on NumPy arrays that offers a variety of useful data manipulation functionality similar to what we've shown here, as well as much, much more."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "HoIwTvrtVGfS"
},
"source": [
"## Creating Structured Arrays\n",
"\n",
"Structured array data types can be specified in a number of ways.\n",
"Earlier, we saw the dictionary method:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "-_JrEBJ1VGfU"
},
"outputs": [
{
"data": {
"text/plain": [
"dtype([('name', '<U10'), ('age', '<i4'), ('weight', '<f8')])"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.dtype({'names':('name', 'age', 'weight'),\n",
" 'formats':('U10', 'i4', 'f8')})"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Cm6uDC7XVGfb"
},
"source": [
"For clarity, numerical types can be specified using Python types or NumPy ``dtype``s instead:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "WZ4hWV3gVGfe"
},
"outputs": [
{
"data": {
"text/plain": [
"dtype([('name', '<U10'), ('age', '<i4'), ('weight', '<f4')])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.dtype({'names':('name', 'age', 'weight'),\n",
" 'formats':((np.str_, 10), int, np.float32)})"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Nh6Wv-_rVGfl"
},
"source": [
"A compound type can also be specified as a list of tuples:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "k4J505_pVGfo"
},
"outputs": [
{
"data": {
"text/plain": [
"dtype([('name', 'S10'), ('age', '<i4'), ('weight', '<f8')])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.dtype([('name', 'S10'), ('age', 'i4'), ('weight', 'f8')])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "zWQ8dF8VVGfv"
},
"source": [
"If the names of the types do not matter to you, you can specify the types alone in a comma-separated string:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "HJJhxX_qVGfy"
},
"outputs": [
{
"data": {
"text/plain": [
"dtype([('f0', 'S10'), ('f1', '<i4'), ('f2', '<f8')])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.dtype('S10,i4,f8')"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5Qp2KNniVGf5"
},
"source": [
"The shortened string format codes may seem confusing, but they are built on simple principles.\n",
"The first (optional) character is ``<`` or ``>``, which means \"little endian\" or \"big endian,\" respectively, and specifies the ordering convention for significant bits.\n",
"The next character specifies the type of data: characters, bytes, ints, floating points, and so on (see the table below).\n",
"The last character or characters represents the size of the object in bytes.\n",
"\n",
"| Character | Description | Example |\n",
"| --------- | ----------- | ------- | \n",
"| ``'b'`` | Byte | ``np.dtype('b')`` |\n",
"| ``'i'`` | Signed integer | ``np.dtype('i4') == np.int32`` |\n",
"| ``'u'`` | Unsigned integer | ``np.dtype('u1') == np.uint8`` |\n",
"| ``'f'`` | Floating point | ``np.dtype('f8') == np.int64`` |\n",
"| ``'c'`` | Complex floating point| ``np.dtype('c16') == np.complex128``|\n",
"| ``'S'``, ``'a'`` | String | ``np.dtype('S5')`` |\n",
"| ``'U'`` | Unicode string | ``np.dtype('U') == np.str_`` |\n",
"| ``'V'`` | Raw data (void) | ``np.dtype('V') == np.void`` |"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "W6fXzzT6VGf8"
},
"source": [
"## More Advanced Compound Types\n",
"\n",
"It is possible to define even more advanced compound types.\n",
"For example, you can create a type where each element contains an array or matrix of values.\n",
"Here, we'll create a data type with a ``mat`` component consisting of a $3\\times 3$ floating-point matrix:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "52eUWALIVGgB"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(0, [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.]])\n",
"[[0. 0. 0.]\n",
" [0. 0. 0.]\n",
" [0. 0. 0.]]\n"
]
}
],
"source": [
"tp = np.dtype([('id', 'i8'), ('mat', 'f8', (3, 3))])\n",
"X = np.zeros(1, dtype=tp)\n",
"print(X[0])\n",
"print(X['mat'][0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "oC4jqjB4VGgT"
},
"source": [
"Now each element in the ``X`` array consists of an ``id`` and a $3\\times 3$ matrix.\n",
"Why would you use this rather than a simple multidimensional array, or perhaps a Python dictionary?\n",
"The reason is that this NumPy ``dtype`` directly maps onto a C structure definition, so the buffer containing the array content can be accessed directly within an appropriately written C program.\n",
"If you find yourself writing a Python interface to a legacy C or Fortran library that manipulates structured data, you'll probably find structured arrays quite useful!"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "SvdBUpvVVGgW"
},
"source": [
"## RecordArrays: Structured Arrays with a Twist\n",
"\n",
"NumPy also provides the ``np.recarray`` class, which is almost identical to the structured arrays just described, but with one additional feature: fields can be accessed as attributes rather than as dictionary keys.\n",
"Recall that we previously accessed the ages by writing:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "cQy_hZgHVGgY"
},
"outputs": [
{
"data": {
"text/plain": [
"array([25, 45, 37, 19])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['age']"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "qv1PIr0JVGgi"
},
"source": [
"If we view our data as a record array instead, we can access this with slightly fewer keystrokes:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "fWKE4lTfVGgr"
},
"outputs": [
{
"data": {
"text/plain": [
"array([25, 45, 37, 19])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_rec = data.view(np.recarray)\n",
"data_rec.age"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "khKEGTLNVGgz"
},
"source": [
"The downside is that for record arrays, there is some extra overhead involved in accessing the fields, even when using the same syntax. We can see this here:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "2hkmj0NkVGg2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"154 ns ± 20.2 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n",
"3.68 µs ± 395 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n",
"6.1 µs ± 694 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
]
}
],
"source": [
"%timeit data['age']\n",
"%timeit data_rec['age']\n",
"%timeit data_rec.age"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "SIT742P02B-StructuredDataNumPy.ipynb",
"provenance": [
{
"file_id": "https://github.com/tulip-lab/sit742/blob/master/Jupyter/SIT742P02B-StructuredDataNumPy.ipynb",
"timestamp": 1551763832358
}
],
"version": "0.3.2"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 1
}