From bb3fa240a2112697224b963556696b5d742fd6cc Mon Sep 17 00:00:00 2001 From: shilpa nalkande Date: Thu, 30 Jan 2025 10:13:13 +0000 Subject: [PATCH 1/4] Recipe notebook for BinarySeriesOp --- .../ClearScape_Functions/BinarySeriesOp.ipynb | 346 ++++++++++++++++++ 1 file changed, 346 insertions(+) create mode 100644 Recipes/ClearScape_Functions/BinarySeriesOp.ipynb diff --git a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb new file mode 100644 index 00000000..68694da5 --- /dev/null +++ b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb @@ -0,0 +1,346 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "bc549e6c-0cc4-4188-94a3-a9bdd3ae3dfa", + "metadata": {}, + "source": [ + "
\n", + "

\n", + " BinarySeriesOp function in Vantage\n", + "
\n", + " \"Teradata\"\n", + "

\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "7ae7611a-0795-4168-b716-01fee6880cbd", + "metadata": {}, + "source": [ + "

Introduction

\n", + "\n", + "

BinarySeriesOp

\n", + "

The BinarySeriesOp() function performs a point wise mathematical operation on two time series of equal size. The principal mathematical operation can be subtraction, addition, multiplication and division. It is called a point wise operation because it performs the mathematical operation one sample point at a time.

\n", + "

Common uses of BinarySeriesOp() are:

\n", + "
  • Subtracting trends from a time series to create a model from it.
    • \n", + "
  • Restoring trends to a time series before using the model for forecasting.
    • \n", + "
  • As a building block to formulate more complex functions.

    • \n", + " \n", + "

    The following procedure is an example of how to use BinarySeriesOp() to convolve two series with digital signal processing:

    \n", + "
  • Use DFFT() function on series 1 and series 2 to get dataframes named 'dfftRes1' and 'dfftRes2', respectively.
    • \n", + "
  • Use BinarySeriesOp() to do point-wise multiplication using 'dfftRes1' and 'dfftRes2'.
    • \n", + "
  • Use IDFFT() on the output of BinarySeriesOp() to get the convolved result of the two series.
    • \n", + "

    " + ] + }, + { + "cell_type": "markdown", + "id": "6b3a00b4-6661-4c91-9b2d-cb7b0b403140", + "metadata": {}, + "source": [ + "
    \n", + "1. Initiate a connection to Vantage" + ] + }, + { + "cell_type": "markdown", + "id": "2346857f-e0d3-488a-8a3f-ac6dff752c2b", + "metadata": {}, + "source": [ + "

    In the section, we import the required libraries and set environment variables and environment paths (if required)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5af5af3-29d5-4f6a-8334-9df6924e7787", + "metadata": {}, + "outputs": [], + "source": [ + "from teradataml import (\n", + " create_context,\n", + " execute_sql,\n", + " load_example_data,\n", + " DataFrame,\n", + " in_schema,\n", + " TDSeries,\n", + " BinarySeriesOp,\n", + " db_drop_view,\n", + " remove_context,\n", + " copy_to_sql,\n", + " db_drop_table\n", + " )\n", + "\n", + "from teradatasqlalchemy.types import *\n", + "\n", + "from teradataml import to_numeric\n", + "# Modify the following to match the specific client environment settings\n", + "display.max_rows = 5" + ] + }, + { + "cell_type": "markdown", + "id": "ad3dd7b4-831c-4fb3-ab71-719c8c99a71c", + "metadata": {}, + "source": [ + "

    \n", + "

    1.1 Connect to Vantage

    \n", + "

    You will be prompted to provide the password. Enter your password, press the Enter key, and then use the down arrow to go to the next cell.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2742444c-4349-4b0f-b4e5-b068a8785cd9", + "metadata": {}, + "outputs": [], + "source": [ + "%run -i ../../UseCases/startup.ipynb\n", + "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n", + "print(eng)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "01c4a128-d106-46ea-8dee-34acc5abd29f", + "metadata": {}, + "outputs": [], + "source": [ + "%%capture\n", + "execute_sql('''SET query_band='DEMO=PP_BinarySeriesOp_Python.ipynb;' UPDATE FOR SESSION; ''')" + ] + }, + { + "cell_type": "markdown", + "id": "efe2fd2d-63ff-4278-9157-8b9110d682e8", + "metadata": {}, + "source": [ + "

    Begin running steps with Shift + Enter keys.

    " + ] + }, + { + "cell_type": "markdown", + "id": "4a6e3852-3b15-4cbd-844d-e7194b3bd204", + "metadata": {}, + "source": [ + "
    \n", + "\n", + "

    1.2 Getting Data for This Demo

    \n", + "\n", + "

    Here, we will get the time series data which is available in the teradataml library and use the same to show the usage of the function.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "398fa0aa-c0df-4130-9dea-53b776fbaf80", + "metadata": {}, + "outputs": [], + "source": [ + "load_example_data(\"uaf\", [\"binary_complex_left\", \"binary_complex_right\"])" + ] + }, + { + "cell_type": "markdown", + "id": "2a3762ac-ba27-4fa3-adba-d577262a4290", + "metadata": {}, + "source": [ + "
    \n", + "2. Data Exploration\n", + "

    Create a \"Virtual DataFrame\" that points to the data set in Vantage. Check the shape of the dataframe as check the datatype of all the columns of the dataframe.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9672d80d-cdcc-4f4c-9399-7c367e7970b1", + "metadata": {}, + "outputs": [], + "source": [ + "data1 = DataFrame.from_table(\"binary_complex_left\")\n", + "data2 = DataFrame.from_table(\"binary_complex_right\")\n", + "data1 " + ] + }, + { + "cell_type": "markdown", + "id": "4a2f79a0-eb16-4c3b-bec2-c313e97ac940", + "metadata": {}, + "source": [ + "
    \n", + "3. BinarySeriesOp\n", + "

    The BinarySeriesOp() function can be configured to operate in one of three input modes - ONE2ONE, MANY2ONE, and MATCH. These modes determine the number of primary series and number of secondary series involved in the operation, as well as determining how the primary and secondary series will be matched together.

    \n", + "\n", + "

    The first step is to convert the data into TDSeries, which is required for the input time series which are passed to the LinearRegr function.

    \n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a711f58c-fbd3-4457-9f6a-9cd5c1b192dc", + "metadata": {}, + "outputs": [], + "source": [ + "data1_series_df = TDSeries(data=data1,\n", + " id=\"id\",\n", + " row_index=\"seq\",\n", + " row_index_style=\"SEQUENCE\",\n", + " payload_field=[\"real_val\", \"imaginary_val\"],\n", + " payload_content=\"COMPLEX\") " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "73e8def9-f8a7-40b0-bbf0-4cb3e4295794", + "metadata": {}, + "outputs": [], + "source": [ + "data2_series_df = TDSeries(data=data2,\n", + " id=\"id\",\n", + " row_index=\"seq\",\n", + " row_index_style= \"SEQUENCE\",\n", + " payload_field=[\"real_val\", \"imaginary_val\"],\n", + " payload_content=\"COMPLEX\")" + ] + }, + { + "cell_type": "markdown", + "id": "d33d2bcc-c87c-4fd4-913a-79e330bd2da2", + "metadata": {}, + "source": [ + "

    Than, we form the filter expressions to filter the series with id=1.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d9cc417c-b6f2-40a1-810b-bbfa893b10ba", + "metadata": {}, + "outputs": [], + "source": [ + "data1_filter_expr=data1_series_df.id==1\n", + "data2_filter_expr=data2_series_df.id==1" + ] + }, + { + "cell_type": "markdown", + "id": "0bd6859c-b85a-4599-90f5-1f6621980fab", + "metadata": {}, + "source": [ + "

    We use the BinarySeriesOp to perform addition of two time series of equal size.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8965392c-404e-4e25-b5b8-c7ef06069154", + "metadata": {}, + "outputs": [], + "source": [ + "uaf_out = BinarySeriesOp(data1=data1_series_df,\n", + " data1_filter_expr=data1_filter_expr,\n", + " data2=data2_series_df,\n", + " data2_filter_expr=data2_filter_expr,\n", + " math_op=\"ADD\",\n", + " input_fmt_input_mode=\"MANY2ONE\")\n", + "\n", + "binaryseries_df = uaf_out.result\n", + "binaryseries_df" + ] + }, + { + "cell_type": "markdown", + "id": "c25fdaf1-098b-4dea-984b-579c6412557e", + "metadata": {}, + "source": [ + "
    \n", + "4. Cleanup" + ] + }, + { + "cell_type": "markdown", + "id": "a5d460af-1a32-4681-8094-360c3f25dc1f", + "metadata": {}, + "source": [ + "

    Work Tables

    \n", + "

    The following code will clean up tables created above.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9dff9677-dae1-4cc5-b78e-3d197cc41e0c", + "metadata": {}, + "outputs": [], + "source": [ + "db_drop_table(\"binary_complex_left\")\n", + "db_drop_table(\"binary_complex_right\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9b96168c-7973-4259-b3e0-668b92fd846e", + "metadata": {}, + "outputs": [], + "source": [ + "remove_context()" + ] + }, + { + "cell_type": "markdown", + "id": "4317a6cf-1479-4aa8-b30a-ee0a3b5231a8", + "metadata": {}, + "source": [ + "
    \n", + "\n", + "

    Links:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "b2dcca28-5de5-44d7-88cb-45a12153b3f8", + "metadata": {}, + "source": [ + "" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From dc8b80ce3058f87443303e3f8fa606a26b1aef74 Mon Sep 17 00:00:00 2001 From: shilpa nalkande Date: Fri, 31 Jan 2025 09:47:42 +0000 Subject: [PATCH 2/4] corrected a minor error in Markdown --- Recipes/ClearScape_Functions/BinarySeriesOp.ipynb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb index 68694da5..7de68efd 100644 --- a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb +++ b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb @@ -174,7 +174,7 @@ "3. BinarySeriesOp\n", "

    The BinarySeriesOp() function can be configured to operate in one of three input modes - ONE2ONE, MANY2ONE, and MATCH. These modes determine the number of primary series and number of secondary series involved in the operation, as well as determining how the primary and secondary series will be matched together.

    \n", "\n", - "

    The first step is to convert the data into TDSeries, which is required for the input time series which are passed to the LinearRegr function.

    \n" + "

    The first step is to convert the data into TDSeries, which is required for the input time series which are passed to the BinarySeriesOp function.

    \n" ] }, { @@ -252,6 +252,14 @@ "binaryseries_df" ] }, + { + "cell_type": "markdown", + "id": "2f66bf78-4ea4-4387-b078-a91f491931e6", + "metadata": {}, + "source": [ + "

    The result series always inherits the identifiers, series_id from the primary series.

    " + ] + }, { "cell_type": "markdown", "id": "c25fdaf1-098b-4dea-984b-579c6412557e", From 38e92cd238e877013c8623e692bdeea819581d2c Mon Sep 17 00:00:00 2001 From: shilpa-nalkande Date: Mon, 24 Feb 2025 06:00:53 +0000 Subject: [PATCH 3/4] Added cell for help --- .../ClearScape_Functions/BinarySeriesOp.ipynb | 82 +++++++++++-------- 1 file changed, 49 insertions(+), 33 deletions(-) diff --git a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb index 7de68efd..6c9fad23 100644 --- a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb +++ b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb @@ -7,7 +7,7 @@ "source": [ "
    \n", "

    \n", - " BinarySeriesOp function in Vantage\n", + " BinarySeriesOp Function in Vantage\n", "
    \n", " \"Teradata\"\n", "

    \n", @@ -19,19 +19,19 @@ "id": "7ae7611a-0795-4168-b716-01fee6880cbd", "metadata": {}, "source": [ - "

    Introduction

    \n", + "

    Introduction

    \n", "\n", - "

    BinarySeriesOp

    \n", - "

    The BinarySeriesOp() function performs a point wise mathematical operation on two time series of equal size. The principal mathematical operation can be subtraction, addition, multiplication and division. It is called a point wise operation because it performs the mathematical operation one sample point at a time.

    \n", - "

    Common uses of BinarySeriesOp() are:

    \n", - "
  • Subtracting trends from a time series to create a model from it.
    • \n", - "
  • Restoring trends to a time series before using the model for forecasting.
    • \n", - "
  • As a building block to formulate more complex functions.

    • \n", + "

    BinarySeriesOp

    \n", + "

    The BinarySeriesOp() function performs a point wise mathematical operation on two time series of equal size. The principal mathematical operation can be subtraction, addition, multiplication and division. It is called a point wise operation because it performs the mathematical operation one sample point at a time.

    \n", + "

    Common uses of BinarySeriesOp() are:

    \n", + "
  • Subtracting trends from a time series to create a model from it.
    • \n", + "
  • Restoring trends to a time series before using the model for forecasting.
    • \n", + "
  • As a building block to formulate more complex functions.

    • \n", " \n", - "

    The following procedure is an example of how to use BinarySeriesOp() to convolve two series with digital signal processing:

    \n", - "
  • Use DFFT() function on series 1 and series 2 to get dataframes named 'dfftRes1' and 'dfftRes2', respectively.
    • \n", - "
  • Use BinarySeriesOp() to do point-wise multiplication using 'dfftRes1' and 'dfftRes2'.
    • \n", - "
  • Use IDFFT() on the output of BinarySeriesOp() to get the convolved result of the two series.
    • \n", + "

    The following procedure is an example of how to use BinarySeriesOp() to convolve two series with digital signal processing:

    \n", + "
  • Use DFFT() function on series 1 and series 2 to get dataframes named 'dfftRes1' and 'dfftRes2', respectively.
    • \n", + "
  • Use BinarySeriesOp() to do point-wise multiplication using 'dfftRes1' and 'dfftRes2'.
    • \n", + "
  • Use IDFFT() on the output of BinarySeriesOp() to get the convolved result of the two series.
    • \n", "

    " ] }, @@ -41,7 +41,7 @@ "metadata": {}, "source": [ "
    \n", - "1. Initiate a connection to Vantage" + "1. Initiate a connection to Vantage" ] }, { @@ -49,7 +49,7 @@ "id": "2346857f-e0d3-488a-8a3f-ac6dff752c2b", "metadata": {}, "source": [ - "

    In the section, we import the required libraries and set environment variables and environment paths (if required)." + "

    In the section, we import the required libraries and set environment variables and environment paths (if required)." ] }, { @@ -86,8 +86,8 @@ "metadata": {}, "source": [ "

    \n", - "

    1.1 Connect to Vantage

    \n", - "

    You will be prompted to provide the password. Enter your password, press the Enter key, and then use the down arrow to go to the next cell.

    " + "

    1.1 Connect to Vantage

    \n", + "

    You will be prompted to provide the password. Enter your password, press the Enter key, and then use the down arrow to go to the next cell.

    " ] }, { @@ -118,7 +118,7 @@ "id": "efe2fd2d-63ff-4278-9157-8b9110d682e8", "metadata": {}, "source": [ - "

    Begin running steps with Shift + Enter keys.

    " + "

    Begin running steps with Shift + Enter keys.

    " ] }, { @@ -128,9 +128,9 @@ "source": [ "
    \n", "\n", - "

    1.2 Getting Data for This Demo

    \n", + "

    1.2 Getting Data for This Demo

    \n", "\n", - "

    Here, we will get the time series data which is available in the teradataml library and use the same to show the usage of the function.

    " + "

    Here, we will get the time series data which is available in the teradataml library and use the same to show the usage of the function.

    " ] }, { @@ -149,8 +149,8 @@ "metadata": {}, "source": [ "
    \n", - "2. Data Exploration\n", - "

    Create a \"Virtual DataFrame\" that points to the data set in Vantage. Check the shape of the dataframe as check the datatype of all the columns of the dataframe.

    " + "2. Data Exploration\n", + "

    Create a \"Virtual DataFrame\" that points to the data set in Vantage. Check the shape of the dataframe as check the datatype of all the columns of the dataframe.

    " ] }, { @@ -167,14 +167,30 @@ }, { "cell_type": "markdown", - "id": "4a2f79a0-eb16-4c3b-bec2-c313e97ac940", + "id": "70d2f836-93e4-4fe3-a9de-2815d95bed5f", "metadata": {}, "source": [ "
    \n", - "3. BinarySeriesOp\n", - "

    The BinarySeriesOp() function can be configured to operate in one of three input modes - ONE2ONE, MANY2ONE, and MATCH. These modes determine the number of primary series and number of secondary series involved in the operation, as well as determining how the primary and secondary series will be matched together.

    \n", - "\n", - "

    The first step is to convert the data into TDSeries, which is required for the input time series which are passed to the BinarySeriesOp function.

    \n" + "3. BinarySeriesOp\n", + "

    The BinarySeriesOp() function can be configured to operate in one of three input modes - ONE2ONE, MANY2ONE, and MATCH. These modes determine the number of primary series and number of secondary series involved in the operation, as well as determining how the primary and secondary series will be matched together. Detailed help can be found by passing function name to built-in help function.

    " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c17c456b-edfd-44a3-a1ec-8596b7abd9a2", + "metadata": {}, + "outputs": [], + "source": [ + "help(BinarySeriesOp)" + ] + }, + { + "cell_type": "markdown", + "id": "874fc607-7e68-4ca7-b7b4-227300147749", + "metadata": {}, + "source": [ + "

    The first step is to convert the data into TDSeries, which is required for the input time series which are passed to the BinarySeriesOp function.

    " ] }, { @@ -212,7 +228,7 @@ "id": "d33d2bcc-c87c-4fd4-913a-79e330bd2da2", "metadata": {}, "source": [ - "

    Than, we form the filter expressions to filter the series with id=1.

    " + "

    Than, we form the filter expressions to filter the series with id=1.

    " ] }, { @@ -231,7 +247,7 @@ "id": "0bd6859c-b85a-4599-90f5-1f6621980fab", "metadata": {}, "source": [ - "

    We use the BinarySeriesOp to perform addition of two time series of equal size.

    " + "

    We use the BinarySeriesOp to perform addition of two time series of equal size.

    " ] }, { @@ -257,7 +273,7 @@ "id": "2f66bf78-4ea4-4387-b078-a91f491931e6", "metadata": {}, "source": [ - "

    The result series always inherits the identifiers, series_id from the primary series.

    " + "

    The result series always inherits the identifiers, series_id from the primary series.

    " ] }, { @@ -266,7 +282,7 @@ "metadata": {}, "source": [ "
    \n", - "4. Cleanup" + "4. Cleanup" ] }, { @@ -274,8 +290,8 @@ "id": "a5d460af-1a32-4681-8094-360c3f25dc1f", "metadata": {}, "source": [ - "

    Work Tables

    \n", - "

    The following code will clean up tables created above.

    " + "

    Work Tables

    \n", + "

    The following code will clean up tables created above.

    " ] }, { @@ -306,7 +322,7 @@ "source": [ "
    \n", "\n", - "

    Links:

    \n", + "

    Links:

    \n", "
      \n", "
    • Teradataml Python reference: here
      • \n", "
    • BinarySeriesOp function reference: here
      • \n", From 9620222987e5492dbe96de6ddda8b672faa0ed37 Mon Sep 17 00:00:00 2001 From: shilpa-nalkande Date: Tue, 25 Feb 2025 06:23:16 +0000 Subject: [PATCH 4/4] Updated formating as per dark theme changes --- .../ClearScape_Functions/BinarySeriesOp.ipynb | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb index 6c9fad23..973f2045 100644 --- a/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb +++ b/Recipes/ClearScape_Functions/BinarySeriesOp.ipynb @@ -40,7 +40,7 @@ "id": "6b3a00b4-6661-4c91-9b2d-cb7b0b403140", "metadata": {}, "source": [ - "
      \n", + "
      \n", "1. Initiate a connection to Vantage" ] }, @@ -85,7 +85,7 @@ "id": "ad3dd7b4-831c-4fb3-ab71-719c8c99a71c", "metadata": {}, "source": [ - "
      \n", + "
      \n", "

      1.1 Connect to Vantage

      \n", "

      You will be prompted to provide the password. Enter your password, press the Enter key, and then use the down arrow to go to the next cell.

      " ] @@ -126,7 +126,7 @@ "id": "4a6e3852-3b15-4cbd-844d-e7194b3bd204", "metadata": {}, "source": [ - "
      \n", + "
      \n", "\n", "

      1.2 Getting Data for This Demo

      \n", "\n", @@ -148,7 +148,7 @@ "id": "2a3762ac-ba27-4fa3-adba-d577262a4290", "metadata": {}, "source": [ - "
      \n", + "
      \n", "2. Data Exploration\n", "

      Create a \"Virtual DataFrame\" that points to the data set in Vantage. Check the shape of the dataframe as check the datatype of all the columns of the dataframe.

      " ] @@ -170,7 +170,7 @@ "id": "70d2f836-93e4-4fe3-a9de-2815d95bed5f", "metadata": {}, "source": [ - "
      \n", + "
      \n", "3. BinarySeriesOp\n", "

      The BinarySeriesOp() function can be configured to operate in one of three input modes - ONE2ONE, MANY2ONE, and MATCH. These modes determine the number of primary series and number of secondary series involved in the operation, as well as determining how the primary and secondary series will be matched together. Detailed help can be found by passing function name to built-in help function.

      " ] @@ -281,7 +281,7 @@ "id": "c25fdaf1-098b-4dea-984b-579c6412557e", "metadata": {}, "source": [ - "
      \n", + "
      \n", "4. Cleanup" ] }, @@ -320,7 +320,7 @@ "id": "4317a6cf-1479-4aa8-b30a-ee0a3b5231a8", "metadata": {}, "source": [ - "
      \n", + "
      \n", "\n", "

      Links:

      \n", "
        \n", @@ -335,7 +335,7 @@ "id": "b2dcca28-5de5-44d7-88cb-45a12153b3f8", "metadata": {}, "source": [ - "