notebooks/training_with_remote_filesystem.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdfec37b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e814db9",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_hugectr_training-with-hdfs/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# HugeCTR Training with Remote File System Example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dfa44b99",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7166fb3",
   "metadata": {},
   "source": [
    "HugeCTR supports reading Parquet data, loading and saving models from/to remote file systems like HDFS, AWS S3, and GCS. Users can read their data stored in these remote file systems and train with it. And after training, users can choose to dump the trained parameters and optimizer states into these file systems."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025dd2e5",
   "metadata": {},
   "source": [
    "## Setup HugeCTR\n",
    "\n",
    "To setup the environment, refer to [HugeCTR Example Notebooks](../notebooks) and follow the instructions there before running the following."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4e0b59",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Training with HDFS Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4483d4-c559-48df-acc7-665e350d4b44",
   "metadata": {},
   "source": [
    "Hadoop is not pre-installe din the Merlin Training Container. To help you build and install HDFS, we provide a script [here](https://github.com/NVIDIA-Merlin/HugeCTR/tree/master/sbin). Please build and install Hadoop using these two scripts. Make sure you have hadoop installed in your Container by running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "74f2ed11-6379-4232-96f3-0bfc46b84db7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hadoop 3.3.2\n",
      "Source code repository https://github.com/apache/hadoop.git -r 0bcb014209e219273cb6fd4152df7df713cbac61\n",
      "Compiled by root on 2022-07-25T09:53Z\n",
      "Compiled with protoc 3.7.1\n",
      "From source with checksum 4b40fff8bb27201ba07b6fa5651217fb\n",
      "This command was run using /opt/hadoop/share/hadoop/common/hadoop-common-3.3.2.jar\n"
     ]
    }
   ],
   "source": [
    "!hadoop version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "399717d1",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19c4321d-c27a-481a-89d9-a2360e1b1fc0",
   "metadata": {},
   "source": [
    "Users can use the [DataSourceParams](https://nvidia-merlin.github.io/HugeCTR/master/api/python_interface.html#data-source-api) to setup file system configurations. Currently, we support `Local`, `HDFS`, `S3`, and `GCS`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ce1453-379f-4789-865f-be7d17e515f6",
   "metadata": {},
   "source": [
    "**Firstly, we want to make sure that we have train and validation datasets ready:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f81abc7b-1600-4545-b5c4-d2d2f9eaf0c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 8 items\n",
      "-rw-r--r--   1 root supergroup  112247365 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_0.parquet\n",
      "-rw-r--r--   1 root supergroup  112243637 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_1.parquet\n",
      "-rw-r--r--   1 root supergroup  112251207 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_2.parquet\n",
      "-rw-r--r--   1 root supergroup  112241764 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_3.parquet\n",
      "-rw-r--r--   1 root supergroup  112247838 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_4.parquet\n",
      "-rw-r--r--   1 root supergroup  112244076 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_5.parquet\n",
      "-rw-r--r--   1 root supergroup  112253553 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_6.parquet\n",
      "-rw-r--r--   1 root supergroup  112249557 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_7.parquet\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -ls hdfs://10.19.172.76:9000/dlrm_parquet/train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3e09246d-e14b-47ff-b43b-b7e3a285ad78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 items\n",
      "-rw-r--r--   1 root supergroup  112239093 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/val/gen_0.parquet\n",
      "-rw-r--r--   1 root supergroup  112249156 2022-07-27 06:19 hdfs://10.19.172.76:9000/dlrm_parquet/val/gen_1.parquet\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -ls hdfs://10.19.172.76:9000/dlrm_parquet/val"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3b8175-f2a8-41b4-ad24-a9ee60027e8e",
   "metadata": {},
   "source": [
    "**Secondly, create `file_list.txt and file_list_test.txt`:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1ee9b926-ce91-4450-8acd-6d1181438bfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir /dlrm_parquet\n",
    "!mkdir /dlrm_parquet/train\n",
    "!mkdir /dlrm_parquet/val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "eab450f3-3a33-4446-9667-856c3f390fdd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /dlrm_parquet/file_list.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /dlrm_parquet/file_list.txt\n",
    "8\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_0.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_1.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_2.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_3.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_4.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_5.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_6.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/train/gen_7.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cb442cb7-12c3-4a77-9ee3-bb8813e40982",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /dlrm_parquet/file_list_test.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /dlrm_parquet/file_list_test.txt\n",
    "2\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/val/gen_0.parquet\n",
    "hdfs://10.19.172.76:9000/dlrm_parquet/val/gen_1.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeec5dfd-019e-46ad-a796-40a5ef0579d7",
   "metadata": {},
   "source": [
    "**Lastly, create `_metadata.json` for both train and validation dataset to specify the feature information of your dataset:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3ae0075a-24ae-4548-9a46-85bd958b62a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /dlrm_parquet/train/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /dlrm_parquet/train/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"./dlrm_parquet/train/gen_0.parquet\", \"num_rows\":1000000}, {\"file_name\": \"./dlrm_parquet/train/gen_1.parquet\", \"num_rows\":1000000}, \n",
    "                 {\"file_name\": \"./dlrm_parquet/train/gen_2.parquet\", \"num_rows\":1000000}, {\"file_name\": \"./dlrm_parquet/train/gen_3.parquet\", \"num_rows\":1000000}, \n",
    "                 {\"file_name\": \"./dlrm_parquet/train/gen_4.parquet\", \"num_rows\":1000000}, {\"file_name\": \"./dlrm_parquet/train/gen_5.parquet\", \"num_rows\":1000000}, \n",
    "                 {\"file_name\": \"./dlrm_parquet/train/gen_6.parquet\", \"num_rows\":1000000}, {\"file_name\": \"./dlrm_parquet/train/gen_7.parquet\", \"num_rows\":1000000} ], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, \n",
    "            {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, \n",
    "            {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, \n",
    "           {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, {\"col_name\": \"C19\", \"index\":19}, \n",
    "           {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, \n",
    "           {\"col_name\": \"C23\", \"index\":23}, {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, \n",
    "           {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "           {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, \n",
    "           {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, {\"col_name\": \"C34\", \"index\":34}, \n",
    "           {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, \n",
    "           {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0bbbeec7-6861-45e0-b680-035709ff63d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /dlrm_parquet/val/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /dlrm_parquet/val/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"./dlrm_parquet/val/gen_0.parquet\", \"num_rows\":1000000}, \n",
    "                 {\"file_name\": \"./dlrm_parquet/val/gen_1.parquet\", \"num_rows\":1000000} ], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, \n",
    "            {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, \n",
    "            {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, \n",
    "           {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, {\"col_name\": \"C19\", \"index\":19}, \n",
    "           {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, \n",
    "           {\"col_name\": \"C23\", \"index\":23}, {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, \n",
    "           {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "           {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, \n",
    "           {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, {\"col_name\": \"C34\", \"index\":34}, \n",
    "           {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, \n",
    "           {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "814a4622-9ee4-402e-a4bf-34ef3353953a",
   "metadata": {},
   "source": [
    "### Training a DLRM model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3999dd8-dbf0-4f72-8931-c3789bcbbed9",
   "metadata": {},
   "source": [
    "**Important APIs used in the following script:**\n",
    "1. We use the [DataSourceParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datasourceparams-class) to define the remote file system to read data from\n",
    "2. In [DataReaderParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datareaderparams), we specify the `DataSourceParams`.\n",
    "3. In [fit()](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#fit-method) method, we specify HDFS path in the `snapshot_prefix` parameters to dump trained models to HDFS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "02510bae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_with_hdfs.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_with_hdfs.py\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "from hugectr.data import DataSourceParams\n",
    "\n",
    "# Create a file system configuration \n",
    "data_source_params = DataSourceParams(\n",
    "    source = hugectr.DataSourceType_t.HDFS, #use HDFS\n",
    "    server = '10.19.172.76', #your HDFS namenode IP\n",
    "    port = 9000, #your HDFS namenode port\n",
    ")\n",
    "\n",
    "# DLRM train\n",
    "solver = hugectr.CreateSolver(max_eval_batches = 1280,\n",
    "                              batchsize_eval = 1024,\n",
    "                              batchsize = 1024,\n",
    "                              lr = 0.01,\n",
    "                              vvgpu = [[1]],\n",
    "                              i64_input_key = True,\n",
    "                              use_mixed_precision = False,\n",
    "                              repeat_dataset = True,\n",
    "                              use_cuda_graph = False)\n",
    "reader = hugectr.DataReaderParams(data_reader_type = hugectr.DataReaderType_t.Parquet,\n",
    "                                  source = [\"/dlrm_parquet/file_list.txt\"],\n",
    "                                  eval_source = \"/dlrm_parquet/file_list_test.txt\",\n",
    "                                  slot_size_array = [405274, 72550, 55008, 222734, 316071, 156265, 220243, 200179, 234566, 335625, 278726, 263070, 312542, 203773, 145859, 117421, 78140, 3648, 156308, 94562, 357703, 386976, 238046, 230917, 292, 156382],\n",
    "                                  data_source_params = data_source_params, #file system config for data reading\n",
    "                                  check_type = hugectr.Check_t.Non)\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type = hugectr.Optimizer_t.SGD,\n",
    "                                    update_type = hugectr.Update_t.Local,\n",
    "                                    atomic_update = True)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(hugectr.Input(label_dim = 1, label_name = \"label\",\n",
    "                        dense_dim = 13, dense_name = \"dense\",\n",
    "                        data_reader_sparse_param_array = \n",
    "                        [hugectr.DataReaderSparseParam(\"data1\", 1, True, 26)]))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash,\n",
    "                            workspace_size_per_gpu_in_mb = 10720,\n",
    "                            embedding_vec_size = 128,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding1\",\n",
    "                            bottom_name = \"data1\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"dense\"],\n",
    "                            top_names = [\"fc1\"],\n",
    "                            num_output=512))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc1\"],\n",
    "                            top_names = [\"relu1\"]))                           \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu1\"],\n",
    "                            top_names = [\"fc2\"],\n",
    "                            num_output=256))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc2\"],\n",
    "                            top_names = [\"relu2\"]))                            \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu2\"],\n",
    "                            top_names = [\"fc3\"],\n",
    "                            num_output=128))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc3\"],\n",
    "                            top_names = [\"relu3\"]))                              \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Interaction,\n",
    "                            bottom_names = [\"relu3\",\"sparse_embedding1\"],\n",
    "                            top_names = [\"interaction1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"interaction1\"],\n",
    "                            top_names = [\"fc4\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc4\"],\n",
    "                            top_names = [\"relu4\"]))                              \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu4\"],\n",
    "                            top_names = [\"fc5\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc5\"],\n",
    "                            top_names = [\"relu5\"]))                              \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu5\"],\n",
    "                            top_names = [\"fc6\"],\n",
    "                            num_output=512))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc6\"],\n",
    "                            top_names = [\"relu6\"]))                               \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu6\"],\n",
    "                            top_names = [\"fc7\"],\n",
    "                            num_output=256))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc7\"],\n",
    "                            top_names = [\"relu7\"]))                                                                              \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu7\"],\n",
    "                            top_names = [\"fc8\"],\n",
    "                            num_output=1))                                                                                           \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "                            bottom_names = [\"fc8\", \"label\"],\n",
    "                            top_names = [\"loss\"]))\n",
    "model.compile()\n",
    "model.summary()\n",
    "\n",
    "model.fit(max_iter = 2020, display = 200, eval_interval = 1000, snapshot = 2000, snapshot_prefix = \"hdfs://10.19.172.76:9000/model/dlrm/\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b29f1042",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 3.8\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][07:51:52.502][WARNING][RK0][main]: The model name is not specified when creating the solver.\n",
      "[HCTR][07:51:52.502][INFO][RK0][main]: Global seed is 3218787045\n",
      "[HCTR][07:51:52.505][INFO][RK0][main]: Device to NUMA mapping:\n",
      "  GPU 1 ->  node 0\n",
      "[HCTR][07:51:55.607][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][07:51:55.607][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][07:51:55.609][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][07:51:56.529][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][07:51:56.530][INFO][RK0][main]: Device 1: NVIDIA A10\n",
      "[HCTR][07:51:56.531][INFO][RK0][main]: num of DataReader workers for train: 1\n",
      "[HCTR][07:51:56.531][INFO][RK0][main]: num of DataReader workers for eval: 1\n",
      "[HCTR][07:51:57.695][INFO][RK0][main]: Using Hadoop Cluster 10.19.172.76:9000\n",
      "[HCTR][07:51:57.740][INFO][RK0][main]: Using Hadoop Cluster 10.19.172.76:9000\n",
      "[HCTR][07:51:57.740][INFO][RK0][main]: Vocabulary size: 5242880\n",
      "[HCTR][07:51:57.741][INFO][RK0][main]: max_vocabulary_size_per_gpu_=21954560\n",
      "[HCTR][07:51:57.755][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "===================================================Model Compile===================================================\n",
      "[HCTR][07:52:04.336][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][07:52:04.411][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][07:52:04.413][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Warm-up done\n",
      "===================================================Model Summary===================================================\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: label                                   Dense                         Sparse                        \n",
      "label                                   dense                          data1                         \n",
      "(None, 1)                               (None, 13)                              \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "DistributedSlotSparseEmbeddingHash      data1                         sparse_embedding1             (None, 26, 128)               \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            dense                         fc1                           (None, 512)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (None, 512)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu1                         fc2                           (None, 256)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc2                           relu2                         (None, 256)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu2                         fc3                           (None, 128)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc3                           relu3                         (None, 128)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Interaction                             relu3                         interaction1                  (None, 480)                   \n",
      "                                        sparse_embedding1                                                                         \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            interaction1                  fc4                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc4                           relu4                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu4                         fc5                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc5                           relu5                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu5                         fc6                           (None, 512)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc6                           relu6                         (None, 512)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu6                         fc7                           (None, 256)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc7                           relu7                         (None, 256)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu7                         fc8                           (None, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  fc8                           loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Use non-epoch mode with number of iterations: 2020\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Evaluation interval: 1000, snapshot interval: 2000\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: False\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: lr: 0.010000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Training source file: /dlrm_parquet/file_list.txt\n",
      "[HCTR][07:52:04.415][INFO][RK0][main]: Evaluation source file: /dlrm_parquet/file_list_test.txt\n",
      "[HCTR][07:52:05.134][INFO][RK0][main]: Iter: 200 Time(200 iters): 0.716815s Loss: 0.69327 lr:0.01\n",
      "[HCTR][07:52:05.856][INFO][RK0][main]: Iter: 400 Time(200 iters): 0.719486s Loss: 0.693207 lr:0.01\n",
      "[HCTR][07:52:06.608][INFO][RK0][main]: Iter: 600 Time(200 iters): 0.750294s Loss: 0.693568 lr:0.01\n",
      "[HCTR][07:52:07.331][INFO][RK0][main]: Iter: 800 Time(200 iters): 0.721128s Loss: 0.693352 lr:0.01\n",
      "[HCTR][07:52:09.118][INFO][RK0][main]: Iter: 1000 Time(200 iters): 1.78435s Loss: 0.693352 lr:0.01\n",
      "[HCTR][07:52:11.667][INFO][RK0][main]: Evaluation, AUC: 0.499891\n",
      "[HCTR][07:52:11.668][INFO][RK0][main]: Eval Time for 1280 iters: 2.5486s\n",
      "[HCTR][07:52:12.393][INFO][RK0][main]: Iter: 1200 Time(200 iters): 3.2728s Loss: 0.693178 lr:0.01\n",
      "[HCTR][07:52:13.116][INFO][RK0][main]: Iter: 1400 Time(200 iters): 0.720984s Loss: 0.693292 lr:0.01\n",
      "[HCTR][07:52:13.875][INFO][RK0][main]: Iter: 1600 Time(200 iters): 0.756448s Loss: 0.693053 lr:0.01\n",
      "[HCTR][07:52:14.603][INFO][RK0][main]: Iter: 1800 Time(200 iters): 0.725832s Loss: 0.693433 lr:0.01\n",
      "[HCTR][07:52:16.382][INFO][RK0][main]: Iter: 2000 Time(200 iters): 1.77763s Loss: 0.693193 lr:0.01\n",
      "[HCTR][07:52:18.959][INFO][RK0][main]: Evaluation, AUC: 0.500092\n",
      "[HCTR][07:52:18.959][INFO][RK0][main]: Eval Time for 1280 iters: 2.57548s\n",
      "[HCTR][07:52:19.575][INFO][RK0][main]: Rank0: Write hash table to file\n",
      "[HDFS][INFO]: Write to HDFS /model/dlrm/0_sparse_2000.model/key successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/dlrm/0_sparse_2000.model/emb_vector successfully!\n",
      "[HCTR][07:52:31.132][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][07:52:31.132][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/dlrm/_dense_2000.model successfully!\n",
      "[HCTR][07:52:31.307][INFO][RK0][main]: Dumping dense weights to HDFS, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/dlrm/_opt_dense_2000.model successfully!\n",
      "[HCTR][07:52:31.365][INFO][RK0][main]: Dumping dense optimizer states to HDFS, successful\n",
      "[HCTR][07:52:31.430][INFO][RK0][main]: Finish 2020 iterations with batchsize: 1024 in 27.02s.\n"
     ]
    }
   ],
   "source": [
    "!python train_with_hdfs.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1dec161-b9e7-40b3-9f59-d00130f386ca",
   "metadata": {},
   "source": [
    "**Check that our model files are saved in HDFS:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "93ee7d71-4f51-4011-bde4-600e7cb5b96a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 items\n",
      "drwxr-xr-x   - root supergroup          0 2022-07-27 07:52 hdfs://10.19.172.76:9000/model/dlrm/0_sparse_2000.model\n",
      "-rw-r--r--   3 root supergroup    9479684 2022-07-27 07:52 hdfs://10.19.172.76:9000/model/dlrm/_dense_2000.model\n",
      "-rw-r--r--   3 root supergroup          0 2022-07-27 07:52 hdfs://10.19.172.76:9000/model/dlrm/_opt_dense_2000.model\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -ls hdfs://10.19.172.76:9000/model/dlrm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c03a8ae-77d1-4e68-9b0a-ae4449ac5743",
   "metadata": {},
   "source": [
    "## Training a DCN model with AWS S3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e30f3277",
   "metadata": {},
   "source": [
    "**Before you start:**\n",
    "Please note that AWS S3 SDKs are NOT preinstalled in the NGC docker. To use S3 related functionalites, please do the following steps to customize the building of HugeCTR:\n",
    "1. git clone https://github.com/NVIDIA/HugeCTR.git\n",
    "2. cd HugeCTR\n",
    "3. git submodule update --init --recursive\n",
    "4. mkdir -p build && cd build\n",
    "5. cmake -DCMAKE_BUILD_TYPE=Release -DSM=70 -DENABLE_S3=ON .. #ENABLE_S3 option will install AWS S3 SDKs for you.\n",
    "6. make -j && make install"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bfcdab6-50c8-4c8d-b3fa-0cffb2e8e35e",
   "metadata": {},
   "source": [
    "### Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdc602d7-c9f7-48cb-84e0-1f7b1bf85335",
   "metadata": {},
   "source": [
    "**Create `file_list.txt and file_list_test.txt`:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4979d23b-7392-4a3c-b6a1-960ea7637a16",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p /hugectr-io-test/data/dcn_parquet/train\n",
    "!mkdir -p /hugectr-io-test/data/dcn_parquet/val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2fe17b12-c704-4962-a1ca-ffabd8625a6e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /hugectr-io-test/data/dcn_parquet/file_list.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/file_list.txt\n",
    "16\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_0.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_1.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_2.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_3.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_4.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_5.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_6.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_7.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_8.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_9.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_10.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_11.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_12.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_13.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_14.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/train/gen_15.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "334060b5-74ee-481c-9ab3-1ef65902c2b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n",
    "4\n",
    "s3://hugectr-io-test/data/dcn_parquet/val/gen_0.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/val/gen_1.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/val/gen_2.parquet\n",
    "s3://hugectr-io-test/data/dcn_parquet/val/gen_3.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0c94830e-4284-4a8a-9ca7-ce487ad90f59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /hugectr-io-test/data/dcn_parquet/train/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/train/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_0.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_1.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_2.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_3.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_4.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_5.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_6.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_7.parquet\", \"num_rows\":40960},\n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_8.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_9.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_10.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_11.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_12.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_13.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_14.parquet\", \"num_rows\":40960}, {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/train/gen_15.parquet\", \"num_rows\":40960}], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, \n",
    "            {\"col_name\": \"C19\", \"index\":19}, {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, {\"col_name\": \"C23\", \"index\":23}, \n",
    "            {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "            {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, \n",
    "            {\"col_name\": \"C34\", \"index\":34}, {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "06691beb-01b4-4bd9-b5c5-0cdace72e789",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /hugectr-io-test/data/dcn_parquet/val/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/val/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/val/gen_0.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/val/gen_1.parquet\", \"num_rows\":40960},\n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/val/gen_2.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"s3://hugectr-io-test/data/dcn_parquet/val/gen_3.parquet\", \"num_rows\":40960}], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, \n",
    "            {\"col_name\": \"C19\", \"index\":19}, {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, {\"col_name\": \"C23\", \"index\":23}, \n",
    "            {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "            {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, \n",
    "            {\"col_name\": \"C34\", \"index\":34}, {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0c300b-802d-451e-9e59-8f53c77423cc",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6ddbd3-668b-4f2d-92f6-a65271906af2",
   "metadata": {},
   "source": [
    "**Important APIs used in the following script:**\n",
    "1. We use the [DataSourceParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datasourceparams-class) to define the remote file system to read data from, in this case, S3.\n",
    "2. In [DataReaderParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datareaderparams), we specify the `DataSourceParams`.\n",
    "3. In [fit()](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#fit-method) method, we specify S3 path in the `snapshot_prefix` parameters to dump trained models to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0fbf62db-aa0b-4b69-a043-e19ec994ad03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_with_s3.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_with_s3.py\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "from hugectr.data import DataSourceParams\n",
    "\n",
    "# Create a file system configuration for data reading\n",
    "data_source_params = DataSourceParams(\n",
    "    source = hugectr.FileSystemType_t.S3, #use AWS S3\n",
    "    server = 'us-east-1', #your AWS region\n",
    "    port = 9000, #with be ignored\n",
    ")\n",
    "\n",
    "solver = hugectr.CreateSolver(\n",
    "    max_eval_batches=1280,\n",
    "    batchsize_eval=1024,\n",
    "    batchsize=1024,\n",
    "    lr=0.001,\n",
    "    vvgpu=[[0]],\n",
    "    i64_input_key=True,\n",
    "    repeat_dataset=True,\n",
    ")\n",
    "reader = hugectr.DataReaderParams(\n",
    "    data_reader_type=hugectr.DataReaderType_t.Parquet,\n",
    "    source=[\"/hugectr-io-test/data/dcn_parquet/file_list.txt\"],\n",
    "    eval_source=\"/hugectr-io-test/data/dcn_parquet/file_list_test.txt\",\n",
    "    slot_size_array=[39884,39043,17289,7420,20263,3,7120,1543,39884,39043,17289,7420,20263,3,7120,1543,63,63,39884,39043,17289,7420,20263,3,7120,1543],\n",
    "    data_source_params=data_source_params, # Using the S3 configurations\n",
    "    check_type=hugectr.Check_t.Non,\n",
    ")\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type=hugectr.Optimizer_t.SGD)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(\n",
    "    hugectr.Input(\n",
    "        label_dim=1,\n",
    "        label_name=\"label\",\n",
    "        dense_dim=13,\n",
    "        dense_name=\"dense\",\n",
    "        data_reader_sparse_param_array=[\n",
    "            hugectr.DataReaderSparseParam(\"data1\", 1, True, 26)\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.SparseEmbedding(\n",
    "        embedding_type=hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash,\n",
    "        workspace_size_per_gpu_in_mb=150,\n",
    "        embedding_vec_size=16,\n",
    "        combiner=\"sum\",\n",
    "        sparse_embedding_name=\"sparse_embedding1\",\n",
    "        bottom_name=\"data1\",\n",
    "        optimizer=optimizer,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Reshape,\n",
    "        bottom_names=[\"sparse_embedding1\"],\n",
    "        top_names=[\"reshape1\"],\n",
    "        leading_dim=416,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Concat, bottom_names=[\"reshape1\", \"dense\"], top_names=[\"concat1\"]\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Slice,\n",
    "        bottom_names=[\"concat1\"],\n",
    "        top_names=[\"slice11\", \"slice12\"],\n",
    "        ranges=[(0, 429), (0, 429)],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.MultiCross,\n",
    "        bottom_names=[\"slice11\"],\n",
    "        top_names=[\"multicross1\"],\n",
    "        num_layers=6,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"slice12\"],\n",
    "        top_names=[\"fc1\"],\n",
    "        num_output=1024,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc1\"], top_names=[\"relu1\"])\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Dropout,\n",
    "        bottom_names=[\"relu1\"],\n",
    "        top_names=[\"dropout1\"],\n",
    "        dropout_rate=0.5,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Concat,\n",
    "        bottom_names=[\"dropout1\", \"multicross1\"],\n",
    "        top_names=[\"concat2\"],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"concat2\"],\n",
    "        top_names=[\"fc2\"],\n",
    "        num_output=1,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "        bottom_names=[\"fc2\", \"label\"],\n",
    "        top_names=[\"loss\"],\n",
    "    )\n",
    ")\n",
    "model.compile()\n",
    "model.summary()\n",
    "\n",
    "model.fit(max_iter = 1100, display = 100, eval_interval = 500, snapshot = 1000, snapshot_prefix = \"https://s3.us-east-1.amazonaws.com/hugectr-io-test/pipeline_test/dcn_model/\")\n",
    "model.graph_to_json(graph_config_file = \"dcn.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "29838415-6be5-46f8-9ee7-435b0910cee0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 4.1\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][06:54:55.819][WARNING][RK0][main]: The model name is not specified when creating the solver.\n",
      "[HCTR][06:54:55.819][INFO][RK0][main]: Global seed is 569406237\n",
      "[HCTR][06:54:55.822][INFO][RK0][main]: Device to NUMA mapping:\n",
      "  GPU 0 ->  node 0\n",
      "[HCTR][06:54:57.710][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][06:54:57.710][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][06:54:57.710][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][06:54:57.711][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][06:54:57.712][INFO][RK0][main]: Device 0: Tesla V100-SXM2-32GB\n",
      "[HCTR][06:54:57.713][INFO][RK0][main]: num of DataReader workers for train: 1\n",
      "[HCTR][06:54:57.713][INFO][RK0][main]: num of DataReader workers for eval: 1\n",
      "[HCTR][06:54:57.714][INFO][RK0][main]: Using S3 file system backend.\n",
      "[HCTR][06:54:59.762][INFO][RK0][main]: Using S3 file system backend.\n",
      "[HCTR][06:55:01.777][INFO][RK0][main]: Vocabulary size: 397821\n",
      "[HCTR][06:55:01.777][INFO][RK0][main]: max_vocabulary_size_per_gpu_=2457600\n",
      "[HCTR][06:55:01.780][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "===================================================Model Compile===================================================\n",
      "[HCTR][06:55:03.407][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][06:55:03.408][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][06:55:03.409][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][06:55:03.411][INFO][RK0][main]: Warm-up done\n",
      "===================================================Model Summary===================================================\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Model structure on each GPU\n",
      "Label                                   Dense                         Sparse                        \n",
      "label                                   dense                          data1                         \n",
      "(1024,1)                                (1024,13)                               \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "DistributedSlotSparseEmbeddingHash      data1                         sparse_embedding1             (1024,26,16)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding1             reshape1                      (1024,416)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  reshape1                      concat1                       (1024,429)                    \n",
      "                                        dense                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Slice                                   concat1                       slice11                       (1024,429)                    \n",
      "                                                                      slice12                       (1024,429)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "MultiCross                              slice11                       multicross1                   (1024,429)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            slice12                       fc1                           (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu1                         dropout1                      (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  dropout1                      concat2                       (1024,1453)                   \n",
      "                                        multicross1                                                                               \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            concat2                       fc2                           (1024,1)                      \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  fc2                           loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Use non-epoch mode with number of iterations: 1100\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Evaluation interval: 500, snapshot interval: 1000\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: True\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: lr: 0.001000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Training source file: /hugectr-io-test/data/dcn_parquet/file_list.txt\n",
      "[HCTR][06:55:03.412][INFO][RK0][main]: Evaluation source file: /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n",
      "[HCTR][06:55:04.668][INFO][RK0][main]: Iter: 100 Time(100 iters): 1.25574s Loss: 0.712926 lr:0.001\n",
      "[HCTR][06:55:06.839][INFO][RK0][main]: Iter: 200 Time(100 iters): 2.16987s Loss: 0.701584 lr:0.001\n",
      "[HCTR][06:55:08.066][INFO][RK0][main]: Iter: 300 Time(100 iters): 1.22653s Loss: 0.696012 lr:0.001\n",
      "[HCTR][06:55:10.229][INFO][RK0][main]: Iter: 400 Time(100 iters): 2.16121s Loss: 0.698167 lr:0.001\n",
      "[HCTR][06:55:11.653][INFO][RK0][main]: Iter: 500 Time(100 iters): 1.42367s Loss: 0.695641 lr:0.001\n",
      "[HCTR][06:55:29.727][INFO][RK0][main]: Evaluation, AUC: 0.500979\n",
      "[HCTR][06:55:29.727][INFO][RK0][main]: Eval Time for 1280 iters: 18.0735s\n",
      "[HCTR][06:55:32.311][INFO][RK0][main]: Iter: 600 Time(100 iters): 20.6575s Loss: 0.696028 lr:0.001\n",
      "[HCTR][06:55:33.349][INFO][RK0][main]: Iter: 700 Time(100 iters): 1.03696s Loss: 0.693602 lr:0.001\n",
      "[HCTR][06:55:35.089][INFO][RK0][main]: Iter: 800 Time(100 iters): 1.73903s Loss: 0.693618 lr:0.001\n",
      "[HCTR][06:55:36.191][INFO][RK0][main]: Iter: 900 Time(100 iters): 1.10101s Loss: 0.696232 lr:0.001\n",
      "[HCTR][06:55:37.789][INFO][RK0][main]: Iter: 1000 Time(100 iters): 1.59704s Loss: 0.693168 lr:0.001\n",
      "[HCTR][06:55:53.378][INFO][RK0][main]: Evaluation, AUC: 0.50103\n",
      "[HCTR][06:55:53.378][INFO][RK0][main]: Eval Time for 1280 iters: 15.5882s\n",
      "[HCTR][06:55:53.378][INFO][RK0][main]: Using S3 file system backend.\n",
      "[HCTR][06:55:55.410][INFO][RK0][main]: Rank0: Write hash table to file\n",
      "[HCTR][06:55:56.473][DEBUG][RK0][main]: Successfully write to AWS S3 location:  https://s3.us-east-1.amazonaws.com/hugectr-io-test/pipeline_test/dcn_model/0_sparse_1000.model/key\n",
      "[HCTR][06:55:57.348][DEBUG][RK0][main]: Successfully write to AWS S3 location:  https://s3.us-east-1.amazonaws.com/hugectr-io-test/pipeline_test/dcn_model/0_sparse_1000.model/emb_vector\n",
      "[HCTR][06:55:57.360][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][06:55:57.360][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HCTR][06:55:57.361][INFO][RK0][main]: Using S3 file system backend.\n",
      "[HCTR][06:56:00.462][DEBUG][RK0][main]: Successfully write to AWS S3 location:  https://s3.us-east-1.amazonaws.com/hugectr-io-test/pipeline_test/dcn_model/_dense_1000.model\n",
      "[HCTR][06:56:00.467][INFO][RK0][main]: Dumping dense weights to file, successful\n",
      "[HCTR][06:56:00.467][INFO][RK0][main]: Using S3 file system backend.\n",
      "[HCTR][06:56:02.839][DEBUG][RK0][main]: Successfully write to AWS S3 location:  https://s3.us-east-1.amazonaws.com/hugectr-io-test/pipeline_test/dcn_model/_opt_dense_1000.model\n",
      "[HCTR][06:56:02.843][INFO][RK0][main]: Dumping dense optimizer states to file, successful\n",
      "[HCTR][06:56:06.987][INFO][RK0][main]: Finish 1100 iterations with batchsize: 1024 in 63.58s.\n",
      "[HCTR][06:56:06.988][INFO][RK0][main]: Save the model graph to dcn.json successfully\n"
     ]
    }
   ],
   "source": [
    "!python train_with_s3.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4bb2959-219f-4a2e-b354-bae387fd5a0d",
   "metadata": {},
   "source": [
    "## Training a DCN model with Google Cloud Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91273c8a-510b-4dd5-8f6d-7d88888faef8",
   "metadata": {},
   "source": [
    "**Before you start:**\n",
    "Please note that GCS SDK are NOT preinstalled in the NGC docker. To use GCS related functionalites, please do the following steps to customize the building of HugeCTR:\n",
    "1. git clone https://github.com/NVIDIA/HugeCTR.git\n",
    "2. cd HugeCTR\n",
    "3. git submodule update --init --recursive\n",
    "4. mkdir -p build && cd build\n",
    "5. cmake -DCMAKE_BUILD_TYPE=Release -DSM=70 -DENABLE_GCS=ON .. #ENABLE_GCS option will install GCS SDKs for you.\n",
    "6. make -j && make install"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67e5b4b8-c98e-4aa8-857f-4bed3e9aab4a",
   "metadata": {},
   "source": [
    "### Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85b14051-f904-4d8b-9248-da0d1f9d5843",
   "metadata": {},
   "source": [
    "**Create `file_list.txt and file_list_test.txt`:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "000d1d50-22d3-4ac0-827c-2d396af75f7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p /hugectr-io-test/data/dcn_parquet/train\n",
    "!mkdir -p /hugectr-io-test/data/dcn_parquet/val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e4d7cba3-fa8c-4766-8824-290d44b44e3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /hugectr-io-test/data/dcn_parquet/file_list.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/file_list.txt\n",
    "16\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_0.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_1.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_2.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_3.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_4.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_5.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_6.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_7.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_8.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_9.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_10.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_11.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_12.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_13.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_14.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/train/gen_15.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b0be2965-45c0-4d6b-ad01-543bdc2ab111",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n",
    "4\n",
    "gs://hugectr-io-test/data/dcn_parquet/val/gen_0.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/val/gen_1.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/val/gen_2.parquet\n",
    "gs://hugectr-io-test/data/dcn_parquet/val/gen_3.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "30ead2b8-658a-4ab2-97fa-47cca12843f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /hugectr-io-test/data/dcn_parquet/train/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/train/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_0.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_1.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_2.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_3.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_4.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_5.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_6.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_7.parquet\", \"num_rows\":40960},\n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_8.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_9.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_10.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_11.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_12.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_13.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_14.parquet\", \"num_rows\":40960}, {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/train/gen_15.parquet\", \"num_rows\":40960}], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, \n",
    "            {\"col_name\": \"C19\", \"index\":19}, {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, {\"col_name\": \"C23\", \"index\":23}, \n",
    "            {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "            {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, \n",
    "            {\"col_name\": \"C34\", \"index\":34}, {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "175feb1d-8773-41c4-8c52-ad3bae1086cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /hugectr-io-test/data/dcn_parquet/val/_metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile /hugectr-io-test/data/dcn_parquet/val/_metadata.json\n",
    "{ \"file_stats\": [{\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/val/gen_0.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/val/gen_1.parquet\", \"num_rows\":40960},\n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/val/gen_2.parquet\", \"num_rows\":40960}, \n",
    "                 {\"file_name\": \"gs://hugectr-io-test/data/dcn_parquet/val/gen_3.parquet\", \"num_rows\":40960}], \n",
    "  \"labels\": [{\"col_name\": \"label0\", \"index\":0} ], \n",
    "  \"conts\": [{\"col_name\": \"C1\", \"index\":1}, {\"col_name\": \"C2\", \"index\":2}, {\"col_name\": \"C3\", \"index\":3}, {\"col_name\": \"C4\", \"index\":4}, {\"col_name\": \"C5\", \"index\":5}, {\"col_name\": \"C6\", \"index\":6}, \n",
    "            {\"col_name\": \"C7\", \"index\":7}, {\"col_name\": \"C8\", \"index\":8}, {\"col_name\": \"C9\", \"index\":9}, {\"col_name\": \"C10\", \"index\":10}, {\"col_name\": \"C11\", \"index\":11}, {\"col_name\": \"C12\", \"index\":12}, \n",
    "            {\"col_name\": \"C13\", \"index\":13} ], \n",
    "  \"cats\": [{\"col_name\": \"C14\", \"index\":14}, {\"col_name\": \"C15\", \"index\":15}, {\"col_name\": \"C16\", \"index\":16}, {\"col_name\": \"C17\", \"index\":17}, {\"col_name\": \"C18\", \"index\":18}, \n",
    "            {\"col_name\": \"C19\", \"index\":19}, {\"col_name\": \"C20\", \"index\":20}, {\"col_name\": \"C21\", \"index\":21}, {\"col_name\": \"C22\", \"index\":22}, {\"col_name\": \"C23\", \"index\":23}, \n",
    "            {\"col_name\": \"C24\", \"index\":24}, {\"col_name\": \"C25\", \"index\":25}, {\"col_name\": \"C26\", \"index\":26}, {\"col_name\": \"C27\", \"index\":27}, {\"col_name\": \"C28\", \"index\":28}, \n",
    "            {\"col_name\": \"C29\", \"index\":29}, {\"col_name\": \"C30\", \"index\":30}, {\"col_name\": \"C31\", \"index\":31}, {\"col_name\": \"C32\", \"index\":32}, {\"col_name\": \"C33\", \"index\":33}, \n",
    "            {\"col_name\": \"C34\", \"index\":34}, {\"col_name\": \"C35\", \"index\":35}, {\"col_name\": \"C36\", \"index\":36}, {\"col_name\": \"C37\", \"index\":37}, {\"col_name\": \"C38\", \"index\":38}, {\"col_name\": \"C39\", \"index\":39} ] }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb17b86c-101f-461c-9bc8-56f0fa7e02d7",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0c674b6-3ccd-4d8b-81f3-c74026fc46a4",
   "metadata": {},
   "source": [
    "**Important APIs used in the following script:**\n",
    "1. We use the [DataSourceParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datasourceparams-class) to define the remote file system to read data from, in this case, GCS.\n",
    "2. In [DataReaderParams](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#datareaderparams), we specify the `DataSourceParams`.\n",
    "3. In [fit()](https://nvidia-merlin.github.io/HugeCTR/main/api/python_interface.html#fit-method) method, we specify GCS path in the `snapshot_prefix` parameters to dump trained models to GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "05525093-0120-4dd1-926b-8248d6169f02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/gcs_key.json\n"
     ]
    }
   ],
   "source": [
    "#You need to set the GCP credentials environmental variable to access the GCS.\n",
    "\n",
    "%env GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/gcs_key.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4fe99bf1-ac3b-4422-ad4f-4a8e1a631023",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_with_gcs.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_with_gcs.py\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "from hugectr.data import DataSourceParams\n",
    "\n",
    "# Create a file system configuration for data reading\n",
    "data_source_params = DataSourceParams(\n",
    "    source = hugectr.FileSystemType_t.GCS, #use Google Cloud Storage\n",
    "    server = 'storage.googleapis.com', #your endpoint override, usually storage.googleapis.com or storage.google.cloud.com\n",
    "    port = 9000, #with be ignored\n",
    ")\n",
    "\n",
    "solver = hugectr.CreateSolver(\n",
    "    max_eval_batches=1280,\n",
    "    batchsize_eval=1024,\n",
    "    batchsize=1024,\n",
    "    lr=0.001,\n",
    "    vvgpu=[[0]],\n",
    "    i64_input_key=True,\n",
    "    repeat_dataset=True,\n",
    ")\n",
    "reader = hugectr.DataReaderParams(\n",
    "    data_reader_type=hugectr.DataReaderType_t.Parquet,\n",
    "    source=[\"/hugectr-io-test/data/dcn_parquet/file_list.txt\"],\n",
    "    eval_source=\"/hugectr-io-test/data/dcn_parquet/file_list_test.txt\",\n",
    "    slot_size_array=[39884,39043,17289,7420,20263,3,7120,1543,39884,39043,17289,7420,20263,3,7120,1543,63,63,39884,39043,17289,7420,20263,3,7120,1543],\n",
    "    data_source_params=data_source_params, # Using the GCS configurations\n",
    "    check_type=hugectr.Check_t.Non,\n",
    ")\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type=hugectr.Optimizer_t.SGD)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(\n",
    "    hugectr.Input(\n",
    "        label_dim=1,\n",
    "        label_name=\"label\",\n",
    "        dense_dim=13,\n",
    "        dense_name=\"dense\",\n",
    "        data_reader_sparse_param_array=[\n",
    "            hugectr.DataReaderSparseParam(\"data1\", 1, True, 26)\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.SparseEmbedding(\n",
    "        embedding_type=hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash,\n",
    "        workspace_size_per_gpu_in_mb=150,\n",
    "        embedding_vec_size=16,\n",
    "        combiner=\"sum\",\n",
    "        sparse_embedding_name=\"sparse_embedding1\",\n",
    "        bottom_name=\"data1\",\n",
    "        optimizer=optimizer,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Reshape,\n",
    "        bottom_names=[\"sparse_embedding1\"],\n",
    "        top_names=[\"reshape1\"],\n",
    "        leading_dim=416,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Concat, bottom_names=[\"reshape1\", \"dense\"], top_names=[\"concat1\"]\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Slice,\n",
    "        bottom_names=[\"concat1\"],\n",
    "        top_names=[\"slice11\", \"slice12\"],\n",
    "        ranges=[(0, 429), (0, 429)],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.MultiCross,\n",
    "        bottom_names=[\"slice11\"],\n",
    "        top_names=[\"multicross1\"],\n",
    "        num_layers=6,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"slice12\"],\n",
    "        top_names=[\"fc1\"],\n",
    "        num_output=1024,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc1\"], top_names=[\"relu1\"])\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Dropout,\n",
    "        bottom_names=[\"relu1\"],\n",
    "        top_names=[\"dropout1\"],\n",
    "        dropout_rate=0.5,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Concat,\n",
    "        bottom_names=[\"dropout1\", \"multicross1\"],\n",
    "        top_names=[\"concat2\"],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"concat2\"],\n",
    "        top_names=[\"fc2\"],\n",
    "        num_output=1,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "        bottom_names=[\"fc2\", \"label\"],\n",
    "        top_names=[\"loss\"],\n",
    "    )\n",
    ")\n",
    "model.compile()\n",
    "model.summary()\n",
    "\n",
    "model.fit(max_iter = 1100, display = 100, eval_interval = 500, snapshot = 1000, snapshot_prefix = \"https://storage.googleapis.com/hugectr-io-test/pipeline_test/\")\n",
    "model.graph_to_json(graph_config_file = \"dcn.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e8ef0130-7d98-4274-9aab-3b7ba08cd991",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 4.1\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][03:15:35.248][WARNING][RK0][main]: The model name is not specified when creating the solver.\n",
      "[HCTR][03:15:35.248][INFO][RK0][main]: Global seed is 1008636636\n",
      "[HCTR][03:15:35.251][INFO][RK0][main]: Device to NUMA mapping:\n",
      "  GPU 0 ->  node 0\n",
      "[HCTR][03:15:37.306][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][03:15:37.306][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][03:15:37.306][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][03:15:37.307][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][03:15:37.308][INFO][RK0][main]: Device 0: Tesla V100-SXM2-32GB\n",
      "[HCTR][03:15:37.308][INFO][RK0][main]: num of DataReader workers for train: 1\n",
      "[HCTR][03:15:37.308][INFO][RK0][main]: num of DataReader workers for eval: 1\n",
      "[HCTR][03:15:37.309][INFO][RK0][main]: Using GCS file system backend.\n",
      "[HCTR][03:15:37.323][INFO][RK0][main]: Using GCS file system backend.\n",
      "[HCTR][03:15:37.328][INFO][RK0][main]: Vocabulary size: 397821\n",
      "[HCTR][03:15:37.329][INFO][RK0][main]: max_vocabulary_size_per_gpu_=2457600\n",
      "[HCTR][03:15:37.331][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "[HCTR][03:15:37.331][WARNING][RK0][main]: using multi-cross v1\n",
      "[HCTR][03:15:37.331][WARNING][RK0][main]: using multi-cross v1\n",
      "===================================================Model Compile===================================================\n",
      "[HCTR][03:15:39.005][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][03:15:39.006][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][03:15:39.008][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][03:15:39.010][INFO][RK0][main]: Warm-up done\n",
      "===================================================Model Summary===================================================\n",
      "[HCTR][03:15:39.010][INFO][RK0][main]: Model structure on each GPU\n",
      "Label                                   Dense                         Sparse                        \n",
      "label                                   dense                          data1                         \n",
      "(1024,1)                                (1024,13)                               \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "DistributedSlotSparseEmbeddingHash      data1                         sparse_embedding1             (1024,26,16)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding1             reshape1                      (1024,416)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  reshape1                      concat1                       (1024,429)                    \n",
      "                                        dense                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Slice                                   concat1                       slice11                       (1024,429)                    \n",
      "                                                                      slice12                       (1024,429)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "MultiCross                              slice11                       multicross1                   (1024,429)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            slice12                       fc1                           (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu1                         dropout1                      (1024,1024)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  dropout1                      concat2                       (1024,1453)                   \n",
      "                                        multicross1                                                                               \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            concat2                       fc2                           (1024,1)                      \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  fc2                           loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Use non-epoch mode with number of iterations: 1100\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Evaluation interval: 500, snapshot interval: 1000\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: True\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: lr: 0.001000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Training source file: /hugectr-io-test/data/dcn_parquet/file_list.txt\n",
      "[HCTR][03:15:39.011][INFO][RK0][main]: Evaluation source file: /hugectr-io-test/data/dcn_parquet/file_list_test.txt\n",
      "[HCTR][03:15:40.236][INFO][RK0][main]: Iter: 100 Time(100 iters): 1.22452s Loss: 0.786299 lr:0.001\n",
      "[HCTR][03:15:41.872][INFO][RK0][main]: Iter: 200 Time(100 iters): 1.6347s Loss: 0.738846 lr:0.001\n",
      "[HCTR][03:15:43.102][INFO][RK0][main]: Iter: 300 Time(100 iters): 1.22938s Loss: 0.711017 lr:0.001\n",
      "[HCTR][03:15:44.736][INFO][RK0][main]: Iter: 400 Time(100 iters): 1.63355s Loss: 0.708317 lr:0.001\n",
      "[HCTR][03:15:45.850][INFO][RK0][main]: Iter: 500 Time(100 iters): 1.11226s Loss: 0.697101 lr:0.001\n",
      "[HCTR][03:15:59.880][INFO][RK0][main]: Evaluation, AUC: 0.501301\n",
      "[HCTR][03:15:59.880][INFO][RK0][main]: Eval Time for 1280 iters: 14.0298s\n",
      "[HCTR][03:16:01.456][INFO][RK0][main]: Iter: 600 Time(100 iters): 15.6054s Loss: 0.698077 lr:0.001\n",
      "[HCTR][03:16:02.201][INFO][RK0][main]: Iter: 700 Time(100 iters): 0.744573s Loss: 0.697804 lr:0.001\n",
      "[HCTR][03:16:03.244][INFO][RK0][main]: Iter: 800 Time(100 iters): 1.04207s Loss: 0.695543 lr:0.001\n",
      "[HCTR][03:16:04.007][INFO][RK0][main]: Iter: 900 Time(100 iters): 0.761465s Loss: 0.695323 lr:0.001\n",
      "[HCTR][03:16:05.289][INFO][RK0][main]: Iter: 1000 Time(100 iters): 1.28151s Loss: 0.695319 lr:0.001\n",
      "[HCTR][03:16:17.647][INFO][RK0][main]: Evaluation, AUC: 0.501347\n",
      "[HCTR][03:16:17.647][INFO][RK0][main]: Eval Time for 1280 iters: 12.3576s\n",
      "[HCTR][03:16:17.647][INFO][RK0][main]: Using GCS file system backend.\n",
      "[HCTR][03:16:17.664][INFO][RK0][main]: Rank0: Write hash table to file\n",
      "[HCTR][03:16:18.623][DEBUG][RK0][main]: Successfully write to GCS location:  https://storage.googleapis.com/hugectr-io-test/pipeline_test/0_sparse_1000.model/key\n",
      "[HCTR][03:16:20.289][DEBUG][RK0][main]: Successfully write to GCS location:  https://storage.googleapis.com/hugectr-io-test/pipeline_test/0_sparse_1000.model/emb_vector\n",
      "[HCTR][03:16:20.294][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][03:16:20.294][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HCTR][03:16:20.294][INFO][RK0][main]: Using GCS file system backend.\n",
      "[HCTR][03:16:21.254][DEBUG][RK0][main]: Successfully write to GCS location:  https://storage.googleapis.com/hugectr-io-test/pipeline_test/_dense_1000.model\n",
      "[HCTR][03:16:21.255][INFO][RK0][main]: Dumping dense weights to file, successful\n",
      "[HCTR][03:16:21.255][INFO][RK0][main]: Using GCS file system backend.\n",
      "[HCTR][03:16:21.803][DEBUG][RK0][main]: Successfully write to GCS location:  https://storage.googleapis.com/hugectr-io-test/pipeline_test/_opt_dense_1000.model\n",
      "[HCTR][03:16:21.804][INFO][RK0][main]: Dumping dense optimizer states to file, successful\n",
      "[HCTR][03:16:22.606][INFO][RK0][main]: Finish 1100 iterations with batchsize: 1024 in 43.60s.\n",
      "[HCTR][03:16:22.607][INFO][RK0][main]: Save the model graph to dcn.json successfully\n"
     ]
    }
   ],
   "source": [
    "!python train_with_gcs.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}