Nemo and xgboost integration

[yidong72]

Create the inital PR to start the revew process. I still need to update the notebook text to make it ready

[GPUtester]

Please update the changelog in order to start CI tests.

View the gpuCI docs here.

[avolkov1]

I think we should squash the commits a bit so it's cleaner going forward what is going on this branch. I rebased as follows:

# $ git remote -vv
# ghub-gquant     git@github.com:rapidsai/gQuant.git (fetch)
# ghub-gquant     git@github.com:rapidsai/gQuant.git (push)

# ghub-gquant is my name for remote of gquant github repo
$ git rebase -Xtheirs ghub-gquant/develop

# above gave me a 5 conflicts I had to resolve during rebase

# first conflict
$ mv modules/custom_port_nodes.py~ghub-gquant_develop modules/custom_port_nodes.py
$ git add modules/custom_port_nodes.py
$ git rebase --continue

# got three more rename/add conflicts
$ mv gquantlab/notebooks/full_example.ipynb~ghub-gquant_develop gquantlab/notebooks/full_example.ipynb
$ mv gquantlab/notebooks/test.ipynb~ghub-gquant_develop gquantlab/notebooks/test.ipynb
$ mv gquantlab/notebooks/test_dask.ipynb~ghub-gquant_develop gquantlab/notebooks/test_dask.ipynb
$ git add gquantlab/notebooks/full_example.ipynb
$ git add gquantlab/notebooks/test.ipynb
$ git add gquantlab/notebooks/test_dask.ipynb
$ git rebase --continue

# last conflict
$ mv taskgraphs/simple_trade.gq.yaml~ghub-gquant_develop taskgraphs/simple_trade.gq.yaml
$ git add taskgraphs/simple_trade.gq.yaml
$ git rebase --continue

After above rebase the develop commit "d82ebef" appears cleanly before all the other commits. I look at the log and take the commit right after d82ebef.

$ git log | less

The SHA for me after d82ebef was 78cbb6d. Yours will be different after d82ebef. You need to find that SHA. The following will squash all the commits from 78cbb6d to latest.

$ export OLD_SHA=78cbb6d
$ git reset --soft $OLD_SHA && git add -u  && git commit --amend --no-edit

And one more rebase to change the commit message.

$ git rebase -i d82ebef
# choose reword
# I deleted all other text and just wrote: "Add nemo examples."

$ git push --force

This is what commit history will look like after above steps:

commit ebce7d2be37a266169bc0c6c65f43b3326b54bd8
Author: Yi Dong <yidong@nvidia.com>
Date:   Wed May 13 16:28:45 2020 -0700

    Add nemo examples.

commit d82ebef2bb11b2849c67b6c0749b9cbc624b626c
Author: yidong72 <43824965+yidong72@users.noreply.github.com>
Date:   Wed Sep 9 16:27:37 2020 -0400

    [REVIEW] gQuant UI, first version (#89)
. 
.
.

In my fork it looks like this: https://github.com/avolkov1/gQuant/commits/branch-nemo

[avolkov1]

Please refer to comments so far. I created a PR (yidong72#2) to refactor gquant nodes for nemo into a python package module. Please take a look and test.

I left a few comments regarding cleaning up the usage of _PortTypesMixin. If it's not needed don't use it.

I'm still reviewing. I haven't looked at all the files yet.

[yidong72]

In my latest commit, I address your comments. I change the UI a bit to add the log console button in the toolbar. Fixed the result tab size issue.

[avolkov1]

A few minor notes and comments. Let's resolve these or take note to fix in the future, and I'll merge.

[yidong72]

done the changes.

[avolkov1]

One more change to fix spurious warning about ignoring delayed_process setting when delayed processing is not applicable.

[avolkov1]

Looks great.

I forgot about the text in "10_nemo_chatbot.ipynb". The example for loading nemo modules in gquant there says

from gquant.plugin_nodes.nemo_util.nemoBaseNode import NeMoBase
import nemo
import nemo.backends.pytorch.tutorials

class EncoderRNNNode(NeMoBase):
    def init(self):
        super().init(nemo.backends.pytorch.tutorials.chatbot.modules.EncoderRNN)

. . .

nemo.asr= %(MODULEPATH)s/asr.py
nemo.common= %(MODULEPATH)s/common.py
nemo.cv= %(MODULEPATH)s/cv.py
nemo.nlp= %(MODULEPATH)s/nlp.py
nemo.simple_gan= %(MODULEPATH)s/simple_gan.py
nemo.tts= %(MODULEPATH)s/tts.py
nemo.tutorials= %(MODULEPATH)s/tutorials.py

Needs to be updated to:

from gquant.dataframe_flow import Node
# within <>/modules/nemo_gquant_modules
from .nemoBaseNode import NeMoBase
import nemo
import nemo.backends.pytorch.tutorials


class EncoderRNNNode(NeMoBase, Node):
    def init(self):
        NeMoBase.init(self, nemo.backends.pytorch.tutorials.chatbot.modules.EncoderRNN)

. . .
# gquantrc file

nemo_modules= %(MODULEPATH)s/nemo_gquant_modules

Functionally everything worked great. I'll merge this and we can fix up the notebook in another PR.

Thanks.