CUQ-5: Mortgage example using gQuant.

[avolkov1]

Draft pull request for review. I still need to clean up the code and finish up the jupyter notebook. But implementation wise it's all there.

[avolkov1]

Review my current progress so far.
https://github.com/avolkov1/gQuant/tree/CUQ-5-Mortgage-Example-Using-qQuant/notebook/mortgage_e2e_gquant

Jupyter notebook:

• mortgage_e2e_gquant.ipynb

modules:

• mortgage_common.py
• mortgage_gquant_plugins.py

scripts to run (python <script>):

• mortgage_run_workflow_local.py
• mortgage_run_workflow_daskdistrib.py

TODO before ready to merge:
1.
Investigate the TypeError being thrown in node.py
https://github.com/rapidsai/gQuant/blob/b94c987516f2b60118b4147f179541ec86574247/gquant/dataframe_flow/node.py#L215

Type checking not being handled correctly for a corner case with cudf dataframes when cudf.read_csv is used. The cudf.read_csv differs a bit from pandas version hence I think type mismatch.

Finish the notebook. Add a conclusion/summary, etc.

Just go over the code and add docstrings, comments, etc.

[yidong72]

there are some little things that need to be fixed like qquant and it is better to remove those commented out code for readability.

[avolkov1]

I will push the latest edits very soon. Been working on the various corner cases with logger in dask distributed workers when running in jupyter lab/notebook.

[avolkov1]

I squashed and pushed all the latest changes.
avolkov1@3276b57

If you cloned my CUQ-5-Mortgage-Example-Using-qQuant branch, you might have to delete and re-clone.

I finished the TODOs I listed for myself.

1. Investigate the TypeError being thrown in node.py

https://github.com/avolkov1/gQuant/blob/3276b578dadedad64809ae4171cfb83644224b71/gquant/dataframe_flow/node.py#L198-L221

You have to be careful comparing types. Comparing a pandas dtype with numpy dtype throws a type error. Convert to common comparison type i.e. string then compare.

2. Finished the notebook.
   https://github.com/avolkov1/gQuant/blob/CUQ-5-Mortgage-Example-Using-qQuant/notebook/mortgage_e2e_gquant/mortgage_e2e_gquant.ipynb

3. Just go over the code and add docstrings, comments, etc.
   I still need/should add docstrings to various functions and classes.

[yidong72]

It looks good to me now.

I am going to merge it.

[avolkov1]

Thanks!

Going over the code today adding docstrings to various functions and classes, and cleaning up the logger logic a bit. I'll make another PR on another branch when I finish.