Skip to content

Make VS hail independent for the python API #237

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Open
rocreguant opened this issue Feb 13, 2024 · 0 comments
Open

Make VS hail independent for the python API #237

rocreguant opened this issue Feb 13, 2024 · 0 comments

Comments

@rocreguant
Copy link
Collaborator

No description provided.

NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Created standalone class for Random Forest Models (#237)
5edfbfa 
FEAT: Implemented RF class method for fitting the model

FEAT: Implemented RF class method for obtaining importance analysis
from a fitted RF

FEAT: Implemented RF class method for returning oob error

FEAT: Implemented RF class method for obtaining FDR
from a fitted model

FEAT: Implemented RF class method for exporting forest to JSON

REFACTOR: Make RF model available at package level

CHORE: Added type checking to all methods
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Updated varspark python wrapper (#237)
80a9c59 
REFACTOR: Removed FeatureSource and
ImportanceAnalysis classes from core

REFACTOR: Added FeatureSource import so features
can be returned as a class instantiation
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Created standalone FeatureSource class in separate file (#237)
23520ec 
REFACTOR: Removed imp analysis and model training

FEAT: Added conversion from feature to RDD (python)

FEAT: Added conversion from feature to RDD (scala)

CHORE: Added type checking
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
REFACTOR: Remove unecessary hail import for hail rf wrapper
4560998 
due to import order warning (#237)
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Created standalone ImportanceAnalysis class in
b8b39fd 
separate wrapper file (#237)

REFACTOR: Updated important_variables and variable_importance
methods to convert to pandas DataFrames
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Updated ImportanceAnalysis scala class (#237)
4bfaac9 
REFACTOR: Removed model training from object instantation and
updated class to accept a model as a parameter

REFACTOR: Added normalisation as an optional parameter for
variable importance methods

FEAT: Updated variableImportance method to include splitCount in return as it is required for local FDR analysis
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Created scala function that trains a forest
0fc736f 
and passes back to python context (#237)
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
REFACTOR: Removed model definition and training
ea069d6 
from importAnalysis method of AnalyticsFunctions (#237)
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Create no-hail equivalent of JSON model export (#237)
e08f12a 
FIX: Update export function to process trees in batches,
instead of collecting the whole forest as a map as this
led to OOM errors on large forests
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
REFACTOR: Update importance API test cases to reflect changes (#237)
ddc5912
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
REFACTOR: Update reproducibility test case to reflect changes (#237)
f6d40d4
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Update python unit testing (#237)
3356d9a 
REFACTOR: Refactor to mirror changes to python wrapper

FEAT: Include FDR calculation in unit test
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Create no hail lfdr class (#237)
59f40bc 
FEAT: Implement function for manhattan plotting negative log p values
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Create temp hail notebook for testing JSON export OOM (#237)
3f8066b
NickEdwards7502 added a commit that referenced this issue Sep 11, 2024
@NickEdwards7502
DEV: Create temp notebook for demonstrating VS functionality
de29b45 
without hail (#237)
NickEdwards7502 added a commit that referenced this issue Sep 13, 2024
@NickEdwards7502
DEV: Add covariate import wrapper function (#237)
fe2db4c 
STYLE: Format with black
NickEdwards7502 added a commit that referenced this issue Sep 13, 2024
@NickEdwards7502
DEV: Create python class for covariate imports (#237)
a9b9570
NickEdwards7502 added a commit that referenced this issue Sep 13, 2024
@NickEdwards7502
STYLE: Format with black (#237)
3ea4c8c
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
REFACTOR: Remove covariatesource as not required (#237)
8f11e62
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
STYLE: Format with black (#237)
d671f35
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Add wrapper functions for covariate support (#237)
209a463 
FEAT: Add wrapper class for importing covariates

FEAT: Add wrapper class for unioning features and covariates
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
STYLE: Format with black (#237)
b94afcc
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
STYLE: Format with black (#237)
04daae2
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Update lfdr to support covariates (#237)
30732ba 
REFACTOR: Include covariate filtering in manhattan plot function

STYLE: Format with black (#237)
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
STYLE: Format with scalamft (#237)
3381e68
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Update VSContext to support covariates (#237)
37f4193 
FEAT: Add functions for importing std and transposed CSVs

FEAT: Add function for unioning features and covariates
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Update std CSV features to support optional variable type specs (#…
dfae3c2 
…237)
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Create class for returning union of features and covariates (#237)
9733844
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
CHORE: Add reproducibility test that includes covariates (#237)
dd32e0f
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
CHORE: Remove print statements from RF reproducibility test (#237)
769ce76
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
CHORE: Update no hail notebook to include covariates (#237)
e9a23cf
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Update fdr estimation to return p-value cutoff (#237)
4379b0a
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
FIX: Correct significance line plot to use cutoff p-value (#237)
d2048d0
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
REFACTOR: Update pairwise operation tests based on import changes (#237)
2416b8e 
Reference changed to importTransposedCSV
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
CHORE: Update nohail notebook with significance line changes (#237)
1529bd8
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
FIX: Add covariate type parameter to python wrapper for std csv (#237)
12d6137
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Replace to_df functionality with head (#237)
4df6e32 
REFACTOR: Remove python component of converting Feature RDD to pandas

FEAT: Add RDD slice to DF function
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
DEV: Update FeatureSource dataframe conversion (#237)
b1fe760 
REFACTOR: Remove conversion of whole RDD to DataFrame

FEAT: Add function for slicing rows and columns and converting to DF
NickEdwards7502 added a commit that referenced this issue Sep 19, 2024
@NickEdwards7502
CHORE: Update nohail demo to include df slicing (#237)
4506139
NickEdwards7502 added a commit that referenced this issue Sep 20, 2024
@NickEdwards7502
REFACTOR: Make ExportModel compatible with JsonRFAnalyser (#237)
07cd144
NickEdwards7502 added a commit that referenced this issue Sep 23, 2024
@NickEdwards7502
REFACTOR: Make head concrete & inherited by all feature sources (#237)
cef13f4
NickEdwards7502 added a commit that referenced this issue Sep 23, 2024
@NickEdwards7502
FIX: Update regex import for lfdr DataFrame parsing(#237)
b6f4e3b
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove cutoff pvalue print statement (#237)
9b7f83b
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove VSHail python wrapper files (#237)
f66f445
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove VSHail Random Forest scala class (#237)
e6e637c
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Update spark context test to use non-hail Kryo registrator (#237)
8f82d29
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove test cases specifically for hail (#237)
1be8d66
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove hail from maven build (#237)
bd53f27
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
REFACTOR: Update lfdr filename and python wrapper reference (#237)
45ae200
NickEdwards7502 added a commit that referenced this issue Oct 2, 2024
@NickEdwards7502
CHORE: Remove hail unit tests (#237)
fe70285
NickEdwards7502 added a commit that referenced this issue Oct 17, 2024
@NickEdwards7502
FIX: Update LocalFDR import statement in rfmodel.py (#237)
279bd5b
NickEdwards7502 added a commit that referenced this issue Oct 17, 2024
@NickEdwards7502
DEV: Integrate bgzipped file support in VCF import API (#237)
5ad8cc0 
* .bgz loader function implemented by Christina
NickEdwards7502 added a commit that referenced this issue Oct 17, 2024
@NickEdwards7502
DEV: Implement imputation for VCF features (#237)
b686d75 
* Update python wrapper to include imputation strategy parameter

* Update scala API to pass imputation strategy to VCFFeatureSource

* Create functions to handle mode and zero imputation strategies

* Added imputation strategy to test cases

* Added imputation strategy to FeatureSource cli

* Remove sparkPar from test cases due to changes in class signature

* Updated DefVariantToFeatureConverterTest to use zeros imputation
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rocreguant