Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

replace plyr / reshape2 functionality with data.table equivalent #161

Closed
29 tasks done
dylanbeaudette opened this issue Jan 21, 2021 · 2 comments
Closed
29 tasks done

Comments

@dylanbeaudette
Copy link
Member

dylanbeaudette commented Jan 21, 2021

Keeping track of changes / testing related to data.table replacement of plyr / reshape functionality. Each change should be checked and a new test added if there isn't sufficient coverage. Handy bash script:

grep 'join(' * | awk '{print "  - [ ] `"$1"`"}' | tr -d ':.R' | sort | uniq

plyr

join

  • fetchHenry
  • fetchNASIS_pedons
  • fetchNASIS_report
  • fetchPedonPC
  • getHzErrorsNASIS
  • getHzErrorsPedonPC
  • get_colors_from_pedon_db
  • get_component_data_from_NASIS_db
  • get_extended_data_from_NASIS_db
  • get_hz_data_from_pedon_db

ddply

  • fetchHenry
  • fetchNASISLabData
  • fetchNASIS_components
  • fetchNASIS_report
  • fetchPedonPC
  • fetchSCAN
  • getHzErrorsNASIS
  • getHzErrorsPedonPC
  • get_colors_from_pedon_db
  • get_component_from_LIMS
  • get_component_from_SDA
  • get_phlabresults_data_from_NASIS_db
  • simplifyArtifactData

ldply

  • SDA-spatial
  • fetchSCAN

reshape2

reshape2::dcastdata.table::dcast

  • fetchHenry
  • simplfyFragmentData
  • simplifyArtifactData

Note that value argument becomes value.var.

meltdata.table::melt

  • fetchSCAN

Caveats

  • It appears that data.table::melt will not preserve factor type / levels unless asked to do so explicitly via value.factor argument. A reasonable solution is to set it after the reshape operation when appropriate.
  • id.vars should not contain duplicated column names
  • melt.data.table expects that all measure.vars are of the same type (e.g. all numeric) otherwise warning issued
@dylanbeaudette dylanbeaudette changed the title replace reshape2 functionality with data.table equivalent replace plyr / reshape2 functionality with data.table equivalent Jan 21, 2021
@brownag brownag mentioned this issue May 18, 2021
7 tasks
@brownag brownag mentioned this issue Oct 28, 2021
3 tasks
brownag added a commit that referenced this issue Nov 2, 2021
@dylanbeaudette
Copy link
Member Author

Nice work on those last changes. Looks like all that is left is get_colors_from_pedon_db?

@brownag
Copy link
Member

brownag commented Nov 5, 2021

Thanks, yes that was all that was left, now done in 66499a7

The conversion from plyr 2ad40e7 was fine but I was unable to verify the correctness of result due to some other dangling, previously un-detected issues with (un)coded moisture state resulting in a 0-length results even with data in the DB.

Testing any MS Access stuff takes some finagling on my work machine because I am still stuck with only 32-bit MS Access drivers (boo)

I also took this opportunity to convert from the (deprecated) mix_and_clean_colors to the newer estimateColorMixture. So now the mixing should be done in LAB (woo) and I have verified that fetchPedonPC / get_colors_from_pedon_db are working as expected.

@brownag brownag closed this as completed in 9587120 Nov 5, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants