Merge recOrder into waveorder
#183
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Remove offline mode
Turn off autoshutter before bg capture
Simplify acquisition settings
Bump napari-hub-preview-action version to fix CI
…233) * unpin recorder version * Revert "unpin recorder version" This reverts commit 838e9a1bb49ac5256c876cbad12c12d48da102c7. * add waveorder co-dev guide & fix markdown syntax * fix markdown syntax * bump napari-hub-preview-action version to fix CI * add waveorder co-dev guide & fix markdown syntax * Revert "Merge branch 'waveorder-co-dev' of https://github.com/mehta-lab/recOrder into waveorder-co-dev" This reverts commit 3e09466c4b94828da557c8fe0d25a7eb69f0db0d, reversing changes made to b399c36cae7cdbb1bf0c4bbc08d2888fe4237a87. Co-authored-by: Talon Chandler <talonchandler@gmail.com>
* load "Phase3D" reconstruction in target reconstruction * displey (10 MB) in sample contribution menu * add zebrafish embryo contribution (local) * First draft README * example data movies * < 100 MB * update links to v1.5 * Update README.md Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * slightly changed wording * add installation movie * delete old data * fix tests --------- Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com>
* direct link * use raw link --------- Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com>
* first draft append channels * bugfix * test create empty * pass output channel indices * test append mode * add config name to transfer_function.zarr * channel names from metadata * fix test create emtpy * Update recOrder/cli/utils.py Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * only open `output_dataset` once * document appending behavior * CLI-level documentation of append behavior --------- Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com>
also bump the suggested version in install guides to 3.10
* make qt, napari, and pycromanager optional installed only when the 'acq' flag is specified * fix ci and docs * fix tox * need `acq` for microscope installation --------- Co-authored-by: Talon Chandler <talonchandler@gmail.com>
* test transfer functions > 2147483647 bytes * YX chunks for transfer functions * Revert "test transfer functions > 2147483647 bytes" This reverts commit 150c5a56d3122f6b57d2e90f5552bf47120302b1.
* pass plate metadata through reconstruction if it exists * simple test * bug fix * use zattrs.update Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * add all metadata then remove `plate` Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> --------- Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com>
* move overlays to io.visualization * refactor and test ret_ori_overlay * first-pass copy of `ret_ori_phase_overlay` * Add `ret_ori_phase_overlay` * clean unused code * modify dask mapping to handle czyx * remove unused colormapping functions * simple test for ret_ori_phase_overlay * remove obsolete binning * add ret_min argument
* raise -> warning * fix background consistency check
* default HSV colormap * reinstate original fix * remove experimental flag for pyqt6 --------- Co-authored-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com>
* update iohub dependency * bump waveorder * update workflow and format yaml * bump python * bump tox env * update workflow * remove unused arguments * prepare to unpin pydantic --------- Co-authored-by: Aaron Alvarez <aaron.alvarez@czbiohub.org>
* depend on submitit * bare-bones submission * simple status handling * refactor `monitor_jobs` * parsing * simple resource estimates * add ram-multiplier * back to array jobs * correct elapsed timer * more informative messages * message handling * better local debugging * refactor debug messages * fix tests * drop 3.12 * refactor monitoring * ignore logs * fix tests * fix test * print the first failure and # of successful jobs * max 50 jobs at a time * monitor when more jobs than terminal lines * cpu_request = num_processes * imrpved headline * add `-rx` option to `reconstruct` * better color handling
* update badges * fix mistaken waveorder hangers-on
* first-pass buyer's guide * edited buyer's guide * link to build video * link to openpolscope stl files * add meadowlark alternative part number --------- Co-authored-by: Shalin Mehta <shalin.mehta@czbiohub.org>
* pydantic-model initial commit - pydantic model UI widget creation based on CLI model - does no processing, creates yaml files * Updated prototype - partially working - refactored model to GUI implementation - added table to hold processing info, entries purge on completion - using click testing for now in prototyping - creates yaml to GUI - using napari notifications for msgs - testing on Windows * first working alpha version - Multi processing using pydantic models - Implemented a client/server approach for managing jobs queue - Updates each job status based on logs - Implemented Unique ID in CLI to associate Jobs with Process - Tested on Windows with locally modified submitit * possible fix for macOS - import OS specific pydantic imports for ModelMetaclass * use ver specific fix for pydantic instead of OS - pydantic possibly needs to be pinned at 1.10.19 @talonchandler - implemented ver specific import for now * set a timeout limit on job update thread - check for update on the job file, if not - notify user in process table and exit after a 2 min timeout * fixes & enhancements - fixes newly created yaml files to reside besides output zarrs - fixes Model numbers not correctly representing on Build & Run button - fixes "reconstruction algorithm" to show up as dropdown box - fixes "regularization strength" spinbox to accept 5 significant decimal place values - background path now accepts Path and str in model and shows up as directory selector button - each model container setting is now a collapsible item - fixes issues when widget is closed/hidden and then started again - all model(s) validation errors are not collected and presented in notification and highlighted on GUI in one go - each collapsible model entry also highlights if any validation errors are present - added confirm dialog for certain actions - added clear results button - job update thread will exit after a timeout limit and no update detected in job output file * Update setup.cfg - pinning pydantic lib ver to 1.10.19 * use PyQt6 for pyqtSignal use PyQt6 for pyqtSignal * standalone - fixes standalone widget * fixes for BG, etc - fixes BG selection and resetting if Path is invalid - fixes removal of row/stability - using Queued List to counter race conditions - added .zarr data validation check for Input, also used for BG - make uniqueID more unique - break thread update operations when user has cleared the results table - standalone GUI niceties * - fixes for user initiated row delete while processing another check if result row is deleted by user action while processing * major refactor to support positions in datasets - timeout of 5 min per Job - BF button error handling when not available - GUI initiates clearing logs on first run to avoid file exist errors - a single processing run now supports the CLI spawning multiple jobs * - fixes & enhancements - stand-alone GUI cmd - refactored UI based on suggestions - fixes cyclic import for stand-alone GUI - duplicates prev model settings when available - clears Model when changing Input datasets - load model looks for .yml files * checking for RuntimeWarning value * on-the-fly processing - checks if data is being acquired and utilizes polling (every 10s) to gather processing information and submits processing jobs once a minimum dim has been acquired - added a script to simulate acq of .zarr storage - added group-box for sections - added scrolling for tab so that it does not block horizontal resizing * Delete recOrder/tests/widget_tests/test_pydantic_model_widget.py * Delete recOrder/tests/widget_tests/test_simulate_acq.py * ditching v1.main from pydantic and reverting pydantic>=1.10.17 to test workflows * dont initialize server listening in main init of worker class - moved socket listening to sub method from init() and initialize when required for the first time (testing build & deploy) * incorporating discussed GUI changes - GUI changes to reflect sketch as discussed with @talonchandler @ieivanov - Output directory and validation - Open dataset after reconstruction based on user choice - pinning pydantic lib back to 1.10.19 * exit polling loop when closing app before finish * implemented a Stop method for On-The-Fly polling reconstructions implemented a Stop method for On-The-Fly polling reconstructions if required * GUI related - added info icon next to Input Store label when a dataset path is set which displays channel names for convenience, more metadata could be displayed that might be relevant for reconstruction - removed line widget at end of each model container to make scrollbar more apparent * create logs dir if it does not exist - we are only reading the location here but it needs to exist, the CLI is creating the logs based on the path * fixes output path not setting correctly * added pixel size meta to info icon * update output dir in defined models when changed * make on-the-fly entry scrollable and not block resizing * added script to simulate a "fake" recOrder acquisition - script to test on-the-fly reconstruction POC * fix for checking output path existing * fix for checking output path existing * logs folder to be created besides dataset - logs folder will reside next to the output dataset - fixed an issue with reconstructed data showing irrespective of selection * display SLURM related errors if Jobs output txt is empty * top scrollbar for model, container sizing now does not need second vert scrollbar * multi-pos bugfix + multiple enhancements fixes: - multi-pos dataset would be displayed after single pos processing enhancements: - CLI will print Job status when used as cmd line, not for GUI - use single socket connection when multi-pos is spawned by a request - added "rx" field to model-container - minor GUI tweaks * code formatting, minor refactoring, comments * with rx default as 1, catch and report OOM errors
* remove pyqt6 dependency and re-worked setup cfgs - removed pyqt6 dependency - streamline different setup needs to address #498 - added position metadata to info label #500 - bug report template to include relevant logs which are created next to the output dir, name of the dataset, yaml file(s) if encountering reconstruction errors - CLI can be used without QtBindings (PyQT or PySide) but will show a warning * Update pytests.yml changed install flag to "all" * update tox extras * restore dev dependencies for tests * refined dependencies, updated docs - moved pycromanager and magicgui to [all] flag * re-worked essential dependencies and catching MM connection error - mostly applies when barebone recOrder is installed - in order to lead a user to install the correct missing package (pycromanager) * MM connection failed msg * added Reconstruction Tab section to docs --------- Co-authored-by: Talon Chandler <talonchandler@gmail.com>
* fix dependencies * move key pieces back to setup.cfg * update imports * move entirely to .toml --- basic working installation
|
* fix acq test * fix test_reconstruct * need all for tests * fix filename * remove all * copy iohub gh actions; remove tox * revert * all for pr * match recOrder's setup * remove tox dependency * try to match recOrder's test-and-deploy w/o tox * reinstate tox * only style with a single python version * skip example scripts on github actions * don't limit parallelism * no mps test on ghactions * debug ghactions * remove false debug * run subprocess in os.environ * replace python with sys.executable * remove debug * skip examples temporarily
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #183 +/- ##
=======================================
Coverage ? 36.56%
=======================================
Files ? 49
Lines ? 10201
Branches ? 0
=======================================
Hits ? 3730
Misses ? 6471
Partials ? 0 ☔ View full report in Codecov by Sentry. |
|
Thanks @mattersoflight. I now have all of the non-example tests passing on gh actions. The I am proceeded to more complex non-automated tests tomorrow---checking the installation, examples, guis, etc. I think I'm on track for a successful merge. Help with merging the README is much appreciated. You can commit directly to this branch or work in your own branch then merge when you're ready (e.g. see my messy #186). |
talonchandler
commented
Feb 23, 2025
* store .npz files in correct place * reduce clutter from local testing * fix qlipp example bug * clean pti tests
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR merges
recOrderintowaveorder.TODO before this branch merges into
main:recOrder's commit history intowaveorderrecOrderrecOrdertowaveorderrecOrdertowaveorder(mostly manual...only small PRs are open)TODO after this branch merges into
main:recOrderrepository as archived, and point towaveorder