Tags: DanielAvdar/ml-orchestrator
Tags
Add MetaComponentV2 (#74) * Refactor component parsing and update dependencies Revised `ComponentParser` and `MetaComponent` structure to improve maintainability and introduced support for `MetaComponentV2`. Updated `ruff` dependency to version 0.8.4, added `mypy` target to the `Makefile`, and switched GitHub workflow to `make` for uniform task execution. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Use `_MetaComponent` instead of `MetaComponent` types Updated the `create_kfp_str` method's parameter to `_MetaComponent` to ensure compatibility with the updated imports. This change aligns type usage throughout the codebase for consistency and clarity. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor component structure and integrate MetaComponentV2 Reorganized components and introduced `MetaComponentV2` to improve modularity and code clarity. Updated tests to accommodate new component structures, including fixtures for both legacy and V2 meta-components. Added `dummy_components_v2` to extend support for enhanced component implementations. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor MetaComponentV2 and streamline code checks. Removed redundant return statement in `MetaComponentV2.env` method for improved readability. Simplified the GitHub Actions workflow by replacing commented-out commands with `make` commands for better maintainability and consistency. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Fixes (#73) * Update dependencies, configs, and imports; add Makefile Upgraded pre-commit hooks, Ruff, and Poetry dependencies to newer versions. Removed unused `section-order` option in `isort` config. Fixed import order in several test files and added type handling for metrics. Introduced a Makefile to streamline installation, testing, and code checks. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Update Makefile to include all extras during install Changed the Poetry install command to use the `--all-extras` flag. This ensures that all optional dependencies are installed, improving compatibility and reducing setup issues. Pre-commit hooks remain updated and installed as before. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add support for parsing field defaults in FunctionParser Refactored `comp_vars` in `FunctionParser` to centralize the handling of field default values via the new `parse_field_default` method. Updated `get_func_params` to prioritize parameters with default values. Added test coverage for field default parsing to ensure correctness and edge case handling. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor and enhance notebook push functionality. Updated the `PushNotebook` class to include GPU and competition source options. Improved environment variable handling and streamlined metadata creation for better configurability. Removed unused code and enhanced metric logging during the notebook push process. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor test structure and improve field handling logic Moved and reorganized test files into a `test_dummies` directory to improve clarity and separation of test cases. Enhanced field handling in function parameter utilities to properly address default and missing values. Introduced new fixtures for streamlined component testing. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Add return type handling for component execution (#68) * Add return type handling for component execution Enhanced the function parser to include return type annotations in function definitions and component execution. This update ensures that the code accurately reflects the return types of components by adding the appropriate return annotations in both definition and execution contexts. Additionally, new test cases were added for components with different return types. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add type hint to exe_return method in func_parser.py This change adds a type hint to the exe_return method, specifying that the comp parameter should be of Type[ComponentProtocol]. It improves code readability and helps with static type checking. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * fix bug with list default values Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * mypy Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Update publish (#58) * test imports. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Move pytest import inside ProtocolCompSuite class This change ensures pytest is only imported when the ProtocolCompSuite class is instantiated, reducing the global import footprint. It helps in keeping the module's namespace cleaner and may prevent potential import-related issues. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove ProtocolCompSuite import and correct fixture placement The `ProtocolCompSuite` import was removed from `main.py` and `__init__.py`. Additionally, the fixture placement was corrected by fixing the indentation and import location within `t_suites.py`. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor poetry commands in publish workflow Removed redundant `poetry install` and specific test run commands. Consolidated to a single `poetry install --all-extras` and `poetry run pytest` for simplicity and efficiency. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Move pytest import inside ProtocolCompSuite class This change ensures pytest is only imported when the ProtocolCompSuite class is instantiated, reducing the global import footprint. It helps in keeping the module's namespace cleaner and may prevent potential import-related issues. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove ProtocolCompSuite import and correct fixture placement The `ProtocolCompSuite` import was removed from `main.py` and `__init__.py`. Additionally, the fixture placement was corrected by fixing the indentation and import location within `t_suites.py`. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor poetry commands in publish workflow Removed redundant `poetry install` and specific test run commands. Consolidated to a single `poetry install --all-extras` and `poetry run pytest` for simplicity and efficiency. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove obsolete ISO tests Deleted outdated ISO test script and adjusted pytest configurations by removing references to the old `test_iso` directory. This cleanup simplifies the test suite and eliminates redundant checks. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
test imports. (#57) * test imports. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Move pytest import inside ProtocolCompSuite class This change ensures pytest is only imported when the ProtocolCompSuite class is instantiated, reducing the global import footprint. It helps in keeping the module's namespace cleaner and may prevent potential import-related issues. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove ProtocolCompSuite import and correct fixture placement The `ProtocolCompSuite` import was removed from `main.py` and `__init__.py`. Additionally, the fixture placement was corrected by fixing the indentation and import location within `t_suites.py`. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
redesign. (#54) * Add mypy to GitHub Actions workflow This commit adds `mypy` type checking to the GitHub Actions workflow in `code-checks.yml`. Additionally, the redundant local `mypy` hook was removed from the pre-commit configuration to streamline type checking. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * redesign. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Handle type instantiation in test_flows_protocol Instantiates comp_fixture if it is a class type to ensure it is a ComponentProtocol instance. This prevents type errors during tests and makes the test method more robust. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor class naming and comment out unnecessary code Renamed `TestProtocolCompSuite` to `ProtocolCompSuite` for clarity and consistency in naming conventions. Additionally, commented out an unused `isinstance` check in `test_flows_protocol` to simplify the code and avoid unnecessary checks. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Enforce abstract methods in ProtocolCompSuite Added abc.ABC to ensure ProtocolCompSuite mandates abstract methods. Marked comp_fixture with abc.abstractmethod to enforce implementation in subclasses. Commented out unused test_flows_protocol method to clean up the code. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove test suite import and definitions Commented out the import statement and class definitions related to `ProtocolCompSuite`. This effectively disables test cases for both `TestProtocolCompSuite` and `TestInvalidProtocolCompSuite`. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Update Poetry install commands to include all extras Modified the Codecov and CI workflows to use `poetry install --all-extras` ensuring all optional dependencies are installed. Adjusted the positioning of `poetry check` in the CI workflow for proper execution order. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Reinstate commented-out test suite Re-added previously commented-out test suite for `ProtocolCompSuite`. This will ensure proper testing coverage and validation of both valid and invalid component classes in the `ml_orchestrator` module. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor and clean up test suites and dummy components Replaced commented test in `t_suites.py` with executable code to check instance type. Modified `conftest.py` to remove an unused component class from fixture parameters. Fixed assertion mark in `test_suites.py`. Simplified `ComponentTestC` in dummy components. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add type hints and fix mypy configuration Enhanced type safety by adding type hints across various methods in the `comp_protocol` and related modules. Adjusted the mypy command in the CI workflow to check all files, ensuring consistency and preventing type mismatches. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add new components and improve function validation Introduced `FunctionParser` and `ProtocolCompSuite` components. Enhanced test cases to include checks for component initialization and function invocations, ensuring better coverage and validation. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add unit tests for artifact classes and schema titles. Introduced new test cases in `test_artifacts.py` to verify the integrity of artifact classes against KFP's `dsl` module, ensuring schema titles and metadata consistency. Enhanced artifact classes with `schema_title` attributes where necessary. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add new dummy component and update protocol implementations Introduced `ComponentTestC` in dummy components with an override for the `execute` method. Updated protocol fixture methods to use single-line implementations. Also added `ComponentTestC` to the test fixture parameters for extended coverage. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add mypy to GitHub Actions workflow This commit adds `mypy` type checking to the GitHub Actions workflow in `code-checks.yml`. Additionally, the redundant local `mypy` hook was removed from the pre-commit configuration to streamline type checking. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * redesign. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Handle type instantiation in test_flows_protocol Instantiates comp_fixture if it is a class type to ensure it is a ComponentProtocol instance. This prevents type errors during tests and makes the test method more robust. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor class naming and comment out unnecessary code Renamed `TestProtocolCompSuite` to `ProtocolCompSuite` for clarity and consistency in naming conventions. Additionally, commented out an unused `isinstance` check in `test_flows_protocol` to simplify the code and avoid unnecessary checks. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Enforce abstract methods in ProtocolCompSuite Added abc.ABC to ensure ProtocolCompSuite mandates abstract methods. Marked comp_fixture with abc.abstractmethod to enforce implementation in subclasses. Commented out unused test_flows_protocol method to clean up the code. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Remove test suite import and definitions Commented out the import statement and class definitions related to `ProtocolCompSuite`. This effectively disables test cases for both `TestProtocolCompSuite` and `TestInvalidProtocolCompSuite`. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Update Poetry install commands to include all extras Modified the Codecov and CI workflows to use `poetry install --all-extras` ensuring all optional dependencies are installed. Adjusted the positioning of `poetry check` in the CI workflow for proper execution order. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Reinstate commented-out test suite Re-added previously commented-out test suite for `ProtocolCompSuite`. This will ensure proper testing coverage and validation of both valid and invalid component classes in the `ml_orchestrator` module. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Refactor and clean up test suites and dummy components Replaced commented test in `t_suites.py` with executable code to check instance type. Modified `conftest.py` to remove an unused component class from fixture parameters. Fixed assertion mark in `test_suites.py`. Simplified `ComponentTestC` in dummy components. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add type hints and fix mypy configuration Enhanced type safety by adding type hints across various methods in the `comp_protocol` and related modules. Adjusted the mypy command in the CI workflow to check all files, ensuring consistency and preventing type mismatches. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add new components and improve function validation Introduced `FunctionParser` and `ProtocolCompSuite` components. Enhanced test cases to include checks for component initialization and function invocations, ensuring better coverage and validation. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add unit tests for artifact classes and schema titles. Introduced new test cases in `test_artifacts.py` to verify the integrity of artifact classes against KFP's `dsl` module, ensuring schema titles and metadata consistency. Enhanced artifact classes with `schema_title` attributes where necessary. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add new dummy component and update protocol implementations Introduced `ComponentTestC` in dummy components with an override for the `execute` method. Updated protocol fixture methods to use single-line implementations. Also added `ComponentTestC` to the test fixture parameters for extended coverage. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Update Python version constraints and CI matrix. Changed the Python version range in pyproject.toml to properly include 3.13. Added Python 3.13 to the CI workflow's testing matrix to ensure compatibility. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> * Add type ignore comment to execute method in ComponentTestC This change suppresses type checking for the execute method, returning a constant value of 1. This can help to temporarily bypass typing issues during development or refactoring. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com> --------- Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
PreviousNext