[Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example #1325

dsikka · 2023-10-17T20:26:29Z

Summary

Further flush out the Pipeline, Router, and Operator functionality
Add an EngineOperator which can be used for majority of our pipelines
Created a LinearRouter
Use the base classes to create Operators for Image Classification, specifically for pre-processing and post-processing and tie the Operators together using a v2 ImageClassificationPipeline

Testing

Updated the initial test case
Test ImageClassificationPipeline creation and inference runs:

from deepsparse.v2.image_classification.pipeline import ImageClassificationPipeline

image_path = "/home/dsikka/deepsparse/src/deepsparse/v2/image_classification/buddy.jpeg"

model_path = (
    "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95-none"
)

# Pipeline creation
pipeline = ImageClassificationPipeline(model_path=model_path)

# run using kwargs; will apply schema if defined
inference = pipeline(images=[image_path, image_path])
print(inference)

# run using arg and schema

from deepsparse.v2.image_classification.preprocess_operator import ImageClassificationInput
inference = pipeline(ImageClassificationInput(images=[image_path, image_path]))

dbogunowicz

Excellent job @dsikka!
I left a lot of comments, but in general I am very happy about the refactor and the direction that we are heading towards!

src/deepsparse/v2/image_classification/pipeline.py

src/deepsparse/v2/image_classification/postprocess_operator.py

src/deepsparse/v2/pipeline.py

src/deepsparse/v2/routers/router.py

src/deepsparse/v2/image_classification/__init__.py

src/deepsparse/v2/image_classification/pipeline.py

src/deepsparse/v2/operators/engine_operator.py

src/deepsparse/v2/operators/operator.py

tests/deepsparse/v2/test_basic_pipeline.py

src/deepsparse/v2/pipeline.py

src/deepsparse/v2/image_classification/pipeline.py

src/deepsparse/v2/operators/engine_operator.py

tests/deepsparse/v2/test_basic_pipeline.py

src/deepsparse/v2/routers/router.py

Satrat

Nice work here, had a few clarifying questions but overall this is really clean!

src/deepsparse/v2/operators/operator.py

src/deepsparse/v2/routers/router.py

dbogunowicz

Few minor comments.

… more generic; remove context

* Pipelines Refactor - Initial Impl (#1287) * [Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix * [v2] EngineOperator updates to make continuous batching easier (#1371) * [v2] EngineOperator updates to make continuous batching easier * test fixes * [Pipeline Refactor] Update routes, text generation initial functionality (#1348) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * [Pipeline Refactor] Additional Operators, Route update and completed generation functionality (#1356) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * fix capacity settting again * typo fixes * [Pipeline Refactor] Split/Join Functionality for multiple prompts (#1384) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class * [Pipeline Refactor] Unit Testing for Text Generation Operators (#1392) * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * fix name * [Continuous Batching] Queue Implementation to support batching grouping and prioritization (#1373) * [Continuous Batching] Queue Implementation to support batching grouping and prioritization * has_key method * thread safety * add blocking option for pop_batch * update docstring * allow mutex to be shared across continuous batching objects * revert last commit * [Continuous Batching] Executor thread for running continuous batching (#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack * [ContinuousBatching] ContinuousBatchingScheduler Implementation (#1375) * [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition * [continuous batching] singleton pattern for scheduler (#1391) * [continuous batching] singleton pattern for scheduler * catch from review * [Pipeline Refactor][Text-Generation] Create a helper function for creating engine_inputs (#1364) * rebasing off my initial commit * cleanups * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions (#1394) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * pipeline runs, but incorrectly * Revert "pipeline runs, but incorrectly" This reverts commit 51c4ee6. * PR review comments --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Text Generation][V2] End-to-end tests (#1402) * initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review * [Pipeline Refactor][Text Generation][Continuous Batching] Integration (#1409) * update split/join * use map * update * run end-to-end * clean-up * fix bug with batch size, introduce SplitRoute dataclass * update tests to use new inputs/outputs * use the normal scheduler for internal kv_cache * add pipeline inpuits * clean-up * change engine type, update docstrings, update override function to be more generic * move subgraph functionality to its own function; clean-up cont batching in text gen pipeline * update linear pathway to also use subgraph execution * rebase fix * fix tests * [Pipeline Refactor] Operator Registry (#1420) * initial registry functionality * use sparsezoo mixin * [Pipeline Refactor] Fix Operator scheduling to fix issue with slow execution (#1453) * fix scheduling to fix issue with engine running very slowly; introduce new completed attribute for Subgraph instead of checking instance type * fix warning message * [Pipeline Refactor] Add `Pipeline.create` method to initialize pipelines (#1457) * add pipeline create method for pipeline creation using the operator registry * add instance check * [Pipeline Refactor] async (#1380) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * async initial functionality * fix capacity settting again * add blocking * more testing * update to use split/join * fix * rebase fix * remove index * change event loop * rebase fix * update async run to use new operator scheduling properly * rebase fixes (#1458) * more fixes (#1459) --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>

… router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix

* Pipelines Refactor - Initial Impl (#1287) * [Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix * [v2] EngineOperator updates to make continuous batching easier (#1371) * [v2] EngineOperator updates to make continuous batching easier * test fixes * [Pipeline Refactor] Update routes, text generation initial functionality (#1348) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * [Pipeline Refactor] Additional Operators, Route update and completed generation functionality (#1356) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * fix capacity settting again * typo fixes * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * [Pipeline Refactor] Split/Join Functionality for multiple prompts (#1384) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * [Pipeline Refactor] Unit Testing for Text Generation Operators (#1392) * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * fix name * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * [Continuous Batching] Queue Implementation to support batching grouping and prioritization (#1373) * [Continuous Batching] Queue Implementation to support batching grouping and prioritization * has_key method * thread safety * add blocking option for pop_batch * update docstring * allow mutex to be shared across continuous batching objects * revert last commit * [Continuous Batching] Executor thread for running continuous batching (#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack * [ContinuousBatching] ContinuousBatchingScheduler Implementation (#1375) * [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition * [continuous batching] singleton pattern for scheduler (#1391) * [continuous batching] singleton pattern for scheduler * catch from review * [Pipeline Refactor][Text-Generation] Create a helper function for creating engine_inputs (#1364) * rebasing off my initial commit * cleanups * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * pipeline runs, but incorrectly * it works for a single sequence * cleanup. now lets figure out how to run multiple sequences * [Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions (#1394) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * pipeline runs, but incorrectly * Revert "pipeline runs, but incorrectly" This reverts commit 51c4ee6. * PR review comments --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Text Generation][V2] End-to-end tests (#1402) * initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review * integration tests pass * [Pipeline Refactor][Text Generation][Continuous Batching] Integration (#1409) * update split/join * use map * update * run end-to-end * clean-up * fix bug with batch size, introduce SplitRoute dataclass * update tests to use new inputs/outputs * use the normal scheduler for internal kv_cache * add pipeline inpuits * clean-up * change engine type, update docstrings, update override function to be more generic * move subgraph functionality to its own function; clean-up cont batching in text gen pipeline * update linear pathway to also use subgraph execution * rebase fix * fix tests * [Pipeline Refactor] Operator Registry (#1420) * initial registry functionality * use sparsezoo mixin * fix tricky rebase * one more cleanup * got tests to work after rebase. implementing SPLIT and JOIN in linearouter now * pipeline working, with GraphRouter. Needs some more testing * ready for review * cleanup * simplify after PR review round * [Pipeline Refactor] Fix Operator scheduling to fix issue with slow execution (#1453) * fix scheduling to fix issue with engine running very slowly; introduce new completed attribute for Subgraph instead of checking instance type * fix warning message * [Pipeline Refactor] Add `Pipeline.create` method to initialize pipelines (#1457) * add pipeline create method for pipeline creation using the operator registry * add instance check * [Pipeline Refactor] async (#1380) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * async initial functionality * fix capacity settting again * add blocking * more testing * update to use split/join * fix * rebase fix * remove index * change event loop * rebase fix * update async run to use new operator scheduling properly * rebase fixes (#1458) * more fixes (#1459) * bring back functionalities that were lost in v2 during rebasing * Update src/deepsparse/transformers/helpers.py * ready for review * bring tests back" * quality * original readme * addressing Dipikas comments * Update src/deepsparse/transformers/pipelines/text_generation/pipeline_no_kv_cache.py * addressing PR review --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>

dsikka marked this pull request as ready for review October 17, 2023 20:28

dsikka changed the title ~~[Pipeline Refactor] Additional functionality, engine operator and image classification example~~ [Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example Oct 17, 2023

dbogunowicz reviewed Oct 18, 2023

View reviewed changes

bfineran reviewed Oct 18, 2023

View reviewed changes

tests/deepsparse/v2/test_basic_pipeline.py Show resolved Hide resolved

Base automatically changed from pipelines-refactor to v2 October 26, 2023 17:22

dsikka added 5 commits October 27, 2023 11:57

initial functionality and working example with image classification

cbd57a3

remove testing image

b0e4267

update args

5037530

initial functionality and working example with image classification

2f98616

remove testing image

f3841bb

dsikka force-pushed the ds_refactor branch from fe9ed2a to f3841bb Compare October 27, 2023 16:05

pr comments

e522025

tdg5 reviewed Oct 27, 2023

View reviewed changes

src/deepsparse/v2/pipeline.py Outdated Show resolved Hide resolved

defines schemas for operators and test

6e70340

dsikka requested review from bfineran, dbogunowicz and rahul-tuli October 27, 2023 20:46

bfineran approved these changes Oct 30, 2023

View reviewed changes

src/deepsparse/v2/image_classification/pipeline.py Outdated Show resolved Hide resolved

src/deepsparse/v2/operators/engine_operator.py Show resolved Hide resolved

tests/deepsparse/v2/test_basic_pipeline.py Show resolved Hide resolved

Satrat reviewed Oct 30, 2023

View reviewed changes

src/deepsparse/v2/routers/router.py Outdated Show resolved Hide resolved

Satrat approved these changes Oct 30, 2023

View reviewed changes

dbogunowicz reviewed Oct 30, 2023

View reviewed changes

src/deepsparse/v2/operators/operator.py Outdated Show resolved Hide resolved

dbogunowicz reviewed Oct 30, 2023

View reviewed changes

src/deepsparse/v2/routers/router.py Show resolved Hide resolved

dbogunowicz reviewed Oct 30, 2023

View reviewed changes

dsikka added 5 commits October 31, 2023 14:23

add image classification test, PR comments

79503b3

fix input/output handling in pipeline and operator base classes to be…

da93fa5

… more generic; remove context

add additional operator input message

5eb035b

typo fix

f7fcd5f

another typo fix

eba1cb4

dsikka merged commit 224e116 into v2 Oct 31, 2023

dsikka deleted the ds_refactor branch October 31, 2023 20:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example #1325

[Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example #1325

dsikka commented Oct 17, 2023 •

edited

Loading

dbogunowicz left a comment

Satrat left a comment

dbogunowicz left a comment

[Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example #1325

[Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example #1325

Conversation

dsikka commented Oct 17, 2023 • edited Loading

Summary

Testing

dbogunowicz left a comment

Choose a reason for hiding this comment

Satrat left a comment

Choose a reason for hiding this comment

dbogunowicz left a comment

Choose a reason for hiding this comment

dsikka commented Oct 17, 2023 •

edited

Loading