PyTorch profiler integration #1297

shrinath-suresh · 2021-10-27T16:54:18Z

Signed-off-by: Shrinath Suresh shrinath@ideas2it.com

Description

This pull request contains changes in base handler to integrate PyTorch profiler - https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/ with torchserve

When user opts for the profiler, the preprocess/inference/postprocess calls are profiled and the traces are saved as a json file . User can load the trace json into pytorch profiler and view the detailed output.

This feature will help users to debug the issues in preprocess or inference.

the basis structure of the implementation is in place. More detailed requirements and unit tests to be captured. once it is done, WIP tag will be removed.

Initial Requirements:

The profiler integration is done only for the custom handler. User can opt for the profiler flow by setting the class variable.
User can override torch.profiler.profiler arguments - This helps user to add conditions based on the environment. For example: by default, the profiler is set to trace only CPU. In case, user wants to profiler CUDA activity as well, profiler arguments can be overridden using class variable.
Saving the profiler output and ability to load the profiler output using pytorch profiler plugin - https://github.com/pytorch/kineto

Fixes #(issue)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Feature/Issue validation/testing

Please describe the tests [UT/IT] that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Test B
UT/IT execution results

Yet to be added

Logs

Checklist:

Have you added tests that prove your fix is effective or that this feature works?
New and existing unit tests pass locally with these changes?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

sagemaker-neo-ci-bot · 2021-10-27T16:59:27Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 1da9607
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-10-27T16:59:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 1da9607
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-10-27T17:22:06Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 1da9607
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-10-27T23:28:07Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: c8676c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-10-27T23:36:32Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: c8676c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-10-27T23:39:36Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: c8676c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

yqhu

Looks good.

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

…resh/serve into profiler-integration

sagemaker-neo-ci-bot · 2021-11-12T18:58:43Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 259899a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-12T19:08:25Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 259899a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

test/pytest/resnet_profiler_override.py

sagemaker-neo-ci-bot · 2021-11-12T19:13:15Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 259899a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

shrinath-suresh · 2021-11-12T19:13:35Z

test/pytest/test_model_archiver.py

@@ -19,37 +19,8 @@ def teardown_module(module):
    test_utils.torchserve_cleanup()


-def model_archiver_command_builder(model_name=None, version=None, model_file=None, serialized_file=None, handler=None, extra_files=None, force=False):


moved this method to test_utils.py to reuse

HamidShojanazeri

Thanks @shrinath-suresh, it looks great. I think adding some notes to docs probably in configuration would be helpful.

HamidShojanazeri · 2021-11-17T02:48:44Z

ts/torch_handler/base_handler.py

+            self.profiler_args["record_shapes"] = True
+
+        if "on_trace_ready" not in self.profiler_args:
+            result_path = "/tmp/pytorch_profiler"


@shrinath-suresh I wonder if we give this option to user to choose the result_path and if not provided we fall back to default path.

Also adding/extending one of the examples, like Resnet or BERT on using this profilers would be very helpful to show the usage.

@HamidShojanazeri Sure Hamid.. I will pick any one example (mnist or resnet) to showcase profiler integration and update the readme accordingly. Thanks for the suggestion

@HamidShojanazeri Regarding the profiler path.. By default the outputs get stored into "/tmp/pytorch_profiler". User can override this behaviour like this.

@HamidShojanazeri Do you have any suggestions for default path ? At base_handler.py i couldn't find any default path (model store?) to store profiler trace files. /tmp/pytorch_profiler looks hardcoded.. Any suggestion ?

For the profiler path I'd suggest using the same path the pytorch profiler uses by default. If 2 succesive runs of a profile just create new directories in the /tmp/pytorch_profiler then there is no need to overwrite but if they do then it's best to allow users to configure what the output path is

I will pick any one example (mnist or resnet) to showcase profiler integration and update the readme accordingly. Thanks for the suggestion

let's have a separate PR on the example.

HamidShojanazeri · 2021-11-17T02:54:00Z

ts/torch_handler/base_handler.py

-        if not self._is_explain():
-            output = self.inference(data_preprocess)
-            output = self.postprocess(output)
+        is_profiler_enabled = os.environ.get("ENABLE_TORCH_PROFILER", None)


@shrinath-suresh I wonder if this could be added to the config.properties, this would help to make sure all the configs are gathered in one place. An example of adding to the config can be found here.

@chauhang your thoughts on this ?

I agree with Hamid, it's a simple change that'll make sure people don't need to think about both OS environment variables and config.properties. It's how we manage all configs. Let us know if you have any questions about how this would work

Hi @shrinath-suresh any update on this? it's much cleaner for us to use config.properties so users can discover all the available configuration options. That way it'll be much easier for people to discover this feature too

shrinath-suresh · 2021-11-18T06:09:32Z

Thanks @shrinath-suresh, it looks great. I think adding some notes to docs probably in configuration would be helpful.

Thanks for your review comments. I will work on addressing them

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

sagemaker-neo-ci-bot · 2021-11-19T03:20:38Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: dd0f807
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

sagemaker-neo-ci-bot · 2021-11-19T03:27:42Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: dd0f807
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-19T03:32:01Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: dd0f807
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-19T03:35:11Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: c916d3c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-19T03:45:30Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: c916d3c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T19:39:06Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: a0e1539
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T19:47:50Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: a0e1539
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T19:53:38Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: a0e1539
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T20:17:31Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 19a5211
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T20:22:04Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 19a5211
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-13T20:28:44Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 19a5211
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

HamidShojanazeri · 2021-12-13T22:18:32Z

examples/image_classifier/mnist/README.md

+export ENABLE_TORCH_PROFILER=true
+```
+
+Note: Ensure`enable_envvars_config` is set to `true` in torchserve config


@shrinath-suresh is this gonna be added to the config.properties?

I added a note here, since the env based setup doesnt work unless user sets enable_envvars_config to true in config.properties.

I will add config.properties file in this example and update the instructions accordingly.

@shrinath-suresh as mentioned in previous comment, you can add a setting to the config following this example. This basically would not work for now.

HamidShojanazeri · 2021-12-13T22:23:51Z

examples/image_classifier/mnist/README.md

+Install torch-tb-profiler and run the following command to view the results in UI
+
+```
+tensorboard --logdir /tmp/pytorch_profiler/mnist/


@shrinath-suresh in the handler below seems, "on_trace_ready" to override the path is missing.

on_trace_ready is not a mandatory argument. If user provides the argument, the trace files will get generated into the user specified location. if the user doesn't provide the argument, the trace files get generated inside "/tmp/pytorch_profiler/<model_name>" folder. Its handled here

For demonstration, we can add on_trace_ready in handler file. Again, we cannot set the on_trace_ready to os.getcwd() as it will be a temp directory in run time. May be we can try use home directory or something .. Your thoughts ?

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

sagemaker-neo-ci-bot · 2021-12-14T18:49:12Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: d9cbaff
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T18:49:23Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: a3abd42
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T19:01:29Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: a3abd42
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T19:04:09Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: d9cbaff
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T19:10:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: d9cbaff
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T19:15:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: a3abd42
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T22:53:16Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 66ae377
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T23:02:07Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 66ae377
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-14T23:08:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 66ae377
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-15T03:37:00Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 0807e3c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-15T03:54:42Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 0807e3c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-15T04:00:21Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 0807e3c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-15T07:00:00Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 0807e3c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-12-15T07:22:34Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 0807e3c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

HamidShojanazeri

@shrinath-suresh please trying adding the config using the example shared in the comments.

HamidShojanazeri · 2021-12-15T20:08:48Z

examples/image_classifier/mnist/README.md

+export ENABLE_TORCH_PROFILER=true
+```
+
+Note: Ensure`enable_envvars_config` is set to `true` in torchserve config


@shrinath-suresh as mentioned in previous comment, you can add a setting to the config following this example. This basically would not work for now.

Updating handler to use pytorch profiler

1da9607

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

HamidShojanazeri self-requested a review October 27, 2021 19:02

Merge branch 'master' into profiler-integration

c8676c6

msaroufim requested a review from yqhu November 2, 2021 21:43

yqhu approved these changes Nov 6, 2021

View reviewed changes

shrinath-suresh added 3 commits November 13, 2021 00:13

Adding unit tests

add780f

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Merge remote-tracking branch 'upstream/master' into profiler-integration

7dc80e1

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Merge branch 'profiler-integration' of https://github.com/shrinath-su…

259899a

…resh/serve into profiler-integration

shrinath-suresh commented Nov 12, 2021

View reviewed changes

test/pytest/resnet_profiler_override.py Outdated Show resolved Hide resolved

shrinath-suresh commented Nov 12, 2021

View reviewed changes

msaroufim self-requested a review November 16, 2021 06:39

HamidShojanazeri reviewed Nov 17, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into profiler-integration

dd0f807

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Clean up

c916d3c

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Merge branch 'master' into profiler-integration

19a5211

chauhang changed the title ~~[WIP] PyTorch profiler integration~~ PyTorch profiler integration Dec 13, 2021

HamidShojanazeri reviewed Dec 13, 2021

View reviewed changes

shrinath-suresh added 2 commits December 14, 2021 23:57

Merge remote-tracking branch 'upstream/master' into profiler-integration

d9cbaff

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Adding config properties

a3abd42

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Merge branch 'master' into profiler-integration

66ae377

Merge branch 'master' into profiler-integration

0807e3c

chauhang merged commit 04c873f into pytorch:master Dec 15, 2021

HamidShojanazeri reviewed Dec 15, 2021

View reviewed changes

		@@ -19,37 +19,8 @@ def teardown_module(module):
		test_utils.torchserve_cleanup()


		def model_archiver_command_builder(model_name=None, version=None, model_file=None, serialized_file=None, handler=None, extra_files=None, force=False):

PyTorch profiler integration #1297

PyTorch profiler integration #1297

Conversation

shrinath-suresh commented Oct 27, 2021

Description

Type of change

Feature/Issue validation/testing

Checklist:

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Oct 27, 2021

AWS CodeBuild CI Report

yqhu left a comment

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Nov 12, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 12, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 12, 2021

AWS CodeBuild CI Report

Choose a reason for hiding this comment

HamidShojanazeri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HamidShojanazeri Dec 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrinath-suresh commented Nov 18, 2021

sagemaker-neo-ci-bot commented Nov 19, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 19, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 19, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 19, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 19, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 13, 2021

AWS CodeBuild CI Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HamidShojanazeri Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

shrinath-suresh Dec 14, 2021 • edited Loading

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Dec 14, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 14, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 14, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Dec 14, 2021

AWS CodeBuild CI Report

HamidShojanazeri Dec 9, 2021 •

edited

Loading

HamidShojanazeri Dec 13, 2021 •

edited

Loading

shrinath-suresh Dec 14, 2021 •

edited

Loading