Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Integration of intel extension for pytorch #1312

Closed
wants to merge 5 commits into from
Closed

Integration of intel extension for pytorch #1312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5c0213e
[draft] integrate intel extension for pytorch to torchserve base_handler
jingxu10 Nov 2, 2021
8031300
removed runtime jit trace
jingxu10 Nov 4, 2021
856d105
revert inference and handler function back in base_handler.py
jingxu10 Nov 9, 2021
2057e63
format fine-tune
jingxu10 Nov 9, 2021
2a91c71
Merge branch 'master' into jingxu10/intel_extension_for_pytorch
jingxu10 Nov 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerLifeCycle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,19 @@ public void startWorker(int port) throws WorkerInitializationException, Interrup
throw new WorkerInitializationException("Failed get TS home directory", e);
}

String[] args = new String[6];
String[] args = new String[12];
args[0] = EnvironmentUtils.getPythonRunTime(model);
args[1] = new File(workingDir, "ts/model_service_worker.py").getAbsolutePath();
args[2] = "--sock-type";
args[3] = connector.getSocketType();
args[4] = connector.isUds() ? "--sock-name" : "--port";
args[5] = connector.getSocketPath();
args[1] = "-m";
args[2] = "intel_extension_for_pytorch.cpu.launch";
args[3] = "--ninstances";
args[4] = "1";
args[5] = "--ncore_per_instance";
args[6] = "22";
Comment on lines +54 to +58
Copy link
Collaborator

@lxning lxning Nov 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to config these parameters in config.properties, instead of hard coded.

Are ninstances and ncore_per_instance model level configuration?

Copy link
Contributor Author

@jingxu10 jingxu10 Nov 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @lxning , Could you guide us where to add the code to read new config knobs from the config.properties file?
Yes, both of them are model-level configuration. They only take effects on the task launched by ts/model_service_worker.py script.

Copy link
Collaborator

@HamidShojanazeri HamidShojanazeri Nov 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lxning how about importing functionalities of "intel_extension_for_pytorch.cpu.launch" into based handler as utilities instead of calling the script here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HamidShojanazeri this launch script is actually a wrapper of numactl command. handler script is supposed to be launched by it to make CPU binding take effect.

args[7] = new File(workingDir, "ts/model_service_worker.py").getAbsolutePath();
args[8] = "--sock-type";
args[9] = connector.getSocketType();
args[10] = connector.isUds() ? "--sock-name" : "--port";
args[11] = connector.getSocketPath();

String[] envp =
EnvironmentUtils.getEnvString(
Expand Down
15 changes: 14 additions & 1 deletion ts/torch_handler/base_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@
import time
import torch

ipex_enabled = False
try:
import intel_extension_for_pytorch as ipex
ipex_enabled = True
except:
pass

from ..utils.util import list_classes_from_module, load_label_mapping

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -73,6 +80,9 @@ def initialize(self, context):
self.model = self._load_torchscript_model(model_pt_path)

self.model.eval()
if ipex_enabled:
self.model = self.model.to(memory_format=torch.channels_last)
self.model = ipex.optimize(self.model, dtype=torch.float32, level='O1')

logger.debug('Model file %s loaded successfully', model_pt_path)

Expand Down Expand Up @@ -141,7 +151,10 @@ def preprocess(self, data):
Returns:
tensor: Returns the tensor data of the input
"""
return torch.as_tensor(data, device=self.device)
t = torch.as_tensor(data, device=self.device)
if ipex_enabled:
t = t.to(memory_format=torch.channels_last)
return t
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HamidShojanazeri currently this only apply to vision models, because in PyTorch only 4D tensor can be changed to NHWC.


def inference(self, data, *args, **kwargs):
"""
Expand Down