Could you help me with adding multiple models to keras-serving

[jerminal]

Could you help me with adding multiple models for keras-serving ? Just reply we can discuss it via email or skype.

[krystianity]

Hi jerminal, I would like to help you on this. Are you talking about changing the model's feature vectors (input) and breaking the dimension apis in GRPC nodejs requests?

I will need a description of the input vector of your network to help you.

[krystianity]

Or do you just want to add multiple versions of a model to the same serving-server?

[jerminal]

I have both problems. I need to classify images (so to change the input vector). And I have multiple keras models that I want to apply to each image. So maybe you can add me to skype: jermuk_skype and I'll describe it with more details?

[jerminal]

Or maybe you can tell me your skype or email?

[jerminal]

For example to server models for face recognition.
Some examples from here: https://github.com/oarriaga/face_classification

[krystianity]

Hi @jerminal I will try to setup an example using the face_classification soon.
I am sorry that I cannot give you support through Skype or Email.

Serving multiple versions of a model at once should be no problem, you should only have to name the exports differently and send the version as paramter to the grpc service during requests.

[jerminal]

Oh. Great. But is it possible to check 1 image for several models at 1 request to docker server?

And about support - ok, no problem. You can add me to twitter or anything else. And I can pay for such improvements of your model if you'd like.

[krystianity]

I would probably suggest to load multiple versions of the model into the serving-server and then use the nodejs service to make parallel requests to both/multiple models at the same time, its very easy to do this in node. Il try to setup the face-recog. example with such a parallel request endpoint.

Haha cheers! Il pm you on twitter, when I am working on it ;)

[krystianity]

Actually the question here is, are you actually using the node.js service simplified API? If yes (thats cool) how many requests are u going to run against it? (You could scale the services to unlimited instances,, but its still a valid question if you are trying to predict in realtime using web-services)

[jerminal]

I have about 10-15 requests per minute. It's not very much :)

[jerminal]

And yes - I use the node.js service simplified API.

[krystianity]

Cool, just hang on ;)

[jerminal]

👍

[krystianity]

So I have already pushed my current state into the remote master, but I am not done yet.
I still have to finalise the grpc node communications so that both models are used to predict on behalf of an image that is uploaded to the node server via client. I think I can finish this until mid next week :)

[krystianity]

I have created a sub project as directory: https://github.com/krystianity/keras-serving/tree/master/face-recog

[jerminal]

Thanks. Will be waiting for this very much!!!

[krystianity]

@jerminal alright. It is up an running :)
Enjoy the awesomeness 🍸 and drop me beer if you like https://www.paypal.me/5cf 🍺

Let me know if you need or want to try something else.

[jerminal]

Hi. Thanks a lot!!! I'll check it in the evening. And of course I've sent drop for a beer for you.

And promise when I come to Germany again, I'll to treat you beer personally :)

[jerminal]

Oh.. I have a strange problem. When follow the instructions:
https://github.com/krystianity/keras-serving/tree/master/face-recog#4-test-api-via-client

at p. 4 when I run: "npm start ./../images/cutouts/4.png" I got strange error :(

10 verbose lifecycle client@1.0.0~start: CWD: /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
11 silly lifecycle client@1.0.0~start: Args: [ '-c', 'node request.js "./../images/cutouts/4.png"' ]
12 silly lifecycle client@1.0.0~start: Returned: code: 1  signal: null
13 info lifecycle client@1.0.0~start: Failed to exec start script
14 verbose stack Error: client@1.0.0 start: `node request.js "./../images/cutouts/4.png"`
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/lifecycle.js:283:16)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at EventEmitter.emit (events.js:172:7)
14 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at ChildProcess.emit (events.js:172:7)
14 verbose stack     at maybeClose (internal/child_process.js:818:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
15 verbose pkgid client@1.0.0
16 verbose cwd /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
17 verbose Darwin 16.6.0
18 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" "./../images/cutouts/4.png"
19 verbose node v5.1.0
20 verbose npm  v5.0.3
21 error code ELIFECYCLE
22 error errno 1
23 error client@1.0.0 start: `node request.js "./../images/cutouts/4.png"`
23 error Exit status 1
24 error Failed at the client@1.0.0 start script.
24 error This is probably not a problem with npm. There is likely additional logging output above.
25 verbose exit [ 1, true ]

[krystianity]

No worries, your running node 5.10; (with npm 5.0.3 o.O) Can you upgrade you node version to at least 7.10 ?

[krystianity]

If its not possible for you to update node on the root system, I can wrap the client in a Docker container as well :)

[jerminal]

Updated node. Now strange thing with running npm start ./../images/cutouts/4.png
I got response:

> client@1.0.0 start /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
> node request.js "./../images/cutouts/4.png"

Reading from path: /Users/siarheiparashchanka/sites/keras-serving/face-recog/images/cutouts/4.png
Loaded image with size: 2304
Making request to CNN GRPC wrapper.
{
  "error": {
    "code": 3,
    "metadata": {
      "_internal_repr": {}
    }
  }
}

[krystianity]

After upgrading nodejs, did you delete the node_modules folder in the face-recog/client dir and ran "npm install again" ? (just to make sure)

Additionally, I have a feeling that the models you servers are serving arent correct e.g. cached.
How are you starting the servers? When you are using my script (start-dual-servers.sh) it should make sure that nothing is cached.

To understand whats going wrong I need you to send me the error that is logged by the server, when your client retrieves the error.

[krystianity]

Additionally.
Are you executing via python 2.7?
Which docker and docker-compose versions are you running?
Could you dump the whole output of docker-compose when you start the dual servers and make a request with the client?

[jerminal]

Oh. I'll check it now on different computer. A few hours and will write back.

[jerminal]

I've checked. The same problem.
I recompiled all.
And after I run "npm start ./../images/cutouts/4.png"

Got the same error.
Looks like problem with input tensor size:

node-server_1     | npm info it worked if it ends with ok
node-server_1     | npm info using npm@5.0.3
node-server_1     | npm info using node@v8.1.2
node-server_1     | npm info lifecycle node-keras-server@1.0.0~prestart: node-keras-server@1.0.0
node-server_1     | npm info lifecycle node-keras-server@1.0.0~start: node-keras-server@1.0.0
node-server_1     | 
node-server_1     | > node-keras-server@1.0.0 start /usr/src/app
node-server_1     | > node server.js
node-server_1     | 
emotion-server_1  | 2017-06-16 13:55:14.421089: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:145] Restoring SavedModel bundle.
gender-server_1   | 2017-06-16 13:55:14.691086: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:145] Restoring SavedModel bundle.
node-server_1     | listening @ http://localhost:8080
emotion-server_1  | 2017-06-16 13:55:15.425129: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:180] Running LegacyInitOp on SavedModel bundle.
emotion-server_1  | 2017-06-16 13:55:15.459184: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:274] Loading SavedModel: success. Took 2047432 microseconds.
emotion-server_1  | 2017-06-16 13:55:15.460264: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: emotion_model version: 1}
emotion-server_1  | 2017-06-16 13:55:15.471674: I tensorflow_serving/model_servers/main.cc:298] Running ModelServer at 0.0.0.0:9000 ...
gender-server_1   | 2017-06-16 13:55:15.608845: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:180] Running LegacyInitOp on SavedModel bundle.
gender-server_1   | 2017-06-16 13:55:15.648949: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:274] Loading SavedModel: success. Took 2080879 microseconds.
gender-server_1   | 2017-06-16 13:55:15.650862: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: gender_model version: 1}
gender-server_1   | 2017-06-16 13:55:15.663076: I tensorflow_serving/model_servers/main.cc:298] Running ModelServer at 0.0.0.0:9000 ...
node-server_1     | { Error: input must be 4-dimensional[]
node-server_1     | 	 [[Node: image_array_2/convolution = Conv2D[T=DT_FLOAT, _output_shapes=[[-1,48,48,16]], data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_image_array_input_2_0_0, image_array_2/kernel/read)]]
node-server_1     |     at /usr/src/app/node_modules/grpc/src/node/src/client.js:442:17 code: 3, metadata: Metadata { _internal_repr: {} } }

[jerminal]

And yes, I execute via python 2.7.

Which docker and docker-compose versions are you running?

Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.12.0, build b31ff33

Could you dump the whole output of docker-compose when you start the dual servers and make a request with the client?

Yes, I've done it above. Hope it will help you.

[krystianity]

Alright,

the versions look fine, the output is strange as it relates to the dimensions defined for the interface.
Of which both are 4 dimensional for predictEmotion(1,48,48,1) and predictGender(1,48,48,3).

I have added additional logging to the node-server and the client, could please run the following and
send me the output of the docker-compose, as well as the client here again:

cd /keras-prod
git pull
rm -r ./export
python face-recog/export.py
./start-dual-servers.sh
#..
cd face-recog/client
npm start ./../images/cutouts/4.png

[krystianity]

This is how it should actually look like.

[jerminal]

Here log:

node-server_1     | npm info it worked if it ends with ok
node-server_1     | npm info using npm@5.0.3
node-server_1     | npm info using node@v8.1.2
node-server_1     | npm info lifecycle node-keras-server@1.0.0~prestart: node-keras-server@1.0.0
node-server_1     | npm info lifecycle node-keras-server@1.0.0~start: node-keras-server@1.0.0
node-server_1     | 
node-server_1     | > node-keras-server@1.0.0 start /usr/src/app
node-server_1     | > node server.js
node-server_1     | 
node-server_1     | listening @ http://localhost:8080
node-server_1     | Input array size: 2304, example value: 0.9882352941176471.
node-server_1     | Tensor Dim: [{"size":1},{"size":48},{"size":48},{"size":3}].
node-server_1     | Input array size: 2304, example value: 0.9882352941176471.
node-server_1     | Tensor Dim: [{"size":1},{"size":48},{"size":48},{"size":1}].
node-server_1     | emotion prediction failed.
node-server_1     | { Error: input must be 4-dimensional[]
node-server_1     | 	 [[Node: image_array_2/convolution = Conv2D[T=DT_FLOAT, _output_shapes=[[-1,48,48,16]], data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_image_array_input_2_0_0, image_array_2/kernel/read)]]
node-server_1     |     at /usr/src/app/node_modules/grpc/src/node/src/client.js:442:17 code: 3, metadata: Metadata { _internal_repr: {} } }
node-server_1     | gender prediction failed.

And request:

MBPJ-2:client siarheiparashchanka$ npm start ./../images/cutouts/4.png

> client@1.0.0 start /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
> node request.js "./../images/cutouts/4.png"

Reading from path: /Users/siarheiparashchanka/sites/keras-serving/face-recog/images/cutouts/4.png
image flat size: 48.
Loaded image with size: 2304
Making request to CNN GRPC wrapper.
POST to http://localhost:8080/predict-emotion-and-gender with 43601 body length.
{
  "error": {
    "code": 3,
    "metadata": {
      "_internal_repr": {}
    }
  }
}

[krystianity]

Strange :/

On which OS are you running this? MacOSX?
I will try to set this up on a clean VM tomorrow, I cannot reproduce this error using Ubuntu 16.04

[krystianity]

Okay so I have spun up a clean Ubuntu 16.04 LTS VM and ran through the setup as defined in the READMEs. I have found 2 potential errors regarding the naming of the input vectors of the gender model as well as the output vectors, its possibly related to some strange docker-compose caching behavior that I wasnt able to see these locally.

The bugs are fixed in the current master, so please go ahead and run:

cd /keras-prod
git checkout .
git pull
python ./face-recog/export.py
./start-dual-servers.sh
#..
cd face-recog/client
npm start ./../images/cutouts/4.png

again :)

[jerminal]

Cool. Now works!

[krystianity]

Awesome :)

Let me know if you want to take this in another direction.

I am currently trying to:

• setup a Google Cloud BigQuery large dataset batch-training example
• reduce the size of the tf-model-server image, 3,4 GB is way too large
• *?

[jerminal]

Maybe some preprocessing images. For example, I send big ".jpg" image and it reshapes it and responses with the output of models.

And also maybe more details how to add your own models and make it work.