On-demand "run now" extra command line args not used by default Mesos executor?

[mdscruggs]

This question is about expected behavior of the "extra command line arguments" provided to this endpoint:
POST /api/requests/request/{requestId}/run

Are these extra command line args ignored when using the default Mesos executor? When I provide extra args they appear in the JSON/UI for the task, but I don't see them being appended to the command and aren't in the stdout or stderr logs.

For example, if I create an on-demand request and deploy it with the command "echo" and provide the extra arg "hello world", I expect to see "hello world" appended to the command in stdout ("echo hello world") and also printed to stdout as a result; this does not happen (but I see the extra args in the JSON/UI). It DOES behave as I expect when using a custom executor type with /usr/local/bin/singularity-executor, and I'm confused why there's a difference in behavior there. It seems like the API (and UI) should prevent extra command line args when using the default Mesos executor, but maybe I just don't get what's going on or how it's intended to work!

I observed this behavior in Singularity 0.4.7 and 0.4.11 (in a local dev setup using docker and docker-compose per the instructions, checking out the relevant git tags).

[ssalinas]

Interesting, I was able to reproduce this locally as well. I see the extra args showing up in the arguments field for the mesos task info, but I am seeing the same as you for the actually output. I'll look into how the arguments field is treated in the default executor and see what's up. Thanks for reporting this.

[mdscruggs]

Sure thing @ssalinas, let me know if I can assist.

[ssalinas]

Found the issue for this. Mesos will ignore all extra arguments sent unless shell is set to false.
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L384

We were already setting this if arguments were specified in the deploy, but not if args were specified as extra in the ui. This will be fixed in #953