Mesos with Spark trying to download non-existent libmesos/ubuntu:14.04 #37
Comments
|
FYI, I also tried updating deimos to 0.4.0, which was available by doing `pip install deimos --upgrade). Nothing has changed with the above issue. I might have allowed Ubuntu to update the kernel through a standard upgrade. Would I need to reinstall anything to do with Mesos after that? |
|
Try with |
|
I tried that and it still is either ignoring that option or passing in ubuntu:14.04 directly:
|
|
@rcsenkbeil I believe though I may be wrong, the deimos config file default image takes precedence over the one you configure on the mesos slave. Are you using the example config? https://github.com/mesosphere/deimos/blob/master/example.cfg#L24. |
|
Yeah, I also added the example config. Since there wasn't too much documentation on each individual option, I just guessed on what to do. The config located at /home/mesos/.deimos:
The results of running
|
|
Any chance you can dump your deimos logs here? I notice you have |
|
Another thing I can think of is that mesos master/slave/marathon are all running as root since I installed them all through deb packages rather than building from source. I guess I'd need to move the deimos config to the root level for it to apply to the running instances? But, if that's the case, there shouldn't be any config affecting mesos-master and mesos-slave, right? The results of running
|
|
@tarnfeld Sorry, I literally installed and got everything running yesterday. Where would the log you want be found? |
|
Looks like the console logging is written to the stderr file in the sandbox, can you paste the full output of that? You can get to that through the Mesos Web UI. |
|
Ah, okay. Also, it's showing up in var/log/syslog:
In terms of the stderr file, this is what I see:
|
|
A new update. I moved the .deimos file to /root since mesos was running under root. I deployed a new spark job and this is what I see: I'm going to update the VM with more resources to see if it is just really sluggish. |
|
And, now I can launch Spark, but all of the tasks become lost. From one of the tasks, I see the following in stderr: And the following in stdout: [EDIT] Below is the output from running spark-submit: |
|
Ok, progress :-) So it looks like the docker container is failing to launch (by that I mean the executor is not launching successfully inside it) – You can see the container ID as |
|
Weird, it claims the container doesn't exist. There isn't any sort of cleanup of containers, is there? I tried it with a short segment [EDIT] In fact, there doesn't seem to be any previously-run containers:
|
|
So, I tried updating the deimos config that is being used and replacing This is the new error log: To reiterate, the previous stderr from using |
|
Another interesting thing to note is that the containers only show up when listing all processes (including non-running ones) for a split second. So, I'm guessing they are being removed rather than being allowed to persist?
[EDIT] [EDIT2] For me, I'd love to have the ability to let Marathon use deimos and the Spark framework use the default containerization, which I see is being worked on. It would allow me to deploy standard Docker containers through a nice RESTful API. |
|
I believe that You could change |
|
Thanks for post. I went through the same process. Creating a new Docker container with libmesos in /usr/lib solved the UnsatisfiedLinkError. |
|
Yes. I also managed to work spark with deimos, the only thing I couldn't do is, didn't know how to pass parameters when submitting a job (different from deimos.cfg). Eg: which image should be used. (I only ran with the same image) |
|
Thats cool, thanks! |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
So, I was fiddling with Mesos on a single-node machine. I'd been using it with Spark and then found out about Deimos to utilize Docker as an external containerization medium.
It worked up until today. I've got this installed on an Ubuntu 14.04 64-bit installation in Virtualbox. Mesos master and slave are working. I also have Marathon running. Now, when I try to submit a Spark job, the first stage in the pipeline has a stderr output of the following:
I added to /etc/mesos-slave the file default_container_image with libmesos/ubuntu:13.10 inside it. After rebooting the VM, I can see that the mesos slave is started with that option:
So, I'm not sure if ubuntu:14.04 is suddenly being fed into the system or what, but I haven't touched anything aside from rebooting the image. I can't figure out how to get the Spark submission to not try to download libmesos/ubuntu:14.04 (and I'm not going through Marathon for this, obviously).
The text was updated successfully, but these errors were encountered: