Skip to content
This repository has been archived by the owner on Nov 9, 2020. It is now read-only.

deb Installation doesn't survive VM restart #1039

Closed
lachnerd opened this issue Mar 16, 2017 · 5 comments · Fixed by #1047
Closed

deb Installation doesn't survive VM restart #1039

lachnerd opened this issue Mar 16, 2017 · 5 comments · Fixed by #1047

Comments

@lachnerd
Copy link

Hi,

i'm using "docker-volume-vsphere" on some VMs running Ubuntu 16.04.2 LTS.

I installed them as follows:
sudo dpkg -i docker-volume-vsphere_0.12.3f64805_amd64.deb

Everythings working fine i see my vsphere Volumes and i'm able to use them etc.

But if i restart the VM vpshere support for docker is gone.

docker volume ls doesn't show me any vsphere volumes, i have to install the deb again although the installed files from it are still existing (for example "/usr/local/bin/docker-volume-vsphere")

So i have to install the docker-volume-vsphere everytime system reboots wich is pretty annoying.

Any tipps on that ?

@govint
Copy link
Contributor

govint commented Mar 16, 2017

@lachnerd can you please post the file /var/log/docker-volume-vsphere.log from the VM thats showing this behavior. Post a restart the docker-volume-vsphere service should be available and can be checked either via "ps -ef|grep docker" or preferrably "service --status-all" and check the status of Docker and docker-volume-vsphere services.

@lachnerd
Copy link
Author

Hi,

thanks for answering heres the Log File:
docker-volume-vsphere.zip

service --status-all shows me only a docker service but no "docker-volume-vsphere" service
image

But docker-volume-vsphere is working:
image

@shuklanirdesh82 shuklanirdesh82 added this to the 0.13 milestone Mar 16, 2017
@msterin msterin added P1 P0 and removed P1 labels Mar 16, 2017
@msterin
Copy link
Contributor

msterin commented Mar 16, 2017

@lachnerd - reading through the message you seem to be doing the following steps (please confirm so we can try to reproduce and investigate) :

  1. install docker (BTW - which version ?)
  2. Install docker-volume-plugin Release 0.12 with dpkg -i command (no apt-get install -f was run)
  3. checked that the plugin works (docker volume ls ,docker volume create work)
  4. rebooted Ubuntu 16.04.2 LTS.
  5. after reboot, docker volume ls does not show vsphere volumes (BTW - is there a delay in command output ?)
  6. reinstalled the plugin with dpkg -i command
  7. checked service --status-all - no plugin there
  8. checked docker volume ls again and it worked.

Is it correct ? What steps are missing / were different ?

The plugin missing in service --status-all is a bug - we won't fix it since we are moving to Docker Managed Plugins anyways, and the current plan is to have in in 0.13 by the end of March. Meanwhile systemctl status docker-volume-vsphere can be used to check for the status

if you can send the results of journalctl -u docker it would help us to compare with internal test logs.

@lachnerd
Copy link
Author

Hi,

  1. OS: Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64) - Docker: Docker version 17.03.0-ce, build 3a232c8 - previously installed 1.13.1
  2. yes - sudo dpkg -i docker-volume-vsphere_0.12.3f64805_amd64.deb
  3. correct
  4. correct
  5. correct - and i noticed a delay from about maybe 20 s
  6. correct
  7. yes no volume plugin service only docker
  8. yap

systemctl status docker-volume-vsphere result:
image

Results of journalctl -u docker:
docker-journal.zip

I tried to reproduce this on a brandnew ubuntu vm, installed 17.03.0-ce and tried to install the deb File and got an error:
image
Unfortunately its in german - it says that it can't be installed becaus of missing docker-engine - so i think youre forced to have a migrated system from docker-engine to docker 17.03.0-ce to install it.

@msterin
Copy link
Contributor

msterin commented Mar 17, 2017

@lachnerd - thanks for the info. The last issue you are hitting is #1023 . It is fixed in the master so if you build the .deb you'd get it fixed. Our current plan is to have 0.13 release around end of March and this will be fixed there too.

Unfortunately the original issue is caused by Docker in 1.13 API hanging if called before full initialization . We are changing out code to accommodate an will update the issue when the build is available

msterin pushed a commit that referenced this issue Mar 17, 2017
Volume discovery was written to address chalenges with docker/plugin startup order,
and as potential auto-recovery for plugin crashes.

* We do not see crashes often (as a matter of fact, only once so far , with VMCI memory issues  - and
it' fixed).
* Docker now manages the order of plugin invocation/startup with
Managed Plugins.
* and, the last but not the least, during the discovery we make assumption about Docker API availablity,
but Docker makes API available only AFTER full startup including plugin initializion, so it could lead to
bug like #1039, where docker either hangs for 15-20 sec (like with unmanaged plugin) or steps on it's own
bugs and crash (like with managed plugin)

In the spirit of taking care of the "sunny day" first, when everything works OK and our code does not crash,
this commit turns off discovery on startup.

Managed plugin handes all restrts and remounts fine. Unmanaged plugin may have issues if we crash, but first of all
we never crash :-) and second unmanaged plugins are getting depreciated anyways
msterin pushed a commit that referenced this issue Mar 17, 2017
Volume discovery was written to address challenges with docker/plugin startup order,
and as potential auto-recovery for plugin crashes.

* We do not see crashes often (as a matter of fact, only once so far , with VMCI memory issues  - and
it' fixed).
* Docker now manages the order of plugin invocation/startup with
Managed Plugins.
* and, the last but not the least, during the discovery we make assumption about Docker API availability,
but Docker makes API available only AFTER full startup including plugin initialization, so it could lead to
bug like #1039, where docker either hangs for 15-20 sec (like with legacy plugin) or steps on it's own
bugs and crash (like with managed plugin)

In the spirit of taking care of the "sunny day" first, when everything works OK and our code does not crash,
this commit turns off discovery on startup.

Managed plugin handles all restarts and remounts fine. Legacy plugin may have issues if we crash, but first of all
we never crash :-) and second legacy plugins are getting depreciated anyways
msterin pushed a commit that referenced this issue Mar 20, 2017
Volume discovery was written to address challenges with docker/plugin startup order,
and as potential auto-recovery for plugin crashes.

* We do not see crashes often (as a matter of fact, only once so far , with VMCI memory issues  - and
it' fixed).
* Docker now manages the order of plugin invocation/startup with
Managed Plugins.
* and, the last but not the least, during the discovery we make assumption about Docker API availability,
but Docker makes API available only AFTER full startup including plugin initialization, so it could lead to
bug like #1039, where docker either hangs for 15-20 sec (like with legacy plugin) or steps on it's own
bugs and crash (like with managed plugin)

In the spirit of taking care of the "sunny day" first, when everything works OK and our code does not crash,
this commit turns off discovery on startup.

Managed plugin handles all restarts and remounts fine. Legacy plugin may have issues if we crash, but first of all
we never crash :-) and second legacy plugins are getting depreciated anyways
msterin pushed a commit that referenced this issue Mar 20, 2017
Volume discovery was written to address challenges with docker/plugin startup order,
and as potential auto-recovery for plugin crashes.

* We do not see crashes often (as a matter of fact, only once so far , with VMCI memory issues  - and
it' fixed).
* Docker now manages the order of plugin invocation/startup with
Managed Plugins.
* and, the last but not the least, during the discovery we make assumption about Docker API availability,
but Docker makes API available only AFTER full startup including plugin initialization, so it could lead to
bug like #1039, where docker either hangs for 15-20 sec (like with legacy plugin) or steps on it's own
bugs and crash (like with managed plugin)

In the spirit of taking care of the "sunny day" first, when everything works OK and our code does not crash,
this commit turns off discovery on startup.

Managed plugin handles all restarts and remounts fine. Legacy plugin may have issues if we crash, but first of all
we never crash :-) and second legacy plugins are getting depreciated anyways
msterin pushed a commit that referenced this issue Mar 21, 2017
…1047)

* Disabled volume discovery and added env var to enable it if needed

Volume discovery was written to address challenges with docker/plugin startup order,
and as potential auto-recovery for plugin crashes.

* We do not see crashes often (as a matter of fact, only once so far , with VMCI memory issues  - and
it' fixed).
* Docker now manages the order of plugin invocation/startup with
Managed Plugins.
* and, the last but not the least, during the discovery we make assumption about Docker API availability,
but Docker makes API available only AFTER full startup including plugin initialization, so it could lead to
bug like #1039, where docker either hangs for 15-20 sec (like with legacy plugin) or steps on it's own
bugs and crash (like with managed plugin)

In the spirit of taking care of the "sunny day" first, when everything works OK and our code does not crash,
this commit turns off discovery on startup.

Managed plugin handles all restarts and remounts fine. Legacy plugin may have issues if we crash, but first of all
we never crash :-) and second legacy plugins are getting depreciated anyways

* commented out crash recovery test to match removing of crash recovery
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Projects
None yet
5 participants