uevent listening goroutine is not (always) working #372
Comments
|
@bergwolf I'll take a look at this. |
|
@bergwolf I have not been able to to reproduce this issue at all on my local setup. Looks like the device gets hotplugged well in advance on my setup. I havent had any luck reproducing this on an Azure instance while running the test for 10 min. I'll try to reproduce this issue on Azure tomorrow. Meanwhile, I had a look at the changes for uevent and I think change this could be causing the uevent listening loop to be locked permanently: Lines 125 to 132 in 7caf1c8 With that change the sandbox lock is never released. |
|
@amshinde Thanks for looking into it. Nice catch on the missing lock! I guess I never hit that code path though. I can see
And I tried both initrd ( Here is my complete |
|
@bergwolf I see the important difference from your environment is that you are using "q35" machine type, while I am using "pc" machine type. Just running the test for netmon, I dont think the network hotplug for q35 is even successful. I dont see any uevent related to network device for q35 at all. @sboeuf Were you able to reproduce this issue at all with "pc" machine type? |
|
@bergwolf Can you give this a try with "pc" machine type and paste the logs here if you see any issues with "pc". |
|
@bergwolf This PR should solve the hotplug issue for q35: For pc, I am wondering if we are running into similar SHPC related issues in our CI (although I cant reproduce it on my setup). We have shpc turned on for pci-bridges in virtcontainers. So I am not sure if it has to do with the timing of scanning the PCI bus. and |
|
Thanks @amshinde ! You do find the root cause. I am able to pass the test as long as I change the machine type back to And if I disable I'll build a new kernel w/ |
|
@amshinde In my environment, with |
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
|
@amshinde I added pci bus rescanning and fixed an issue with the listening channel. See #380. With that I can pass the test w/ and w/o And the test fails if I take out the pci bus rescan commit. The device shows up a few seconds after the waiting fails. So I think we would always want to scan the pci bus for new devices to find out new devices timely. |
|
@bergwolf I'd like to understand what's going on with And yes, adding PCI rescan works since we don't need any hotplug mechanism, as we simply force PCI enumeration every time we add a new device from Qemu. |
|
And just to add a comment here, I am not against forcing the PCI rescan, but I want to make sure we're not hiding a potential lack of support regarding PCI hotplug. |
|
/cc @mcastelino But here is the error I get when I try to hotplug through the bridge: Any idea why there's not enough space for the PCI holes defined by the BARs? |
|
@sboeuf As I tested it,
We enable |
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
|
I was testing the network vfio device hotplug localy based on kata I have seen the device in the container, but the device has no driver: Machine type is |
|
@miaoyq The network hotplug of a VFIO device is not supported yet, but should land soon. I just want to make sure the standard |
@sboeuf Yeah, works very well :-) We were implementing the network hotplug of a VFIO device locally based on |
|
@miaoyq Oh okay! So I think the issue might come from the fact that your device needs a lot of memory and because it is hotplugged behind a PCI bridge, the reserved space is not large enough. hotplug_vfio_on_root_bus = truefrom the |
|
Thanks @sboeuf It seems that root bus does not support hotplug. |
|
@miaoyq are you using |
Otherwise the channel we are waiting on is not the one we put in the device watchers map. Fixes: kata-containers#372 Signed-off-by: Peng Tao <bergwolf@gmail.com>
When the client tries to connect sometimes a race condition could happen when the between bind and listen calls in the agent vsock side. This will block the hypervisor wanting for a response and as consequence the agent client where it checks for an OK response. This case needs to be handled by the guest kernel, see https://lore.kernel.org/netdev/668b0eda8823564cd604b1663dc53fbaece0cd4e.camel@intel.com/ As an extra protection make the agent client timeout if no OK response is given. The response should be quick so is OK to wait a few seconds and then timeout. This also allow to return an error from the dialler function so retry does not fallback on grpc retry making retries faster. Fixes: kata-containers#372 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
When the client tries to connect sometimes a race condition could happen when the between bind and listen calls in the agent vsock side. This will block the hypervisor wanting for a response and as consequence the agent client where it checks for an OK response. This case needs to be handled by the guest kernel, see https://lore.kernel.org/netdev/668b0eda8823564cd604b1663dc53fbaece0cd4e.camel@intel.com/ As an extra protection make the agent client timeout if no OK response is given. The response should be quick so is OK to wait a few seconds and then timeout. This also allow to return an error from the dialler function so retry does not fallback on grpc retry making retries faster. Fixes: kata-containers#372 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
When the client tries to connect sometimes a race condition could happen when the between bind and listen calls in the agent vsock side. This will block the hypervisor wanting for a response and as consequence the agent client where it checks for an OK response. This case needs to be handled by the guest kernel, see https://lore.kernel.org/netdev/668b0eda8823564cd604b1663dc53fbaece0cd4e.camel@intel.com/ As an extra protection make the agent client timeout if no OK response is given. The response should be quick so is OK to wait a few seconds and then timeout. This also allow to return an error from the dialler function so retry does not fallback on grpc retry making retries faster. Fixes: kata-containers#372 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
When the client tries to connect sometimes a race condition could happen when the between bind and listen calls in the agent vsock side. This will block the hypervisor wanting for a response and as consequence the agent client where it checks for an OK response. This case needs to be handled by the guest kernel, see https://lore.kernel.org/netdev/668b0eda8823564cd604b1663dc53fbaece0cd4e.camel@intel.com/ As an extra protection make the agent client timeout if no OK response is given. The response should be quick so is OK to wait a few seconds and then timeout. This also allow to return an error from the dialler function so retry does not fallback on grpc retry making retries faster. Fixes: kata-containers#372 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
As I was adding network hotplug support for vm factory and found out that the uevent listening goroutine is not working for me. After hotplugging a new nic, however long I wait in
UpdateInterface, the uevent listening goroutine cannot get the new device sysfs uevent.Then I tested the network monitor process (kata-containers/runtime#534), and it failed on getting the new uevent as well:
It's not that the uevent is not sent to the agent. If I remove the waiting and ignore UpdateInterface error, the device shows up very quickly and the uevent listening goroutine is able to pick up uevents for it. So it's like the goroutine is not woke up when agent is waiting in
UpdateInterfacecall. But I don't know why that is happening.Also since netmon works in @sboeuf 's local environment, I put up a testing PR (kata-containers/tests#743) to get a baseline for environment comparison. And it turns out that two times out of three, it failed on getting the IP of new hotplugged nic. So the failure/success do not seem constant even for CI.
It's not clear to me what has caused the failure/success. @sboeuf @devimc @amshinde @caoruidong help to take a look.
The text was updated successfully, but these errors were encountered: