/v1.32/networks/create "failed to respond" #17640
Comments
github-actions
bot
added
macos
|
If you make a single network create call from that java library does it work? Your jounald output seems incomplete (at least it did not log the API calls from your testsuite), You can filter the podman output with |
Yes, ordinarily it works without issue. Earlier in the tests I see client side output like this: but then later I see a failure like this. And like say above, I I fix testcontainers to that it is capable of retrying a failed network create, it works.
I'm not seeing any output when the tests are run. The tests using testcontainers create containers and networks via the Docker API. journalctl --user -u podman.service |
|
I think I'm being hit by the same or similar issue. The tests run fine for the first use of test containers. However, on the second attempt to start a Kafka container, we're getting the following failure: |
|
same issue for me, with testcontainers |
|
Still seeing this issue on newer podman: |
|
Do you all use podman machine on macos? Or does this also happen on linux? |
|
@Luap99 Amongst our team (both Mac and Linux developer environments), we are only seeing this problem on the Mac (and the problem is seen by >1 developers.) I reiterate my offer above to help get the problem tracked down. If I can run with extra logging turned up to help I will. Just let us know what you need. |
|
updating to testcontainers 1.18.0 fixed it for me |
|
The gvproxy process is used to proxy all the traffic between VM and host on macos. So I assume you would hit a bug in gvproxy if you do not see it on linux. |
@abentele - you are probably benefiting from testcontainers/testcontainers-java#6667 which was part of 1.18.0. That fix allows testcontainers to retry the network creation if it fails. Previously network creation was not subjected to the retry. We are running 1.18.0 and confirm we still see the issue in podman/mac. We are working around by setting |
|
Facing the same issue on Mac (Intel) trying to start container:
|
|
We new use cases in our software, we are seeing this problem more frequently and affecting other Docker API calls. Unfortunately a simple retry is not a viable workaround in all cases. I'm confirming that I'm still seeing this, even on 4.5.1. I've been unsuccessful in reproducing the problem outside of Testcontainers.
Are there instructions on how to turn how to turn up logging in gvproxy? |
|
I suspect testcontainers/testcontainers-java#7310 is related if not the root cause of the issues with Network creation. |
|
To summarise that ticket ^ we think it's a bad interaction between the podman api server HTTP idle timeout and the default connection keepalive timeout in the apache HTTP client that is used by testcontainers. Connections are pooled in the client with a default keepalive of 3 minutes. Idle connections are closed on the podman side after idling 10 seconds, which is only discovered by the HTTP client when it tries to use the connection again. By default there is 1 retry in the HTTP client, so if that retry also picks another pooled connection in this state, then the exception is thrown. This server side 10 second idle timeout on the connections by default is quite different from Docker where connections look to be very long lived (no idle timeout?). Is there any appetite for increasing or removing the idle timeout in the podman api server so it behaves more like the docker one? |
|
Thanks for the investigation. If you want to increase the timeout you need to increase the inactivity timeout of the service, i.e. |
|
Thanks @Luap99, there's also a non-standard response header FYI: We are also exploring client-side fixes as it's perhaps better to make the client resilient, just wondering if there's any improvements on the server. |
|
Podman adding Keep-Alive headers to the response sounds like a good idea to me. It would help Podman will play nicely with existing tooling such as docker-java and things built on top, such as https://github.com/testcontainers/testcontainers-java. That can only help Podman adoption. I do agree, of course, that the client side should be made resilient too. |
@Luap99 separately, feeding back on the suggestion to try What's the simplest way to tell podman (on the Mac) that you want a machine in that configuration? I don't see any podman machine ssh
sudo su
systemctl disable podman.socket
socat - UNIX-CONNECT:/run/podman/podman.sock. # expecting No such file or directory
/usr/bin/podman --log-level=info system service -t 0I couldn't get a I found a couple of files within the VM of that name. I wasn't sure which I was to modify or exactly how. |
|
add this to There is no way to do that by default with podman machine commands. Copying existing config files from the host to the VM on creating time is worth a separate RFE. As far as setting the |
I confirm that works. Thank you.
I can open a separate GitHub issue if that helps. Let me know.
I've been learning more about the I see now that it is prohibited from HTTP2/3 and looking around, I actually see few open implementations actually using it (https://issues.apache.org/jira/browse/HTTPCLIENT-781 seems to be an outlier). As far as I can see, there's no replacement for the |
Up to you, if you want this as a feature then it is worth to discus it. |
|
I am going to close this given this seems to be mostly a client issue. I don't think we should mess with the keep-alive headers in podman as this is managed by go. |
github-actions
bot
added
the
locked - please file new issue/PR
Issue Description
I have a suite of integration tests using testcontainers to automate Apache Kafka tests. Each test creates containers for Kafka and Zookeeper and a Podman Network which is torn down after each test. After succession of several passing tests, I see a
/v1.32/networks/createcall fail. Podman appears to close the socket without sending a HTTP response.The problem is 100% repeatable for me and always occurs in exactly the same spot in the test suite.
If I fix a bug in testcontainers that allows it to retry the network POST, the problem disappears.
Steps to reproduce the issue
I haven't yet been able to reproduce the issue outside test suite. There's something about the workload of the test that appears to trip the defect.
Steps to reproduce the issue
Describe the results you received
No response to network creation HTTP request.
Describe the results you expected
Network created successfully.
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
Additional environment details
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
I checked the podman machine's journalctl, I see nothing suspicious in that log.
podman-machine-journalctl.txt
The text was updated successfully, but these errors were encountered: