Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

exec: fix file handle leak with container.exec_* APIs #2320

Merged
merged 2 commits into from
Jan 13, 2023

Conversation

Lekensteyn
Copy link
Contributor

Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).

Tested with:

# Test exec_run (stream=False) - observe one less leak
make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
# Test exec_start (stream=True, fully reads from CancellableStream)
make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'

After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().

Fixes #1293
(Regression from #1130)

Signed-off-by: Peter Wu pwu@cloudflare.com


Relevant documentation:
https://2.python-requests.org//en/master/user/advanced/#body-content-workflow

Note: it is not easy to write a test to catch this issue. For one, lot of other tests also suffer from the resource leak issue (client.close() not being called). The other issue is that this error is only written to stderr.

While pytest supports the capsys fixture to capture stderr, this is not easily possible due to the use of unittest which does not allow fixtures as function parameters: https://docs.pytest.org/en/latest/unittest.html
(Potential workarounds include using request.getfixturevalue, pytest-dev/pytest#4143 or manually changing sys.stderr.)

Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).

Tested with:

    # Test exec_run (stream=False) - observe one less leak
    make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
    # Test exec_start (stream=True, fully reads from CancellableStream)
    make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'

After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().

Fixes docker#1293
(Regression from docker#1130)

Signed-off-by: Peter Wu <pwu@cloudflare.com>
@jgfoster
Copy link

This fixes the problem I observed and I hope it gets incorporated soon!

@inkychris
Copy link

Are there any technical reasons why this hasn't been merged, or has it just slipped under the radar? Tentatively tagging @shin- in case they may be able to help!

@Nitrooo
Copy link

Nitrooo commented Jan 11, 2023

@milas can you please review and possibly merge this fix? Thank you.

@milas milas requested review from milas and removed request for ulyssessouza and aiordache January 13, 2023 20:08
@milas milas self-assigned this Jan 13, 2023
Copy link
Contributor

@milas milas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR & fix!

@milas milas merged commit 34e6829 into docker:main Jan 13, 2023
felixfontein added a commit to felixfontein/community.docker that referenced this pull request Feb 11, 2023
…py#2320)

Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).

Tested with:

    # Test exec_run (stream=False) - observe one less leak
    make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
    # Test exec_start (stream=True, fully reads from CancellableStream)
    make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'

After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().

Fixes docker/docker-py#1293
(Regression from docker/docker-py#1130)

Cherry-picked from docker/docker-py@34e6829

Co-authored-by: Peter Wu <pwu@cloudflare.com>
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
felixfontein added a commit to ansible-collections/community.docker that referenced this pull request Feb 12, 2023
…py#2320) (#582)

Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).

Tested with:

    # Test exec_run (stream=False) - observe one less leak
    make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
    # Test exec_start (stream=True, fully reads from CancellableStream)
    make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'

After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().

Fixes docker/docker-py#1293
(Regression from docker/docker-py#1130)

Cherry-picked from docker/docker-py@34e6829

Co-authored-by: Peter Wu <pwu@cloudflare.com>
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ResourceWarning: unclosed <socket.socket>
5 participants