Allow option to keep the containers alive

[Can-Sahin]

It would be nice to keep the docker containers alive for speeding up the test runs. Now, in every test run containers are re-created and can't keep them alive since this block

  def __del__(self):
        """
        Try to remove the container in all circumstances
        """
        if self._container is not None:
            try:
                self.stop()
            except:  # noqa: E722
                pass

removes containers when the instance variable is deleted (when the program terminates) and I cannot override it because this also runs without using with as block.

I don't want my Mysql container to be recreated everytime. I am running tests very frequently and I am waiting couple seconds everytime. Pretty annoying.

I can make a small PR if that makes sense?

[stonecharioteer]

You could subclass it and overwrite the __del__, since that gives you an option to inject the behaviour that you require. That's the approach we took when we needed an alternative behaviour for the Selenium container wrapper.

[tillahoffmann]

@Can-Sahin, makes sense to want to keep the container alive to speed up tests. Having said that, I'm not sure how you'd be able to connect to the existing container after the reference to the instance variable has been discarded.

If you're using pytest, you can use global fixtures to reuse the same container across all test runs within the same process. Reusing an existing container across different test runs/processes would require persisting information about the container across runs---container management should probably be outside the scope of this project.

[tillahoffmann]

Closing this one for now but feel free to reopen if using fixtures with a different scope cannot address the problem.

[mausch]

I figured out the same workaround mentioned in #109 (comment) , then found this issue about keeping containers alive.

I think this is worth documenting at least. I was surprised to see that my containers were getting deleted even when I explicitly didn't start them with a context. I had to dig around the code of testcontainers to figure out that it was __del__ doing this.

Also note that the .NET implementation of testcontainers doesn't auto-delete containers if the instance isn't wrapped in the equivalent of a with statement. There's no right or wrong here but it would be nice if testcontainers behaved roughly the same across languages. I'm not saying which impl should be changed 🙂

I'm not sure how you'd be able to connect to the existing container after the reference to the instance variable has been discarded.

I've used the docker client for Python to get the existing container if available+running then get its port, it's just a few lines of code.

container management should probably be outside the scope of this project.

Isn't this what testcontainers-java are managing with testcontainers/testcontainers-java#781 ?

[tillahoffmann]

Sounds like there is sufficiently broad interest in this feature. We could add a remove: bool = True keyword argument to the constructor and keep the container alive if not remove. Note that a PR would now need to modify the atexit registration once #208 is merged.

[n1ngu]

I am testing a sort of ETL that connects to quite a lot of testcontainers and this feature would be awesome.

I imagine a workflow similar to the reutilization of DB schemas in pytest-django: an optional flag telling whether to try to reuse already running containers and a second flag telling to stop and respawn them. https://pytest-django.readthedocs.io/en/latest/database.html#example-work-flow-with-reuse-db-and-create-db Sorry if this idea is very tailored for pytest.

Yet, for CI environments it would be ideal if any containers were dropped anyway, because some CI systems like jenkins may use long-lived build agents and littering a bunch of containers in every test run would come unhealthy. Maybe the somehow ubiquitous CI environment variable could be honored as an override?

Maybe this should be up to the user, but this library could document a canonical snippet to get it working, as well as redesigning the __del__ method so that it was possible out-of-the-box without major hacks.

[delicb]

I had a need for this (well, not a need, just annoyance that running even a single test takes couple of seconds). Since I was not ready to give up testing using real database (in my case Postgres) and wanted to continue using testcontainer, my solution is below. I am sharing it in case someone else needs it, but I am happy to contribute with PR if maintainers think solution is acceptable (which would be more simple to implement directly in the project, since there would be no need to subclass DockerClient and no need to reimplement same things in subclass if base class can be changed, no accessing to semi-private members and functions, etc).

My solution requires that containers have name defined. Than, it uses that name to reuse container across the runs. That way, there is no need for testcontainers to know which container to reuse, it is moved to user.

I am using python 3.12, so some syntax might not work in older versions, that can be easily fixed in a real PR.

from __future__ import annotations

import os
import logging
import atexit

from docker.models.containers import Container
from testcontainers.postgres import PostgresContainer
from testcontainers.core.container import DockerContainer
from testcontainers.core.docker_client import DockerClient as OriginalDockerClient, _stop_container
from typing import override, Self


logger = logging.getLogger(__name__)


class DockerClient(OriginalDockerClient):
    @override
    def run(self, image: str,
            command: str = None,
            environment: dict = None,
            ports: dict = None,
            detach: bool = False,
            stdout: bool = True,
            stderr: bool = False,
            remove: bool = False,
            stop_at_exit: bool = True,
            **kwargs) -> Container:
        """
        Default implementation with configurable container stopping atexit.
        """
        container = self.client.containers.run(
            image,
            command=command,
            stdout=stdout,
            stderr=stderr,
            remove=remove,
            detach=detach,
            environment=environment,
            ports=ports,
            **kwargs
            )
        if stop_at_exit:
            atexit.register(_stop_container, container)
        return container

    def find_container_by_name(self, name: str) -> Container | None:
        for cnt in self.client.containers.list(all=True):
            if cnt.name == name:
                return cnt
        return None


class PermanentContainer(DockerContainer):
    """
    Docker testing container that has ability to keep using same container
    for multiple test runs.

    If it is configured so, it does not destroy container
    at the end of test run. Same container is reused next time. It is required to
    configure container name when this feature is used, since name of the container
    is used to find container from previous run.

    To enable, pass "keep_container" to init or use "with_keep_container" function.

    Default value is based on detection of CI environment. CI environment is considered
    every environment that has "CI" environment variable set. If current env is CI,
    keep_container is False by default, otherwise it is True by default.
    """

    def __init__(self, *args, **kwargs):
        self._keep_container = kwargs.pop("keep_container",  self._default_keep_container())
        super().__init__(*args, **kwargs)

    @override
    def start(self) -> Self:
        # return as fast as possible, if not using keep container
        if not self._keep_container:
            self._create_and_start_container()
            return self

        if self._keep_container and not self._name:
            raise Exception("If keep_container is used, name of container must be set")

        existing_container = self.get_docker_client().find_container_by_name(self._name)
        if existing_container:
            if existing_container.status != "running":
                existing_container.start()
            logger.info("Using existing container %s", existing_container.id)
            self._container = existing_container
            return self

        # since container is not found, this is probably the first run
        self._create_and_start_container()
        return self

    def _create_and_start_container(self):
        # copy of super().start() but with some parameter overrides.
        self._container = self.get_docker_client().run(
            self.image,
            command=self._command,
            detach=True,
            environment=self.env,
            ports=self.ports,
            name=self._name,
            volumes=self.volumes,
            stop_at_exit=not self._keep_container,
            **self._kwargs
        )

    @override
    def stop(self, force=True, delete_volume=True):
        if self._keep_container:
            return
        return super().stop(force, delete_volume)

    @override
    def get_docker_client(self) -> DockerClient:
        return DockerClient()

    def with_keep_container(self, keep: bool = True):
        self._keep_container = keep

    @staticmethod
    def _default_keep_container() -> bool:
        return os.getenv("CI", None) is None


class PermanentPostgresContainer(PostgresContainer, PermanentContainer):
    """
    Postgres variant of permanent container. See docs for PermanentContainer for details.
    """
    pass

[alexanderankin]

#314

[emersonfoni]

In case it's a good source of inspiration, Testcontainers for Java implemented reusable containers (description, PR).

My use case is a bit different than the one discussed above. I'd like to run a pytest test suite. When I'm running the tests locally, I'd like to bring up a Postgres container if it's not already running, or reuse it if it is running to speed up my test run. When I'm running the tests on CI, I'd like to use a GitHub Actions Postgres service container, and not try to bring up a Postgres container at all. I figure both use cases can be solved by "don't bring up a container if one exists".

[alexanderankin]

When i was introduced to testcontainers at work the first time, we implemented logic to use local db and fall back on testcontainers; the api has changed and reuse is great - PRs would be welcome to implement it.

[matthiasschaub]

We also have a need for keeping testcontainer alive across consecutive test runs with the goal of making reducing test run times on developers machines.

We took a look at the solution @delicb proposed and with a little tweaking it worked well for our use case.

Furthermore, we also took a look at the Java implementation of this feature, which is very similar.

Would a PR, implementing a "reuse" feature along the lines of the Java implementation, be welcomed?

Couple of details which would need to be decided on.

• Class Container should have a reusable: bool attribute do signal if it can be reused. What should be the default value?
• Enable or disable reuse should be possible by setting a environment variable (e.g. TESTCONTAINERS_REUSE_DISABLE=true) or setting a property in testcontainers.properties (e.g. testcontainers.reuse.enable=true). Which option is preferred? What should the default value be?
• How should existing containers be identified? The Java implementation uses a label containing a hash of the docker run command as value, which is probably better the enforcing setting a name on the container.

Anything else we should take into account when implementing a prototype?

[alexanderankin]

@matthiasschaub it sounds like a great plan - happy to review if PR is submitted

https://stackoverflow.com/a/73192262