Comfortable way of performing Ray tests (unit/integration).

[jose-turintech]

Pre-context

This discussion mainly gets triggered by the need of including Ray support on some black-widow-data-api features, in order to solve some memory issues that system is experiencing because of the expected way of python memory management on single-process apps.

Context:

Black-widow code, in the form of complex endpoints having logic or endpoints calling services with business logic, that need to be wrapped as ray tasks/actors in order to minimise memory footprint (enforce memory deallocation to the OS) after that business process finishes execution.

Dependency injection was included on services in order to make it easy to play around with pieces, and with the intention of making tests easier on the pytest side (looking forward injecting mock instances instead of patching basically the entire codebase in order to have unit tests running).

Ray tasks (a few of them) are in place and working already after solving a problem with Database management for the business logic operations.

Problem to solve:

Execution of "unit" tests, that are way more alike to integration tests, for the services, wrapped as Ray tasks/actors, which involves running that code remotely in a Ray cluster (self-spawn or stand-alone). Problem here is how to mock pieces of that code since workingdir or py_modules will send the code to the cluster but doesn't let us mock/patch objects IN the cluster in order to control mocked responses of methods/elements etc, so basically we do ship to the cluster the real impl, with the inconvenience of external API calls, etc, and also with the blocking factor of being unable to control dependency injection happening in the Ray cluster.

Options/attemps:

Initial approach/intention was to split interfaces/impls for the services module, and being able to exclude the real impl section of desired parts of the code, while executing a remote call or the ray.init call, and send the mocked impl equivalent instead. This approach didn't work as classes in order to work with how kink works for dependency injection, need to be processed at init.py import scope, so in order to achieve that, impls should share the exact same package name in order to fool the code to think that mocked impls are real impls (same import path), and since the mocked impls are in the same project I couldn't achieve that.

Options I do see to solve the problem and have good tests for the features (in the Ray scope):

• Proper solution: real integration tests. Use testcontainers with a docker-compose that boots up the set of containers needed in order to test the features. Problem with this: black widow interacts with so many systems, that the docker-compose will be huge, number of elements will be huge, and my machine will not manage to boot everything up. Discarded for now because of such reasons.

• Creative solution on DI management 1: on the services module of the real impl, include also impls for mocks, inject a Environment Variable that we can populate with different values in order to control what elements are instantiated/injected into the execution (PRD/TEST). On the init.py manage dependency injection of elements accordingly. Downside: messy approach, really hard to manage this to be flexible on mocking specific elements in a per-test basis.

• Creative solution on DI management 2: create a new project, with the same module structure than for the impls for mocked ones, and specify that mudle on py_modules config for the ray.init on tests. Same downsides than previous option, but without the hustle to manage the env vars hell and mixing code.

• Creative solution 3: execute tests with a self-spawned cluster or a stand-alone one with testcontainers, and configure a runtime_env by provided a custom-tailored conda-env crafted with the modules needed for proper testing (including the structure swapping real impl components with mocked components, even if that means replacing files manually etc). Dowside of this: really non-trivial to manage and configure tests as real behaviour will be determined mostly by the contents of the conda env rather than exclusively what you see in the pytest code. Really hard to manage and articulate for big projects and obviously lacks the flexibility to coordinate different test scenarios (choosing different mock scopes, configurations etc) as this would imply several conda envs articulated differently. I think this would work, but would be hell on earth.

• Another "solution": assume services tested without Ray support, are correct and just perform manual tests of impls on DEV environment. Not really a solution as this would discard any tests on the cluster.

Before entering further discussion on this thread, lessons learned from this (imho):

I think we should rely on proper middleware interaction for communicating between services in order to decrease/remove the couplement between components and define proper bounding contexts for them (services talking with kafka making them event-driven instead of using internal http clients), also making it easier to test/mock microservices based on Ray. Mocking or creating 1 component that is able to mimic kafka messages in the exepcted way, would be way easier than mocking or booting N set of microservices and corresponding persistent middleware.

DI is really comfortable and works fine with standard code implementation, but it doesn't work for this kind of approach for tests, unless I'm missing some idea regarding how to arrange Di dynamically on python (maybe we could play around with a root "bootstrap_di" on root project level that arranges all di on the project and that solves this kind of issue, providing such element on tests by py_modules dynamically on the ray.init, but until now I've been unable to do that.