feat: capacity to deploy sharded MongoDB #1321
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tschneider-aneo
force-pushed
the
ts/integrate-mongodb-sharding
branch
from
fb06fc5 to
e69c53c
Compare
tschneider-aneo
changed the title
This was referenced Oct 1, 2024
Comment on lines
200
to
201
| min_size = 5 | ||
| desired_size = 5 |
There was a problem hiding this comment.
You could make it depend on whether we deploy the sharded mongodb or not?
There was a problem hiding this comment.
I guess so, I agree on the fact that we should design the "MongoDB nodes" to be kind of proportionnal with the resources needed for MongoDB. We'll talk about this
There was a problem hiding this comment.
You could keep min_size and desired_size to 1 and rely on the cluster autoscaler to start more nodes when you have more shards and replicas.
There was a problem hiding this comment.
Seems smart. Is there a similar mechanism for GCP deployment ?
tschneider-aneo
force-pushed
the
ts/integrate-mongodb-sharding
branch
from
e00df89 to
9bf2e6c
Compare
tschneider-aneo
force-pushed
the
ts/integrate-mongodb-sharding
branch
from
f13e115 to
be0b089
Compare
|
lemaitre-aneo
approved these changes
Nov 14, 2024
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Following aneoconsulting/ArmoniK.Infra#168 and aneoconsulting/ArmoniK.Core#760, this PR aims to enable the deployment of a sharded MongoDB as ArmoniK's database.
A more complete motivation is provided in AEP 004 (aneoconsulting/ArmoniK.Community#48). This PR is meant to treat a previsible bottleneck on ArmoniK's database by enabling a sharded architecture to it.
Description
This PR enables the possibility to deploy a sharded MongoDB in a configurable way. A new variable
mongodb_shardingdescribing the configuration wanted is introduced. It allows to define the number of shards, labels, node selectors, resources and replicas for each MongoDB entities.If this new variable is not null, a sharded MongoDB is deployed, else it is a classical one. Only one "version" is deployed at each deployment.
Testing
It has been tested manually in AWS and localhost. GCP deployment is still to be tested. Workflows testing a deployment of ArmoniK with a sharded MongoDB are currently being studied.
Impact
It is expected to enhance ArmoniK's strong scalability at the cost of simple database configuration. A sharded database is indeed more complex.
Warning
Theses developments were made using Bitnami's mongodb-sharded image and Helm chart. It is not guaranteed that they work without using these particular resources.
As Bitnami's images are not verified with Docker Scout, we advice to adapt Bitnami's mongodb-sharded image to make it compliant with your personnal or business security requirements.
As quick-deploy is an example of ArmoniK deployment meant to help understanding what ArmoniK needs to run correctly, we are not responsible for warrantying a full operationality in production environment, especially in a security-compliant point of view.
Additional Information
To better understand how MongoDB sharding works see MongoDB's documentation.
Checklist