feat: migrate concurrency check to redis

[fregataa]

the implementation of 1st approach in lablup/backend.ai#365

Test

TL; DR

Migrating concurrency check to redis works without malfunctions. Other tables such as kernels causes a bottleneck but this is another issue.
Creating a keypair_resource_usages table occurs many bottlenecks and consequently raises canceled error.

1

python run.py start --cpu=0.5 --mem=0.5g -n 100 -d 2 --max-parallel=10 sleep

The number of sessions is too big to examine the change.

The bottleneck caused by keypairs.concurrency_used does not occur but another error appears.

All sessions raise TimeoutError and CancelledError during methods manager.registry.AgentRegistry.sync_kernel_stats and manager.registry.AgentRegistry.mark_agent_terminated

2

python run.py start --cpu=0.5 --mem=0.5g -n 20 -d 0.1 --max-parallel=16 sleep

Result

Error

One error occurs in DB, kernels table.

3

python run.py start --cpu=0.5 --mem=0.5g -n 20 -d 0.1 --max-parallel=32 sleep

Result

Error

Only one error occurs. This also occurs during updating kernels table.

[codecov]

Codecov Report

Merging #535 (045027d) into main (768aff2) will decrease coverage by 0.04%.
The diff coverage is 15.38%.

❗ Current head 045027d differs from pull request most recent head a19d9d2. Consider uploading reports for the commit a19d9d2 to get more accurate results

@@            Coverage Diff             @@
##             main     #535      +/-   ##
==========================================
- Coverage   49.33%   49.28%   -0.05%     
==========================================
  Files          55       55              
  Lines        8991     9008      +17     
==========================================
+ Hits         4436     4440       +4     
- Misses       4555     4568      +13     

Impacted Files	Coverage Δ
src/ai/backend/manager/api/auth.py	47.92% <ø> (ø)
src/ai/backend/manager/api/manager.py	37.26% <ø> (ø)
src/ai/backend/manager/models/agent.py	52.04% <ø> (ø)
src/ai/backend/manager/models/vfolder.py	33.57% <ø> (ø)
src/ai/backend/manager/scheduler/dispatcher.py	24.73% <0.00%> (ø)
src/ai/backend/manager/registry.py	18.20% <5.00%> (-0.09%)	⬇️
src/ai/backend/manager/scheduler/predicates.py	50.00% <16.66%> (+2.72%)	⬆️
src/ai/backend/manager/models/keypair.py	50.38% <22.22%> (-1.22%)	⬇️
src/ai/backend/manager/models/kernel.py	44.18% <25.00%> (-0.27%)	⬇️
src/ai/backend/manager/models/user.py	35.49% <33.33%> (+0.04%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 768aff2...a19d9d2. Read the comment docs.