mvcc: database space exceeded

[CH-anhngo]

we set --auto-compaction-mode revision --auto-compaction-retention 25000 but usually getting mvcc: database space exceeded

+--------------------------------------------------+------------------+---------+---------+-----------+-----------+------------+
|                     ENDPOINT                     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://localhost:2379 | 404484f5ef90cf4c |  3.3.13 |  2.1 GB |     false |      2975 |   53109325 |
+--------------------------------------------------+------------------+---------+---------+-----------+-----------+------------+

should we increase the quota-backend-bytes to bigger size ? or do folks have to run daily/hourly defrag ?
what is best practice to benchmark the capacity ..etc ? how do we know we are under capacity regarding the db size ? #10312 #8009

[jingyih]

should we increase the quota-backend-bytes to bigger size ?

Probably. The actual answer depends on how much space is needed in your use case. This can be estimated by: How many keys are stored in etcd? How big are they? How often do they get updated?

or do folks have to run daily/hourly defrag ?

Defrag blocks all reads and writes, so probably don't defrag frequently. I think a good practice is to monitor the db size frequently and only defrag if needed.

[WIZARD-CXY]

@CH-anhngo the old etcd's storage size limit is small. Try to use the latest version, it can support 100GB data, I almost not run defrag any more since it's absolutely enough for my case.

[CH-anhngo]

yep we are running the latest v3.3.13 and bump the size limit to 3G for now ...it is staying at 2.7G now. :(

[spzala]

@CH-anhngo I guess we can close this issue but please reopen or create a new one if you feel a need. As it's mentioned in the issue comments, there aren't best practices that fits all but the documentations below discusses compaction, defrag, quota in more details which you might already read.
https://etcd.io/docs/v3.3.12/op-guide/maintenance/
https://etcd.io/docs/v3.4.0/op-guide/maintenance/
Thanks!

[zorro786]

Probably. The actual answer depends on how much space is needed in your use case. This can be estimated by: How many keys are stored in etcd? How big are they? How often do they get updated?

@jingyih Curious to know more about estimation. Do you have any docs on how estimation can be done? Mainly interested to map frequency of updates to estimated size.