Removing records from kv that exceed the limit

[SergioWhite]

Hello!

Please tell me if there is a way to remove records from kv that exceed the set limit? For example, I want to store 100 records in kv. As soon as the 101st record is added, it should displace the oldest one, as it is done in the LRU cache. I think this is a useful option.

I could find outdated records myself, but in this case I cannot guarantee that the oldest record will be removed, since after receiving it, it can be immediately updated by another user. In this unpleasant case, I will remove the current record with useful information.

I could also remove based on revision. However, you can't remove from kv based on revision, only from the stream. But I also can't remove records from the stream created with kv, since, apparently, you can't remove records directly from system streams.

At the moment, I don't see a beautiful solution to this problem. Please tell me if you have any advice. Thank you!

[derekcollison]

MaxMsgsPerSubject will do what you want and that is the default when creating a KV. You can use history to ask for N number of records per subject, e.g. 100.

[SergioWhite]

My apologies for the lack of attention! Streams do have more options, and MaxMsgsPerSubject is exactly what I need. Thank you very much!

[SergioWhite]

Maybe I was a bit hasty. Besides MaxMsgsPerSubject I saw the MaxMsgs option, and it seemed right to me. But then I realized that, for example, a message about deleting a non-existent record would displace an existing record, and I would lose useful data.

Then I tried MaxMsgsPerSubject, but as far as I understand, this option sets the limit within one key, not KV.

Just an example to demonstrate:

    const kv = await nats.jetstream().views.kv('my_kv')
    const kvStatus = await kv.status()
    const streamName = kvStatus.streamInfo.config.name

    const jsm = await this.nats.jetstreamManager()
    await jsm.streams.update(streamName, { max_msgs_per_subject: 1 }) // Here I set the limits for the stream

    await kv.put('1', '1')
    await kv.put('2', '2') // Here I expect the message '1' to be deleted from KV due to the limit.
    const value = (await kv.get('1'))?.string()
    console.log(value) // 1 <- But the message is not deleted. We can get its value

    const stream = await jsm.streams.get(streamName)
    const message1 = await stream.getMessage({ seq: 1 })
    const message2 = await stream.getMessage({ seq: 2 })
    // Here we can see that each message has different subjects, so limiting MaxMsgsPerSubject does not solve my original problem
    console.log(message1.subject) // $KV.my_kv.1
    console.log(message2.subject) // $KV.my_kv.2

Could you give me an example of how MaxMsgsPerSubject could solve my problem?

[jnmoyne]

If you want to keep a max of N messages per subject then use MaxMsgsPerSubject limit if you want to keep a max of N messages in the stream (and delete the oldest message in the stream to make room for a new one) then use the MaxMsgs limit, they are two different limits and you can set both of them at the same time if you need to, and in both cases you will want to set the discard policy to 'old' (so it deletes automatically an old message to make room for a new one).

Remember that with KV buckets, when you explicitly delete a key, it actually places a marker 'deleted' message in the stream, and those markers could end up being the ones deleted to make room because a limit is reached (may or may not matter depending on your use case, if it does just directly use a stream or use a stream.purge directly on the KV's stream to delete keys).