Why JetStream doesn't achieve linear scalability through partitioning?

[penghuazhou]

If a topic has a data volume of tens of millions, does it need to be split into multiple streams to support it. Because streams cannot be split and stored on multiple machines, they can only be split into multiple streams from the business side, as shown in the following example.

nats stream add ORDERS --subjects "orders.partition0,orders.partition1"

package main

import (
    "fmt"
    "github.com/nats-io/nats.go"
)

func main() {
    nc, _ := nats.Connect(nats.DefaultURL)
    js, _ := nc.JetStream()

    // 模拟消息
    messages := []string{"msg1", "msg2", "msg3", "msg4"}

    for _, msg := range messages {
        // 分区规则：轮询分区
        partition := len(msg) % 2
        subject := fmt.Sprintf("orders.partition%d", partition)

        // 发布消息
        _, err := js.Publish(subject, []byte(msg))
        if err != nil {
            fmt.Println("Error publishing message:", err)
        } else {
            fmt.Printf("Published '%s' to %s\n", msg, subject)
        }
    }

    nc.Close()
}

[derekcollison]

A partition in Kafka is equivalent to a stream in NATS.

[penghuazhou]

thanks

[jnmoyne]

You can also leverage the built-in core NATS subject mapping functionality to automatically have NATS insert a partition number as part of the subject when the message is published (meaning you can control and adjust the number of partitions (or which part(s) of the subject the partition number is calculated from) administratively rather than doing the partitioning at the client code level (if you would rather avoid that)).
https://docs.nats.io/nats-concepts/subject_mapping

[penghuazhou]

@jnmoyne， If there is only one replicas of the stream, will this stream be deployed only to one NATS server? Will it still be deployed to multiple NATS servers?

[jnmoyne]

If the stream is R1 it's only handled by one NATS server. If it's R3 by 3 server, R5 by 5 servers, etc... You can also use 'placement tags' and tag your servers and then affect which server(s) is(are) selected to participate in the stream. See https://docs.nats.io/nats-concepts/jetstream/streams#placement

[penghuazhou]

thanks