Ingester zone questions

[changliu-wk]

We enabled ingester zoneAwareReplication when we started mimir at the first place, and each zone has three pods. But now we find within each zone, not all pods belong to one aws availability zone,

PVC Name                   AWS Availability Zone 
storage-mimir-ingester-zone-a-0	us-east-1b
storage-mimir-ingester-zone-a-1	us-east-1b
storage-mimir-ingester-zone-a-2	us-east-1c
storage-mimir-ingester-zone-b-0	us-east-1b
storage-mimir-ingester-zone-b-1	us-east-1b
storage-mimir-ingester-zone-b-2	us-east-1c
storage-mimir-ingester-zone-c-0	us-east-1b
storage-mimir-ingester-zone-c-1	us-east-1b
storage-mimir-ingester-zone-c-2	us-east-1c

From my understanding, this is a wrong state for the zone, zone-a should go to one aws availability zone, right?

[changliu-wk]

Could someone help? I think ideally, it should be something like this:

PVC Name                   AWS Availability Zone 
storage-mimir-ingester-zone-a-0	us-east-1a
storage-mimir-ingester-zone-a-1	us-east-1a
storage-mimir-ingester-zone-a-2	us-east-1a
storage-mimir-ingester-zone-b-0	us-east-1b
storage-mimir-ingester-zone-b-1	us-east-1b
storage-mimir-ingester-zone-b-2	us-east-1b
storage-mimir-ingester-zone-c-0	us-east-1c
storage-mimir-ingester-zone-c-1	us-east-1c
storage-mimir-ingester-zone-c-2	us-east-1c

Am I right? But currently our configuration is:

 topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: 'topology.kubernetes.io/zone'
      whenUnsatisfiable: ScheduleAnyway
  zoneAwareReplication:
    enabled: true
    maxUnavailable: 1
    topologyKey: 'kubernetes.io/hostname'

This generate yaml file like this:

 affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: rollout-group
                operator: In
                values:
                - ingester
              - key: zone
                operator: NotIn
                values:
                - zone-c
            topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: mimir
            app.kubernetes.io/instance: mimir
            app.kubernetes.io/component: ingester

Which just make sure the pod not deploying with the ingester pods coming from other zones in one node, and the 3 pods from one zone can't be deployed to just one aws az.
What is the best practice for ingester zone? Any advice is appreciate.