GCP stateless HA support via Firestore and GCS #2821

joshdurbin · 2019-07-02T21:00:08Z

This follows information in this issue and
adds support for stateless HA clusters backed by GCP services (Firestore and GCS). Screen recordings / sessions are stored in GCS while event information and backend support is implemented in Firestore.

Firestore must be configured for "Native" mode, which allows for document snapshotting.

Firestore Indexes

Firestore automatically indexes all individual fields and requires custom indexes be created for compound queries. Both the Firestore Backend and Firestore Events "Backend/Handler" check, create, and block until indexes are created. This is done to prevent query errors on cluster startup.

Two composite indexes are required for the events collection:

EventNamespace ascending, then on CreatedAt ascending
SessionID ascending, then on EventIndex ascending

One composite index is required for the cluster-data collection:

key ascending, then on expires ascending

Firestore Doc IDs

Firestore Document IDs must be unique, cannot start with periods, and cannot contain forward
slashes. In order to support more straight forward fetching but work within the requirements
of Firestore, Event Document IDs are the concatenation of the session ID (a UUID) and event type joined with a dash -, ex: 13498a42-69a8-4fa2-b39d-b0c49e346713-user.login. Firestore Backend Document IDs are a SHA1 hash of the document's key.

Both Firestore Event and Firestore Backend's support record purging; events are purged based on the created timestamp of the event as they're not updated, while records stored in the Firestore Backend are purged based on and only if they have expires set.

Realtime updates

Firestore Backend watches collection and document snapshots and reports those to all watchers. Document added and modified events are pushed as put operations to the buffer. Removal events are also pushed. Retrieval events are NOT pushed to the buffer.

gravitational-jenkins · 2019-07-02T21:00:09Z

Can one of the admins verify this patch?

klizhentas · 2019-07-16T21:11:39Z

lib/backend/firestore/firestorebk.go

+						Key: []byte(r.Key),
+						Value: []byte(r.Value),
+						Expires: expires,
+						ID: r.ID,


Research the revision property, can we use some incremental unique doc id from firestore?

klizhentas · 2019-07-16T21:14:22Z

lib/backend/firestore/firestorebk.go

+	return &lease
+}
+
+// hashKey hashes keys using SHA1 enabling fast lookups


Let's use reversible encoding here, instead of sha1 - e.g. hex or base64 depending on what works for firebase, alternatively, we can use replacement escaping symbol for \

Going w/ string replacement so that document IDs are easy to find in the Firestore UI.

klizhentas · 2019-07-16T21:17:45Z

lib/backend/firestore/firestorebk.go

+func RetryingAsyncFunctionRunner(ctx context.Context, retryConfig utils.LinearConfig, logger *log.Logger, task func() error, taskName string) {
+	retry, err := utils.NewLinear(retryConfig)
+	if err != nil {
+		logger.Errorf("bad retry parameters: %v, returning and not running", err)


General note on style log entries are proper english sentences - Bad retry parameters: %v, returning and not running. and also for errors it's the opposite -they are lowercase with no dot at the end.

klizhentas · 2019-07-16T21:21:42Z

lib/backend/firestore/firestorebk.go

+
+// watchCollection watches a firestore collection for changes and pushes those changes, events into the buffer for watchers
+func (b *FirestoreBackend) watchCollection() error {
+	snaps := b.svc.Collection(b.CollectionName).Snapshots(b.clientContext)


Need to test this on 10K nodes (you don't have to, we will do it later after merge)

I tested a new instance, created an account, and created 5 sessions for a few minutes and watched the prometheus metrics in two cases; one with just that strategy and two with the code below, generating 100_000 junk records.

The prometheus metrics without 100_000 junk records are:

# HELP audit_failed_disk_monitoring Number of times disk monitoring failed. # TYPE audit_failed_disk_monitoring counter audit_failed_disk_monitoring 0 # HELP audit_failed_emit_events Number of times emitting audit event failed. # TYPE audit_failed_emit_events counter audit_failed_emit_events 0 # HELP audit_percentage_disk_space_used Percentage disk space used. # TYPE audit_percentage_disk_space_used gauge audit_percentage_disk_space_used 16 # HELP audit_server_open_files Number of open audit files # TYPE audit_server_open_files gauge audit_server_open_files 0 # HELP auth_generate_requests Number of current generate requests # TYPE auth_generate_requests gauge auth_generate_requests 0 # HELP auth_generate_requests_throttled_total Number of throttled requests to generate new server keys # TYPE auth_generate_requests_throttled_total counter auth_generate_requests_throttled_total 0 # HELP auth_generate_requests_total Number of requests to generate new server keys # TYPE auth_generate_requests_total counter auth_generate_requests_total 3 # HELP auth_generate_seconds Latency for generate requests # TYPE auth_generate_seconds histogram auth_generate_seconds_bucket{le="0.001"} 3 auth_generate_seconds_bucket{le="0.002"} 3 auth_generate_seconds_bucket{le="0.004"} 3 auth_generate_seconds_bucket{le="0.008"} 3 auth_generate_seconds_bucket{le="0.016"} 3 auth_generate_seconds_bucket{le="0.032"} 3 auth_generate_seconds_bucket{le="0.064"} 3 auth_generate_seconds_bucket{le="0.128"} 3 auth_generate_seconds_bucket{le="0.256"} 3 auth_generate_seconds_bucket{le="0.512"} 3 auth_generate_seconds_bucket{le="1.024"} 3 auth_generate_seconds_bucket{le="2.048"} 3 auth_generate_seconds_bucket{le="4.096"} 3 auth_generate_seconds_bucket{le="8.192"} 3 auth_generate_seconds_bucket{le="16.384"} 3 auth_generate_seconds_bucket{le="32.768"} 3 auth_generate_seconds_bucket{le="+Inf"} 3 auth_generate_seconds_sum 2.5200000000000003e-07 auth_generate_seconds_count 3 # HELP backend_batch_read_requests_total Number of read requests to the backend # TYPE backend_batch_read_requests_total counter backend_batch_read_requests_total{component="backend"} 205 backend_batch_read_requests_total{component="cache"} 278 # HELP backend_batch_read_seconds Latency for batch read operations # TYPE backend_batch_read_seconds histogram backend_batch_read_seconds_bucket{component="backend",le="0.001"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.002"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.004"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.008"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.016"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.032"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.064"} 117 backend_batch_read_seconds_bucket{component="backend",le="0.128"} 200 backend_batch_read_seconds_bucket{component="backend",le="0.256"} 204 backend_batch_read_seconds_bucket{component="backend",le="0.512"} 205 backend_batch_read_seconds_bucket{component="backend",le="1.024"} 205 backend_batch_read_seconds_bucket{component="backend",le="2.048"} 205 backend_batch_read_seconds_bucket{component="backend",le="4.096"} 205 backend_batch_read_seconds_bucket{component="backend",le="8.192"} 205 backend_batch_read_seconds_bucket{component="backend",le="16.384"} 205 backend_batch_read_seconds_bucket{component="backend",le="32.768"} 205 backend_batch_read_seconds_bucket{component="backend",le="+Inf"} 205 backend_batch_read_seconds_sum{component="backend"} 13.914725686999988 backend_batch_read_seconds_count{component="backend"} 205 backend_batch_read_seconds_bucket{component="cache",le="0.001"} 276 backend_batch_read_seconds_bucket{component="cache",le="0.002"} 276 backend_batch_read_seconds_bucket{component="cache",le="0.004"} 277 backend_batch_read_seconds_bucket{component="cache",le="0.008"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.016"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.032"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.064"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.128"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.256"} 278 backend_batch_read_seconds_bucket{component="cache",le="0.512"} 278 backend_batch_read_seconds_bucket{component="cache",le="1.024"} 278 backend_batch_read_seconds_bucket{component="cache",le="2.048"} 278 backend_batch_read_seconds_bucket{component="cache",le="4.096"} 278 backend_batch_read_seconds_bucket{component="cache",le="8.192"} 278 backend_batch_read_seconds_bucket{component="cache",le="16.384"} 278 backend_batch_read_seconds_bucket{component="cache",le="32.768"} 278 backend_batch_read_seconds_bucket{component="cache",le="+Inf"} 278 backend_batch_read_seconds_sum{component="cache"} 0.058935650000000006 backend_batch_read_seconds_count{component="cache"} 278 # HELP backend_batch_write_requests_total Number of batch write requests to the backend # TYPE backend_batch_write_requests_total counter backend_batch_write_requests_total{component="cache"} 70 # HELP backend_batch_write_seconds Latency for backend batch write operations # TYPE backend_batch_write_seconds histogram backend_batch_write_seconds_bucket{component="cache",le="0.001"} 68 backend_batch_write_seconds_bucket{component="cache",le="0.002"} 69 backend_batch_write_seconds_bucket{component="cache",le="0.004"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.008"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.016"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.032"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.064"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.128"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.256"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.512"} 70 backend_batch_write_seconds_bucket{component="cache",le="1.024"} 70 backend_batch_write_seconds_bucket{component="cache",le="2.048"} 70 backend_batch_write_seconds_bucket{component="cache",le="4.096"} 70 backend_batch_write_seconds_bucket{component="cache",le="8.192"} 70 backend_batch_write_seconds_bucket{component="cache",le="16.384"} 70 backend_batch_write_seconds_bucket{component="cache",le="32.768"} 70 backend_batch_write_seconds_bucket{component="cache",le="+Inf"} 70 backend_batch_write_seconds_sum{component="cache"} 0.015011324000000003 backend_batch_write_seconds_count{component="cache"} 70 # HELP backend_read_requests_total Number of read requests to the backend # TYPE backend_read_requests_total counter backend_read_requests_total{component="backend"} 1548 backend_read_requests_total{component="cache"} 2623 # HELP backend_read_seconds Latency for read operations # TYPE backend_read_seconds histogram backend_read_seconds_bucket{component="backend",le="0.001"} 1548 backend_read_seconds_bucket{component="backend",le="0.002"} 1548 backend_read_seconds_bucket{component="backend",le="0.004"} 1548 backend_read_seconds_bucket{component="backend",le="0.008"} 1548 backend_read_seconds_bucket{component="backend",le="0.016"} 1548 backend_read_seconds_bucket{component="backend",le="0.032"} 1548 backend_read_seconds_bucket{component="backend",le="0.064"} 1548 backend_read_seconds_bucket{component="backend",le="0.128"} 1548 backend_read_seconds_bucket{component="backend",le="0.256"} 1548 backend_read_seconds_bucket{component="backend",le="0.512"} 1548 backend_read_seconds_bucket{component="backend",le="1.024"} 1548 backend_read_seconds_bucket{component="backend",le="2.048"} 1548 backend_read_seconds_bucket{component="backend",le="4.096"} 1548 backend_read_seconds_bucket{component="backend",le="8.192"} 1548 backend_read_seconds_bucket{component="backend",le="16.384"} 1548 backend_read_seconds_bucket{component="backend",le="32.768"} 1548 backend_read_seconds_bucket{component="backend",le="+Inf"} 1548 backend_read_seconds_sum{component="backend"} 0.0026599250000000035 backend_read_seconds_count{component="backend"} 1548 backend_read_seconds_bucket{component="cache",le="0.001"} 2623 backend_read_seconds_bucket{component="cache",le="0.002"} 2623 backend_read_seconds_bucket{component="cache",le="0.004"} 2623 backend_read_seconds_bucket{component="cache",le="0.008"} 2623 backend_read_seconds_bucket{component="cache",le="0.016"} 2623 backend_read_seconds_bucket{component="cache",le="0.032"} 2623 backend_read_seconds_bucket{component="cache",le="0.064"} 2623 backend_read_seconds_bucket{component="cache",le="0.128"} 2623 backend_read_seconds_bucket{component="cache",le="0.256"} 2623 backend_read_seconds_bucket{component="cache",le="0.512"} 2623 backend_read_seconds_bucket{component="cache",le="1.024"} 2623 backend_read_seconds_bucket{component="cache",le="2.048"} 2623 backend_read_seconds_bucket{component="cache",le="4.096"} 2623 backend_read_seconds_bucket{component="cache",le="8.192"} 2623 backend_read_seconds_bucket{component="cache",le="16.384"} 2623 backend_read_seconds_bucket{component="cache",le="32.768"} 2623 backend_read_seconds_bucket{component="cache",le="+Inf"} 2623 backend_read_seconds_sum{component="cache"} 0.0059324000000000095 backend_read_seconds_count{component="cache"} 2623 # HELP backend_watcher_queues_total Watcher queue sizes # TYPE backend_watcher_queues_total gauge backend_watcher_queues_total{component="auth:cache"} 0 # HELP backend_watchers_total Number of active backend watchers # TYPE backend_watchers_total gauge backend_watchers_total{component="backend"} 1 backend_watchers_total{component="cache"} 5 # HELP backend_write_requests_total Number of write requests to the backend # TYPE backend_write_requests_total counter backend_write_requests_total{component="backend"} 710 backend_write_requests_total{component="cache"} 111 # HELP backend_write_seconds Latency for backend write operations # TYPE backend_write_seconds histogram backend_write_seconds_bucket{component="backend",le="0.001"} 0 backend_write_seconds_bucket{component="backend",le="0.002"} 0 backend_write_seconds_bucket{component="backend",le="0.004"} 0 backend_write_seconds_bucket{component="backend",le="0.008"} 0 backend_write_seconds_bucket{component="backend",le="0.016"} 0 backend_write_seconds_bucket{component="backend",le="0.032"} 0 backend_write_seconds_bucket{component="backend",le="0.064"} 1 backend_write_seconds_bucket{component="backend",le="0.128"} 454 backend_write_seconds_bucket{component="backend",le="0.256"} 675 backend_write_seconds_bucket{component="backend",le="0.512"} 706 backend_write_seconds_bucket{component="backend",le="1.024"} 709 backend_write_seconds_bucket{component="backend",le="2.048"} 709 backend_write_seconds_bucket{component="backend",le="4.096"} 709 backend_write_seconds_bucket{component="backend",le="8.192"} 709 backend_write_seconds_bucket{component="backend",le="16.384"} 709 backend_write_seconds_bucket{component="backend",le="32.768"} 710 backend_write_seconds_bucket{component="backend",le="+Inf"} 710 backend_write_seconds_sum{component="backend"} 114.26281696300003 backend_write_seconds_count{component="backend"} 710 backend_write_seconds_bucket{component="cache",le="0.001"} 75 backend_write_seconds_bucket{component="cache",le="0.002"} 81 backend_write_seconds_bucket{component="cache",le="0.004"} 95 backend_write_seconds_bucket{component="cache",le="0.008"} 111 backend_write_seconds_bucket{component="cache",le="0.016"} 111 backend_write_seconds_bucket{component="cache",le="0.032"} 111 backend_write_seconds_bucket{component="cache",le="0.064"} 111 backend_write_seconds_bucket{component="cache",le="0.128"} 111 backend_write_seconds_bucket{component="cache",le="0.256"} 111 backend_write_seconds_bucket{component="cache",le="0.512"} 111 backend_write_seconds_bucket{component="cache",le="1.024"} 111 backend_write_seconds_bucket{component="cache",le="2.048"} 111 backend_write_seconds_bucket{component="cache",le="4.096"} 111 backend_write_seconds_bucket{component="cache",le="8.192"} 111 backend_write_seconds_bucket{component="cache",le="16.384"} 111 backend_write_seconds_bucket{component="cache",le="32.768"} 111 backend_write_seconds_bucket{component="cache",le="+Inf"} 111 backend_write_seconds_sum{component="cache"} 0.175271155 backend_write_seconds_count{component="cache"} 111 # HELP etcd_backend_batch_read_requests Number of read requests to the database # TYPE etcd_backend_batch_read_requests counter etcd_backend_batch_read_requests 0 # HELP etcd_backend_batch_read_seconds Latency for etcd read operations # TYPE etcd_backend_batch_read_seconds histogram etcd_backend_batch_read_seconds_bucket{le="0.001"} 0 etcd_backend_batch_read_seconds_bucket{le="0.002"} 0 etcd_backend_batch_read_seconds_bucket{le="0.004"} 0 etcd_backend_batch_read_seconds_bucket{le="0.008"} 0 etcd_backend_batch_read_seconds_bucket{le="0.016"} 0 etcd_backend_batch_read_seconds_bucket{le="0.032"} 0 etcd_backend_batch_read_seconds_bucket{le="0.064"} 0 etcd_backend_batch_read_seconds_bucket{le="0.128"} 0 etcd_backend_batch_read_seconds_bucket{le="0.256"} 0 etcd_backend_batch_read_seconds_bucket{le="0.512"} 0 etcd_backend_batch_read_seconds_bucket{le="1.024"} 0 etcd_backend_batch_read_seconds_bucket{le="2.048"} 0 etcd_backend_batch_read_seconds_bucket{le="4.096"} 0 etcd_backend_batch_read_seconds_bucket{le="8.192"} 0 etcd_backend_batch_read_seconds_bucket{le="16.384"} 0 etcd_backend_batch_read_seconds_bucket{le="32.768"} 0 etcd_backend_batch_read_seconds_bucket{le="+Inf"} 0 etcd_backend_batch_read_seconds_sum 0 etcd_backend_batch_read_seconds_count 0 # HELP etcd_backend_read_requests Number of read requests to the database # TYPE etcd_backend_read_requests counter etcd_backend_read_requests 0 # HELP etcd_backend_read_seconds Latency for etcd read operations # TYPE etcd_backend_read_seconds histogram etcd_backend_read_seconds_bucket{le="0.001"} 0 etcd_backend_read_seconds_bucket{le="0.002"} 0 etcd_backend_read_seconds_bucket{le="0.004"} 0 etcd_backend_read_seconds_bucket{le="0.008"} 0 etcd_backend_read_seconds_bucket{le="0.016"} 0 etcd_backend_read_seconds_bucket{le="0.032"} 0 etcd_backend_read_seconds_bucket{le="0.064"} 0 etcd_backend_read_seconds_bucket{le="0.128"} 0 etcd_backend_read_seconds_bucket{le="0.256"} 0 etcd_backend_read_seconds_bucket{le="0.512"} 0 etcd_backend_read_seconds_bucket{le="1.024"} 0 etcd_backend_read_seconds_bucket{le="2.048"} 0 etcd_backend_read_seconds_bucket{le="4.096"} 0 etcd_backend_read_seconds_bucket{le="8.192"} 0 etcd_backend_read_seconds_bucket{le="16.384"} 0 etcd_backend_read_seconds_bucket{le="32.768"} 0 etcd_backend_read_seconds_bucket{le="+Inf"} 0 etcd_backend_read_seconds_sum 0 etcd_backend_read_seconds_count 0 # HELP etcd_backend_tx_requests Number of transaction requests to the database # TYPE etcd_backend_tx_requests counter etcd_backend_tx_requests 0 # HELP etcd_backend_tx_seconds Latency for etcd transaction operations # TYPE etcd_backend_tx_seconds histogram etcd_backend_tx_seconds_bucket{le="0.001"} 0 etcd_backend_tx_seconds_bucket{le="0.002"} 0 etcd_backend_tx_seconds_bucket{le="0.004"} 0 etcd_backend_tx_seconds_bucket{le="0.008"} 0 etcd_backend_tx_seconds_bucket{le="0.016"} 0 etcd_backend_tx_seconds_bucket{le="0.032"} 0 etcd_backend_tx_seconds_bucket{le="0.064"} 0 etcd_backend_tx_seconds_bucket{le="0.128"} 0 etcd_backend_tx_seconds_bucket{le="0.256"} 0 etcd_backend_tx_seconds_bucket{le="0.512"} 0 etcd_backend_tx_seconds_bucket{le="1.024"} 0 etcd_backend_tx_seconds_bucket{le="2.048"} 0 etcd_backend_tx_seconds_bucket{le="4.096"} 0 etcd_backend_tx_seconds_bucket{le="8.192"} 0 etcd_backend_tx_seconds_bucket{le="16.384"} 0 etcd_backend_tx_seconds_bucket{le="32.768"} 0 etcd_backend_tx_seconds_bucket{le="+Inf"} 0 etcd_backend_tx_seconds_sum 0 etcd_backend_tx_seconds_count 0 # HELP etcd_backend_write_requests Number of wrtie requests to the database # TYPE etcd_backend_write_requests counter etcd_backend_write_requests 0 # HELP etcd_backend_write_seconds Latency for etcd write operations # TYPE etcd_backend_write_seconds histogram etcd_backend_write_seconds_bucket{le="0.001"} 0 etcd_backend_write_seconds_bucket{le="0.002"} 0 etcd_backend_write_seconds_bucket{le="0.004"} 0 etcd_backend_write_seconds_bucket{le="0.008"} 0 etcd_backend_write_seconds_bucket{le="0.016"} 0 etcd_backend_write_seconds_bucket{le="0.032"} 0 etcd_backend_write_seconds_bucket{le="0.064"} 0 etcd_backend_write_seconds_bucket{le="0.128"} 0 etcd_backend_write_seconds_bucket{le="0.256"} 0 etcd_backend_write_seconds_bucket{le="0.512"} 0 etcd_backend_write_seconds_bucket{le="1.024"} 0 etcd_backend_write_seconds_bucket{le="2.048"} 0 etcd_backend_write_seconds_bucket{le="4.096"} 0 etcd_backend_write_seconds_bucket{le="8.192"} 0 etcd_backend_write_seconds_bucket{le="16.384"} 0 etcd_backend_write_seconds_bucket{le="32.768"} 0 etcd_backend_write_seconds_bucket{le="+Inf"} 0 etcd_backend_write_seconds_sum 0 etcd_backend_write_seconds_count 0 # HELP firestore_events_backend_batch_read_requests Number of batch read requests to firestore events # TYPE firestore_events_backend_batch_read_requests counter firestore_events_backend_batch_read_requests 7 # HELP firestore_events_backend_batch_read_seconds Latency for firestore events batch read operations # TYPE firestore_events_backend_batch_read_seconds histogram firestore_events_backend_batch_read_seconds_bucket{le="0.001"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.002"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.004"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.008"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.016"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.032"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.064"} 3 firestore_events_backend_batch_read_seconds_bucket{le="0.128"} 7 firestore_events_backend_batch_read_seconds_bucket{le="0.256"} 7 firestore_events_backend_batch_read_seconds_bucket{le="0.512"} 7 firestore_events_backend_batch_read_seconds_bucket{le="1.024"} 7 firestore_events_backend_batch_read_seconds_bucket{le="2.048"} 7 firestore_events_backend_batch_read_seconds_bucket{le="4.096"} 7 firestore_events_backend_batch_read_seconds_bucket{le="8.192"} 7 firestore_events_backend_batch_read_seconds_bucket{le="16.384"} 7 firestore_events_backend_batch_read_seconds_bucket{le="32.768"} 7 firestore_events_backend_batch_read_seconds_bucket{le="+Inf"} 7 firestore_events_backend_batch_read_seconds_sum 0.5001422440000001 firestore_events_backend_batch_read_seconds_count 7 # HELP firestore_events_backend_batch_write_requests Number of batch write requests to firestore events # TYPE firestore_events_backend_batch_write_requests counter firestore_events_backend_batch_write_requests 14 # HELP firestore_events_backend_batch_write_seconds Latency for firestore events batch write operations # TYPE firestore_events_backend_batch_write_seconds histogram firestore_events_backend_batch_write_seconds_bucket{le="0.001"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.002"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.004"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.008"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.016"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.032"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.064"} 2 firestore_events_backend_batch_write_seconds_bucket{le="0.128"} 13 firestore_events_backend_batch_write_seconds_bucket{le="0.256"} 14 firestore_events_backend_batch_write_seconds_bucket{le="0.512"} 14 firestore_events_backend_batch_write_seconds_bucket{le="1.024"} 14 firestore_events_backend_batch_write_seconds_bucket{le="2.048"} 14 firestore_events_backend_batch_write_seconds_bucket{le="4.096"} 14 firestore_events_backend_batch_write_seconds_bucket{le="8.192"} 14 firestore_events_backend_batch_write_seconds_bucket{le="16.384"} 14 firestore_events_backend_batch_write_seconds_bucket{le="32.768"} 14 firestore_events_backend_batch_write_seconds_bucket{le="+Inf"} 14 firestore_events_backend_batch_write_seconds_sum 1.293583958 firestore_events_backend_batch_write_seconds_count 14 # HELP firestore_events_backend_write_requests Number of write requests to firestore events # TYPE firestore_events_backend_write_requests counter firestore_events_backend_write_requests 2 # HELP firestore_events_backend_write_seconds Latency for firestore events write operations # TYPE firestore_events_backend_write_seconds histogram firestore_events_backend_write_seconds_bucket{le="0.001"} 0 firestore_events_backend_write_seconds_bucket{le="0.002"} 0 firestore_events_backend_write_seconds_bucket{le="0.004"} 0 firestore_events_backend_write_seconds_bucket{le="0.008"} 0 firestore_events_backend_write_seconds_bucket{le="0.016"} 0 firestore_events_backend_write_seconds_bucket{le="0.032"} 0 firestore_events_backend_write_seconds_bucket{le="0.064"} 0 firestore_events_backend_write_seconds_bucket{le="0.128"} 1 firestore_events_backend_write_seconds_bucket{le="0.256"} 1 firestore_events_backend_write_seconds_bucket{le="0.512"} 2 firestore_events_backend_write_seconds_bucket{le="1.024"} 2 firestore_events_backend_write_seconds_bucket{le="2.048"} 2 firestore_events_backend_write_seconds_bucket{le="4.096"} 2 firestore_events_backend_write_seconds_bucket{le="8.192"} 2 firestore_events_backend_write_seconds_bucket{le="16.384"} 2 firestore_events_backend_write_seconds_bucket{le="32.768"} 2 firestore_events_backend_write_seconds_bucket{le="+Inf"} 2 firestore_events_backend_write_seconds_sum 0.36986882 firestore_events_backend_write_seconds_count 2 # HELP gcs_event_storage_downloads Number of downloads from the GCS backend # TYPE gcs_event_storage_downloads counter gcs_event_storage_downloads 0 # HELP gcs_event_storage_downloads_seconds Latency for GCS download operations # TYPE gcs_event_storage_downloads_seconds histogram gcs_event_storage_downloads_seconds_bucket{le="0.001"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.002"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.004"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.008"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.016"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.032"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.064"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.128"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.256"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.512"} 0 gcs_event_storage_downloads_seconds_bucket{le="1.024"} 0 gcs_event_storage_downloads_seconds_bucket{le="2.048"} 0 gcs_event_storage_downloads_seconds_bucket{le="4.096"} 0 gcs_event_storage_downloads_seconds_bucket{le="8.192"} 0 gcs_event_storage_downloads_seconds_bucket{le="16.384"} 0 gcs_event_storage_downloads_seconds_bucket{le="32.768"} 0 gcs_event_storage_downloads_seconds_bucket{le="+Inf"} 0 gcs_event_storage_downloads_seconds_sum 0 gcs_event_storage_downloads_seconds_count 0 # HELP gcs_event_storage_uploads Number of uploads to the GCS backend # TYPE gcs_event_storage_uploads counter gcs_event_storage_uploads 0 # HELP gcs_event_storage_uploads_seconds Latency for GCS upload operations # TYPE gcs_event_storage_uploads_seconds histogram gcs_event_storage_uploads_seconds_bucket{le="0.001"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.002"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.004"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.008"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.016"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.032"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.064"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.128"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.256"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.512"} 0 gcs_event_storage_uploads_seconds_bucket{le="1.024"} 0 gcs_event_storage_uploads_seconds_bucket{le="2.048"} 0 gcs_event_storage_uploads_seconds_bucket{le="4.096"} 0 gcs_event_storage_uploads_seconds_bucket{le="8.192"} 0 gcs_event_storage_uploads_seconds_bucket{le="16.384"} 0 gcs_event_storage_uploads_seconds_bucket{le="32.768"} 0 gcs_event_storage_uploads_seconds_bucket{le="+Inf"} 0 gcs_event_storage_uploads_seconds_sum 0 gcs_event_storage_uploads_seconds_count 0 # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 1.1855e-05 go_gc_duration_seconds{quantile="0.25"} 2.3168e-05 go_gc_duration_seconds{quantile="0.5"} 2.8492e-05 go_gc_duration_seconds{quantile="0.75"} 4.2054e-05 go_gc_duration_seconds{quantile="1"} 0.000201878 go_gc_duration_seconds_sum 0.002210812 go_gc_duration_seconds_count 54 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 381 # HELP go_info Information about the Go environment. # TYPE go_info gauge go_info{version="go1.12.7"} 1 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge go_memstats_alloc_bytes 3.674728e+07 # HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. # TYPE go_memstats_alloc_bytes_total counter go_memstats_alloc_bytes_total 4.4393692e+08 # HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. # TYPE go_memstats_buck_hash_sys_bytes gauge go_memstats_buck_hash_sys_bytes 1.652192e+06 # HELP go_memstats_frees_total Total number of frees. # TYPE go_memstats_frees_total counter go_memstats_frees_total 4.711685e+06 # HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started. # TYPE go_memstats_gc_cpu_fraction gauge go_memstats_gc_cpu_fraction 6.643587413729206e-05 # HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. # TYPE go_memstats_gc_sys_bytes gauge go_memstats_gc_sys_bytes 2.63168e+06 # HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. # TYPE go_memstats_heap_alloc_bytes gauge go_memstats_heap_alloc_bytes 3.674728e+07 # HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. # TYPE go_memstats_heap_idle_bytes gauge go_memstats_heap_idle_bytes 1.8825216e+07 # HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. # TYPE go_memstats_heap_inuse_bytes gauge go_memstats_heap_inuse_bytes 4.3892736e+07 # HELP go_memstats_heap_objects Number of allocated objects. # TYPE go_memstats_heap_objects gauge go_memstats_heap_objects 171191 # HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. # TYPE go_memstats_heap_released_bytes gauge go_memstats_heap_released_bytes 0 # HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. # TYPE go_memstats_heap_sys_bytes gauge go_memstats_heap_sys_bytes 6.2717952e+07 # HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection. # TYPE go_memstats_last_gc_time_seconds gauge go_memstats_last_gc_time_seconds 1.563806766329105e+09 # HELP go_memstats_lookups_total Total number of pointer lookups. # TYPE go_memstats_lookups_total counter go_memstats_lookups_total 0 # HELP go_memstats_mallocs_total Total number of mallocs. # TYPE go_memstats_mallocs_total counter go_memstats_mallocs_total 4.882876e+06 # HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. # TYPE go_memstats_mcache_inuse_bytes gauge go_memstats_mcache_inuse_bytes 13888 # HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. # TYPE go_memstats_mcache_sys_bytes gauge go_memstats_mcache_sys_bytes 16384 # HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. # TYPE go_memstats_mspan_inuse_bytes gauge go_memstats_mspan_inuse_bytes 462384 # HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. # TYPE go_memstats_mspan_sys_bytes gauge go_memstats_mspan_sys_bytes 557056 # HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. # TYPE go_memstats_next_gc_bytes gauge go_memstats_next_gc_bytes 5.0030032e+07 # HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. # TYPE go_memstats_other_sys_bytes gauge go_memstats_other_sys_bytes 1.565464e+06 # HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator. # TYPE go_memstats_stack_inuse_bytes gauge go_memstats_stack_inuse_bytes 4.390912e+06 # HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. # TYPE go_memstats_stack_sys_bytes gauge go_memstats_stack_sys_bytes 4.390912e+06 # HELP go_memstats_sys_bytes Number of bytes obtained from system. # TYPE go_memstats_sys_bytes gauge go_memstats_sys_bytes 7.353164e+07 # HELP go_threads Number of OS threads created. # TYPE go_threads gauge go_threads 32 # HELP http_request_duration_microseconds The HTTP request latencies in microseconds. # TYPE http_request_duration_microseconds summary http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1569.894 http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 1863.958 http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 1863.958 http_request_duration_microseconds_sum{handler="prometheus"} 6918.215 http_request_duration_microseconds_count{handler="prometheus"} 4 # HELP http_request_size_bytes The HTTP request sizes in bytes. # TYPE http_request_size_bytes summary http_request_size_bytes{handler="prometheus",quantile="0.5"} 325 http_request_size_bytes{handler="prometheus",quantile="0.9"} 325 http_request_size_bytes{handler="prometheus",quantile="0.99"} 325 http_request_size_bytes_sum{handler="prometheus"} 1300 http_request_size_bytes_count{handler="prometheus"} 4 # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",handler="prometheus",method="get"} 4 # HELP http_response_size_bytes The HTTP response sizes in bytes. # TYPE http_response_size_bytes summary http_response_size_bytes{handler="prometheus",quantile="0.5"} 3759 http_response_size_bytes{handler="prometheus",quantile="0.9"} 3793 http_response_size_bytes{handler="prometheus",quantile="0.99"} 3793 http_response_size_bytes_sum{handler="prometheus"} 15076 http_response_size_bytes_count{handler="prometheus"} 4 # HELP remote_clusters Number inbound connections from remote clusters and clusters stats # TYPE remote_clusters gauge remote_clusters{cluster="C02WG09CHTDH"} 0 # HELP rx Number of bytes received. # TYPE rx counter rx 4527 # HELP server_interactive_sessions_total Number of active sessions # TYPE server_interactive_sessions_total gauge server_interactive_sessions_total 5 # HELP tx Number of bytes transmitted. # TYPE tx counter tx 4038

The prometheus metrics for the same activity PLUS the 100_000 junk records are:

# HELP audit_failed_disk_monitoring Number of times disk monitoring failed. # TYPE audit_failed_disk_monitoring counter audit_failed_disk_monitoring 0 # HELP audit_failed_emit_events Number of times emitting audit event failed. # TYPE audit_failed_emit_events counter audit_failed_emit_events 0 # HELP audit_percentage_disk_space_used Percentage disk space used. # TYPE audit_percentage_disk_space_used gauge audit_percentage_disk_space_used 16 # HELP audit_server_open_files Number of open audit files # TYPE audit_server_open_files gauge audit_server_open_files -1 # HELP auth_generate_requests Number of current generate requests # TYPE auth_generate_requests gauge auth_generate_requests 0 # HELP auth_generate_requests_throttled_total Number of throttled requests to generate new server keys # TYPE auth_generate_requests_throttled_total counter auth_generate_requests_throttled_total 0 # HELP auth_generate_requests_total Number of requests to generate new server keys # TYPE auth_generate_requests_total counter auth_generate_requests_total 0 # HELP auth_generate_seconds Latency for generate requests # TYPE auth_generate_seconds histogram auth_generate_seconds_bucket{le="0.001"} 0 auth_generate_seconds_bucket{le="0.002"} 0 auth_generate_seconds_bucket{le="0.004"} 0 auth_generate_seconds_bucket{le="0.008"} 0 auth_generate_seconds_bucket{le="0.016"} 0 auth_generate_seconds_bucket{le="0.032"} 0 auth_generate_seconds_bucket{le="0.064"} 0 auth_generate_seconds_bucket{le="0.128"} 0 auth_generate_seconds_bucket{le="0.256"} 0 auth_generate_seconds_bucket{le="0.512"} 0 auth_generate_seconds_bucket{le="1.024"} 0 auth_generate_seconds_bucket{le="2.048"} 0 auth_generate_seconds_bucket{le="4.096"} 0 auth_generate_seconds_bucket{le="8.192"} 0 auth_generate_seconds_bucket{le="16.384"} 0 auth_generate_seconds_bucket{le="32.768"} 0 auth_generate_seconds_bucket{le="+Inf"} 0 auth_generate_seconds_sum 0 auth_generate_seconds_count 0 # HELP backend_batch_read_requests_total Number of read requests to the backend # TYPE backend_batch_read_requests_total counter backend_batch_read_requests_total{component="backend"} 188 backend_batch_read_requests_total{component="cache"} 304 # HELP backend_batch_read_seconds Latency for batch read operations # TYPE backend_batch_read_seconds histogram backend_batch_read_seconds_bucket{component="backend",le="0.001"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.002"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.004"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.008"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.016"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.032"} 0 backend_batch_read_seconds_bucket{component="backend",le="0.064"} 88 backend_batch_read_seconds_bucket{component="backend",le="0.128"} 173 backend_batch_read_seconds_bucket{component="backend",le="0.256"} 183 backend_batch_read_seconds_bucket{component="backend",le="0.512"} 187 backend_batch_read_seconds_bucket{component="backend",le="1.024"} 188 backend_batch_read_seconds_bucket{component="backend",le="2.048"} 188 backend_batch_read_seconds_bucket{component="backend",le="4.096"} 188 backend_batch_read_seconds_bucket{component="backend",le="8.192"} 188 backend_batch_read_seconds_bucket{component="backend",le="16.384"} 188 backend_batch_read_seconds_bucket{component="backend",le="32.768"} 188 backend_batch_read_seconds_bucket{component="backend",le="+Inf"} 188 backend_batch_read_seconds_sum{component="backend"} 15.471601142000003 backend_batch_read_seconds_count{component="backend"} 188 backend_batch_read_seconds_bucket{component="cache",le="0.001"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.002"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.004"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.008"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.016"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.032"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.064"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.128"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.256"} 304 backend_batch_read_seconds_bucket{component="cache",le="0.512"} 304 backend_batch_read_seconds_bucket{component="cache",le="1.024"} 304 backend_batch_read_seconds_bucket{component="cache",le="2.048"} 304 backend_batch_read_seconds_bucket{component="cache",le="4.096"} 304 backend_batch_read_seconds_bucket{component="cache",le="8.192"} 304 backend_batch_read_seconds_bucket{component="cache",le="16.384"} 304 backend_batch_read_seconds_bucket{component="cache",le="32.768"} 304 backend_batch_read_seconds_bucket{component="cache",le="+Inf"} 304 backend_batch_read_seconds_sum{component="cache"} 0.083000583 backend_batch_read_seconds_count{component="cache"} 304 # HELP backend_batch_write_requests_total Number of batch write requests to the backend # TYPE backend_batch_write_requests_total counter backend_batch_write_requests_total{component="cache"} 70 # HELP backend_batch_write_seconds Latency for backend batch write operations # TYPE backend_batch_write_seconds histogram backend_batch_write_seconds_bucket{component="cache",le="0.001"} 64 backend_batch_write_seconds_bucket{component="cache",le="0.002"} 67 backend_batch_write_seconds_bucket{component="cache",le="0.004"} 69 backend_batch_write_seconds_bucket{component="cache",le="0.008"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.016"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.032"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.064"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.128"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.256"} 70 backend_batch_write_seconds_bucket{component="cache",le="0.512"} 70 backend_batch_write_seconds_bucket{component="cache",le="1.024"} 70 backend_batch_write_seconds_bucket{component="cache",le="2.048"} 70 backend_batch_write_seconds_bucket{component="cache",le="4.096"} 70 backend_batch_write_seconds_bucket{component="cache",le="8.192"} 70 backend_batch_write_seconds_bucket{component="cache",le="16.384"} 70 backend_batch_write_seconds_bucket{component="cache",le="32.768"} 70 backend_batch_write_seconds_bucket{component="cache",le="+Inf"} 70 backend_batch_write_seconds_sum{component="cache"} 0.036286640999999994 backend_batch_write_seconds_count{component="cache"} 70 # HELP backend_read_requests_total Number of read requests to the backend # TYPE backend_read_requests_total counter backend_read_requests_total{component="backend"} 1626 backend_read_requests_total{component="cache"} 2893 # HELP backend_read_seconds Latency for read operations # TYPE backend_read_seconds histogram backend_read_seconds_bucket{component="backend",le="0.001"} 1626 backend_read_seconds_bucket{component="backend",le="0.002"} 1626 backend_read_seconds_bucket{component="backend",le="0.004"} 1626 backend_read_seconds_bucket{component="backend",le="0.008"} 1626 backend_read_seconds_bucket{component="backend",le="0.016"} 1626 backend_read_seconds_bucket{component="backend",le="0.032"} 1626 backend_read_seconds_bucket{component="backend",le="0.064"} 1626 backend_read_seconds_bucket{component="backend",le="0.128"} 1626 backend_read_seconds_bucket{component="backend",le="0.256"} 1626 backend_read_seconds_bucket{component="backend",le="0.512"} 1626 backend_read_seconds_bucket{component="backend",le="1.024"} 1626 backend_read_seconds_bucket{component="backend",le="2.048"} 1626 backend_read_seconds_bucket{component="backend",le="4.096"} 1626 backend_read_seconds_bucket{component="backend",le="8.192"} 1626 backend_read_seconds_bucket{component="backend",le="16.384"} 1626 backend_read_seconds_bucket{component="backend",le="32.768"} 1626 backend_read_seconds_bucket{component="backend",le="+Inf"} 1626 backend_read_seconds_sum{component="backend"} 0.0034168239999999993 backend_read_seconds_count{component="backend"} 1626 backend_read_seconds_bucket{component="cache",le="0.001"} 2893 backend_read_seconds_bucket{component="cache",le="0.002"} 2893 backend_read_seconds_bucket{component="cache",le="0.004"} 2893 backend_read_seconds_bucket{component="cache",le="0.008"} 2893 backend_read_seconds_bucket{component="cache",le="0.016"} 2893 backend_read_seconds_bucket{component="cache",le="0.032"} 2893 backend_read_seconds_bucket{component="cache",le="0.064"} 2893 backend_read_seconds_bucket{component="cache",le="0.128"} 2893 backend_read_seconds_bucket{component="cache",le="0.256"} 2893 backend_read_seconds_bucket{component="cache",le="0.512"} 2893 backend_read_seconds_bucket{component="cache",le="1.024"} 2893 backend_read_seconds_bucket{component="cache",le="2.048"} 2893 backend_read_seconds_bucket{component="cache",le="4.096"} 2893 backend_read_seconds_bucket{component="cache",le="8.192"} 2893 backend_read_seconds_bucket{component="cache",le="16.384"} 2893 backend_read_seconds_bucket{component="cache",le="32.768"} 2893 backend_read_seconds_bucket{component="cache",le="+Inf"} 2893 backend_read_seconds_sum{component="cache"} 0.00754319799999999 backend_read_seconds_count{component="cache"} 2893 # HELP backend_watcher_queues_total Watcher queue sizes # TYPE backend_watcher_queues_total gauge backend_watcher_queues_total{component="auth:cache"} 0 # HELP backend_watchers_total Number of active backend watchers # TYPE backend_watchers_total gauge backend_watchers_total{component="backend"} 1 backend_watchers_total{component="cache"} 5 # HELP backend_write_requests_failed_total Number of failed write requests to the backend # TYPE backend_write_requests_failed_total counter backend_write_requests_failed_total{component="backend"} 2 # HELP backend_write_requests_total Number of write requests to the backend # TYPE backend_write_requests_total counter backend_write_requests_total{component="backend"} 877 backend_write_requests_total{component="cache"} 135 # HELP backend_write_seconds Latency for backend write operations # TYPE backend_write_seconds histogram backend_write_seconds_bucket{component="backend",le="0.001"} 0 backend_write_seconds_bucket{component="backend",le="0.002"} 0 backend_write_seconds_bucket{component="backend",le="0.004"} 0 backend_write_seconds_bucket{component="backend",le="0.008"} 0 backend_write_seconds_bucket{component="backend",le="0.016"} 0 backend_write_seconds_bucket{component="backend",le="0.032"} 0 backend_write_seconds_bucket{component="backend",le="0.064"} 1 backend_write_seconds_bucket{component="backend",le="0.128"} 460 backend_write_seconds_bucket{component="backend",le="0.256"} 846 backend_write_seconds_bucket{component="backend",le="0.512"} 867 backend_write_seconds_bucket{component="backend",le="1.024"} 877 backend_write_seconds_bucket{component="backend",le="2.048"} 877 backend_write_seconds_bucket{component="backend",le="4.096"} 877 backend_write_seconds_bucket{component="backend",le="8.192"} 877 backend_write_seconds_bucket{component="backend",le="16.384"} 877 backend_write_seconds_bucket{component="backend",le="32.768"} 877 backend_write_seconds_bucket{component="backend",le="+Inf"} 877 backend_write_seconds_sum{component="backend"} 126.3226184850002 backend_write_seconds_count{component="backend"} 877 backend_write_seconds_bucket{component="cache",le="0.001"} 81 backend_write_seconds_bucket{component="cache",le="0.002"} 98 backend_write_seconds_bucket{component="cache",le="0.004"} 111 backend_write_seconds_bucket{component="cache",le="0.008"} 135 backend_write_seconds_bucket{component="cache",le="0.016"} 135 backend_write_seconds_bucket{component="cache",le="0.032"} 135 backend_write_seconds_bucket{component="cache",le="0.064"} 135 backend_write_seconds_bucket{component="cache",le="0.128"} 135 backend_write_seconds_bucket{component="cache",le="0.256"} 135 backend_write_seconds_bucket{component="cache",le="0.512"} 135 backend_write_seconds_bucket{component="cache",le="1.024"} 135 backend_write_seconds_bucket{component="cache",le="2.048"} 135 backend_write_seconds_bucket{component="cache",le="4.096"} 135 backend_write_seconds_bucket{component="cache",le="8.192"} 135 backend_write_seconds_bucket{component="cache",le="16.384"} 135 backend_write_seconds_bucket{component="cache",le="32.768"} 135 backend_write_seconds_bucket{component="cache",le="+Inf"} 135 backend_write_seconds_sum{component="cache"} 0.25040869699999996 backend_write_seconds_count{component="cache"} 135 # HELP etcd_backend_batch_read_requests Number of read requests to the database # TYPE etcd_backend_batch_read_requests counter etcd_backend_batch_read_requests 0 # HELP etcd_backend_batch_read_seconds Latency for etcd read operations # TYPE etcd_backend_batch_read_seconds histogram etcd_backend_batch_read_seconds_bucket{le="0.001"} 0 etcd_backend_batch_read_seconds_bucket{le="0.002"} 0 etcd_backend_batch_read_seconds_bucket{le="0.004"} 0 etcd_backend_batch_read_seconds_bucket{le="0.008"} 0 etcd_backend_batch_read_seconds_bucket{le="0.016"} 0 etcd_backend_batch_read_seconds_bucket{le="0.032"} 0 etcd_backend_batch_read_seconds_bucket{le="0.064"} 0 etcd_backend_batch_read_seconds_bucket{le="0.128"} 0 etcd_backend_batch_read_seconds_bucket{le="0.256"} 0 etcd_backend_batch_read_seconds_bucket{le="0.512"} 0 etcd_backend_batch_read_seconds_bucket{le="1.024"} 0 etcd_backend_batch_read_seconds_bucket{le="2.048"} 0 etcd_backend_batch_read_seconds_bucket{le="4.096"} 0 etcd_backend_batch_read_seconds_bucket{le="8.192"} 0 etcd_backend_batch_read_seconds_bucket{le="16.384"} 0 etcd_backend_batch_read_seconds_bucket{le="32.768"} 0 etcd_backend_batch_read_seconds_bucket{le="+Inf"} 0 etcd_backend_batch_read_seconds_sum 0 etcd_backend_batch_read_seconds_count 0 # HELP etcd_backend_read_requests Number of read requests to the database # TYPE etcd_backend_read_requests counter etcd_backend_read_requests 0 # HELP etcd_backend_read_seconds Latency for etcd read operations # TYPE etcd_backend_read_seconds histogram etcd_backend_read_seconds_bucket{le="0.001"} 0 etcd_backend_read_seconds_bucket{le="0.002"} 0 etcd_backend_read_seconds_bucket{le="0.004"} 0 etcd_backend_read_seconds_bucket{le="0.008"} 0 etcd_backend_read_seconds_bucket{le="0.016"} 0 etcd_backend_read_seconds_bucket{le="0.032"} 0 etcd_backend_read_seconds_bucket{le="0.064"} 0 etcd_backend_read_seconds_bucket{le="0.128"} 0 etcd_backend_read_seconds_bucket{le="0.256"} 0 etcd_backend_read_seconds_bucket{le="0.512"} 0 etcd_backend_read_seconds_bucket{le="1.024"} 0 etcd_backend_read_seconds_bucket{le="2.048"} 0 etcd_backend_read_seconds_bucket{le="4.096"} 0 etcd_backend_read_seconds_bucket{le="8.192"} 0 etcd_backend_read_seconds_bucket{le="16.384"} 0 etcd_backend_read_seconds_bucket{le="32.768"} 0 etcd_backend_read_seconds_bucket{le="+Inf"} 0 etcd_backend_read_seconds_sum 0 etcd_backend_read_seconds_count 0 # HELP etcd_backend_tx_requests Number of transaction requests to the database # TYPE etcd_backend_tx_requests counter etcd_backend_tx_requests 0 # HELP etcd_backend_tx_seconds Latency for etcd transaction operations # TYPE etcd_backend_tx_seconds histogram etcd_backend_tx_seconds_bucket{le="0.001"} 0 etcd_backend_tx_seconds_bucket{le="0.002"} 0 etcd_backend_tx_seconds_bucket{le="0.004"} 0 etcd_backend_tx_seconds_bucket{le="0.008"} 0 etcd_backend_tx_seconds_bucket{le="0.016"} 0 etcd_backend_tx_seconds_bucket{le="0.032"} 0 etcd_backend_tx_seconds_bucket{le="0.064"} 0 etcd_backend_tx_seconds_bucket{le="0.128"} 0 etcd_backend_tx_seconds_bucket{le="0.256"} 0 etcd_backend_tx_seconds_bucket{le="0.512"} 0 etcd_backend_tx_seconds_bucket{le="1.024"} 0 etcd_backend_tx_seconds_bucket{le="2.048"} 0 etcd_backend_tx_seconds_bucket{le="4.096"} 0 etcd_backend_tx_seconds_bucket{le="8.192"} 0 etcd_backend_tx_seconds_bucket{le="16.384"} 0 etcd_backend_tx_seconds_bucket{le="32.768"} 0 etcd_backend_tx_seconds_bucket{le="+Inf"} 0 etcd_backend_tx_seconds_sum 0 etcd_backend_tx_seconds_count 0 # HELP etcd_backend_write_requests Number of wrtie requests to the database # TYPE etcd_backend_write_requests counter etcd_backend_write_requests 0 # HELP etcd_backend_write_seconds Latency for etcd write operations # TYPE etcd_backend_write_seconds histogram etcd_backend_write_seconds_bucket{le="0.001"} 0 etcd_backend_write_seconds_bucket{le="0.002"} 0 etcd_backend_write_seconds_bucket{le="0.004"} 0 etcd_backend_write_seconds_bucket{le="0.008"} 0 etcd_backend_write_seconds_bucket{le="0.016"} 0 etcd_backend_write_seconds_bucket{le="0.032"} 0 etcd_backend_write_seconds_bucket{le="0.064"} 0 etcd_backend_write_seconds_bucket{le="0.128"} 0 etcd_backend_write_seconds_bucket{le="0.256"} 0 etcd_backend_write_seconds_bucket{le="0.512"} 0 etcd_backend_write_seconds_bucket{le="1.024"} 0 etcd_backend_write_seconds_bucket{le="2.048"} 0 etcd_backend_write_seconds_bucket{le="4.096"} 0 etcd_backend_write_seconds_bucket{le="8.192"} 0 etcd_backend_write_seconds_bucket{le="16.384"} 0 etcd_backend_write_seconds_bucket{le="32.768"} 0 etcd_backend_write_seconds_bucket{le="+Inf"} 0 etcd_backend_write_seconds_sum 0 etcd_backend_write_seconds_count 0 # HELP firestore_events_backend_batch_read_requests Number of batch read requests to firestore events # TYPE firestore_events_backend_batch_read_requests counter firestore_events_backend_batch_read_requests 8 # HELP firestore_events_backend_batch_read_seconds Latency for firestore events batch read operations # TYPE firestore_events_backend_batch_read_seconds histogram firestore_events_backend_batch_read_seconds_bucket{le="0.001"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.002"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.004"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.008"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.016"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.032"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.064"} 0 firestore_events_backend_batch_read_seconds_bucket{le="0.128"} 8 firestore_events_backend_batch_read_seconds_bucket{le="0.256"} 8 firestore_events_backend_batch_read_seconds_bucket{le="0.512"} 8 firestore_events_backend_batch_read_seconds_bucket{le="1.024"} 8 firestore_events_backend_batch_read_seconds_bucket{le="2.048"} 8 firestore_events_backend_batch_read_seconds_bucket{le="4.096"} 8 firestore_events_backend_batch_read_seconds_bucket{le="8.192"} 8 firestore_events_backend_batch_read_seconds_bucket{le="16.384"} 8 firestore_events_backend_batch_read_seconds_bucket{le="32.768"} 8 firestore_events_backend_batch_read_seconds_bucket{le="+Inf"} 8 firestore_events_backend_batch_read_seconds_sum 0.694322218 firestore_events_backend_batch_read_seconds_count 8 # HELP firestore_events_backend_batch_write_requests Number of batch write requests to firestore events # TYPE firestore_events_backend_batch_write_requests counter firestore_events_backend_batch_write_requests 18 # HELP firestore_events_backend_batch_write_seconds Latency for firestore events batch write operations # TYPE firestore_events_backend_batch_write_seconds histogram firestore_events_backend_batch_write_seconds_bucket{le="0.001"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.002"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.004"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.008"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.016"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.032"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.064"} 0 firestore_events_backend_batch_write_seconds_bucket{le="0.128"} 13 firestore_events_backend_batch_write_seconds_bucket{le="0.256"} 18 firestore_events_backend_batch_write_seconds_bucket{le="0.512"} 18 firestore_events_backend_batch_write_seconds_bucket{le="1.024"} 18 firestore_events_backend_batch_write_seconds_bucket{le="2.048"} 18 firestore_events_backend_batch_write_seconds_bucket{le="4.096"} 18 firestore_events_backend_batch_write_seconds_bucket{le="8.192"} 18 firestore_events_backend_batch_write_seconds_bucket{le="16.384"} 18 firestore_events_backend_batch_write_seconds_bucket{le="32.768"} 18 firestore_events_backend_batch_write_seconds_bucket{le="+Inf"} 18 firestore_events_backend_batch_write_seconds_sum 1.896814674 firestore_events_backend_batch_write_seconds_count 18 # HELP firestore_events_backend_write_requests Number of write requests to firestore events # TYPE firestore_events_backend_write_requests counter firestore_events_backend_write_requests 6 # HELP firestore_events_backend_write_seconds Latency for firestore events write operations # TYPE firestore_events_backend_write_seconds histogram firestore_events_backend_write_seconds_bucket{le="0.001"} 0 firestore_events_backend_write_seconds_bucket{le="0.002"} 0 firestore_events_backend_write_seconds_bucket{le="0.004"} 0 firestore_events_backend_write_seconds_bucket{le="0.008"} 0 firestore_events_backend_write_seconds_bucket{le="0.016"} 0 firestore_events_backend_write_seconds_bucket{le="0.032"} 0 firestore_events_backend_write_seconds_bucket{le="0.064"} 0 firestore_events_backend_write_seconds_bucket{le="0.128"} 3 firestore_events_backend_write_seconds_bucket{le="0.256"} 6 firestore_events_backend_write_seconds_bucket{le="0.512"} 6 firestore_events_backend_write_seconds_bucket{le="1.024"} 6 firestore_events_backend_write_seconds_bucket{le="2.048"} 6 firestore_events_backend_write_seconds_bucket{le="4.096"} 6 firestore_events_backend_write_seconds_bucket{le="8.192"} 6 firestore_events_backend_write_seconds_bucket{le="16.384"} 6 firestore_events_backend_write_seconds_bucket{le="32.768"} 6 firestore_events_backend_write_seconds_bucket{le="+Inf"} 6 firestore_events_backend_write_seconds_sum 0.8536411470000002 firestore_events_backend_write_seconds_count 6 # HELP gcs_event_storage_downloads Number of downloads from the GCS backend # TYPE gcs_event_storage_downloads counter gcs_event_storage_downloads 0 # HELP gcs_event_storage_downloads_seconds Latency for GCS download operations # TYPE gcs_event_storage_downloads_seconds histogram gcs_event_storage_downloads_seconds_bucket{le="0.001"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.002"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.004"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.008"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.016"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.032"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.064"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.128"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.256"} 0 gcs_event_storage_downloads_seconds_bucket{le="0.512"} 0 gcs_event_storage_downloads_seconds_bucket{le="1.024"} 0 gcs_event_storage_downloads_seconds_bucket{le="2.048"} 0 gcs_event_storage_downloads_seconds_bucket{le="4.096"} 0 gcs_event_storage_downloads_seconds_bucket{le="8.192"} 0 gcs_event_storage_downloads_seconds_bucket{le="16.384"} 0 gcs_event_storage_downloads_seconds_bucket{le="32.768"} 0 gcs_event_storage_downloads_seconds_bucket{le="+Inf"} 0 gcs_event_storage_downloads_seconds_sum 0 gcs_event_storage_downloads_seconds_count 0 # HELP gcs_event_storage_uploads Number of uploads to the GCS backend # TYPE gcs_event_storage_uploads counter gcs_event_storage_uploads 0 # HELP gcs_event_storage_uploads_seconds Latency for GCS upload operations # TYPE gcs_event_storage_uploads_seconds histogram gcs_event_storage_uploads_seconds_bucket{le="0.001"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.002"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.004"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.008"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.016"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.032"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.064"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.128"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.256"} 0 gcs_event_storage_uploads_seconds_bucket{le="0.512"} 0 gcs_event_storage_uploads_seconds_bucket{le="1.024"} 0 gcs_event_storage_uploads_seconds_bucket{le="2.048"} 0 gcs_event_storage_uploads_seconds_bucket{le="4.096"} 0 gcs_event_storage_uploads_seconds_bucket{le="8.192"} 0 gcs_event_storage_uploads_seconds_bucket{le="16.384"} 0 gcs_event_storage_uploads_seconds_bucket{le="32.768"} 0 gcs_event_storage_uploads_seconds_bucket{le="+Inf"} 0 gcs_event_storage_uploads_seconds_sum 0 gcs_event_storage_uploads_seconds_count 0 # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 8.99e-06 go_gc_duration_seconds{quantile="0.25"} 1.4424e-05 go_gc_duration_seconds{quantile="0.5"} 2.2126e-05 go_gc_duration_seconds{quantile="0.75"} 3.2661e-05 go_gc_duration_seconds{quantile="1"} 0.000160642 go_gc_duration_seconds_sum 0.001752287 go_gc_duration_seconds_count 53 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 229 # HELP go_info Information about the Go environment. # TYPE go_info gauge go_info{version="go1.12.7"} 1 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge go_memstats_alloc_bytes 5.3468128e+07 # HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. # TYPE go_memstats_alloc_bytes_total counter go_memstats_alloc_bytes_total 4.50613008e+08 # HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. # TYPE go_memstats_buck_hash_sys_bytes gauge go_memstats_buck_hash_sys_bytes 1.63228e+06 # HELP go_memstats_frees_total Total number of frees. # TYPE go_memstats_frees_total counter go_memstats_frees_total 4.795068e+06 # HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started. # TYPE go_memstats_gc_cpu_fraction gauge go_memstats_gc_cpu_fraction 5.7552149064962624e-05 # HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. # TYPE go_memstats_gc_sys_bytes gauge go_memstats_gc_sys_bytes 2.639872e+06 # HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. # TYPE go_memstats_heap_alloc_bytes gauge go_memstats_heap_alloc_bytes 5.3468128e+07 # HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. # TYPE go_memstats_heap_idle_bytes gauge go_memstats_heap_idle_bytes 5.472256e+06 # HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. # TYPE go_memstats_heap_inuse_bytes gauge go_memstats_heap_inuse_bytes 5.7245696e+07 # HELP go_memstats_heap_objects Number of allocated objects. # TYPE go_memstats_heap_objects gauge go_memstats_heap_objects 334068 # HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. # TYPE go_memstats_heap_released_bytes gauge go_memstats_heap_released_bytes 0 # HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. # TYPE go_memstats_heap_sys_bytes gauge go_memstats_heap_sys_bytes 6.2717952e+07 # HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection. # TYPE go_memstats_last_gc_time_seconds gauge go_memstats_last_gc_time_seconds 1.563811319909208e+09 # HELP go_memstats_lookups_total Total number of pointer lookups. # TYPE go_memstats_lookups_total counter go_memstats_lookups_total 0 # HELP go_memstats_mallocs_total Total number of mallocs. # TYPE go_memstats_mallocs_total counter go_memstats_mallocs_total 5.129136e+06 # HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. # TYPE go_memstats_mcache_inuse_bytes gauge go_memstats_mcache_inuse_bytes 13888 # HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. # TYPE go_memstats_mcache_sys_bytes gauge go_memstats_mcache_sys_bytes 16384 # HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. # TYPE go_memstats_mspan_inuse_bytes gauge go_memstats_mspan_inuse_bytes 584352 # HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. # TYPE go_memstats_mspan_sys_bytes gauge go_memstats_mspan_sys_bytes 655360 # HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. # TYPE go_memstats_next_gc_bytes gauge go_memstats_next_gc_bytes 5.7333232e+07 # HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. # TYPE go_memstats_other_sys_bytes gauge go_memstats_other_sys_bytes 1.47888e+06 # HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator. # TYPE go_memstats_stack_inuse_bytes gauge go_memstats_stack_inuse_bytes 4.390912e+06 # HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. # TYPE go_memstats_stack_sys_bytes gauge go_memstats_stack_sys_bytes 4.390912e+06 # HELP go_memstats_sys_bytes Number of bytes obtained from system. # TYPE go_memstats_sys_bytes gauge go_memstats_sys_bytes 7.353164e+07 # HELP go_threads Number of OS threads created. # TYPE go_threads gauge go_threads 31 # HELP http_request_duration_microseconds The HTTP request latencies in microseconds. # TYPE http_request_duration_microseconds summary http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1607.427 http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 2152.303 http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 2223.088 http_request_duration_microseconds_sum{handler="prometheus"} 27694.036 http_request_duration_microseconds_count{handler="prometheus"} 17 # HELP http_request_size_bytes The HTTP request sizes in bytes. # TYPE http_request_size_bytes summary http_request_size_bytes{handler="prometheus",quantile="0.5"} 325 http_request_size_bytes{handler="prometheus",quantile="0.9"} 325 http_request_size_bytes{handler="prometheus",quantile="0.99"} 325 http_request_size_bytes_sum{handler="prometheus"} 5525 http_request_size_bytes_count{handler="prometheus"} 17 # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",handler="prometheus",method="get"} 17 # HELP http_response_size_bytes The HTTP response sizes in bytes. # TYPE http_response_size_bytes summary http_response_size_bytes{handler="prometheus",quantile="0.5"} 3800 http_response_size_bytes{handler="prometheus",quantile="0.9"} 3811 http_response_size_bytes{handler="prometheus",quantile="0.99"} 3815 http_response_size_bytes_sum{handler="prometheus"} 64460 http_response_size_bytes_count{handler="prometheus"} 17 # HELP remote_clusters Number inbound connections from remote clusters and clusters stats # TYPE remote_clusters gauge remote_clusters{cluster="C02WG09CHTDH"} 0 # HELP rx Number of bytes received. # TYPE rx counter rx 83799 # HELP server_interactive_sessions_total Number of active sessions # TYPE server_interactive_sessions_total gauge server_interactive_sessions_total 5 # HELP tx Number of bytes transmitted. # TYPE tx counter tx 610546

The code for generating records:

// Returns an int >= min, < max func randomInt(min, max int) int { return min + rand.Intn(max-min) } // Generate a random string of A-Z chars with len = l func randomString(len int) string { bytes := make([]byte, len) for i := 0; i < len; i++ { bytes[i] = byte(randomInt(65, 90)) } return string(bytes) } func (b *FirestoreBackend) batchInsertTestData() { rand.Seed(time.Now().UnixNano()) batch := b.svc.Batch() outerCounter := 0 innerCounter := 0 for { if outerCounter >= 100000 { break } if innerCounter >= 490 { b.Logger.Infof("Writing outer counter: %v, inner counter: %v ...", outerCounter, innerCounter) _, _ = batch.Commit(b.clientContext) innerCounter = 0 } slices := make([]string, 0) slicesNo := randomInt(1, 8) sliceCount := 0 for { if sliceCount >= slicesNo { break } sliceCount = sliceCount + 1 slices = append(slices, randomString(randomInt(5, 12))) } key := strings.Join(slices, documentNameReplacementCharacter) var r record r.Key = key r.Value = string(randomString(randomInt(30, 200))) r.Timestamp = b.clock.Now().UTC().Unix() r.ID = b.clock.Now().UTC().UnixNano() batch.Create(b.svc.Collection(b.CollectionName).Doc(key), r) outerCounter = outerCounter + 1 innerCounter = innerCounter + 1 } }

klizhentas · 2019-07-16T21:26:13Z

lib/backend/firestore/firestorebk.go

+	b := &FirestoreBackend{
+		svc:              	firestoreClient,
+		Entry:           	l,
+		BKConfig:			*cfg,


use gofmt

and goimports :)

lib/backend/firestore/firestorebk.go

klizhentas · 2019-07-16T21:29:47Z

lib/backend/firestore/firestorebk.go

+
+	if !doc.Exists() {
+		return trace.NotFound("key %s does not exist", string(key))
+	} else {


because here you return, you don't need else clause

https://dave.cheney.net/2019/07/09/clear-is-better-than-clever <- check it out

klizhentas · 2019-07-16T21:33:55Z

lib/events/gcssessions/gcshandler.go

+	"github.com/gravitational/teleport"
+	"github.com/gravitational/teleport/lib/session"
+
+	"cloud.google.com/go/storage"


(
"std packages"

"teleport packages" "vendored stuff"

)

lib/events/gcssessions/gcshandler.go

klizhentas · 2019-07-16T21:41:38Z

lib/service/service.go

@@ -2415,4 +2441,4 @@ func initSelfSignedHTTPSCert(cfg *Config) (err error) {
 		return trace.Wrap(err, "error writing key PEM")
 	}
 	return nil
-}
+}


missing newlines at the end of file

…store-backed events, and firestore backend storage

Update e ref to bring in: * [v12] ci: Remove 'v' from OCI version labels (#2821) * [v12] chore: Bump Go to v1.20.12 (#2839) * [v12] ci: Build single-arch images for legacy/operator OCI (#2856)

* dronegen: enumerate linux tag build pipelines Unroll the loops over architectures, fips and package type and just list all the pipelines explicitly. This makes it a bit easier to understand the combinations and to replace them with GitHub actions workflows. Running `make dronegen` results in no changes to `.drone.yml`. * dronegen: Convert some linux tag pipelines to GitHub Actions Convert some of the linux-based tag build pipelines to run on GitHub Actions. The following pipelines have been converted: build-linux-amd64 build-linux-amd64-centos7 build-linux-amd64-centos7-fips build-linux-386 build-linux-arm build-linux-amd64-deb build-linux-amd64-deb-fips build-linux-amd64-centos7-rpm build-linux-amd64-centos7-fips-rpm build-linux-386-deb build-linux-386-rpm build-linux-arm-deb build-linux-arm-rpm The GHA workflows builds tarballs as well as deb/rpm packages in the one workflow, so the `-deb` and `-rpm` pipelines will need to be manually removed from `.drone.yml`. The amd64 centos7 and non-centos7 pipelines have been combined as they were calling the same `make` target duplicating work. The amd64 build is always done on centos7. As a result, we do not name the pipeline with -centos7 any more, but we do still specify it as the build.assets `Makefile` still has a centos7 target which is called, and we do still release an asset named with "centos7". Still remaining of the linux-based tag build pipelines are the arm64 pipelines which are already converted using a different workflow and the non-native windows build. * dronegen: Convert linux-arm64 pipeline to common pipeline Convert the build-linux-arm64 and push-build-linux-arm64 pipelines to use the common `release-linux` workflow instead of the arm64-specific `release-linux-arm64` workflow. This aligns it with the other linux build pipelines and allows us to get rid of the specific workflow. The pipelines for building the arm64 rpm and deb packages have not been generated by dronegen for some time now - since the arm64 build was converted to GitHub Actions. The OS packages were still built as dronegen does not remove pipelines, so the existing pipelines from before the GHA migration remained in `.drone.yml` and continued to run. These os packaging pipelines will be manually removed in a subsequent commit. * dronegen: Add pipeline for building legacy AMIs Generate a pipeline for calling a GitHub Actions workflow to generate the legacy AMIs. There were two existing manually added pipelines - `build-oss-amis` and `build-ent-amis` - that are replaced by this. The new pipeline needs to be manually added and the old ones manually removed. * dronegen: Replace oci build with GitHub Actions workflow Replace the `teleport-container-images-branch-tag` workflow that builds the legacy and operator OCI images with a call to the GitHub Actions workflow that does the same on GitHub Actions. This requires the manual addition of the `build-oci` pipeline and manual removal of the `teleport-container-images-branch-tag` pipeline, followed by running `make dronegen` to flesh out `build-oci` and sign .drone.yml. * dronegen: Remove now-unused functions and vars Remove the now-unused functions and vars after converting pipelines to calling GitHub Actions instead of running stuff on Drone. * drone: Update .drone.yml for GHA workflows Update .drone.yml by running `make dronegen` to update the following pipelines to call GitHub Actions to build instead of building on Drone: build-linux-amd64 build-linux-amd64-fips build-linux-386 build-linux-arm64 build-linux-arm Add two new pipelines for building AMIs and OCIs on GHA: build-legacy-amis build-oci Remove the following pipelines as the build of deb/rpm packages are done within the above pipelines on GitHub Actions now and the ami/oci pipelines have been replaced: build-linux-amd64-deb build-linux-amd64-fips-deb build-linux-amd64-centos7-rpm build-linux-amd64-centos7-fips-rpm build-linux-386-deb build-linux-386-rpm build-linux-arm64-deb build-linux-arm64-rpm build-linux-arm-deb build-linux-arm-rpm build-oss-amis build-ent-amis teleport-container-images-branch-tag Remove the following pipelines as AMD64 builds are always centos7 builds, but we were just doing it twice. No need for these any more, as the GHA workflow will build the release artifacts for these with the centos7 targets: build-linux-amd64-centos7 build-linux-amd64-centos7-fips The pipelines were added/removed using the following script, followed by `make dronegen`: AWK_SCRIPT=' /^---$/ { printf "%s", accumulator; accumulator = "" } /^---$/ || accumulator { accumulator = accumulator $0 "\n" } /^name: / { drop = $2 == to_remove if ($2 == before && to_add) { printf "---\nname: %s\n", to_add } if (!drop) { printf "%s", accumulator } accumulator = "" next } !drop && !accumulator { print } ENDFILE { printf "%s", accumulator }' toremove=( build-linux-amd64-{centos7,centos7-fips} build-linux-amd64-{deb,fips-deb,centos7-rpm,centos7-fips-rpm} build-linux-386-{deb,rpm} build-linux-arm64-{deb,rpm} build-linux-arm-{deb,rpm} build-{oss,ent}-amis teleport-container-images-branch-tag ) add_before=build-buildboxes toadd=( build-legacy-amis build-oci ) for pipeline in "${toremove[@]}"; do gawk -i inplace -v to_remove=$pipeline "$AWK_SCRIPT" .drone.yml done for pipeline in "${toadd[@]}"; do gawk -i inplace -v to_add=$pipeline -v before=$add_before "$AWK_SCRIPT" .drone.yml done * Update e ref for updated workflows and Go update Update e ref to bring in: * [v12] ci: Remove 'v' from OCI version labels (#2821) * [v12] chore: Bump Go to v1.20.12 (#2839) * [v12] ci: Build single-arch images for legacy/operator OCI (#2856)

joshdurbin force-pushed the gcp_ha_support branch 2 times, most recently from 7a5da14 to 5740611 Compare July 3, 2019 11:52

joshdurbin changed the title ~~GCP stateless HA support via Firestore and GCS~~ WIP: GCP stateless HA support via Firestore and GCS Jul 3, 2019

joshdurbin force-pushed the gcp_ha_support branch 4 times, most recently from cd2e73f to 654babe Compare July 10, 2019 20:43

joshdurbin force-pushed the gcp_ha_support branch from c3260d9 to a788443 Compare July 16, 2019 06:47

joshdurbin changed the title ~~WIP: GCP stateless HA support via Firestore and GCS~~ GCP stateless HA support via Firestore and GCS Jul 16, 2019

joshdurbin force-pushed the gcp_ha_support branch from a788443 to 1b59526 Compare July 16, 2019 06:55