[improve] [pip] PIP-355: Enhancing Broker-Level Metrics for Pulsar

pip/pip-355.md

@@ -0,0 +1,39 @@
+# PIP-355: Enhancing Broker-Level Metrics for Pulsar
+
+# Background Knowledge
+Pulsar provides broker-level, namespace-level, and topic-level metrics to monitor and analyze the behavior of the Pulsar service. These metrics are accessible through the Prometheus metrics endpoint. Detailed explanations of all metrics can be found on the Pulsar website: [Pulsar Metrics Reference](https://pulsar.apache.org/docs/3.2.x/reference-metrics/)
+
+# Motivation
+Within Pulsar's current metrics framework, the `pulsar_out_bytes_total` metric is utilized to expose the total bytes dispatched by the broker to consumers. However, there are notable limitations and challenges associated with this metric:
+- Inclusion of system subscriptions in the total bytes out, alongside user subscriptions, complicates accurate calculation of user-specific data.
+- The granularity of the metric (namespace-level vs. topic-subscription level) impacts the scalability and resource consumption when calculating cluster-level total out bytes.
+
+# Goals
+This proposal aims to address the following objectives:
+- Simplify the process of calculating cluster-level total out bytes.
+- Enable the calculation of total out bytes dispatched to system subscriptions.
+
+# High-Level Design
+To achieve the outlined goals, the proposal introduces three new broker-level metrics:
+- `pulsar_broker_out_bytes_total`: Represents the total out bytes dispatched by the broker to consumers, encompassing both user and system subscriptions.
+- `pulsar_broker_out_bytes_total_internal`: Captures the total out bytes dispatched to system consumers, configurable via `additionalSystemCursorNames` introduced in PIP-349.
+- `pulsar_broker_in_bytes_total`: Tracks the total in bytes sent by producers to the broker.
+
+# Detailed Design
+The implementation involves the introduction of the following broker-level metrics:
+- `pulsar_broker_out_bytes_total`: Aggregates the total out bytes from all topics, presented as a broker-level metric.
+- `pulsar_broker_out_bytes_total_internal`: Summation of system cursors across topics, including those configured through `additionalSystemCursorNames` and the default compaction service cursor.
+- `pulsar_broker_in_bytes_total`: Calculation of total in bytes across all topics.
+
+# Metrics
+The proposal includes the addition of three new broker-level metrics:
+- `pulsar_broker_out_bytes_total`
+- `pulsar_broker_out_bytes_total_internal`
+- `pulsar_broker_in_bytes_total`
+
+# Backward & Forward Compatibility
+The proposed changes ensure full compatibility with existing systems and pave the way for seamless integration with future enhancements.
+
+# Links
+- Mailing List discussion thread:
+- Mailing List voting thread: