Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

359913994 third pr csm #1571

Draft
wants to merge 340 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
340 commits
Select commit Hold shift + click to select a range
ef8e3fe
Don’t require retries to be passed into metrics
danieljbruce Jan 24, 2025
c68a76f
Remove testMeterProvider
danieljbruce Jan 24, 2025
47a24b1
Remove the attributesToString function
danieljbruce Jan 24, 2025
b2600f2
Eliminate unused class
danieljbruce Jan 24, 2025
d4d3f6c
Generate documentation for the IMetricsHandler
danieljbruce Jan 24, 2025
b8dff1c
Generate documentation for GCPMetricsHandler
danieljbruce Jan 24, 2025
d50384f
Restrict attributes interfaces and solve compile
danieljbruce Jan 24, 2025
b5fc1f2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 24, 2025
1e5dc82
use undefined instead of null
danieljbruce Jan 24, 2025
c2ffbc6
Introduce enums for allowable values
danieljbruce Jan 24, 2025
1200b3c
Merge branch '359913994-first-PR-add-infrastructure' of https://githu…
danieljbruce Jan 24, 2025
9320149
Add more headers
danieljbruce Jan 24, 2025
4023361
Remove only
danieljbruce Jan 24, 2025
ef91733
Use null to pass values around. Not undefined
danieljbruce Jan 24, 2025
52b570c
Modify test step
danieljbruce Jan 24, 2025
6a6774f
Add metrics
danieljbruce Jan 24, 2025
10b6d30
Don’t provide first response latency
danieljbruce Jan 24, 2025
33c17c6
Remove firstResponseLatency from operation metrics
danieljbruce Jan 24, 2025
fbf2314
Expose interface allowing undefined not null
danieljbruce Jan 24, 2025
39fe861
Better explanations for design decision inline
danieljbruce Jan 24, 2025
8f13100
Use attempt start time not operation start time
danieljbruce Jan 27, 2025
48e0e95
Adjust tests for first response latency
danieljbruce Jan 27, 2025
66c4ab1
Remove TODO
danieljbruce Jan 27, 2025
e7c5b5f
Use the MethodName enum instead of string
danieljbruce Jan 28, 2025
98be351
Don’t use enum for streaming operation
danieljbruce Jan 28, 2025
efdfcea
Remove copy/pasted comment
danieljbruce Jan 28, 2025
4a6a476
Rename to OperationMetricsCollector
danieljbruce Jan 28, 2025
edfcf8a
Rename the method to getOperationAttributes
danieljbruce Jan 28, 2025
bc4998f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 28, 2025
7e6374d
Merge branch '359913994-first-PR-add-infrastructure' of https://githu…
gcf-owl-bot[bot] Jan 28, 2025
10b72ec
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 28, 2025
a72b51a
Merge branch '359913994-first-PR-add-infrastructure' of https://githu…
gcf-owl-bot[bot] Jan 28, 2025
47fd9d0
Add aggregate views to the GCP metrics handler
danieljbruce Jan 28, 2025
2a42162
Merge branch '359913994-first-PR-add-infrastructure' of https://githu…
danieljbruce Jan 28, 2025
9073f07
Adjust test based on enum changes
danieljbruce Jan 28, 2025
32d3983
Update the documentation to be more descriptive
danieljbruce Jan 28, 2025
9716c4a
Add the state machine to the metrics collector
danieljbruce Jan 29, 2025
d2b93ee
Use grpc code to report attempt/operation status
danieljbruce Jan 29, 2025
99f9577
Remove parameters from JS Documentation
danieljbruce Jan 29, 2025
c82e72d
Update interfaces and some metrics
danieljbruce Jan 30, 2025
759e829
Documentation for all the different interfaces
danieljbruce Jan 30, 2025
76b6f5a
use operation start time as the benchmark
danieljbruce Jan 30, 2025
1e840a4
Final operation status shouldn’t be included per a
danieljbruce Jan 30, 2025
7bf62e9
Move OnAttemptCompleteInfo
danieljbruce Jan 31, 2025
fca55b7
Provide AttemptOnlyAttributes in the only file
danieljbruce Jan 31, 2025
51afdce
Move over the OperationOnlyAttributes
danieljbruce Jan 31, 2025
57b1dc1
Adjust the guard so that it is earlier
danieljbruce Jan 31, 2025
0f850b7
Adjust the test output file
danieljbruce Jan 31, 2025
6c1e01b
Change streaming back to STREAMING/UNARY
danieljbruce Jan 31, 2025
2910408
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 31, 2025
2781561
Change metrics handler interface to support each metric
danieljbruce Jan 31, 2025
0b4d93e
Revert "Change metrics handler interface to support each metric"
danieljbruce Jan 31, 2025
1b6681b
Supply the projectId later in the client side
danieljbruce Feb 5, 2025
b6f1302
Remove the GCPMetricsHandler file
danieljbruce Feb 5, 2025
1ae82ff
Change location of the client-side-metrics-attribu
danieljbruce Feb 6, 2025
3ee5604
Change common test utilities folder name
danieljbruce Feb 6, 2025
124ed30
Remove aliases for grpc status
danieljbruce Feb 6, 2025
ef36a6f
Should be MethodName type
danieljbruce Feb 6, 2025
6829224
Rename variable as it expands beyond latency
danieljbruce Feb 6, 2025
dd603f1
Remove private methods for building attributes
danieljbruce Feb 6, 2025
b493c0d
Replace the logger class with a simple object
danieljbruce Feb 6, 2025
2f19f31
Remove only
danieljbruce Feb 6, 2025
dfe7d57
Remove the logger classes
danieljbruce Feb 6, 2025
02d752a
Add stubs
danieljbruce Feb 6, 2025
19d1d81
Revert "Remove the GCPMetricsHandler file"
danieljbruce Feb 6, 2025
48ff706
Add exporter and helper function
danieljbruce Feb 7, 2025
883ea1a
Add headers and fixture
danieljbruce Feb 7, 2025
ad9c85b
Add milliseconds unit
danieljbruce Feb 7, 2025
5a3bac2
Record to a histogram
danieljbruce Feb 7, 2025
4740c62
Add necessary elements to the GCPMetricsHandler
danieljbruce Feb 7, 2025
ee404f1
Pass an exporter into the GCPMetricsHandler
danieljbruce Feb 7, 2025
c997f0f
Move file to tests
danieljbruce Feb 7, 2025
3719257
Remove unused import
danieljbruce Feb 7, 2025
a6498f2
Adapt the fixture to include the projectId
danieljbruce Feb 7, 2025
e7d631d
More exporter code fixes
danieljbruce Feb 7, 2025
382ebef
Add the fixture. We are going to use it later
danieljbruce Feb 7, 2025
8ba3347
Solved compiler errors. Test almost passes
danieljbruce Feb 7, 2025
13382d2
Address the failing export test
danieljbruce Feb 7, 2025
948a3a3
Fixed the exporter. It now writes data properly.
danieljbruce Feb 10, 2025
c8bb0a8
Test should complete when the export doesnt error
danieljbruce Feb 10, 2025
78ec2e8
Add the fixture to the shared folder
danieljbruce Feb 10, 2025
e5ec89f
Remove two files not in this PR
danieljbruce Feb 10, 2025
5403a1b
delete empty file
danieljbruce Feb 10, 2025
176ed02
Remove files that are already in the other PR
danieljbruce Feb 10, 2025
08c1c1b
Remove the metrics handler fixture
danieljbruce Feb 10, 2025
be7673f
Use 3 metrics instead of 1
danieljbruce Feb 11, 2025
e6d6606
Replace with proper buckets
danieljbruce Feb 11, 2025
285454f
Change the metrics handler fixture
danieljbruce Feb 12, 2025
12a5cc7
Stop using stub in exporter
danieljbruce Feb 13, 2025
54239f3
Use more realistic buckets
danieljbruce Feb 13, 2025
86890aa
Remove the go request to export
danieljbruce Feb 13, 2025
a0fa7e4
Modify the fixtures to be more realistic
danieljbruce Feb 13, 2025
f5267c1
Change the mean
danieljbruce Feb 13, 2025
887d988
Remove only
danieljbruce Feb 13, 2025
075bf9b
Add the export input fixture
danieljbruce Feb 13, 2025
94f422c
Reducing export interval makes time complete
danieljbruce Feb 13, 2025
1c3c290
Add the GCPMetricsHandler test
danieljbruce Feb 13, 2025
b118bc1
Add a stub for the otel export
danieljbruce Feb 14, 2025
8aefe11
Test calling export is now working
danieljbruce Feb 14, 2025
a30d3ec
Remove old unused code
danieljbruce Feb 14, 2025
c3f2963
Change tests and json structure to work
danieljbruce Feb 14, 2025
67b6478
Update all the fixtures
danieljbruce Feb 14, 2025
8544410
Fix the view creation code
danieljbruce Feb 19, 2025
96dbc1c
Starting test for gcp-metrics-handler
danieljbruce Feb 19, 2025
416e18c
Put tests in the proper places
danieljbruce Feb 19, 2025
991f5c8
Replace start and end time with more recent values
danieljbruce Feb 19, 2025
09389ee
Utilize the new metrics handler interface
danieljbruce Feb 20, 2025
87d5592
Solve compiler errors resulting from metrics handl
danieljbruce Feb 20, 2025
9ad2ef8
Update the fixture
danieljbruce Feb 20, 2025
c59fcab
rewrite the metric to request method
danieljbruce Feb 20, 2025
5848588
Add interfaces to work with type guards
danieljbruce Feb 21, 2025
9c35dfb
Correct the compile error problems
danieljbruce Feb 21, 2025
07ec90d
Fix expected OTEL output test
danieljbruce Feb 21, 2025
bf54c8c
Remove TODOs
danieljbruce Feb 21, 2025
f226b5f
Fix test to compare pointwise
danieljbruce Feb 21, 2025
b42b4f4
connectivity error count corrections
danieljbruce Feb 21, 2025
54ac764
Correct the fixtures
danieljbruce Feb 21, 2025
eb8f14b
Eliminate tests using the old export input fixture
danieljbruce Feb 21, 2025
6ecb1a6
Add headers
danieljbruce Feb 21, 2025
fa0a56e
run linter
danieljbruce Feb 21, 2025
fcef83d
Modify the gcp-metrics-handler and test the proces
danieljbruce Feb 21, 2025
cd2efac
Remove only
danieljbruce Feb 21, 2025
eba027c
Use a fake projectId
danieljbruce Feb 21, 2025
5929a9d
Only call export once
danieljbruce Feb 21, 2025
3b48c8e
Ensure test suite completes
danieljbruce Feb 21, 2025
8edc4ab
Remove shutdown
danieljbruce Feb 21, 2025
8c9d23f
remove async
danieljbruce Feb 21, 2025
7b49f01
Don’t export the data twice
danieljbruce Feb 21, 2025
2ec12b6
Merge branch 'main' of https://github.com/googleapis/nodejs-bigtable …
danieljbruce Feb 24, 2025
b4f7705
Increase the timeout
danieljbruce Feb 24, 2025
2a32459
Use the PushMetricExporter interface
danieljbruce Feb 24, 2025
e5caa9e
Removed some interfaces that are not used anymore
danieljbruce Feb 24, 2025
fc114ff
Update JSdoc
danieljbruce Feb 24, 2025
6fb5944
Move fake start time and fake end time
danieljbruce Feb 24, 2025
ca6f05e
Remove the TODO
danieljbruce Feb 24, 2025
4bec216
Update documentation
danieljbruce Feb 24, 2025
bd4b0ac
Add additional information to the error reported
danieljbruce Feb 25, 2025
c191614
Move start time and end time
danieljbruce Feb 25, 2025
86be1ea
Try to use timestamps in order
danieljbruce Feb 25, 2025
3b0f081
Reduce timestamp delay
danieljbruce Feb 25, 2025
3ebb9ff
Remove only
danieljbruce Feb 25, 2025
cf32131
Inline addFakeRecentTimestamps
danieljbruce Feb 25, 2025
78a20d4
Move replace timestamps into the only file it
danieljbruce Feb 25, 2025
105b58b
Fix comment
danieljbruce Feb 25, 2025
d4022fd
Rename the metric types
danieljbruce Feb 26, 2025
7ea28d2
Generate documentation for the new classes
danieljbruce Feb 26, 2025
7848643
Add documentation for monitored resource
danieljbruce Feb 26, 2025
7229174
Generate documentation for the other metrics
danieljbruce Feb 26, 2025
7f4e167
Generate documentation for the constructor
danieljbruce Feb 26, 2025
c86196a
Get documentation for replaceTimestamps, fixtures
danieljbruce Feb 26, 2025
41c9b2d
Merge branch 'main' of https://github.com/googleapis/nodejs-bigtable …
danieljbruce Feb 26, 2025
d76fa14
Reduce the interval time
danieljbruce Feb 27, 2025
2f3b4e5
Make view list based on latencies
danieljbruce Feb 27, 2025
7a4b33e
Add a guard for count
danieljbruce Feb 28, 2025
eaf8879
Add a CSM test for Bigtable
danieljbruce Mar 3, 2025
22d25fa
Add handlers to the middleware layer
danieljbruce Mar 3, 2025
a3068bf
Add the metrics to the tabular api surface
danieljbruce Mar 3, 2025
61187ea
Move the tests
danieljbruce Mar 3, 2025
a1276b4
Use the GCPMetricsHandler in the tabular api surface
danieljbruce Mar 3, 2025
d16d3ec
Add a guard in the metrics collector
danieljbruce Mar 5, 2025
874af79
Fix the proxyquire issue
danieljbruce Mar 5, 2025
c000a1c
Export every 60 seconds instead
danieljbruce Mar 5, 2025
9f28d3e
Add a guard to the metadata collector
danieljbruce Mar 5, 2025
8cc0757
Add a guard to the export function
danieljbruce Mar 5, 2025
88e0527
Make sure to await to avoid a race condition
danieljbruce Mar 5, 2025
d6d409e
Adjust headers
danieljbruce Mar 5, 2025
6b99e09
Add the right assertions for the TestMetricsHandle
danieljbruce Mar 6, 2025
f4cb545
Send the project id to the mock
danieljbruce Mar 6, 2025
5a09c33
Invert the for and if
danieljbruce Mar 6, 2025
aa18c1e
Pull all attributes out
danieljbruce Mar 6, 2025
791e70d
Eliminate the need for the type guard
danieljbruce Mar 6, 2025
2e21503
Eliminate the data points interface
danieljbruce Mar 6, 2025
e936aaa
only
danieljbruce Mar 6, 2025
ab07a58
Revert "Eliminate the need for the type guard"
danieljbruce Mar 6, 2025
7fe9a46
Pull timeseries into one variable
danieljbruce Mar 6, 2025
26b9ca7
Eliminate an unused import
danieljbruce Mar 6, 2025
29ef6b0
Add a comment that explains the usefulness
danieljbruce Mar 6, 2025
8fd59f4
Remove the gce instance setting
danieljbruce Mar 6, 2025
183bdb7
Eliminate the gcp cloud provider setting
danieljbruce Mar 6, 2025
92059aa
run all tests
danieljbruce Mar 6, 2025
af3aa73
Eliminate duplicate project id from monitored
danieljbruce Mar 6, 2025
422060e
Change the two metric names
danieljbruce Mar 6, 2025
f93a721
Extend the timeout so that the exporter has
danieljbruce Mar 7, 2025
ea230aa
Use spread syntax
danieljbruce Mar 7, 2025
5ecfb70
Changed metric to otel instruments
danieljbruce Mar 7, 2025
303c57c
Remove optional on Otel instruments
danieljbruce Mar 7, 2025
a822961
Rename initialize to getMetrics
danieljbruce Mar 7, 2025
31fe184
Pin promisify to version 4
danieljbruce Mar 7, 2025
30152d6
cast to string - compiler errors
danieljbruce Mar 7, 2025
ccaaa07
Cast to string when building buffer
danieljbruce Mar 13, 2025
236c42a
Remove a TODO - it is done
danieljbruce Mar 13, 2025
6c28f25
Delete the test that writes to metrics collected
danieljbruce Mar 13, 2025
59f9c51
eliminate unnecessary spacing
danieljbruce Mar 13, 2025
ef322d5
Remove console logs from the code
danieljbruce Mar 13, 2025
b17680c
Eliminate the TODO in index
danieljbruce Mar 13, 2025
c587722
Add comment to mention mocha
danieljbruce Mar 13, 2025
f00993c
Add another comment for code 0
danieljbruce Mar 13, 2025
8f61f4b
Add a comment clarifying what the test does
danieljbruce Mar 13, 2025
4fb25eb
Pin promisify to solve the linting issues
danieljbruce Mar 13, 2025
5fafe96
Rename test to ClientSideMetricsToMetricsHandler
danieljbruce Mar 13, 2025
68c2a1a
Rename another test suite
danieljbruce Mar 13, 2025
1a2ace7
Remove these console logs
danieljbruce Mar 13, 2025
886f999
Add guards to the after hooks on this test
danieljbruce Mar 13, 2025
bb3b177
Remove only
danieljbruce Mar 13, 2025
c3d918c
Increase the GCPMetricsHandler time
danieljbruce Mar 13, 2025
99c6271
Fix samples test issue
danieljbruce Mar 13, 2025
3843e7b
Add a comment to test that works with the mock ser
danieljbruce Mar 13, 2025
05787e5
Revert "Fix samples test issue"
danieljbruce Mar 13, 2025
78550a8
Increase the timeout
danieljbruce Mar 13, 2025
ee022ce
Catch failures to load ResponseParams
danieljbruce Mar 13, 2025
9584c69
Change the latency buckets
danieljbruce Mar 13, 2025
082c049
Updated the resultCallback comment
danieljbruce Mar 13, 2025
f2e46a4
Change the test description
danieljbruce Mar 14, 2025
dfcf56e
Add a comment explaining what the test is about
danieljbruce Mar 14, 2025
35da5c6
Completed the test for two metrics handlers
danieljbruce Mar 14, 2025
b5ae964
remove only
danieljbruce Mar 14, 2025
b575010
Cast as histogram to make compile error go away
danieljbruce Mar 14, 2025
822c14e
Move the duplicate copies of replaceTimestamps
danieljbruce Mar 14, 2025
561889c
Take steps to eliminate the ExportInput interface
danieljbruce Mar 14, 2025
f423c57
Add a header
danieljbruce Mar 14, 2025
9076be1
Remove only
danieljbruce Mar 14, 2025
ead3f5e
Eliminate ExportInput and dependencies
danieljbruce Mar 14, 2025
fa6c3fd
Eliminate constant
danieljbruce Mar 14, 2025
a587d0c
Push conversion to status back to metrics collect
danieljbruce Mar 14, 2025
311d555
Change property names to match MetricServiceClient
danieljbruce Mar 14, 2025
117473b
Solve the app_profile_id issue
danieljbruce Mar 14, 2025
e4154af
It is actually app_profile not app_profile_id
danieljbruce Mar 14, 2025
0ccec5b
Add guards that stop the export call
danieljbruce Mar 17, 2025
9f94b9e
Remove only
danieljbruce Mar 17, 2025
7e76c39
Add a new test for ten metrics handlers
danieljbruce Mar 17, 2025
bb64fbe
Remove only
danieljbruce Mar 17, 2025
27f5bcd
Do not pass data through the Resource object
danieljbruce Mar 17, 2025
d1a292f
Add a test for writing duplicate points to MH
danieljbruce Mar 17, 2025
23a4d39
Eliminate interface
danieljbruce Mar 17, 2025
55dbd8f
Set connectivity error count to 1 (not increment)
danieljbruce Mar 17, 2025
69abb9a
Merge branch '359913994-exporter-PR' of https://github.com/googleapis…
danieljbruce Mar 17, 2025
71b2d48
Edit the fixtures
danieljbruce Mar 17, 2025
c581129
Remove the guard
danieljbruce Mar 20, 2025
ba04208
Change attempt status to string
danieljbruce Mar 20, 2025
9c3e5b1
Add a comment - undefined in conformance tests
danieljbruce Mar 20, 2025
ab4545e
Make metrics handlers per client
danieljbruce Mar 21, 2025
2d23a83
Add coment about latency
danieljbruce Mar 21, 2025
efa147d
Wrap in try
danieljbruce Mar 21, 2025
5761892
Adjust the proxyquire mocks to mock handler in
danieljbruce Mar 21, 2025
333c5b9
Eliminate TODO
danieljbruce Mar 21, 2025
3b71eff
Eliminate TODO
danieljbruce Mar 21, 2025
5f67eef
Merge branch 'main' of https://github.com/googleapis/nodejs-bigtable …
danieljbruce Mar 24, 2025
4ab9084
Add some code so that first response latencies gets recorded
danieljbruce Mar 27, 2025
7daeb0f
Run the linter
danieljbruce Mar 27, 2025
25ab7c4
Reduce latency when collect metrics is false
danieljbruce Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Generate documentation for GCPMetricsHandler
danieljbruce committed Jan 24, 2025
commit b8dff1c85088fdc4cba6d8038fa4e831da058954
25 changes: 25 additions & 0 deletions src/client-side-metrics/gcp-metrics-handler.ts
Original file line number Diff line number Diff line change
@@ -29,10 +29,21 @@ interface Metrics {
clientBlockingLatencies: typeof Histogram;
}

/**
* A metrics handler implementation that uses OpenTelemetry to export metrics to Google Cloud Monitoring.
* This handler records metrics such as operation latency, attempt latency, retry count, and more,
* associating them with relevant attributes for detailed analysis in Cloud Monitoring.
*/
export class GCPMetricsHandler implements IMetricsHandler {
private initialized = false;
private otelMetrics?: Metrics;

/**
* Initializes the OpenTelemetry metrics instruments if they haven't been already.
* Creates and registers metric instruments (histograms and counters) for various Bigtable client metrics.
* Sets up a MeterProvider and configures a PeriodicExportingMetricReader for exporting metrics to Cloud Monitoring.
* @param {string} [projectId] The Google Cloud project ID. Used for metric export. If not provided, it will attempt to detect it from the environment.
*/
private initialize(projectId?: string) {
if (!this.initialized) {
this.initialized = true;
@@ -114,6 +125,12 @@ export class GCPMetricsHandler implements IMetricsHandler {
}
}

/**
* Records metrics for a completed Bigtable operation.
* This method records the operation latency and retry count, associating them with provided attributes.
* @param {onOperationCompleteMetrics} metrics Metrics related to the completed operation.
* @param {Attributes} attributes Attributes associated with the completed operation.
*/
onOperationComplete(
metrics: onOperationCompleteMetrics,
attributes: Attributes
@@ -125,6 +142,14 @@ export class GCPMetricsHandler implements IMetricsHandler {
);
this.otelMetrics?.retryCount.add(metrics.retryCount, attributes);
}

/**
* Records metrics for a completed attempt of a Bigtable operation.
* This method records attempt latency, connectivity error count, server latency, and first response latency,
* along with the provided attributes.
* @param {onAttemptCompleteMetrics} metrics Metrics related to the completed attempt.
* @param {Attributes} attributes Attributes associated with the completed attempt.
*/
onAttemptComplete(metrics: onAttemptCompleteMetrics, attributes: Attributes) {
this.initialize();
this.otelMetrics?.attemptLatencies.record(
Loading