Skip to content

Commit da373b5

Browse files
danieljbrucegcf-owl-bot[bot]kevkim-codes
authoredNov 7, 2024
feat: Bigtable authorized views requests on the Data plane (#1509)
* Create the new tabular api service file * feat: Bigtable authorized views - move the code over to the TabularApiSurface class (#1463) * Move the constructor over to TabularApiService * Move sampleRowKeys over * Move sampleRowKeys functions over and use promisify * Adjust the proxyquire to work with TabularAPIserv * Move all the ReadRows functionality over * Solve the issue with the is dependency * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add header for new class * Only include Table mocks that are necessary * Remove TODO * Rename TabularApiService to TabularApiSurface * Change all imports to tabular-api-surface * surface. not service --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * feat: Bigtable Authorized Views - Allow checkAndMutate and ReadModifyWriteRow calls on Authorized Views (#1464) * Move the constructor over to TabularApiService * Move sampleRowKeys over * Move sampleRowKeys functions over and use promisify * Adjust the proxyquire to work with TabularAPIserv * Move all the ReadRows functionality over * Solve the issue with the is dependency * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add header for new class * Trying out the FilterInformation class * Add a DataUtils module for the shared row function * Outsource functionality of filter to helper * debugging * Adjust proxyquire to include mocks for moved fn * Remove imports * Outsource code to a createRulesUtil function * Change proxyquire to include createRulesUtil * Move increment over to the utils folder * Move the functions into a static class for mocks * Remove mockCreateRules and shorten mock * Stub out FakeRowDataUtil * Remove unused dependencies * Add the functions to work with checkAndMutate and readWriteModifyRow * Fix regressions from the merge * Add documentation for the class * Document the new methods of table * Change the interface of the rowUtils * Pull the generateProperties code out avoid duplica * Move duplicate code out into a getProperties object * Remove console.log * Add method for creating views * Object for making grpc calls for authorized views * Remove import * Update the documentation for the Table * More specific type * Add documentation for each of the functions * Remove TODO * Add headers * Add documentation for new class * Remove imports * Reintroduce before * Remove unused import --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * feat: Create a view on the instance, not on the table (#1476) * Move view creation from the table class to instanc * feat: Create a view on the instance, not on the table * Add table name parameter to the documentation * Remove Table * feat: Create the unit tests for Authorized Views ensuring that requests are consistent (#1501) * Create a test for authorized views * Set the ReadRows options * Add authorized view test for createreadstream * Pass authorizedViewName along for createReadStream * Add the unit test for getRows * Add the unit test for getRows * Take out the test mock for the request function * Mutate rows request - test setup * Made corrections and src code changes so unit test passes * Fix the readrows test to end the stream Also add a test for insert and make the test easier to debug. * Finish the sampleRowKeys tests * Refactor part of the test for getting request opts * Authorized views for readRows test should be one assert * Add the mock function for request * Make mutate rows use the mockRequest function * Setup the sampleRowKeys setup fn to use mockReq * Add comments to the mocking functions * Fix comments. Move functions to right place * Setup the readModifyWriteRow tests * For createRules test use new mockCallbackRequest You need to call the callback in order to end the operation to createRules. * Use the new mockCallbackRequest function * Get rid of the unused mock function * Create a test file for createRules not got * increment * Add the test for increment * Change the response value for authorized view * Add the console logs and change return type * Flesh out response so that view call works * Exclude appropriate properties using promisify. * Removed console logs * Finished the filter test * Add the view call for filter * Remove only * Change to more readable values * Remove only * Add header to the test * For tests, expect view to be excluded from promis * run linter * Add a few @params and @returns * Add a comment about making requests for auth views * Added more comments - auth views vs table Also added a comment for calling increment * feat: Bigtable authorized views integration tests (#1504) * Create a test for authorized views * Set the ReadRows options * Add authorized view test for createreadstream * Pass authorizedViewName along for createReadStream * Add the unit test for getRows * Add the unit test for getRows * Take out the test mock for the request function * Mutate rows request - test setup * Made corrections and src code changes so unit test passes * Fix the readrows test to end the stream Also add a test for insert and make the test easier to debug. * Finish the sampleRowKeys tests * Refactor part of the test for getting request opts * Authorized views for readRows test should be one assert * Add the mock function for request * Make mutate rows use the mockRequest function * Setup the sampleRowKeys setup fn to use mockReq * Add comments to the mocking functions * Fix comments. Move functions to right place * Setup the readModifyWriteRow tests * For createRules test use new mockCallbackRequest You need to call the callback in order to end the operation to createRules. * Use the new mockCallbackRequest function * Get rid of the unused mock function * Create a test file for createRules not got * increment * Add the test for increment * Change the response value for authorized view * Add the console logs and change return type * Flesh out response so that view call works * Exclude appropriate properties using promisify. * Removed console logs * Finished the filter test * Add the view call for filter * Remove only * Change to more readable values * Remove only * Add header to the test * For tests, expect view to be excluded from promis * run linter * Create the before hook for auth views * Note to self on auth views - TODO * Add an insert statement * Make a test for getRows * Add the test for the getRows function. * Finish ‘should fail when writing to a row not in v * Add another mutate test, preserve table after test Add after hook to make sure table values stay the same. Add test for modifying from different column * Remove TODO * refactor the error message in the test * Add mutate, insert and sampleRowKeys tests * Fix sampleRowKeys test to require a fix upstream * Create samplerowkeys and createreadstream tests * Surround all tests in a try block This allows for better error reporting * Fixed the column filter so it works * Reduce verbosity in tests In many cases, 3 lines can be reduced to one variable. This is crucial for making the integration tests shorter. * Replace verbose data with data variable * Add an insert for multiple rows * Fix the check and mutate test to ignore corrupt d * Add the readModifyWriteRow tests * Add awaits * wrap columnFamily and columnIdInView in brackets * Eliminate the TODO * Remove only * Add key back to filter config option --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Kevin Kim <kevkim@google.com>
1 parent c4e2356 commit da373b5

13 files changed

+2614
-1006
lines changed
 

‎src/authorized-view.ts

+260
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
// Copyright 2024 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// https://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
import {TabularApiSurface} from './tabular-api-surface';
16+
import {CallOptions} from 'google-gax';
17+
import {
18+
CreateRulesCallback,
19+
CreateRulesResponse,
20+
FilterCallback,
21+
FilterConfig,
22+
FilterResponse,
23+
IncrementCallback,
24+
IncrementResponse,
25+
Rule,
26+
} from './row';
27+
import {RowDataUtils, RowProperties} from './row-data-utils';
28+
import {RawFilter} from './filter';
29+
import {Family} from './chunktransformer';
30+
import {Instance} from './instance';
31+
import {promisifyAll} from '@google-cloud/promisify';
32+
33+
interface FilterInformation {
34+
filter: RawFilter;
35+
rowId: string;
36+
}
37+
38+
interface CreateRulesInformation {
39+
rules: Rule | Rule[];
40+
rowId: string;
41+
}
42+
43+
interface IncrementInformation {
44+
column: string;
45+
rowId: string;
46+
}
47+
48+
/**
49+
* The AuthorizedView class is a class that is available to the user that
50+
* contains methods the user can call to work with authorized views.
51+
*
52+
* @class
53+
* @param {Table} table The table that the authorized view exists on.
54+
* @param {string} id Unique identifier of the authorized view.
55+
*
56+
*/
57+
export class AuthorizedView extends TabularApiSurface {
58+
private readonly rowData: {[id: string]: {[index: string]: Family}};
59+
60+
constructor(instance: Instance, tableName: string, viewName: string) {
61+
super(instance, tableName, viewName);
62+
this.rowData = {};
63+
}
64+
65+
createRules(
66+
createRulesInfo: CreateRulesInformation,
67+
options?: CallOptions
68+
): Promise<CreateRulesResponse>;
69+
createRules(
70+
createRulesInfo: CreateRulesInformation,
71+
options: CallOptions,
72+
callback: CreateRulesCallback
73+
): void;
74+
createRules(
75+
createRulesInfo: CreateRulesInformation,
76+
callback: CreateRulesCallback
77+
): void;
78+
/**
79+
* Update a row with rules specifying how the row's contents are to be
80+
* transformed into writes. Rules are applied in order, meaning that earlier
81+
* rules will affect the results of later ones.
82+
*
83+
* @throws {error} If no rules are provided.
84+
*
85+
* @param {CreateRulesInformation} createRulesInfo The rules to apply to a row
86+
* along with the row id of the row to update.
87+
* @param {object} [gaxOptions] Request configuration options, outlined here:
88+
* https://googleapis.github.io/gax-nodejs/CallSettings.html.
89+
* @param {function} callback The callback function.
90+
* @param {?error} callback.err An error returned while making this
91+
* request.
92+
* @param {object} callback.apiResponse The full API response.
93+
*
94+
* @example <caption>include:samples/api-reference-doc-snippets/row.js</caption>
95+
* region_tag:bigtable_api_create_rules
96+
*/
97+
createRules(
98+
createRulesInfo: CreateRulesInformation,
99+
optionsOrCallback?: CallOptions | CreateRulesCallback,
100+
cb?: CreateRulesCallback
101+
): void | Promise<CreateRulesResponse> {
102+
this.initializeRow(createRulesInfo.rowId);
103+
RowDataUtils.createRulesUtil(
104+
createRulesInfo.rules,
105+
this.generateProperties(createRulesInfo.rowId),
106+
optionsOrCallback,
107+
cb
108+
);
109+
}
110+
111+
/**
112+
* Mutates a row atomically based on the output of a filter. Depending on
113+
* whether or not any results are yielded, either the `onMatch` or `onNoMatch`
114+
* callback will be executed.
115+
*
116+
* @param {FilterInformation} filter Filter to be applied to the contents of
117+
* the row along with the row id of the affected row.
118+
* @param {object} config Configuration object.
119+
* @param {?object[]} config.onMatch A list of entries to be ran if a match is
120+
* found.
121+
* @param {object[]} [config.onNoMatch] A list of entries to be ran if no
122+
* matches are found.
123+
* @param {object} [config.gaxOptions] Request configuration options, outlined
124+
* here: https://googleapis.github.io/gax-nodejs/global.html#CallOptions.
125+
* @param {function} callback The callback function.
126+
* @param {?error} callback.err An error returned while making this
127+
* request.
128+
* @param {boolean} callback.matched Whether a match was found or not.
129+
*
130+
* @example <caption>include:samples/api-reference-doc-snippets/row.js</caption>
131+
* region_tag:bigtable_api_row_filter
132+
*/
133+
filter(
134+
filterInfo: FilterInformation,
135+
config?: FilterConfig
136+
): Promise<FilterResponse>;
137+
filter(
138+
filterInfo: FilterInformation,
139+
config: FilterConfig,
140+
callback: FilterCallback
141+
): void;
142+
filter(filterInfo: FilterInformation, callback: FilterCallback): void;
143+
filter(
144+
filterInfo: FilterInformation,
145+
configOrCallback?: FilterConfig | FilterCallback,
146+
cb?: FilterCallback
147+
): void | Promise<FilterResponse> {
148+
this.initializeRow(filterInfo.rowId);
149+
RowDataUtils.filterUtil(
150+
filterInfo.filter,
151+
this.generateProperties(filterInfo.rowId),
152+
configOrCallback,
153+
cb
154+
);
155+
}
156+
157+
/**
158+
* Generates request properties necessary for making an rpc call for an
159+
* authorized view.
160+
*
161+
* @param {string} id The row id to generate the properties for.
162+
* @private
163+
*/
164+
private generateProperties(id: string): RowProperties {
165+
return {
166+
requestData: {
167+
data: this.rowData[id],
168+
id,
169+
table: this,
170+
bigtable: this.bigtable,
171+
},
172+
reqOpts: {
173+
authorizedViewName: this.name + '/authorizedViews/' + this.viewName,
174+
},
175+
};
176+
}
177+
178+
increment(
179+
columnInfo: IncrementInformation,
180+
value?: number
181+
): Promise<IncrementResponse>;
182+
increment(
183+
columnInfo: IncrementInformation,
184+
value: number,
185+
options?: CallOptions
186+
): Promise<IncrementResponse>;
187+
increment(
188+
columnInfo: IncrementInformation,
189+
options?: CallOptions
190+
): Promise<IncrementResponse>;
191+
increment(
192+
columnInfo: IncrementInformation,
193+
value: number,
194+
options: CallOptions,
195+
callback: IncrementCallback
196+
): void;
197+
increment(
198+
columnInfo: IncrementInformation,
199+
value: number,
200+
callback: IncrementCallback
201+
): void;
202+
increment(
203+
columnInfo: IncrementInformation,
204+
options: CallOptions,
205+
callback: IncrementCallback
206+
): void;
207+
increment(
208+
columnInfo: IncrementInformation,
209+
callback: IncrementCallback
210+
): void;
211+
/**
212+
* Increment a specific column within the row. If the column does not
213+
* exist, it is automatically initialized to 0 before being incremented.
214+
*
215+
* @param {IncrementInformation} columnInfo The column we are incrementing a
216+
* value in along with the row id of the affected row.
217+
* @param {number} [value] The amount to increment by, defaults to 1.
218+
* @param {object} [gaxOptions] Request configuration options, outlined here:
219+
* https://googleapis.github.io/gax-nodejs/CallSettings.html.
220+
* @param {function} callback The callback function.
221+
* @param {?error} callback.err An error returned while making this
222+
* request.
223+
* @param {number} callback.value The updated value of the column.
224+
* @param {object} callback.apiResponse The full API response.
225+
*
226+
* @example <caption>include:samples/api-reference-doc-snippets/row.js</caption>
227+
* region_tag:bigtable_api_row_increment
228+
*/
229+
increment(
230+
columnInfo: IncrementInformation,
231+
valueOrOptionsOrCallback?: number | CallOptions | IncrementCallback,
232+
optionsOrCallback?: CallOptions | IncrementCallback,
233+
cb?: IncrementCallback
234+
): void | Promise<IncrementResponse> {
235+
this.initializeRow(columnInfo.rowId);
236+
RowDataUtils.incrementUtils(
237+
columnInfo.column,
238+
this.generateProperties(columnInfo.rowId),
239+
valueOrOptionsOrCallback,
240+
optionsOrCallback,
241+
cb
242+
);
243+
}
244+
245+
/**
246+
* Sets the row data for a particular row to an empty object
247+
*
248+
* @param {string} id An string with the key of the row to initialize.
249+
* @private
250+
*/
251+
private initializeRow(id: string) {
252+
if (!this.rowData[id]) {
253+
this.rowData[id] = {};
254+
}
255+
}
256+
}
257+
258+
promisifyAll(AuthorizedView, {
259+
exclude: ['initializeRow', 'generateProperties'],
260+
});

‎src/chunktransformer.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ export interface Data {
3434
chunks: Chunk[];
3535
lastScannedRowKey?: Buffer;
3636
}
37-
interface Family {
37+
export interface Family {
3838
[qualifier: string]: Qualifier[];
3939
}
4040
export interface Qualifier {

‎src/instance.ts

+12
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ import {Bigtable} from '.';
6868
import {google} from '../protos/protos';
6969
import {Backup, RestoreTableCallback, RestoreTableResponse} from './backup';
7070
import {ClusterUtils} from './utils/cluster';
71+
import {AuthorizedView} from './authorized-view';
7172

7273
export interface ClusterInfo extends BasicClusterConfig {
7374
id: string;
@@ -1475,6 +1476,16 @@ Please use the format 'my-instance' or '${bigtable.projectName}/instances/my-ins
14751476
}
14761477
);
14771478
}
1479+
1480+
/**
1481+
* Gets an Authorized View object for making authorized view grpc calls.
1482+
*
1483+
* @param {string} tableName The name for the Table
1484+
* @param {string} viewName The name for the Authorized view
1485+
*/
1486+
view(tableName: string, viewName: string): AuthorizedView {
1487+
return new AuthorizedView(this, tableName, viewName);
1488+
}
14781489
}
14791490

14801491
/*! Developer Documentation
@@ -1490,6 +1501,7 @@ promisifyAll(Instance, {
14901501
'getBackupsStream',
14911502
'getTablesStream',
14921503
'getAppProfilesStream',
1504+
'view',
14931505
],
14941506
});
14951507

0 commit comments

Comments
 (0)