Skip to content

feat: add explain support for cursor commands #2622

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions src/cmap/wire_protocol/query.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import { command, CommandOptions } from './command';
import { Query } from '../commands';
import { MongoError } from '../../error';
import { maxWireVersion, collectionNamespace, Callback } from '../../utils';
import { maxWireVersion, collectionNamespace, Callback, decorateWithExplain } from '../../utils';
import { getReadPreference, isSharded, applyCommonQueryOptions } from './shared';
import { Document, pluckBSONSerializeOptions } from '../../bson';
import type { Server } from '../../sdam/server';
import type { ReadPreferenceLike } from '../../read_preference';
import type { FindOptions } from '../../operations/find';
import { Explain } from '../../explain';

/** @internal */
export interface QueryOptions extends CommandOptions {
Expand Down Expand Up @@ -43,7 +44,14 @@ export function query(
}

const readPreference = getReadPreference(cmd, options);
const findCmd = prepareFindCommand(server, ns, cmd);
let findCmd = prepareFindCommand(server, ns, cmd);

// If we have explain, we need to rewrite the find command
// to wrap it in the explain command
const explain = Explain.fromOptions(options);
if (explain) {
findCmd = decorateWithExplain(findCmd, explain);
}

// NOTE: This actually modifies the passed in cmd, and our code _depends_ on this
// side-effect. Change this ASAP
Expand All @@ -62,7 +70,7 @@ export function query(
}

function prepareFindCommand(server: Server, ns: string, cmd: Document) {
let findCmd: Document = {
const findCmd: Document = {
find: collectionNamespace(ns)
};

Expand Down Expand Up @@ -146,14 +154,6 @@ function prepareFindCommand(server: Server, ns: string, cmd: Document) {
if (cmd.collation) findCmd.collation = cmd.collation;
if (cmd.readConcern) findCmd.readConcern = cmd.readConcern;

// If we have explain, we need to rewrite the find command
// to wrap it in the explain command
if (cmd.explain) {
findCmd = {
explain: findCmd
};
}

return findCmd;
}

Expand Down Expand Up @@ -195,7 +195,7 @@ function prepareLegacyFindQuery(
if (typeof cmd.showDiskLoc !== 'undefined') findCmd['$showDiskLoc'] = cmd.showDiskLoc;
if (cmd.comment) findCmd['$comment'] = cmd.comment;
if (cmd.maxTimeMS) findCmd['$maxTimeMS'] = cmd.maxTimeMS;
if (cmd.explain) {
if (options.explain !== undefined) {
// nToReturn must be 0 (match all) or negative (match N and close cursor)
// nToReturn > 0 will give explain results equivalent to limit(0)
numberToReturn = -Math.abs(cmd.limit || 0);
Expand Down
37 changes: 17 additions & 20 deletions src/cursor/cursor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,13 @@ import { PromiseProvider } from '../promise_provider';
import type { OperationTime, ResumeToken } from '../change_stream';
import type { CloseOptions } from '../cmap/connection_pool';
import type { CollationOptions } from '../cmap/wire_protocol/write_command';
import type { Hint, OperationBase } from '../operations/operation';
import { Aspect, Hint, OperationBase } from '../operations/operation';
import type { Topology } from '../sdam/topology';
import type { CommandOperationOptions } from '../operations/command';
import { CommandOperation, CommandOperationOptions } from '../operations/command';
import type { ReadConcern } from '../read_concern';
import type { Server } from '../sdam/server';
import type { ClientSession } from '../sessions';
import { Explain, ExplainVerbosityLike } from '../explain';

const kCursor = Symbol('cursor');

Expand Down Expand Up @@ -1300,26 +1301,22 @@ export class Cursor<
/**
* Execute the explain for the cursor
*
* @param verbosity - The mode in which to run the explain.
* @param callback - The result callback.
*/
explain(): Promise<unknown>;
explain(callback: Callback): void;
explain(callback?: Callback): Promise<unknown> | void {
// NOTE: the next line includes a special case for operations which do not
// subclass `CommandOperationV2`. To be removed asap.
// TODO NODE-2853: This had to be removed during NODE-2852; fix while re-implementing
// cursor explain
// if (this.operation && this.operation.cmd == null) {
// this.operation.options.explain = true;
// return executeOperation(this.topology, this.operation as any, callback);
// }

this.cmd.explain = true;

// Do we have a readConcern
if (this.cmd.readConcern) {
delete this.cmd['readConcern'];
}
explain(verbosity?: ExplainVerbosityLike): Promise<unknown>;
explain(verbosity?: ExplainVerbosityLike, callback?: Callback): Promise<unknown> | void {
if (typeof verbosity === 'function') (callback = verbosity), (verbosity = true);
if (verbosity === undefined) verbosity = true;

// TODO: For now, we need to manually do these checks. This will change after cursor refactor.
if (
!(this.operation instanceof CommandOperation) ||
!this.operation.hasAspect(Aspect.EXPLAINABLE)
) {
throw new MongoError('This command cannot be explained');
}
this.operation.explain = new Explain(verbosity);

return maybePromise(callback, cb => nextFunction(this, cb));
}
Expand Down
5 changes: 3 additions & 2 deletions src/explain.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ export const ExplainVerbosity = {
} as const;

/**
* For backwards compatibility, true is interpreted as
* "allPlansExecution" and false as "queryPlanner".
* For backwards compatibility, true is interpreted as "allPlansExecution"
* and false as "queryPlanner". Prior to server version 3.6, aggregate()
* ignores the verbosity parameter and executes in "queryPlanner".
* @public
*/
export type ExplainVerbosityLike = keyof typeof ExplainVerbosity | boolean;
Expand Down
2 changes: 1 addition & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ export type {
export type { DbPrivate, DbOptions } from './db';
export type { AutoEncryptionOptions, AutoEncryptionLoggerLevels, AutoEncrypter } from './deps';
export type { AnyError, ErrorDescription } from './error';
export type { ExplainOptions, ExplainVerbosity, ExplainVerbosityLike } from './explain';
export type { Explain, ExplainOptions, ExplainVerbosity, ExplainVerbosityLike } from './explain';
export type {
GridFSBucketReadStream,
GridFSBucketReadStreamOptions,
Expand Down
12 changes: 3 additions & 9 deletions src/operations/aggregate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,8 @@ export class AggregateOperation<T = Document> extends CommandOperation<Aggregate
this.readPreference = ReadPreference.primary;
}

if (options?.explain && (this.readConcern || this.writeConcern)) {
throw new MongoError(
'"explain" cannot be used on an aggregate call with readConcern/writeConcern'
);
if (this.explain && this.writeConcern) {
throw new MongoError('"explain" cannot be used on an aggregate call with writeConcern');
}

if (options?.cursor != null && typeof options.cursor !== 'object') {
Expand Down Expand Up @@ -111,10 +109,6 @@ export class AggregateOperation<T = Document> extends CommandOperation<Aggregate
command.hint = options.hint;
}

if (options.explain) {
command.explain = options.explain;
}

command.cursor = options.cursor || {};
if (options.batchSize && !this.hasWriteStage) {
command.cursor.batchSize = options.batchSize;
Expand All @@ -124,4 +118,4 @@ export class AggregateOperation<T = Document> extends CommandOperation<Aggregate
}
}

defineAspects(AggregateOperation, [Aspect.READ_OPERATION, Aspect.RETRYABLE]);
defineAspects(AggregateOperation, [Aspect.READ_OPERATION, Aspect.RETRYABLE, Aspect.EXPLAINABLE]);
7 changes: 6 additions & 1 deletion src/operations/command.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,12 @@ export abstract class CommandOperation<
}

if (this.hasAspect(Aspect.EXPLAINABLE) && this.explain) {
cmd = decorateWithExplain(cmd, this.explain);
if (serverWireVersion < 6 && cmd.aggregate) {
// Prior to 3.6, with aggregate, verbosity is ignored, and we must pass in "explain: true"
cmd.explain = true;
} else {
cmd = decorateWithExplain(cmd, this.explain);
}
}

server.command(
Expand Down
7 changes: 6 additions & 1 deletion src/operations/find.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,11 @@ export class FindOperation extends CommandOperation<FindOptions, Document> {
findCommand.allowDiskUse = options.allowDiskUse;
}

if (this.explain) {
// TODO: For now, we need to manually ensure explain is in the options. This will change after cursor refactor.
this.options.explain = this.explain.verbosity;
}

// TODO: use `MongoDBNamespace` through and through
server.query(
this.ns.toString(),
Expand All @@ -222,4 +227,4 @@ export class FindOperation extends CommandOperation<FindOptions, Document> {
}
}

defineAspects(FindOperation, [Aspect.READ_OPERATION, Aspect.RETRYABLE]);
defineAspects(FindOperation, [Aspect.READ_OPERATION, Aspect.RETRYABLE, Aspect.EXPLAINABLE]);
3 changes: 3 additions & 0 deletions src/operations/find_one.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import type { FindOptions } from './find';
import { MongoError } from '../error';
import type { Server } from '../sdam/server';
import { CommandOperation } from './command';
import { Aspect, defineAspects } from './operation';

/** @internal */
export class FindOneOperation extends CommandOperation<FindOptions, Document> {
Expand Down Expand Up @@ -36,3 +37,5 @@ export class FindOneOperation extends CommandOperation<FindOptions, Document> {
}
}
}

defineAspects(FindOneOperation, [Aspect.EXPLAINABLE]);
16 changes: 4 additions & 12 deletions test/functional/aggregation.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -386,12 +386,7 @@ describe('Aggregation', function () {
* @example-class Collection
* @example-method aggregate
*/
it.skip('should correctly return a cursor and call explain', {
// TODO NODE-2853: This had to be skipped during NODE-2852; un-skip while re-implementing
// cursor explain

// Add a tag that our runner can trigger on
// in this case we are setting that node needs to be higher than 0.10.X to run
it('should correctly return a cursor and call explain', {
metadata: {
requires: {
mongodb: '>2.5.3',
Expand Down Expand Up @@ -461,7 +456,7 @@ describe('Aggregation', function () {
cursor.explain(function (err, result) {
expect(err).to.not.exist;
expect(result.stages).to.have.lengthOf.at.least(1);
expect(result.stages[0]).to.have.key('$cursor');
expect(result.stages[0]).to.have.property('$cursor');

client.close(done);
});
Expand Down Expand Up @@ -928,7 +923,7 @@ describe('Aggregation', function () {
}
});

it('should fail if you try to use explain flag with readConcern/writeConcern', {
it('should fail if you try to use explain flag with writeConcern', {
metadata: {
requires: {
mongodb: '>3.6.0',
Expand All @@ -938,12 +933,9 @@ describe('Aggregation', function () {

test: function (done) {
var databaseName = this.configuration.db;
var client = this.configuration.newClient(this.configuration.writeConcernMax(), {
poolSize: 1
});
var client = this.configuration.newClient({ poolSize: 1 });

const testCases = [
{ readConcern: { level: 'local' } },
{ writeConcern: { j: true } },
{ readConcern: { level: 'local' }, writeConcern: { j: true } }
];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was passing because the write concern was inherited from the client, making it look like using read concern with explain would fail on the server, when really it was the inherited write concern causing the server to return an error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused about this, why do we want to stop throwing an error if you use explain + readConcern? To me it looks like this test would still be passing due to the an error generated by the driver if not for the change here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server doesn't actually throw an error when you just use explain & readConcern, so it is unnecessary to throw from the driver in this case. If you remove the driver error, the server would return an error for this test case because the write concern is inherited from the client, meaning explain is being used with writeConcern, not because there is any issue with explain & readConcern. Does that help?

Copy link
Contributor

@emadum emadum Nov 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify what I'm confused about: this test was explicitly expecting the failure to happen client-side rather than server-side, and from what I can see it was only the change to AggregateOperation that caused this client-side error to stop happening, so I'm not sure I see how the server throwing an error because of the inherited write concern would effect this test. I'd expect a server-side error to cause the test to fail with the Expected aggregation to fail on client instead of server for option message.

I think we were using this error as a hint to the user that read concern isn't supported by the explain operation, rather than merely as a way of avoiding a server-side error, and that we're no longer providing that guidance to the user with this change. I can see an argument in favor of this - they just want to get their explain results, and we should just ignore the read concern the way the server does; just wanted to fully understand the reason for the change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks for the clarification, I was misunderstanding.

I agree with what you're saying at the end. I think it makes sense to not error, since the server also ignores the read concern and since it's not really an error condition, in my opinion, in that nothing has gone wrong/is in an invalid state. I could imagine including a warning or possibly something in the documentation (for all of the explainable operations, not just aggregate), if we feel like the user needs this guidance. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it seems unnecessary to warn about a read concern on explain operations. Is there any additional context you could provide about this @mbroadst ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I can't recall the rationale for that check, I think it was added before my time and we don't have any specifications covering this. We have a similar situation here where we throw a client-side error if a non-local readConcern is provided for servers which don't support it - the server would otherwise ignore the value, it's not clear why we throw an error client-side.

I'm pro this change if it doesn't break any of our tests

Expand Down
Loading