[BUG] Airtable triggers are getting OOM

[andrewjschuang]

Describe the bug
Depending on the number of records (and its contents) being fetched in the Airtable triggers, it can get an OOM error (for example > 150).

This request is to add an optional prop:

    maxRecordsPerExecution: {
      type: "integer",
      label: "Maximum Records Per Execution",
      description: "Set a maximum number of records to fetch per execution if your trigger is failing",
      optional: true,
    }

to the triggers, and it translates to the param maxRecords in the Airtable API:

    const params = {
      ...,
      maxRecords: this.maxRecordsPerExecution,
    }

Not sure if this should be added to the base common file or to each individual trigger code.

Additional context
Reported by a user

[vunguyenhung]

Hello everyone, I have tested this PR and there're some test cases failed or needed improvement.

Please check the test report below for more information
https://vunguyenhung.notion.site/BUG-Airtable-triggers-are-getting-OOM-1c1bf548bb5e81f7a4d4e04b5b249eed

[jcortes]

Hi @vunguyenhung I've just pushed some changes where I'm not using the maxRecords prop. I've tested them and seems like I couldn't reproduce the OOM error so can you please give it a try if with this change the error got fixed. Thanks!

[vunguyenhung]

Hey @jcortes, I also couldn't reproduce either, though I think it's best to include the maxRecords to our API calls, maybe we can hard code it to 50, then we can add the info box to inform user that maximum the source will emit 50 events each, new runs will emit remaining events

[vunguyenhung]

Hello everyone, I have tested this PR and there're some test cases failed or needed improvement.

Please check the test report below for more information
https://vunguyenhung.notion.site/BUG-Airtable-triggers-are-getting-OOM-1c1bf548bb5e81f7a4d4e04b5b249eed

[jcortes]

@vunguyenhung please try with latest push, I saw your report and you tested with the one before the latest sorry for that.
On the other hand about your suggestion on maxRecords I think in this case doesn't apply because it will only response with those number of records potentially leaving out records as you pointed out in a previous QA test

[vunguyenhung]

Hi everyone, all test cases are passed! Ready for release!

Test report
https://vunguyenhung.notion.site/BUG-Airtable-triggers-are-getting-OOM-1c1bf548bb5e81f7a4d4e04b5b249eed