Skip to content

HTTP-63 Add caching #109

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 3 commits into from
Jul 30, 2024
Merged

HTTP-63 Add caching #109

merged 3 commits into from
Jul 30, 2024

Conversation

grzegorz8
Copy link
Member

@grzegorz8 grzegorz8 commented Jul 29, 2024

Description

Introduce caching similar to the Flink JDBC connector, so long running synchronous Flink jobs can cache content resulting in less API calls.

Resolves #63

I took over this PR: #94 due to author inactivity.

PR Checklist

davidradl and others added 2 commits July 29, 2024 11:29
Signed-off-by: David Radley <david_radley@uk.ibm.com>
 * Use LookupFunction and AsyncLookupFunction interfaces
 * Add integration tests
@grzegorz8 grzegorz8 force-pushed the feature/63-add-cache branch from 6be2698 to 24ae7d0 Compare July 29, 2024 13:53
@grzegorz8 grzegorz8 marked this pull request as ready for review July 29, 2024 13:54
@grzegorz8 grzegorz8 requested review from kristoffSC and MarekMaj July 29, 2024 14:10
Copy link
Collaborator

@kristoffSC kristoffSC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im super happy to see this finally added to the connector :)
Thanks @grzegorz8 !!!

@grzegorz8
Copy link
Member Author

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

@MarekMaj
Copy link
Contributor

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

Looks like jacoco report is not generated, logs show:

05:46:48,640 [INFO] --- jacoco-maven-plugin:0.8.7:report (jacoco-report) @ flink-http-connector ---
05:46:48,643 [INFO] Skipping JaCoCo execution due to missing execution data file.

The reason may be related to this added configuration of maven-surefire-plugin

<argLine>--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED</argLine>

Documentation suggest to add @{argLine} in maven-surefire-plugin if we want to keep this arguments

Copy link
Contributor

File Coverage [98.86%] 🍏
HttpLookupTableSourceFactory.java 100% 🍏
AsyncHttpTableLookupFunction.java 100% 🍏
HttpLookupTableSource.java 99.23% 🍏
JavaNetHttpPollingClient.java 98.59% 🍏
HttpTableLookupFunction.java 92.65% 🍏
Total Project Coverage 94.39% 🍏

@grzegorz8
Copy link
Member Author

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

Looks like jacoco report is not generated, logs show:

05:46:48,640 [INFO] --- jacoco-maven-plugin:0.8.7:report (jacoco-report) @ flink-http-connector ---
05:46:48,643 [INFO] Skipping JaCoCo execution due to missing execution data file.

The reason may be related to this added configuration of maven-surefire-plugin

<argLine>--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED</argLine>

Documentation suggest to add @{argLine} in maven-surefire-plugin if we want to keep this arguments

Thanks. It helped.

@grzegorz8 grzegorz8 merged commit da0b0ca into main Jul 30, 2024
3 checks passed
@grzegorz8 grzegorz8 deleted the feature/63-add-cache branch July 30, 2024 15:07
@kristoffSC kristoffSC mentioned this pull request Aug 1, 2024
2 tasks
PeatBoy pushed a commit to hyperpaas/flink-http-connector that referenced this pull request Jan 21, 2025
Co-authored-by: David Radley <david_radley@uk.ibm.com>
Co-authored-by: Grzegorz Kołakowski <grzegorz.kolakowski@getindata.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement Cache in Lookup Http Source
4 participants