Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP-63 Add caching #109

Merged
merged 3 commits into from
Jul 30, 2024
Merged

HTTP-63 Add caching #109

merged 3 commits into from
Jul 30, 2024

Conversation

grzegorz8
Copy link
Member

@grzegorz8 grzegorz8 commented Jul 29, 2024

Description

Introduce caching similar to the Flink JDBC connector, so long running synchronous Flink jobs can cache content resulting in less API calls.

Resolves #63

I took over this PR: #94 due to author inactivity.

PR Checklist

davidradl and others added 2 commits July 29, 2024 11:29
Signed-off-by: David Radley <david_radley@uk.ibm.com>
 * Use LookupFunction and AsyncLookupFunction interfaces
 * Add integration tests
@grzegorz8 grzegorz8 force-pushed the feature/63-add-cache branch from 6be2698 to 24ae7d0 Compare July 29, 2024 13:53
@grzegorz8 grzegorz8 marked this pull request as ready for review July 29, 2024 13:54
@grzegorz8 grzegorz8 requested review from kristoffSC and MarekMaj July 29, 2024 14:10
Copy link
Collaborator

@kristoffSC kristoffSC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im super happy to see this finally added to the connector :)
Thanks @grzegorz8 !!!

@grzegorz8
Copy link
Member Author

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

@MarekMaj
Copy link
Contributor

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

Looks like jacoco report is not generated, logs show:

05:46:48,640 [INFO] --- jacoco-maven-plugin:0.8.7:report (jacoco-report) @ flink-http-connector ---
05:46:48,643 [INFO] Skipping JaCoCo execution due to missing execution data file.

The reason may be related to this added configuration of maven-surefire-plugin

<argLine>--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED</argLine>

Documentation suggest to add @{argLine} in maven-surefire-plugin if we want to keep this arguments

Copy link
Contributor

File Coverage [98.86%] 🍏
HttpLookupTableSourceFactory.java 100% 🍏
AsyncHttpTableLookupFunction.java 100% 🍏
HttpLookupTableSource.java 99.23% 🍏
JavaNetHttpPollingClient.java 98.59% 🍏
HttpTableLookupFunction.java 92.65% 🍏
Total Project Coverage 94.39% 🍏

@grzegorz8
Copy link
Member Author

@kristoffSC Build for 1.18.1 fails due to:


Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
    at async open (node:internal/fs/promises:636:25)
    at async Object.readFile (node:internal/fs/promises:1246:14)
    at async /home/runner/work/_actions/madrapps/jacoco-report/v1.2/dist/index.js:12458:25
    at async Promise.all (index 0) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'
}

Any thoughts?

Looks like jacoco report is not generated, logs show:

05:46:48,640 [INFO] --- jacoco-maven-plugin:0.8.7:report (jacoco-report) @ flink-http-connector ---
05:46:48,643 [INFO] Skipping JaCoCo execution due to missing execution data file.

The reason may be related to this added configuration of maven-surefire-plugin

<argLine>--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED</argLine>

Documentation suggest to add @{argLine} in maven-surefire-plugin if we want to keep this arguments

Thanks. It helped.

@grzegorz8 grzegorz8 merged commit da0b0ca into main Jul 30, 2024
3 checks passed
@grzegorz8 grzegorz8 deleted the feature/63-add-cache branch July 30, 2024 15:07
@kristoffSC kristoffSC mentioned this pull request Aug 1, 2024
2 tasks
PeatBoy pushed a commit to hyperpaas/flink-http-connector that referenced this pull request Jan 21, 2025
Co-authored-by: David Radley <david_radley@uk.ibm.com>
Co-authored-by: Grzegorz Kołakowski <grzegorz.kolakowski@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement Cache in Lookup Http Source
4 participants