HADOOP-19348. Integrate analytics accelerator into S3A. #7433
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
force-pushed
the
aal-integration
branch
from
944fbbb to
a8da9b2
Compare
4 tasks
|
ok, squash didn't take either |
|
build was terminated after 12 hours starts ends raise on common-dev, assume it's a yetus/config/timeout issue. |
steveloughran
approved these changes
Feb 26, 2025
|
weird that this one is also checking out: |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Feb 27, 2025
| @@ -636,7 +636,8 @@ public void initialize(URI name, Configuration originalConf) | |||
| // If encryption method is set to CSE-KMS or CSE-CUSTOM then CSE is enabled. | |||
| isCSEEnabled = CSEUtils.isCSEEnabled(getS3EncryptionAlgorithm().getMethod()); | |||
|
|
|||
| isAnalyticsAccelaratorEnabled = StreamIntegration.determineInputStreamType(conf).equals(InputStreamType.Analytics); | |||
| isAnalyticsAccelaratorEnabled = StreamIntegration.determineInputStreamType(conf) | |||
|
|
||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); | ||
| } | ||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); |
There was a problem hiding this comment.
add a check for the filesystem iostats too, to make sure it trickles up
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
commented
Feb 27, 2025
|
|
||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); | ||
|
|
||
| verifyStatisticCounterValue(getFileSystem().getIOStatistics(), |
There was a problem hiding this comment.
@steveloughran can't get this one to work..
There was a problem hiding this comment.
Ok think it's to do with the test base class maybe, and these one's don't aggregate to the FS. I added an assertion in ITestS3AOpenCost which extends AbstractS3ACostTest and can see it get's propagated.
s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:dumpFileSystemIOStatistics(130)) - Aggregate FileSystem Statistics counters=((action_file_opened=1)
(action_http_head_request=3)
(audit_request_execution=13)
....
(stream_read_analytics_opened=1)
....
Think this is enough for now, and I'll remove this assertion and merge as is. Once we support statistics properly, we can add in tests in ITestS3AOpenCost
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Copy of #7334, checking if a new PR from a new branch keeps yetus happy.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?