HDDS-12542. Support S3 signed multi chunks payload verification#10006
Draft
chungen0126 wants to merge 1 commit intoapache:masterfrom
Draft
HDDS-12542. Support S3 signed multi chunks payload verification#10006chungen0126 wants to merge 1 commit intoapache:masterfrom
chungen0126 wants to merge 1 commit intoapache:masterfrom
Conversation
chungen0126
changed the title
henrybear327
suggested changes
Mar 30, 2026
|
|
||
| # Context & Motivation | ||
|
|
||
| Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. |
Contributor
There was a problem hiding this comment.
Suggested change
| Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. | |
| Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. |
| To guarantee the correctness, stability, and security of the newly introduced chunk verification logic, a comprehensive testing strategy will be executed. This plan covers both granular unit testing for the stream parsing logic and end-to-end integration testing using official AWS SDKs. | ||
|
|
||
| - Unit Tests (SignedChunksInputStream.java): There is an existing test class for SignedChunksInputStream. To complete it, we should make sure it works well on different signatures. | ||
| - Integration Tests: |
Contributor
There was a problem hiding this comment.
We should likely also add integration tests for STREAMING-AWS4-HMAC-SHA256-PAYLOAD and STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER.
|
|
||
| ## Incremental Hashing | ||
|
|
||
| To maintain a low memory footprint during the continuous buffering process, the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams. This ensures that we validate the payload on the fly without allocating massive temporary byte arrays, avoiding Garbage Collection (GC) spikes during large multi-gigabyte uploads. |
Contributor
There was a problem hiding this comment.
I think the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams won't work.
I was reading Task 3: Calculate Signature, and I saw that the signing is done through StringToSign, which needs to be generated by following the instructions in Create a string to sign
| - Native Module Compilation: Similar to Ozone's existing rocks-native module, we will introduce a dedicated module (e.g., hdds-aws-crt-native). This module will compile the AWS CRT C/C++ source code using CMake directly during the Ozone Maven build process. | ||
| - Dynamic Loading: We will leverage Ozone's existing NativeLibraryLoader infrastructure to safely extract and load the compiled dynamic libraries (.so, .dylib) at runtime. | ||
| - JNI Integration: A Java wrapper will be implemented to pass the secret key to the native KDF function to derive the ECDSA public key, which will then be used by a new ECDSAChunkSignatureVerifier. | ||
|
|
Contributor
There was a problem hiding this comment.
Maybe it's also worth calling out in the design that for -PAYLOAD-TRAILER types, e.g., we need to:
- include
x-amz-trailerin the header and specify the trailing header names as a string in a comma-separated list - all trailing headers are written after the final chunk. In the case of uploading the data in multiple chunks, we must send a final chunk with 0 bytes of data before sending the trailing header.
Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a design doc for supporting s3 multi chunks upload verification.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12542
How was this patch tested?
None.