HDDS-13919. S3 Conditional Writes (PutObject) [1/2] by peterxcli · Pull Request #9815 · apache/ozone

peterxcli · 2026-02-24T11:32:38Z

What changes were proposed in this pull request?

Add createKeyIfNotExists, rewriteKeyIfMatch to RpcClient API, prevent the logic mix wit the existing rewriteKey
Object Endpoint put endpoint handler multiplex createKeyIfNotExists, rewriteKeyIfMatch and createKey
OM side validation logic of etag matching in key creation and commit request
Error translation from OM exception S3 Error for conditional request

ozone/hadoop-hdds/docs/content/design/s3-conditional-requests.md

Lines 59 to 82 in 6c8a08e

    
           ## Specification 
        
           ### AWS S3 Conditional Write Specification 
        
           #### If-None-Match Header 
        
           ``` 
        
           If-None-Match: "*" 
        
           ``` 
        
           - Succeeds only if object does NOT exist 
        
           - Returns `412 Precondition Failed` if object exists 
        
           - Primary use case: Create-only semantics 
        
           #### If-Match Header 
        
           ``` 
        
           If-Match: "<etag>" 
        
           ``` 
        
           - Succeeds only if object EXISTS and ETag matches 
        
           - Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches 
        
           - Primary use case: Atomic updates (compare-and-swap)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13919

How was this patch tested?

Clean CI:
https://github.com/peterxcli/ozone/actions/runs/23634401199/job/68841474397

unit tests for affected components
rpc client integration test for newly introduced two Bucket API
s3 sdk test
robot test

…not exists" (If-None-Match) semantics.

…-create-if-not-exists

…nt tests and protocol definition.

…-create-if-not-exists

…ion handling - Implemented tests for `createKey` and `rewriteKey` methods to validate behavior when using the `EXPECTED_GEN_CREATE_IF_NOT_EXISTS` constant. - Added scenarios for key creation when the key is absent and when it already exists. - Enhanced the `rewriteFailsWhenKeyExists` test to cover cases for both committed and uncommitted keys. - Updated error handling to ensure correct responses for key existence checks.

…not exists" (If-None-Match) semantics.

…nt tests and protocol definition.

…ion handling - Implemented tests for `createKey` and `rewriteKey` methods to validate behavior when using the `EXPECTED_GEN_CREATE_IF_NOT_EXISTS` constant. - Added scenarios for key creation when the key is absent and when it already exists. - Enhanced the `rewriteFailsWhenKeyExists` test to cover cases for both committed and uncommitted keys. - Updated error handling to ensure correct responses for key existence checks.

jojochuang

from a quick glance it looks mostly reasonable.

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

…-conditional-writes

Copilot

Pull request overview

Adds end-to-end support for S3 conditional PUT semantics (If-None-Match: * and If-Match: <etag>) by extending the client RPC API, S3 Gateway PUT handling, and Ozone Manager (create/commit) validation and error mapping.

Changes:

Extend Ozone client APIs with createKeyIfNotExists and rewriteKeyIfMatch and wire them through RpcClient, ClientProtocol, and OzoneBucket.
Add S3 Gateway conditional header parsing/multiplexing for PUT and translate OM conditional failures to S3 412 PreconditionFailed.
Add OM create/commit validation for expectedETag, new OM exception/status codes, and comprehensive unit/integration/robot/S3 SDK tests.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java	Adds unit tests for conditional PUT behavior and ETag parsing.
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/OzoneBucketStub.java	Extends test bucket stub with conditional create/rewrite behaviors.
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java	Wires new bucket APIs through the client protocol test stub.
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Consts.java	Defines constants for conditional request headers.
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java	Implements conditional PUT header handling and OM→S3 error translation.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java	Adds OM create-phase tests for expected ETag validation.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java	Adds OM commit-phase tests for expected ETag validation.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java	Propagates expected ETag from request into key info preparation.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java	Adds create-phase validation for `expectedETag` against existing key metadata.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequestWithFSO.java	Clears `expectedETag` before persisting committed keys (FSO path).
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java	Clears `expectedETag` on commit and validates ETag at commit time.
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto	Adds new status codes and `expectedETag` fields to protocol messages.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/OzoneRpcClientTests.java	Adds integration tests for the new bucket APIs and ETag-based rewrite.
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java	Adds AWS SDK v2 coverage for conditional PUT behaviors.
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java	Adds AWS SDK v1 coverage for conditional PUT behaviors.
hadoop-ozone/dist/src/main/smoketest/s3/conditionalput.robot	Adds robot smoketests validating conditional PUT via awscli.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java	Propagates expected ETag from client args into OM RPC requests.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java	Persists/serializes the new `expectedETag` field on key info.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java	Adds `expectedETag` to OM key args and protobuf conversion.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java	Adds new OM exception result codes for ETag mismatch/unavailable.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java	Implements new conditional APIs using `expectedDataGeneration` / `expectedETag`.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java	Exposes new conditional write methods in the client protocol API.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java	Adds bucket-level wrappers for the new conditional APIs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

jojochuang

just a few code style comments otherwise looks good to me.

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

ivandika3

Thanks @peterxcli for the patch, left initial comments.

hadoop-ozone/dist/src/main/smoketest/s3/conditionalput.robot

...gration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java

hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

…and rewriteIfMatch key in RpcClient

ivandika3

@peterxcli thanks for the update, left additional comments.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

peterxcli · 2026-04-01T08:23:37Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

        return Response.ok().status(HttpStatus.SC_OK).build();
      }

+      String ifNoneMatch = getHeaders().getHeaderString(


I’m still thinking about how to refactor this(line 283-314), as the conditional request validation logic dominates the function.

https://issues.apache.org/jira/browse/HDDS-14958

…dk test

peterxcli · 2026-04-01T10:09:47Z

@ivandika3 please take another look, Thanks!

ivandika3

@peterxcli Thanks for the update. LGTM +1.

jojochuang · 2026-04-01T15:40:14Z

Merged. Thanks @peterxcli and @ivandika3

peterxcli · 2026-04-01T15:42:36Z

Thanks @ivandika3, @jojochuang for reviewing!

peterxcli added 17 commits November 20, 2025 11:52

Recognizes expectedDataGeneration = -1L as a sentinel for "create if …

afa8b6c

…not exists" (If-None-Match) semantics.

correct the new result code position

7868a86

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

d630699

…-create-if-not-exists

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

f7ad505

…-create-if-not-exists

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

2fdb6de

…-create-if-not-exists

HDDS-14070. Update key generation mismatch handling in Ozone RPC clie…

da492ce

…nt tests and protocol definition.

Introduce EXPECTED_GEN_CREATE_IF_NOT_EXISTS constant

0d4dfb0

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

f238ade

…-create-if-not-exists

revert the new KEY_GENERATION_MISMATCH error code

4e9665c

Recognizes expectedDataGeneration = -1L as a sentinel for "create if …

817e33f

…not exists" (If-None-Match) semantics.

correct the new result code position

021747b

HDDS-14070. Update key generation mismatch handling in Ozone RPC clie…

0fd6730

…nt tests and protocol definition.

Introduce EXPECTED_GEN_CREATE_IF_NOT_EXISTS constant

c64beca

revert the new KEY_GENERATION_MISMATCH error code

6b33766

Implement S3 conditional write support

6c8a08e

peterxcli changed the title ~~Hdds 13919 conditional writes~~ HDDS-13919. S3 Conditional Writes (PutObject) Feb 24, 2026

jojochuang reviewed Mar 7, 2026

View reviewed changes

peterxcli added 9 commits March 13, 2026 01:09

Add aws kit tests for S3 conditional object put

57d7788

Add tests for S3 object put with conditional headers

6d85ee1

Merge branch 'master' into HDDS-13919-conditional-writes

c15cd6a

Merge branch 'master' into HDDS-13963-atomic-create-if-not-exists

88aa08e

Add ATOMIC_CREATE_IF_NOT_EXISTS version gate

a1898c0

suggestion

e5f5e3d

fix pmd

86489b6

Merge branch 'HDDS-13963-atomic-create-if-not-exists' into HDDS-13919…

9081e96

…-conditional-writes

fix pmd

d6395a9

peterxcli mentioned this pull request Mar 25, 2026

HDDS-13963. Atomic Create-If-Not-Exists #9332

Merged

fix new smoke test

c10fb9e

peterxcli requested a review from jojochuang March 28, 2026 03:10

jojochuang requested a review from Copilot March 31, 2026 00:23

Copilot started reviewing on behalf of jojochuang March 31, 2026 00:24 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

jojochuang reviewed Mar 31, 2026

View reviewed changes

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java Show resolved Hide resolved

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java Show resolved Hide resolved

ivandika3 reviewed Mar 31, 2026

View reviewed changes

copilot suggestion

5764a70

peterxcli marked this pull request as draft March 31, 2026 17:02

peterxcli added 2 commits April 1, 2026 01:04

Use a unified builder to refactor create, rewrite, createIfNotExists …

e9a45cf

…and rewriteIfMatch key in RpcClient

enhance check in robot test and s3 test

e455b88

peterxcli marked this pull request as ready for review April 1, 2026 01:07

peterxcli requested review from ivandika3 and jojochuang April 1, 2026 01:18

ivandika3 reviewed Apr 1, 2026

View reviewed changes

peterxcli added 3 commits April 1, 2026 14:16

allow streaming key to use conditional put request

b496634

handle * etag matching

d929ed9

set error message when validating the conditional request

d5d74c9

peterxcli requested a review from ivandika3 April 1, 2026 08:14

peterxcli commented Apr 1, 2026

View reviewed changes

peterxcli added 3 commits April 1, 2026 16:39

use standard sdk request builder to build conditional request in v2 s…

77aa2e6

…dk test

add conditional put endpoints for streaming

2ed823d

checkstyle

3a5e880

ivandika3 approved these changes Apr 1, 2026

View reviewed changes

jojochuang merged commit e9cceba into apache:master Apr 1, 2026
45 checks passed

peterxcli changed the title ~~HDDS-13919. S3 Conditional Writes (PutObject)~~ HDDS-13919. S3 Conditional Writes (PutObject) [1/2] Apr 1, 2026

peterxcli deleted the HDDS-13919-conditional-writes branch April 1, 2026 17:44

peterxcli mentioned this pull request Apr 1, 2026

HDDS-13919. S3 Conditional Writes (PutObject) [2/2] - Reuse Atomic Rewrite at Commit Path #10023

Open

	## Specification

	### AWS S3 Conditional Write Specification

	#### If-None-Match Header

	```
	If-None-Match: "*"
	```

	- Succeeds only if object does NOT exist
	- Returns `412 Precondition Failed` if object exists
	- Primary use case: Create-only semantics

	#### If-Match Header

	```
	If-Match: "<etag>"
	```

	- Succeeds only if object EXISTS and ETag matches
	- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches
	- Primary use case: Atomic updates (compare-and-swap)

Conversation

peterxcli commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jojochuang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterxcli Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterxcli Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

peterxcli commented Apr 1, 2026

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang commented Apr 1, 2026

Uh oh!

peterxcli commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

peterxcli commented Feb 24, 2026 •

edited

Loading

jojochuang left a comment •

edited

Loading

peterxcli Apr 1, 2026 •

edited

Loading