feat(server): add integration test for message deduplication#3099
Open
seokjin0414 wants to merge 6 commits intoapache:masterfrom
Open
feat(server): add integration test for message deduplication#3099seokjin0414 wants to merge 6 commits intoapache:masterfrom
seokjin0414 wants to merge 6 commits intoapache:masterfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3099 +/- ##
=============================================
- Coverage 73.17% 54.54% -18.64%
Complexity 943 943
=============================================
Files 1123 1121 -2
Lines 97892 87279 -10613
Branches 75065 64471 -10594
=============================================
- Hits 71632 47602 -24030
- Misses 23671 37110 +13439
+ Partials 2589 2567 -22
🚀 New features to boost your workflow:
|
seokjin0414
force-pushed
the
2872-add-integration-test-for-message-deduplication
branch
4 times, most recently
from
6fdbeb7 to
46db1ea
Compare
tungtose
reviewed
Apr 12, 2026
tungtose
suggested changes
Apr 12, 2026
… calculation - Add empty batch guard after prepare_for_persistence() to prevent server panic when all messages in a batch are duplicates - Fix partition offset calculation to use actual last offset from batch instead of arithmetic that ignores gaps created by dedup removal - Create message deduplicator for lazily-initialized partitions in init_partition_inner() instead of hardcoding None Signed-off-by: shin <sars21@hanmail.net>
Add 7-step scenario testing the full deduplication pipeline: - Auto-generated IDs (id=0) all pass through with unique UUIDs - Explicit IDs are accepted on first send - Duplicate IDs are rejected, original payload preserved - All-duplicate batch does not crash server (regression for empty batch) - Mixed batch with partial duplicates only accepts new IDs - Offsets are monotonically increasing after dedup removal - TTL expiry allows previously seen IDs to be accepted again Signed-off-by: shin <sars21@hanmail.net>
Signed-off-by: shin <sars21@hanmail.net>
…urnal offset tracking - When all messages in a batch are duplicates, advance partition offset past the assigned (but removed) offset range to prevent offset reuse in subsequent batches - Fix journal current_offset to use actual last offset from batch instead of arithmetic that ignores gaps created by dedup removal Signed-off-by: shin <sars21@hanmail.net>
- Reduce builder duplication with conditional .id() call - Add comment linking DEDUP_TTL_SECS to server config in specific.rs Signed-off-by: shin <sars21@hanmail.net>
seokjin0414
force-pushed
the
2872-add-integration-test-for-message-deduplication
branch
from
46db1ea to
e3efc2c
Compare
The conditional mut builder pattern causes a type mismatch because bon's type-state changes the builder type after calling .id(). Use maybe_id() which accepts Option<u128> in a single chain instead. Signed-off-by: shin <sars21@hanmail.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2872
Bug fixes
Empty batch panic (
messages.rs): Afterprepare_for_persistence()removes all duplicate messages, subsequent.unwrap()calls onfirst_timestamp(),last_timestamp(),last_offset()panic. Added empty batch guard.Offset calculation (
messages.rs):last_offsetwas computed ascurrent_offset + count - 1, which doesn't account for offset gaps created by dedup removal. Changed to usesegment.end_offset(the actual last offset from batch).Deduplicator not created (
partitions.rs):init_partition_inner()hardcodedNonefor themessage_deduplicatorparameter. Addedcreate_message_deduplicator()call matching the bootstrap path.Integration test scenario
Test plan
cargo fmt --all -- --checkcargo clippy -p server -p integration --all-targets -- -D warningscargo test -p integration --test mod -- message_deduplication(CI)