Conversation
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
9 times, most recently
from
29f47f3 to
264aa7f
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
a30cbfb to
1e7bdbc
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
b5e980f to
67d47c2
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
95285b0 to
4b2d345
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
cba29a3 to
db1fe83
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
db1fe83 to
35f9ec2
Compare
In preparation for trampoline, rust-lightning has been updated to report multiple HTLCs incoming and outgoing in PaymentForwarded. Support is not yet complete, so we can still expect that we only get a single HTLC in/out for now. This change writes the first entry of the new fields to our PaymentForwarded event, because we don't yet have the serialization macros we need to persist a vec with a default value (as the trait and the type Vec<T> are not owned by this crate). We're not losing any information here, as we only expect regular forwards with this version of LDK. We'll follow with a change that writes the full vec once upstream changes are made to LDK's serialization macros.
Bump rust-lightning to include trampoline changes
Update rust-lightning dependency to rev 49912057895ddfbd6 and bitcoin-payment-instructions to tnull/bitcoin-payment-instructions rev e9d7c07d7affc7714b02. Adapt to API changes: - Remove `ChainSource` generic parameter from `LiquidityManager` type alias as it was dropped upstream - Remove `chain_source` from `LiquiditySourceBuilder` and its call sites since `LiquidityManager::new` no longer accepts it - Add new `lsps1_service_config` field to `LiquidityServiceConfig` Co-Authored-By: HAL 9000
Bump LDK to `49912057` and `bitcoin-payment-instructions` to `e9d7c07d`
Introduces TierStore, a KVStore implementation that manages data across three storage layers: - Primary: Main data store for critical node data - Ephemeral: Secondary store for non-critical, easily-rebuildable data (e.g., network graph) with fast local access - Backup: Tertiary store for disaster recovery with async/lazy operations to avoid blocking primary store - Unit tests for TierStore core functionality
Adds TierStoreConfig and two configuration methods to NodeBuilder: - set_backup_store: Configure backup store for disaster recovery - set_ephemeral_store: Configure ephemeral store for non-critical data Modifies build_with_store to wrap the provided store in a TierStore, as the primary store, applying any configured ephemeral and backup stores. Note: Temporal dead code allowance will be removed in test commit.
Introduce FFI-safe abstractions and builder APIs to allow foreign language targets to configure custom backup and ephemeral stores when constructing nodes with a custom store. Major changes include: - Addition of FfiDynStoreTrait, an FFI-safe equivalent of DynStoreTrait, working around uniffi's lack of support for Pin<Box<T>> - Addition of FfiDynStore, a concrete wrapper for foreign language store implementations - Provision of FfiDynStoreTrait implementation for DynStoreWrapper to bridge native Rust stores to FFI layer (useful in testing) - Extension of ArcedNodeBuilder with methods for configuring backup and ephemeral stores - Exposure of build_with_store so foreign targets can build nodes with custom store implementations - Addition of build_node_with_store test helper to abstract uniffi-gated store wrapping at build_with_store call sites
- Add Rust integration test verifying correct routing to storage tiers - Add Python in-memory KV store and FFI test for tiered storage
Introduce FfiDynStoreInner with per-key write version locks that ensure write ordering and skip stale versions in both sync and async code paths. Test changes: - Unify tier store test helpers to use TestSyncStore for all tiers, replacing mixed SqliteStore/FilesystemStore/TestStore usage that caused test hangs due to TestStore's async write blocking - Split build_node_with_store into cfg-gated versions for uniffi vs non-uniffi feature flags
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
35f9ec2 to
14dbb22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
We introduce
TierStore, aKVStoreimplementation that manages data acrossthree distinct storage layers.
The layers are:
(e.g., network graph). This tier aims to improve latency by leveraging a
local
KVStoredesigned for fast/local access.asynchronously/lazily to avoid blocking primary store operations.
We also permit the configuration of
Nodewith these stores allowingcallers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the
Nodewith TierStore's primary store. These configurationoptions also extend to our foreign interface, allowing bindings target to build the
Node with their own
ffi::KVStoreimplementations.A sample Python implementation is added and tested.
Additionally, we add comprehensive testing for
TierStoreby introducingTierStorecore functionality.Nodebuilt with tiered storage.ffi::KVStoreimplementations.Concerns
It is worth considering the way retry logic is handled, especially because of nested
retries.
TierStorecomes with a basic one by default but there areKVStoreimplementationsthat come with them baked-in (e.g.
VssStore), and thus would have no need forthe wrapper-store's own logic.