Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions docs/lightning-primer-for-qa.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,13 @@ The **fix path** uses **`accept_stale_channel_monitors`** so ldk-node can align

## What to test when Lightning / LDK storage changes

| Area | Why |
|------|-----|
| **Cold start** | Any path that reads/writes ChannelManager, monitors, or VSS must not pair **new** manager with **old** monitor. |
| **Backup / restore** | Restoring must be **consistent snapshots**; partial or older monitor alone is high risk. |
| **Migration** | RN → native or schema changes: avoid overwriting live data with **stale** remote copies. |
| **Recovery** | After `DangerousValue` / `accept_stale`: peers reconnect, chain sync completes, **inbound and outbound** payments work, **second launch** does not repeat recovery forever. |
| **Infra noise** | On regtest, **stale RGS** / gossip can cause transient **“route not found”** — distinguish from persistence bugs (see logs for `DangerousValue` vs routing errors). |
| Area | Why |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cold start** | Any path that reads/writes ChannelManager, monitors, or VSS must not pair **new** manager with **old** monitor. |
| **Backup / restore** | Restoring must be **consistent snapshots**; partial or older monitor alone is high risk. |
| **Migration** | RN → native or schema changes: avoid overwriting live data with **stale** remote copies. |
| **Recovery** | After `DangerousValue` / `accept_stale`: peers reconnect, chain sync completes, **inbound and outbound** payments work, **second launch** does not repeat recovery forever. |
| **Infra noise** | On regtest, **stale RGS** / gossip can cause transient **“route not found”** — distinguish from persistence bugs (see logs for `DangerousValue` vs routing errors). |

## Risks of incorrect “fixes”

Expand All @@ -50,13 +50,13 @@ The **fix path** uses **`accept_stale_channel_monitors`** so ldk-node can align

## Glossary

| Term | Meaning |
|------|--------|
| **Commitment update** | New off-chain state (balances + HTLC set). |
| **`update_id`** | LDK’s persisted notion of how far the ChannelMonitor has advanced vs the ChannelManager for that channel. |
| **HTLC** | **Hash Time-Locked Contract** — conditional payment inside a commitment (hash lock + time lock). |
| **ChannelMonitor** | Per-channel persisted state for chain watching and dispute handling. |
| **DangerousValue** | LDK/ldk-node refusing to load because continuing would violate safety assumptions (e.g. stale monitor). |
| Term | Meaning |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Commitment update** | New off-chain state (balances + HTLC set). |
| **`update_id`** | LDK’s persisted notion of how far the ChannelMonitor has advanced vs the ChannelManager for that channel. |
| **HTLC** | **Hash Time-Locked Contract** — conditional payment inside a commitment (hash lock + time lock). |
| **ChannelMonitor** | Per-channel persisted state for chain watching and dispute handling. |
| **DangerousValue** | LDK/ldk-node refusing to load because continuing would violate safety assumptions (e.g. stale monitor). |
| **accept_stale_channel_monitors** | Explicit recovery mode to load despite mismatch, then heal via protocol + sync (use only in controlled recovery). |

## See also
Expand Down
67 changes: 37 additions & 30 deletions docs/repro-channel-monitor-desync.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@
See also: [Lightning primer for QA](./lightning-primer-for-qa.md) (monitors, HTLCs, gaps, risks).

Related issues:

- [#847 (bitkit-android)](https://github.com/synonymdev/bitkit-android/issues/847)
- iOS support ticket (user logs from 2026-03-18)

Fix branches:

- **iOS**: `fix/stale-monitor-recovery-release`
- **Android**: `fix/stale-monitor-recovery-v2`

Expand All @@ -17,6 +19,7 @@ Build 182 (v2.1.0) introduced `fetchOrphanedChannelMonitorsIfNeeded` which fetch
## Root Cause

On v2.1.0 startup:

1. `fetchOrphanedChannelMonitorsIfNeeded` fetches stale channel monitor from RN backup server
2. Injects it via `setChannelDataMigration` with `channelManager: nil` (monitors only)
3. ldk-node persists the stale monitor to VSS/local storage
Expand All @@ -33,6 +36,7 @@ Failed to read channel manager from store: Value would be dangerous to continue
```

In app logs:

```
Running pre-startup channel monitor recovery check
Found 1 monitors on RN backup for pre-startup recovery
Expand Down Expand Up @@ -89,6 +93,7 @@ RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum).
**Critical**: The RN app's `.env.production` must point the backup server to **staging** (not localhost), because the native apps have `rnBackupServerHost` hardcoded to staging. If the RN app pushes to `127.0.0.1:3003` but the native app queries `bitkit.stag0.blocktank.to`, it will never find the channel monitors and the bug won't trigger.

In `.env.production` for the RN v1.1.6 build, set:

```
BACKUPS_SERVER_HOST=https://bitkit.stag0.blocktank.to/backups-ldk
BACKUPS_SERVER_PUBKEY=02c03b8b8c1b5500b622646867d99bf91676fac0f38e2182c91a9ff0d053a21d6d
Expand Down Expand Up @@ -130,6 +135,7 @@ The app's channel has all balance on the app side. LND needs outbound liquidity
```

Verify with a test payment:

```bash
./bitcoin-cli payinvoice <bolt11_invoice> 10
```
Expand Down Expand Up @@ -166,6 +172,7 @@ Install v2.1.0 **over** the native app → app fails to start LN node (see error
Upgrading from a broken v2.1.0 wallet to v2.1.2 (fix candidate) recovers the wallet. Channels are healed and LN transactions work after recovery.

Fix branches:

- **iOS**: `fix/stale-monitor-recovery-release`
- **Android**: `fix/stale-monitor-recovery-v2`

Expand All @@ -192,34 +199,34 @@ Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be

### Blocktank channel (staging regtest)

| # | Scenario | Result |
|---|----------|--------|
| B1 | v2.0.6 (wallet with 21+ payment gap) → v2.1.0 → confirm broken | Reproduces |
| B2 | Restore broken v2.1.0 wallet into v2.1.2 (clean install + restore) | ✅ Recovered |
| B3 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered |
| B4 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues |
| B5 | v2.0.6 (wallet with gap) → v2.1.1 → v2.1.2 | ✅ No issues |
| B6 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues |
| B7 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered |
| # | Scenario | Result |
| --- | -------------------------------------------------------------------- | ------------ |
| B1 | v2.0.6 (wallet with 21+ payment gap) → v2.1.0 → confirm broken | Reproduces |
| B2 | Restore broken v2.1.0 wallet into v2.1.2 (clean install + restore) | ✅ Recovered |
| B3 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered |
| B4 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues |
| B5 | v2.0.6 (wallet with gap) → v2.1.1 → v2.1.2 | ✅ No issues |
| B6 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues |
| B7 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered |

### 3rd-party channel (local docker)

| # | Scenario | Result |
|---|----------|--------|
| T1 | v2.0.6 (wallet with 30+ payment gap) → v2.1.0 → confirm broken | Reproduces |
| T2 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered |
| T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues |
| T4 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues |
| T5 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered |
| # | Scenario | Result |
| --- | -------------------------------------------------------------------- | ------------ |
| T1 | v2.0.6 (wallet with 30+ payment gap) → v2.1.0 → confirm broken | Reproduces |
| T2 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered |
| T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues |
| T4 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues |
| T5 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered |

### Version reference

| Version | iOS branch | Android branch |
|---------|-----------|---------------|
| v1.1.6 | tag `v1.1.6` (RN) | tag `v1.1.6` (RN) |
| v2.0.6 | `chore/e2e-updater-url` | — |
| v2.0.3 | — | `chore/e2e-updater-url` |
| v2.1.0 | build 182 | build 182 |
| Version | iOS branch | Android branch |
| ------------ | ------------------------------------ | ------------------------------- |
| v1.1.6 | tag `v1.1.6` (RN) | tag `v1.1.6` (RN) |
| v2.0.6 | `chore/e2e-updater-url` | — |
| v2.0.3 | — | `chore/e2e-updater-url` |
| v2.1.0 | build 182 | build 182 |
| v2.1.2 (fix) | `fix/stale-monitor-recovery-release` | `fix/stale-monitor-recovery-v2` |

---
Expand All @@ -233,11 +240,11 @@ Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be

## Files

| File | Purpose |
|------|---------|
| `docs/lightning-primer-for-qa.md` | Background: ChannelManager vs ChannelMonitor, HTLCs, gaps, test focus |
| `test/specs/receive-ln-payments.e2e.ts` | Automated spec to receive N Lightning payments |
| `wdio.no-install.conf.ts` | WDIO config that attaches to existing app (no reinstall) |
| `docker/bitcoin-cli` | Local docker CLI with `openchannel`, `payinvoice`, `mine`, `send` commands |
| `scripts/pay-lightning-address.sh` | Shell script to pay BOLT11/BIP21/LN address via Blocktank |
| `scripts/pay-lightning-address-loop.sh` | Shell script to send N payments to a Lightning address |
| File | Purpose |
| --------------------------------------- | -------------------------------------------------------------------------- |
| `docs/lightning-primer-for-qa.md` | Background: ChannelManager vs ChannelMonitor, HTLCs, gaps, test focus |
| `test/specs/receive-ln-payments.e2e.ts` | Automated spec to receive N Lightning payments |
| `wdio.no-install.conf.ts` | WDIO config that attaches to existing app (no reinstall) |
| `docker/bitcoin-cli` | Local docker CLI with `openchannel`, `payinvoice`, `mine`, `send` commands |
| `scripts/pay-lightning-address.sh` | Shell script to pay BOLT11/BIP21/LN address via Blocktank |
| `scripts/pay-lightning-address-loop.sh` | Shell script to send N payments to a Lightning address |
43 changes: 9 additions & 34 deletions test/helpers/actions.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
import type { ChainablePromiseElement } from 'webdriverio';
import { reinstallApp } from './setup';
import { deposit, mineBlocks } from './regtest';
import { doNavigationClose, doTriggerTimedSheet, openSettings } from './navigation';

export { doNavigationClose, doTriggerTimedSheet } from './navigation';

export const sleep = (ms: number) => browser.pause(ms);

Expand Down Expand Up @@ -603,16 +606,8 @@ export async function handleAndroidAlert(
}
}

export async function doNavigationClose() {
await tap('HeaderMenu');
await tap('DrawerWallet');
await sleep(500);
}

export async function getSeed(): Promise<string> {
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('BackupSettings');
await openSettings('security');
await tap('BackupWallet');

// get seed from SeedContainer
Expand Down Expand Up @@ -810,9 +805,7 @@ async function assertAddressTypeSwitchFeedback() {
}

export async function switchPrimaryAddressType(nextType: addressTypePreference) {
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('AdvancedSettings');
await openSettings('advanced');
await tap('AddressTypePreference');
await tap(nextType);
await assertAddressTypeSwitchFeedback();
Expand Down Expand Up @@ -1165,6 +1158,8 @@ export type ToastId =
| 'TransactionRemovedToast'
| 'InvalidAddressToast'
| 'ExpiredLightningToast'
| 'DevModeEnabledToast'
| 'DevModeDisabledToast'
| 'InsufficientSpendingToast'
| 'InsufficientSavingsToast';

Expand Down Expand Up @@ -1204,23 +1199,6 @@ export async function acknowledgeExternalSuccess() {
await sleep(300);
}

/**
* Triggers the timed backup sheet by navigating to settings and back.
* Since timed sheets are sometimes triggered by user behavior (when user goes back to home screen),
* we need to trigger them manually.
*
* @example
* // Trigger backup sheet before testing dismissal
* await doTriggerTimedSheet();
*/
export async function doTriggerTimedSheet() {
await sleep(700); // wait for any previous animations to finish
await tap('HeaderMenu');
await tap('DrawerSettings');
await sleep(500); // wait for the app to settle
await doNavigationClose();
}

export async function dismissBackgroundPaymentsTimedSheet({
triggerTimedSheet = false,
}: { triggerTimedSheet?: boolean } = {}) {
Expand Down Expand Up @@ -1363,9 +1341,7 @@ export async function acknowledgeHighBalanceWarning({
// enable/disable widgets in settings
export async function toggleWidgets() {
await sleep(3000);
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('GeneralSettings');
await openSettings();
await tap('WidgetsSettings');
const widgets = await elementsByText('Widgets');
await widgets[1].click();
Expand Down Expand Up @@ -1476,8 +1452,7 @@ export async function attemptRefreshOnHomeScreen() {
}

export async function waitForBackup() {
await tap('HeaderMenu');
await tap('DrawerSettings');
await openSettings('security');
await tap('BackupSettings');
await elementById('AllSynced').waitForDisplayed();
await doNavigationClose();
Expand Down
9 changes: 3 additions & 6 deletions test/helpers/lnd.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
typeText,
} from './actions';
import { LndConfig } from './constants';
import { openSettings } from './navigation';
import createLndRpc, { LnRpc, WalletUnlockerRpc } from '@radar/lnrpc';

export async function setupLND(
Expand Down Expand Up @@ -94,9 +95,7 @@ export async function waitForActiveChannel(
}

export async function getLDKNodeID(): Promise<string> {
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('AdvancedSettings');
await openSettings('advanced');
// wait for LDK to start
await sleep(5000);
await tap('LightningNodeInfo');
Expand All @@ -123,9 +122,7 @@ export async function connectToLND(lndNodeID: string, { navigationClose = true }
}

export async function checkChannelStatus({ size = '100 000' } = {}) {
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('AdvancedSettings');
await openSettings('advanced');
await tap('Channels');
await sleep(1000);
await tap('Channel');
Expand Down
46 changes: 46 additions & 0 deletions test/helpers/navigation.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
import { tap, sleep } from './actions';

export type SettingsTab = 'general' | 'security' | 'advanced';

/**
* Opens the Settings screen at the given tab.
* General is the default tab so no extra tap is needed for it.
*/
export async function openSettings(tab: SettingsTab = 'general') {
await tap('HeaderMenu');
await tap('DrawerSettings');
if (tab !== 'general') {
await tap(`Tab-${tab}`);
await sleep(300);
}
}

/**
* Opens the Support screen from the drawer menu.
*/
export async function openSupport() {
await tap('HeaderMenu');
await tap('DrawerSupport');
}

/**
* Closes the drawer and navigates back to the Wallet home screen.
*/
export async function doNavigationClose() {
await tap('HeaderMenu');
await tap('DrawerWallet');
await sleep(500);
}

/**
* Triggers the timed backup sheet by navigating to settings and back.
* Since timed sheets are sometimes triggered by user behavior (when user goes back to home screen),
* we need to trigger them manually.
*/
export async function doTriggerTimedSheet() {
await sleep(700);
await tap('HeaderMenu');
await tap('DrawerSettings');
await sleep(500);
await doNavigationClose();
}
8 changes: 5 additions & 3 deletions test/helpers/regtest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ async function localMineBlocks(count: number): Promise<void> {

function lndRestRequest(
path: string,
body: Record<string, unknown>,
body: Record<string, unknown>
): Promise<Record<string, unknown>> {
const tlsCert = fs.readFileSync(lndConfig.tls);
const macaroon = fs.readFileSync(lndConfig.macaroonPath).toString('hex');
Expand All @@ -74,15 +74,17 @@ function lndRestRequest(
},
(res) => {
let data = '';
res.on('data', (chunk: Buffer) => { data += chunk.toString(); });
res.on('data', (chunk: Buffer) => {
data += chunk.toString();
});
res.on('end', () => {
if (res.statusCode && res.statusCode >= 200 && res.statusCode < 300) {
resolve(JSON.parse(data) as Record<string, unknown>);
} else {
reject(new Error(`LND REST ${res.statusCode}: ${data}`));
}
});
},
}
);
req.on('error', reject);
req.write(payload);
Expand Down
5 changes: 2 additions & 3 deletions test/specs/backup.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import {
} from '../helpers/actions';
import { ciIt } from '../helpers/suite';
import { ensureLocalFunds } from '../helpers/regtest';
import { openSettings } from '../helpers/navigation';

describe('@backup - Backup', () => {
let electrum: Awaited<ReturnType<typeof initElectrum>> | undefined;
Expand Down Expand Up @@ -66,9 +67,7 @@ describe('@backup - Backup', () => {
await tap('NavigationBack');

// - change settings (currency to GBP) //
await tap('HeaderMenu');
await tap('DrawerSettings');
await tap('GeneralSettings');
await openSettings();
await tap('CurrenciesSettings');
const gbp_opt = await elementByText('GBP (£)');
await gbp_opt.waitForDisplayed();
Expand Down
Loading