Extend nitro-testnode to run consensus and execution separately

README.md

@@ -1,6 +1,6 @@
 # Nitro Testnode
 
-Nitro-testnode brings up a full environment for local nitro testing (with Stylus support) including a dev-mode geth L1, and multiple instances with different roles.
+Nitro-testnode brings up a full environment for local Nitro testing (with Stylus support) including a dev-mode Geth L1, and multiple node instances with different roles.
 
 ### Requirements
 
@@ -44,9 +44,40 @@ Initialize the node in dev-mode (this will build the docker images from source)
 ```
 To see more options, use `--help`.
 
+## Node topology flags
+
+By default, the testnode runs a single **sequencer** node that handles both consensus and execution in one process.
+
+### `--follower-node`
+
+Adds a **regular follower node** (`regular-follower-node`) that runs consensus and execution together in a single process, following the sequencer. Useful for testing read-only RPC nodes.
+
+```bash
+./test-node.bash --init --dev --follower-node
+```
+
+| Service | Ports |
+|---------|-------|
+| `regular-follower-node` | HTTP `8547` -> `7447`, WS `8548` -> `7548` |
+
+### `--run-consensus-and-execution-in-different-processes`
+
+Adds a **split-process follower** where consensus and execution run as separate containers communicating over authenticated RPC.
+
+```bash
+./test-node.bash --init --dev --run-consensus-and-execution-in-different-processes
+```
+
+| Service | Role | Ports |
+|---------|------|-------|
+| `consensus-follower-node` | Consensus (inbox, feed, validation) | HTTP `8547` -> `7147`, WS `8552` -> `8552` |
+| `execution-follower-node` | Execution (EVM, RPC, GraphQL) | HTTP `8547` -> `7247`, WS `9682` -> `9682` |
+
+Both flags can be combined to run all three follower configurations simultaneously.
+
 ## Further information
 
-### Branch Selection Guide (for devs working *on* nitro-testnode)
+### Branch selection guide (for devs working *on* nitro-testnode)
 
 This repository maintains two main branches with distinct purposes.
 
@@ -86,26 +117,66 @@ Target branch for changes supporting unreleased Nitro features.
 
 ### Working with docker containers
 
-**sequencer** is the main docker to be used to access the nitro testchain. It's http and websocket interfaces are exposed at localhost ports 8547 and 8548 ports, respectively.
+**sequencer** is the main docker container used to access the Nitro testchain. Its HTTP and WebSocket interfaces are exposed at localhost ports 8547 and 8548, respectively.
 
-Stopping, restarting nodes can be done with docker-compose.
+Stopping and restarting nodes can be done with docker-compose.
 
 ### Helper scripts
 
 Some helper scripts are provided for simple testing of basic actions.
 
-To fund the address 0x1111222233334444555566667777888899990000 on l2, use:
+To fund the address 0x1111222233334444555566667777888899990000 on L2, use:
 
 ```bash
 ./test-node.bash script send-l2 --to address_0x1111222233334444555566667777888899990000
 ```
 
+To get the private key of an account, run:
+```bash
+./test-node.bash script print-private-key --account funnel
+```
+
 For help and further scripts, see:
 
 ```bash
 ./test-node.bash script --help
 ```
 
+## Options reference
+
+| Flag | Description |
+|------|-------------|
+| `--init` | Initialize the testnode (removes previous data, prompts for confirmation) |
+| `--init-force` | Initialize without confirmation prompt |
+| `--dev` | Build from local source in dev-mode (implies `--no-simple`) |
+| `--build` | Rebuild all docker images |
+| `--no-build` | Skip all docker image rebuilds |
+| `--simple` | Single node as sequencer/batch-poster/staker (default without `--dev`) |
+| `--no-simple` | Full configuration with separate sequencer/batch-poster/validator/relayer |
+| `--follower-node` | Run a follower node (single process, consensus + execution combined) |
+| `--run-consensus-and-execution-in-different-processes` | Run split consensus and execution follower nodes communicating over RPC |
+| `--validate` | Enable validation (implies `--no-simple`) |
+| `--batchposters` | Number of batch posters [0-3] (implies `--no-simple`) |
+| `--redundantsequencers` | Number of redundant sequencers [0-3] (implies `--no-simple`) |
+| `--l3node` | Enable an L3 node |
+| `--l3-fee-token` | Use a custom fee token on L3 (requires `--l3node`) |
+| `--l3-fee-token-decimals` | Set fee token decimals [0-36] (requires `--l3-fee-token`) |
+| `--l3-fee-token-pricer` | Enable fee token pricer (requires `--l3-fee-token`) |
+| `--l3-token-bridge` | Deploy L2-L3 token bridge (requires `--l3node`) |
+| `--l2-anytrust` | Enable AnyTrust DA on L2 |
+| `--l2-referenceda` | Enable reference DA on L2 |
+| `--l2-timeboost` | Enable TimeBOOST express lane auctions on L2 |
+| `--l2-tx-filtering` | Enable transaction filtering on L2 |
+| `--tokenbridge` | Deploy L1-L2 token bridge |
+| `--no-tokenbridge` | Skip token bridge deployment |
+| `--blockscout` | Build or launch Blockscout explorer |
+| `--pos` | Use proof-of-stake consensus client on L1 |
+| `--detach` | Detach from nodes after running them |
+| `--nowait` | Don't wait for nodes to be ready (requires `--detach`) |
+| `--no-run` | Don't launch nodes (useful with `--build` or `--init`) |
+| `--no-l2-traffic` | Disable L2 spam transaction traffic |
+| `--no-l3-traffic` | Disable L3 spam transaction traffic |
+
 ## Named accounts
 
 ```bash
@@ -116,7 +187,7 @@ sequencer:                  0xe2148eE53c0755215Df69b2616E552154EdC584f
 validator:                  0x6A568afe0f82d34759347bb36F14A6bB171d2CBe
 l2owner:                    0x5E1497dD1f08C87b2d8FE23e9AAB6c1De833D927
 l3owner:                    0x863c904166E801527125D8672442D736194A3362
-l3sequencer:                0x3E6134aAD4C4d422FF2A4391Dc315c4DDf98D1a5
+l3sequencer:                0x3E6134aAD4C4d422FF2A4391Dc315c4DDf98F1a5
 user_l1user:                0x058E6C774025ade66153C65672219191c72c7095
 user_token_bridge_deployer: 0x3EaCb30f025630857aDffac9B2366F953eFE4F98
 user_fee_token_deployer:    0x2AC5278D230f88B481bBE4A94751d7188ef48Ca2
@@ -133,5 +204,3 @@ To run the metrics stack (Prometheus + Grafana) read the instructions in the [me
 Discord - [Arbitrum](https://discord.com/invite/5KE54JwyTs)
 
 Twitter: [Arbitrum](https://twitter.com/arbitrum)
-
-

docker-compose.yaml

@@ -154,6 +154,95 @@ services:
       - "consensus:/consensus"
       - "config:/config"
 
+  execution-follower-node:
+    pid: host # allow debugging
+    image: nitro-node-dev-testnode
+    entrypoint: /usr/local/bin/nitro
+    ports:
+      - "127.0.0.1:7247:8547"
+      - "127.0.0.1:9682:9682"
+    volumes:
+      - "exec-follower-data:/home/user/.arbitrum/local/nitro"
+      - "l1keystore:/home/user/l1keystore"
+      - "config:/config"
+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/base_node_config.json
+      - --node.feed.output.enable
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --node.seq-coordinator.my-url=http://sequencer:8547
+      - --graphql.enable
+      - --graphql.vhosts=*
+      - --graphql.corsdomain=*
+      - --ws.addr=0.0.0.0
+      - --ws.port=9682
+      - --ws.api=net,web3,eth,txpool,debug,nitroexecution
+      - --execution.rpc-server.enable
+      - --execution.rpc-server.public
+      - --execution.rpc-server.authenticated=false
+      - --execution.consensus-rpc-client.url=ws://consensus-follower-node:8552
+      - --execution.consensus-rpc-client.jwtsecret=/config/jwt.hex
+      - --execution.consensus-rpc-client.retries=3
+      - --execution.consensus-rpc-client.connection-wait=15s
+      - --execution.consensus-rpc-client.retry-delay=1s
+    depends_on:
+      - geth
+
+  consensus-follower-node:
+    pid: host # allow debugging
+    image: nitro-node-dev-testnode
+    entrypoint: /usr/local/bin/nitro
+    ports:
+      - "127.0.0.1:7147:8547"
+      - "127.0.0.1:8552:8552"
+    volumes:
+      - "consensus-follower-data:/home/user/.arbitrum/local/nitro"
+      - "l1keystore:/home/user/l1keystore"
+      - "config:/config"
+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/base_node_config.json
+      - --node.seq-coordinator.my-url=http://sequencer:8547
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --graphql.enable
+      - --graphql.vhosts=*
+      - --graphql.corsdomain=*
+      - --ws.addr=0.0.0.0
+      - --ws.port=8552
+      - --ws.api=net,web3,eth,txpool,debug,nitroconsensus
+      - --node.rpc-server.enable
+      - --node.rpc-server.public
+      - --node.rpc-server.authenticated=false
+      - --node.execution-rpc-client.url=ws://execution-follower-node:9682
+      - --node.execution-rpc-client.jwtsecret=/config/jwt.hex
+      - --node.execution-rpc-client.retries=3
+      - --node.execution-rpc-client.connection-wait=15s
+      - --node.execution-rpc-client.retry-delay=1s
+    depends_on:
+      - geth
+
+  regular-follower-node:
+    pid: host # allow debugging
+    image: nitro-node-dev-testnode
+    entrypoint: /usr/local/bin/nitro
+    ports:
+      - "127.0.0.1:7447:8547"
+      - "127.0.0.1:7548:8548"
+    volumes:
+      - "regular-follower-data:/home/user/.arbitrum/local/nitro"
+      - "l1keystore:/home/user/l1keystore"
+      - "config:/config"
+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/base_node_config.json
+      - --node.seq-coordinator.my-url=http://sequencer:8547
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --graphql.enable
+      - --graphql.vhosts=*
+      - --graphql.corsdomain=*
+    depends_on:
+      - geth
+
   sequencer:
     pid: host # allow debugging
     image: nitro-node-dev-testnode
@@ -519,6 +608,9 @@ volumes:
   seqdata_b:
   seqdata_c:
   seqdata_d:
+  exec-follower-data:
+  consensus-follower-data:
+  regular-follower-data:
   unsafestaker-data:
   validator-data:
   poster-data:

scripts/config.ts

@@ -427,6 +427,12 @@ function writeConfigs(argv: any) {
         fs.writeFileSync(path.join(consts.configpath, "poster_config.json"), JSON.stringify(posterConfig))
     }
 
+    let consensusConfig = JSON.parse(baseConfJSON)
+    // Use a different persistent.chain name than the sequencer ("local") to avoid
+    // storage lock conflicts between services sharing the same volume.
+    consensusConfig.persistent.chain = "base-node-local"
+    fs.writeFileSync(path.join(consts.configpath, "base_node_config.json"), JSON.stringify(consensusConfig))
+
     let l3Config = JSON.parse(baseConfJSON)
     delete l3Config["init"]
     l3Config["parent-chain"].connection.url = argv.l2url

test-node.bash

@@ -50,6 +50,8 @@ blockscout=false
 tokenbridge=false
 l3node=false
 consensusclient=false
+run_consensus_and_execution_in_different_processes=false
+follower_node=false
 redundantsequencers=0
 l3_custom_fee_token=false
 l3_custom_fee_token_pricer=false
@@ -175,6 +177,14 @@ while [[ $# -gt 0 ]]; do
             build_utils=true
             shift
             ;;
+        --run-consensus-and-execution-in-different-processes)
+            run_consensus_and_execution_in_different_processes=true
+            shift
+            ;;
+        --follower-node)
+            follower_node=true
+            shift
+            ;;
         --validate)
             simple=false
             validate=true
@@ -343,6 +353,8 @@ while [[ $# -gt 0 ]]; do
             echo --no-build-dev-blockscout  don\'t rebuild dev blockscout docker image
             echo --build-utils         rebuild scripts, rollupcreator, token bridge docker images
             echo --no-build-utils      don\'t rebuild scripts, rollupcreator, token bridge docker images
+            echo --follower-node       run a follower node 
+            echo --run-consensus-and-execution-in-different-processes  run consensus and execution nodes in different processes communicating over RPC
             echo --force-build-utils   force rebuilding utils, useful if NITRO_CONTRACTS_BRANCH or TOKEN_BRIDGE_BRANCH changes
             echo
             echo script runs inside a separate docker. For SCRIPT-ARGS, run $0 script --help
@@ -353,6 +365,14 @@ done
 NODES="sequencer"
 INITIAL_SEQ_NODES="sequencer"
 
+if $run_consensus_and_execution_in_different_processes; then
+    NODES="$NODES consensus-follower-node execution-follower-node"
+fi
+
+if $follower_node; then
+    NODES="$NODES regular-follower-node"
+fi
+
 if ! $simple; then
     NODES="$NODES redis"
 fi
@@ -623,10 +643,13 @@ if $force_init; then
         run_script redis-init --redundancy $redundantsequencers
     fi
 
-    echo == Funding l2 funnel and dev key
+    echo == Spinning up sequencer nodes
     docker compose up --wait $INITIAL_SEQ_NODES
-    sleep 45 # in case we need to create a smart contract wallet, allow for parent chain to recieve the contract creation tx and process it
+    echo == Sleeping for 45s allow for parent chain to recieve the contract creation tx and process it
+    sleep 45
+    echo == Funding l2 funnel and dev key
     run_script bridge-funds --ethamount 100000 --wait
+    echo == Funding l2owner
     run_script send-l2 --ethamount 100 --to l2owner --wait
     rollupAddress=`docker compose run --rm --entrypoint sh poster -c "jq -r '.[0].rollup.rollup' /config/deployed_chain_info.json | tail -n 1 | tr -d '\r\n'"`