Indexer Technical Documentation

Real-time XRPL ledger indexing for DEX routing. Current state only, DEX transactions only, deterministic and reproducible.

Overview

The LucenDEX indexer monitors the XRPL blockchain for DEX-related transactions and maintains a database with current liquidity state for the router.

Core Capabilities

Real-time streaming: WebSocket connection to rippled API node (256 ledger history)
Transaction filtering: Processes AMM and orderbook transactions
State management: Current AMM reserves and active orderbook offers
Trade tracking: Records Lucendex-executed trades via QuoteHash in memo
Gap detection: Smart backfill for small gaps (within 256-ledger buffer)

What Gets Indexed

✓ Indexed

Current State: AMM pool reserves (all pools on XRPL)
Current State: Active orderbook offers (all offers on XRPL)
Lucendex Trades: Completed trades with QuoteHash in memo (90-day retention)

✗ NOT Indexed

Payments between accounts
Account balances or trust lines
User settings or account data
Any personally identifiable information (PII)

Data Model

AMM Pools (`core.amm_pools`)

CREATE TABLE core.amm_pools (
  account        TEXT PRIMARY KEY,
  asset1         TEXT NOT NULL,
  asset2         TEXT NOT NULL,
  amount1        NUMERIC NOT NULL,
  amount2        NUMERIC NOT NULL,
  trading_fee    INTEGER NOT NULL,
  ledger_index   INTEGER NOT NULL,
  last_updated   TIMESTAMP DEFAULT NOW(),
  meta           JSONB
);

Orderbook Offers (`core.orderbook_state`)

CREATE TABLE core.orderbook_state (
  account        TEXT,
  sequence       INTEGER,
  base_asset     TEXT NOT NULL,
  quote_asset    TEXT NOT NULL,
  side           TEXT CHECK (side IN ('bid','ask')),
  price          NUMERIC NOT NULL,
  base_amount    NUMERIC NOT NULL,
  status         TEXT DEFAULT 'active',
  ledger_index   INTEGER NOT NULL,
  last_updated   TIMESTAMP DEFAULT NOW(),
  meta           JSONB,
  PRIMARY KEY (account, sequence)
);

Gap Detection Strategy

On restart, the indexer checks for missing ledgers between the last processed ledger and current validated ledger:

Small Gaps (within 256-ledger buffer)

Backfill sequentially to maintain continuity. Typical case after short downtime.

Gap detected: 100 ledgers (within API node history)
Backfilling: 100 ledgers
Backfill: 50/100 (50%) - ETA: 1m30s

Large Gaps (>256 ledgers)

Skip backfill and resume from current ledger. Historical data isn't needed—only current state matters.

Large gap detected: 2000 ledgers (> API node retention)
Resuming from current ledger: 100025000

Why skip large gaps? DEX routing only needs current liquidity state (AMM reserves, active offers). State rebuilds quickly from live stream. Completed Lucendex trades are stored with 90-day retention.

Performance Metrics

Processing Time	100-500ms per ledger
Transactions per Ledger	40-150 (typical XRPL mainnet)
Memory Usage	~5-10MB
CPU Usage	<1% (16 vCPU VM)
Database Connections	1 active (pooled)
Ledger Cutoff	Nov 1, 2025 (START_LEDGER)

Security & Access Control

Database role: indexer_rw (read/write)
Router role: router_ro (read-only)
API role: api_ro (read-only)
Encryption: SSL/TLS required for all database connections
Audit trail: All transactions stored with meta JSONB

Data Integrity

Duplicate Prevention

Idempotent upserts prevent data corruption on retries:

ON CONFLICT (account, sequence) 
DO UPDATE SET 
  price = EXCLUDED.price,
  status = EXCLUDED.status,
  ledger_index = EXCLUDED.ledger_index

Checkpoint Continuity

Ledger checkpoints verify chain integrity. Each checkpoint records:

Ledger index and hash
Timestamp and transaction count
Processing duration

Version History

v1.1.0 (2025-11-11):

Single API node architecture (256 ledger history)
Trade tracking with QuoteHash detection
completed_trades table with 90-day retention
Optimized for reduced resource usage

v1.0.0-alpha (2025-11-06):

Initial production release
Gap detection with smart backfill
Build timestamp versioning
Channel buffer optimization (main: 100, backfill: 10K)
START_LEDGER cutoff enforcement