Why Substreams Make Traditional ETL Obsolete

The Graph is sunsetting hosted subgraphs in favor of Substreams-powered Subgraphs. The legacy system required developers to write custom mappings for each chain, a process taking weeks to months.\n- Unified Indexing: Write a Substream once, deploy it to any supported chain (Ethereum, Polygon, Arbitrum) instantly.\n- Real-Time Data: Enables sub-second data availability for dApps, versus the multi-block confirmation delays of traditional subgraphs.\n- Cost Efficiency: Eliminates the need to run and sync a dedicated Graph Node for each chain, reducing infrastructure overhead by ~70%.

Goldsky uses Substreams to power high-frequency data products for protocols like Uniswap, Aave, and Compound. Traditional ETL pipelines batch data in ~15 minute intervals, making real-time dashboards and alerts impossible.\n- Streaming-First: Delivers blockchain state changes as they occur, with ~500ms end-to-end latency.\n- Deterministic Outputs: Ensures every consumer gets the exact same data stream, critical for financial applications and MEV analysis.\n- Modular Consumption: Clients subscribe only to the specific data streams they need (e.g., swap events, liquidations), avoiding the cost of processing full blocks.

Pinax provides liquidity intelligence across Ethereum, Solana, and Avalanche. Aggregating DEX data across heterogeneous chains with traditional methods requires maintaining separate, brittle indexing infra for each.\n- Single Codebase: A Substream module for DEX trades compiles to native code for each chain's execution environment.\n- Time-Travel Queries: Analysts can rewind the stream to any block to backtest strategies or audit historical states, a feature prohibitively slow with Postgres-based ETL.\n- Scalability: Adding support for a new chain (e.g., Base, Blast) is a configuration change, not a re-engineering project.

Financial institutions and regulatory tech firms need to monitor transactions for sanctions or illicit finance. Legacy chain analysis tools rely on delayed, batched data, creating compliance gaps.\n- Sub-Second Alerts: Substreams can trigger real-time alerts for transactions involving sanctioned addresses (e.g., OFAC lists) before they are confirmed in multiple blocks.\n- Full-Data Fidelity: Processes every transaction and internal trace, unlike RPC-based methods that miss complex, nested calls.\n- Audit Trail: The deterministic, versioned nature of Substreams provides an immutable record for compliance audits, superior to custom database logs.

Top NFT markets need to index complex events like bulk listings, trait bids, and royalty payments across millions of contracts. Monolithic indexers struggle with schema changes and data consistency.\n- Modular Schemas: Different teams can own Substreams for Listings, Bids, and Sales, merging outputs into a unified sink.\n- Handles Forking: NFT markets are prone to wash trading and chain reorgs. Substreams' linear processing model and deterministic outputs guarantee data consistency after a reorg.\n- Developer Velocity: New features like Blur's lending integration can be indexed and served in days, not quarters.

Risk models for protocols like Aave and Compound require calculating collateralization ratios and liquidation thresholds in real-time. Batch ETL creates dangerous lag between on-chain state and risk metrics.\n- Live Risk Parameters: Streams position health and market volatility data, enabling dynamic parameter adjustment proposals based on live feeds.\n- Parallelized Computation: Heavy calculations (e.g., VaR simulations) are offloaded to parallel Substream modules, scaling horizontally.\n- Protocol Integration: Outputs can be consumed directly by keeper networks like Chainlink Automation to trigger protective measures.

Traditional ETL pipelines operate on a poll-and-batch model, creating a fundamental delay between on-chain events and application state. This is fatal for DeFi, gaming, or any system requiring sub-second updates.

• Latency: Batch processing introduces 5-60 second delays, missing critical arbitrage or liquidation windows.
• Inefficiency: Repeatedly re-scanning the chain for each new query wastes 90%+ of compute cycles on redundant work.

Substreams provides a firehose of pre-indexed data the moment a block is finalized. Developers subscribe to streams of decoded events, calls, or state changes, treating the blockchain as a real-time database.

• Performance: Applications react in ~100ms from block finalization, enabling high-frequency on-chain logic.
• Efficiency: Data is computed once, shared globally—eliminating the redundant work of siloed indexers like The Graph.

Substreams modules are written in Rust and execute in a massively parallel pipeline. This unlocks performance and scalability impossible for sequential ETL jobs.

• Scale: Processes 10,000+ blocks per second by parallelizing across historical data and new blocks simultaneously.
• Portability: The same Substream runs identically across nodes (e.g., Pinax, StreamingFast), ensuring verifiable, consistent data without vendor lock-in.

Protocols like Uniswap, Aave, and Blur cannot rely on minute-old data. Substreams powers the next generation of intent-based systems (e.g., UniswapX, CowSwap) and NFT marketplaces that need instant floor price updates and trait filtering.

• Use Case: MEV bots and liquidators require the fastest possible data feed to capture value.
• Ecosystem: Used by Across Protocol for fast bridging proofs and Goldsky for instant NFT indexing.

Maintaining a reliable, low-latency ETL pipeline requires a dedicated team and six-figure cloud bills. Substreams commoditizes this layer.

• OpEx Reduction: Shifts cost from continuous DevOps overhead to a predictable consumption model.
• Developer Focus: Teams ship product logic, not data infrastructure, reducing time-to-market by months.

Substreams' architecture is chain-agnostic. Supporting Ethereum, Polygon, Arbitrum, and Base today, it is the logical substrate for a unified multi-chain data layer. This contrasts with siloed solutions like LayerZero's messaging or chain-specific indexers.

• Vision: A single query interface for all EVM chains, making multi-chain app development trivial.
• Standardization: Positions Substreams as the TCP/IP for blockchain state, similar to how libp2p standardized networking.