What is a Retrieval Deal?

definition

BLOCKCHAIN STORAGE

A Retrieval Deal is a paid, on-demand transaction for accessing data stored on a decentralized storage network, distinct from the initial storage agreement.

A Retrieval Deal is a transaction in a decentralized storage network, such as Filecoin or IPFS, where a client pays a provider to fetch and deliver a specific piece of stored data. Unlike a Storage Deal, which secures data for a fixed duration, a retrieval deal is a short-term, on-demand contract focused on data accessibility and transfer speed. The terms, including price and bandwidth, are typically negotiated peer-to-peer via a retrieval market, making it analogous to a content delivery network (CDN) service for decentralized data.

The technical mechanism often involves a payment channel or micropayment system, where the client streams small, incremental payments to the provider as data is delivered in chunks. This pay-as-you-stream model ensures the provider is incentivized to deliver data quickly and completely, while protecting the client from paying for failed or slow transfers. Protocols like Graphsync and Bitswap in IPFS facilitate the actual data exchange, with retrieval deals providing the economic layer on top.

Key components of a retrieval deal include the retrieval query (finding which provider has the data), price discovery (negotiating a fee per byte), and data transfer. This model is crucial for applications requiring low-latency access, such as serving website assets, streaming video, or querying datasets. It decouples the cost of long-term archival from the cost of frequent access, creating a more efficient market for data utility.

how-it-works

DATA RETRIEVAL

How a Retrieval Deal Works

A retrieval deal is the on-chain or off-chain agreement that governs the process of fetching and delivering stored data to a user on a decentralized storage network.

A retrieval deal is a transaction between a client (data requester) and a retrieval provider (often a storage miner) to fetch and deliver a specific piece of stored data. Unlike a storage deal, which secures data for the long term, a retrieval deal is focused on low-latency, on-demand access. The deal's core parameters are the Payment Channel used for incremental, pay-as-you-deliver microtransactions, the Price per Byte, and the required Retrieval Peer ID for establishing a direct connection. This mechanism is fundamental to making decentralized storage networks like Filecoin and IPFS practically usable for applications.

The process begins when a client queries the network's Distributed Hash Table (DHT) or a retrieval market indexer to find providers hosting the desired content, identified by its Content Identifier (CID). Once a provider is located, the two parties negotiate terms off-chain, typically using the Graphsync or Bitswap data transfer protocols. The client establishes a payment channel and funds it with the network's native token (e.g., FIL). The provider then begins streaming the data in small chunks, with the client automatically verifying and paying for each chunk received, ensuring a trustless exchange.

Key technical components enable this efficient flow. The Payment Channel allows for rapid, numerous small payments without incurring blockchain transaction fees for each one. Data transfer protocols like Graphsync manage the actual exchange of data blocks and associated payments. The entire deal is designed for speed and cost-effectiveness, often completing in seconds, with costs calculated based on the actual amount of data delivered. This contrasts with the slower, contract-heavy process of initiating a storage deal, which is optimized for durability rather than instant access.

In practice, retrieval deals power user-facing applications. When you load a website or video stored on IPFS via a public gateway, a retrieval deal (often subsidized by the gateway operator) is executed in the background. Decentralized applications (dApps) integrate retrieval client libraries to programmatically fetch data. The ecosystem also includes specialized retrieval marketplaces and indexers that help clients find the fastest or cheapest provider, creating a competitive landscape for data delivery services on decentralized networks.

key-features

FILECOIN MECHANICS

Key Features of Retrieval Deals

Retrieval deals are on-chain agreements that govern the payment and delivery of stored data to clients, distinct from the storage deal that initially secures the data.

01

Pay-Per-Use Billing

Unlike storage deals which involve prepayment for a fixed duration, retrieval deals use a pay-as-you-go model. Clients pay for the data they actually retrieve, typically measured in bytes transferred. This is facilitated by payment channels that enable fast, incremental micropayments for each data segment delivered.

02

Retrieval Market Participants

The retrieval market involves distinct roles:

Retrieval Clients: End-users or applications requesting data.
Retrieval Providers: Nodes (often separate from storage providers) that fetch data from storage miners and serve it to clients.
Indexers: Services that help clients discover which providers hold the data they need, often using the Filecoin Content Identifier (CID).

03

Incentive & Pricing Mechanism

Pricing is dynamic and set by the retrieval provider, creating a competitive market. Key mechanisms include:

Uncapped Pricing: Providers can set any price, encouraging competition.
Bid-Ask Model: Clients request data, providers respond with offers.
Bandwidth Incentives: Providers are rewarded for high-speed, reliable data delivery, unlike storage deals which reward proven long-term storage.

04

Technical Protocols & Speed

Retrieval is optimized for low latency and is protocol-agnostic. The primary protocol is Graphsync, but the newer Bitswap protocol is favored for retrieval due to its faster performance. Deals are executed off-chain for speed, with on-chain settlement only for final payment channel closure. This separation allows for sub-second data delivery initiation.

05

Contrast with Storage Deals

Retrieval and storage deals are complementary but fundamentally different:

Purpose: Storage deals secure data longevity; retrieval deals enable data access.
Payment: Storage is prepaid, time-based (FIL per epoch); retrieval is pay-per-byte.
On-Chain Footprint: Storage deals are heavy, on-chain commitments; retrieval deals are lightweight, often off-chain agreements.
Providers: A storage miner is not required to also be a retrieval provider.

06

Use Cases & Importance

Retrieval deals are critical for practical data utility. They enable:

Content Delivery Networks (CDNs): Serving web content, videos, and datasets on-demand.
DApp Data Access: Allowing decentralized applications to fetch user or state data quickly.
Data Monetization: Allowing dataset owners to sell access via retrieval markets.
Archival Access: Retrieving data from cold storage for analysis or use.

FILECOIN PROTOCOL

Storage Deal vs. Retrieval Deal

A comparison of the two primary transaction types on the Filecoin network, focusing on their distinct purposes, participants, and economic models.

Feature	Storage Deal	Retrieval Deal
Primary Purpose	Persistent, long-term data storage	On-demand, short-term data delivery
Core Participants	Client (Data Owner) & Storage Provider	Client (Data User) & Retrieval Provider
Agreement Type	On-chain, verifiable smart contract	Off-chain, payment-channel-based agreement
Duration	Fixed term (e.g., 1+ years)	Instantaneous (seconds to minutes)
Payment Model	Pre-committed, locked collateral	Pay-as-you-go, per-byte/segment
Data Transfer Protocol	GraphSync (for sealing data)	GraphSync or Bitswap (for retrieval)
Network Consensus Role	Central to Proof-of-Spacetime (PoSt)	Independent of consensus, layer-2 service
Typical Pricing Factors	Duration, redundancy, provider reputation	Bandwidth, latency, market competition

ecosystem-usage

RETRIEVAL DEAL

Ecosystem Usage & Examples

A retrieval deal is a marketplace transaction where a client pays a storage provider to fetch and deliver specific data from the Filecoin network. This section details its core mechanics and real-world applications.

01

Core Transaction Flow

A retrieval deal follows a structured, peer-to-peer process:

Client Query: A client discovers a storage provider's data offering via the Filecoin Retrieval Market.
Deal Proposal: The client sends a proposal specifying the Data CID, desired retrieval speed, and price.
Payment Channel: A secure, off-chain payment channel is established for incremental, trustless micropayments.
Data Transfer: The provider streams the data in exchange for ongoing payments, using protocols like Graphsync or Bitswap.

02

Key Technical Components

The deal is enabled by several Filecoin-specific protocols and structures:

Payment Channels: Off-chain state channels that enable pay-as-you-go streaming without on-chain latency or fees for each payment.
Retrieval Market: A decentralized discovery layer where providers advertise their available data and terms.
Data Transfer Protocols: Graphsync is commonly used for efficient retrieval of large DAGs, while Bitswap serves as a fallback.
Piece CID: The unique content identifier that the client uses to request the exact data segment.

03

Use Case: Decentralized CDN

Retrieval deals power content delivery by allowing platforms to fetch media from geographically distributed providers.

Example: A video streaming dApp stores film reels across global providers. When a user hits play, the app initiates a retrieval deal with the lowest-latency provider, streaming the video in real-time via micropayments.
Benefit: This bypasses centralized CDN costs and creates a competitive market for bandwidth, potentially lowering delivery costs.

04

Use Case: Dataset Access for AI/ML

Machine learning teams use retrieval deals to access large, curated datasets stored on Filecoin.

Example: An AI startup needs a 10TB image dataset for model training. Instead of downloading the entire archive, they use retrieval deals to stream specific batches of data directly into their training pipeline, paying only for the data transferred.
Benefit: Enables efficient, on-demand access to massive datasets without prohibitive upfront storage or bandwidth costs.

05

Use Case: Blockchain Data Archival

Blockchain explorers and analytics firms use retrieval deals to access historical chain data.

Example: An analytics platform needs three-year-old Ethereum state data to run a historical analysis. The data is archived on Filecoin. The platform initiates a retrieval deal to pull the specific state trie nodes required, paying for the precise data fetched.
Benefit: Provides cost-effective, verifiable access to immutable historical data that is too large to store on-chain or in traditional cloud storage.

06

Contrast with Storage Deal

It's critical to distinguish retrieval from the primary storage deal:

Purpose: A storage deal is for long-term persistence (years). A retrieval deal is for short-term data access (seconds/minutes).
Payment Model: Storage uses upfront, locked collateral and scheduled proofs. Retrieval uses streaming micropayments via payment channels.
Incentives: Storage rewards providers for durability and time. Retrieval rewards providers for bandwidth and latency.
On-chain Footprint: Storage deals are on-chain contracts. Retrieval deals are primarily off-chain arrangements.

technical-details-retrieval-market

DATA RETRIEVAL

Technical Details: The Retrieval Market

An overview of the decentralized market where users pay to access data stored on Filecoin, focusing on the mechanics of retrieval deals.

A retrieval deal is a short-lived, micropayment-based agreement between a client and a retrieval provider (a storage provider or specialized retrieval miner) for the fast, on-demand delivery of specific data stored on the Filecoin network. Unlike a long-term storage deal, which focuses on persistent archival, a retrieval deal is optimized for low-latency access, where the provider streams the requested data to the client in exchange for incremental payments, typically using the Graphsync or Bitswap data transfer protocols. The deal is considered complete once the full dataset has been delivered and all payments are settled.

The retrieval market operates on a pay-as-you-go model, which is fundamentally different from the prepaid, long-term model of storage. Clients initiate a deal by finding a provider who hosts the desired data, identified by its Data CID (Content Identifier). The two parties then negotiate terms, including price per byte and payment intervals. As the provider streams data chunks, the client sends small, frequent payments, often using payment channels to enable instant, trustless transactions. This model incentivizes providers to offer high-bandwidth, low-latency service and allows clients to pay only for the data they successfully receive.

Key technical components enabling this market include payment channels and data transfer protocols. A payment channel, often implemented as a state channel, allows for a series of off-chain micropayments that are later settled on-chain, minimizing transaction fees and delays. For data transfer, Graphsync is commonly used for efficient bulk retrieval from Filecoin storage providers, while Bitswap, the native protocol of IPFS, is used for more granular, peer-to-peer data exchange. These protocols manage the actual streaming of data blocks alongside the payment logic.

The performance and economics of the retrieval market are driven by competition among providers. Factors like geographic proximity, network bandwidth, latency, and price determine which provider a client selects. Specialized retrieval miners may emerge who do not offer long-term storage but instead maintain caches of popular data to provide ultra-fast retrieval services. This creates a robust, competitive market for data access that complements Filecoin's core storage function, ensuring that stored data remains readily usable and valuable.

security-considerations

RETRIEVAL DEAL

Security & Incentive Considerations

A retrieval deal is a payment-for-data agreement between a client and a storage provider, distinct from long-term Filecoin storage deals, focusing on the secure and incentivized delivery of content.

01

Payment Channel Security

Retrieval deals use off-chain payment channels (like state channels) to enable incremental, trust-minimized micropayments. This prevents the 'pay and pray' problem by allowing the client to pay for each piece of data as it is received. The security model relies on the ability to submit the latest signed payment voucher to the blockchain if the provider stops delivering, ensuring the client only pays for what they get.

02

Incentive Alignment

The protocol aligns incentives by making payment contingent on verifiable data delivery. Key mechanisms include:

Pay-as-you-go: Providers earn revenue per byte delivered, incentivizing fast, complete service.
Unilateral completion: The client can unilaterally close the payment channel with the last valid voucher, preventing provider hold-up.
Reputation systems: Reliable providers attract more business, creating a market for quality retrieval.

03

Sybil & Censorship Resistance

Retrieval markets must be resistant to Sybil attacks (where a single entity creates many fake nodes) and censorship. Decentralization is key:

Many providers: Content should be retrievable from multiple, independent storage providers.
Content addressing (CIDs): Clients request data by its cryptographic hash, not a provider-controlled URL, preventing gatekeeping.
Peer discovery: Using decentralized protocols (e.g., libp2p) to find providers reduces reliance on centralized registries that could be censored.

04

Data Authenticity & Integrity

Clients must cryptographically verify that the received data is correct. This is achieved through:

Merkle proofs: Providers send cryptographic proofs (e.g., Piece Inclusion Proofs) that the delivered blocks correspond to the requested Content Identifier (CID).
On-chain verification: The deal's terms, including the root CID, can be anchored on-chain, providing a verifiable record of the agreement.
Client-side validation: The client's node validates all proofs before accepting data and releasing payment.

05

Liveness & Retrievability Guarantees

Unlike storage deals with slashing, retrieval deals lack strong liveness guarantees. The primary assurance is economic: providers are paid for being online and serving data. Considerations include:

No collateral at risk: Providers don't post collateral for retrieval, so penalties for being offline are limited to lost income.
Redundancy: Clients should ensure data is stored with multiple providers to increase the chance of availability.
Fallback mechanisms: Systems may use retrieval markets to bid for service or cache content in Content Delivery Networks (CDNs).

06

Protocol Examples & Evolution

Different networks implement retrieval with varying security models:

Filecoin Retrieval: Uses Graphsync and Bitswap over libp2p with payment channels.
Ethereum's Portal Network: A lightweight protocol for retrieving historical chain data without incentives, relying on altruism.
Arweave: Uses Proof of Access where miners must prove they hold random past blocks, incentivizing them to store and serve the entire dataset. The field is evolving towards retrieval markets with explicit bidding.

RETRIEVAL DEAL

Frequently Asked Questions (FAQ)

Common questions about the process of fetching data from the Filecoin network, covering mechanics, costs, and key participants.

A retrieval deal is a paid transaction where a client pays a storage provider to fetch and deliver a specific piece of data stored on the Filecoin network. Unlike a storage deal, which is a long-term commitment, a retrieval deal is a short-lived, on-demand transaction focused on data delivery speed and bandwidth. The deal is brokered and facilitated by the Filecoin Retrieval Market, which connects clients seeking data with providers who have it. Payment is typically made per byte delivered, and the data is transferred directly from the provider's storage to the client using efficient protocols like Graphsync or Bitswap.

Retrieval Deal