With over 600 ecosystem projects and 632k+ max transaction throughput, Polkadot has become one of the major blockchains with a thriving ecosystem. Building on Polkadot is fairly straighforward via the Polkadot SDK. The SDK is a versatile developer kit designed to facilitate building on the Polkadot network by making the required components for building chains, parachains, rollups, and more, available to interested developers.

The Polkadot SDK is composed of five major components made up of Substrate, FRAME, Cumulus, XCM, and the node implementation protocol. Substrate is a highly customizable Rust framework for building application-specific chains from modular and extensible components. It includes default implementations of core infrastrasture components, allowing developers to focus on application logic.

Polkadot SDK

As chains mature and handle increasingly complex data and higher transaction volumes, the efficiency of data retrieval and storage becomes paramount for maintaining performance and scalability. Traditional on-chain data storage, while providing immutability and transparency, can become a bottleneck when dealing with frequent and complex queries, especially for large datasets.

To address these challenges, off-chain indexing presents a compelling solution for optimizing data handling in Substrate-based chains. This guide aims to provide a thorough understanding of off-chain indexing within the Polkadot ecosystem, offering detailed implementation guidance and exploring the various considerations for its effective utilization.

Background

Substrate provides a highly flexible and modular environment for blockchain development. Traditionally, blockchain data resides on-chain, ensuring immutability and network-wide consensus. However, for applications dealing with large datasets or requiring complex and frequent queries, this on-chain paradigm can become a performance bottleneck and significantly increase storage costs.

Polkadot Node

Technical Terms

  • On-chain Storage: Data that is stored directly on the chain and is part of the network’s state, secured by consensus.

  • Off-chain Storage: Data that is stored outside the main chain, typically on individual nodes’ local storage or external databases.

  • Offchain Workers (OCWs): Substrate’s feature allowing for the execution of long-running, potentially non-deterministic tasks off the main chain, with the ability to read on-chain state and interact with the external world.

  • Off-chain Indexing: A Substrate feature enabling the runtime to directly write data to a node’s local off-chain storage, which can then be queried by Offchain Workers or via RPC.

  • Runtime: The state transition logic of a Substrate-based chain, defining how the chain’s state changes.

  • Extrinsic: A signed transaction that can trigger state changes in the chain’s runtime.

In today’s rapidly evolving tech landscape, chain applications are moving beyond simple token transfers to power complex ecosystems like DeFi, NFTs, and decentralized social media. These applications generate vast amounts of data, and efficient data retrieval is crucial for user experience, analytics, and overall platform scalability. Off-chain indexing offers a vital tool to decouple frequently queried or derived data from the core, expensive on-chain storage, leading to faster query times, reduced on-chain storage burden, and the ability to support more sophisticated data-driven features.

Deep Dive

The Power of Off-Chain Indexing in Substrate

Substrate’s off-chain indexing allows the chain’s runtime logic itself to write data directly to the local storage of the node executing that logic. This is distinct from Offchain Workers performing external computations and then submitting transactions. The data written through off-chain indexing is not subject to the same consensus mechanisms as on-chain data, making writes significantly faster and less resource-intensive. This indexed data can then be accessed by Offchain Workers for further processing or served via custom RPC endpoints.

Consider a decentralized marketplace where users create listings with numerous attributes (price, description, categories, etc.). Storing and querying all these attributes directly on-chain for every search can be inefficient. With off-chain indexing, when a new listing is created (an on-chain event), the runtime can extract relevant attributes and write them to the off-chain storage, indexed for efficient searching. An Offchain Worker or a custom RPC handler can then query this off-chain index to quickly retrieve listings based on various criteria without needing to iterate through all on-chain events.

This direct write capability from the runtime provides a highly efficient way to create searchable and filterable datasets derived from on-chain events or state changes. It offloads query-intensive tasks from the core chain, improving overall performance and responsiveness for end-users.

Implementing Off-Chain Indexing

Implementing off-chain indexing in Substrate involves defining what data to index and where to store it. Substrate provides a local storage API within the runtime that can be used for this purpose. When a relevant on-chain event occurs or a specific state transition happens, the runtime logic can extract the necessary information and use the offchain::local_storage_set function to store it under a specific key in the node’s local storage. It’s crucial to design a well-structured key-value system for efficient retrieval.

Let’s say we want to index transfer events in a simple token pallet. When a Transfer event is emitted, the runtime can extract the from, to, and value and store this information off-chain using a composite key like (block_number, event_index) or a key based on the to address for quick balance lookups. An Offchain Worker could then periodically aggregate these off-chain transfer records to calculate user balances or generate transaction history summaries. The following snippet illustrates a simplified example within a Substrate pallet’s on_finalize hook:

#![cfg_attr(not(feature = "std"), no_std)]

// Import necessary modules and traits from the frame_support and frame_system crates.
use frame_support::{
    decl_module, decl_storage, decl_event, decl_error, dispatch, ensure,
    traits::{Currency, ExistenceRequirement},
};
use frame_system::{ensure_signed, offchain::StorageKind}; // Import StorageKind for off-chain storage.
use sp_std::prelude::*; // Import standard library prelude for common types like Vec.
use codec::{Encode, Decode}; // Import Encode and Decode for serializing and deserializing data.

#[cfg(test)]
mod mock; // Define a module for unit tests.

#[cfg(test)]
mod tests; // Define a module containing the test cases.

// Define the Trait for this pallet. It specifies the types this pallet depends on.
pub trait Trait: frame_system::Trait + 'static {
    // Define the Event type for this pallet. It must be convertible to the frame_system's Event type.
    type Event: From<Event<Self>> + Into<<Self as frame_system::Trait>::Event>;
    // Define the Currency type used for token transfers.
    type Currency: Currency<Self::AccountId>;
}

// Declare the events that this pallet can emit.
decl_event!(
    pub enum Event<T> where
        AccountId = <T as frame_system::Trait>::AccountId,
        Balance = <T as Trait>::Currency::Balance,
    {
        // Event emitted when a successful transfer occurs.
        Transfer(AccountId, AccountId, Balance),
    }
);

// Declare the errors that this pallet can return during dispatchable calls.
decl_error! {
    pub enum Error for Module<T: Trait> {
        // Error returned when the sender has insufficient balance.
        InsufficientBalance,
    }
}

// Declare the storage items for this pallet.
decl_storage! {
    pub trait Store for Module<T: Trait> as Example {
        // A simple storage item to hold an optional u32 value.
        pub Something get(fn something): Option<u32>;
    }
}

// Declare the module logic.
decl_module! {
    pub struct Module<T: Trait>(_) {
        // Define the error type for this module.
        type Error = Error<T>;

        // Define the deposit event function used to emit events.
        fn deposit_event<T>() = default;

        // A dispatchable function for transferring tokens.
        #[pallet::weight(10_000)]
        pub fn transfer(origin, to: T::AccountId, value: T::Currency::Balance) -> dispatch::DispatchResult {
            // Ensure that the caller of this function is a signed account.
            let from = ensure_signed(origin)?;

            // Get the free balance of the sender.
            let balance = T::Currency::free_balance(&from);

            // Ensure that the sender has enough balance to make the transfer.
            ensure!(balance >= value, Error::<T>::InsufficientBalance);

            // Perform the token transfer.
            T::Currency::transfer(&from, &to, value, ExistenceRequirement::KeepAlive)?;

            // Emit a Transfer event.
            Self::deposit_event(Event::Transfer(from, to, value));

            // Return Ok to indicate success.
            Ok(())
        }

        // The on_finalize hook is called at the end of each block.
        fn on_finalize(block_number: T::BlockNumber) {
            // Get the list of events that occurred in the current block.
            if let Some(last_event) = frame_system::Module::<T>::events().last() {
                // Match on the last event to see if it's a Transfer event.
                if let Event::Transfer(from, to, value) = last_event.event {
                    // Create a unique key for storing the transfer information off-chain.
                    // The key includes the event name, block number, and event index in the block.
                    let key = ("transfer", block_number, frame_system::Module::<T>::event_count() - 1).encode();
                    // Encode the transfer information (from, to, value) to be stored.
                    let value_to_store = (from, to, value).encode();
                    // Store the encoded transfer information in the node's local off-chain storage.
                    // StorageKind::PERSISTENT ensures the data persists across node restarts.
                    sp_io::offchain::local_storage_set(StorageKind::PERSISTENT, &key, &value_to_store);
                }
            }
        }
    }
}

This direct integration within the runtime ensures that the off-chain index is updated atomically with on-chain state changes. By carefully choosing the data to index and designing an efficient key structure, developers can create highly optimized data access layers for their applications.

Accessing and Utilizing Off-Chain Indexed Data

Data stored via off-chain indexing using local_storage_set can be accessed by Offchain Workers using the offchain::local_storage_get function. This allows OCWs to perform complex queries or aggregations on the indexed data without burdening the on-chain runtime. Additionally, Substrate provides an RPC endpoint, offchain_localStorageGet, which allows external clients (like frontend applications) to query this off-chain data directly, bypassing the need to fetch and process numerous on-chain events.

Continuing the decentralized marketplace example, an Offchain Worker could periodically scan the off-chain index of listings, filter them based on certain criteria (e.g., recently expired), and then potentially submit on-chain transactions to update the status of these listings. A frontend application could use the offchain_localStorageGet RPC to display a paginated and filtered list of active listings to users, providing a much faster and more responsive experience than querying on-chain events directly.

// Example of an Offchain Worker accessing and utilizing off-chain indexed data.
// This code snippet would typically reside within an `impl` block for a pallet
// that has declared an `impl OffchainWorker` trait.

impl<T: Trait> Module<T> {
    fn offchain_worker(block_number: T::BlockNumber) {
        // Use a constant prefix for keys to organize off-chain storage.
        let prefix = b"transfer";

        // Iterate through a range of recent block numbers to retrieve transfer events.
        // This is a simplified example; a more robust approach might involve
        // tracking the last processed block number.
        for bn in block_number.saturating_sub(T::BlockNumber::from(10))..block_number {
            // Iterate through potential event indices within the block.
            // The actual number of events per block can vary.
            for index in 0..100 { // Example: Check up to 100 events per block
                // Construct the key to retrieve the off-chain data.
                let key = (prefix, bn, index).encode();

                // Retrieve the data from the local off-chain storage.
                let result = sp_io::offchain::local_storage_get(StorageKind::PERSISTENT, &key);

                if let Some(encoded) = result {
                    // Decode the retrieved data.
                    if let Ok((from, to, value)) = <(T::AccountId, T::AccountId, T::Currency::Balance)>::decode(&mut &encoded[..]) {
                        // Now you can utilize the retrieved transfer information.
                        // For example, you could:
                        log::info!("OCW: Found off-chain transfer at block {:?}, index {:?}: From {:?}, To {:?}, Value {:?}",
                            bn, index, from, to, value);

                        // Perform some off-chain computation or aggregation with this data.
                        // For instance, you could track the total value transferred to a specific account off-chain.
                        // (Implementation of such logic is beyond the scope of this example).
                    } else {
                        log::warn!("OCW: Failed to decode off-chain transfer data at block {:?}, index {:?}", bn, index);
                    }
                } else {
                    // No data found for this key. Continue to the next potential event.
                }
            }
        }

        // Example of accessing off-chain data via RPC (this wouldn't be inside the OCW itself,
        // but demonstrates how an external client might fetch the data).
        // To enable this, you would need to define custom RPC methods in your runtime API.
        // The client would then call `offchain_localStorageGet` with the appropriate key.
        // For instance:
        // ```
        // // In your custom RPC implementation:
        // pub fn get_offchain_transfer(
        //     _at: Option<<Block as BlockT>::Hash>,
        //     block_number: <Block as BlockT>::BlockNumber,
        //     event_index: u32,
        // ) -> Result<Option<Bytes>, sp_rpc::error::Error> {
        //     let key = ("transfer", block_number, event_index).encode();
        //     let result = sp_io::offchain::local_storage_get(StorageKind::PERSISTENT, &key);
        //     Ok(result.map(|v| v.into()))
        // }
        // ```
        // A frontend could then call this RPC method to retrieve the encoded data.
    }
}

The ability for both Offchain Workers and external clients to efficiently access off-chain indexed data unlocks powerful possibilities for building feature-rich and performant decentralized applications. It allows for complex data presentation, real-time analytics, and efficient background processing without impacting the core blockchain’s performance.

Challenges and Solutions

Challenges

  • Data Consistency: Ensuring the off-chain index remains consistent with the on-chain state is crucial. If the blockchain experiences a rollback, the off-chain index might become out of sync.

  • Scalability of Local Storage: Relying solely on individual node’s local storage might present scalability challenges for very large datasets or high-traffic applications.

  • Query Complexity: While off-chain indexing improves query speed for predefined indexed fields, complex ad-hoc queries might still require more sophisticated off-chain database solutions.

  • Data Availability: Data stored off-chain is not inherently replicated across the network in the same way as on-chain data. Ensuring data availability might require additional mechanisms.

Solutions

  • Event-Driven Updates: Trigger off-chain index updates directly from on-chain events to maintain near real-time consistency.

  • Periodic Reconciliation: Implement mechanisms for Offchain Workers to periodically verify and reconcile the off-chain index with the on-chain state.

  • External Databases: For very large datasets or complex querying needs, consider using Offchain Workers to populate and query external databases (e.g., PostgreSQL, Elasticsearch) based on on-chain events.

  • Redundancy and Backups: Implement strategies for backing up and potentially replicating off-chain indexed data across multiple nodes or using distributed off-chain storage solutions.

Best Practices

  • Index Relevant Data: Only index data that is frequently queried or requires optimized retrieval. Avoid indexing everything off-chain.

  • Design Efficient Key Structures: Choose key structures that allow for fast and targeted lookups.

  • Handle Rollbacks Gracefully: Implement logic to detect and handle potential inconsistencies due to blockchain rollbacks.

  • Monitor Off-Chain Storage: Regularly monitor the storage usage of your off-chain index.

  • Consider Data Privacy: Be mindful of any sensitive data being stored off-chain and implement appropriate security measures.

Looking Ahead

The integration of off-chain indexing with more sophisticated off-chain computation and storage solutions is likely to evolve. We might see more standardized patterns and libraries emerging to simplify the implementation and management of off-chain indexes.

Exploring decentralized off-chain storage solutions (like IPFS or Arweave) in conjunction with off-chain indexing could provide both performance benefits and enhanced data availability and immutability for the indexed data.

Off-chain indexing will become an increasingly essential tool for building scalable and user-friendly decentralized applications on Substrate. As appchains become more data-intensive, the ability to efficiently manage and query data off-chain will be a key differentiator for successful projects.

Conclusion

Implementing off-chain indexing in Substrate offers a powerful way to optimize data retrieval and storage by leveraging the node’s local storage for frequently queried or derived data. By allowing the runtime to directly write indexed information and enabling Offchain Workers and external clients to efficiently access it, developers can build more performant and scalable blockchain applications.

While off-chain indexing introduces its own set of challenges, the benefits in terms of performance and scalability often outweigh the complexities. Understanding and strategically implementing off-chain indexing is a crucial skill for any Substrate developer looking to build robust and user-friendly decentralized solutions.

References