Block Crawler Dataset Configuration

Username

Block Crawler // ZettaFi Labs

Grant Description

We built an open source high-performance ETL framework for blockchain data designed to process blockchain data and transform it into useful datasets in the easiest way possible. We will configure it for free to collect any dataset for you.

Overview

Block Crawler creates datasets from blockchains and provides it to those who are building applications. Its high-performance, open-source ETL framework is designed to process massive amounts of blockchain data and transform it into useful datasets in the easiest and most expedient way possible. With Block Crawler, anyone can extract any blockchain data from EVM RPC APIs like Ethereum, Polygon, and more, transform it into desired datasets, and persist it into databases to support data access requests on historical and real-time data. Furthermore, the Block Crawler's modular components can be combined with dataset specific components to cater to the unique requirements of every user.

Whether a blockchain looking to improve data availability or a blockchain application needing a specific dataset, we are creating a product where data consumers tell us what data they need, and we configure the Block Crawler for them and work with them to extract, transform, and load the data.

Value to Synthetix

The Block Crawler will allow you to extract any dataset from any blockchain you might find valuable now and into the future. This will ensure you have access to the data you need to display the complex analytics dashboards, make important internal business decisions and analyze on chain behavior. We can load any historical dataset you want into a database to perform data science as well as loading real-time data into a database you can query with an API.

Project Implementation Plan

We are hoping to accomplish the goal of improving accessibility to valuable datasets. We would define success for this grant to be extracting, transforming and loading valuable datasets from any blockchain into a database of your choice. The scope of the grant would depend on the volume and complexity of the datasets requested. We would consider this project successful if we improve data accessibility in the blockchain industry. While open-source data extraction tools for overly generic or specific niches are emerging, there is currently no high-performance framework for creating and maintaining robust datasets from Blockchain data.

The initial consultation and configuration of Block Crawler is free. We will work with your team to identify important datasets, configure the Block Crawler for you to extract those datasets and assist you with loading those datasets into the database of your choice. After this initial milestone, the long-term engagement would depend highly on the project's scope. The volume and complexity of the data extraction would impact the cost and future milestones.

Additional Information

We are currently doing market research into what datasets are difficult to extract from blockchains and are considered valuable. Our open source software is capable of being configured to extract, transform and load any dataset but we are still unsure about which datasets the industry considers most important to extract. In addition to financial support, or even instead of financial support, we would appreciate the ability to ask some questions about your data needs and any pain points you have experienced when dealing with blockchain data.

Funding Request

We are offering the initial configuration and consultation for free and the software is open source. After the free initial engagement, we would be looking for anywhere from $10k to $50k depending on the scope of the ongoing work.