Request for funds: Beefy Data Project - Phase 1 - Infra

Funds requested: $15,000

Beefy currently runs revenue based analysis within Google Sheets and uses a semi-automated process for updating revenue metrics. Input data is gathered from the Beefy API endpoints via Google Apps Script into the Google Sheets, processed using formulae to create outputs, which are then pulled into a PowerBI setup. This architecture is not sustainable as a solution; we will run out of room in the sheets, we cannot scale user access whilst preserving the integrity of the data; it is very difficult to debug; the process is not transparent; output tools must be able to pull from Google Sheets.

TL;DR

This project proposes to move the Beefy Data infrastructure forward by using code backed and scalable data services. We will align our architecture with the data that backs the Beefy v2 app to simplify and avoid duplicate sources of truth. We will create new metrics for a future token terminal integration.

Detail

The project lead is tumbledyer, from the discord server.

There are 2 named developers on the project. They will be tumbledyer and MrExchangee, both from the discord server. The project outputs are valued at $15,000 and the split between the developers will be made based on value of contribution through the project after delivery. They will be paid on the completion of the following key deliverables.

There are 6 key deliverables:

Raw data history storage

We will store off the history of /vaults, /apy/breakdown, /tvl, /prices, /lps raw JSON, so that we can always update our data logic in the future and have this reflect through history.

Single public Beefy database

We will provision a database to store the data after processing and then use this to automate the transformations needed for revenue metrics. There is also currently a PostgreSQL database backing the v2 app within beefy-db-dev. This was planned as a temporary solution for pulling history. We would like to have a single database setup for storing all our important Beefy data.

To enable this, we will perform a database migration as part of this project, exposing the current v2 format of the data as well as a new format for the v2 app to move onto in whatever timeline they like to. We will not decommission the old setup until the v2 app has moved across, of course.

Single source of truth for revenue metrics

The revenue metrics are currently created within a Google Sheets and this architecture is difficult to debug and not a sustainable architecture. We will move to a code backed set of revenue logic that uses SQL, the most common data language and Git for audit and change trail plus safe collaboration.

Current data reporting backed by sustainable architecture

We will provision the data in the same format as Google Sheets, but using sustainable architecture. We will move the current PowerBI reporting to point to this new location. We will then be able to think about moving out of PowerBI easily down the line. A change of reporting / BI tool(s) is not scoped in.

Provision outputs ready for token terminal

Token terminal is a well known and trusted web tool for tracking defi projects. It has been requested and discussed multiple times in this server. We will look at the token terminal set of metrics available and provision these for Beefy, as outputs in our database and publicly available. We are still trying to get in contact with token terminal via various methods (email, Twitter, via partners already on the platform). Until we know the full details on integrating with them, we cannot scope the full integration.

Infrastructure as code

We will build the infrastructure in Google Cloud Platform, as that is where existing architecture for data work lives. We will use Terraform, meaning that all of our infrastructure is managed by code, rather than human clicks in a console. Terraform is an industry standard, open source tool for infrastructure as code. This will then mean we can manage our architecture ourselves, via GitHub. Currently this will live in beefy-data-dev. This setup will also allow us to have as many users managing the infrastructure as we like, with no extra costs.

Full Timeline

All work will be tracked in the Beefy Height application.

The planned delivery for the project is published here and will be updated weekly with progress. It will also be exported into the #beefy-data channel on the discord server for any interested stakeholders.