Retro Funding S7 – Dev Tooling – Eval Algos

Hey everyone, Carl from Open Source Observer here, sharing the evaluation algorithms for Retroactive Public Goods Funding (Retro Funding) Season 7: Dev Tooling.

The Dev Tooling mission seeks to reward projects based on the utility they provide to builders in developing onchain applications. See the here for more on the mission’s objectives.

This season, Optimism’s Retro Funding is pioneering a "metrics-driven, humans-in-the-loop" approach. Instead of voting on projects, citizens vote on evaluation algorithms. Each algorithm represents a different strategy for rewarding projects. Algorithms will evolve in response to voting, community feedback, and new ideas over the course of the season.

Here’s a tl;dr of the three algorithms:

|Algorithm|Goal|Best For|Emphasis| | --- | --- | --- | --- | |Arcturus|Reward widely adopted projects|Established, high-impact devtooling projects|Prioritizes total dependents and the economic weight of those dependents| |Bellatrix|Prioritize fast-growing tools|New or rapidly expanding devtooling projects|Applies a steep decay factor on older forms of GitHub engagement, favors Rust over npm| |Canopus|Balance various forms of contribution|Tools with high developer collaboration & contributions|Puts stronger emphasis on GitHub engagement and developer reputation|

Metrics Overview

The algorithms all rely on a value chain graph linking dev tools to onchain builders and use OpenRank’s EigenTrust implementation to distribute trust between projects iteratively.

Nodes

There are three types of core nodes in the graph:

• Onchain Projects, which hold economic pretrust and pass it on.
• Developers, who gain partial trust from the onchain projects they contribute to.
• Devtooling Projects, which receive trust both directly from onchain projects, via package dependencies, and indirectly from developers’ GitHub engagement.

Edges

Links between nodes, i.e., edges, are derived from the following relationships:

1. Onchain Projects → Devtooling Projects (Package Dependencies). For each package in the onchain project’s Software Bill of Materials (SBOM), if that package is owned by a devtooling project, we add an edge from the onchain project to the devtooling project.
2. Onchain Projects → Developers (Commits). Whenever a developer commits code (or merges PRs) in an onchain project’s GitHub repository, we add an edge from the onchain project to that developer.
3. Developers → Devtooling Projects (GitHub Engagement). Whenever a developer (from the set recognized above) engages with a devtooling project (commits, PRs, issues, forks, stars, or comments), we add an edge from that developer to the devtooling project.

Pre-trust Assumptions

Metrics about projects and developers are used as a way of seeding the EigenTrust algorithm with pretrust assumptions. All other things being equal, edges that involve nodes with higher pretrust values will be weighted more heavily in the EigenTrust algorithm.

Pretrust metrics are applied to each node in the graph:

1. Onchain Projects. Transaction volumes, gas fees, and user counts over the past 180 days.
2. Devtooling Projects. Count of published packages and GitHub metrics (stars, forks).
3. Developer Reputation. Active months contributing to one or more onchain projects.

Weighting Assumptions

After constructing the graph and computing pretrust scores, we assign final weights to each edge before EigenTrust runs:

1. Alpha: the portion of the EigenTrust output taken from the pretrust values vs. the iterative trust propagation step.
2. Time Decay: an exponential decay factor so that newer events get weighted more heavily than older events.
3. Link and Event Types: coefficients for weighting certain types of links more heavily (e.g., Rust imports more heavily than npm imports).

Each algorithm has its own set of weighting assumptions.

Proposed Algorithms

We’ve developed an initial three candidate algorithms for evaluating impact. Each algorithm highlights different priorities: Arcturus focuses on established adoption, Bellatrix puts more emphasis on emerging toolchains, and Canopus aims to look more closely at the quality of the developer community around a project.

We’ve included details on each algorithm below. We also have a Hex dashboard where you can see data on each algorithm.

Important: as projects are still signing up for the current measurement period, we simulated the results for each algorithm using past Retro Funding participants and historical data. The “top projects” below are based on onchain builders applications from RF4 and devtooling projects from RF3.

Arcturus

Arcturus rewards projects with significant current adoption by focusing on total dependents and downstream impact. It emphasizes the number of developers and onchain users benefiting from these tools, ensuring that cornerstone infrastructure is properly recognized. By applying only a modest discount to older events, Arcturus is best suited for rewarding established projects that many builders rely on.

Top projects (using simulated data):

1. Ethers.js
2. OpenZeppelin
3. wevm
4. Hardhat
5. Prettier Solidity

Bellatrix

Bellatrix is designed to spotlight projects and toolchains that are gaining in momentum. It applies a steep decay factor on older GitHub engagements and favors modern tooling preferences, notably prioritizing Rust over npm packages. This approach makes Bellatrix ideal for giving an edge to devtooling projects that are on the rise.

Top projects (using simulated data):

1. wevm
2. Ethers.js
3. OpenZeppelin
4. Hardhat
5. Foundry

Canopus

Canopus recognizes projects that have large active developer communities. It considers package dependencies alongside developer interactions, but puts extra weight on key GitHub engagement signals – such as commits, pull requests, and issue discussions. The result is a more balanced distribution of rewards across a broader variety of projects (not just the very biggest or newest). This approach can be a bit more susceptible to noise, since it values a wider range of contribution types, but it aims to ensure broader open source contributions are well-represented.

Top projects (using simulated data):

1. wevm
2. Foundry
3. Ethers.js
4. OpenZeppelin
5. DefiLlama

How to Vote

One of the proposed evaluation algorithm for the Dev Tooling mission will be chosen using approval voting. This means you can approve one or multiple options. For an algorithm to be selected, it must meet the following conditions:

• Quorum**:** At least 30% (44) of badgeholders must participate in the vote.
• Approval Threshold**:** An algorithm must receive approval from at least 51% of quorum (22 badgeholders)

The algorithm with the most approvals, provided it meets the thresholds, will be used for the remainder of Season 7.

To learn more about this mission, please see the Dev Tooling Mission Details. We also have more in-depth documentation on these algorithms in the Retro Funding repo.

If you have ideas on refining these models, let us know in the comments. And, if you want to contribute to making these algorithms better, submit a PR or open an issue here.