Effect of a Grant Program on Developer Activity

Key Points

On average, there was an observed increase of approximately 150 to 200 monthly active developers.

Background

• Open Source Observer (OSO) is exploring advanced metrics to better measure the impact of certain types of interventions on public goods ecosystems.
• For example, it aims to compare the performance of projects or users who received token incentives against those who did not.
• However, in real-world economies, it is impossible to randomly assign treatment and control groups like in controlled A/B tests.
• Therefore, advanced statistical techniques must be employed to estimate the causal effect of a treatment on a target cohort, while controlling for other factors such as market conditions, competing incentives, and geopolitical events.

Analysis Method

Dataset

• The synthetic control work is part of a broader initiative to build a flexible analytics engine capable of analyzing virtually all metrics over time.
• OSO is currently rolling out a suite of timeseries metrics. These models allow metrics to be computed for any cohort over any time period, enabling "time travel" to evaluate past performance.
• Most timeseries metrics are computed using a rolling window with daily buckets. For example, rather than measuring monthly active developers as a static monthly count, OSO uses 30-day and 90-day rolling windows to provide a more detailed view of cohort performance.
• Sample SQL queries show the use of tables such as timeseries_metrics_by_collection_v0, metrics_v0, and collections_v1.

Intervation / Explanatory Variable

• OSO is interested in measuring the impact of specific types of interventions on public goods ecosystems.
• Specifically, it seeks to assess how grants and incentives affect outcomes such as developer retention, user activity, and network TVL (Total Value Locked).
• The initial experiment evaluates an intervention targeting a cohort of projects that received Optimism Retro Funding in January 2024.
• The explanatory variable is new_contributors_over_90_day (new contributors over 90 days), commits_over_90_day (commits over 90 days), and issues_opened_over_90_day (issues opened over 90 days).

Dependent Variable

• The dependent variable is active_developers_over_90_day (90-day active developers)

Identification Strategy

• As an early experiment in crypto network economics, OSO is exploring methods such as synthetic controls and causal inference.
• Synthetic control methods are widely used to assess the impact of interventions in complex systems.
• In this approach, a synthetic control is a weighted average of several units, constructed to replicate the trajectory that the treated unit would have followed in the absence of the intervention.
• Weights are selected in a data-driven way so that the resulting synthetic control closely resembles the treated unit with respect to key predictors of the outcome variable.
• Unlike difference-in-differences approaches, this method allows for adjustments for time-varying confounders by weighting the control group to better match the treatment group in the pre-intervention period.
• Economists frequently use synthetic controls to evaluate policy impacts in non-experimental settings (e.g., Abadie and Gardeazabal’s study on the economic impact of the Basque separatist conflict).
• One key advantage of synthetic controls is the ability to systematically select comparison groups. In OSO’s case, this means comparing grant recipients to similar non-recipient projects.
• Inspired by work from Counterfactual Labs, OSO uses the pysyncon package to estimate treatment effects across the range of timeseries metrics available within OSO.
• Each analysis request includes a pre-period start and end date, an optimization period start and end date, a dependent variable, treatment identifier, control identifiers, and predictor variables.

Results

• In early findings, OSO analyzed monthly active developers over a 90-day rolling window for a cohort of projects that received Optimism Retro Funding in January 2024.
• The results indicate that the gap between the treated group and the synthetic control group reflects the treatment effect, with an average increase of approximately 150 to 200 monthly active developers.
• OSO is in the early stages of applying advanced metrics like synthetic control to measure the impact of incentives in crypto networks and plans to share further insights in the future.