Supabase, the open-source backend-as-a-service provider, has officially released Supabase Evals, a comprehensive open-source benchmark and testing framework designed to evaluate the proficiency of AI coding agents. This move addresses a growing need in the software development industry to move beyond general-purpose large language model (LLM) benchmarks toward specialized, task-oriented evaluations that reflect real-world engineering workflows. By open-sourcing the framework under the Apache-2.0 license, Supabase is providing the developer community with a standardized methodology to measure how effectively autonomous agents—such as Claude Code, Codex, and OpenCode—can navigate complex backend environments, manage databases, and debug cloud infrastructure.
The release of Supabase Evals comes at a pivotal moment in the evolution of artificial intelligence, as the industry shifts from simple chat-based interfaces to "agentic" workflows where AI models are granted the authority to execute code, manage files, and interact with live services. However, assessing the reliability of these agents has historically been difficult due to the lack of specialized testing environments. Supabase Evals fills this gap by subjecting agents to a battery of tests grounded in actual developer experiences, including the construction of database schemas, the debugging of failed Edge Functions, and the rectification of broken Row Level Security (RLS) policies.
The Architecture of the Evaluation Harness
The technical core of Supabase Evals is a sophisticated testing harness designed to simulate a complete development environment. Unlike traditional benchmarks that may only test a model’s ability to generate a code snippet in isolation, Supabase Evals requires agents to operate within a "real" ecosystem. The framework utilizes a Docker-based sandbox to boot a hosted-like stack and a local Command Line Interface (CLI) project in containers. This allows agents to interact with the actual Model Context Protocol (MCP) server and the Supabase CLI, mirroring the exact tools a human developer would use.
To ensure the environment remains lightweight yet functional, the framework incorporates a "platform-lite" runtime. This runtime exposes a surface compatible with the Supabase Management API, backed by the @supabase/lite library. This architecture allows the framework to run locally via pnpm, making it accessible for individual developers and enterprise teams alike to run their own internal regression suites.
The evaluation process follows a structured methodology. Every scenario is defined by a directory containing three primary components: a PROMPT.md file, which outlines the specific task and frontmatter; an EVAL.ts file, which serves as the automated scorer; and optional remote/ and local/ directories that establish the starting state of the project. If an evaluation requires a local workspace or specifies a CLI interface, the framework automatically provisions a Docker container with the necessary tools pre-installed.
Defining the Three Dimensions of Agent Proficiency
To provide a holistic view of an AI agent’s capabilities, Supabase structured the evaluation framework across three distinct dimensions: products, topics, and stages. This multi-dimensional approach ensures that a model is not just good at writing SQL, but also understands how that SQL interacts with authentication systems, storage buckets, and deployment pipelines.
The "Products" dimension encompasses the entire Supabase ecosystem, including the core PostgreSQL database, Auth, Storage, Edge Functions, Realtime, Cron, Queues, Vectors, and the Data API. By covering such a broad range, the benchmark tests whether an agent can maintain context across disparate services.
The "Topics" dimension focuses on specific technical domains that frequently present challenges to developers. These include Row Level Security (RLS), security configurations, database migrations, SQL syntax, SDK integration, observability, self-hosting configurations, automated tests, and declarative schemas. These topics were selected based on an analysis of common friction points found in support tickets, bug reports, and GitHub issues.
Finally, the "Stages" dimension tracks the lifecycle of a development task: Build, Deploy, Investigate, and Resolve. This allows the framework to distinguish between an agent that is good at generating new code (Build) versus one that is skilled at diagnosing a production error (Investigate) or fixing a regression (Resolve).
Analysis of Initial Findings and Model Performance
Initial testing using Supabase Evals has yielded significant data regarding the current state of AI coding agents. The benchmark separates scenarios into two distinct categories: the public "Benchmark" suite, which covers the breadth of the framework and populates the public leaderboard, and an "Internal Regression" suite, which focuses on known failure modes and is used by the Supabase team to monitor daily performance without affecting public scores.
The data suggests that modern AI models are becoming increasingly capable of handling "Build" stage tasks even without specialized tools or "skills" loaded. For instance, both Opus 5 and Kimi K3 achieved a 100% pass rate in the Build stage unaided. However, the introduction of "skills"—specific tools or documentation access provided to the agent—proved to be the deciding factor for other models. Sonnet 5 saw its performance jump from 78% to 100% when equipped with skills. Similarly, GPT-5.6 Sol rose from 89% to 100%, while GPT-5.4 mini improved from 78% to 89%.
Despite these high scores in construction, the evaluations revealed three consistent weaknesses across nearly all tested agents:
- Preference for Imperative Migrations: Agents frequently default to hand-writing migration files rather than utilizing modern declarative database schemas. This discovery has already led Supabase to update its "skill guidance" to encourage agents to adopt more sustainable, declarative workflows.
- Manual Auth Verification: Instead of utilizing the streamlined
@supabase/serverpackage, many agents attempted to verify authentication manually by writing complex, error-prone logic. This prompted the creation of a new package selection guide to better inform both human and AI developers. - Variable Documentation Usage: The framework tracked how often agents consulted official documentation. The results showed a sharp divide in behavior. Codex and GPT-5.6 were found to read an average of eight documentation pages per scenario. In contrast, Claude Code checked documentation in fewer than 40% of scenarios, averaging only two pages, yet maintained high accuracy, suggesting differences in pre-training depth or internal reasoning strategies.
Chronology of Development and Strategic Intent
The development of Supabase Evals was driven by the company’s internal need to ensure that the AI tools they were building for their users were actually effective. The project began as a series of manual tests derived from real-world support interactions. Over time, these tests were formalized into the current automated harness.
By open-sourcing the tool, Supabase is attempting to set a standard for "agentic" benchmarks. Most existing benchmarks, like HumanEval or MBPP (Mostly Basic Python Problems), focus on small, isolated functions. Supabase Evals represents a shift toward "System-Level Evaluations," where the success of the agent is measured by the final state of a running system rather than just the text of the code it produced.
The framework uses a hybrid scoring mechanism: deterministic checks (e.g., "Does the database table exist?" or "Does the API return a 200?") combined with "LLM-as-a-judge" for more nuanced assessments. To prevent noise from transient errors, agents are typically granted one retry before a final grade is assigned.
Industry Implications and Future Outlook
The release of Supabase Evals is expected to have a ripple effect across the developer tools industry. As more companies move toward providing "AI-first" experiences, the ability to prove the reliability of these agents becomes a competitive advantage. The public leaderboard at supabase.com/evals provides a transparent, data-driven way for developers to choose which AI models to integrate into their workflows.
Furthermore, the decision to use the Apache-2.0 license ensures that other platform providers—whether they offer database services, cloud infrastructure, or frontend frameworks—can fork the repository and adapt the harness for their own specialized environments. This could lead to a fragmented but highly accurate landscape of "Domain-Specific Evals," providing a much clearer picture of AI capabilities than general benchmarks currently offer.
For Supabase, this framework is more than just a public relations tool; it is a critical piece of their infrastructure. By running a daily regression suite against their own internal tools, they can ensure that updates to their CLI or API do not break the "reasoning" of the AI agents that their customers rely on. As AI agents become the primary interface for many developers, maintaining "AI-compatibility" is becoming as important as maintaining backward compatibility in traditional software.
The findings regarding documentation usage and the tendency of agents to ignore declarative schemas also highlight a new challenge for technical writers and developer advocates: "LLM Optimization." As agents become the primary consumers of documentation, the way information is structured may need to change to ensure that AI models can quickly find and implement the "best practice" path rather than the "path of least resistance."
In the long term, Supabase Evals represents a commitment to the "Agentic Workflow" as a permanent fixture of modern software engineering. By providing the tools to measure, debug, and improve these agents, Supabase is helping to transition AI from a experimental coding assistant to a reliable, autonomous collaborator in the cloud development process.
