Artificial Intelligence1 min read

Genie ZeroOps, now in public beta: An autonomous agent that fixes data pipelines, with lessons from 80,000 customer issues

By · Published by Everything Blog

In short

The tool actively monitors data infrastructure by using direct access to run history, source code, metadata, and Unity Catalog lineage within established workspace permissions. When issues like delayed table commits or low row counts appear, the agent formulates a solution and pushes the suggested fix to a separate review branch. New workflow capabilities such as custom agent instructions, notifications, and shared inboxes were added after testing showed that most incidents rely on enterprise-specific context. Human oversight remains mandatory, ensuring that neither production code nor live data is modified without explicit approval.

Key points

  • Automated investigation triggers immediately when disruptions like schema changes or abno…: Automated investigation triggers immediately when disruptions like schema changes or abnormal row counts are identified.
  • Enterprise teams retain full control because the agent cannot alter production code or da…: Enterprise teams retain full control because the agent cannot alter production code or data without human approval.
  • Workflow integrations like one-click pull requests and shared inboxes help teams resolve…: Workflow integrations like one-click pull requests and shared inboxes help teams resolve pipeline incidents using organization-specific rules.

Sources checked by our analysis pipeline: 1

Over half of the pipeline issues we saw needed company-specific knowledge to fix. So we built shared inboxes, custom agent instructions, and one-click PRs to help data teams triage faster.

by Jeff Chou, Kristóf Molnár, Lennart Kats, Danny Chiao and Ray Zhu

In June we introduced Genie ZeroOps, an autonomous agent that detects, investigates, and resolves data pipeline issues, such as upstream schema changes or data quality anomalies. Based on extensive private preview testing with hundreds of customers, we’ve added new features to help data teams triage issues faster - including inbox sharing, notifications, one-click PR creation, and the ability to add custom business instructions to follow specific company workflows. Today, Genie ZeroOps is available in Public Beta, and your workspace admin can turn it on from the workspace Previews page.

Data engineering teams are underwater, managing many thousands of pipelines a day. Some pipeline failures are easy to find and fix - a job fails, a pipeline update errors out, and someone gets paged. Others are silent - a table stops updating, or last night's load arrives with half…

Artificial Intelligence