Bring your docs, tickets, emails, and AI tools into a single, governed foundation with Lakeflow Connect to fuel AI agents and Genie analytics
by Sonia Bendre and Giselle Goicochea
This is the second post in our series on how Lakeflow Connect brings fully managed data ingestion to every line of business. Following our deep dive into unifying marketing data, this post focuses on unlocking the enterprise knowledge trapped within your organization's productivity stack. We’ll explore how native connectors for wikis, project management tools, and AI foundation models give your downstream AI agents the complete business context they need.
Lakeflow Connect provides a comprehensive suite of fully managed native connectors across SaaS applications, databases, and file sources that ingest your data directly into the Databricks Platform. Each connector can be set up via a point-and-click UI or a simple API. From there, your data is incrementally ingested as governed, managed tables in Unity Catalog, with no infrastructure for you to provision, manage, or scale.
Prioritized by customer demand, Lakeflow Connect's ingestion connectors span every line of business, including Customer 360, Operations, Finance, and Productivity—the stack closest to day-to-day work.
The average company's productivity stack is a sprawl of project trackers, wikis, document stores, inboxes, and AI tools, each holding only a slice of how work gets done. Getting that data out has historically required a patchwork of custom pipelines that are slow to build and painful to maintain. The bar is high enough that the most impactful use cases stay out of reach, for example: AI agents powered by your enterprise's internal knowledge or visibility into operational analytics.
Lakeflow Connect resolves this by providing built-in connectors to ingest seamlessly from across the productivity stack:
With this breadth of connectors, Lakeflow Connect becomes the ingestion engine that brings scattered productivity data into one governed foundation, without the integration bottlenecks that traditionally slow teams down.
For example, Toyota Finance Australia, the financing arm of one of the world's largest automakers, uses Lakeflow Connect to bring its Jira data into Databricks as a fully managed, zero-maintenance pipeline.
“Lakeflow Connect for Jira has transformed how we onboard and consume Jira data within our enterprise data management platform. Moving to the fully managed, UI-driven experience offered by Databricks provided a seamless, zero-maintenance pipeline that automatically handles API complexity and schema changes while delivering governed, analytics-ready Delta tables in the Databricks Lakehouse with minimal effort and maximum reliability.”—Charbel Khoury, Head of Data Office, Toyota Finance Australia Limited
Another customer, VaxCare, a vaccine management company runs the largest primary-care vaccination network in the U.S., and uses Lakeflow Connect's Confluence connector to feed its internal knowledge base into Databricks and ground an AI agent in it.
"Lakeflow Connect's Confluence connector gives us a seamless way to feed our internal knowledge base into Databricks, making that data searchable and actionable for our chatbot today—and ready to power future AI solutions as we build them." —Yasaman Malone, VP of Optimization and AI, VaxCare
AI agents can provide instant, grounded answers from your company's collective knowledge without employees having to hunt across wikis, drives, and inboxes. But an agent is only as good as the data behind it, and most of that is unstructured. With Lakeflow Connect, this is not a problem; our connectors can land unstructured content like files, pages, and messages as governed tables in Unity Catalog. The exact data, and the format it arrives in, varies by source:
content
column (either as BINARY
or FILE
type), with per-file metadata stored alongside. Structured files (Excel files, CSVs, etc.) can also be landed directly as Delta tables. content
column. Structured files (Google Sheets, CSVs, etc.) can also be landed directly as Delta tables. STRUCT
type in a body
field of the pages
and blogposts
tables. full_body_content
column as a text string (either HTML or plain text), and it stores the attachments in an attachments
column as BINARY
type. Both live in the email_messages
table.messages
table. It also ingests a reference to each attachment as an attachmentId
.content
column of the repo_contents
table.Whatever the source, the content lands in Unity Catalog as a governed table, where you can begin to transform it for downstream AI use cases.
ai_parse_document
, Databricks' built-in document intelligence.Either way you end up with text you can index for retrieval and put behind an agent. For a deeper look at document processing, watch this on-demand session, Intelligent Document Processing with Lakeflow; and for more on how to get started, read the downstream RAG guide.
M3 Accounting, a leading provider of hospitality software, uses Lakeflow Connect's GitHub and SharePoint connectors to unify its engineering and product documentation into Databricks.
"We had the Lakeflow Connect GitHub connector up and running in minutes. It's saving us about an hour a week by automatically blending our GitHub data with our other engineering operations data, giving us a full picture of developer activity we can actually act on.We pointed the Lakeflow Connect SharePoint connector at our product documentation library and had our internal knowledge assistant pulling from live, up-to-date content within a week. No custom crawlers, no file sync scripts, just a connector and a pipeline." —Ross Serven, VP of Data & AI, M3 Accounting
As AI agents and generative tools spread across teams, so do the questions about the models powering them: what are we spending, who's using what, is it compliant? To help answer these, Lakeflow Connect provides connectors for the biggest foundation model providers:
Concretely, this enables a single, unified view of how AI usage and adoption are trending within your organiztion, so you can measure and report the ROI of your AI investments to stakeholders.
Lakeflow Connect’s depth of ecosystem coverage also extends to Atlassian—one of the most widely used suites in any engineering or product org:
With both connectors, you get a complete picture of your Atlassian data, the delivery and the documentation that surrounds it, side by side. From there you can track launches, bottlenecks, and resolution times across projects. You can also build AI/BI dashboards, and ask Genie about any of it in natural language.
For example, Digicert, a global leader in digital security, uses Lakeflow Connect’s Confluence connector to reliably power its internal AI Knowledge Assistants without the overhead of custom pipelines.
"Lakeflow Connect's Confluence connector has been a great help in keeping our Knowledge Assistants in Databricks continuously in sync with our latest documentation. It gives us a simple, reliable way to bring Confluence data into Databricks without the overhead of building and maintaining custom pipelines. This has been especially valuable for our Product Bots, helping them stay current and provide our teams with accurate, up-to-date knowledge." —Nikita Raje, Senior Director, Data Engineering, DigiCert
Another customer, Orizon, a Brazilian healthcare technology company, uses Lakeflow Connect's Jira connector to feed its leadership dashboards.
"With the Lakeflow Connect Jira connector, we brought over one million tickets from 168 projects into Databricks, all synchronized daily with 100% success. From this data, we built AI/BI dashboards that automatically feed our Steering Committee's cockpit. Genie has been fundamental: we ask questions in natural language and it delivers answers instantly." —Gustavo Lima, Data Engineering Manager, Orizon
For a step-by-step setup and tuning guide for the Jira connector, see Unify Your Engineering Data: Jira Ingestion Best Practices. Once you’ve ingested data from Jira, you can also leverage Jira-Genie solution accelerator to transform bronze tables into an out-of-the-box AI/BI dashboard and Genie agent, backed by governed metric views in Unity Catalog in just a few steps.
Because every connector lands its data as governed Delta tables in Unity Catalog, the productivity and AI data you ingest becomes part of the same foundation as the rest of your enterprise data. From there you can query it directly in Databricks SQL alongside your other sources, ask questions of it in natural language with AI/BI Genie, and ground the agents you build with Agent Bricks in current company knowledge.
And because governance, lineage, and access control all come from Unity Catalog, access stays consistent regardless of the team or tool from where the data originally came.
TRegs, a financial technology platform for treasury, risk, and regulatory reporting, is already building on that foundation.
"We needed to see all our costs and projections in one place. By bringing our Monday.com and QuickBooks data into Databricks with Lakeflow Connect, we can now power our Genie AI/BI dashboards to get a full 360-degree view of our current and projected expenses, making our lives super easy." —Smriti Raj, President and CTO, TRegs
Lakeflow Connect is actively rolling out new connectors across every line of business, with support for even more connectors coming soon! Try Lakeflow Connect for free now.
Watch for the next post in this Lakeflow Connect blog series to keep learning how to bring all your enterprise data into Databricks and give your AI agents the full context they need.
Subscribe to our blog and get the latest posts delivered to your inbox.