How to combine sensitive on-premise portfolio data with third-party financial intelligence without compromising security or compliance
by Meghana Padmanabhan, Isaac Gritz and Max Nienu
In today's enterprise landscape, data rarely sits in one place. Financial institutions, asset managers, and individual investors face a common challenge: their most sensitive data including portfolio holdings, trading positions, and client information often resides in on-premises databases behind corporate firewalls, in the cloud, or across both environments. That challenge is compounded by the fact that the third-party data they need for analysis, including market data, analyst estimates, and financial fundamentals increasingly lives in the cloud.
Traditional approaches force an uncomfortable choice: copy sensitive data to the cloud, stand up brittle ETL pipelines, or give up on integrating the two altogether.
What if you could just federate the on-premises source, subscribe to the third-party data on the cloud, and go straight from those two catalogs to a dashboard and a natural-language chat, with no pipeline, no materialized views, and no persisted objects in between?
That's Databricks Lakehouse Federation at its simplest.
Databricks Lakehouse Federation lets you create federated connections to external databases like SQL Server, PostgreSQL, MySQL, Snowflake, and more, and query them directly from Databricks using standard SQL. The data stays exactly where it is. No copying. No syncing. No compliance headaches.
Add a marketplace data share on top of it, and you have two catalogs sitting side by side in Unity Catalog - one federated, one shared, that a plain SQL query can join like they were any two ordinary tables.
Let's walk through a concrete example. Imagine you're a wealth management firm with:
Your goal: Build an investment dashboard that shows your projected earnings, identifies high-risk positions, and recommends actions, all without exposing your holdings data to the cloud.
Setting up federation is straightforward through the Databricks UI. Navigate to Catalog → External Data → Connections and create a new connection to your on-premises database.
The connection securely stores your credentials.
Once the connection is established, create a foreign catalog that maps to your on-premises database. This makes your external tables appear as native Unity Catalog objects.
After this step, your on-premises data appears as a catalog, but the data never leaves your SQL Server. Every query executes against the source in real time.
The Databricks Marketplace provides instant access to FactSet's comprehensive financial data. Search for FactSet, review the available datasets, and request access. Once the provider approves your request, you can subscribe to the relevant datasets.
After subscribing, the data is available as a shared catalog in your workspace, organized into schemas:
These tables appear alongside your federated portfolio catalog in Unity Catalog. You can browse them, preview data, and check schemas, all without any ingestion or data copying.
With both data sources accessible through Unity Catalog, an asset manager can open the SQL editor and write one query that spans both the federated holdings on one side, the FactSet marketplace tables on the other. To the SQL author they look like two ordinary tables in the same warehouse.
There is no pipeline to author, no materialized view to schedule, and no gold layer to maintain. Every time the query runs it re-federates the on-premises source live and re-joins it against the marketplace share.
The dashboard is powered by the same combined query inlined directly into each widget's dataset. No views are created and no tables are persisted. Every widget re-federates the source on each refresh.
It instantly answers questions like:
The final piece: a Genie agent wired directly to the same federated holdings and FactSet marketplace tables. Genie receives the join chain and data-quality filters in its instructions. It translates natural language into the same queries the dashboard runs.
The result: democratized data access. Portfolio managers, financial advisors, and individual investors can all get answers without writing SQL, and without anyone materializing anything.
Databricks Lakehouse Federation fundamentally changes the economics of hybrid data analytics. Instead of the expensive, risky, and time-consuming process of moving data to the cloud, or the equally burdensome process of building and maintaining pipelines on top of it, you can bring the compute to the data and let a single query drive everything downstream.
In this example, the walkthrough built a complete investment analytics solution that:
All without copying a single row of portfolio data to the cloud.
The future of data analytics isn't about moving all your data to one place and it isn't about building elaborate pipelines to reshape it either. It's about querying data wherever it lives. While this blog used investment portfolios as the example, this pattern – federate sensitive first-party data, subscribe to cloud-based third-party data, and analyze without moving or copying anything – applies broadly. Healthcare organizations can join HIPAA-protected patient records with drug interaction databases for treatment recommendations.Manufacturing firms can federate on-premises IoT telemetry with cloud-based supply chain data. Retailers can combine in-store POS systems with third-party demographic datasets.The common thread: sensitive first-party data stays protected while enabling rich analytics with third-party data.
Ready to get started? Explore the following links:
Solutions, Tutorials, Financial Services, Insights and Customers
Subscribe to our blog and get the latest posts delivered to your inbox.