Implementing this architecture requires providers to publish source tables to a share using metastore-level privileges, optionally enabling Change Data Feed for downstream tracking. Recipients then register the share, mount it as a catalog, and use a checkpointed Structured Streaming reader to process incoming updates. Features like the AvailableNow trigger allow recipients to ingest newly committed data in discrete batches before terminating execution. Furthermore, Databricks Runtime 19 and above automatically computes Change Data Feeds for tables with row tracking enabled, capturing inserts, updates, and deletes seamlessly.
Key points
Direct sharing eliminates the operational overhead of managing SFTP endpoints and custom…: Direct sharing eliminates the operational overhead of managing SFTP endpoints and custom data export jobs.
Structured Streaming readers with checkpoints automatically track unprocessed changes wit…: Structured Streaming readers with checkpoints automatically track unprocessed changes without copying data to recipient storage.
Shared table history allows external partners to consume both live data queries and granu…: Shared table history allows external partners to consume both live data queries and granular record modifications.
Sources checked by our analysis pipeline: 1
What the recipient actually does with a shared table
by Nikhil Tallam and Jordan Soldo
Many data-sharing architectures begin with a familiar pattern:
Part 1 of this blog described why a shared table can be a better alternative to a recurring file or API feed. Now let's look at what the recipient actually does with that table.
This walkthrough focuses on Databricks-to-Databricks (D2D) sharing. The recipient has a Unity Catalog enabled Databricks workspace and accesses the shared table through a shared catalog. The provider shares the table with its history, enabling the recipient to use it as a Structured Streaming source. We'll come back to Databricks-to-Open (D2O) sharing later in the post.
A Delta table keeps an ordered history of every change made to it. OpenSharing exposes that table over an open protocol without copying it. The data stays in the provider's storage, and the recipient reads it live.
On the recipient side, you point a Structured Streaming reader at the shared table. Spark records how far it has read in a checkpoint, so each run picks up only the…
Software agents can now construct dynamic rich cards across messaging ecosystems by outputting a JSON specification mapped to a typed component catalog. Individual adapters convert these structured definitions into native layouts like Slack Block Kit and Microsoft Teams Adaptive Cards, saving developers from maintaining custom tooling for each platform. This framework also introduces automated validation to prevent content degradation caused by downstream messaging size limits.
Modern operational environments require a coordinated mix of specialized components to handle large deployments effectively. Systems rely on agents and automated tooling to coordinate with infrastructure-as-code frameworks. This operational structure provides the foundation necessary for maintaining consistency across enterprise environments.
Organizations frequently encounter compounding operational friction, duplicate storage requirements, and data drift when scaling traditional pipeline connections like SFTP drops and APIs. By leveraging open table formats like Iceberg and Delta Lake, modern architectures can query vendor-hosted tables directly at rest. This transition eliminates duplicate storage footprints and grants consumers instantaneous access to updates the moment providers write them.