Technology1 min read

Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

By · Published by Everything Blog

In short

The new continuous execution model processes data up to 100 times faster than legacy micro-batch systems and outperforms Apache Flink by up to 92 percent in benchmarks. Organizations can leverage serverless deployments to eliminate manual infrastructure sizing while paying solely for active cores. The architecture pushes data directly into operational stores such as Redis and Lakebase, demonstrated by the platform's Feature Store achieving a 200-millisecond p99 latency from Kafka.

Key points

  • Unifying batch ETL and low-latency streaming within a single framework removes the overhe…: Unifying batch ETL and low-latency streaming within a single framework removes the overhead of maintaining duplicate systems.
  • Deploying the technology on serverless compute removes manual infrastructure tuning and c…: Deploying the technology on serverless compute removes manual infrastructure tuning and charges users only for consumed cores.
  • Direct integration with transactional sinks like Redis and Lakebase bypasses the need for…: Direct integration with transactional sinks like Redis and Lakebase bypasses the need for offline-to-online synchronization pipelines.

Sources checked by our analysis pipeline: 1

Bring sub-second latency to your most time-critical operational workloads with declarative flows and sinks

by Jerry Peng, Abhay Bothra, Sudhanva Huruli, Navneeth Nair, Anant Pingle and Murali Talluri

We introduced Real-Time Mode (RTM) last year which enabled customers to build low-latency, operational use cases. This was a game changer because it allowed customers to process streaming data in milliseconds, up to 100x faster than traditional micro-batch architecture. Furthermore, they no longer needed to maintain separate, specialized engines like Flink alongside Spark, eliminating duplicate codebases, fragmented governance models, and operational overhead.

Today, we are announcing RTM in Spark Declarative Pipelines (SDP) is now Genearally Available. SDP has become the standard way to build streaming and ETL pipelines on Databricks, automatically handling dependency resolution, incremental processing, orchestration, failure recovery, and data quality enforcement. With RTM, customers can build operational, low-latency pipelines while retaining all of those SDP benefits. Whether for fraud scoring, real-time alerting, or pushing fresh features to operational stores, you can now use SDP across all of these use cases.

RTM is an evolution of the Spark Structured Streaming engine…

Technology

More in Technology