TechnologyThursday, November 5, 20261 min read

Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

By Engineering Blogs · Published by Everything Blog

In short

The update introduces native concurrent stage scheduling and pipelined streaming shuffles, replacing blocking stages so producer and consumer processes can execute simultaneously. These low-latency optimizations delivered sub-150 millisecond P99 processing times across benchmark deduplication, window aggregation, and custom state tasks. Furthermore, the community is shifting its release schedule from a six-month cycle to quarterly deliveries to incorporate user feedback more rapidly.

Key points

Stateful streaming operations can now achieve sub-150 millisecond latency within the plat…: Stateful streaming operations can now achieve sub-150 millisecond latency within the platform's native Real-Time Mode execution engine.
Engineers can implement low-latency pipelines like personalization and fraud detection wi…: Engineers can implement low-latency pipelines like personalization and fraud detection without provisioning an entirely separate streaming engine.
Shortening the feature release cycle to every three months will provide the community mor…: Shortening the feature release cycle to every three months will provide the community more frequent opportunities to submit feedback.

Sources checked by our analysis pipeline: 1

Stateful real-time processing, history-preserving pipelines, and faster small-data development

by Jungtaek Lim, Wenchen Fan, Andreas Neumann, Serge Rielau, Hyukjin Kwon, Jerry Peng, DB Tsai, Daniel Tenedorio, Xiao Li and Reynold Xin

Apache Spark 4.3 brings stateful processing to Real-Time Mode, achieving sub-150 ms P99 end-to-end latency across three representative stateful benchmarks. Streaming aggregations, deduplication, and custom stateful processing can now run in Structured Streaming's low-latency execution mode, extending it to applications that need to remember context across events.

The impact goes beyond faster pipelines. For supported workloads, you no longer need a separate streaming engine just to achieve millisecond-scale latency. From fraud detection to continuously updated user activity and device monitoring, developers can build stateful applications with familiar Structured Streaming APIs in the same open engine they use for batch processing, analytics, and AI.

Stateful real-time streaming is the headline of Spark 4.3, but the improvements extend across the engine. The release delivers five key benefits:

transformWithState

in Real-Time Mode to maintain context as events arrive.Together, these changes help developers use one open engine to explore data, maintain changing context,…

AI Analysis

Tone

Positive

In short

Key Points

Stateful streaming operations can now achieve sub-150 millisecond latency within the plat…: Stateful streaming operations can now achieve sub-150 millisecond latency within the platform's native Real-Time Mode execution engine.
Engineers can implement low-latency pipelines like personalization and fraud detection wi…: Engineers can implement low-latency pipelines like personalization and fraud detection without provisioning an entirely separate streaming engine.
Shortening the feature release cycle to every three months will provide the community mor…: Shortening the feature release cycle to every three months will provide the community more frequent opportunities to submit feedback.

Topics

Technology

Was this summary useful?

AI Analysis

Tone

Positive

In short

Key Points

Stateful streaming operations can now achieve sub-150 millisecond latency within the plat…: Stateful streaming operations can now achieve sub-150 millisecond latency within the platform's native Real-Time Mode execution engine.
Engineers can implement low-latency pipelines like personalization and fraud detection wi…: Engineers can implement low-latency pipelines like personalization and fraud detection without provisioning an entirely separate streaming engine.
Shortening the feature release cycle to every three months will provide the community mor…: Shortening the feature release cycle to every three months will provide the community more frequent opportunities to submit feedback.

Topics

Technology

Was this summary useful?

X says its updated web app is way faster
The platform overhauled its underlying infrastructure to streamline web access across varying network speeds. Engineered to aid visitors on sluggish connections, the rewrite unifies fragmented systems to cut loading friction worldwide. Early feedback indicates noticeable responsiveness gains as the rollout gradually expands across international regions.
1 min read
Australia data centre inquiry finds inadequate checks, stops short of moratorium
Lawmakers in New South Wales warn that current planning guidelines critically underestimate the surging water and power demands of technology infrastructure projects. The findings highlight a contentious political battle ahead of the state's upcoming March election as major operators expand their footprints. Despite severe resource warnings, officials chose policy reform over halting ongoing sector developments.
1 min read
Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg
The automated capability enables data engineering teams to transition large analytics environments to modern lakehouse formats without provisioning dedicated compute clusters. Because operations run asynchronously alongside active systems, organizations avoid substantial downtime and bypass custom pipeline development. This streamlined shift unlocks advanced query capabilities while significantly reducing manual infrastructure management and operational expenses.
4 min read

Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

In short

Key points

More in Technology