Artificial Intelligence1 min read

Doing More by Storing Less: Speeding Up Real-Time Machine Learning

By · Published by Everything Blog

In short

Standard real-time transaction monitoring relies on continuous read-modify-write cycles that quickly overwhelm disk storage systems. To resolve this, Feedzai implemented a technique that evaluates every event in memory for fraud but selectively skips saving updates to disk as entity event volumes rise. The system calculates an inclusion probability by dividing a target threshold by the entity's actual transaction rate to discard excess writes. Meanwhile, low-volume entity profiles retain every single database update to preserve complete statistical accuracy.

Key points

  • Discarding persistent storage updates for high-volume entities significantly reduces proc…: Discarding persistent storage updates for high-volume entities significantly reduces processing latency and database contention.
  • Operational write limits can be managed probabilistically by dividing a target threshold…: Operational write limits can be managed probabilistically by dividing a target threshold by an entity's actual transaction rate.
  • Skipping non-essential database writes cuts infrastructure costs without degrading downst…: Skipping non-essential database writes cuts infrastructure costs without degrading downstream fraud detection machine learning accuracy.

Sources checked by our analysis pipeline: 1

Modern fraud detection systems rely heavily on tracking behavior over time. For example:

How much money has a person spent this month? Or, how many purchases have they made in the last week?

Any event-streaming system worth its salt tracks hundreds of such metrics. Every time a new transaction even arrives, these are retrieved from a database, updated, and then written back to persistent storage. This process is known as the read-modify-write cycle, and its final write stage is a major source of latency, database contention, and infrastructure cost.

But here is the catch: In many financial applications, a small set of heavy-hitter entities triggers most of these cycles and accounts for a disproportionate number of transactions.

Statistically, the more events we process for any entity, the less information each individual event adds to their overall profile. Thus, here at Feedzai we developed a process and system that automatically stops writing every single event for large financial entities, saving significant compute without losing downstream ML accuracy.

A friendly introduction to feature profiles

Before we go any deeper into discarding events,…

Artificial Intelligence