Real-time Fraud Detection pipeline

What I built

An event-driven pipeline that generates synthetic card transactions, scores them for suspicious activity, and stores the results for review. I built this to explore the backend of a fraud-detection system: streaming ingestion, stateful rules, cloud infrastructure, and the trade-off between throughput and latency.

The detection path is deliberately explainable rather than ML-based. Each transaction is checked for a high amount, unusual local hour, high-risk merchant category, and repeated activity on the same card within a ten-minute window. The result includes which rules fired, not just a fraud flag.

Architecture

Python transaction generator
          ↓
Kinesis Data Streams
          ↓
       Lambda
       ├─ S3: raw transaction archive
       └─ Rule engine → DynamoDB: scored transactions

• Ingestion: The generator emits transactions at a configurable rate and batches writes to Kinesis, using card_id as the partition key.
• Processing: A Kinesis enhanced fan-out consumer feeds a Python Lambda function. Records within a batch are processed concurrently, including their storage calls.
• Storage: S3 archives raw events under time-based paths. DynamoDB stores scores and rule flags, with a secondary index for recent flagged transactions and a 30-day TTL.
• Infrastructure: Terraform modules define the stream, consumer, Lambda, storage, IAM permissions, and CloudWatch dashboard.

There is also a cloud-free development mode: an in-process scorer, SQLite storage, and a live terminal dashboard. Separate source, engine, and sink interfaces let the local and AWS paths share the same pipeline structure.

Engineering decisions

Batch writes without waiting indefinitely. The producer flushes up to 500 records at once, or every 100 ms. That reduces ingestion API calls while putting a time limit on how long a lightly loaded buffer waits. A bounded producer–consumer queue separates transaction generation from scoring and output.

Tune delivery and processing together. Enhanced fan-out avoids relying on the standard polling consumer. The Lambda event source uses batches of 50 records and configurable infrastructure for shard count and memory. Inside the handler, a thread pool overlaps I/O, with the AWS connection pool sized to match the worker limit.

Make failures visible. The handler reports failed record sequence numbers through Lambda's partial-batch response mechanism. S3 archive failures are logged separately and do not block scoring, explicitly favoring continued processing over a guaranteed complete archive.

Measuring behavior

I added CloudWatch latency percentiles, Lambda duration, errors, and throttles, plus command-line tools for checking generator throughput and reporting latency distributions from DynamoDB.

The load generator defaults to 1,000 transactions per second in AWS mode, and the latency report checks a p50 target of 200 ms or less. These are configured targets, not published benchmark results. The recorded latency runs from transaction creation to just before the DynamoDB write, so it does not include that write's completion time.

A separate evaluation harness uses a seeded generator and virtual clock to examine precision, recall, and per-rule false positives. Those results describe behavior on synthetic data constructed around the rules—not real-world fraud-detection accuracy.

Trade-offs and next steps

The velocity rule reads recent card activity from DynamoDB before writing each score. This avoids introducing another state store, but adds a database read to the processing path. Concurrent records for the same card also need stronger coordination before the velocity check could serve as a production decision mechanism.

The implemented review interface is the local terminal dashboard. An AWS-backed web dashboard and an ML scoring path are future extensions; the current project focuses on the streaming system and its observability.