Back to home

Fraud Triage: deciding what to do with a fraud score

A LightGBM model plus a cost-based policy that decides approve, review or block under a fixed daily analyst budget. 66% lower total cost than reviewing the highest-probability cases.

Fraud Triage: deciding what to do with a fraud score

Type

Personal project

Role

ML + full stack, solo

Duration

Live demo

Tools & Technologies

PythonLightGBMSHAPDuckDBFastAPIPostgreSQLNext.jsPlaywright

Problem

Fraud models are easy to train and hard to act on. Fraud is rare here, about 1 in 190 transactions, so a model that never flags anything is 99.5% accurate and useless. Analysts can only check a few cases a day, so the real question isn't "is this fraud?" but "which cases deserve a person's time?"

The usual answer is to review the highest-scoring transactions. That spends review slots on small purchases that happen to look suspicious.

What I built

A system that turns a fraud probability into one of three actions by comparing expected dollar costs:

  • Approve costs probability × amount, the fraud you would lose
  • Review costs $5 of analyst time
  • Block costs (1 − probability) × $10 for turning away a real customer

Each transaction gets the cheapest action, and each day's 30 review slots go to the cases where a review saves the most. Analysts work through a review app that shows the three reasons behind each score and the cost of each action.

The daily review queue, ranked by expected loss
A single case: why the model flagged it and what the policy decided

Results

Tested on three held-out months, October to December 2020: 281,521 transactions, 936 frauds worth $489,894. Every policy gets the same 30 reviews a day.

PolicyFraud lostReviewsCustomers declinedTotal cost
Approve everything$489,89400$489,894
Review highest probabilities$24,1642,7600$37,964
Review highest expected dollars$10,5632,0520$20,823
Cost-based: approve, review or block$4,5271,419114$12,762

The model itself reaches a test PR-AUC of 0.838 against 0.376 for a logistic regression baseline, and reviewing the riskiest 1% of transactions catches 93.3% of fraud.

Fraud lost as the daily review budget changes, for each policy

Key decisions

Time-based splits only

Train on January 2019 to June 2020, tune on July to August, calibrate on September, test on October to December. A random split would let the model learn from the future.

Leak-free features, with a test that proves it

Every history window ends one microsecond before the current transaction. A test adds a later transaction and checks that no earlier feature changes.

Calibrated probabilities

The policy multiplies probability by amount, so probabilities have to match real fraud rates. Training weighted fraud 13 times, which made raw scores about twice too high; Platt scaling on a held-out month fixes that.

Probability ranking wastes review slots on small purchases. Ranking by expected dollars, with the same budget, cuts fraud lost by more than half.

Score offline, serve precomputed results

The deployed API never loads LightGBM or SHAP, so it fits on free hosting. A test checks that production mode never imports them.

Limitations

  • No fraud in the dataset is above $1,376, so the model learned that very large purchases are safe. Real fraudsters would exploit that.
  • The $5 review and $10 decline costs are assumptions. The conclusion holds when the review cost is halved or doubled and when the decline cost rises to $50 or $100, though blocking becomes much more cautious.
  • The data is the public Kaggle Sparkov simulation, not real card data.