SecureBank fraud

The SecureBank case in EN.705.603 at Johns Hopkins asks for a card-fraud model that reaches 70 percent precision and 70 percent recall on unseen months. I ran this as the model selection pass. I picked two offline metrics tied to the requirements, trained three models on the same chronological split, and read the comparison for where the signal lives. Codex reviewed the first version adversarially and I reworked the leakage paths, the ablations, and the claims it flagged.

I score every model on average precision over the whole ranking and on precision and recall at one operating point. Average precision is my model selection number, since it does not depend on a threshold I have not chosen yet. The operating point is the number I'd show the bank. The alert threshold is set on the validation month at the lowest score that keeps precision at 70 percent, which is the analyst capacity constraint from my earlier posts turned into a rule. A model that never reaches 70 percent on November gets no qualifying operating point; I score it at a fixed 10 alerts a day instead and say so.

The split is the one I argued for in the dataset design week. I train on January 2019 through October 2020, choose the threshold on November, and score December once. The models train on the full training frame with balanced class weights; I kept the 10 percent downsample from my first pass as an ablation and it costs the boosted model 0.03 of average precision. Validation and test stay at the fraud rate among labelled rows, because precision on a rebalanced set is a number the bank never sees. The 10 percent of transactions with no label are out of every metric; their median amount and night share match the labelled rows, which is the only check I have that they are missing at random.

December doubles the volume and keeps the fraud count flat, so the base rate halves. Every precision number I report below was harder to earn than the validation month suggested.

I built seventeen features, all computable at authorization. The amount bands and the gap sentinel were fixed before any fitting. The velocity features use every transaction, including the unlabelled ones, but only rows with a strictly earlier timestamp. The scorecard adds one cross bin, category by amount band.

Numeric: log_amt, amt_vs_prior_mean, n_prev_1h, n_prev_24h, log_secs_since_prev, first_txn, hour_sin, hour_cos, night, dow_sin, dow_cos, month_sin, month_cos, age, log_city_pop, home_merchant_km. Categorical: category, and cat_amt (category by amount band) in the scorecard.

I fit a logistic regression on the standardized features, a weight-of-evidence scorecard (optbinning bins each numeric feature on the training rows, the category and cross-bin columns get a smoothed WOE per level, and a logistic regression fits on the WOE columns), and a gradient boosting classifier.

Gradient boosting reaches average precision 0.83 and, at the November threshold, 57 percent precision with 84 percent recall on 9 alerts a day, catching 94 percent of the fraud dollars. The scorecard reaches 0.56 and the raw logistic regression 0.10, and the logistic regression never qualifies. The whiskers are card-cluster bootstrap intervals with the model and threshold held fixed, so they cover the December card mix and nothing else. The precision miss against the 70 percent target has the halved base rate as one plausible cause; ROC AUC barely moves from November to December (0.999 to 0.998), but average precision drops from 0.93 to 0.83 and I have not separated prevalence from drift in the fraud mix.

Shuffling feature groups together on November shows what the boosted model relies on. Amount costs 0.92 of average precision, category 0.81, hour 0.31, and nothing else more than 0.13. That is model reliance, not a statement about how the features combine, so I tested the combination directly.

On the training rows, a $300 to $500 purchase runs 100 percent fraud in grocery_pos and 0 percent in travel, so the same amount means different things in different categories. The ablation makes the point in numbers. A scorecard on amount alone scores 0.02 on December, category alone 0.003, the two added together 0.02, and the single category-by-amount cross bin 0.37. All eleven main effects with category reach 0.11; adding the cross bin lifts that to 0.56. The tree model learns the same interaction and the rest on its own and scores 0.83. That gap is what I took from the exercise.

On the training rows 85 percent of the fraud sits on the 23 percent of volume between 22:00 and 04:00, a 1.7 percent rate against 0.09 percent by day. Gradient boosting holds 90 percent recall at night and 55 percent in the day on the same threshold. The scorecard raises 9 daytime alerts in the whole month. A model that only works at night is a model a fraudster learns to beat by waiting until morning, so I'd report the daytime slice to the bank separately.

• Hold the threshold rule and re-fit monthly on a rolling window, to see whether the December precision drop is seasonal volume or drift in the fraud mix.
• Give the scorecard the hour-by-category and amount-by-hour cross bins and check whether an explainable model can close the rest of the gap to the tree.
• Replace the 70 percent precision rule with an alert budget per analyst per day and read recall off that, which is how the fraud operations teams I have worked with set thresholds.
• Daytime fraud as its own evaluation slice, with its own threshold if the cost of a missed daytime case justifies more alerts.

Method notes: scikit-learn 1.9, optbinning 1.0, one run of 115 seconds on a laptop. Thresholds and bins come from the training and validation months only. Intervals resample cardholders, 1000 draws, model and threshold fixed. The oracle recall at 70 percent precision in the tables uses December labels and is an upper bound, not an operating result.