ML System Design: Abuse Detection — Account Takeover, Bots, and Velocity Beyond Spam
Design a cross-product abuse platform distinct from content spam — credential stuffing, account takeover (ATO), synthetic accounts, scraping, and collusion rings. Covers device graphs, velocity features in Redis, challenge escalation (CAPTCHA, step-up auth), feedback loops when labels are delayed, and why Meta-style integrity teams separate abuse from policy-violating content classifiers.
Abuse Is Behavioral and Graphy — Not Just Text
Spam classifiers score bodies of text or media. Abuse systems score actions over time — login failures, password resets, IP / ASN churn, device fingerprint reuse, headless signals, API rate anomalies. Labels are sparse, delayed (chargebacks, victim reports), and adversarial — attackers A/B your defenses.
Interviewers want latency budgets for pre-auth checks, post-auth async graph jobs, and human review queues with SLA.
Differentiation from Fraud and Spam
Fraud optimizes money loss with transaction context. Spam optimizes content policy. Abuse optimizes platform integrity — accounts, APIs, social graphs — metrics: ATO rate, fake signup %, scrape bandwidth.
Clarifying Questions
Abuse class
ATO vs scraping vs fake accounts — feature families differ.
Latency band
Login path vs async post-login — sub-20ms vs seconds.
User friction budget
Step-up auth rate cap — product will veto high FPR.
Global vs regulated markets
SMS OTP legality, data localization.
Label source
Support tickets, chargebacks, honeypots — bias audit.
Feature Families — Velocity, Device, Graph
Velocity: Counts per IP / device / user in rolling windows — Redis with TTL buckets (1m, 10m, 24h). Signals: failed logins, distinct accounts per device, geo velocity impossibility.
Device intelligence: Fingerprint stability, emulator flags, TLS fingerprint (JA3), header order anomalies for bots.
Graph: Shared phone across accounts, referral cycle motifs — offline GNN or union-find components for ring detection; online scoring uses cached embeddings.
Abuse Scoring Path — Sync Gate vs Async Graph
Actions by Risk Tier
| Risk score band | User experience | Engineering cost |
|---|---|---|
| Low | Silent allow | Baseline traffic |
| Medium | CAPTCHA or SMS step-up | Support tickets rise if miscalibrated |
| High | Block plus cooldown | False positives viral on social |
| Graph-confirmed ring | Bulk disable with appeal | Legal review |
| Recommended | Calibrated thresholds shadow mode first | Iterate with PR curve |
Suppression and Delayed Labels
Blocking stops the negative label stream — models starve on hard positives. Use shadow scoring, honeypot credentials, and external breach intel as weak labels.
Interview Closer
"I'd combine sub-10ms velocity and device signals with a compact GBM in the request path, enqueue graph enrichers asynchronously for account clusters, calibrate thresholds on dollarized harm plus UX friction, and run shadow mode before tightening blocks."