Home About Projects Papers Awards Blog
Back to Blog
Paper

Comparative Study of Machine Learning Methodsfor Credit Card Fraud Detection

Credit card fraud detection is an imbalanced classification problem in which legitimate transactions greatly outnumber fraudulent ones.
This paper applies nine machine learning methods to two public datasets under the same conditions and compares both training and inference time.

Key Takeaways

XGBoost and LGBM achieved the strongest classification performance on both datasets.
In particular, LGBM maintained accuracy comparable to XGBoost while detecting fraudulent transactions more than 30 times faster, making it a stronger candidate for real-time detection environments.
  • On CCFD, XGBoost achieved the highest AUPRC and AUROC.
  • On IEEE-CIS FD, LGBM achieved the highest AUPRC, while XGBoost achieved the highest AUROC.
  • For imbalanced data, evaluating AUPRC alongside AUROC revealed differences between models more clearly.
  • Practical deployment requires considering retraining frequency and inference latency as well as predictive accuracy.

1. Problem Definition

Fraud detection is a binary classification problem in which a model trained on legitimate and fraudulent transactions determines whether a new transaction is suspicious.
Because transaction features such as amount and timestamp may contain sensitive personal information, real financial data is difficult to release, and the meanings of most features in public datasets are anonymized.

The paper focuses on the following challenges.

  • Class imbalance: Fraud accounts for only 0.173% of CCFD and 3.499% of IEEE-CIS FD.
  • Concept drift: As new attack patterns emerge, models trained on historical data may lose accuracy.
  • Time complexity: A production system must support periodic retraining and process thousands or tens of thousands of transactions per second.
  • Limited data: Privacy constraints make it difficult to obtain sufficient real transaction data and meaningful feature descriptions.
Classification of machine learning methods used for credit card fraud detection
Major machine learning methods used for credit card fraud detection

2. Data and Evaluation Methodology

Datasets

  • Credit Card Fraud Detection (CCFD): 492 fraudulent transactions among 284,807 total transactions. View dataset
  • IEEE-CIS Fraud Detection: 590,540 total transactions and 433 original features. View dataset

CCFD was standardized using the mean and variance calculated from the training data.
For IEEE-CIS FD, the authors considered missing values, correlations, feature importance, and temporal stability. They also generated derived features and used 263 features in the final dataset.

Nine Compared Methods

  • Linear models: LDA, Lasso, Ridge, and Elastic Net
  • Tree and kernel models: XGBoost, LGBM, Random Forest, and SVM
  • Neural network model: A classifier pretrained with an autoencoder

Metrics and Validation

The models were evaluated using stratified five-fold cross-validation repeated ten times. Within each training set, hyperparameters were optimized through stratified three-fold cross-validation and grid search.
The reported results are the means and standard deviations from a total of 50 evaluations.

  • AUROC: Evaluates overall classification performance across thresholds, including the majority class.
  • AUPRC: Uses precision and recall, making it more sensitive when identifying a minority class such as fraudulent transactions is important.

3. Classification Performance

AUPRC and AUROC performance of machine learning methods on CCFD
Performance comparison on CCFD
AUPRC and AUROC performance of machine learning methods on IEEE-CIS FD
Performance comparison on IEEE-CIS FD

CCFD

XGBoost achieved the best performance, with an AUPRC of 0.8612 and an AUROC of 0.9827.
LGBM and Random Forest followed closely, both recording AUPRC values above 0.85.
Although most methods achieved AUROC values above 0.9, the difference between linear and nonlinear models was more apparent in AUPRC.
This result demonstrates why AUPRC is important when evaluating imbalanced data.

IEEE-CIS FD

LGBM achieved the highest AUPRC at 0.9249, while XGBoost achieved the highest AUROC at 0.9836.
Gradient boosting methods again outperformed the other approaches overall.

4. Training and Inference Time

In production, both retraining speed and per-transaction inference latency are important.
Models must be updated periodically as attack patterns change, and fraudulent transactions may occur repeatedly within a short period.

Training and inference time of machine learning methods on CCFD
Training and inference time on CCFD
Training and inference time of machine learning methods on IEEE-CIS FD
Training and inference time on IEEE-CIS FD
  • SVM required the longest training time on both datasets.
  • XGBoost required 0.0668 seconds per prediction on CCFD and 0.1248 seconds on IEEE-CIS FD.
  • LGBM required only 0.0016 and 0.0037 seconds, respectively, making it more than 30 times faster than XGBoost.
  • Although LGBM took slightly longer to train than XGBoost, it provided a substantial advantage for real-time inference.

5. Reflections

A strength of this paper is that it compares multiple models on two different public datasets using the same validation procedure while evaluating both predictive performance and processing time.
Rather than simply selecting the model with the highest score, the paper considers the response time required by a production fraud-detection system, making its recommendation of LGBM persuasive.

In a real service, however, model performance must be considered alongside the cost of reviewing false positives, detection thresholds, concept drift caused by emerging fraud patterns, and explainability.
Future comparisons could use time-based validation and cost-sensitive learning to better approximate production conditions.

Attachment