Abstract

This paper presents a large-scale empirical comparison of machine learning approaches for detecting botnet Command-and-Control (C2) traffic using network flow analysis. We evaluate three distinct methods on the CTU-13 dataset comprising 13.46 million network flows: Random Forest, XGBoost (supervised learning), and an LSTM Autoencoder (anomaly detection). Our comparative analysis demonstrates that Random Forest achieves the highest performance with 99.21% accuracy, 94.31% recall, and 0.9968 ROC-AUC, while XGBoost attains 98.93% accuracy and 90.54% recall. The LSTM Autoencoder shows limited effectiveness with 0.679 ROC-AUC. These findings provide empirical evidence that flow-based supervised learning can effectively detect encrypted C2 traffic without payload inspection, achieving false positive rates below 1%, making it practical for real-time botnet detection in operational networks.