Abstract

The rapid escalation of sophisticated cyber-attacks, including advanced persistent threats, ransomware, and zero-day exploits, has outpaced the capabilities of traditional signature-based detection tools. This paper presents a comprehensive study on leveraging machine learning and deep learning techniques for real-time anomaly detection in system logs. System logs from diverse sources—networks, hosts, applications, and security devices—contain temporal, sequential, and semantic features that reveal patterns of cyber threats. We review state-of-the-art approaches, including LSTM-based models like DeepLog, Transformer-based models such as LogBERT, and hybrid architectures incorporating graph neural networks and large language models. Evaluations on benchmark datasets (HDFS, BGL, and Thunderbird) demonstrate superior precision, recall, and reduced false positives compared to conventional methods. Challenges such as data variability, evolving threats, and interpretability are addressed through explainable AI enhancements. Future directions include generative AI models, privacy-preserving federated learning, and edge computing for IoT environments.