Abstract

The use of new malware detection techniques has been made more difficult by the increase in use of encryption protocols like HTTPS, QUIC, and TLS 1.3. With the introduction of the QUIC protocol, there has been an improvement in the experience of end-users, however, there is a trade off with regard to new security risks. QUIC, which was designed by Google in 2012, is an example of an encrypted, low-latency, and connection-oriented protocol. In addition to encryption, QUIC is able to address several limitations present in current transport protocols like the TCP protocol, which tends to have high-latency when establishing a connection. Nonetheless, there have been some studies conducted on the security of QUIC that have brought to light some shortcomings. Because of its cryptographic security, QUIC has the ability to mask C2 packets with legitimate QUIC traffic. We present a machine-learning-based C2 traffic detection technique that utilizes fingerprinting which is commonly used for intrusion detection systems. Although encryption is important for privacy, it also cloaks C2 communication and data exfiltration, which is unfortunate. Our methodology utilizes an attention-based deep learning approach, including Convolutional Neural Networks (CNN) and a Transformer encoder. This is coupled with a 45-dimensional feature set derived from TLS fingerprinting, temporal patterns, DNS characteristics, and certificate metadata. Evaluated on a dataset of 1.2 million real-world encrypted flows, the system achieved a detection accuracy of over 89% with a low false positive rate of 0.7%. With an average latency of less than 100 ms, the framework is optimized for real-time enterprise deployment.