Abstract

Ransomware attacks surged 50% in 2025 with over 7,874 incidents recorded globally. The lateral propagation phase—where malware spreads across enterprise networks—remains largely undetected by existing security tools that rely on closed-set classification and cannot identify previously unknown ransomware families. This paper proposes OpenRansom, a novel open-set detection framework combining Heterogeneous Provenance Graphs with Prototypical Graph Neural Networks to detect both known and unknown ransomware propagation patterns. Unlike current graph-based ransomware methods which mainly work under closed-set assumptions, OpenRansom uses a prototype-based mechanism to spot new, unknown ransomware behaviors. We validate the framework through three experimental implementations on the MLRan dataset spanning 65 ransomware families: a Random Forest prototype achieving 93.8% detection of unknown ransomware families, a KNN-Graph Neural Network achieving 54.6% unknown detection with improved known-family accuracy, and a Propagation-GNN architecture designed for provenance graph input. Full validation of the provenance-graph-based Graph Attention Network architecture remains future work due to the absence of suitable public enterprise-scale propagation datasets.