Dağ, HasanDemirkiran, FerhatDemirkıran, FerhatCayir, AykutUnal, GurDag, Hasan2024-06-232024-06-232022160167-40481872-6208https://doi.org/10.1016/j.cose.2022.102846https://hdl.handle.net/20.500.12469/5644Demirkiran, Ferhat/0000-0001-7335-9370; Unal, Ugur/0000-0001-6552-6044Classification of malware families is crucial for a comprehensive understanding of how they can infect devices, computers, or systems. Hence, malware identification enables security researchers and incident responders to take precautions against malware and accelerate mitigation. API call sequences made by malware are widely utilized features by machine and deep learning models for malware classification as these sequences represent the behavior of malware. However, traditional machine and deep learning models remain incapable of capturing sequence relationships among API calls. Unlike traditional machine and deep learning models, the transformer-based models process the sequences in whole and learn relationships among API calls due to multi-head attention mechanisms and positional embeddings. Our experiments demonstrate that the Transformer model with one transformer block layer surpasses the performance of the widely used base architecture, LSTM. Moreover, BERT or CANINE, the pre-trained transformer models, outperforms in classifying highly imbalanced malware families according to evaluation metrics: F1-score and AUC score. Furthermore, our proposed bagging-based random transformer forest (RTF) model, an ensemble of BERT or CANINE, reaches the state-of-the-art evaluation scores on the three out of four datasets, specifically it captures a state-of-the-art F1-score of 0.6149 on one of the commonly used benchmark dataset. (C) 2022 Elsevier Ltd. All rights reserved.eninfo:eu-repo/semantics/openAccessTransformerTokenization-freeAPI CallsImbalancedMulticlassBERTCANINEEnsembleMalware classificationArticle121WOS:00088154130000510.1016/j.cose.2022.1028462-s2.0-85136643921Q1Q1