Adversaries develop adversarial malware examples that are designed to evade detection systems, which can significantly decrease the performance of malware classifiers. To defend against adversarial malware examples, a technique called adversarial training is implemented by additionally train the detection models with adversarial examples, making the models more robust to such attacks. This thesis discusses an approach to generating adversarial examples using Particle Swarm Optimization (PSO). We explore a technique called AdversarialPSO proposed by Rayan Mosli [1], which crafts adversarial malware examples by replacing API calls with alternative APIs that have equivalent functionality leveraging PSO to navigate substitutions. In this thesis, we propose a new approach for finding alternative APIs using the state-of-art Sentence Transformers (SBERT) to generate API mapping. Additionally, we categorize the mapping into four categories, making the attacks more practicability, and then manually verify the substitution in practice, resulting in two mappings: Simple Replacement and Complete Replacement. The study finds that the PSO attack using these mappings approximately achieves a success rate of 78% and 82%, respectively, outperforming the naive method that replaces all potential alternatives in the two mappings. Moreover, the attacks also require 28.8% and 24.42% fewer substitutions from the naivety replacement utilizing Simple Replacement and Complete Replacement, respectively. Additionally, a method of mapping augmentation is introduced to enhance the results, enabling the success rate to rise up to 86.44%. The study concludes that using SBERT to generate comprehensive mappings and leveraging PSO to guide the replacement location is an impactful and practical approach for generating adversarial malware examples with a low likelihood of errors or unexpected behavior.

Library of Congress Subject Headings

Malware (Computer software)--Prevention; Computer security; Swarm intelligence; Mathematical optimization

Publication Date


Document Type


Student Type



Matthew Wright

Advisor/Committee Member

Yin Pan

Advisor/Committee Member

Sumita Mishra


RIT – Main Campus

Plan Codes