Abstract
Modern malware evolves rapidly, rendering static signatures and heuristic-based detection methods less effective. Deep neural networks offer richer pattern recognition but require careful hyperparameter configuration, which manual or naïve search cannot achieve efficiently. To address this, automated methods are often needed to tune hyperparameters effectively. This dissertation explores biologically inspired optimisation methods for automating hyperparameter tuning in deep learning algorithms for malware classification. The study introduces BIMDO, a modular model that combines four deep learning algorithms (ANN, CNN, LSTM, Transformer) with six biologically inspired optimisers: three artificial immune system techniques (clonal selection, immune network, dendritic cell) and three evolutionary algorithms (genetic algorithm, particle swarm optimisation, differential evolution). Experiments were conducted on three malware data representations (PE file headers, API call sequences, and opcode n‑grams) under a fixed evaluation budget. For each combination of optimiser, deep learning algorithm, and dataset, approximately 60 function evaluations were performed and repeated with five independent random seeds, providing matched and statistically robust comparisons of performance. Results show that all optimisers discovered competitive configurations, with performance driven mainly by the synergy between data representation and deep learning architecture. LSTMs and transformers excelled on API call sequences (accuracy above 0.99), ANNs on PE file headers (approximately 0.989), while CNNs outperformed ANNs on opcodes but with lower overall accuracy. AIS methods, particularly dendritic cell and immune network, often achieved the best results but at higher computational cost, whereas differential evolution and particle swarm optimisation reached near‑optimal solutions faster. The research contributes a controlled comparison of AIS and EA approaches, a reusable prototype to study the use of biologically inspired algorithms for hyperparameter optimisation, and practical guidance for selecting deep learning-optimiser-dataset combinations under equivalent compute budgets, while highlighting challenges such as configuration failures and representation limits when integrating biologically inspired optimisation into AutoML pipelines for cybersecurity.