Antivirus Engines

From Methods to Innovations, Design, and Applications

1st Edition - October 21, 2024
Author: Paul A. Gagniuc
Language: English
Paperback ISBN:
9 7 8 - 0 - 4 4 3 - 3 2 9 5 2 - 4
eBook ISBN:
9 7 8 - 0 - 4 4 3 - 3 2 9 5 3 - 1

Antivirus Engines: From Methods to Innovations, Design, and Applications offers an in-depth exploration of the core techniques employed in modern antivirus software. It provides… Read more

Purchase options

SUSTAINABLE DEVELOPMENT

Innovate. Sustain. Transform.

Save up to 30% on top Physical Sciences & Engineering titles!

Explore now

Physical science & engineering for sustainable development book covers

Institutional subscription on ScienceDirect

Request a sales quote

Antivirus Engines: From Methods to Innovations, Design, and Applications offers an in-depth exploration of the core techniques employed in modern antivirus software. It provides a thorough technical analysis of detection methods, algorithms, and integration strategies essential for the development and enhancement of antivirus solutions. The examples provided are written in Python, showcasing foundational, native implementations of key concepts, allowing readers to gain practical experience with the underlying mechanisms of antivirus technology.

The text covers a wide array of scanning techniques, including heuristic and smart scanners, hexadecimal inspection, and cryptographic hash functions such as MD5 and SHA for file integrity verification. These implementations highlight the crucial role of various scanning engines, from signature-based detection to more advanced models like behavioral analysis and heuristic algorithms. Each chapter provides clear technical examples, demonstrating the integration of modules and methods required for a comprehensive antivirus system, addressing both common and evolving threats.

Beyond simple virus detection, the content illustrates how polymorphic malware, ransomware, and state-sponsored attacks are tackled using multi-layered approaches. Through these examples, students, researchers, and security professionals gain practical insight into the operation of antivirus engines, enhancing their ability to design or improve security solutions in a rapidly changing threat environment.

Antivirus Engines
Cover image
Title page
Table of Contents
Copyright
Dedication
Personal notes
Foreword
Chapter 1 History
Abstract
Keywords
1.1 Introduction
1.2 Early days and mainframe security
1.3 Rise of PCs and networked systems
1.4 Internet Age and Global Connectivity
1.5 Threats and cybercrime
1.6 Current and future challenges
1.7 Antivirus engines
1.8 Back to the future
1.9 Conclusions
References
Chapter 2 The internet
Abstract
Keywords
2.1 Introduction
2.2 The structure of the internet
2.2.1 Physical infrastructure
2.2.2 IP addressing
2.2.3 Routing
2.2.4 Domain name system
2.2.5 Internet protocols
2.2.6 Internet services and applications
2.3 Key components
2.3.1 End-user devices
2.3.2 Internet service providers
2.3.3 Protocols and standards
2.3.4 Internet services and applications
2.4 Internet layers
2.5 Internet visibility
2.5.1 Surface web
2.5.2 Deep web
2.5.3 Dark web
2.6 Domains
2.6.1 Domain names
2.6.2 Domain name system
2.6.3 Domain name registrars
2.6.4 DNS records
2.6.5 Domain name resolution
2.6.6 Malware and domains
2.6.7 Domain blacklisting
2.6.8 Domain redirection and traffic hijacking
2.7 Conclusions
Chapter 3 Operating systems
Abstract
Keywords
3.1 Introduction
3.1.1 Operating system types
3.1.2 The Windows OS
3.2 Structure of the Windows OS
3.2.1 Hardware abstraction layer
3.2.2 Windows kernel
3.2.3 System processes
3.2.4 Windows Subsystems
3.2.5 Windows API
3.2.6 Windows Services
3.2.7 Registry
3.2.8 New Technology File System
3.2.9 Drivers
3.2.10 Graphical user interface
3.2.11 Security
3.3 The OS structure and malware implications
3.3.1 Kernel modes vs malware
3.3.2 HAL vs malware
3.3.3 Device drivers vs malware
3.3.4 File system vs malware
3.3.5 Process management vs malware
3.4 Memory management
3.4.1 The link to malware
3.4.2 Targeting memory
3.4.3 Process injection
3.5 File management
3.5.1 The link to malware
3.6 User interface
3.6.1 Malware visual tricks
3.7 Networking
3.7.1 Networking: Link to malware
3.7.2 Malware tactics
3.7.3 Frequently attacked ports
3.7.4 About low-level protocols
3.7.5 Transmission control protocol/internet protocol
3.7.6 User datagram protocol
3.7.7 Cybersecurity of TCP/IP
3.7.8 Cybersecurity of UDP
3.8 Conclusions
Chapter 4 Operating system APIs
Abstract
Keywords
4.1 Introduction
4.2 The Windows API
4.3 Key libraries
4.3.1 Kernel32.dll
4.3.2 User32.dll
4.3.3 Gdi32.dll
4.3.4 Advapi32.dll
4.3.5 Comctl32.dll
4.3.6 Shell32.dll
4.3.7 Netapi32.dll
4.4 APIs and malware scanners
4.4.1 Malware signatures
4.4.2 Heuristics analysis
4.4.3 Behavior monitoring
4.4.4 Real-time protection
4.4.5 Windows Event Log API
4.5 Malware tactics and API libraries
4.6 Conclusions
Chapter 5 Cryptography
Abstract
Keywords
5.1 Introduction
5.2 Confidentiality and encryption
5.2.1 Symmetric and asymmetric encryption
5.2.2 Management and confidentiality
5.2.3 About cryptographic standards and protocols
5.3 Integrity and hash functions
5.3.1 Cryptographic hash functions
5.3.2 Examples on SHA-3, SHA-256, MD5
5.3.3 Verification of data integrity
5.4 Authentication and digital signatures
5.4.1 Public key infrastructure
5.4.2 Digital certificates
5.4.3 Digital signatures
5.4.4 Message digest and hash functions
5.4.5 Signing with private key and verification with public key
5.4.6 Nonrepudiation
5.5 Key exchange and secure communication
5.5.1 Encryption basics
5.5.2 Key exchange challenges
5.5.3 Asymmetric cryptography
5.5.4 Diffie-Hellman key exchange
5.5.5 Public key infrastructure
5.5.6 Secure communication protocols
5.5.7 Forward secrecy
5.6 Password security
5.7 Cryptographic techniques in malware
5.7.1 Code obfuscation and encryption
5.7.2 Polymorphism and cryptographic keys
5.7.3 Concealing command-and-control communication
5.7.4 Ransomware and encryption
5.7.5 Detection and solution strategies
5.8 Concealment of communication channels
5.8.1 Encryption
5.8.2 Tunneling
5.8.3 Steganography
5.8.4 Covert channels and defense strategies
5.9 Malware resistance to analysis
5.9.1 Code obfuscation
5.9.2 Packing and encryption
5.9.3 Antidebugging techniques
5.9.4 Polymorphism and metamorphism
5.9.5 Challenges and future trends
5.10 Cryptography as a defensive measure
5.11 Future implications and challenges
5.12 The impact of artificial intelligence
5.12.1 AI in threat detection
5.12.2 Incident response and threat intelligence
5.13 Conclusions
References
Chapter 6 Exploits
Abstract
Keywords
6.1 Introduction
6.2 Code injection exploits
6.2.1 Code injection techniques
6.2.2 History of code injection exploits
6.3 Buffer overflow exploits
6.3.1 History of buffer overflow exploits
6.3.2 Buffer overflow main steps
6.3.3 Low-level computer languages
6.3.4 High-level computer languages
6.3.5 C and C++: At the base of everything
6.4 Format string exploits
6.4.1 Practical example
6.4.2 The format specifiers
6.4.3 The meaning of the output
6.4.4 A striking demonstration
6.4.5 The limits of the exploit
6.4.6 Non-printf examples
6.5 SQL injection exploits
6.5.1 Working example
6.5.2 History of SQL injection exploits
6.6 Command injection
6.6.1 A command injection on a UNIX-like OS
6.6.2 A command injection on a Windows OS
6.6.3 A clear and dangerous exemplification
6.6.4 History of command injection
6.7 Cross-site scripting (XSS) exploits
6.7.1 Mechanisms of XSS exploitation
6.7.2 A test in a web server environment
6.7.3 A cross-site scripting exploit
6.7.4 Evaluate user input as an expression
6.7.5 History of cross-site scripting exploits
6.8 Remote code execution exploits
6.8.1 The mechanisms of RCE
6.8.2 Example of a simple RCE
6.8.3 How to test it?
6.8.4 A second example of a simple RCE
6.8.5 History of remote code execution exploits
6.9 Privilege escalation exploits
6.9.1 The mechanisms of privilege escalation
6.9.2 History of privilege escalation
6.10 Man-in-the-Middle (MitM) exploits
6.10.1 The mechanisms of MitMexploits
6.10.2 History of MitM exploits
6.11 Zero-day exploits
6.11.1 The mechanisms of zero-day exploits
6.11.2 History of zero-day exploits
6.12 Social engineering exploits
6.12.1 The mechanisms of social engineering exploits
6.12.2 History of social engineering exploits
6.13 Denial-of-service (DoS) and distributed DoS exploits
6.13.1 Mechanisms of DoS and DDoS attacks
6.13.2 Brief history of DoS and DDoS attacks
6.14 Conclusions
References
Chapter 7 Malware types and classification
Abstract
Keywords
7.1 Introduction
7.2 Computer languages and malware designs
7.3 List of malware types
7.4 Malware classification
7.4.1 Malware notation criteria
7.5 Paradigms in computer virus infection
7.5.1 File infectors
7.5.2 Macro viruses
7.5.3 Boot sector infectors
7.5.4 Network viruses
7.5.5 Email viruses
7.5.6 Polymorphic viruses
7.5.7 Worms: The mirror of the past
7.5.8 Malware: Polymorphic vs metamorphic
7.6 Paradigms in antivirus disinfection
7.6.1 Quarantine
7.6.2 Removal/cleaning
7.6.3 System restore
7.6.4 Repair/recovery
7.6.5 Patching/vulnerability remediation
7.7 Conclusions
References
Chapter 8 Antivirus engines
Abstract
Keywords
8.1 Introduction
8.2 Detection methods
8.2.1 Signature-based detection
8.2.2 Behavior-based detection
8.2.3 Heuristic detection
8.2.4 Sandbox detection
8.2.5 Machine learning detection
8.2.6 Cloud-based detection
8.2.7 Reputation-based detection
8.3 Antivirus solutions: The structure
8.3.1 Proactive monitoring
8.3.2 File verification and scanning
8.3.3 Decision and mitigation
8.3.4 Scheduled scans and updates
8.4 Signature databases
8.5 Antivirus files
8.5.1 Organization
8.6 Antivirus persistence
8.6.1 The invasive OS dependent method
8.6.2 The noninvasive OS independent method
8.6.3 File integrity
8.6.4 Antivirus self-protection model
8.6.5 File encryption and decryption
8.7 Interprocess communication (IPC)
8.7.1 IPC methods
8.7.2 SendMessage
8.8 The clients, the server, and the laboratory
8.8.1 A historical link
8.9 Conclusions
References
Chapter 9 Algorithms
Abstract
Keywords
9.1 Introduction
9.2 Signature matching with some examples
9.3 The signature lookup process
9.3.1 Hash tables
9.3.2 Hash tables (native)
9.3.3 Aho-Corasick algorithm
9.4 The hundred million hash signature file
9.4.1 Hash function: Size and speed
9.4.2 Bloom filters, caching, and search optimizations
9.4.3 Hardware methods
9.4.4 Hash tables vs binary search
9.5 Conclusions
Chapter 10 MD5/SHA signatures and scanners
Abstract
Keywords
10.1 Introduction
10.2 Single MD5 scanners
10.2.1 Whole file MD5 signatures and file path
10.2.2 Whole file MD5 signatures and file name (I)
10.2.3 A malware scanner using single MD5 signatures (II)
10.2.4 Signature generator on dual regions and single MD5 signatures (I)
10.2.5 A malware scanner for dual regions and single MD5 signatures (II)
10.2.6 Another approach to the same problem
10.3 Dual signatures vs one signature on dual file regions
10.3.1 Advantages and disadvantages of using signatures for each region
10.3.2 Advantages and disadvantages of a single signature for multiple regions
10.4 Dual MD5 scanners
10.4.1 The reference system
10.4.2 Signature generator for dual MD5 signatures (I)
10.4.3 Single file scanner for mono MD5 signatures (II)
10.4.4 Single file scanner for dual MD5 signatures (III)
10.5 Recursive malware scanners
10.5.1 Signature generator for balanced binary tree search (I)
10.5.2 Scanner for dual MD5 signatures with linear search (II)
10.6 Discussions on speed and efficiency
10.6.1 Why not more than two MD5 signatures?
10.6.2 Linear search vs binary search
10.7 Binary search on signature files
10.7.1 Signature generator for MD5 binary search(I)
10.7.2 Scanner using binary search in memory-loaded signatures (II)
10.7.3 Scanner using binary search in memory and extension filter (III)
10.7.4 Additional considerations
10.8 Hash tables on signature files
10.8.1 Scanner using hash table on dual MD5 signatures
10.8.2 Time complexity O(n) vs O(1)
10.8.3 Large scale signature files
10.9 Conclusions
Chapter 11 Disinfections, banks, and vaults
Abstract
Keywords
11.1 Introduction
11.2 The malware bank considerations
11.2.1 Malware bank and automatic signature extraction
11.2.2 The end of file problem
11.2.3 Self-sequence alignment
11.2.4 Elimination of potential confusions
11.2.5 Signature prevalidation scanner: The curator
11.3 The malware vault
11.3.1 Malware activation and inactivation by encryption
11.4 Malware (virus) disinfection
11.4.1 Baseline—The executable critical area
11.4.2 Virus disinfection—A prefix restoration approach
11.4.3 Mono vs dual regions
11.5 Disinfection by forced deletion
11.5.1 Privileged processes and their files
11.5.2 Malware persistence
11.5.3 The malware process and file terminator
11.5.4 WARNING: Use the code with extreme caution
11.6 Infection in progress: What now?
11.6.1 Exclusive file lock on a set of files
11.6.2 Direct API access: The LockEx example
11.6.3 A wise note on prevention
11.7 Conclusions
Chapter 12 Hexadecimal signatures & scanners
Abstract
Keywords
12.1 Introduction
12.2 Hexadecimal inspection
12.2.1 Alignment of multiple files in hexadecimal (raw)
12.2.2 Header alignment of two files (inf vs normal)
12.2.3 Pairwise sequence alignment of 2 files (infected vs clean)
12.3 Automatic signature extraction
12.3.1 The hexadecimal signature generator
12.3.2 Signature quality: Measurements
12.4 Signature detection
12.4.1 A primitive malware scanner with wildcard support
12.5 Malware scanners with Aho-Corasick algorithm
12.5.1 In-deep—Malware scanner
12.5.2 High speed—Malware scanner
12.6 Scanner speed and malware location
12.6.1 In-depth scanner (with libraries)
12.6.2 Optimized scanner (with libraries)
12.6.3 Optimized scanner (native Aho-Corasick)
12.6.4 Results and discussion
12.7 Polymorphic malware scanner: Wildcards
12.7.1 The meaning of wildcards
12.7.2 The regex malware scanner
12.7.3 Aho-Corasick vs regular expressions
12.8 Conclusions
Chapter 13 Heuristic signatures & scanners
Abstract
Keywords
13.1 Introduction
13.2 Heuristic detection on a multiparameter strategy
13.2.1 Signature generator for a multiparametric approach (I)
13.2.2 Malware scanner using multiparametric signatures (II)
13.3 Heuristic detection on discrete file regions
13.3.1 Computation of indices on 10 successive file regions
13.3.2 Computation of a global index for each file region
13.3.3 Signature generator for discrete file regions (I)
13.3.4 Malware scanner using discrete file regions (II)
13.4 Heuristic in-depth use of Information Content
13.4.1 The mathematical model
13.4.2 A concrete example
13.4.3 Model dissection
13.4.4 The information content
13.4.5 Signature generator for self-sequence alignment (I)
13.4.6 Malware scanner using self-sequence alignment (II)
13.5 Heuristic detection on strings
13.5.1 String extraction from portable executables
13.5.2 The use of extracted strings from executable files
13.5.3 API function name detection in PE files
13.5.4 A malware scanner based on weighted findings
13.5.5 A malware scanner based on the index of suspicion
13.5.6 The whole reasoning behind the API function names
13.6 Heuristic detection using PWMs
13.6.1 The Entry Point
13.6.2 What follows the Entry Point?
13.6.3 The bytes at the Entry Point
13.6.4 The 200 bytes coverage
13.6.5 The PWM generator based on PE Entry Points (I)
13.6.6 The PWM size on malware detection
13.6.7 How many positions for a PWM?
13.6.8 Byte consensus of a malware type
13.6.9 The conceptualization of ideal byte regions
13.6.10 The use of byte frequency distribution in the Entry Point
13.6.11 Malware family classification
13.6.12 Manual vs automatic signature extraction
13.6.13 About integration
13.6.14 A malware scanner using PWMs (II)
13.6.15 Virus infection experiments and detection refinement
13.7 Heuristic detection using Markov chains
13.7.1 Markov chains for static analysis
13.7.2 A Markov chains signature
13.7.3 Markov chains signature generator (I)
13.7.4 Markov chains signature scanner (II)
13.8 Heuristic detection using transition matrices
13.8.1 Transition matrix generator for binary data
13.8.2 Transition matrix signature generator (I)
13.8.3 Single file malware scanner (II)
13.8.4 Malware scanner with transition matrix signature-based detection
13.9 Heuristics by using perceptrons
13.9.1 Perceptron data generator for a malware scanner (I)
13.9.2 Malware scanner—Perceptron level-up binary signature training
13.10 Heuristics by random sampling convergence
13.10.1 Random sampling—Signature generator
13.10.2 Signature generator based on random sampling (I)
13.10.3 Malware signature scanner with signature file and linear search (II)
13.10.4 Malware scanner with fault tolerant matching and binary search (III)
13.10.5 Signature generator vs malware scanner
13.10.6 Signature convergence via genetic algorithms
13.11 Heuristics by searching for opcodes
13.11.1 The opcodes
13.11.2 The conditional vs unconditional jumps
13.11.3 The Entry Point and unconditional jumps
13.11.4 Types of jumps
13.11.5 Heuristic PE file analyzer
13.11.6 Single PE file scanner with weighted jumps
13.11.7 A heuristic scanner using weighted jumps
13.12 Heuristics of Entry Point obfuscation
13.12.1 Entry Point: High entropy
13.12.2 Entry Point: Low entropy
13.12.3 Entry Point: Moderate entropy
13.12.4 Entry Point obfuscation
13.12.5 The baseline scanner (I)
13.12.6 The EPO scanner (II)
13.12.7 EPO: Concentration vs dilution
13.12.8 The NOP scanner
13.13 Conclusions
References
Chapter 14 Smart scanners and rabbit holes
Abstract
Keywords
14.1 Introduction
14.2 Heuristics on header vs extension
14.2.1 Multifile header alignment for signature identification
14.2.2 File scanner: Type vs extension
14.2.3 Folder scanner for type vs extension verification
14.3 Heuristic multiple file headers
14.3.1 Folder scanner for multiple file headers
14.3.2 How to test the scanner?
14.4 Heuristic multiple file extensions
14.4.1 A scanner for detection of multiple file extensions
14.5 Heuristics on hidden files
14.5.1 Hidden file scanner
14.6 Scanning the archive content
14.6.1 A simple look in the archives
14.6.2 Scanning executable files from inside archives
14.6.3 Detection workflow in compressed archives
14.7 Registry monitoring
14.7.1 What is the windows registry?
14.7.2 Physical storage
14.7.3 Windows registry change detector (II)
14.8 Conclusions
Chapter 15 Process and user behavior
Abstract
Keywords
15.1 Introduction
15.2 The extension frequency
15.2.1 General file extension count
15.2.2 File extension count by folder (path)
15.2.3 File extension relative and overall frequency: Base line reference
15.2.4 Extension frequency monitoring of suspicious activity
15.2.5 Directory monitor via Win32 API
15.2.6 The system folder default paths
15.3 User monitor
15.3.1 Primitive active window to process path monitor
15.3.2 Active window to process path monitor for malware detection
15.3.3 Real-time user activity tracking
15.3.4 Processes and network connections
15.4 Process monitor
15.4.1 Frequent processes—Baseline map generator (I)
15.4.2 Anomaly detection through initial process maps (II)
15.4.3 Process path validation for proactive malware identification (III)
15.4.4 The overall monitoring strategy
15.4.5 Process alarm for CPU usage
15.4.6 A general download and upload network monitoring
15.4.7 Established connections—Network connection monitoring
15.4.8 Scan all processes under established connection
15.4.9 Process monitoring for suspicious activity over the network
15.4.10 Adaptive suspicious scoring thresholds for network monitoring
15.4.11 Detection of hidden windows
15.5 Conclusions
Chapter 16 The network
Abstract
Keywords
16.1 Introduction
16.1.1 Communication protocols: In brief
16.1.2 Ports and their significance_ In brief
16.1.3 Processes and networks
16.2 The firewall
16.2.1 Stop connection by PID and IP
16.2.2 List all processes and ports/IP
16.2.3 List only processes with connections
16.2.4 Monitor processes with open ports
16.2.5 Stop process connection with GUI
16.2.6 The command prompt tools
16.2.7 Integration of netstat output into a GUI
16.2.8 Pièce de résistance_ The PE file vs the network
16.3 Conclusions
Glossary
Appendices
Appendix 1—Executable file extensions
Appendix 2—PWM to JSON
Appendix 3—File headers
Appendix 4—Running processes
Appendix 5—Hidden windows
Appendix 6—Registry snapshots
Appendix 7—Registry key list
Appendix 8—Built-in tools and utilities
Appendix 9—CMD built-in commands
Appendix 10—A general overview of Scut Antivirus
The webpage
The laboratory
The setup wizard
The on-demand scanner
Scanner settings
Full and critical scans
Tests and results
The output of the scan
File/process organization
The communication protocol
Firewall (I)
Firewall (II)
Process monitor and scanning
Network monitoring and scanning
Proactive firewall (III)
CPU monitor and process role
Security logs
Graphical user interface
The VX Heaven website
Appendix 10—Words of wisdom
References
References
Index

Life Sciences

Physical Sciences & Engineering

Social Sciences & Humanities

Health

Antivirus Engines

From Methods to Innovations, Design, and Applications

Purchase options

Innovate. Sustain. Transform.

Institutional subscription on ScienceDirect

Paul A. Gagniuc

Related books