Testing methodology

The Malware
All malware samples are obtained and verified through a virtual machine that has all built-in security mechanisms disabled. This to make sure that OS X can not affect the samples while they are being validated and sorted. The validity of the samples is determined by uploading the files to VirusTotal and by installing/testing it’s effects and behaviors in another isolated virtual machine. Once samples are proven to be valid they are sorted into named folders that make it easier to read out the scan and detection results in AV logs later on. For example:
Pintsized =1=
Pintsized =2=
Pintsized =3=
Leverage =1=
Leverage =2=
etc.

Each named folder contains the following:
– The original sample, unaltered.
– A duplicate of the original sample, possibly with a file extension added.

If a sample is an archive or disk image, the sample is duplicated, placed into a new folder and unarchived. Once unarchived, the duplicate is deleted from the folder so only the unarchived files remain. The additional folders are named too so it may look something like this:
Leverage =1=
Leverage =1= Files
Leverage =2=
etc.

When all the files have been sorted, the entire folder containing all files as described above are saved to a read-only 128-bit encrypted disk image through Disk Utility. This disk image is used to infect the test virtual machines.

I make no distinction between active or no longer active malware. Malware is malware and as such it should be detected. If old malware that is no longer active is not detected today, it was not able to detect it at the time it was active either. This will show an application’s reliability and ability to protect you in the past and may show how it will behave in the future. Also if you run an older system (PPC and/or old OS) you may still be able to get infected by old malware whether it is active or not. I do show detection results by year so you can see how good an antivirus application is with recent malware samples.

Detection test machine
All tests are done inside a VMware Fusion virtual machine on a Mac Pro Server. The virtual machine is given the following resources:
– 4 CPU cores of 2.92 GHz each.
– 4 GB RAM.
– Whatever hard drive space it needs. VMware uses a 2TB dedicated internal hard drive.
– OS X 10.8.5 and latest software updates available at the time of testing.
– Latest Flash and Java available at the time of testing.
– Fresh copy of Little Snitch, latest version demo.
– File Quarantine set to allow applications to be opened from anywhere.
With this clean install done, a snapshot of the system is made (Snapshot 1).

Xprotect / File Quarantine
The system is then purposely infected with all the malware installers and scripts I have. Whatever is caught by XProtect will be noted down and can be found in the XProtect test results. XProtect is then disabled and all available malware is opened/installed again. I test and verify infection by looking in the known locations the malware installs itself and watching for certain behaviors that are known to be the result of infection. Xprotect is then enabled and updated to see if it detects already installed malware (I found it never does). At this point a second snapshot is made of a thoroughly infected machine (Snapshot 2).

When new malware samples are obtained that contain any type of executables I restore Snapshot 1 and start over with the procedure so a new Snapshot 2 can be created.

The antivirus applications
The virtual machine will be restored to Snapshot 2. All malware that does not have installers is placed in the user’s download folder. Now the antivirus application is installed, configured for best possible detection results and , when possible, set to quarantine rather than delete or cure. The logs and contents of the quarantine are used to determine detection. As soon as a scan is done another snapshot is made (Snapshot Results #), these snapshots can be used for further testing of new malware and analysis. Of every AV product only two snapshots are saved; the current one and the one before. Older snapshots are deleted but log files are saved.

Future tests / updates
The malware detection results are updated as soon as new samples become available. Every time an application is tested or re-tested Snapshot 1 is restored and a brand new copy of the product is downloaded, installed, configured and updated.

Results
Applications are tested on the following:
– On-Demand scan detection.
– On-Access scan detection (for top performing products, when possible).
– Clean-up results.

A malware sample or trace file is marked as detected in the PDF if the logs, quarantine and/or notifications confirm this. At least one file in every folder must be detected (Leverage =1= > original sample or the duplicate sample with file extension when present). In the case of unarchived contents (Leverage =1= Files) all malicious files must be detected. Both the On-Demand and On-Access scans are verified this way. The Clean-up results are pretty straight forward, an infection was cleaned up or it wasn’t. Quarantine, logs and notifications are used to determine this.