I have received a good amount of feedback in answer to my previous post. There is plenty of work still to be done but I have made some changes to the malware sample set and the PDF. These changes are:
– Two linux malware samples were removed. A few products benefitted from this and now have slightly higher detection rates.
– The Tetracycle (2005) sample was removed as it was pointed out to me it’s not an OS X but an OS 9 sample. The sample set goes back pretty far but OS 9 samples have no place in this test. (Thanks F-Secure for pointing this out)
– Some samples are now listed under Potentially Unwanted Applications (PUA). This was done because not everybody agrees on the classification of these samples. Adware, spyware and key loggers will now be in the PUA list. I personally think detection of these samples is important so even though they were reclassified they still count to the total detection percentage. (Thanks Thomas Reed for the feedback and suggestion)
– As updating this PDF is usually done over the course of a few days you can now see exactly when each product was updated. This is mentioned underneath the percentages. (thanks Eddie) I aim to update all products in the same percentage category on the same day.
– With all this shuffling around I re-counted all results to make sure nothing got messed up. Minor tweaks were made but nothing got majorly affected (just my eyes). I did find I miscalculated Comodo’s percentage after adding the new samples on October 9th. After re-counting their result was adjusted from 93.4% to 86%
X-Protect has been removed from the PDF completely. I added it at first to show how poorly OS X protects you from malware compared to real AV products but after discussing it with a few people (Thomas Reed being one of them) I now agree it should not be included in this PDF. I do believe that a lot of OS X users out there bypass the default gatekeeper settings either by adjusting it in System Preferences > Security or by right-clicking applications to open them. In these cases a user still has to count on X-Protect as a last line of defense against most malware and X-Protect will not always protect a user because the malware is not recognized or Apple takes a very long time to add it to the list. Excluding X-Protect from the PDF does not change my opinion of AV, I still recommend using it on OS X. Changed my mind again, it’s still in there.
– Three trace files were added based on recommendations from F-Secure. The files are already present as samples but I have also added them to the trace list to replicate where they would have been on a system infected by that malware. One of the files is a .plist from the Boonana malware, it has been added to Users/[USER]/Library/LaunchAgents folder, the other is a .plist file from the Inqtana malware and was added to Users/[USER]/Library/LaunchAgents and /Users/ folders. All three of these files should be detected as samples but it will be interesting to see if they are found in their original locations as well. Ironically none of these samples are actually detected by the F-Secure AV.
– One of the trace files was removed (/.fseventsd/00000000001a0016) as it was only detected by one AV and has not been ever since. It was a false positive or a cache file. Either way it’s no longer relevant. (Thanks Al)
– Added a few more trace files based on detection.
– Until now, samples that are archives or disk images were opened and it’s contents placed in the same folder as the sample. Any hit on that folder counted as that sample being detected. I have now split those samples into parts. The original sample (archive or disk image) and the unarchived or disk image contents. This will more clearly show an AV’s ability to look inside archive or disk image files. These additional files are not counted towards the total number of samples. While it is unlikely an AV will detect one and not the other (though Comodo only detected the unarchived Imunizator sample), it leaves no doubt in the way these samples are tested. I did not do this for the PUA files.
– The virtual machines were updated and adjusted with the above described changes, Java and Flash updates. New VM’s now get 4Gb of Ram instead of 8GB. This allows me to run more of them at the same time.
That’s it for now. All of the top performing AV (80% or higher detection rates) have been updated and the rest of the list will be done in the upcoming couple of days. Thanks to all that provided feedback and comments, keep it coming! 🙂 The PDF can be found here.