Malware test results – Sample and PDF changes
I have received a good amount of feedback in answer to my previous post. There is plenty of work still to be done but I have made some changes to the malware sample set and the PDF. These changes are:
– Two linux malware samples were removed. A few products benefitted from this and now have slightly higher detection rates.
– The Tetracycle (2005) sample was removed as it was pointed out to me it’s not an OS X but an OS 9 sample. The sample set goes back pretty far but OS 9 samples have no place in this test. (Thanks F-Secure for pointing this out)
– Some samples are now listed under Potentially Unwanted Applications (PUA). This was done because not everybody agrees on the classification of these samples. Adware, spyware and key loggers will now be in the PUA list. I personally think detection of these samples is important so even though they were reclassified they still count to the total detection percentage. (Thanks Thomas Reed for the feedback and suggestion)
– As updating this PDF is usually done over the course of a few days you can now see exactly when each product was updated. This is mentioned underneath the percentages. (thanks Eddie) I aim to update all products in the same percentage category on the same day.
– With all this shuffling around I re-counted all results to make sure nothing got messed up. Minor tweaks were made but nothing got majorly affected (just my eyes). I did find I miscalculated Comodo’s percentage after adding the new samples on October 9th. After re-counting their result was adjusted from 93.4% to 86%
– X-Protect has been removed from the PDF completely. I added it at first to show how poorly OS X protects you from malware compared to real AV products but after discussing it with a few people (Thomas Reed being one of them) I now agree it should not be included in this PDF. I do believe that a lot of OS X users out there bypass the default gatekeeper settings either by adjusting it in System Preferences > Security or by right-clicking applications to open them. In these cases a user still has to count on X-Protect as a last line of defense against most malware and X-Protect will not always protect a user because the malware is not recognized or Apple takes a very long time to add it to the list. Excluding X-Protect from the PDF does not change my opinion of AV, I still recommend using it on OS X. Changed my mind again, it’s still in there.
– Three trace files were added based on recommendations from F-Secure. The files are already present as samples but I have also added them to the trace list to replicate where they would have been on a system infected by that malware. One of the files is a .plist from the Boonana malware, it has been added to Users/[USER]/Library/LaunchAgents folder, the other is a .plist file from the Inqtana malware and was added to Users/[USER]/Library/LaunchAgents and /Users/ folders. All three of these files should be detected as samples but it will be interesting to see if they are found in their original locations as well. Ironically none of these samples are actually detected by the F-Secure AV.
– One of the trace files was removed (/.fseventsd/00000000001a0016) as it was only detected by one AV and has not been ever since. It was a false positive or a cache file. Either way it’s no longer relevant. (Thanks Al)
– Added a few more trace files based on detection.
– Until now, samples that are archives or disk images were opened and it’s contents placed in the same folder as the sample. Any hit on that folder counted as that sample being detected. I have now split those samples into parts. The original sample (archive or disk image) and the unarchived or disk image contents. This will more clearly show an AV’s ability to look inside archive or disk image files. These additional files are not counted towards the total number of samples. While it is unlikely an AV will detect one and not the other (though Comodo only detected the unarchived Imunizator sample), it leaves no doubt in the way these samples are tested. I did not do this for the PUA files.
– The virtual machines were updated and adjusted with the above described changes, Java and Flash updates. New VM’s now get 4Gb of Ram instead of 8GB. This allows me to run more of them at the same time.
That’s it for now. All of the top performing AV (80% or higher detection rates) have been updated and the rest of the list will be done in the upcoming couple of days. Thanks to all that provided feedback and comments, keep it coming! 🙂 The PDF can be found here.
While still in in the pdf I see, I disagree with removing X-Protect from the pdf.
I used that specific “key” part of the pdf to change the thinking of the CEO of a company where I work. Our IT guy is a Mac guy and he doesn’t follow security as a whole. He fells Mac’s are safe because Apple has X-Protect to save the day. While so far most are way safer on a Mac these days, I believe Macs have not been targeted by aggressive advanced malware coders. That day will come. Flashback was not even done by a high level malware coder.
Because you had the X-Protect findings in the pdf I could show him statistical evidence that the Macs on our companies network need protection. Many of the Macs are all over the internet surfing news, shopping sites, Google images, ect…, not to mention our company has zero filtering, so third party ad servers are feeding the viewers web pages. In time, this may be a security issue.
No, the X-Protect stats don’t really matter to AV companies, and Mac security geeks, but to anyone else I think it is a “key” statistic that is needed. I have used that exact column to show many co-works that thought Apple was going to save the day and protect them. They were surprised it was so bad.
I am one who thinks you are doing a disservice to take that “key” statistic out of the pdf. You would be losing an over all picture of “security on the Mac” to anyone but the AV companies and the enlightened Mac geek that accepts that OS X IS NOT OpenBSD safe.
The way I test the malware is already ‘worst case scenario’ because applications are set to open from ‘anywhere’. As was pointed out to me this is a user’s choice (as is right-click opening an app). With default gatekeeper settings it should be very hard for any malicious applications to sneak through.
However in the few instances where a malicious app was signed with a valid certificate, X-Protect did not flag the malware as it generally takes Apple a while to disable these certificates.
In the end, the test is an antivirus application test and comparison and X-Protect is not an antivirus application. It’s a built-in ‘aid’ or rudimentary additional warning system.
I believe it’s strenghts lie in the blocking of vulnerable Flash and Java plugins and it’s malware ‘capabilities’ are laughable. Pride probably prevents Apple from admitting Macs are vulnerable but antivirus/antimalware should be handled by the pros.
I may do a follow up post later on to further explain why X-Protect was removed and my opinions of it.