Letting competent people do their jobs

Firs of all - the usual disclaimer applies - this is my personal opinion, blah, blah

The first positive comment to my VirusTotal uploader came in which is cool, however it brought up two issues:

The fist would be: please don't use this tool to scan your entire collection, performing a small DoS attack on VirusTotal. It was written to be as gentle as possible to the service including:

• no multithreading, samples are submitted one by one
• it waits until the previous sample is fully scanned before it moves on to the next sample
• it uses a custom user agent string, so that VirusTotal can filter it / prioritize it if they wish

However the main topic of this post is the idiotic test (if you can call it that - it was more a marketing spin) carried out by Untagle. If you didn't hear about it yet, the gist of it was: pull out around 30 samples from our a** (one of which was EICAR!), scan them with some AV engines and declare that ClamAV (which coincidently is used in their product) is good enough. This is wrong on so many levels. You can read the a good writeup on the McAfee AVERT blog, however the most infuriating thing (for me) was the constant pondering on the fact that AV testing is not open, AV testing needs to be peer reviewed. My response is:

• Don't try to climb out the s*** hole you put yourself into. You've made some (very) bad moves, now admit to them
• Have you've heart about AV-Comparatives (full disclosure: I have no relation with them)? It is a venue whicg (as opposed to your little show) does tests that are fully independent, recognized industry wide and fully documented (as far as the methodology).
• There has been many claims (including the McAfee blog and this result - generated with my script by a third party) - which seems to be true - that the scanners were misconfigured and the detection rate would have been much higher, would you have taken the time to configure them properly
• Making malware publicly available is stupid at best, illegal at worst

I agree that many AV tests in magazines are completely irrelevant and bogus, but - congratulations - you've managed to make something even less valuable and accurate.

PS. This criticism is not directed towards ClamAV, the open source movement, etc. Its sole target is the Untangle test. ClamAV is a reasonably good AV engine with its main focus being threats which arrive in the inbox (it being more a gateway product rather than a desktop product)