McAfee error triggers massive manual PC clean-up

News reports suggest thousands of Windows PCs in large organizations around the globe were thrown into a fit of rebooting yesterday after McAfee distributed a routine antivirus update carrying an egregious error.

Now each one of those computers will have to be manually cleaned. Affected organizations can expect to expend a minimum of 30 minutes of manual labor per PC to get each one back into working order, security experts say.

“There’s no way to automate the process,” says Amrit Williams, CTO of security management system company Big Fix. “It will take however long it takes to touch each single machine. The companies affected by this could be dealing with this for days or weeks.”

In a blog posting late Wednesday, McAfee executive vice president Barry McPherson said “less than one half of one percent of our enterprise accounts globally” were affected. “McAfee teams are working with the highest priority to support impacted customers,” he says.

The incident unfolded after McAfee somehow classified a well-known, legit Windows operating system file, called svchost.exe, as a malicious program. Svchost.exe has long been a crucial part of the Windows operating system. Without it, a PC cannot be networked with other PCs.

Legit files like, svchost.exe, can get intermingled with the tens of thousands of slightly different variants of malicious programs antivirus researchers cull through each day, says Immunet CEO Oliver Friedrichs. “It doesn’t help that some viruses actually masquerade as svchost.exe, leading to confusion and the submission of the legitimate svchost.exe process for analysis,” says Friedrichs.

But quality assurance testing processes are  well developed and most of the time prevents antivirus companies from designating legit files as a “false positive,” that ends up quarantined or scrubbed out. “As for why the false positive was not detected during quality assurance, McAfee will have to answer that,” says Friedrichs. “I can definitely sympathize with McAfee. Nobody wants to have this problem while striving to protect people.”

McAfee declined to answer questions, instead directing reporters to McPherson’s post. Here’s what unfolded at about noon Pacific on Wednesday, 21April2010:

As it does several times each week, McAfee sent updated virus signatures to its corporate clients. This is all part of a time-honored cat-and-mouse game in which hackers create slightly different new versions of viruses, thousands of new variants each day. Antivirus companies compete against each other to be the first to detect the latest variants. They then  hustle to create fresh virus “signatures,” then push out these protective  signatures to corporate customers.

A standard test — running the update on an in-house Windows PC — should have caught the glitch, says Big Fix CTO Williams. He  should know. Williams  worked at McAfee through 2001 and says he helped develop basic quality assurance tests for signature updates. “It’s very basic testing, not something weird or intricate,” says Williams. “The fact that McAfee didn’t see this as part of normal testing is really shocking.”

Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly  determine that the poisonous update from McAfee  threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that “remediation time is estimated  to be 30 minutes per user, ” says Steve Shillingford, CEO of Solera.

“Estimating $100 per hour, this organization’s lost time alone can be conservatively estimated to cost more than $2.5 million,” says Shillingford. “And that does not factor in lost productivity while users are down.”

Security experts say false positives are impossible to completely eliminate in the frenetic cat-and-mouse world of antivirus  protection. McAfee’s gaffe suggests traditional antivirus signature protection may be at its limits, says Ashar Aziz founder and CEO of network security firm FireEye.

“While I’d like to say this is an anomaly, this has happened to several other antivirus vendors and the problem is that antivirus is an antiquated technology that is requiring them to literally process tens of thousands of malware daily,” says Aziz.  “What we are seeing is that this technology framework is collapsing under the weight of maintaining a broken signature approach to security.”

by Byron Acohido

April 22nd, 2010 | For consumers | For technologists | Top Stories