News reports suggest thousands of Windows PCs in large organizations around the globe were thrown into a fit of rebooting yesterday after McAfee distributed a routine antivirus update carrying an egregious error.
Now each one of those computers will have to be manually cleaned. Affected organizations can expect to expend a minimum of 30 minutes of manual labor per PC to get each one back into working order, security experts say.
“There’s no way to automate the process,” says Amrit Williams, CTO of security management system company Big Fix. “It will take however long it takes to touch each single machine. The companies affected by this could be dealing with this for days or weeks.”
In a blog posting late Wednesday, McAfee executive vice president Barry McPherson said “less than one half of one percent of our enterprise accounts globally” were affected. “McAfee teams are working with the highest priority to support impacted customers,” he says.
The incident unfolded after McAfee somehow classified a well-known, legit Windows operating system file, called svchost.exe, as a malicious program. Svchost.exe has long been a crucial part of the Windows operating system. Without it, a PC cannot be networked with other PCs.
Legit files like, svchost.exe, can get intermingled with the tens of thousands of slightly different variants of malicious programs antivirus researchers cull through each day, says Immunet CEO Oliver Friedrichs. “It doesn’t help that some viruses actually masquerade as svchost.exe, leading to confusion and the submission of the legitimate svchost.exe process for analysis,” says Friedrichs.
But quality assurance testing processes are well developed and most of the time prevents antivirus companies from designating legit files as a “false positive,” that ends up quarantined or scrubbed out. “As for why the false positive was not detected during quality assurance, McAfee will have to answer that,” says Friedrichs. “I can definitely sympathize with McAfee. Nobody wants to have this problem while striving to protect people.”
McAfee declined to answer questions, instead directing reporters to McPherson’s post. Here’s what unfolded at about noon Pacific on Wednesday, 21April2010:
As it does several times each week, McAfee sent updated virus signatures to its corporate clients. This is all part of a time-honored cat-and-mouse game in which hackers create slightly different new versions of viruses, thousands of new variants each day. Antivirus companies compete against each other to be the first to detect the latest variants. They then hustle to create fresh virus “signatures,” then push out these protective signatures to corporate customers.
A standard test — running the update on an in-house Windows PC — should have caught the glitch, says Big Fix CTO Williams. He should know. Williams worked at McAfee through 2001 and says he helped develop basic quality assurance tests for signature updates. “It’s very basic testing, not something weird or intricate,” says Williams. “The fact that McAfee didn’t see this as part of normal testing is really shocking.”
Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly determine that the poisonous update from McAfee threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that “remediation time is estimated to be 30 minutes per user, ” says Steve Shillingford, CEO of Solera.
“Estimating $100 per hour, this organization’s lost time alone can be conservatively estimated to cost more than $2.5 million,” says Shillingford. “And that does not factor in lost productivity while users are down.”
Security experts say false positives are impossible to completely eliminate in the frenetic cat-and-mouse world of antivirus protection. McAfee’s gaffe suggests traditional antivirus signature protection may be at its limits, says Ashar Aziz founder and CEO of network security firm FireEye.
“While I’d like to say this is an anomaly, this has happened to several other antivirus vendors and the problem is that antivirus is an antiquated technology that is requiring them to literally process tens of thousands of malware daily,” says Aziz. “What we are seeing is that this technology framework is collapsing under the weight of maintaining a broken signature approach to security.”
by Byron Acohido
Discuss this Article
4 Comments on "McAfee error triggers massive manual PC clean-up"
Interesting point about how this relates to perhaps unanticipated soft spots in cloud computing.
I have bad experience about svchost.exe deleted by auto and my PC need re-installed operating system.
Now I not use McAfee anymore
*sorry my english bad