False positives are the real cost of detection
A detector that is right 99.9% of the time can still be wrong about most of what it flags. The arithmetic explains why release discipline matters more than clever rules.
Die Artikel erscheinen auf Englisch.
A security product is judged by what it catches. It is abandoned because of what it wrongly blocks. A false alarm on a business-critical program stops work, and it teaches users to ignore warnings or switch the product off.
The arithmetic of rare events
Take a detector with good figures. These numbers are chosen for illustration:
- it catches 99% of malicious files;
- it wrongly flags 0.1% of clean files;
- one file in 10,000 is actually malicious.
Now scan one million files. About 100 are malicious, and the detector catches 99 of them. Of the 999,900 clean files, 0.1% are flagged, which is about 1,000 false alarms.
So out of roughly 1,099 alerts, only 99 are real. About nine alerts in a hundred are correct, even though the detector sounds excellent on paper. Malicious files are rare, so small error rates on the huge clean population dominate the result.
| Flagged | Not flagged | |
|---|---|---|
| Malicious (100) | 99 | 1 |
| Clean (999,900) | about 1,000 | about 998,900 |
The figure to watch
The useful question is not “what percentage of malware do we detect?” It is “when we raise an alert, how often are we right?” That figure, precision, is what the person handling the alert experiences.
How to control false positives
Test against clean files. Every rule should run against a large, current collection of legitimate software before release, including installers, drivers, developer tools and the unusual programs used in specific industries.
Trust signed software carefully. A valid signature from a known publisher is strong evidence of a clean file, although signing certificates do get stolen, so it should never be an absolute pass.
Roll out in stages. Release a new rule to a small share of machines, in a mode that logs without blocking. Watch what it matches, and only then widen it.
Keep a way back. Every rule needs a fast withdrawal path that does not depend on the update that caused the problem. Security products have taken down large numbers of computers with a bad update, and the recovery is much harder when the machines can no longer boot.
Give users a route. A clear way to report a wrong detection, with a quick response, turns a bad day into a minor incident. Software vendors need the same route for their own products.
Choosing the threshold
Lowering a detection threshold catches more malware and raises more false alarms. Different customers want different points on that curve. A home user wants quiet, while a security team may accept more alerts in return for coverage. A product that lets administrators choose, with sensible defaults, serves both.
Verfasst vom Engineering-Team von Avtim. Dies sind allgemeine Informationen; prüfen Sie sie an Ihrem eigenen System, bevor Sie sich darauf verlassen.