Hacker News
points by Lerc 3 days ago | 0 comments
Have you given thought on how you can use this info to estimate how many vulnerabilities there are?

It will be difficult to claim that the problem has been solved without an idea of how big the problem is.

Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.

Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?

[–] ky1ebot1337 3 days ago
I think it is hard to estimate what we don't know. In our first scan when we found 166 vulnerabilities, we wouldn't be able to guess there were 900+ more vulns to find in the code base. We are in the same situation again: now we find 900+ vulns, are there no more vulns to find, 1k more, or even more than that. I think it'd be an interesting benchmark problem if we can have a formally proved safe, large program as the ground truth, inject synthetic vulnerabilities to it, then ask the model to exhaustively find vulns, and use that as a benchmark results.
[–] asdff 3 days ago
How can they estimate the existence of a vulnerability that has not been discovered and documented? The hubris to assume zero latent factors!
[–] BoppreH 3 days ago
From what I understand, biologists have the same problem when estimating the number of undiscovered species. One technique is to look at very popular, well studied biomes, and extrapolate the results to less studied biomes. Or ask niche specialists to guess different groups of species and add up the results, which is usually better than trying to estimate the population as a whole.

The errors bars will be huge, but it's better than "zero or more".