Hacker News
points by svachalek 3 days ago | 0 comments
This doesn't necessarily require proving everything mathematically, it also helps to design things in a way that doesn't rely on discipline to avoid gaping holes. For example, languages that allow pointer arithmetic, unchecked array access, or use-after-free have huge opportunities to make security errors that don't exist in languages without them.
[–] hinkley 3 days ago
A problem I frequently have with setting up testing policies is in convincing people to test what they can reliably test with the tools, and know what they cannot test.

Inconclusive tests to increase automation and coverage accomplish less than nothing. The opportunity cost, the loss of trust, the conflation of the manual testing surface area, are all worse than doing nothing.

The point of testing is to establish a level of trust. That level doesn’t need to be 100%. It can’t be 100% regardless. So you do what you can, and you solve the remaining problem a different way. But you need to know it’s there to address it, and the smoke and mirrors fight that.

[–] annzabelle 3 days ago
I used to work for a company that enforced 80% code coverage for our CI/CD pipelines. Both teams I worked with there, in wildly different parts of the company, were playing games with excluding most files in the repo and writing dozens of tests that didn't actually test anything, just ran the code in the testing container so the lines would count as having been tested. I think LLMs have probably cleaned it all up a lot in the end, because the tests they write do actually test the code (against the code at the point you told the LLM to write the test a lot of the time, rather than a third party spec, but at least it's not just return true all the time).

Every 6 months there was a new testing initiative, which seemed to all be responded to by writing wrappers that returned true to the test automation system we were ordered to use.

[–] hinkley 2 days ago
I’ve seen that behavior but it usually doesn’t kick in until the mandate 90%, so that’s a shame. Usually I push back on it being a rule because if you have low coverage, you have a team that’s not good at automated testing, and forcing them to write a lot of tests before they’ve gotten good at it always ends poorly. But typically nobody listens and the shitshow begins.
[–] latentsea 3 days ago
Seems like a culture problem at both ends of the org.
[–] hinkley 2 days ago
It’s a pretty common failure mode in my experience. Keep management out of testing as long as you can.
[–] latentsea 2 days ago
Keep shitty tests out. PRs with vacuous tests should be rejected. You can have high coverage and good tests. These aren't mutually exclusive.
[–] gopher_space 1 day ago
Demonstrate the good vs bad test cases to people by doing an obviation pass over your test suite. Lots of things to look for, like misunderstanding the flow of data authority or testing against things that are fundamentally Somebody Else's Problem.

A base exercise in this vein would be asking someone to describe how a failure state they're checking for could come to exist in your current system.

[–] skydhash 3 days ago
> For example, languages that allow pointer arithmetic, unchecked array access, or use-after-free have huge opportunities to make security errors

All languages are built on top of an ISA, whether directly (assembler), or by mapping it to an abstract machine. The you have the dreaded “eval” function. Then because a lot of software lives on top of an OS, there’s all the virtualization that is going on (timesharing, virtual memory, filesystems,…). Memory safety is nice, but it doesn’t help you when the attacker send valid, but malicious data from a compromised source, or by exploiting a logic error.

And for kernel code, where everything is accessible to everyone, it’s much a matter of discipline, especially with so much hardware interrupts and syscalls from userspace going around. Most kernel code has very short lifecycle, it’s the data that is long lived, and it’s a shapeless blob in memory that you have to trust the other party kept in good shape.

[–] eru 3 days ago
Using these languages is pretty close to a mathematical proof.