Hacker News
points by swatcoder 3 days ago | 0 comments
Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while.

It totally feels like the former if you watch it find and fix vulnerabilities or other debt-tasks one after another.

But only so many of these "technical debt" issue are little off-by-one errors that get remedied by a surgically narrow patch.

Many enough require some kind of restructuring or redevelopment of their own and it's going to take time to figure out if AI faces the same dilemma as human developers: where each non-trivial change carries the risk of some new issue, ad infinitum.

If that happens to be the case, and AI isn't able to magically produce perfect software in finite time, then what's really happened is that the threat of hostile AI exploiting our technical debt will force us to always be burning and churning code just to remain a moving target.

It's another "we'll see" thing for now, I think.

[–] pessimizer 3 days ago
> Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work?

My problem with even talking about this is that there's absolutely no evidence for it. The idea that coming up with a technology to find holes in the hulls of boats starts with a flurry of holes being found and a bunch of patching to be done does not even hint at the theory that there are an infinite number of holes in all boats, and now that we can find them we will have to spend the rest of our lives patching them.

Code was bad, and we knew it was bad. We would have found most of these bugs if instead of coming up with LLMs, a bunch of countries got together and decided that they were going to put $10B into hiring programmers to do nothing but review all the code in the world for bugs. With AI, we managed to do it for only a trillion dollars.

[–] intrasight 3 days ago
The original technical debt, which cannot be paid off, was the decision to use a Von Neumann architectures instead of a Harvard architectures for CPUs.
[–] joe_the_user 3 days ago
This argument is much worse than "we should have made functional programming the standard back in 196X" (which might some merit but overdone imo). The Harvard architecture is just a hardware separation of instructions and data - the argument is virtually an urban legend. Hardwave code/data separations are relatively easy, have existed at time but change nothing since dynamic language just makes its code into data and the self-modification is unhindered.

see: https://www.computer.org/csdl/magazine/an/2022/03/09779481/1...

[–] Joker_vD 3 days ago
Separating the call stack (and probably inventing a very specific "the only valid reads/writes in this area are the ones made by the CALL/RET instructions" permission bit for it) from the data stack would make almost all buffer overflows useless for taking over the control flow: the normal STORE instruction simply can't overwrite the return address, period.

Of course, the interpreters would still be vulnerable, but even for them having separate (and distant) stacks make this particular vulnerability much harder to exploit. And again, a generalized technique having different processor instructions for manipulating differently "painted" memory regions would help there too — at least, it would be there so the implementers of the interpreters could reach for it if they decided to.

In the end, of course, nothing can save you from having a program that insists on literally doing eval(fetch(url=input())), or curl "$(read ; printf "%s" "$REPLY")" | "$SHELL", or something else to this effect.

[–] victorbjorklund 3 days ago
Explain? (Genuine question because I wanna know)
[–] GTP 3 days ago
Basically, have an explicit distinction between data and code at the architectural level, to enforce "Write xor Execute" in hardware. In this way, you would prevent e.g. a buffer overflow letting an attacker injecting shellcode that gets executed later by somehow pointing at it. AFAIK the issue would still remain with some programming languages though.
[–] sznio 3 days ago
don't we already have the NX bit?

and it's not sufficient anyways. even with NX you could have the attacker control a function pointer you call. in any sufficiently complex program you'll have enough widgets to build anything you want just out of data.

code and data are the same thing. whatever inputs your system receives is "code" that your application interprets, just like your application is the code that the CPU interprets. turtles all the way down.

[–] intrasight 3 days ago
In a Harvard architecture, it's not possible to point the CPU at data.
[–] TeMPOraL 3 days ago
But that's just irrelevant confusion of terms.

There is no fundamental distinction between code and data. This separation does not exist as category in actual reality (and I vividly remember that being one of the bigger points GEB made, early on). They're fundamentally one and the same, and classifying some bits as data and some as code is a point of view that really depends on what you're doing and why.

As for Harvard architecture - every time you branch on your "data", you're effectively executing it, and your "code" can be seen as emulating a system where your "data" is the actual code.

[–] rootlocus 3 days ago
So JIT is not possible on a Harvard architecture?
[–] mxey 3 days ago
What about return oriented programming?
[–] ihavesomeexampl 3 days ago
But.. alter data that gets executed (interpeter?), I mean, how does this work even in theory? All data is potentially code?
[–] AdamH12113 3 days ago
To clarify what intrasight is talking about: A Harvard architecture CPU has two physically separate memory busses for code and data. This makes it possible to build a computer with a program memory that cannot be altered through the data bus. In practice, this isn't normally done -- even on Harvard architecture microcontrollers, the flash and RAM are connected to both busses because memory is expensive. And having inalterable program memory doesn't save you from stack-smashing attacks that alter a return address.
[–] intrasight 3 days ago
All data is potentially code only in a Von Neumann architecture, but not in a Harvard architecture, where there's a separation of data and code with different memory access to each.
[–] esterna 3 days ago
Code and data being equivalent is a consequence of Turing completeness; you can always write a Python interpreter, point it at "data" and have it execute that via interpretation.
[–] lelanthran 3 days ago
> All data is potentially code only in a Von Neumann architecture, but not in a Harvard architecture, where there's a separation of data and code with different memory access to each.

Says you. I wrote a program for a Hardvard architecture chip (PIC) that read bytecode from the data memory and executed it. Updates were made by replacing the data block with a new data block.

In practice, there is no way to separate data and instructions. After all, JS downloaded inside a webpage is literally data, and it gets executed.

[–] AlienRobot 3 days ago
How do you turn text into code in this architecture? Through the kernel?
[–] SSLy 3 days ago
you don't. I also believe that JITs would be impossible under strict Harvard.
[–] AlienRobot 3 days ago
Wait, how do you create programs then? How do you compile?

That doesn't even make sense, to be honest. Interpreted languages like Python aren't running machine code. They are taking data and executing machine code based on the data. Is Python safe?

[–] thephyber 3 days ago
According to Wikipedia, Modified Harvard Architecture is a common thing and appears to have properties of both Harvard and Von Neumann architectures. It's probably worth discussing a specific device, OS, and programming language rather than having a vague discussion where both parties are constantly describing different goalposts.

It's not that hard to imagine that in a specific embedded device the kernel / compiler have access to write to the instruction memory space while user space programs don't have any API / permissions to do that.

[–] SSLy 3 days ago
My understanding is that all hard-Harvards are special-purpose ROM-driven systems with static payloads developed on von Neumann computers.
[–] AdamH12113 3 days ago
This sort of thing is much more common in microcontrollers, where the code is cross-compiled on another computer and written to flash memory through a debug interface.

In practice, it's usually possible to write to the embedded flash memory via software, but it's much more difficult and disruptive than a simple CPU memory access.

[–] Merad 3 days ago
A compiler writes machine code do a file. The OS then loads the code from the file into executable memory before it runs the program. The compiler isn't actually trying to execute the machine code it produces. This is why a JIT wouldn't work - the JIT bypasses the interpreter and produces machine code _that executes directly on the CPU_ inside of the same app.

I don't know any details about Python's architecture, but it's totally possible to write an interpreter for machine code; that's basically what an emulator is. The trade off is that pure interpreters tend to be very slow.

[–] tosapple 3 days ago
and we don't believe a sterilized virtual machine creates a similar situation?

any respective machine doesn't require full Turing completeness?

any ad-hoc machine allows for mutations and flexibility, which I'm sure is an added strength.

2nd edit,

Also, didn't we have a weaker model of this with segmented non-flat memory models on i386-

[–] layer8 3 days ago
That won’t help with Lisp programs. ;)
[–] intrasight 3 days ago
Good point, but not much infrastructure runs on Lisp
[–] imtringued 3 days ago
Modern CPUs already split code and data between two types of L1 caches. So they are technically both "Von Neumann" and "Harvard" simultaneously.

It's also missing the point because modern buffer overflow exploits no longer write code into the stack, they overwrite the return address on the stack and assemble jumps to the end of functions known as "return oriented programming", but the underlying issue still applies to function pointers on the heap.

[–] 01HNNWZ0MV43FF 3 days ago
PHP would still be a C interpreter even if you disabled JITs.

Really, we never should have left the Garden of Eden. Things were great there, other than the maternal mortality rate.

[–] sfink 3 days ago
Given that there were no mothers in the Garden (other than arguably Adam who gave birth from his rib), the maternal mortality rate was undefined behavior (0/0). Which checks out -- what did they do during all their time in the Garden, anyway? I guess now we know that we cannot know, and that Lilith must have been of the nasal variety.
[–] hashmap 3 days ago
the equilibrium state would be having found a good pattern for making patterns good. we probably aren't too far from that, so i think your worry may show up in the short term but not the long one
[–] worldsayshi 3 days ago
You're thinking of provable code? Or something else?
[–] hashmap 3 days ago
that would do it too, and i suppose people are working on that. but even without that, there arent infinite low hanging fruit problems to find, and any given code can just sort of be play-doh'd into other code. so the llms will need a few idk years worth of rounds of being trained on the things that pop up that wed never have the time to harden, and things will settle. like, its hard patching stuff up quick right now but we're managing and society hasnt fallen in to chaos.
[–] asa123 3 days ago
it appears they are thinking of a universal 'framework' for all things we may consider insecure as of present
[–] vjvjvjvjghv 3 days ago
"Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while."

Yes, let's fix the foundation. It will make adding features easier once the debt is paid off. And we will probably learn how to do better in the future. It reminds me of settings compiler warning to be treated as errors. Once you do that, you have a ton of work to do. But if you are smart (not all devs are), you learn why the warnings are there and avoid these problems in the future.

[–] khafra 2 days ago
> It will make adding features easier once the debt is paid off.

I'm not so sure: DJB-style secure microkernel/microservice architectures are hard to extend, and I have little hope that today's monolithic code bases are securable without retreating to a small, proven, and well-tested TCB--even by ASI.

[–] keybored 3 days ago
Max AI-glazing perspective in the title, as expected. We need to fix bugs that no mere humans could successfully find and exploit. Thankfully we have AI that can fix those bugs that wouldn’t be a problem without superhuman code agents.

But there are potential upsides like the already mentioned possible nation state exploits (sleeper exploits).

[–] naasking 3 days ago
They would absolutely be a problem at some point, the timeline is just uncertain. That means defenders have to fix them all just in case.
[–] keybored 2 days ago
That’s a legitimate and sensible opinion to have. But FYI I wasn’t inventing my hyperbolic retelling about bots-protecting-from-bots. See paragraph “Jana says actually exploiting the bug requires carefully working out a way to jump to a specific point in the memory for a particular program running on the server”.