I swear most AI booster arguments can work about as well if we replace AI with the Holy Spirit.

(Apologies if this was sarcasm, I genuinely can't tell these days...)

The law neither allows nor disallows clean room reverse engineering. Clean room reverse engineering enables a defendant to logically prove they have not copied the any protected expression, because the implementers have never seen any protected expression.

Not exactly, and that's an unhelpful way to think about it. Copyright law, technically speaking, DGAF about whether you are using an LLM or not. Yes, there has been a lot of talk about whether LLM outputs qualify for copyright protection. And the answer isn't completely straightforward, but the ultimate answer doesn't have anything to do with LLMs -- the law requires a certain modicum of human creativity to qualify for copyright protection. So the real legal answer is that it depends less on whether you are using an LLM and more about how you as a human are using it. That line isn't well litigated yet -- but in general, if you want a copyright, you need to be ready to explain how human creativity was involved -- that's, by definition, the only thing copyright protects.

BUT, this doesn't matter at all in this situation, we're not questioning the copyright protection status of the LLM output, we're questioning whether it violates someone else's IP.

The point of clean room reverse engineering is a defense to prove you could not have violated IP rights of others (not just copyright, but also trade secrets, and patents). The entire point of the exercise is that the implementation team has never seen the original work, so it is impossible for them to have copied the expression. They only could have come to it coincidentally by implementing the idea in the same way. This protects you from a copyright lawsuit because copyright only protects expression, not ideas.

LLMs, are proven to possess the capability to reproduce expression: exhibited as overfitting and/or memorization. So, to form the same defense with an LLM, you'd have to prove that the LLM couldn't possibly be overfitting or repeating from memory. And the best way to do that is to do it the same way it is done with humans: you need to ensure they never saw (i.e. wasn't trained on) the subject of the reverse engineering.

If you don't do this, here's what happens:

You go to court, the plaintiff says "look, this code is basically the same as ours!". The defense says "no, we super duper promise we didn't copy it, we just coincidentally came up with very close to the same thing". They ask "well did your developers ever see our code before?" ...and... <drumroll> the outcome depends on that answer.

It's the exact same situation with an LLM. If it almost exactly produces something that is already copyrighted, the plain simple answer is that the model was overfitting or memorizing the training inputs. You need a believable defense to that.