So far OpenAI's security work has focused on identifying vulnerabilities in open-source code. It occurs to me that using LLMs to scan binaries for supply chain issues and implants, and publicizing the results/certifying particular hashes as safe, could also be very valuable.

Presumably, the barrier to sophisticated supply chain attacks is falling rapidly due to LLM agents. Such attacks could even be utilized by a rogue AI. OpenAI could easily develop internal expertise in detecting such attacks via the following method:

* Train an LLM agent to create binaries with subtle backdoors/bugdoors

* Train another agent which ingests data in the form of clean binaries and poisoned binaries, and learns to tell the difference

* Train in a loop, where the first agent "wins" if it fools the second agent with an implant judged useful by a 3rd un-incentivized agent; the second agent "wins" if it's able to detect increasingly obfuscated backdoors/bugdoors

Over time you could train a small model for high performance on this task, which continuously scans open-source package repositories, firmware, OS release images, etc. flagging issues, and certifying hashes as free of apparent malicious code. To save on hardware resources during continuous monitoring, it could also learn to scan binary diffs. A certification API could then be exposed for use by maintainers of apt-get and other package tools.

This could be excellent as a goodwill project to help prevent security apocalypse and harden against rogue AI.

Originally had this idea in this subthread: https://news.ycombinator.com/item?id=50379912