Hacker News

points by mlmonkey 4 days ago | 0 comments

As someone who has been working in ML for 30+ years, and has designed and trained too many NNs to keep count, this is the first time I've seen anyone twiddling the weights directly. Usually you play with the training set, exemplar weights, model's hyperparameters, etc. I've never seen a trained network being modified by hand.

[–] numeri 4 days ago

Nothing Astra did is even slightly surprising or novel in mechanistic interpretability, which has been a fairly big field for a few years now.

[–] senderista 4 days ago

You are unfamiliar with abliteration?

[–] mlmonkey 4 days ago

There are a 1000 papers published on arXiv in ML every single day. It's hard to keep abreast of everything that's going on in this space.

[–] larodi 4 days ago

ablation is not manual, and comes after evaluating which parts to turn off against given desired result.

[–] prophesi 4 days ago

it is manual, just automated via running inferences on a dataset of assumed refusals, and is a small rank-1 weight edit that doesn't require fine-tuning or training. Perf hit is much smaller than baking in these refusals in the first place.

[–] larodi 3 days ago

small _automated_ rank-1 edit is not manual edit, sorry. automated is the contrary of manual by definition (which is "thinking by itself", definitely not something done by hand (main in french, thus manual)).