HN Reader

Show HN: Llama 3.2 Interpretability with Sparse Autoencoders

305

I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3].

I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback.

Cheers

[1] https://transformer-circuits.pub/2024/scaling-monosemanticit...

[2] https://arxiv.org/abs/2406.04093

[3] https://arxiv.org/abs/2408.05147

For anyone who hasn’t seen this before, mechanistic interpretability solves a very common problem with LLMs: when you ask a model to explain itself, you’re playing a game of rhetoric where the model tries to “convince” you of a reason for what it did by generating a plausible-sounding answer based on patterns in its training data. But unlike most trends of benchmark numbers getting better as models improve, more powerful models often score worse on tests designed to self-detect “untruthfulness” because they have stronger rhetoric, and are therefore more compelling at justifying lies after the fact. The objective is coherence, not truth.

Rhetoric isn’t reasoning. True explainability, like what overfitted Sparse Autoencoders claim they offer, basically results in the causal sequence of “thoughts” the model went through as it produces an answer. It’s the same way you may have a bunch of ephemeral thoughts in different directions while you think about anything.

10 hours agoby foundry27

Incredible, well-documented work -- this is an amazing effort!

Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.

[1] https://arxiv.org/abs/2408.00657

10 hours agoby jwuphysics

Thank you for posting this PaulPauls,

can I please ask a wacky question that I have about mech.interp. ?

we know that when we use a metric to measure humans, we game the metric, I wonder about future ai, gaming mech.interp.

for simplicity let me propose a 2d matrix to encode tokens, and hope my understanding of neural networks is good enough for this to be a valid question I understand that higher dimensions have a lot more 'space', so accidental collisions are easily avoided, but 2d makes my example easier to explain.

if we had the following tokens mapped in 2d space

  Apple 1a
  Pear  1b
  Donkey 2a
  Horse 2b

it becomes impossible to understand if the neurons 1,2,a,b, all excited represents apple and horse or donkey and pear?

I wonder if a future, vastly more competent AI overseeing its own training might use this form of ambiguity as means to create a model in which the weights are selected so the potential for encoding collisions exist, so that it is possible to deceive an mech.int. observer as to the true thoughts of the model, essentially enabling the ai to think in euphemisms?

7 hours agoby monkeycantype

Hey - Thanks for sharing!

Will take a closer look later but if you are hanging around now, it might be worth asking this now. I read this blog post recently:

https://adamkarvonen.github.io/machine_learning/2024/06/11/s...

And the author talks about challenges with evaluating SAEs. I wonder how you tackled that and where to look inside your repo for understanding the your approach around that if possible.

Thanks again!

12 hours agoby curious_cat_163

This seems like decent alignment-positive work on a glance, though I haven't checked full details yet. I probably can't make it happen, but how much would someone need to pay you to make up your time, expense, and risk?

7 hours agoby Eliezer

This is incredible!

PaulPauls, how would you like us to cite your work?

6 hours agoby batterylake

This is awesome! I really appreciate the time you took to document everything!

13 hours agoby jaykr_

Very cool work! Any plans to integrate it with SAELens?

12 hours agoby JackYoustra