Josh Engels

@JoshAEngels

Interp @GoogleDeepMind | on leave from my PhD @ MIT

2 037Followers

141Following

192Posts total

77.4KViews on collected posts

Neueste Beiträge

Josh Engels@JoshAEngels20 Jun

@JoshAEngels Cool research. Is the code available by any chance? Would be awesome to play with it. 157 views · 1 likes · 0 reposts · 0 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Finally, it's unclear if these results are an artifact of current nascent text diffusion training paradigms rather than a lasting property of latent reasoning architectures. We thus hope that our work serves as a template for evaluations of future latent reasoning models. 478 views · 14 likes · 0 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Another neat result is "token smearing": when DiffusionGemma is confident that a token will exist somewhere, but doesn't know exactly where the token will go, it will maintain a "smeared" probability distribution over adjacent positions. https://t.co/XHLeLT1AqX 538 views · 15 likes · 1 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

In one case study, we ask DiffusionGemma to count the number of perfect squares between 400 and 800 and give its answer first followed by the list of squares. The model will guess wrong, list the squares, and then go back and correct its mistake. https://t.co/nVXfez5iIh 378 views · 9 likes · 1 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Next, in a series of case studies, we study algorithmic transparency: whether we can use intermediates to reconstruct the process by which the model arrived at its outputs. We introduce neat visualizations too! I won't get to all case studies here, so go see the paper for more! 415 views · 9 likes · 1 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

We also test monitorability, a key application of transparency that measures whether model outputs are useful for downstream tasks. We find that Gemma and DiffusionGemma are similarly monitorable. https://t.co/FfARuptCHv 858 views · 10 likes · 1 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Furthermore, for the p = 0.03 ablation (one of the ablations with no effect on performance), most tokens are equal or semantically similar to tokens in the final rollout. So the load bearing diffusion model intermediates are mostly (interpretable) guesses for final tokens! https 524 views · 10 likes · 2 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

But all hope isn't lost! The reason that the opaque serial depth is so high is the non-interpretable vector in between denoising steps. We project this vector into token space and restrict to the top few tokens; performance is unharmed; we can look at just these tokens. https:/ 633 views · 11 likes · 2 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

As we expected, DiffusionGemmas' opaque serial depth--a numerical estimate of non-transparent reasoning that measures the length of the deepest path through the model that doesn't go through tokens--is empirically and asymptotically larger than the corresponding Gemma 4 model. ht 909 views · 10 likes · 2 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

In our transparency audit of DiffusionGemma, we: - find no decrease in monitorability - reduce opaque serial depth by applying LogitLens to intermediate vectors - discover weird non-autoregressive phenomena We hope audits like this become standard for new model architectures. 1.2K views · 13 likes · 2 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Paper here! https://t.co/cgMhy0qZbd Work done w/ @calsmcdougall @bilalchughtai_ @JanosKramar @sen_r @cindyxywu @ArthurConmy @AsicChen @jean_tarbou @Sophia_NLP @bodonoghue85 @jglo_liveira @rohinmshah and @NeelNanda5 2K views · 41 likes · 5 reposts · 1 replies Open on X →

Josh Engels@JoshAEngels19 Jun

Text diffusion models are fast, but are less transparent than today's LLMs because they do many forward passes before outputting text. We audit the transparency of DiffusionGemma and find that the intermediates are interpretable. This recovers many of the benefits of CoT! 🧵 htt 69.3K views · 250 likes · 47 reposts · 3 replies Open on X →

Growth & engagement

How the posts we collected actually performed: views and reaction rate post by post, what the audience did with them, and where the follower count goes.

Views per post

69.3K19 Jun

1.2K

909

633

524

858

415

378

538

478

15720 Jun

Last 12 collected posts, oldest on the left. The scale is logarithmic: one post can outrun the rest a hundred times over.

Engagement rate per post

0.44%19 Jun

2.41%

1.30%

1.43%

2.21%

2.48%

1.52%

2.65%

2.91%

3.16%

3.14%

0.64%20 Jun

Reactions — likes, reposts, replies and quotes — divided by views.

What the audience does

Likes57.8%393 in total

Reposts9.4%64 in total

Replies1.9%13 in total

Quotes1.2%8 in total

Bookmarks29.7%202 in total

Share of every reaction we collected for this account. Replies mean argument, reposts mean endorsement, bookmarks mean the post was worth keeping.

The follower curve appears once this account has two daily snapshots — we take one a day, and this one is on its first.

Neueste Beiträge

Growth & engagement

Views per post

Engagement rate per post

What the audience does

Ähnliche Konten