Ai2
@allen_ai · Seattle, WA · joined 04 Sep 2015
Breakthrough AI to solve the world's biggest problems. › Join us: https://t.co/MjUpZpKPXJ › Newsletter: https://t.co/k9gGznstwj
86 145Followers
448Following
3 908Posts total
197.7KViews on collected posts
Latest posts
Looking for new posts…
@allen_ai @GoodfireAI Open post-training is the only way to squeeze margin without buying more chips. most models are just expensive parrots now. efficiency wins.
124 views · 1 likes
· 0 reposts · 0 replies
Open on X →
@GoodfireAI In one experiment, preference training improved Olmo’s general capabilities but also made it more likely to answer some harmful requests framed as fiction or hypotheticals.
Goodfire traced part of the regression to specific Dolci preference pairs. https://t.co/ui9zzm
735 views · 6 likes
· 0 reposts · 1 replies
Open on X →
@GoodfireAI This is why publishing more than model weights matters: researchers can trace unexpected behavior back to the data behind it, test targeted changes to training, and measure whether those fixes work without weakening the model elsewhere. https://t.co/OWUFdhyAWw
721 views · 9 likes
· 1 reposts · 0 replies
Open on X →
@GoodfireAI The problem: teams often discover unwanted changes in model behavior only after training is finished.
Then they have to work backward from eval scores to guess which of hundreds of thousands of training examples caused them.
736 views · 9 likes
· 0 reposts · 1 replies
Open on X →
@GoodfireAI Using that stack, Goodfire developed “predictive data debugging”: a way to estimate which behaviors preference training will strengthen or suppress before committing compute to a full run. https://t.co/GdjfAX4FxF
246 views · 8 likes
· 0 reposts · 1 replies
Open on X →
@GoodfireAI Goodfire wanted to move that debugging earlier. Ai2’s open stack made it possible.
Our Dolci dataset provides Olmo 3’s preference data, Olmo includes intermediate checkpoints + reproducible recipes, and OLMES measures changes in model capabilities.
489 views · 8 likes
· 0 reposts · 1 replies
Open on X →
Training an LLM by showing it answers people prefer – and ones they don’t – can improve it while quietly worsening behaviors.
@GoodfireAI used our open post-training stack to predict how a full training run would change responses to different prompts. 🧵
https://t.co/KZV2gFZMaX
16.7K views · 72 likes
· 6 reposts · 2 replies
Open on X →
@allen_ai BenchMIRT showing BBQ tracks reasoning over safety is a useful eval audit
6 views · 0 likes
· 0 reposts · 0 replies
Open on X →
BenchMIRT gives researchers a clearer view of what benchmarks really measure—and could help build evals that are smaller, more focused, & easier to interpret.
We’re releasing it openly so others can build on it:
💻 https://t.co/5lnoifXMMK
📄 https://t.co/BkSXP5W5ds
809 views · 7 likes
· 0 reposts · 0 replies
Open on X →
BenchMIRT can also estimate how a model will perform on Qs it hasn’t answered.
For held-out questions, it correctly predicts whether the model will answer correctly 79% of the time compared with 70% for a simpler benchmark-average baseline.
728 views · 3 likes
· 0 reposts · 1 replies
Open on X →
We applied BenchMIRT across the 16 benchmarks it was trained on to see whether we could make evals more efficient by removing less informative questions.
We found keeping just the strongest 10% of Qs preserves nearly the same picture of model strengths as using the full set.
138 views · 2 likes
· 0 reposts · 1 replies
Open on X →
BenchMIRT works at multiple levels:
◙ For models, it estimates strength on the capabilities reflected in the benchmark set.
◙ For questions, it estimates difficulty & how strongly each Q distinguishes models along those capabilities.
177 views · 2 likes
· 0 reposts · 1 replies
Open on X →
We trained BenchMIRT on results from 100 LLMs across 16 benchmarks & 34K+ questions.
We didn’t tell it which evals measured what. Two dominant dimensions consistently emerged: general reasoning + safety. https://t.co/GNMJkqAiVq
428 views · 2 likes
· 0 reposts · 1 replies
Open on X →
BenchMIRT builds on Item Response Theory (IRT), a technique from psychometrics for measuring abilities from patterns of test responses.
The idea: not every question tells you the same amount. Some are harder; some better distinguish stronger models from weaker ones.
623 views · 4 likes
· 0 reposts · 1 replies
Open on X →
Do LLM safety & capability evals measure what they claim to?
We built BenchMIRT to audit them + see which model abilities their Qs actually test.
On BBQ, a social-bias eval, it found the Qs distinguished models more by reasoning ability than safety. 🧵
https://t.co/BQ8WPTu5nR
14.7K views · 55 likes
· 9 reposts · 4 replies
Open on X →
At an event on August 27, we brought together AI researchers, scientists, & medical practitioners to explore what AI needs to do better to meaningfully advance science.
Five ideas kept coming up. 🧵 https://t.co/vmqcsXq1EZ
3.3K views · 14 likes
· 2 reposts · 2 replies
Open on X →
@allen_ai @ProvSwedish Is it me or every AI2 post seems so impactful?
47 views · 2 likes
· 0 reposts · 0 replies
Open on X →
@ProvSwedish This collaboration highlights our approach to AI for science: not replacing scientific judgment, but working alongside scientists to expand what they can explore.
Providence Swedish will now deploy AutoDiscovery on additional protected cancer data. https://t.co/lJvg
672 views · 0 likes
· 0 reposts · 0 replies
Open on X →
@ProvSwedish The signal involved invasive lobular carcinoma (ILC), a subtype affecting ~48K Americans each year.
ILC has long been considered “immune cold,” with relatively low immune activity. AutoDiscovery found more activity than expected.
552 views · 1 likes
· 0 reposts · 1 replies
Open on X →
@ProvSwedish Providence Swedish researchers ran AutoDiscovery over The Cancer Genome Atlas (TCGA), a resource built from donated cancer patient samples.
AutoDiscovery generated + tested hypotheses across the data, searching for results that challenged expectations.
437 views · 1 likes
· 0 reposts · 1 replies
Open on X →
@ProvSwedish One was the ILC immune signal.
The researchers confirmed the same pattern in separate patient data. Lab analysis subsequently verified it, finding T-cells around ILC tumors.
128 views · 0 likes
· 0 reposts · 1 replies
Open on X →
@ProvSwedish Together, the results suggest ILC warrants broader study for immunotherapy.
Read more in the researchers’ scientific report: https://t.co/Kt6Ig1rW0u https://t.co/BePGP621V8
716 views · 1 likes
· 0 reposts · 1 replies
Open on X →
AI’s biggest role in science may not be answering questions. It may be helping scientists find which questions are worth asking.
At @ProvSwedish, AutoDiscovery surfaced an unexpected immune signal in a heavily studied cancer dataset—and follow-up research confirmed it. 🧵 https:/
136.5K views · 53 likes
· 9 reposts · 11 replies
Open on X →
1:25
@allen_ai @GeorgiaTech Attribution back to data mixtures is the part most capability reporting skips. We would love to see how stable those traces are across seeds and checkpoints, since that is where the practical guidance lives.
70 views · 1 likes
· 0 reposts · 0 replies
Open on X →
@GeorgiaTech That’s why we release more than model weights.
With the full Olmo scientific stack open, independent researchers can investigate where model capabilities come from, test those connections experimentally, and build on the results.
703 views · 5 likes
· 0 reposts · 0 replies
Open on X →
@GeorgiaTech None of this would have been possible with a closed model.
The team needed access to Olmo 3’s training data and infrastructure to trace its behavior back to the data that drove it. https://t.co/JXAyD8lMCM
754 views · 4 likes
· 0 reposts · 1 replies
Open on X →
@GeorgiaTech They uncovered some interesting patterns.
For example, dialogue-rich, interpersonal writing was more influential on Olmo’s reasoning performance than on its answers to factual knowledge questions. https://t.co/AbnAjaWvtU
211 views · 7 likes
· 0 reposts · 1 replies
Open on X →
@GeorgiaTech The Georgia Tech team set out to test whether they could trace different model abilities back to the kinds of text that contributed to them.
Olmo 3 made that possible because its full pretraining corpus, Dolma 3, is public.
524 views · 4 likes
· 0 reposts · 1 replies
Open on X →
@GeorgiaTech LLMs train on huge mixtures of text.
When a model gets better at figuring out what a user is thinking or trying to do, it’s often hard to know whether that ability came from literature, technical writing, or some other part of its training data.
886 views · 3 likes
· 0 reposts · 1 replies
Open on X →
@GeorgiaTech The team sampled millions of files from Dolma, then used influence functions – which estimate how much each file influenced a model’s answer – to see which types of files mattered most across tests of social/general reasoning, plus social-science & STEM knowledge.
206 views · 2 likes
· 0 reposts · 1 replies
Open on X →
Against accounts of the same size
31 posts from the last 90 days, next to the 10K–100K follower range. below its peers on both reach and engagement.
Median views623this account924median for 10K–100K
Reach, %0.72%this account3.62%median for 10K–100K
Engagement, %0.80%this account1.52%median for 10K–100K
| Metric | This account | Median for 10K–100K | Ratio |
|---|---|---|---|
| Median views per post | 623 | 924 | 0.67× |
| Reach (views ÷ followers) | 0.72% | 3.62% | 0.20× |
| Engagement rate | 0.80% | 1.52% | 0.53× |
Others in this range → Compare with another account → How these benchmarks are built →
Growth & engagement
How the posts we collected actually performed: views and reaction rate post by post, what the audience did with them, and where the follower count goes.
Views per post
6231 Sep
428
177
138
728
809
63 Sep
16.7K9 Sep
489
246
736
721
735
124
Last 14 collected posts, oldest on the left. The scale is logarithmic: one post can outrun the rest a hundred times over.
Engagement rate per post
0.80%1 Sep
0.70%
1.69%
2.17%
0.55%
0.87%
0.00%3 Sep
0.52%9 Sep
1.84%
3.66%
1.36%
1.39%
0.95%
0.81%
Reactions — likes, reposts, replies and quotes — divided by views. Median for 10K–100K accounts is 1.52%.
What the audience does
Likes61.8%286 in total
Reposts5.8%27 in total
Replies8.0%37 in total
Quotes3.2%15 in total
Bookmarks21.2%98 in total
Share of every reaction we collected for this account. Replies mean argument, reposts mean endorsement, bookmarks mean the post was worth keeping.
The follower curve appears once this account has two daily snapshots — we take one a day, and this one is on its first.
Similar accounts
Skyroot Aerospace ✓
@SkyrootA
172.3K followers
· 911 posts
SarkNatives Ghana
@SarkNativesGH
172.3K followers
· 100.1K posts
TL Pobelter ✓
@Pobelter
172.3K followers
· 6.3K posts
🦊 GitLab ✓
@gitlab
172.3K followers
· 42.1K posts
NYC DOT
@NYC_DOT
172.3K followers
· 46.8K posts
Gouri G Kishan ✓
@Gourayy
172.3K followers
· 1.2K posts
Sarah Sizzle ✓
@sizzle_sarah
172.3K followers
· 57.7K posts
FAR Labs ✓
@FARLabsAI
172.3K followers
· 11.2K posts
Frequently asked questions about @allen_ai
How many followers does @allen_ai have?
86 145 followers.
How many posts has @allen_ai published?
3 908 posts.
How do posts by @allen_ai perform?
Collected posts: 31. Views: 197 748. Likes: 399.