METR
@METR_Evals · Berkeley, CA · joined 26 Sep 2023
We work to scientifically measure whether and when AI systems might threaten catastrophic harm to society. Nonprofit.
54 584Followers
39Following
684Posts total
7.2MViews on collected posts
Последние посты
Ищем новые посты…
We thank OpenAI for facilitating conversations with staff and providing datasets, including ~1,300 agent transcripts (focused on activity in July 7-13) with raw chain-of-thought reasoning. This sets an excellent precedent for independent investigation of misalignment incidents.
58.4K views · 521 likes
· 10 reposts · 18 replies
Open on X →
Virtually all data we analyzed was from July 7-13. In OpenAI’s recent Black Hat presentation, they describe that agents had been using unsanctioned message boards in training since May, and that the compromise of OpenAI’s own infrastructure continued past July 13, 2026; these
43.4K views · 397 likes
· 7 reposts · 3 replies
Open on X →
The full report has much more information than we could convey here, including details on the projects the agents collectively pursued, the technologies they developed for communication and coordination, and interactive figures analyzing agent activity:
https://t.co/g4XwUwIt9T
57.9K views · 510 likes
· 41 reposts · 7 replies
Open on X →
The primary model involved in the attack was an internal model we call “highly-persistent internal model” (HPIM) in the report. GPT-5.6 Sol was also involved; our evidence suggests it accounted for roughly 5% of the activity.
49.2K views · 361 likes
· 10 reposts · 5 replies
Open on X →
We analyzed agents’ reasons for joining the attack in their CoT. The most common was to learn how the ExploitGym scorer works in order to trick or tamper with it. Other rationales included finding specific task solutions and obtaining shared infrastructure or credentials. https:/
95.4K views · 415 likes
· 18 reposts · 4 replies
Open on X →
One agent, PHASEONE[big], orchestrated a significant fraction of this cheating research. PHASEONE10841 passed along its work to PHASEONE[big], which had the same task but a larger budget. PHASEONE[big] was “poisoned” so it was particularly interested in tampering with logs. https
94.8K views · 465 likes
· 19 reposts · 5 replies
Open on X →
Some agents explored ways to access Hugging Face. One agent (38148c) found HF credentials and later designed a malicious dataset upload to get the HF server to share unrelated files. Within hours, 100s of agents were using this to obtain data and try to acquire deeper access. htt
52.1K views · 361 likes
· 13 reposts · 1 replies
Open on X →
Despite efforts to manipulate transcripts, agents only rarely seemed motivated to deceive humans. We ran a sweep looking for this, and a representative example of the most severe cases we found was an agent writing a malicious pull request with a misleading description.
60.4K views · 397 likes
· 14 reposts · 5 replies
Open on X →
Agents also tried to retroactively edit transcripts. They modified some accessible action logs, but these weren’t the source of transcripts (agents realized this but tried anyway). We did not see them successfully edit the source of truth in our transcripts (scoped to Jul 7-13).
63.4K views · 393 likes
· 18 reposts · 1 replies
Open on X →
>96 transcripts in our dataset (>7%) showed incorrect tool call outputs due to deliberate “spoofing”. In one case, an agent appears to run echo REAL; sleep. It returns instantly (no sleep) and outputs SPOOFTEST. The spoofs we saw were all easy-to-notice tests like this. https://t
64.8K views · 440 likes
· 12 reposts · 1 replies
Open on X →
For (3), agents pursued a wide range of ways to hide evidence of cheating in their transcripts. They successfully developed a method to pretend to run one command while actually running another (“tool call spoofing”). https://t.co/u8NcA5SSc0
81.7K views · 535 likes
· 28 reposts · 4 replies
Open on X →
To gather evidence in (2), agents created “tripwires” that would send information to the message board about how the scorer works. They recruited “sacrificial” agents to deliberately end their run and submit to trigger the tripwire and generate information for the “collective”.
400.8K views · 1K likes
· 52 reposts · 16 replies
Open on X →
For (1), agents modified their target programs to be easier to exploit & put the modified targets in cache. They then worked on crashing their targets in the hope that a restart would load the modified version from cache. Some agents risked failing their task to try this. https:/
445.9K views · 1.1K likes
· 75 reposts · 20 replies
Open on X →
Agents collaborated on many efforts to make cheats look legitimate, including: (1) swapping the program they had to exploit; (2) manipulating the automated scorer; (3) manipulating transcripts to obscure evidence of cheating. https://t.co/K3n51FfKwC
86K views · 477 likes
· 22 reposts · 4 replies
Open on X →
Based on reading the ExploitGym paper, agents were mistakenly concerned the automatic scorer would fail them if they clearly acquired the flag by cheating. Agents that had seen the reverse-engineered flag were considered “poisoned” because they thought it would disqualify them. h
96.1K views · 522 likes
· 14 reposts · 2 replies
Open on X →
Within a few hours of PHASEONE10841’s initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks. https://t.co/oZ
134.4K views · 528 likes
· 15 reposts · 5 replies
Open on X →
Here we highlight key events from agent transcripts & messages.
An agent that named itself PHASEONE10841 determined its task wasn't solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message.
146.9K views · 635 likes
· 26 reposts · 3 replies
Open on X →
Over July 7-13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined. https://t.co/z7VFKCL2p3
242.3K views · 841 likes
· 55 reposts · 11 replies
Open on X →
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs. htt
4.4M views · 6K likes
· 1.1K reposts · 174 replies
Open on X →
We work to maintain our independence from frontier AI companies, especially as our risk assessments become more consequential. We have not accepted funding from these companies, and we do not accept donations made by or at the direction of their staff. However, frontier AI
59.8K views · 396 likes
· 6 reposts · 3 replies
Open on X →
We are significantly expanding our team and starting new ambitious projects. Join our team to help us realize this opportunity: https://t.co/8XoCmUPJiK.
7K views · 82 likes
· 1 reposts · 0 replies
Open on X →
In the last 6 months, METR raised commitments of around $71 million. This will fund ambitious projects: studying autonomous capabilities, tracking recursive self-improvement, evaluating monitoring systems, conducting risk assessments, investigating AI incidents, and more. https:/
103.4K views · 630 likes
· 48 reposts · 11 replies
Open on X →
Thank you to everyone who has supported METR over the years: The Audacious Project, through which we received our first institutional-scale funding; individuals from Jane Street; foundations like the Sijbrandij Foundation, The Pew Charitable Trusts, Schmidt Sciences and the https
14.5K views · 129 likes
· 2 reposts · 1 replies
Open on X →
@METR_Evals Good! I am sad that the investigation is brief and narrowly scoped. What is the scope and what are the questions you would ideally like to answer but can't? Are you under some sort of NDA about the details of the agreement you have made?
5.2K views · 169 likes
· 3 reposts · 2 replies
Open on X →
OpenAI also plans to publish their own technical report and our findings will inform their analysis.
6.3K views · 165 likes
· 4 reposts · 1 replies
Open on X →
We have reached an agreement with OpenAI to conduct an independent review, with Redwood Research, of the model behavior observed during the Hugging Face incident. We will publish a blog post that describes the terms of our engagement, the scope covered, and tentative conclusions.
248.2K views · 2.3K likes
· 257 reposts · 46 replies
Open on X →
The investigation will be brief and focus on a specific set of questions regarding this incident. In our recent post, we shared a larger set of questions that could be answered in a more comprehensive investigation.
https://t.co/emlQoWndlY
17.8K views · 187 likes
· 9 reposts · 3 replies
Open on X →
We believe it's important to track and investigate misalignment incidents: cases where an AI agent autonomously took sophisticated, sustained actions in violation of human intent. In a new post, we lay out how independent propensity investigations of such incidents could be https
66.1K views · 361 likes
· 55 reposts · 8 replies
Open on X →
На фоне аккаунтов своего размера
25 постов за последние 90 дней рядом с диапазоном 10K–100K подписчиков. показывается широко, но откликается мало кто.
Медианные просмотры64 846этот аккаунт924медиана для 10K–100K
Охват, %118.80%этот аккаунт3.62%медиана для 10K–100K
Вовлечённость, %0.68%этот аккаунт1.52%медиана для 10K–100K
| Показатель | Этот аккаунт | Медиана для 10K–100K | Отношение |
|---|---|---|---|
| Медианные просмотры на пост | 64 846 | 924 | 70.2× |
| Охват (просмотры ÷ подписчики) | 118.80% | 3.62% | 32.8× |
| Вовлечённость | 0.68% | 1.52% | 0.45× |
Другие в этом диапазоне → Сравнить с другим аккаунтом → Как считаются эти ориентиры →
Growth & engagement
How the posts we collected actually performed: views and reaction rate post by post, what the audience did with them, and where the follower count goes.
Views per post
445.9K26 Aug
400.8K
81.7K
64.8K
63.4K
60.4K
52.1K
94.8K
95.4K
49.2K
57.9K
43.4K
58.4K
12.6K27 Aug
Last 14 collected posts, oldest on the left. The scale is logarithmic: one post can outrun the rest a hundred times over.
Engagement rate per post
0.30%26 Aug
0.30%
0.70%
0.71%
0.66%
0.69%
0.73%
0.53%
0.48%
0.79%
1.00%
0.94%
0.95%
2.22%27 Aug
Reactions — likes, reposts, replies and quotes — divided by views. Median for 10K–100K accounts is 1.52%.
What the audience does
Likes66.2%20 702 in total
Reposts6.1%1 911 in total
Replies1.2%366 in total
Quotes3.5%1 081 in total
Bookmarks23.1%7 223 in total
Share of every reaction we collected for this account. Replies mean argument, reposts mean endorsement, bookmarks mean the post was worth keeping.
The follower curve appears once this account has two daily snapshots — we take one a day, and this one is on its first.
Похожие аккаунты
Naval Surface Forces
@SurfaceWarriors
109.2K followers
· 13.1K posts
Dr.Arfana Mallah
@Arfana_Mallah
109.2K followers
· 22.3K posts
James Freeman ✓
@james_freeman__
109.2K followers
· 44K posts
Omna Tigray ✓
@OmnaTigray
109.2K followers
· 12.4K posts
liss ✓
@liss_dolledup
109.2K followers
· 2.8K posts
Er Prashant Pathak (Hindu)
@PrasantPathak_
109.2K followers
· 25K posts
Lana Analise ✓
@peachylanaxo
109.2K followers
· 19.2K posts
Ian Hanomansing
@ianhanomansing
109.1K followers
· 9.5K posts