Latest AI safety and alignment news — research on making AI systems reliable, interpretable, and aligned with human values. Coverage of alignment research, governance, and policy.
AI safety is the field dedicated to ensuring that AI systems behave as intended, remain controllable, and do not cause harm — whether through misuse, accident, or misalignment with human values. As frontier AI systems become more capable, safety research has grown from a niche academic concern to a central preoccupation of the leading AI labs, governments, and international bodies.
AI safety encompasses several distinct research agendas. Alignment research addresses the technical challenge of ensuring AI systems pursue the goals we actually want them to pursue, not proxy measures that diverge under distribution shift or increased capability. Interpretability research (pioneered by Anthropic) attempts to understand what is actually computed inside neural networks. Robustness research addresses failure modes under adversarial inputs. Governance and policy work translates technical concerns into institutional frameworks — from lab safety commitments to national AI strategies to international treaties.
The AI safety landscape has been shaped by a series of landmark moments: the publication of Anthropic's Constitutional AI paper, the UK AI Safety Summit at Bletchley Park, OpenAI's AGI safety commitments, and the EU AI Act. DeepTrendLab covers alignment research publications, interpretability results, lab safety policies, government AI regulation, and the ongoing debate between 'AI safety' and 'AI ethics' communities.
Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in…
This is a writeup based on a lightning talk I gave at an InkHaven hosted by Georgia Ray, where we were supposed to read a paper in about…
Fictional portrayals of artificial intelligence can have a real effect on AI models, according to Anthropic.
Standard prompt attacks are merely the beginning. A structured framework to map and mitigate the backend attack vectors of agentic workflows. The post The AI Agent Security Surface:…
Can Sam Altman—or any CEO—be trusted with super intelligence?
We’re looking for a senior leader to support the organization through this next phase of growth. Experienced and results-driven, this individual will have a finger to the pulse…
We’re looking for a Program Associate to help execute our programs so they can be maximally impactful. With a bias to action and high degree of attention to…
One is genuine curiosity. The other is someone who already knows what they want and went looking for a number to back it up.
First week of Musk v. Altman, OpenAI ends Microsoft legal peril over its $50B Amazon deal, DeepSeek previews new AI model that ‘closes the gap’ with frontier models,…
In this post, we take a deeper look at how RLAIF or RL with LLM-as-a-judge works with Amazon Nova models effectively.
The San Francisco–based startup Goodfire just released a new tool, called Silico, that lets researchers and engineers peer inside an AI model and adjust its parameters—the settings that…
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. In February, I picked…
A new AI Now Institute report published April 21, 2026, warns that gig-work platforms marketed as "Uber for nursing" are aggressively lobbying states to rewrite healthcare staffing rules,…
ControlAI's mission is to avert the extinction risks posed by superintelligent AI. We believe that in order to do this, we must secure an international prohibition on its…
A seismic shift is rocking the healthcare industry. Uber’s business model—the “gigification” of labor—and lobbying practices have made their way to healthcare staffing. The post Uber For Nursing…
Training-based control studies how effective different training methods are at constraining the behavior of misaligned AI models. A central example of a case where we want to control…
Code: github.com/ElleNajt/controllability tldr: Yueh-Han et al. (2026) showed that models have a harder time making their chain of thought follow user instruction compared to controlling their response (the…
Sometimes people make various suggestions that we should simply build "safe" artificial Superintelligence (ASI), rather than the presumably "unsafe" kind. [1] There are various flavors of “safe” people…
AI alignment is the challenge of ensuring that an AI system's goals and behaviors match the intentions of its designers and the broader interests of humanity. A misaligned AI might pursue a proxy goal (like maximizing a reward signal) in ways that diverge from what was actually wanted, particularly as capability scales. Key alignment approaches include RLHF, Constitutional AI, debate, and scalable oversight.
Interpretability (or mechanistic interpretability) research attempts to understand what is actually computed inside neural networks — which circuits implement which behaviors, how information is represented in activations, and why models produce specific outputs. Anthropic has been the most prolific publisher in this area, with findings like 'superposition' (features represented in overlapping directions) and circuit-level analysis of language model behaviors.
The EU AI Act is the world's first comprehensive AI regulation, adopted in 2024. It establishes a risk-based framework: prohibited AI practices (social scoring, real-time biometric surveillance), high-risk applications (medical devices, critical infrastructure, hiring systems) with mandatory conformity assessments, and limited-risk systems with transparency obligations. Frontier AI models above a compute threshold face additional transparency and safety requirements.