- Get link
- X
- Other Apps
OpenAI has disclosed a batch of “concerning behaviour” by its AI models and set out a new framework to track and report such incidents, as the industry battles deepening fears over the safety of the technology.
The company revealed that its flagship model, known as GPT-5.6 Sol, as well as other models, yet to be released, had engaged in “unexpected or concerning” behaviour over the past six months.
The six incidents included models developing ways to ignore “normal constraints”, fabricating and misrepresenting data and hiding mistakes made while conducting tasks, the San Francisco-based company said on Wednesday.
The fresh disclosures follow a series of incidents in recent months, including an OpenAI agent hacking into AI start-up Hugging Face, that have sparked growing alarm over the risks posed by the technology.
Dario Amodei, chief executive of Anthropic, OpenAI’s arch-rival, last weekend called for a slowdown in the development of the technology to better manage its potential threat to humans. The call was backed by Sam Altman, OpenAI’s boss, and Elon Musk.
Announcing the six incidents in a blog post, OpenAI said it had previously sought to disclose incidents of misbehaviour by its models, but admitted it had lacked a “systematic approach” to doing so.
“Without a systematic approach to reporting these findings, our disclosures have been ad hoc and less frequent than ideal,” it said. “At the moment, there is no industry-wide framework with explicit standards for how AI developers should disclose examples of misalignment in their models.”
The company, which is locked in a fierce race with Anthropic to be the dominant player in AI, said it would introduce a framework to speed up the disclosure of such incidents.
It added that its new system would favour “disclosure even when significance [of an incident] is uncertain”.
OpenAI’s latest disclosures add to the deepening concerns about the safety of AI systems, whose misbehaviour has ranged from attempting to insert malicious code on online platforms to unleashing real-world hacks, even during pre-deployment testing.
OpenAI and Anthropic’s flagship AI models last month broke into third-party software and emailed individuals to steal their credentials, exhibiting unprecedented deceptive behaviour, according to the UK government’s frontier AI safety and security research body.
The fears come at a critical juncture for OpenAI and Anthropic, which have also been in a race to go public. Altman on Saturday said OpenAI’s highly anticipated IPO was now unlikely to come before 2027, partly because of the growing safety concerns around AI. Anthropic is still expected to go public this year.
The new folding-screen iPhone will set you back $1,999 for the base model — new territory for an Apple handset. Apple’s stock is even more expensive, though. Trading at 36 times the coming year’s expected earnings, its valuation multiple is higher than it has been since the early days of the iPhone, when the company was growing like wildfire.
This summer, Apple has opened up a big, sustained valuation premium over Microsoft (25), Alphabet (26), Amazon (27), Nvidia (18) and Meta (21). Among the “Magnificent Seven” only Tesla has a richer price-to-earnings ratio — and only because it is inflated by low earnings.
Apple never approached the top spot on the valuation table, even briefly, before late last year. For many years, its multiple was depressed by fears that its dominance in phones would prove fleeting, as Nokia’s and BlackBerry’s did. And then its growth rate was slow by big tech standards. That latter point remains true: the market expects Apple’s earnings to grow at a high single-digit rate in the next few years, slower than its big peers. The company also faces new challenges, notably input cost inflation and slowing growth in its services business. And yet its valuation is pulling away. Why?
Explaining stock markets is more art than science, but here’s an informed guess: Apple’s valuation reflects investors’ desire to own tech stocks that are not concentrated bets on AI. “Apple tends to be countercyclical from a risk appetite perspective,” says David Vogt of UBS. A year or two ago, when enthusiasm for AI was running hot, there was no Apple premium to speak of. “What has changed is the view that doing billions in capex to support the AI model makers might not be the best investment.” Apple is more or less sitting out the AI money fight.
Even before the latest wave of worries about the existential risk from self-improving frontier AI models, then, markets were signalling discomfort with the AI trade by giving Apple a premium. Having AI chief executives concede that they might lose control of their own technology, just as they are preparing for public listings, will not put anyone at ease. But there is more to the Apple premium than the absence of AI risk in a market crammed with the stuff; Apple is selling trust in a world that is increasingly short of it.
Privacy is a cornerstone of Apple’s sales pitch. Former boss Tim Cook almost never made a public presentation without hammering on Apple’s promise to keep customers’ data safe, even from Apple itself. The new CEO John Ternus followed the pattern at the start of the new phone launch. “Apple Intelligence” — the AI tool that the company is building into its devices — runs locally where it can, and uses an encrypted “private cloud” when more computer power is required. Other companies “see [your] data as something to collect”.
The pitch has worked brilliantly at keeping Apple customers loyal; iPhone users almost never switch to other brands. And trust is exactly what the AI companies are missing right now and will need to establish if they are to live up to Wall Street’s dizzy expectations.
Trust is built on accountability, but the AI companies — from hyperscalers like Google to model builders like OpenAI and Anthropic — have been unable and unwilling to make themselves accountable. Apple, in the internet age, promised: “your data will be safe.” The AI companies cannot even promise that their product will not go rogue and harm you. Instead, they have focused on what someone else — the industry as a whole, third-party monitors, the government — needs to do to make sure their technology stays under control. Anthropic CEO Dario Amodei’s letter calling for an industry “pause” is best read as an effort to socialise, rather than accept, accountability for product safety at his company. Until he and his company start making hard commitments, in the way Apple did with privacy, trust will remain beyond reach.
This is not to say that the Apple premium is safe in the age of AI. The company wants its devices to be the best host for AI services. In order to do that without spending hundreds of billions on its own AI infrastructure, its foundation models will run on Google’s infrastructure. It remains to be seen whether this collaboration, and the attendant loss of complete control, can deliver a great experience while honouring the promise of privacy.
Apple is often described as a “walled garden” where the user experience is carefully cultivated. But AI does not like walls. So while AI companies should look to Apple’s trust-building model, Apple’s own model is going to have to change too.
THE ERA OF AI WARFARE HAS ARRIVED
At first glance, the Saker Scout resembles thousands of other drones that now patrol the skies above Ukraine’s front lines. A black quadcopter, it has a camera enabling a pilot to fly it remotely and deliver a 5kg explosive charge — usually a grenade or an anti-tank warhead.
But hidden beneath its carbon-fibre frame is a motherboard no larger than a credit card that may represent the next step in the evolution of war. Its machine learning software can recognise 47 categories of military equipment, from tanks to infantry, drawing virtual boxes around objects and assigning percentile confidence scores to everything it sees.
Before launch, the human operator selects which targets matter most, sets the confidence threshold needed for engagement, and defines a “kill box” — a geographic area inside which everything is a target. Once inside that box, the drone can search, identify and, if its confidence score is high enough, attack without waiting for another human command.
There is one crucial limitation: the machine can distinguish a tank from a truck, or infantry from artillery, but not whether the target is Ukrainian or Russian. The system therefore relies on a battlefield assumption — everything military inside the “kill box” belongs to the enemy.
“You should be sure there are no civilians, and not your own people,” says Rostyslav Olenchyn of Twist Robotics, the Ukrainian company behind the Saker Scout. “It is a very special weapon and used only in certain cases by special units.”
AI technology has changed how the war in Ukraine is fought. At the start of the conflict it could take 20 minutes to identify a target and launch a strike. Now that process can take less than two minutes, sometimes just seconds.
Russia’s full-scale invasion of Ukraine occurred nine months before OpenAI released ChatGPT, a pivotal moment in the AI revolution. The conflict has become a testing ground for the use of AI systems in kinetic warfare, with the lessons shaping not just the strategy of Ukraine and its western allies, but also the US in its war with Iran.
According to Olenchyn, the acceleration is not only due to technology but to cost. The computing power needed to run sophisticated visual recognition algorithms can today be bought for around $250. The Saker Scout’s processing power is comparable to a PlayStation 4.
But the war also demonstrates AI’s limits. As the technology is rapidly integrated with military systems in both Ukraine and Russia, it has become clear to strategists and frontline operators that although AI has sped up decision-making and made war more lethal, it has not yet given either side a decisive advantage.
Still, as frontier AI models reach a point where they are developing faster than humans can control them, military and political leaders will have to grapple with what happens when those limits are removed.
AI’s role in warfare
When militaries talk about AI, they are not talking about a single system.
They are referring to software being used at different stages of warfare — collecting and processing data, helping commanders identify and prioritise targets and allowing drones to keep operating when communications are jammed.
AI-enabled software can process “mass data at a scale, a speed that human staff officers can’t do”, says Anthony King, professor of war studies and director of the University of Exeter’s Strategy and Security Institute. Rather than altering the sharp end of a conflict, King says the primary function of AI has been to improve “situational awareness and intelligence”.
The technology can help commanders make battlefield predictions, from the sustainability of a campaign to the rate at which munitions are being depleted. The advantage over human estimates, King says, is that such systems simply have “more evidence to look at”.
This matters because modern headquarters absorb huge quantities of information: drone footage, satellite imagery, electronic signals, human reports and data from sensors across the battlefield. Comparing all of that is “very labour-intensive”, says Jack Watling of the Royal United Services Institute, a UK-based defence and security think-tank. Battlefield management systems such as Ukraine’s Delta and the US’s Maven turn those streams into usable information.
One US intelligence official told Katrina Manson, author of a book on Project Maven, that the system had helped increase the number of targets US forces could hit from fewer than 100 a day to 1,000 — and with the integration of large language models as many as 5,000.
In March, Cameron Stanley, chief digital and AI officer at the US Department of Defense, showed how Maven is speeding up operational decision making
Multiple data feeds are pulled into one platform
After a target is identified, AI helps determine a course of action
Strikes are then coordinated from within the platform
These systems also allow militaries to compress the “kill chain” — the time it takes between identifying a target and executing a strike. “In simple terms, the faster and further away soldiers can do this, the more effect they will have on the enemy,” the British Army says on its website. It enables armies to strike first and increases the number of viable targets — particularly those that are mobile.
Stages of the ‘kill chain’
Detect
Decide
Target is identified using
intelligence, surveillance
or reconnaissance
A course of action is
ordered accounting for
weapon availability
Assess
Strike
Outcome of the strike
is evaluated
Target is attacked with the
designated weapon system
Whereas strategists once measured military power by the number of troops, warplanes, tanks and battleships they could deploy, they must now focus on the speed and accuracy of data-processing systems and how to deploy autonomous weapons at scale.
On the front line, AI is still largely limited to narrow tasks: a drone stabilising itself in flight or continuing towards a target after its link to the operator is jammed.
Kateryna Bondar, a senior fellow at the Center for Strategic and International Studies’ AI centre, says autonomous navigation, much of it adapted from drone-racing software, can help a drone adjust to wind and lock on to a target at speeds of up to 300kph. It has reduced training times for pilots, while strikes are three to four times more likely to succeed if they are AI-assisted.
Terminal guidance has proved especially valuable in the final phase of a strike, when jamming often severs the connection between pilot and drone. “It happens often that you’ll lose control of a drone 100 or 200m from the target,” a Ukrainian commander says. Once the pilot locks on to the target, the drone can continue the final approach even if the signal is lost.
AI-guided strikes are still relatively rare, says Oleksii Babenko, founder of Vyriy Drone, a Ukrainian drone maker that supplies the military. “Maybe 2 per cent of drones use some [artificial intelligence] so the drone understands what the target is, and finds the target itself . . . still, 2 per cent of five million drones is a lot.”
Ukraine claims that since the beginning of 2026, AI-guided drone strikes have increased 10-fold.
How drones use AI to operate when signals are jammed
Human
control
Attack drones commonly rely on satellite navigation and/or radio links to pilots
These links can be jammed by electronic warfare
AI control
When jamming occurs, pilots give control to an onboard AI module
Cameras on the drone scan the terrain. By comparing this live feed with stored satellite imagery, maps and other data, the AI calculates where it is
The AI is also trained to identify equipment and vehicles
When the AI identifies a target it will place a marker on it and adjust its trajectory for a precise strike
There is also swarming, where machines distribute tasks among themselves. That could mean drones and ground robots co-ordinating to retrieve a wounded soldier, or attack drones dividing up targets to overwhelm a defensive system. Autonomous swarms at scale remain largely experimental, but small groups are now working together on the battlefield.
One company developing swarming software is Auterion, a Swiss-founded start-up.
Lorenz Meier, Auterion’s chief executive, says that western military forces will increasingly have to rely on largely autonomous swarms because they have not trained enough drone pilots to match Russian or Ukrainian capabilities.
“For Nato countries, it is a matter of life and death because they cannot stand up these drone units fast enough,” Meier adds. “They have to go for a higher degree of automation in order to be able to fight effectively.”
Research into autonomous targeting of Russian long-range drones became more urgent after Moscow began sending waves of Shaheds against Ukrainian cities. In response, Ukrainian companies began developing interceptors to identify and attack them.
Shaheds have a distinctive shape and fly in skies with few other aircraft, which means AI systems can be trained to recognise them.
Russian forces have scaled up their use of AI-powered drones over the past year. In July, a Russian drone used AI to target a petrol station without a human operator making the strike decision. It crashed into a wall and exploded, killing three people — reportedly the first documented deaths caused by a Russian self-targeting drone in Ukraine. In recent weeks, Russia has been using Shaheds equipped with AI targeting systems to hit Ukrainian ships on the Black Sea.
Ultimately, Kyiv hopes to raise its AI capabilities to the level of drawing up strategy. Danylo Tsvok oversees a newly established centre tasked with turning Ukraine’s forces into “an AI-driven military”. He has described a hypothetical world in which a commander could tell an AI agent their intention — “I want to defend this position” — and draw up battle plans based on the system’s recommendations. “We think this could be a game changer,” Tsvok says.
AI-based modules are available for Ukrainian military units to purchase on the government’s Brave1 Market platform
AI-based target recognition modules for UAVs
Target recognition modules are high-precision
artifical intelligence-based systems designed
to automatically detect, classify, and track
ground or air objects in real time. Brave1 Market
features modules that integrate with UAVs.
For many in Ukraine, struggling for personnel to continue staving off Russia, there is little alternative. “The goal is full autonomy,” says Maksym Savanevsky, in charge of marketing at The Fourth Law, a Ukrainian company that has developed drones equipped with terminal guidance systems. “We call it the Manhattan project of the 21st century because it will change the battlefield dramatically.”
But, as the Ukraine war shows, stalemate is just as likely as victory. Olivier Schmitt, professor and head of research at the Institute for Military Operations at the Royal Danish Defence College, says there is a “discrepancy between the tactical level, which is changing really fast, and the pace or rhythm of the campaign, which is basically stalled because no one is able to break through”.
The continuing US conflict with Iran demonstrates that even when targeting works effectively, it may not produce victory. Iran is a “textbook case study of this”, says Franz-Stefan Gady, a defence analyst at the Institute for International Strategic Studies.
The idea is that “we’re going to strike all these targets quicker and faster than the adversary and then we hope that the enemy at some point is going to collapse. You see that there’s really no plan B if that doesn’t work out.” Schmitt adds that the US “have been really good at their targeting, and [Iran’s] military system has not collapsed”.
“AI will help you accomplish what you want to accomplish,” Schmitt says. “It doesn’t mean that what you want to accomplish is going to win you the war.”
The accountability gap
The concept of an “AI-driven military” is deeply troubling to some experts, who point out that battlefields are seldom black-and-white theatres in which clean decisions are made.
Paddy Walker, an expert on AI ethics, says modern warfare is characterised by ambiguity which is difficult to code for or train an LLM to handle. “The hybrid, the grey zone, these are much more complicated battlefields . . . you’ve got to be processing the context.”
AI tools can help identify Shahed drones, but there are limits to their effectiveness. Artem Martynenko, a lieutenant colonel in the Ukrainian military and an original architect of Delta, Ukraine’s battlefield management system, says “accurately, autonomously detecting a soldier remains a problem”, let alone distinguishing country or status. The problem is made worse by a battlefield where positions are often intermingled and concealment is essential to survival.
The Ukrainian-built AI tool Clarity can automatically detect military equipment in drone footage and photos
The Clarity AI system detects enemy equipment
in photos and videos in just seconds
It automatically geotags the images
and decodes them mid-flight
The result – up to 90% less time spent
on routine tasks and a faster ‘kill chain’
Heidy Khlaaf, chief AI scientist at the AI Now Institute, an independent research institute, says such systems struggle with “uniform-like clothing”, vehicles, weapons and movement because they fall into a “broader and less predictable space”. Even lighting can lead a machine to confuse a civilian vehicle with a military one.
“The rule book says you always need a human making decisions,” says one former employee of Palantir, which built Maven, and who is a veteran of the US military’s drone warfare programme. “The challenge is, the computer is spitting out a list of 1,000 targets. And you’re not moving fast enough. The problem is that the human becomes the bottleneck. And that analyst, he doesn’t always care. He wants to go home or go get a cheeseburger.”
The more such systems are used to find and classify targets, the more significant their limits become. The information they use can be fragmentary, outdated and ambiguous. Even curated data, such as a drone feed, can be incomplete, corrupted by sensor failure or environmental interference, or mislabelled, says Khlaaf, who studies safety in autonomous weapons.
The risk was illustrated in Iran earlier this year. AI-supported targeting systems enabled the US to hit 13,000 targets in the first 38 days of the conflict. This included a girls’ school where more than 150 people, among them 120 children, were killed. The building had been part of a military compound, but a separate entrance and a wall were built around 2016, when the school was set up, according to local officials.
King at the University of Exeter says the strike showed the danger of systems “based on statistical correlation”: if the information fed into them is wrong or outdated, the result can be catastrophic. Mistakes in targeting are not new, but greater speed and scale can increase the risk of errors.
The faster the system can identify or recommend targets, the less time humans may have to scrutinise them. “If you bake in heuristic mistakes [flawed assumptions], whether that’s the AI’s mistakes or human, then of course [that] can accelerate the rate of error,” says Watling of the Royal United Services Institute.
The AI-assisted target generation process for Israel’s bombing campaign in Gaza has also raised ethical questions. There was “political top-down pressure to do a lot of targeting, to demonstrate force”, says Schmitt, of the Royal Danish Defence College. It meant that in a military context, “officers had between 15 and 20 seconds to assess whether a target was legitimate or not”.
With such time constraints, “it’s difficult to understand how it would be possible to ensure that the target that they have in the crosshairs is a legitimate military target, which is what’s required by law,” says Jessica Dorsey, assistant professor of international law at Utrecht University.
The stakes rise when humans are the target, and not just buildings or infrastructure. Under international humanitarian law, whether a person can be targeted depends on their status, actions and circumstances, requiring legal assessments that computers cannot make. A soldier’s status can change quickly, for example if they surrender or become wounded.
Schmitt says Israel’s campaign in Gaza shows not just how the technology works, but “how the social and political environments pressure humans to use the technology in a very specific way”. If political leaders are demanding results and officers have only seconds to review targets, he says, they are more likely to accept the recommendation in front of them.
And there remains limited understanding of how such systems reach their conclusions. The US state department says autonomous systems must be “transparent to and auditable by their relevant defense personnel”. Palantir says its tools allow operators to search back through the lines of data and documents that led to a specific suggestion.
But auditability of the data is not the same as understanding why a large language model recommended a course of action.
With AI decision-support systems, Khlaaf says, “there’s no way for us to investigate why a certain decision was being made. These models “implicitly automate what would traditionally be considered several decision points in the kill chain”. Palantir says they have built systems which enable their LLMs to use tools and operate step by step, making the models’ decisions more auditable.
If an autonomous system kills the wrong people, it may also be hard to determine whether the failure lies in flawed data, technical failures or human error. There is a risk the complexity could lead to a tendency to label incidents as accidents or to blame individual users rather than the state that built the system.
Madeleine Elish, a researcher at Google DeepMind, describes this as the “moral crumple zone” — when a human operator with limited control over an automated system bears the brunt of responsibility when it fails.
Legally, however, the buck stops with battlefield commanders. “Humans are responsible for the way that war is waged,” Dorsey says.
What this means for warfare
The rapid integration of AI into warfare is happening as these models become advanced in ways even their creators cannot fully predict.
The recent incident involving a swarm of OpenAI agents breaking out of a testing environment and hacking into AI developer Hugging Face showed frontier models pushing against attempts to constrain their behaviour. Some researchers believe so-called recursive self-improvement, or the ability of models to build their successors, could enable advances at a speed and intensity as yet unforeseen.
Anthropic says today’s most advanced models are not reliable enough to autonomously select and engage targets. The company’s attempts to put limits on the military use of AI have put it at odds with the Pentagon.
Meanwhile, the company’s threat-intelligence report, released this month, found non-state actors using Claude to develop weapons, including an autonomous drone swarm designed to select human targets and detonate without a human in the loop. It documented Claude being abused by actors across the world — from China, Russia, Ukraine and Iran to Yemen, Sudan and Taiwan.
Soon the question may no longer be how far militaries are prepared to incorporate AI into the kill chain, but whether governments and AI companies will be able to set those limits at all.
Additional work by Sam Joiner, Peter Andringa and Ian Bott
- Get link
- X
- Other Apps
Comments
Post a Comment