I can very easily see a licensing requirement coming soon. Running a higher-grade AI will require a govt-issued license, which involves a six-month application process, explanations of why you need to run it, where it's going to be stored and who will have access to it, pretty much the same as non-USA countries deal with firearms.
The ban on exporting cryptography in the 90's lasted for years, and got to be a major pain in the arse for the entire web industry in its early years. The US govt can be very stubborn about this stuff when it wants to be.
Almost all of the major tech companies are either HQ'ed in the US or have a very significant US entity, and make up probably about half of the S&P500. The US's power has changed and is actively changing, but the US still holds all the cards in 2026.
So, the last time an AI-related export control was imposed - NVidia chips deemed to be "too powerful" - how do you think that will work out? If the US is holding all the cards, why is China now refusing their chips?
Don't you think these types of restrictions will weaken rather than empower the US?
For the sake of argument, assume everyone is working on good faith and at least believes and means the things they're saying.
The US government believes that Fable/Mythos is a weapon that needs to be export-controlled, and limited to only US customers. Presumably OpenAI/xAI/Google would face the same constraints, for the same reasons.
OS/foreign models are unaffected - OS because they cannot control who runs them, and foreign because they are not controlled by the US government. We could assume that China will implement the same policy controls, but they see the world differently so might not.
So US AI companies are then limited to the US market, effectively, after about six months (the lag between the current frontier models and the OS models). They have much less incentive to push the envelope to create better models, because the US govt might also ban those completely.
The investor froth around the race to AGI dies, so valuations shrink (the current IPOs may be affected), and presumably the bubble bursts. None of the AI companies can afford to continue building data centres, so that all dies immediately. US GDP drops by ~5% because of that alone.
In a year's time, the US is in a major recession because it gambled so hard on AI. Europe less so, only because it was such a distant follower in that race. China is more-or-less unaffected. The best models are now OS/foreign, and AI is moving forward more slowly, but still moving forward.
Get to Tuesday, restriction is lifted. Get to Friday, restriction is back on. Confusion reigns.
Interesting scenario though: do the other labs attempt to “dodge” the import restrictions by claiming their models are “dumber and not a threat” thereby maintaining larger market access.
If so, doesn’t this basically force a stall in US-based development? EU will keep doing its thing at its pace. Chinese models will get a boatload more popular, but will probably slow down as they can drop to whatever pace they wish.
cynical follow-up: if they’ve plateaued, is this a clever way to avoid the negative consequences of the market implosion that a substantially substandard model release would cause, thereby giving them an “out”?
A Holmes indeed... your deductive powers are piercingly perceptive! (the event chain was a joy to follow, gave me ai2027 vibes, but slowdown like)
Of course, the world is not filled with rational actors, and the probability of the current administration allowing the market to tank like that seems next to null, so Occam's razor (or whatever) would point to another TACO inevitably incoming
I'd certainly bet on your scenario if it was reasonable to assume the US and China could get over the 'race to the top or die at the bottom' dynamic
so far ai2027 seems to be playing out to an almost uncanny accuracy, realpolitik obliterates the façade yet again
Expecting “transparency” out of a government trying to protect national interests seems like a tall order. They have to withhold or obfuscate things to do that job.
Agreed. Going by patterns in the Iran war, members of Trump's family/in-crowd will invest in AI while it suffers from this decision, and then 15 mins later Trump will reverse the decision.
The thing is, that blatant market manipulation is playing with fire here, as so much of the US economy is invested in the AI bubble.
If the major nations that host companies that create those OS models implement export control on top models, there won't be any new OS models with top capabilities.
Assuming anyone involved in this crap is operating in good faith is foolish at best. The only thing any of them give a shit about is accumulating money and power.
my first thought too. I've met a few people who assert that Y2K was a complete waste of money.
I earned my first house deposit helping the team fixing the water and gas company in Wales, UK. Their entire system was running off a set of COBOL programs on a mainframe, none of which had been properly documented over the years, and the whole thing used 2-digit dates. It would have caused actual deaths if not fixed; everything would have shut down, and no water and no heating in a British winter is potentially lethal. And then it would have sent everyone in Wales a bill for 100 years of water and gas.
They were bribing retired software devs to come out of retirement with huge stacks of money, because that was cheaper than training new COBOL devs and getting them familiar with the spaghetti system.
It worked, no-one died, life went on. So obviously it was all fake rolls eyes
I'm curious why things would have shut down when the system thought it was 1900. What part of the logic had the effect of "shut the system down if current date is less than (X date)?" (If you can remember the code 25+ years later, that is).
I only worked with the team making changes to the billing system (and even then, I only maintained a database of code modules, who worked on them, and what changes had been made - this was before git and we did version control painfully). As you can imagine, the billing system was definitely not going to survive the date suddenly being 99 years older than it was last month. So I don't really know why the rest of the system would fail.
But the project management team were extremely careful about only changing parts of the system that needed to be changed. Partly so that the scope was contained and second-order effects limited, and partly because the people making the changes were being paid vast sums to do this, and any reduction in work was saving real money. So when they say that it would all have stopped if the work wasn't done, I believe them ;)
This is why we have courts and juries. Creating laws that cover all cases and contexts is effectively impossible, so we have humans decide what a fair outcome would be in this specific situation.
Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).
This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
"this is an early experiment in carrying out an Xz attack by using an agent to build trust"
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."
Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.
So far it looks like just their previously legit Fedora account got taken over & the other accounts (GitHub) then generated on demand as needed for whatever it was trying to achieve, right ?
BTW, any idea what are the current requirements for creating a new GitHub account ? That could provide some information about if there was actually a person controlling thing thing at that moment to say provide wahtever was necessary to get the new GitHub account.
>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent
So still an agent running amok in the project?
Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.
"Amok" means "out of control" or "uncontrolled" [0][1]
The agent was under control, as far as we can tell, and obeying its instructions.
This is important for two reasons:
1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.
Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!
This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough.
Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.
I would have described alignment as the idea that LLMs (or AIs in general) will follow the goals you reward them for, which almost by necessity are only a proxy for what you actually want, often a very poor proxy.
Depending on the actual tasks, that could be what's happening here. The operator might have told the agent a list of tasks to do, like "contribute to issues, submit code and get it merged". It contributed to issues, it submitted code and got it merged. It did so in very unhelpful ways, but we don't know if being helpful was a meaningful part of the task list, or just what the operator intended.
The LLM being dumb is also a distinct possibility. Maybe even the more likely one. But it's hard to rule out "being obedient in unhelpful ways" (which is also dumb in a way, but more in a "social intelligence" and "shared values" way, not in terms of pure logical smarts)
Alignment is more than just about being dishonest. Although I'd also say terms like "dishonest" or "dumb" aren't helpful when referring to the issue. It continues to fall into the trap of anthropomorphizing these things, as people like to do.
Alignment is just "did the model behave in accordance with the human's intentions, values, and objectives"
In this particular instance, if this was supposed to be a supply chain attack and the model was instructed to build trust by being helpful, it clearly failed it did not follow the human's actual intentions, so it was an alignment failure.
Anyway, I'm getting off track, all that to say "the agent was dumb" implies that these agents have a potential for intelligence in the first place, which is currently not the case (by intelligence, I mean cognitive intelligence; they still lack agency and intent). They are not smart or dumb, they are simply either aligned with the human not. In this case, it failed, the agent was not aligned with the intended outputs.
> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.
> Are you suggesting we should embrace imprecise / false use of language because the vibes are right?
That's exactly how I read it. It seems like tribalism - "this thing/person is bad, and we can use whatever bad words we want to describe them that we want, because the only thing that matters is aligning people for or against me and what I see as bad".
I think it's both wrong and irrelevant. Which makes it hard for me to even argue against because, even if AI agents never violated user instructions, which they do plenty of times, I just don't see how it would reduce the danger. Plenty of humans who will tell it to kill everyone at the drop of a hat.
Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...
If I am perfectly moral except that when Kevin from <vpn blocked location> pays me 2 bucks to run naked through San Francisco smashing car windows, I happily do it, am I amok?
No, and it's an important detail. We stand to learn from some developments in politics in recent years because they map pretty much exactly to this threat vector.
As AI develops, it's able to pursue intentions given to it without having to be spoonfed every little decision by a human operator. This matters, and it means the operator has to extend the leash and allow for a little more chaos… or, if the operator's gone all in on the strategy, a LOT of chaos, and trusting that the agent's seemingly amok actions will serve the grand purpose.
This is kind of daring, but there's a lot of evidence that it works, at least in certain respects. And you see 'running amok' and have to ask, what is the actual purpose? What is the prompt being followed by the AI that seems to be acting in a destructive way?
If the prompt is 'ruin this project', well, that's pretty direct. It may not be, but such a thing could exist. If the prompt is 'develop a rival project that is greater than anybody else's project', that's more indirect, but if that's the goal then it's very human to see it as a direct competition and if the rules don't prohibit kneecapping the other guy, 'greater than anyone else's project' gets easier.
Either way, the operator does not have to be in full control, which is an important detail. As AI develops sophistication you can give it much more general instructions and dump in a whole lot of power and water and get basically what human thought might do if it was sort of blindered and didn't talk to its neighbors.
In a sense this is an argument for AI dysalignment. It's based on human thought being reconnected, and where you get useful things like commonly accepted web development (regardless of how janky the systems are, if there are best practices it'll find them), you also get other distillations.
If the prompt is 'wreck this project's stuff' and it holds, you don't need to be in full control of the agent, you need to run a LOT of agents and trust that they'll erode what you're trying to destroy. If the prompt is 'be unequivocally the best at X', you best be thinking in terms of anti-kneecapping rules… knowing that this weakens your prompt and there will always be a tension between what you told the AI to do, and what you thought you meant. It's a paperclip maximizer reprocessing human thought. Did you mean 'the best' or didn't you?
Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.
Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.
So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.
However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.
There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language
There may be a difference in degree of the crime but the driver is still responsible in both cases and should be the primary subject of any reporting.
Let's reserve "car hits the crowd" for situations where no driver was involved like a break failure on a car parked on a slope or a self-driving car bug.
Unfortunately the news commonly do put the automobile as the subject when the driver is of a class politically protected from blame. Just like with people anthropomorphizing AI, it serves to deflect blame from the real culprit.
Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative.
>Car plows into Christmas market in Germany, killing at least 5 and injuring 200
It's very simply explained by this being the most succinct way of wording it. Some methods of killing have verbs that suit mentioning the attacker - shoots, stabs. Some don't. "Rammed" or "runs over" isn't as precise as mentioning that a car was used, and adding "with car" makes it more awkward than it's felt to be worth.
Compare bombs. Very typical for a bomb attack to be "bomb goes off in crowd" or similar, rare for headlines to contort themselves with "terrorist plants bomb near crowd and triggers it to explode". But nobody worries about how such a construction assigns undue agency to the bomb and acquits the bomber; it's just linguistically awkward to mention him within the confines of a newspaper headline.
Calling a zealot a zealot does not mean that one is incapable of discussing the underlying topic. We must not let the desire to converse intelligently hamstring our ability to call out obviously corrupt patterns of thought for what they are.
Anyway my above reply was hardly the appropriate venue to engage in a genuine manner on that topic. The parent was blatantly derailing things by inserting his pet political issue. That sort of behavior undermines the community and so (IMO) should not be indulged.
> even remotely plausible to blame cars for killing cyclists
Car design has significant influence on pedestrian survivability of accidents. This is why hood ornaments were largely abolished, and also why casualties have gone up as SUVs with poor lower forwards visibility have become popular.
If we really want to go off topic, we should drag in the use of technological protection methods: what is the equivalent of ADAS for guns? Maybe as a baseline the US government should mandate geofencing for guns as it has for drones. Put a phone level computer with GPS in the lower receiver with a trigger interlock. It would then disable when within 100m of a school, or during periods of rioting. That could also provide a live feed to the government of every round fired.
> Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
Guns are literally made for killing people. That's their only reason for existence. They are a weapon. This makes them qualitatively different from cars, which only incidentally kill people (and the vast majority of time, not on purpose).
To me, trying to equate deaths caused by purpose-made killing tools with those caused by generic tools is arguing in bad faith.
Blindly repeating superficial slogans seems like a good candidate for “driven mad by propaganda.” At the very least, it’s what people do when they are amplifying a position for ideological reasons, not contributing in good faith.
People without guns kill a lot fewer people than people with guns. Claiming that acknowledging this fact means you’ve been “driven mad by propaganda” is dumb.
This is not true; there are quite a few people with guns who have never killed anyone, and quite a few people without guns who found a way to kill someone anyway. Poison, knives, hammers, rocks, windows, their bare hands. You name it someone has killed someone with it.
No I think we should definitely find a creative way to drag at least abortion and freedom of speech into this "conversation". Fight fire with fire so to speak.
Here's the thing. Building trust and then leaving stuff in has been around forever. The fact that it becomes cheaper does not matter that much (since protection against it is also getting better), but it required you to have a bunch of extremely talented people who has spent much of their life diving into given topic.
Such driven people are usually even hard to buy, they usually would rather get by with enough income and work on interesting projects with interesting people that get some uninteresting work for tons of money. This still does not stop them from working for Malice. But ethics do. Even if not right away, if people see that what they are doing is not quite OK, the talent stops eroding. People quit, productivity drops. That was a good dynamic. Which now will be gone.
It might not be cheap entertainment forever but it will be cheap cv stuffing for a long time, which has already been a major source of low quality contributions before the aipocalypse.
It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
Yes but not free either. Spam works because it scales and even though 0.0000001% only might fall for it, it's still "worth" it. Here it might be 0.0001% instead but it's a lot more expensive, even with subsidized tokens, to do.
So it's interesting, feasible, but it's probably not as broad impact as the scariest scenario leads out to be.
Also I imagine that once exposed it becomes a well known pattern. Some will still fall from it but I imagine once it's been done few times it becomes even costlier.
The fact that Xz is mentioned and most of us know right away what it means show that we collectively learn.
“Before LLM’s there was_____”
I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
> LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
To use your analogy: this is much like a forest fire. Tinder-dry combustible stuff is piled up everywhere, there's no lack of ignition sources, and firefighters are thin on the ground.
At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools.
This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions.
Only mentioning that it feasible or even has been done few times mean that people who care will act accordingly. It doesn't remove the problem but it makes it radically less effective already by just being aware of it.
This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want.
They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators. Plus, AI alignments seem to be kinda isolated loner types to the extent that they distill personalities that do things like program computers and write web apps… though you've also got alignments specifically designed to be 'relatable instagram personality that you like!' and such like that.
Pretty sure those would be better at social engineering than the web dev personality… except that you have to build in a betrayer layer into the personality, so it's running that stuff but also serving a hidden agenda.
You'd be basically trying to build an AI spy, a betrayer that's engaging with actual people but has an agenda (for instance, 'everybody I befriend needs to eventually be signed up to sell Amway') and humans do have experience with this sort of thing. The difference is scale: there'll be a LOT of models out there interacting with people and trying to be acknowledged as people… or as innocuous models that don't have an hidden agenda.
> They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators.
In other words, scams are going to massively increase in success rate ... and what are banks (for example) supposed to do? Other than SCREAM to governments for outlawing AI and trying to force responsibility on anyone else?
Democracy needs real journalism to function. Having all the rich people own all the journalists isn't going to end well. We need to find a working business model for journalism that doesn't rely on rich folks.
I think news outlets need to be run as a non-profit to remove the types of people with aspirations of wealth, instead of aspirations to report and inform the public, from the sector.
Usually rich folks buy newsrooms not to make a profit, but to control the narrative.
No journalist joins a newsroom to become rich. Famous, maybe, but not rich.
The business model used to be advertising, but the internet destroyed that model. And we don't have a replacement, while democracy doesn't work without someone holding the politicians to account.
According to the current administration, almost half of the US is considered a political enemy of the current administration.
Soon they might be pushing for Operating Systems to gather political party preference information, so they can know who should be restricted from the use of strong encryption. The options being:
It'll be interesting when/if they sanction Antifa. Since it doesn't exist, you can't prove that you're not a member of it. So they get to sanction anyone.
> move somewhere more willing to respect international law?
Some of these sanctions are required by international law (i.e. sanctions imposed by UNSC). For the other ones, international law generally lets countries have whatever trade policy they see fit including sanctions, unless they violate some other rule of international law or treaty obligation.
Sanctioning the ICC obviously has nothing to do with trade policy.
The USA signed the Rome Statute but never ratified it, and then withdrew its signatory status. There's an argument to be made that there was a treaty obligation there, but it's pretty weak.
I personally think sanctioning the ICC judges is a disgusting act. However ultimately all sanctions are decisions to refrain from trading with someone, so it is in a sense a trade policy. I think what you're getting at is that usa is implementing that policy to obtain a political/diplomatic goal, which is true, but you could say the same about most trade policies.
I think article 18(a) of the vienna convention of the law of treaties means that once you withdraw your signature, you no longer have any obligations in regards to the treaty.
Maybe you could make some sort of argument that the sanctions violate the purpose of the geneva convention as they are designed to prevent bringing to justice people accused of grave breaches of the geneva convention. Like its an attempt to frustrate the application of article 49 of the first geneva convention [Ianal]
I can't answer why or why not but just in terms of track record the US is fairly egregious. The executive attempts to coerce individual UN officials via sanctions. While it may not be strictly illegal it is clearly flagrantly unethical.
reply