> deliberate backdoors will be more

But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?

I'm not sure you understand what is being described here?

If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."

Note that this does certainly already happen a lot, but it's also not something that can happen across the board (like the author points out). Apple, being one of the largest companies in the world and who has one of their biggest selling points being their security, has explicitly refused to do this to the point that the intelligence agencies couldn't break into an iPhone until another company found a way to do so.

>I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases?

The people working on these codebases are "in on it." Of course, we're talking backdoors which are very subtle, target very minimal infrastructure, and are known about by very few people. But, like I said, companies are currently shipping products with backdoors in them knowing they exist already. AI doesn't change that dynamic.

>Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it?

The only people who would have access to the systems that can even be flagged by this stuff would be people who would know about it. Keep in mind that the scale we're talking here is massive. Think about how software development works at companies like Apple, Microsoft, Google, etc. There are devs working in offices all around the world where they only ever have access to a fraction of the code that company owns. These companies are very capable of keeping their stuff locked down. It's a necessary component of their work.

>How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?

Hopefully my explanation at this point is clear, but just to be concrete: backdoors are, by design, very hard to detect. That doesn't mean they're just sneakily written code that humans don't notice as they read over it, but, instead, they're very subtle implementations in very specific parts of huge systems that are already locked down to the point that the number of people who even have access to those portions of the systems are very limited. These agencies don't slip in backdoors without anybody noticing; they convince the minimal number of people needed to know about it to implement it. Again, we're talking about a meeting between the directory of a three letter agency and a CEO, where the CEO then directs the CTO to implement the backdoor who then instructs the handful of very high-ranking engineers to do so.

AI systems in these companies may very well flag these backdoors to the people who already know they exist, then these people can tell the AI "hey, those are their on purpose, so just move on," and the other 99.99% of the company will never know they exist.

I suppose it's important to emphasize, again, that these systems are incredibly massive and complex and most people at these orgs don't have any access to most of these systems, so it's not like you can expect an intern running BugBot across a repo and expecting it to find a backdoor.

I think people take the FBI or CIA too literally. I imagine they don't need to talk to owners and it's probably not even ideal. It might just be easier to get plants in the organization.

I would imagine most big companies, like Microsoft, have dozens of CIA and FBI plants in their organizations. Agents who are legitimate software engineers, tasked with acquiring intelligence and undermining security.

> If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."

And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".

(And you move out of the US.)

And since this is a foreseeable future, you should start acting now to prepare for that future.

With laws like Chatcontrol and all, other jurisdictions are not necessarily any better.

In fact, we're increasingly seeing a desire to build the "backdoor" directly into the software, e.g. mandatory age verification, client-side scanning, etc.

> And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".

> (And you move out of the US.)

And then everybody claps.

Name me a software or hardware company—one big enough that the US government would actually care to force them to add a backdoor—that would be willing to give up the US market? The only one that's shown the least bit of backbone is Apple, and while I like them and appreciate what they've done in that vein so far, they're never going to move to open source software running their stuff, and they're pretty well embedded in the US, and very, very unlikely to try to move regardless of the headwinds there.

I'm fully with you that this would be a wise and moral thing to do, but frankly, our tech companies are neither wise nor moral. They are self-serving, greedy, and many of them have wanted to become the neofeudal overlords of a new order since before Trump started smashing the old one.

I'm not suggesting giving up the US market. I'm suggesting moving out of the US and continuing to serve the US market from elsewhere, because the US does not have a nation-wide firewall. And working with organizations mounting legal challenges to "please destroy your company in order to put in a backdoor for us".

Certificate Transparency has essentially eliminated the problem of backdoored CAs, because attempting to do so would destroy an entire CA. Binary Transparency can do the same for software.

I'm not sure exactly what you think that will accomplish...?

Companies have to follow the laws of the countries they operate in, not just the countries their physical headquarters are in.

That's why, for instance, Apple has to follow the DMA in Europe.

Furthermore, especially for many of the tech companies, where they are located is an integral part of their culture. They are Silicon Valley. You're going to have a very, very hard time convincing any of them to up stakes and move.

And further-furthermore, move where? Europe has, unfortunately, made similar authoritarian noises (eg, Chat Control). China is already more of an authoritarian state than even Trump's USA. Ditto for Russia, and, AIUI, India, though both in somewhat different ways.

The degree to which you have to follow the laws of a place you have no legal nexus in (e.g. no employees) is limited. Non-US companies can and do refuse overreaching requests from the US sometimes.

Apple is subject to the DMA in europe because 1) they have employees in Europe and 2) they want to import and sell physical phones into Europe. It's easy to block imports or fine employees. It is not especially easy to prevent people from spending money on a service provided entirely via the Internet from another country.

It is not at all impossible to impose sanctions on an entire company. But if the US tried to do so over a refusal to put in a backdoor, and the company very loudly made that clear, that would to some extent be exceptional free marketing.

Why can't the backdoors themselves be more sophisticated? I understand that expecting such sophistication from legacy companies is a joke at this point.

There's no such thing. If it can be cracked open, then it's only a question of time until every criminal on earth has a matching crowbar.