I mean, "CI" is a complete misnomer. There is usually no "integration" happening while a CI pipeline runs, it's mostly just running a couple of tests.

Correct me if I'm wrong, but an MCP server still make a lot of sense for remote-controlling a UI application (like a game engine editor) which otherwise doesn't have any 'access points' for remote-control right?

An AmigaOS-style "scripting port" would work just as well, but AFAIK that's essentially what an MCP server is (a socket that implements a discoverable command protocol).

I would actually love if all applications and services could agree on a standard for scripting / remote-control. If the AI hype is what it takes to get there, then so be it ;)

Every few years we're all doomed to reinvent COM again.

The problem is the boundary with, say, windows. In rust they have WTF-8 more or less just to deal with windows filenames.

The internal string encoding and its limitations does leak into the APIs.

That combination is how it already works. You can build combined "characters" (grapheme clusters) from multiple code points, e.g. you could have a "base emoji" followed by a "modifier" emoji, and AFAIK that's how emojis with different skin colors work (one code point for the base emoji (e.g. 'thumbs up'), and a number of skin color modification code points which can be applied to all emojis that involve skin color.

> "UTF-8" support was often just single byte, so it was little more than ASCII

"Single byte UTF-8" is ASCII. That's one of its most important properties.

> Even now it's common for programs to choke on the optional BOM

And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.

Your opinion on the BOM isn't wrong, but it's also not germaine to whether or not you're actually following the spec. The spec is the spec. If you don't like it you can get the spec changed. You don't get to ignore the spec and then claim support. That's not how standards work. "I don't like it," isn't a good explanation.

Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.

> BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing

How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).

Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.

> How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM?

Simple: switch to UTF-8 as the only encoding standard for sharing text data, keep UTF-32 as 'internal' runtime format for random access to codepoints, and get rid of all other legacy encodings (UCS-2, UTF-16, Extended ASCII with code pages, and all the other region specific encodings that popped up in the 70s and 80s because UTF-8 wasn't invented yet.

This general switch to UTF-8 should have happend in the mid-to-late 1990s (e.g. together with the web becoming popular), and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a decade or so, the transition would have been finished by 2010. Alas, that never happened.

And tbh, the file size argument for alphabets that don't fit into 7-bit ASCII doesn't really make sense anymore today where images and videos make up the vast majority of data volume.

CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.

IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely larger. We obviously have both the bandwidth and storage volume to spare for those 30% increased size for text data which among all the other data would account for a tiny fraction of a percent.

OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).

The bonus is synchonizing at arbitrary point in stream and that ASCII is UTF-8

Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.

Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.

UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).

Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.

Truly the worst of all worlds.

Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.

What you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.

The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.

But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.

Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.

It starts like this, and ends up like x86.

> ...24-bit numbers?

Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".

It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.

Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)