> We know CJK languages need UTF-16 for compression.
My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest.
(.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.)
> JSON and CSV are woefully neglectful,
As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1
That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane".
(Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.)
> When the whole world looks to the future, Microsoft will follow.
That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).
You can fit Japanese comfortably in a 16 bit charset but Chinese needs more than that.
My take though is that CSV is not a good thing because the format isn't completely standardized, you just can't count that people did the right thing with escaping, whether a particular column is intended to be handled as strings or numeric values, etc.
Where I work we publish data files in various formats, I'm planning on making a Jupyter notebook to show people how to process our data with Pandas -- one critical point is that I'm going to use one of the commercial statistics data formats (like Stata) because I can load the data right the first time and not look back. (e.g. CSV = good because it is "open" is wrong)
If I am exporting files for Excel users I export an Excel file. Good Excel output libraries have been around for at least 20 years and even if you don't have fun with formulas, formatting and all that it is easy to make a file that people will load right the first time and every time.
> Good Excel output libraries have been around for at least 20 years
I wish that were the case more often. Depends on your ecosystem, of course.
For instance, I've yet to find a good XLSX library for JS that works well "idempotently" (in the browser as well as Node/etc). Every one I tried either had native dependencies and couldn't run in-browser or had a cost (time, money, size) I couldn't budget for at the time.
I have found some XLS libraries for JS that were extremely "mid", but outputting XLS is nearly as bad as CSV in 2024. (Including all the huge messy legacy of character set Encoding problems.)
The best and worst thing about CSV is that it seems "low overhead": it seems really cheap to output. ("How hard can it be, just ','.join(records)?" has been the pit so many of us fall into over and over again and sometimes never come out.) In terms of low overhead: in a world where all my APIs are already talking JSON if I can wrap an existing HTTP API with just two extra headers to get "free" Excel files for my users, that could be a beautiful world:
All the pieces are already there. If you could teach every user to use "Data > From JSON" we could maybe have nice things today instead of yet another CSV export dump. We just need someone on the Excel team to greenlight a "double to click to open an .XLJSON file" feature.
First and foremost to avoid accidents: We don't want a misconfigured HTTP website accidentally opening a new Excel window for every fetch/XHR call or to have to fight Excel defaults to get JSON to open up in our IDE or Dev Tools of choice. We don't want random shell scripts accidentally curl-ing things to Excel. Things like that.
Secondly for "ownership" reasons: We don't want to give non-developers the mistaken impression that Excel "owns" JSON and that it is a Microsoft format. I've had people tell me that CSV must be a Microsoft format because you can double click them in Excel and they show an Excel-like icon (in some ways it has been too long since Lotus existed and Excel was in the "we'll take all of our competitors' file associations too" era). On the one hand it might be nice to blame all of CSV's problems on Microsoft and Excel if that were actually the case, but on the other hand it also confuses people as to the real/valid uses of the format. Unfortunately, too, that transitive relationship goes both ways and I've heard second hand that CSV files are among the reasons the Excel team hopes to never add another file type association again because supposedly they get far too many support requests for CSV file problems that maybe shouldn't be their job to deal with.
A separate file association adds some intent of "this file was actually meant to be opened in Excel and hopefully the developer actually tested it some".
My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest.
(.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.)
> JSON and CSV are woefully neglectful,
As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1
That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane".
(Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.)
> When the whole world looks to the future, Microsoft will follow.
That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).