Rewind to 2002. The internet was a different place back then, Unicode and UTF-8 had not yet taken the foothold it has now, and each language had its own encoding. Some had many. When I started working at Opera Software back in 2000, the browser only did ISO 8859-1, the encoding used in western Europe and the USA. We wanted to support more encodings, like Netscape Navigator and Internet Explorer already did, and decided to move to Unicode all the way through. This meant that we had to convert at the edges.

I took over the project that was doing this shortly after joining Opera Software, and was in charge of supporting all the various encodings used. For version 5, released in late 2000, we released a “Cyrillic version” as an experiment, which supported the various character encodings used for Russian (the one used in MS-DOS, the one used in Windows, the one used on Unix, and the ISO standard that no-one used). We already had a fanbase in the Russian-speaking part of the world, with hacked versions that displayed at least the Windows-encoded pages properly, so we leveraged this to test the conversion routines before switching to full Unicode.

Version 6, released in 2001, added full Unicode support. We used UTF-16 internally. An odd choice these days, but back then it made sense since it was what Windows NT used (and Windows 95 could be co-erced into using it, at least partially). Our Linux version was coming along, and we made that work as well, converting to UTF-8 for text rendering, which was easy enough.

The conversion code was part algorithm, part data table. Some encodings had the same data tables across different encodings (Shift-JIS and EUC-JP encode the same character set, just diffrently). Since we were aiming for a small binary footprint, we tried to make the data tables as compact as possible, so that meant making the code a bit more complex.

Then China happened. They already had a couple of encoding standards that encoded various sets of Chinese characters, with each one adding more characters. So we had one data table and used the subset that each encoding supported.

Then they dropped the GB18030 standard, which encoded everything else. They could of course have just used UTF-8, but wanted to make it compatible with their old standard, so what it did was filling in the gaps from the old standard to encode all remaining Unicode codepoints, whether defined or not. With the encoding standard document only being available in Chinese, a language that I unfortunately do not understand, all I had to work with was a couple of documents and a conversion table from IBM’s ICU (International Components for Unicode). This conversion table was huge, it was released as an XML source file, encoding the entire Unicode codespace. Even converting it to the binary format used in Opera, the table was larger than the entire Opera binary (possibly with the exception of the JavaScript engine, can’t remember now, but it was enormous).

That wouldn’t do. I would have preferred to drop the standard altogether, but China being China, this was set to be mandatory in all products sold in the country sold from 2003 or so, so we had to implement it to get a foothold there. After looking at the table and algorithm for a a while, I found a pattern. They had encoded all the missing characters, but they had added them in Unicode order. They had used a four-byte encoding for this, which I could easily convert into a scalar value, starting at 0 and going upwards. If I mapped this number to the Unicode codepoint, those were also increasing, but with holes in them.

I can’t remember the actual numbers now, but say that 0 mapped to 128, 1 to 129 then 2 suddenly mapped to 132 because 130 and 131 already was encoded. This could be optimized to a list saying that 0 maps to 128 and 2 to 132. From that we can infer that 1 maps to 129 by adding 1 to 128. I wrote a script that analyzed the enormous data table and extracted the points at where the offset changed. This shrank the first part of the table to about 200 bytes!

The second table was even easier. It mapped all Unicode codepoints beyond the first plane, i.e all codepoints that have five or more digits (when written hexadecimally). None of the previous encodings had encoded anything from there, so it was a simple 1-to-1 mapping that didn’t require a conversion table at all, all I had to write was the code to calculate the number, convert it into a Unicode codepoint and output that (as a pair of surrogate characters, as we were using UTF-16).

Writing the reverse converter (Unicode to GB18030), we just had to do the same conversion the other way and could use the same table as the forward. So, in the end we needed a small script, some extra code in the encoding engine and a small data table. The day was saved, and the binary still fit a 1.44 inch floppy disk. Try that today.

When Opera Software started the HTML 5 project, in conjunction with some of the other driving forces for web development, I made sure the algorithm I came up with was described in the encodings spec, but I think it has changed since due to various updates to the spec from after I stopped working there. A lot in that spec was based on my implementation of character encodings in Opera, which of course meant we followed it quite closely…