ARM still lets you switch endianness, but nobody other than me ever seems to (and they claim they will probably deprecate that going forward).
What is really alarming to me is that I occasionally run into middle-endian systems on 64-bit chips (two little-endian doubles in big-endian relative order, to signify a single quad). This is an abomination and must be killed with fire.
Linkbait much? Further down below, Torvalds explains precisely why BE is anything but dead. All that's dead is general-purpose CPUs using it, mostly because x86 happened.
(Edit: which is precisely why I prefer BE for anything, unless there is a pressing - usually hardware- and performance-related - requirement for the opposite. Computers are good at reading bytes in swapped order. I suck at it.)
I think Linus' comment is more on the lines of picking an endian-ness and sticking with it rather than having to check the specific stream and swap bytes conditionally.
He specifically mentions cases where the data is stored in BE and the program reads it, swaps it, operates with it, swaps it back again and writes it.
The BE comment seems to me to be of much less importance than the other message (kind of like a side comment). And as you say, BE is very much alive today.
Well, and IBM switched Power to little endian recently (the machines are dual endian, but all the new OSs are littl endian only). Power was the main BE architecture left. Most arm machines are dual endian, and NetBSD supports both, but almost everyone uses little endian on arm. There is a bit of big endian mips around still, eg the Cavium network appliances.
"Computers are good at reading bytes in swapped order. I suck at it."
Byte endianess is about binary content, which isn't supposed to be read by humans. Humans read only interpretations of the binary data. You shouldn't come nowhere near to the question of if you are or aren't good at reading binary.
I'm not much into low-level CPU architectures, so I have to ask: Is there a technical reason (performance or whatever) why LE would be objectively better than BE?
I do read hex dumps & disassemblies, and there LE is a huge pain in the backend. Everything is in reverse. Why would a sane programmer voluntarily use LE?
I think the main advantage I see in LE architectures is that integral types don't have to relocate. For example, the int32 value 30 is represented in memory as "1E 00 00 00". The int16 value 30 is represented in memory as "1E 00". So casting downwards means that the pointers are the same, and you just ignore the trailing 0s. Contrast this to BE architectures, where you have "00 00 00 1E" vs "00 1E" -- casting from the larger to the smaller type means that you have to move the pointer.
Little-endian has a major advantage over big-endian...on eight-bit architectures: as you expand the size of the number, you can keep using up new digits at the end of the range, rather than the beginning. This means that you can more easily expand field widths.
Beyond that, it's called after two stupid factions in Gulliver's Travels for a reason...
Little-endian is generally better for compilers, to the extent that it matters. For example, with SIMD, it makes most sense to number the lanes in the same direction as addresses in memory, so that lane 0 is at offset 0 when doing a load or store. And if you want to reinterpret the bits of a SIMD value as a different type, little-endian is the only sane way to do it. And so on.
I've become fond of thinking about little-endian becoming known as a universal "CPU byte order", to complement "network byte order". Each order makes the most sense for its domain.
I won't give you a technical reason, but rather respond to the second part about LE and sanity.
Actually LE is the natural order and the question is why anybody sane would prefer BE. The reason you think otherwise is because you view hexdumps the wrong way.
A hexdump shows hexadecimal bytes. Isolate one byte and number the bits -- 76543210, right to left. Number the nibbles -- 10, right to left. Now number the bytes -- 0123..., left to right?
Display your hexdumps with addresses increasing right to left and LE makes perfect sense. This is how numbers are written, so the convention should be followed when displaying numbers.
Big-endian has always made more sense to me, since digits are laid out in memory the same way they are laid out in writing and therefore in my brain. Little-endian just seems like a cheap '70s kludge that accidentally became dominant thanks to Intel.
When Motorola 68000 was out, it was doing 32bits linear addressing.
The mov operation would load directly from the memory by using a direct addressing.
Wiring 32 wires is expensive.
Wiring 2 overlaping address bus of 16 bits partially ovelapping was less. Intel CPU ASM works big endian because we all prefer to write AND 0xFF, DSI to get the 8 lower bit of a number.
The first register would hit segment, and the second the offset.
Hence you would first need to load the page (that would wire the multiplexing) to get the offset in the page/segment.
The addressing bus would be accessed with a stack. So first you push the lower 16 bits, and then the highest 16 bits. Which would revert since its a FIFO in first loading the segment address, then the offset address.
The cost of this "savings" in terms of CPU cycle was marginal while the gain in money was tremenduous.
Going against what Linus suggests, The Dalvik DEX format (used by Android) has a flag representing the endianness of the enclosed bytecode: https://source.android.com/devices/tech/dalvik/dex-format.ht... (see "ENDIAN_CONSTANT and REVERSE_ENDIAN_CONSTANT"). By default, it's Little Endian.
I've always wondered WHY this flag is present - why would an implementation wish to change the byte-order? I could understand if the file contained machine code, and would vary with different architectures - but given it's running in a VM, why bother?
You can blame me for those wasted four bytes (or, perhaps, that wasted terabyte in the aggregate).
The original idea was that on big-endian machines, the resulting .odex would be endian-swapped compared to the original .dex, and this field would provide a reasonably-blatant indication. This would probably help with debugging more than anything else (maybe help with security, but not much), because there's no reason anyone would drop a little-endian .odex file on a big-endian machine (or a big-endian one on a little-endian machine).
We wrote (most of?) the code to get the vm working on big-endian systems, but within Google we never shipped any big-endian hardware AFAIK. I have my doubts that anyone ever did.
Even though the bytecode is being executed on a VM, the VM itself has been ported to the architecture, which has a native endianness. Byte order in the dex that matches the host OS's will likely result in a performance gain as the VM can cast more efficiently.
The Web is now little-endian thanks to ArrayBufferViews exposing endianness and everyone testing only the little-endian case. It doesn't make sense to make big-endian hardware that needs to run a Web browser anymore, so chances are that big-endian will never come back.
We use DataView wherever we can, but it's quite likely we do have endianness bugs in our code. AFAICT, some browsers on BE machines use LE typed buffers.
Do you know of any modern browsers that use BE typed buffers with full WebGL support for a cheap BE machine that we could buy for testing? Does anybody actually use such? Is it worth the bother?
From Wikipedia: [Gulliver's Travels] describes an intra-Lilliputian quarrel over the practice of breaking eggs. Traditionally, Lilliputians broke boiled eggs on the larger end; a few generations ago, an Emperor of Lilliput, the Present Emperor's great-grandfather, had decreed that all eggs be broken on the smaller end after his son cut himself breaking the egg on the larger end. The differences between Big-Endians (those who broke their eggs at the larger end) and Little-Endians had given rise to "six rebellions... wherein one Emperor lost his life, and another his crown". The Lilliputian religion says an egg should be broken on the convenient end, which is now interpreted by the Lilliputians as the smaller end. The Big-Endians gained favour in Blefuscu.
This got me thinking if in general (outside computing) it was a good choice having numbers represented the way they are. What if we had 1234 read as "four thousand, three hundred and twenty one"? Or even more different than how we pronounce it today - in order to read the number left-to-right to spell it "one and twenty and three hundred and four thousand"! This way the last digit lingering in your mind would be the one representing the most significant part of the given number (unlike the current practice of hearing first then tuning off for the less interesting smaller parts). Would it have been practical?
If you take into account the fact that you can obtain the order-of-magnitude for most "human" numbers without actually focusing on them, then putting the most-significant digit first in the stream makes sense. 30 is easy to see, even if you look at the 3 you can at least tell there's two digits. 12,048 is pretty easy. Even 8,293,254 isn't that hard to order-of-magnitude from the near-fovea while you're focused on the 8. By the time you get into the larger numbers that are hard, well, they're hard under either order so it's a wash.
The number system we use was imported to Europe from Arabia. In Arabic, the number 1234 is written with the most significant digit on the left -- but Arabic is written right-to-left, so it's actually little-endian. When they were introduced to Europe (with the help of Fibonacci), the convention of putting the most significant digit on the left was retained, but since European languages are written left-to-right the notation magically became big-endian.
The fact that the Roman numeral system that it replaced was also big-endian was likely an influence.
My gut feeling is that it would have been less so than the current solution.
Our current representation is great for skimming in a language written LtR. Placing the most significant digit all the way to the right would be like putting the lead paragraph of an article last.
I would not follow Linus advice here. To stick with a given endianess for a protocol? Sure! But why to use BE if everything is LE? If you use LE at compile time you can convert the conversion calls to no operations: zero cost. The cost will be incurred only into BE systems that are every day less common. So my suggestion is to stick to a specific endianess in protocols and serialization factors but picking LE instead of BE. Macros to convert the value that will do nothing if the host is already LE are trivial to write.
That's what Linus is saying: hey, BE is basically dead and you shouldn't use it, but if you really really want to use it, that's fine as long as you don't go for dynamic switching.
On modern architectures the byte swap is effectively free, if it is done at the time the data is read or written. Worry more about code that touches data just to swap bytes, which may have to page data in.
When POWER8 came out (ppc64le) the writing was pretty much on the wall. Is there any major or even minor architecture which still defaults to BE? SPARC possibly?
Yes, SPARC and zSeries. Among architectures supported by recently released Debian, zSeries is the only architecture that defaults to BE. (SPARC support is dropped.)
[+] [-] bandrami|11 years ago|reply
What is really alarming to me is that I occasionally run into middle-endian systems on 64-bit chips (two little-endian doubles in big-endian relative order, to signify a single quad). This is an abomination and must be killed with fire.
[+] [-] w8rbt|11 years ago|reply
[+] [-] weland|11 years ago|reply
http://geekandpoke.typepad.com/geekandpoke/2011/09/simply-ex...
(Edit: which is precisely why I prefer BE for anything, unless there is a pressing - usually hardware- and performance-related - requirement for the opposite. Computers are good at reading bytes in swapped order. I suck at it.)
[+] [-] loopbit|11 years ago|reply
He specifically mentions cases where the data is stored in BE and the program reads it, swaps it, operates with it, swaps it back again and writes it.
The BE comment seems to me to be of much less importance than the other message (kind of like a side comment). And as you say, BE is very much alive today.
[+] [-] justincormack|11 years ago|reply
[+] [-] restalis|11 years ago|reply
Byte endianess is about binary content, which isn't supposed to be read by humans. Humans read only interpretations of the binary data. You shouldn't come nowhere near to the question of if you are or aren't good at reading binary.
[+] [-] tempodox|11 years ago|reply
I do read hex dumps & disassemblies, and there LE is a huge pain in the backend. Everything is in reverse. Why would a sane programmer voluntarily use LE?
God Save the Network Byte Order!
[+] [-] andrewla|11 years ago|reply
[+] [-] gecko|11 years ago|reply
Beyond that, it's called after two stupid factions in Gulliver's Travels for a reason...
[+] [-] cliffbean|11 years ago|reply
I've become fond of thinking about little-endian becoming known as a universal "CPU byte order", to complement "network byte order". Each order makes the most sense for its domain.
[+] [-] mungoman2|11 years ago|reply
Actually LE is the natural order and the question is why anybody sane would prefer BE. The reason you think otherwise is because you view hexdumps the wrong way.
A hexdump shows hexadecimal bytes. Isolate one byte and number the bits -- 76543210, right to left. Number the nibbles -- 10, right to left. Now number the bytes -- 0123..., left to right?
Display your hexdumps with addresses increasing right to left and LE makes perfect sense. This is how numbers are written, so the convention should be followed when displaying numbers.
[+] [-] kahirsch|11 years ago|reply
[+] [-] marssaxman|11 years ago|reply
[+] [-] SFjulie1|11 years ago|reply
When Motorola 68000 was out, it was doing 32bits linear addressing.
The mov operation would load directly from the memory by using a direct addressing.
Wiring 32 wires is expensive.
Wiring 2 overlaping address bus of 16 bits partially ovelapping was less. Intel CPU ASM works big endian because we all prefer to write AND 0xFF, DSI to get the 8 lower bit of a number.
The first register would hit segment, and the second the offset.
Hence you would first need to load the page (that would wire the multiplexing) to get the offset in the page/segment.
The addressing bus would be accessed with a stack. So first you push the lower 16 bits, and then the highest 16 bits. Which would revert since its a FIFO in first loading the segment address, then the offset address. The cost of this "savings" in terms of CPU cycle was marginal while the gain in money was tremenduous.
[+] [-] DonHopkins|11 years ago|reply
[+] [-] SeanLuke|11 years ago|reply
http://spimsimulator.sourceforge.net/
[+] [-] JosephRedfern|11 years ago|reply
I've always wondered WHY this flag is present - why would an implementation wish to change the byte-order? I could understand if the file contained machine code, and would vary with different architectures - but given it's running in a VM, why bother?
[+] [-] danfuzz|11 years ago|reply
The original idea was that on big-endian machines, the resulting .odex would be endian-swapped compared to the original .dex, and this field would provide a reasonably-blatant indication. This would probably help with debugging more than anything else (maybe help with security, but not much), because there's no reason anyone would drop a little-endian .odex file on a big-endian machine (or a big-endian one on a little-endian machine).
We wrote (most of?) the code to get the vm working on big-endian systems, but within Google we never shipped any big-endian hardware AFAIK. I have my doubts that anyone ever did.
[+] [-] davtbaum|11 years ago|reply
[+] [-] hsivonen|11 years ago|reply
[+] [-] dtech|11 years ago|reply
[+] [-] blueflow|11 years ago|reply
[+] [-] bryanlarsen|11 years ago|reply
Do you know of any modern browsers that use BE typed buffers with full WebGL support for a cheap BE machine that we could buy for testing? Does anybody actually use such? Is it worth the bother?
[+] [-] krazydad|11 years ago|reply
http://en.wikipedia.org/wiki/Lilliput_and_Blefuscu
[+] [-] restalis|11 years ago|reply
[+] [-] jerf|11 years ago|reply
[+] [-] _kst_|11 years ago|reply
The number system we use was imported to Europe from Arabia. In Arabic, the number 1234 is written with the most significant digit on the left -- but Arabic is written right-to-left, so it's actually little-endian. When they were introduced to Europe (with the help of Fibonacci), the convention of putting the most significant digit on the left was retained, but since European languages are written left-to-right the notation magically became big-endian.
The fact that the Roman numeral system that it replaced was also big-endian was likely an influence.
[+] [-] chucksmash|11 years ago|reply
My gut feeling is that it would have been less so than the current solution.
Our current representation is great for skimming in a language written LtR. Placing the most significant digit all the way to the right would be like putting the lead paragraph of an article last.
[+] [-] antirez|11 years ago|reply
[+] [-] toyg|11 years ago|reply
[+] [-] loopbit|11 years ago|reply
[+] [-] njloof|11 years ago|reply
[+] [-] rwmj|11 years ago|reply
[+] [-] sanxiyn|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] nudpiedo|11 years ago|reply
[+] [-] olm|11 years ago|reply