top | item 26019931

(no title)

wesm | 5 years ago

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.

discuss

order

waynesonfire|5 years ago

Of course there is. There is always deserialization. The data format is most definitely not native to the CPU.

wesm|5 years ago

I challenge you to have a closer look at the project.

Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.