Python: Making ctypes Structures Beautiful

[densh]

People never seem to fully understand namespaces. You don't need any additional prefixes for classes and functions inside of your module. Its name already does separate it from everything else. Moreover "from module import *" is evil. Period.

import newstruct as ns

...

x = ns.long()

[DeepDuh]

Makes me think.. is the NS prefix in cocoa frameworks short for "newstruct"? I always wondered about that one.

[jessedhillon]

That's my module. Now, how can I make it better?

You could have those fields know their own sizes, and then you could have Struct subclasses know the size of their instances, so you don't have to call sizeof. E.g.:

    class DataHeader(Struct):
        address = newstruct.long()
        typ =     newstruct.byte()
        tag =     newstruct.short()
        data =    newstruct.byte(length=40) # 40 bytes

    >>> fp = open('mydata', 'rb')
    >>> dh = DataHeader.read(fp)
    # or perhaps: DataHeader.load(fp.read(len(dh)))
    # this way the side-effect of moving fp's current position is more explicit

    >>> fp.close()
    >>> len(dh)
    12345    # or whatever, sum of sizes for long, 41 bytes, and short

[JoachimSchipper]

Additional issues: is a long guaranteed to be 64 bits, or whatever the platform calls long? Is the structure padded (in the same way that the C compiler pads structs)? And, of course, which endianness is used?

[jevinskie]

My go-to library for binary mucking in Python is the fantastic Construct library. [0] [1] It is a declarative way of specifying binary structures. It handles endianness, serialization, and deserialization for you. You can build structures of structures with Construct. I used it to parse the PS3's pseudo-ELF files (SELFs) and was quite successful except for a few shortcomings in the library. I don't blame Construct though because the SELF format is a wild beast. Corbin Simpson (the author) was quite helpful when I was looking into the possibility of fixing my issues. Look at the ELF parser to see how easy it is to use! [2]

[0]: http://construct.readthedocs.org/en/latest/

[1]: https://github.com/construct/construct

[2]: https://github.com/construct/construct/blob/master/construct...

[wisesage5001]

One thing I really like about the struct module's format string is the ability to easily specify the endian-ness of the data. How does newstruct (or the underlying ctypes) handle endian-ness (if at all)?

[veyron]

endianness only matters when you bother interpreting the data. The storage concept shouldn't care -- after all, 8 bytes is 8 bytes is 8 bytes.

Struct needs it because it is interpreting data. unpack needs to know the endianness to correctly interpret numeric types, and pack needs to know the endianness to know how to serialize.

FYI: the C way of doing this is byteswapping.