Show HN: Ldump – serialize any Lua data

[girvel]

Some time ago, I was implementing saves for my LOVE2D game. I wanted to do a full dump of the game state -- which included closures (AI), complex graphs, sets with tables as keys and also fundamentally non-serializable data (coroutines and userdata), that require user-defined serialization/deserialization logic. I went through every Lua serialization library -- none covered all data types/cases. So I wrote my own.

It is a polished version, thoroughly annotated, tested and documented. It is made to be as functional and customizable as possible (or at least I did everything I could think of). I would be happy to hear suggestions/corrections for both code and documentation -- even nitpicky ones.

[lifthrasiir]

Maybe I'm too pedantic but allowing anything to be "deserialized", which equals to "evaluated" here, is not secure. I think it only has to accept a very limited subset of Lua anyway, so you may switch to a non-Lua format which is made easy to parse. That way the library has a total control over what is being evaluated.

[girvel]

This is an interesting thought. Currently, it is unsafe and intended to load only the files you trust. I should definitely include a warning into README.

Overall, it would be nice to make it safer. I don't think switching to non-Lua format would make it safer, because it is intended to serialize functions too, which can have arbitrary code even if everything else would be stored as data. Maybe it is possible to make a function like `ldump.safe_load` restricting `load`'s environment, so it wouldn't have access to debug/os/io modules.

[ithkuil]

"lump" would have been a nice name

[girvel]

Damn, it really would

[JourneyJourney]

I'm afraid I spent too much time with LUA lately and fell in love with its simplicity. Kinda hard to go back to OOP after that.

[nicoloren]

Same for me, I used Lua for a desktop software for a client and I enjoyed it a lot!

I'm thinking of starting to dev a game with LOVE2D just to have an excuse to use Lua.

[je42]

I find that most LUA code sits between functional, imperative and OOP.

[aldanor]

It's simple until you dig deep into meta tables lol

[koeng]

I've been looking for something similar! Here is what I'd like to do:

I have a long-running script. At several steps, the execution of the script has to pause for a long time for operations to be done in-real-life (biological experiments, so think wait time being like 2 days between running), before getting some data and continuing to run. From what I can see in this, I'd add yielding coroutines at data pause points, right? How would you handle that?

[appleorchard46]

Very cool! I was just needing something like this for my Defold game, this looks way better than my hacky solution.

Semi-unrelated - you say you're using tables as keys in your project. I didn't know you could do that! What are you using it for?

[girvel]

For example, you want to keep a set of objects -- then the objects themselves would be keys, and values would be true/nil. Or there is a good example in one of ldump's recent issues: https://github.com/girvel/ldump/issues/44, where the loaded packages are stored in a table as keys to easily detect external module usage.

[GranPC]

FWIW anything in Lua can be used as a key - including functions, userdata, etc.

[sebstefan]

So it also dumps functions and is able to import them back?

Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)?

Even in the linked test case on the readme you don't show the output/expectation of the serialization

[myrmidon]

> can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?

Yes, it dumps them as bytecode (probably not compatible between completely different interpreters).

It even preserves debug metadata, so stack traces involving serialized/deserialized functions look right, and still show the original source file.

This is really neat.

[lifthrasiir]

Functions are apparently serialized as a bytecode dump contained in a self-extracting expression. So everything can indeed be serialized as a Lua expression. Seems that the author also tried to preserve as many upvalues as possible, though I feel that is way more dangerous than I would like.

[girvel]

The function (even a closure) would be fully recreated on deserialization, it is fully safe to save it to disk. It wouldn't preserve reference equality -- it would be a new function -- but the behaviour and the state (if using closures) would be equivalent.

I didn't include asserts in the linked case, because I thought it would be too verbose. You can see asserts in the test, that is linked below the example. Maybe it was the wrong call, I will think about including asserts into the example itself.

[bflesch]

Nice - I wonder why something like this is not built-in to the language itself. Especially debugging tables is painful to say the least :)

[nicoloren]

You can use inspect.lua to debug tables [1]

[1] https://github.com/kikito/inspect.lua

[sebstefan]

For most things in Lua you have to bring your own wheel

[jhatemyjob]

Try running this in a repl and tell me what you get, OP:

    string.format('%q', 'hi\n')

[myrmidon]

If you insinuate that %q obviates the need for ldump then you are wrong.

There is not even significant overlap in what they do; all that %q does is sufficiently escape Lua strings so the interpreter can read them back. It does not serialize functions nor even tables in any shape or form.

edit: Sorry for being unreasonably harsh after misunderstanding your message.

[girvel]

On my machine it produces an equivalent string, although differently formatted. It seems that ldump preserves all special characters (`"\a\b\f\n\r\t\v\\\"\'"`), although I will need to test in on all supported versions.

[gvx]

Cool to see you were inspired by Ser!

[girvel]

Oh wow, didn't expect to meet the author, thank you!

[brunocroh]

I will try it on my next love2d project, thank you!

[synergy20]

newbie question,when is this useful in practice

[girvel]

It is intended to be used in cases where you need to store data on a disk or transfer it to another machine -- like in a video game save or a network data exchange

[098527222722]

use computer divices