top | item 46464235

FracturedJson

575 points| PretzelFisch | 1 month ago |github.com

164 comments

order

simonw|1 month ago

It looks like there are two maintained implementations of this at the moment - one in C# https://github.com/j-brooke/FracturedJson/wiki/.NET-Library and another in TypeScript/JavaScript https://github.com/j-brooke/FracturedJsonJs. They each have their own test suite.

There's an older pure Python version but it's no longer maintained - the author of that recently replaced it with a Python library wrapping the C# code.

This looks to me like the perfect opportunity for a language-independent conformance suite - a set of tests defined as data files that can be shared across multiple implementations.

This would not only guarantee that the existing C# and TypeScript implementations behaved exactly the same way, but would also make it much easier to build and then maintain more implementations across other languages.

Interestingly the now-deprecated Python library does actually use a data-driven test suite in the kind of shape I'm describing: https://github.com/masaccio/compact-json/tree/main/tests/dat...

That new Python library is https://pypi.org/project/fractured-json/ but it's a wrapper around the C# library and says "You must install a valid .NET runtime" - that makes it mostly a non-starter as a dependency for other Python projects because it breaks the ability to "pip install" them without a significant extra step.

odyssey7|1 month ago

This is a good idea, though I don’t think it would guarantee program equivalence beyond the test cases.

klysm|1 month ago

This is also basically a pure function which makes it super simple to write a harness.

EmilStenstrom|1 month ago

Data driven test suites are really good for building trust in a library. Both the html5lib-tests suite and my recent xss-bench are examples of this!

tonixx|1 month ago

[deleted]

fcoury|1 month ago

I ported it to Rust with a cli tool that allows you to format json in this format:

https://github.com/fcoury/fracturedjson-rs

https://crates.io/crates/fracturedjson

And install with:

cargo install fracturedjson

    > $ fjson --help
    Rust port of FracturedJsonJs: human-friendly JSON formatter with optional comment support.

    Usage: fjson [OPTIONS] [FILE]...

    Arguments:
      [FILE]...  Input file(s). If not specified, reads from stdin

    Options:
      -o, --output <FILE>
              Output file. If not specified, writes to stdout
      -c, --compact
              Minify output (remove all whitespace)
      -w, --max-width <MAX_WIDTH>
              Maximum line length before wrapping [default: 120]
      -i, --indent <INDENT>
              Number of spaces per indentation level [default: 4]
      -t, --tabs
              Use tabs instead of spaces for indentation
          --eol <EOL>
              Line ending style [default: lf] [possible values: lf, crlf]
          --comments <COMMENTS>
              How to handle comments in input [default: error] [possible values: error, remove, preserve]
          --trailing-commas
              Allow trailing commas in input
          --preserve-blanks
              Preserve blank lines from input
          --number-align <NUMBER_ALIGN>
              Number alignment style in arrays [default: decimal] [possible values: left, right, decimal, normalize]
          --max-inline-complexity <MAX_INLINE_COMPLEXITY>
              Maximum nesting depth for inline formatting (-1 to disable) [default: 2]
          --max-table-complexity <MAX_TABLE_COMPLEXITY>
              Maximum nesting depth for table formatting (-1 to disable) [default: 2]
          --simple-bracket-padding
              Add padding inside brackets for simple arrays/objects
          --no-nested-bracket-padding
              Disable padding inside brackets for nested arrays/objects
      -h, --help
              Print help
      -V, --version
              Print version

re|1 month ago

Ports are a derivative work; you should preserve the original author's copyright attribution.

kstenerud|1 month ago

This is great! The more human-readable, the better!

I've also been working in the other direction, making JSON more machine-readable:

https://github.com/kstenerud/bonjson/

It has EXACTLY the same capabilities and limitations as JSON, so it works as a drop-in replacement that's 35x faster for a machine to read and write.

No extra types. No extra features. Anything JSON can do, it can do. Anything JSON can't do, it can't do.

esrauch|1 month ago

This is very interesting, though the limitations for 'security' reasons seem somewhat surprising to me compared to the claim "Anything JSON can do, it can do. Anything JSON can't do, it can't do.".

Simplest example, "a\u0000b" is a perfectly valid and in-bounds JSON string that valid JSON data sets may have in it. Doesn't it end up falling short of 'Anything JSON can do, it can do" to refuse to serialize that string?

kreco|1 month ago

Can you tell me what was the context that lead you to create this?

Unrelated JSON experience:

I worked on a serializer which save/load json files as well as binary file (using a common interface).

From my own use case I found JSON to be restrictive for no benefit (because I don't use it in a Javascript ecosystem)

So I change the json format into something way more lax (optional comma, optional colon, optional quotes, multi line string, comments).

I wish we would stop pretending JSON to be a good human-readable format outside of where it make sense and we would have a standard alternative for those non-json-centric case.

I know a lot of format already exists but none really took off so far.

imiric|1 month ago

That's neat, but I'm much more intrigued by your Concise Encoding project[1]. I see that it only has a single Go reference implementation that hasn't been updated in 3 years. Is the project still relevant?

Thanks for sharing your work!

[1]: https://concise-encoding.org/

gritzko|1 month ago

Probably, we need a formal data format, because JSON is just a notation. It does not mandate the bit width of numbers, for example, or whether ints are different from floats. Once there is such formal model, we can map it 1:1 between representations.

I am writing this because I work on a related topic https://replicated.wiki/blog/args.html

krick|1 month ago

What about compression rates?

zzo38computer|1 month ago

I think JSON is too limited and has some problems, so BONJSON has mostly the same problems. There are many other formats as well, some of which add additional types beyond JSON and some don't. Also, a few programs may expect (and possibly require) that files may contain invalid UTF-8, even though it is not proper JSON (I think it would be better that they should not use JSON, due to this and other issues), so there is that too. Using normalized Unicode has its own problems, as does allowing 64-bit integers when some programs expect it and others don't. JSON and Unicode are just not good formats, in general. (There is also a issue with JSON.stringify(-0) but that is an issue with JavaScript that does not seem to be relevant with BONJSON, as far as I can tell.)

Nevertheless, I believe your claims are mostly accurate, except for a few issues with which things are allowed or not allowed, due to JavaScript and other things (although in some of these cases, the BONJSON specification allows options to control this). Sometimes rejecting certain things is helpful, but not always; for example sometimes you do want to allow mismatched surrogates, and sometimes you might want to allow null characters. (The defaults are probably reasonable, but are often the result of a bad design anyways, as I had mentioned above.) Also, the top of the specification says it is safe against many attacks, but these are a feature of the implementation, which would also be the case if you are implement JSON or other formats (although the specification for BONJSON does specify that implementations are supposed to check for these things to make them safe).

(The issue of overlong UTF-8 encodings in IIS web servers is another security issue, which is using a different format for validation and for usage. In this case there are actually two usages though, because one of these usages is the handling of relative URLs (using the ASCII format) and the other is the handling of file names on the server (which might be using UTF-16 here; in addition to that is the internal format of the file paths into individual pieces with the internal handling of relative file paths). There are reasons to avoid and to check for overlong UTF-8 encodings, although this is a different more general issue than the character encoding.)

Another issue is canonical forms; the canonical form of JSON can be messy, especially for numbers (I don't know what the canonical form for numbers in JSON is, but I read that apparently it is complicated).

I think DER is better. BONJSON is more compact but that also makes the framing more complicated to handle than DER (which uses consistent framing for all types). I also wrote a program to convert JSON to DER (I also made up some nonstandard types, although the conversion from JSON to DER only uses one of these nonstandard types (key/value list); the other types it needs are standard ASN.1 types). Furthermore, DER is already canonical form (and I had made up SDER and SDSER for when you do not want canonical form but also do not want the messiness of BER; SDSER does have chunking and does not require the length to be known ahead of time, so more like BONJSON in these ways). Because of the consistent framing, you can easily ignore any types that you do not use; even though there are many types you do not necessarily need all of them.

polshaw|1 month ago

Is there an option for it to read the contents from a pipe? that's by far my biggest use for the jq app.

simonw|1 month ago

There's a C# CLI app in the repo: https://github.com/j-brooke/FracturedJson/blob/main/Fracture...

  Output is to standard out, or a file specified by the --outfile switch.  Input is from either standard in, or from a file if using the --file switch
It looks like both the JavaScript version and the new Python C# wrapper have equivalent CLI tools as well.

ruuda|1 month ago

RCL (https://github.com/ruuda/rcl) pretty-prints its output by default. Pipe to `rcl e` to pretty-print RCL (which has slightly lighter key-value syntax, good if you only want to inspect it), while `rcl je` produces json output.

It doesn’t align tables like FracturedJson, but it does format values on a single line where possible. The pretty printer is based on the classic A Prettier Printer by Philip Wadler; the algorithm is quite elegant. Any value will be formatted wide if it fits the target width, otherwise tall.

vhcr|1 month ago

I don't know, but you can always use <() process substitution to create a temp file.

pimlottc|1 month ago

You can (usually) specify the input file name as “-“ (single hyphen) to read from stdin

tuetuopay|1 month ago

this would be amazing to be chained with jq, that was my first thought as well.

damnitbuilds|1 month ago

Nice.

And BTW, thanks for supporting comments - the reason given for keeping comments out of standard Json is silly ( "they would be used for parsing directives" ).

Xymist|1 month ago

It's a pretty sensible policy, really. Corollary to Hyrum's Law - do not permit your API to have any behaviours, useful or otherwise, which someone might depend on but which aren't part of your design goals. For programmers in particular, who are sodding munchkins and cannot be trusted not to do something clever but unintended just because it solves a problem for them, that means aggressively hamstringing everything.

A flathead screwdriver should bend like rubber if someone tries to use it as a prybar.

patates|1 month ago

XML people were doing crazy things in the Java/.NET world and "<!--[if IE 6]>" was still a thing in HTML when JSON was being designed.

I also would have wanted comments, but I see why Crockford must have been skeptical. He just didn't want JSON to be the next XML.

frizlab|1 month ago

Unrelated: why spaces inside the parentheses? It’s not the first time I see this, but this is incorrect!

barishnamazov|1 month ago

This is pretty cool, but I hope it isn't used for human-readable config files. TOML/YAML are better options for that. Git diff also can be tricky with realignment, etc.

I can see potential usefulness of this is in debug mode APIs, where somehow comments are sent as well and are rendered nicely. Especially useful in game dev jsons.

actionfromafar|1 month ago

Yaml is the worst. Humans and LLMs alike get it wrong. I used to laugh at XML but Yaml made me look at XML wistfully.

Yaml - just say Norway

silvestrov|1 month ago

Just say Norway to YAML.

_pdp_|1 month ago

Yaml works really well with LLMs (not to generate but to consume). So yes, we use it all the time in our service.

kerblang|1 month ago

I made a silly groovy script called "mommyjson" that doesn't try to preserve JSON formatting but just focuses on giving you the parentage (thus the name) including array indexes, object names, etc., all on the same line, so that when you find something, you know exactly where it is semantically. Not gonna claim that everybody should use it or that it cures insomnia cancer & hangnails, but feel free to borrow it:

https://github.com/zaboople/bin/blob/master/mommyjson.groovy

(btw I would happily upvote a python port, since groovy is not so popular)

asa400|1 month ago

This is good! There are a number of these, so it seems like it's definitely somthing people want. The most popular of which I think is gron[0]. My own is jstream[1]. One tiny point of friendly feedback: you may want to consider adding an example usage/output so folks can see what it does literally.

[0] - https://github.com/tomnomnom/gron [1] - https://github.com/ckampfe/jstream

croisillon|1 month ago

do you have an ouput example?

miguelbemartin|1 month ago

Is JSON a format that needs improvement for human readability? I think there are much better ways to present data to users, and JSON is a format that should be used to transfer data from system to system.

btown|1 month ago

If you're reaching for a tool like this, it's because you don't have a well-defined schema and corresponding dedicated visualization; you're looking at some arbitrary internal or insufficiently-documented transfer-level data with nested structure, perhaps in the midst of a debug breakpoint, and need a quick and concise visualization without the ability (or time) to add substantial code into the running runtime. Especially if you're working on integration code with third parties, it's common to come across this situation daily.

CamouflagedKiwi|1 month ago

I think yes? I fairly often find that I have something in JSON, which probably is from some system to system comms, and I'm trying to read it. Once it's not trivially small I often pipe it through jq or python -m json.tool or whatever, I like the idea of something that just does a better job of that.

setr|1 month ago

If you discard the human-readability component of it, JSON is an incredibly inefficient choice of encoding. Other than its ubiquity, you should only be using JSON because it’s both human and machine readable (and being human-readable is mainly valuable for debugging)

zzo38computer|1 month ago

I think JSON is not really so good either way, due to problems with the data and with the file format.

frizlab|1 month ago

This is interesting. I’d very much like to see a code formatter do that kind of thing; currently formatters are pretty much inflexible, which makes getting structure out of a formatted code sometimes hard.

thechao|1 month ago

I just built a C++ formatter that does this (owned by my employee, unfortunately). There's really only two formatting objects: tab-aligned tables, and single line rows. Both objects also support a right-floating column/tab aligned "//" comment.

Both objects desugar to a sequence of segments (lines).

The result is that you can freely mix expression/assignment blocks & statements. Things like switch-case blocks & macro tables are suddenly trivial to format in 2d.

Because comments are handled as right floating, all comments nicely align.

I vibe coded the base layer in an hour. I'm using with autogenerated code, so output is manually coded based on my input. The tricky bit would be "discovering" tables & block. I'd jus use a combo of an LSP and direct observation of sequential statements.

barishnamazov|1 month ago

Right. In my previous work, I wrote a custom XML formatter for making it look table-like which was our use case. Of course, an ideal solution would have been to move away from XML, but can't run away from legacy.

__MatrixMan__|1 month ago

When I want something more readable than json I usually use nushell. The syntax is almost the same and you can just pipe through "from json" and "to json" to convert: https://gist.github.com/MatrixManAtYrService/9d25fddc15b2494...

What I like about fractured json is the middle ground between too-sparse pretty printing, and too-compact non-pretty printing, nu doesn't give me that by default.

One thing that neither fractured json nor nushell gives me, which I'd like, is the ability to associate an annotation with a particular datum, convert to json, convert back to the first language, and have that comment still be attached to that datum. Of course the intermediate json would need to have some extra fields to carry the annotations, which would be fine.

whoamii|1 month ago

Why reinvent the parser when the role of this library is formatting?

CamouflagedKiwi|1 month ago

I really like this, I think I'd find it useful fairly often and I like the idea of just making something that I use irregularly but not that rarely a bit better.

But then I found it's in C#. And apparently the CLI app isn't even published any more (apparently nobody wanted it? Surprises me but ok). Anyway, I don't think I want this enough to install .NET to get it, so that's that. But I'd have liked a version in Go or Rust or whatever.

j-brooke|1 month ago

I'm the maintainer of FracturedJson. The decision to stop publishing a binary for the CLI version was made a long time ago: fewer features, fewer users, less mature .NET tooling (as far as I knew). And as you say, .NET isn't a common language for distributing CLI tools.

I plan to take a new look at that when I have the time. But a port to a more CLI-friendly platform could probably do a better job.

kayhantolga|1 month ago

These JSON files are actually readable, congrats. I’m wondering whether this could be handled via an additional attached file instead. For example, I could have mycomplexdata.json and an accompanying mycomplexdata.jsonfranc. When the file is opened in the IDE, the IDE would merge the two automatically.

That way, the original JSON file stays clean and isn’t polluted with extra data.

zzo38computer|1 month ago

> That way, the original JSON file stays clean and isn’t polluted with extra data.

FracturedJson does not add any extra data; it only changes the formatting (it is a way of automatically formatting JSON data, not a new file format). However, the documentation mentions that in some cases it does reorder or rewrite things (such as the order of keys, the number of decimal places, etc).

If you set CommentPolicy=TreatAsError then programs that convert it to a canonical form (whether or not that canonical form is JSON or some binary format intended to be like JSON) should (hopefully) result in the same output with the original and the one converted by FracturedJson, depending on what things are considered to be significant. (I tested this with a program I wrote, which converts JSON to DER (which is a canonical form (and, in my opinion, usually the only good one) of ASN.1), and does not consider the order of keys or the representation of numbers to be significant (although the conversion of numbers does not lose any precision and is converted exactly, but e.g. "1.2" is considered the same as "1.200" and "80e1" is considered the same as "800").)

andix|1 month ago

I like this idea a lot. Currently the biggest issue for adoption seems to be the missing packages for most programming languages, and for homebrew/etc.

It should even be possible to compile the dotnet library to a C-compatible shared library and provide packages for many other languages.

jedbrooke|1 month ago

I had to do a double take on the repo author here :)

this tool also looks super useful, I spend so much time at work looking at json logs that this will surely come in handy. It’s the kind of thing I didn’t even know I needed, but now that I saw it it makes perfect sense.

DJBunnies|1 month ago

While I wish JSON formally supported comments, it seems more sensible (compatible) to just nest them inside of a keyed list or object as strings.

  {
    foo: "bar",
    ans: 42,
    comments: {
      ans: "Douglas Adams"
    }
  }

Etheryte|1 month ago

Works right up until you get an entity where the field `comments` is suddenly relevant and then you need to go change everything everywhere. Much better to use the right tool for the job, if you want JSONC, be explicit and use JSONC.

ljm|1 month ago

Personally, I think if your JSON needs comments then it's probably for config or something the user is expected to edit themselves, and at that point you have better options than plain JSON and adding commentary to the actual payload.

If it's purely for machine consumption then I suspect you might be describing a schema and there are also tools for that.

NooneAtAll3|1 month ago

idk... "ans: 42 // an old reference from DA API" seems easier to read than wasting 4 lines of yours

multiply that for a long file... it takes a toll

---

also sometimes one field contains a lot of separate data (because it's straight up easier to deserialize into a single std::vector and then do stuff) - so you need comments between data points

shiandow|1 month ago

This looks very readable. The one example I didn't like is the expanded one where it expanded all but 1 of the elements. I feel like that should be an all or norhing thing, but there's bound to be edge cases.

peterallport|1 month ago

I could see FracturedJson being great for a browser extension or preview extension in IDE, as in just for _viewing_ JSON in a formatted way, not formatting the source.

piskov|1 month ago

The lengths people go to (not) use the XML. It has everything: comments, validation, schema, what have you.

Though, I guess, the only(?) great XML workflow is with C# LINQ

strunz|1 month ago

Generally much larger for the same date and not readable unless using something that indents. Even then, I'd argue it's still less legible.

whoamii|1 month ago

Love the spirit, but the attack-plans example IMO looks worse with this formatting. I don’t love the horizontal scrolling through properties of an object.

ramraj07|1 month ago

I dont know if you spend a fraction of your life scrolling vertically through megabyte sizes json files, but if something can reduce the height of the file thats welcome. We dont need to read every single line fro left to right, we just need to quickly browse through the entire file. If a line in this format is longer than fits the screen, its likely we dont need to know whats in the cut off right corner anyway.

tracker1|1 month ago

Nice... I like using JSON to stdout for logging, this would be a nice formatting option when doing local dev to prettify it without full decomposition.

rednafi|1 month ago

How many times do you actually need to look at large JSONs? The cost of readability is too high, IMO.

Personally, I don't spend much time looking at complex JSON; a binary format like Protobuf along with a typed DSL is often what you need. You can still derive JSON from Proto if you need that. In return, you get faster transport and type safety.

Also, on another note, tools like jq are so ubiquitous that any format that isn't directly supported by jq will have a really hard time seeing mass adoption.

vitaelabitur|1 month ago

I tokenized these and they seem to use around 20% less tokens than the original JSONs. Which makes me think a schema like this might optimize latency and costs in constrained LLM decoding.

I know that LLMs are very familiar with JSON, and choosing uncommon schemas just to reduce tokens hurts semantic performance. But a schema that is sufficiently JSON-like probably won't disrupt model path/patterns that much and prevent unintended bias.

nurumaik|1 month ago

Minified json would use even less tokens

lifetimerubyist|1 month ago

If my json is too complicated to be readable I just use jq to find the things I want out of it.

magius18|1 month ago

All this work and there's no mention of YAML on the repository is kind of funny to me

__MatrixMan__|1 month ago

The trouble with yaml is that it's too hard to keep track of how indented something is if its parent is off the screen. I have to keep a t-square on my desk and hang it from the top of my monitor whenever this comes up.

That, and the fact that it has enough bells and whistles to that there are yaml parser exploits out there.

andix|1 month ago

Let's implement this formatting in all code editors and replace YAML with it :)

Androcles|1 month ago

That is remarkable. I recently implemented this very functionality in Python using roughly 200 lines of code, completely unaware that a pre-built library was available.

jesse__|1 month ago

I love doing this type of formatting with source code. It'll be nice when people start writing linters that format code like this

londons_explore|1 month ago

Great. Now integrate this into every JSON library and tool so I get to see it's output more often

tomtomtom777|1 month ago

I think integration into jq would be both powerful and sufficient.

focom|1 month ago

Great stuff

gritzko|1 month ago

I am a bit saddened by the fact people get obsessed by syntactic innovation or even less than that. Don't we have plenty of urgent problems around us?

teruakohatu|1 month ago

People have a problem and are trying to solve it. We are not all required, nor able, to solve whatever the world’s most urgent problem is today.

In this case they are formatting JSON in an easier to read way. It’s not an alternative to CRDT, it is a totally different issue.