Jaq – A jq clone focused on correctness, speed, and simplicity

[j1elo]

> [[]] | implode crashes jq, and this was not fixed at the time of writing despite being known since five years.

Well, taking into account that jq development has been halted for 5 years and only recently revived again, it's no wonder that bug reports have been sitting there for that time, both well known and new ones. I bet they'll get up to speed and slowly but surely clear the backlog that has built up all this time.

[wwader]

Yeap was fixed in 1.7 https://github.com/jqlang/jq/pull/2646

[thekoma]

Why was it halted?

[gigatexal]

It's so awesome when projects shout out other projects that they're similar to or inspired by or not replacements for. I learned about https://github.com/yamafaktory/jql from the readme of this project and it's what I've been looking for for a long time, thank you!

That's not to take away from JAQ by any means I just find the JQ style syntax uber hard to grokk so jql makes more sense for me.

[Valodim]

Very nice in this regard is gron, too. It simply flattens any json into lines of key value format, making it compatible with grep and other simple stream operations.

https://github.com/tomnomnom/gron

[jjeaff]

Nice find. I think I'll try it out. Although I was hoping for a real SQL type experience. I don't understand why no one just copies SQL so I can write a query like "SELECT * FROM $json WHERE x>1".

Everyone seems to want to invent their own new esoteric symbolic query language as if everything they do is a game of code golf. I really wish everyone would move away from this old Unix mentality of extremely concise, yet not-self-evident syntax and do more like the power shell way.

[PhilippGille]

I can also recommend checking https://github.com/tidwall/jj

[OJFord]

I do sympathise with that a bit, but for me at least it does not look like jql is the solution:

    '|={"b""d"=2, "c"}'

this appears to be something like jq's:

    'select(."b"."d" == 2 or ."c" != null)'

which.. is obviously longer, but I think I prefer it, it's clearer?

(actually it would be `.[] | select(...)`, but I'm not sure something like that isn't true of jql too without trying it, I don't know if the example's intended to be complete - and I don't think it affects my verdict)

[klausnrooster]

jql homoiconicity looks rather ... Lispy. Like you could use it on itself, write "Macros", etc.

[antonvs]

> I just find the JQ style syntax uber hard to grokk

You're not alone. ChatGPT (3.5) is terrible at it also, for anything non-trivial.

I'm not sure if that's because of the nature of the jq syntax, but I do wonder.

[Osiris]

I love the idea of jq but i use it infrequently enough that I have to search the manual for how to use their syntax to get what I want.

Sadly 99% of what I do with jq is “| jq .”

[ruuda]

I have the same problem. Then, unrelated, I started building a configuration language, and it turned out it's quite nice for querying json [1]. Here is an example use case that I couldn't solve in jq but I could in RCL: https://fosstodon.org/@ruuda/111120049523534027

[1]: https://docs.ruuda.nl/rcl/rcl_query/

[dse1982]

I had the same problem, keeping me from really exploiting the power of jq. But for this and similar cases I am really glad about copilot being available to help. I just tell it what I need, together with a reduced sample of the source-json, and it generates a correct jq-script for me. For more complex requirements I usually iterate a bit with Copilot because it is easier and more reliable to guide it to the solution gradually than to word everything out correctly in the question in the first go. Also I myself often get new and better ideas during the iterations than I had in the beginning. Probably works the same with ChatGPT and others.

[mmorearty]

Me too; but recently I used ChatGPT to just quickly me the jq syntax I needed: https://chat.openai.com/share/40b68d73-d2dd-412d-867f-9f375e...

[WhereIsTheTruth]

https://github.com/01mf02/jaq/blob/main/Cargo.lock

That's a lot of dependencies..

[mozey]

Yes it is, compared to gojq https://github.com/itchyny/gojq/blob/main/go.mod

[sgt]

How does that usually play out in the Rust ecosystem? Lots of dependencies tell me there's a huge risk of the dependencies becoming inherently incompatible with each other over time, making maintenance a major task. How will this compile in say, 2 years?

[mgaunard]

While jq is a very powerful tool, I've also been using DuckDB a lot lately.

SQL is a much more natural language if the data is somewhat tabular.

[suchar]

Some time ago I tried Retool and it does have "Query JSON with SQL": https://docs.retool.com/queries/guides/sql/query-json (it is somewhat relevant because it was extremely convenient)

It is somewhat similar to Linq in C# although SQL there is more standardised so I like it more. Also, it would be fantastic to have in-language support for querying raw collections with SQL. Even better: to be able to transparently store collections in Sqlite.

It is always sad to see code which takes some data from db/whatever and then does simple processing using loops/stream api. SQL is much higher level and more concise language for these use cases than Java/Kotlin/Python/JavaScript

[CBLT]

I've found the same. I store all raw json output into a sqlite table, create virtual columns from it, then do a shell loop off of a select. Nested loops become unnested, and debugability is leagues better because I have the exact record in the db to examine and replay.

I've noticed what I'm creating are DAGs, and that I'm constantly restarting it from the last-successfully-proccessed record. Is there a `Make`-like tool to represent this? Make doesn't have sql targets, but full-featured dag processors like Airflow are way too heavyweight to glue together shell snippets.

[cryptonector]

Yes. SQL is much better for relational data with a strict schema. Though you'll still never get a way to express recursive queries in SQL w/o a lot of verbosity.

[MrDrMcCoy]

I like textql [0] better for this use case, as it's simpler in my mind.

[0] https://github.com/dinedal/textql

[lopatin]

Regarding correctness, will it display uint64 numbers without truncating them? That's my biggest pet peeve with jq currently.

[necubi]

Unfortunately JSON numbers are 64 bit floats, so if you're standards compliant you have to treat them as such, which gives you 53 bits of precision for integers.

Also hey, been a while ;)

Edit: I stand corrected, the latest spec (rfc8259) only formally specifies the textual format, but not the semantics of numbers.

However, it does have this to say:

> This specification allows implementations to set limits on the range/and precision of numbers accepted. Since software that implements IEEE 754 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision.

In practice, most implementations treat JSON as a subset of Javascript, which implies that numbers are 64-bit floats.

[re]

I believe this has improved in jq 1.7: https://github.com/jqlang/jq/releases/tag/jq-1.7

> Use decimal number literals to preserve precision. Comparison operations respects precision but arithmetic operations might truncate.

[wwader]

jq 1.7 do preserve large integers but will truncate if any operation is done on them. Unfortunetly it currently truncates to a decimal64 which is a bit confusing, this will be fixed in next release where it follow the suggestion from the JSON spec and truncates to binary64 (double) https://github.com/jqlang/jq/pull/2949

[jhatemyjob]

I switched to jless and never looked back. The user interface is miles ahead of everything else

[Snelius]

It's not the same. The jq is not just a viewer. It's a JSON query lang processor.

[jeffbee]

I guess it's cute that there's some terminal line art library in Rust somewhere, but when I tried to invoke jaq it just pooped megabytes of escape codes into my iTerm and eventually iTerm tried to print to the printer. Too clever.

I tried to do `echo *json | rush -- jaq -rf ./this-program.jq {} | datamash ...` and in that context I don't think it's appropriate to try to get artistic with the tty.

The cause of the errors, for whatever it's worth, is that `jaq` lacks `strftime`.

[olemunch]

My first impression is it has fancy error messages but no halt_error/0

  $ ./jaq-v1.2.0-x86_64-unknown-linux-gnu -sf aoc22-13.jq input.txt
  Error: undefined filter
      ╭─[<unknown>:30:18]
      │
   30 │ ╭─▶          "bad input" | halt_error
   31 │ ├─▶        end;
      │ │
      │ ╰───────────────── undefined filter
  ────╯

and (after commenting out halt_error) slower than both jq and gojq

  $ time jq -sf aoc22-13.jq input.txt
  6415
  20056
  
  real    0m0.023s
  user    0m0.010s
  sys     0m0.010s
  $
  $ time gojq -sf aoc22-13.jq input.txt
  6415
  20056
  
  real    0m0.070s
  user    0m0.030s
  sys     0m0.000s
  $
  $ time ./jaq-v1.2.0-x86_64-unknown-linux-gnu -sf aoc22-13.jq input.txt
  6415
  20056
  
  real    0m0.103s
  user    0m0.065s
  sys     0m0.000s

aoc22-13.jq is here https://pastebin.com/raw/YiUjEu2n and input.txt is here https://pastebin.com/raw/X0FSyTNf

[rad_gruchalski]

I started using yq over jq. Any significant differences?

[MrDrMcCoy]

Which yq? I prefer https://github.com/mikefarah/yq to https://github.com/kislyuk/yq.

[a-nikolaev]

jq feels like a much more robust tool than yq. I understand that the task of processing YAML is much harder than JSON, but:

- yq changed its syntax between version 3 and 4 to be more like jq (but not quite the same for some reason)

- yq has no if-then-else https://github.com/mikefarah/yq/issues/95 which is a poor design (or omission) in my opinion

So yq works when you need to process YAML, it can even handle comments quite well. Buy for pure JSON processing jq is a better tool.

[sgt]

The fact that jq takes almost a second to run on a Pi is crazy[0]. And the tool is written in C.

[0] https://github.com/jqlang/jq/issues/1411

[eyegor]

It was fixed in 2019 though? I don't understand your point.

https://github.com/jqlang/jq/issues/1380

[coldtea]

>nan > nan is false, while nan < nan is true.

If this wrong behavior from jq, or some artifact consistent with how the floating point spec is defined, surprising, but faithful to IEEE 754 nonetheless?

[extraduder_ire]

IIRC, any comparison using a nan must fail (return false) according to the IEEE spec.

[throw555chip]

I used Bard after trying unsuccessfully to decipher the wikipedia page and Bard says, according to IEEE 754, nan < nan should return false (0); while nan > nan should return false (0)

[Yanael]

How have you been using jq? It is more adhoc for exploring JSON files during development/data analysis or in programs that run in production?

[wwader]

Quite a lot! i use it to explode both JSON and tex (parse using jq functions). I also use it for exploring ane debug binary formats (https://github.com/wader/fq). Now a days i also use it for some adhoc programming and a calculator.

[brundolf]

Yeah, I've always liked the idea of jq but personally I find it easier to open a REPL in the language I'm most familiar with (which happens to be JS, which does make a difference) and just paste in the JSON and work with it there

It may be more verbose, but I never have to google anything, which makes a bigger difference in my experience

[delecti]

My most common usage is pretty-printing the output of curl, or getting a list of things from endpoint service/A and then calling service/endpoint B/<entry> to do things for each entry in the list.

[Liskni_si]

I use it as a "JSON library for bash". :-)

Not really in "production", but I have a lot of small-ish shell scripts all over the place, mostly in ~/bin, and some in CI (GitHub Actions) as well.

[jbritton]

The 2nd and 3rd examples make no sense to me.

echo '{"a": 1, "b": 2}' | jaq 'add'

3

Construct an array from an object in two ways and show that they are equal:

$ echo '{"a": 1, "b": 2}' | jaq '[.a, .b] == [.[]]'

true

[wwader]

What might be confusing is that iterating an object iterates its values. add is defined something like this: def add: reduce .[] as $n (0; . + $n)

[vjust]

I find jq's syntax (and docs) kind of opaque, but I guess we have no other options. And I don't think this latest incarnation breaks any new ground there. But it'd be better if I just wrote it myself - "be the change ...."

[stevage]

Well, as pointed out in the jaq docs there is jql.

But I just looked at jql and I liked it even less. The pedantry about requiring all keys in selectors to be double quoted is, um, painful for a CLI tool.

[wrsh07]

ChatGPT or the warp chatbot is pretty good at jq syntax

[jasonlhy]

I think the best alternative for JQ is datawave, but it is not open source. https://dataweave.mulesoft.com/

[anonymoushn]

The latest blog post is about open sourcing it from last September. So the process of open sourcing dataweave takes at least 15 months.

[1vuio0pswjnm7]

All else being equal, does the speed of jaq change with the size of the input.

[bilekas]

> nan > nan is false, while nan < nan is true.

You learn something new everyday. Does anyone have any idea why this might be happening? Seems like more than just a bug..

[linux_whisperer]

I use jq on a daily basis. This is new to me thanks for remarking it

[sesm]

Is there a JS library that is similar to JQ but works on JS objects in memory?

[232kkk33kk]

and in powershell you don't need to learn all those syntaxes for different tools for different formats like jq, xmlstarlet, etc. Just convert everything to an object and query the data by using powershell syntax

[icco]

I use `yq` for this stuff and it handles most of this pretty well.

[phplovesong]

Before a clicked on the link i had this gut feeling. It turned out my gut was right. It was written in rust. Go figure..

[loudmax]

I applaud this project's focus on correctness and efficiency, but I'd also really like a version of `jq` that's easy to understand without having to learn a whole new syntax.

`jq` is a really powerful tool and `jaq` promises to be even more powerful. But, as a system administrator, most lot of the time that I'm dealing with json files, something that behaved more like grep would be sufficient.

[ishandotpage]

Have you tried `gron`?

It converts your nested json into a line by line format which plays better with tools like `grep`

From the project's README:

▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page..." | fgrep "commit.author"

json[0].commit.author = {};

json[0].commit.author.date = "2016-07-02T10:51:21Z";

json[0].commit.author.email = "[e-mail]";

json[0].commit.author.name = "Tom Hudson";

https://github.com/tomnomnom/gron

It was suggested to me in HN comments on an article I wrote about `jq`, and I have found myself using it a lot in my day to day workflow

[jrockway]

One of my coworkers really likes Miller: https://github.com/johnkerl/miller

The idea is that you get awk/grep like commands for operating on structured data.

[zellyn]

ChatGPT excels at producing `jq` incantations; I can actually use `jq` now…

[frou_dh]

> I'd also really like a version of `jq` that's easy to understand without having to learn a whole new syntax.

Since JSON is JavaScript Object Notation, then an obvious non-special-snowflake language for such expressions on the CLI is JavaScript: https://fx.wtf/getting-started#json-processing

[gchamonlive]

It is a little early to say, but I have been learning how nushell deals with structured data and it seems like it is very usable for simple cases to produce readable one-liners, and if you need to bring out the big guns the shell is also a full fledged scripting language. Don't know about how efficient it is though.

It needs to justify moving to a completely different shell, but the way you deal with data in general does not restrict itself to manipulating json, but also the output of many commands, so you kinda have one unified piping interface for all these structured data manipulations, which I think is neat.

[msluyter]

Obligatory reference to "gron" ("make JSON greppable"), which I find to be quite useful for many common tasks:

https://github.com/tomnomnom/gron

[INTPenis]

jq, and yq, are tools you spend an hour figuring out and then leave them in a CI pipeline for 3 years.

[hyperthesis]

Maybe like SQL for relational algebra? Codd made two query languages that were "too difficult for mortals to use". (B-trees for performance was a separate issue)

But jq's strength is its syntax - the difficulty is the semantics.

[notatoad]

there's got to be some syntax though. jq does a unique function that isn't defined in any other syntax. i'm with you, the jq syntax is weird and sometimes difficult to understand. but the replacement would just be some different syntax.

these little one-off unique syntaxes that i'm never going to properly learn are one of my favourite uses of chatGPT.

[stickfigure]

Congratulations! We're almost back to the basic functionality we used to have with XSLT.

[lkuty]

You could use an elaborate filter with jq (see https://stackoverflow.com/a/73040814/452614) to transform JSON to XML and then use an XQuery implementation to process the document. It would be quite powerful, especially if the implementation supports XML Schema. I have not tested it.

Or https://github.com/AtomGraph/JSON2XML which is based on https://www.w3.org/TR/xslt-30/#json-to-xml-mapping

It even looks like we could use an XSLT 3 processor with the json-to-xml function (https://www.w3.org/TR/xslt-30/#func-json-to-xml) and then use XQuery or stay with XSLT 3.

Now I have to test it.

[nurettin]

To be fair, xslt is a lot more verbose than `map(.*2)`

[cryptonector]

Yes. jq is essentially an XPath/XSLT for JSON. I'd say that jq is more powerful than XPath/XSLT, but that's neither here nor there since both can evolve to be as powerful as they need to be.

[visarga]

This language must be the spiritual successor of Perl

[TurboHaskal]

I inherited some piece of code that made use of an extremely long and complicated jq script.

I simply gave up understanding the whole thing, and restored the balance in the universe by rewriting it in Perl.

[Yanael]

jq have been in my toolbox since a while it’s a very great tool. But yet another query language to learn, jaq seems identical on that. I think that’s where LLMs can help a lot to make it easier for adoption, I started a project on that note to manipulate the data just with natural language, https://partial.sh

‘cat’ your json file and describe what you want I think should be the way to go

[LargeTomato]

I usually avoid those types of tools. It looks way too fragile and the examples look a bit magical. Do you think it's stable and easy to use?

[sigmonsays]

why not contribute to the existing jq project instead of starting a new one?

We have so many json query tools now it's insane.

[lilyball]

The obvious reason here is jaq makes some changes to semantics, changes which would be rejected by jq.

Another likely reason is that it seems a motivation for jaq is improving the performance of jq. Any low-hanging fruit there in the jq implementation was likely handled a long time ago, so improving this in jq is likely to be hard. Writing a brand new implementation allows for trying out different ways of implementing the same functionality, and using a different language known for its performance helps too.

Using a language like Rust also helps with the goal of ensuring correctness and safety.

[cryptonector]

jq maintainer here. We love that there are multiple implementations of jq now. It does several things: a) it gives users more choices, b) it helps standardize the language (though we've not yet written a formal specification), c) it brings more energy to jq because the maintainers of the other tools have joined jq as maintainers. I also love that these alternative implementations relieve my growing dislike of C.

[sillysaurusx]

Fun, of course. Existing projects are boring almost by definition. And this is volunteer work.

[anonymoushn]

One reason to do this is that often performance improvements involve architectural overhauls that maintainers are unlikely to approve of.

[pizza_pleb]

Somewhat off-topic, but is there a tool which integrates something like this/jq/fx and API requests? I’d like to be able to do some ETL-like operations and join JSON responses declaratively, without having to write a script.

[awayto]

Is there anything out there like "SELECT * FROM "http://..."?

[dilsmatchanov]

Haven't checked yet, but I am sure it's written in Rust

[anitil]

How could you tell?

[explaininjs]

How would you pronounce `jaq` other than `Jaques`[1]? It seems to be the default pronunciation.

[1] https://www.bing.com/videos/riverview/relatedvideo?q=Jacques...

[wewtyflakes]

How does this relate to navigating structured documents? Even if you use XML, presumably you will want to programmatically navigate/query it at some point.

[johnfn]

I think we all understand this to some degree, but working on open source, outside of a few flashy projects, is some of the most thankless work there is. And contributing an immense amount of difficult work (such as perf and correctness improvements across the board) to a repo that you don't own and won't be recognized for is somehow significantly more thankless than that. For whatever reason, people only really care about the creator of a project, and virtually no one else.

For instance, do you know who Junio Hamano is? Oh, he's just a guy who's been maintaining a fairly minor project called Git for the last 15 years. But everyone can connect Linus Torvalds with git, even though he only worked on it consistently for a year or two before leaving it [1].

Also, and I think we all know this too, but working on someone else's codebase kinda sucks. Greenfield is so much more fun. It's a shame, but I'm really not surprised in the slightest.

[1]: https://github.com/git/git/graphs/contributors

[habitue]

As an outsider, getting your code merged into a popular open source project involves a political process of convincing the maintainers that your fix should be addressed, and then convincing them they should merge your code.

Writing a fork involves sitting down at your laptop and coding it out.

[empath-nirvana]

I think in this case it's for the completely reasonable reason that he wanted to write it in Rust and asking jq to rewrite their whole project in rust would be obnoxious.

[sunbum]

This sounds more like an ad for your own project than a constructive comment to be completely honest.

[fyzix]

I think my benchmark[1] would be a great test for this. The jq[2] version takes 50s on my machine.

[1] : https://github.com/jinyus/related_post_gen

[2]: https://github.com/jinyus/related_post_gen/blob/main/jq/rela...