jq 1.7

[d_t_w]

This is great, JQ is brilliant.

I love JQ so much we implemented a subset of JQ in Clojure so that our users could use it to munge/filter data in our product (JVM and browser based Kafka tooling). One of the most fun coding pieces I've done, though I am a bit odd and I love writing grammars (big shoutout to Instaparse![1]).

I learned through my implementation that JQ is a LISP-2[2] which surprised me as it didn't feel obvious from the grammar.

[1] https://github.com/Engelberg/instaparse

[2] https://github.com/jqlang/jq/wiki/jq-Language-Description#:~....

[chatmasta]

I can't stand jq. I realize this is an unpopular opinion, and our codebase at work has plenty of jq in the bash scripts, some of it even code that I wrote. I begrudgingly use it when it's the best option for me. But something about it rubs me the wrong way - I think it's the unintuitive query syntax and the need to search for every minute step of what I'm trying to do, and the frequency with which that leads to cryptic answers that I can only decipher if I am some sort of jq expert. But I have this instinctive reaction to all DSL languages that embed themselves into strings, like htmx and tailwind (both embedded in attribute string values). I realize some people like it, and it's a well-made piece of software, and I will even admit that sometimes there is no better choice. But I guess I just hate that it's necessary? I guess I could also admit it's the least-bad option, in the sense that it's a vast improvement over various sed/awk/cut monstrosities when it comes to parsing JSON in bash. Certainly once you find the right incantation, it's perfect - it transforms some raw stdin into parsed JSON that you can manipulate into exactly what you need. But for me, it ranks right next to regex in terms of "things I (don't) want to see in my code." I hate that the jq command is always some indecipherable string in the middle of the script. The only real alternative I've ever used is piping to a Python program that I define inline in a heredoc, but that ends up being at least as nasty as the JQ script.

[mingodad]

I also just add a JQ parser/grammar to the online LALR(1)/FLEX grammar editor/tester at https://mingodad.github.io/parsertl-playground/playground/ select "Jq parser (partially working)" from examples then click "Parse" to see a parser tree of the source in "Input source".

Any feedback is welcome !

[dekelpilli]

Just adding that a JQ implementation already exists on the JVM[1], although it's not 100% complete.

[1] https://github.com/eiiches/jackson-jq

[mnutt]

jq is great for letting users munge their data; we do something similar letting users provision an incoming webhook endpoint, send us arbitrary json data, and set up mappings to do useful things with their data, along with regression tests, monitoring, etc. jq makes the majority of cases straightforward (basically json dot notation) and the long tail possible.

[Dionakra]

Is your product available yet? I would be glad to try it out!

[nikolay]

I love jq, but I also use JMESPath (especially with AWS CLI), yq (bundled with tomlq and xq as well), and dasel [2]. I also wish hclq [3] wasn't so dead!

[0]: https://jmespath.org/

[1]: https://kislyuk.github.io/yq/

[2]: https://github.com/TomWright/dasel

[3]: https://hclq.sh/

[js2]

Make JSON greppable!

https://github.com/tomnomnom/gron

I've been using `jq` for years and I'm always able to cobble together what I need, but I have yet to find it intuitive and I'm rarely able to arrive at a solution of any complexity without spending a lot of time reading its documentation. I wish I found it easier to use. :-(

[rmwaite]

I really like the JMESPath interactive tutorial page (https://jmespath.org/tutorial.html). It helped me when I was first learning the syntax and I still go to it if I run into a particularly weird syntax that throws me off.

[commotionfever]

on a similar note I wrote little tool to convert from many formats to many formats

the biggest usecase for me is taking some csv, toml, xml, whatever and converting that to json so I can pipe to jq

https://github.com/sentriz/rsl

[AndyKluger]

A couple more alternatives:

https://github.com/kellyjonbrazil/jello

https://github.com/wwkimball/yamlpath

[wraptile]

Another great alternative is JSONPath[1] which unfortunately not as widely supported and known despite being brilliant!

It's inspired by XPath so it's very familiar instead of a complete new DSL. The killer feature imo is the recursive key lookup so you can write `people..address` and it'll find all "address" keys that descend from "people" anywhere in the JSON. It's by far my favorite parsing language for JSON and I wrote an introduction blog on how to use it in JSON dataset parsing [2] :)

1 - https://github.com/JSONPath-Plus/JSONPath

2 - https://scrapfly.io/blog/parse-json-jsonpath-python/

[tambourine_man]

If you’re like me and use jq infrequently enough that you have to consult the documentation every time, try gron. It’s greppable json

https://github.com/tomnomnom/gron

[preommr]

Wow, that's a pretty cool - so simple but so useful.

I'll also share that I've been using `curl cheat.sh/jq` (cheat.sh in general is a great resource) for years.

Although now I'd probably use something like chatgpt.

[Galanwe]

Damn thats incredibly well thought!

You can even `gron | grep | sed | gron -u`.

Awesome tool thanks for sharing.

[dewey]

I was always struggling with the jq syntax...but I'm consistently impressed by ChatGPT coming up with the right commands from an example JSON.

[ithkuil]

One of the reasons I like/tolerate jq is that it's stable, i.e scripts written for it a few years ago still work the same today.

I have some code around for yq instead that keep breaking because yq keeps improving in non backward compatible ways (I didn't investigate how often yq introduced backwards incompatible changes, but the issue affected me several times in unrelated places, CI scripts or whatnot, that by their nature end up running with different versions of base tooling and update them at various pace)

I was always thus grateful to the great wisdom of the jq maintainers for their understanding of the importance of backwards compatibility.

I hope this announcement doesn't mean that this stability was just an accidental side product of stagnation and that once stagnation is "fixed" it will be done at the expense of stability.

[wwader]

Hey! as one of the maintainers i can safely say that we take backwards compatibility very seriously

[El_RIDO]

Reg. yq/jq incompatibility - unsurprisingly there are many projects that provide yq functionality as wrappers around jq. To list but a few:

- https://kislyuk.github.io/yq/

- https://github.com/kyle-long/yq#yq

- https://github.com/up1/yq-1#yq-command-line-yamlxml-processo...

- https://github.com/onixspot/yq-2#yq-command-line-yamlxmltoml...

- https://github.com/simonrupf/convert2json#convert2json

[ainar-g]

What are you using yq for? Unless you use YAML-only features (e.g. integers, arrays, and objects as object keys), it seems like it would be easier to just pipe-convert your YAML to JSON and process it with jq.

[sumek83]

"Improving in non backwards compatible way" sounds like "deteriorate"

[nikolay]

In addition to my previous comment about jq-like tools, I want to share a couple other interesting tools, which I use alongside jq are jo [0] and jc [1].

[0]: https://github.com/jpmens/jo

[1]: https://github.com/kellyjonbrazil/jc

[goku12]

And jless [1] and gron [2].

This is the first I'm hearing of gron, but adding here for completeness sake. Meanwhile, JSON seems to be becoming a standard for CLI tools. Ideal scenario would be if every CLI tool has a --json flag or something similar, so that jc is not needed anymore.

[1] https://jless.io/

[2] https://github.com/tomnomnom/gron

[mtmail]

Related last month "First release of jq in 5 years" https://news.ycombinator.com/item?id=36951830

Huge fan, I use it all the time.

[cryptonector]

Woohoo!! Finally!

It's really awesome how the community pulled together and helped us recruit new maintainers to revive the project. Special thanks to, well, all involved, but especially @stedolan, @itchyny, and @owenthereal (all GitHub usernames).

[paradox460]

jq and miller[1] are essential parts of my toolbelt, right up there with awk and vim.

[1]: https://github.com/johnkerl/miller

[porridgeraisin]

Thanks for pointing me to miller. Too good.

[black_puppydog]

> Adds new builtin pick(stream) to emit a projection of the input object or array.

> jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | pick(.a, .b.c, .x)'

This is a godsend! Thanks to the contributors! <3

[wwader]

If you don't need to digg deep it's also possible to do:

  $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | {a, e}'  
  {
    "a": 1,
    "e": 4
  }

[nnutter]

I just installed from Git the other day to use this feature. It’s very useful!

[nivekastoreth]

This is a fantastic new feature. I would also love a version of ‘pick’ that works on streaming data, since that doesn’t seem to be possible currently without reassembling the stream first

[lwhsiao]

I'll give a plug for jaq [0], a clone focused on correctness, speed, and simplicity. It only implements a subset of jq, but I've been enjoying it so far.

[0]: https://github.com/01mf02/jaq

[goku12]

Seeing this news today, I decided to give jq another try and ended up discovering jq-mode [1] for emacs. It doesn't just support jq filter file editing, it supports jq in org-mode and something else called 'jq-interactively'. This interactive mode allows you to apply jq interactively on a JSON or YAML (with yq) buffer. The buffer contents become the filtered value when you finish editing the jq filter with a return. This is especially impressive to see in yaml files.

[1] https://github.com/ljos/jq-mode

[G3rn0ti]

Didn’t know this. Thanks for the tip!

Personally, when I test REST APIs, I use „restclient.el“ all the time which also comes with a great JQ integration („jq-set-var“ for example for deriving request variables from responses). For traversing larger responses I use „counsel-jq“ in a customized JSON mode: https://github.com/200ok-ch/counsel-jq

But I’ll give the major mode a try, too.

[pyrolistical]

Love jq. Solved many problems

But at one point I started write long jq modules and while it was pretty straightforward, there are less people familiar with jq.

So I declared jq bankruptcy and rewrote it as a nodejs script. The rest of the team was relived

[cryptonector]

> I started write long jq modules

If you're writing long jq modules, you probably do want a different (faster, better) language.

[ccakes]

There seems to be a few comments from people that dislike jq. If you're that way inclined, gron[0] might be more your thing.

[0] https://github.com/tomnomnom/gron

[jfoutz]

jq is great. I don't know how many times I've had to explain to engineers the invalid numeric literal error means your json is bad. no really, don't trust me, copy it into the ide. it's not jq. your message is malformed.

Strangely, I also have ECMA-404 and RFC8259 open in other tabs. mostly annoyance with the occasional flashes of anger over number formats and duplicate keys.

[dang]

Recent and related:

First release of jq in 5 years - https://news.ycombinator.com/item?id=36951830 - Aug 2023 (27 comments)

[VoodooJuJu]

Works well enough, but it really tripped me up at first with the syntax. Had to constantly look things up while using it.

  jq -r \
  '.apps.http.servers.srv1.routes[0]
  | .match[0].header.[env.AUTH_USER_HEADER][0] = "$username"'

That's an error because you can't select an env var key '[env.AUTH_USER_HEADER]' in the middle of a chain like that, only immediately following a pipe:

  | .match[0].header | .[env.AUTH_USER_HEADER][0] = "$username"'

But then I need to preserve the parent object after the assignment and that pipe throws it out. Thankfully, parentheses fix that:

  | (.match[0].header | .[env.AUTH_USER_HEADER][0]) = "$username"'

After working out the gotchas, it's quite powerful, like regex, but a little clunky, though not nearly as much as regex.

[cryptonector]

> jq -r \ > '.apps.http.servers.srv1.routes[0] > | .match[0].header.[env.AUTH_USER_HEADER][0] = "$username"'

> That's an error because you can't select an env var key '[env.AUTH_USER_HEADER]' in the middle of a chain like that, only immediately following a pipe:

> | .match[0].header | .[env.AUTH_USER_HEADER][0] = "$username"'

You can use `env.AUTH_USER_HEADER` as a key the way you wanted. The issue is that you had to write `... | .match[0].header[env.AUTH_USER_HEADER] ...` -- no `.` between "header" and the index operator!

This complaint is a fairly frequent one, so in fact we did "fix" this in 1.7! You can now write `.a.[0]` and it works.

[rewmie]

> Almost feels like I'm learning regex again.

But isn't that what jq is and always has been? I mean,what led you to believe that an entirely unrelated pattern matching language would work?

[saziba]

I can't stress enough how I love jq. It saved my life countless times as data engineer. Good job guys!

[wanderingmind]

A slight tangential question. I use ripgrep-all along with fzf to interactively search through files from cli. Is it possible to integrate jq (or some equivalent) into this search ecosystem to search through json files?

[lordgrenville]

Yes https://gist.github.com/reegnz/b9e40993d410b75c2d866441add2c...

[faizshah]

I wish there was a faster version of jq, I was only able to get a few mb/s throughput out of jq vs few hundred mb/s throughput out of ripgrep.

I often use ripgrep to setup quick bash pipelines for rapid data analysis, would love to be able to use jq for that purpose. These days I am setting up scripts with simdjson but the cost of writing a script vs quickly setting up jq or ripgrep in a bash pipeline are orders of magnitude different.

[cryptonector]

Did you try gojq or jaq?

[gobdovan]

Thought this is an array programming language considering the name's made out of J and Q. Nice util

[fiddlerwoaroof]

The way it works is sort of reminiscent of array programming.