[charlesdaniels]

I have been working on a project in a similar vein: rq[0]. Mine started out as an attempt to make a jq-like frontend for the Rego[1] language. However, I do find myself using it to simply convert from one format to another, and for pretty printing, quite often.

The interactive mode that qq has is really slick. I didn't torture test it, but it worked pretty smoothly with a quick test.

I see that the XML support is implemented using the same library as rq. This definitely has some problems, because the data model of XML does not map very cleanly onto JSON. In particular, I recall that I had problems using mxj to marshal arrays, and had to use this[2] ugly hack instead. qq seems to have done this a little more cleanly[3]. I may just have to lift this particular trick.

I definitely found CSV output to be challenging. Converting arbitrary JSON style data into something table shaped is pretty tricky. This[4] is my attempt. I have found it works well enough, though it isn't perfect.

I can also see that qq hasn't yet run into the byte order marker problem with CSV inputs yet. Might want to check out spkg/bom[5].

One final breadcrumb I'll drop - I drew a lot of inspiration for rq's input parsers from conftest[6]. That may be a good resource to find more formats and see specific usage examples of them.

Thanks for sharing! It's really interesting to see some of the convergent evolution between rq and qq.

0 - https://git.sr.ht/~charles/rq

1 - https://www.openpolicyagent.org/docs/latest/policy-language/

2 - https://git.sr.ht/~charles/rq/tree/c67df633c0438763956ff8646...

3 - https://github.com/JFryy/qq/blob/2f750f04def47bec9be100b7c89...

4 - https://git.sr.ht/~charles/rq/tree/c67df633c0438763956ff8646...

5 - github.com/spkg/bom

6 - https://github.com/open-policy-agent/conftest

[arandomhuman]

Hi charles,

rq was shared with me yesterday and just wanted to say it's very impressive, I had heard of OPA/Gatekeeper and have looked into Rego before for policy assertions w/ terraform but I was not aware the language was so expressive until I saw rq. Also the amount of codecs rq supports and quality of them is really great.

It is really neat seeing a lot tools solve a similar problem in such unique ways (especially the case with rq) and has been a lot of fun reading your experiences here. Thanks for sharing your experiences and expertise with the de-serializing/serializing content - It is really cathartic to hear you mention the challenges you solved with xml and csv. I really like how you solved for CSV output/input and the conditions on the input data you chose for evaluating it makes a lot of sense and is really comprehensive, it bothered me too since the content would either need to be a matrix or a slice of maps but seeing as jq has string formatting that can convert things to csv and @tsv - I was at a bit of a standstill of how to approach.

Thanks so much for the bread crumbs I look forward to reading this in more detail over the week/weekend :)

[charlesdaniels]

Wow, didn’t realize it had enough legs for people to be hearing about it except via me! Awesome to hear that.

Rego is “for” those authz cases like the ones you mentioned in the sense that it’s definitely designed with those in mind, and I do think it does a good job for those needs. OPA itself is definitely geared for use as a microservice or container sidecar, talking over the wire. That’s kinda hard to use in a shell script though.

Once I learned it I found myself using opa eval for searching and transforming data, eventually so much so that I made a shell script called “rq” that was basically opa eval -I -f pretty… the rest is history.

[iimblack]

This is the beauty of open source. I love that you’re being so collaborative here instead of seeing another tool as competition.

[charlesdaniels]

What's better than 1 nifty tool for querying semistructured data?

2 nifty tools for querying semistructured data!