List of command line tools for manipulating CSV, XML, HTML, JSON, INI, etc.

[chrisweekly]

[lnav](https://www.lnav.org) is a terrific little tool, a "mini-ETL" of sorts with an embedded SQLite client and a clean, powerful interface. Its sweet spot is logfiles, but given regex-based custom formats, works great with any semi-structured input. Lnav easily handles a few million rows at a time. IME it pairs really really well with eg mitmproxy/mitmdump for client request logs, as well as webserver logs.

[hoistbypetard]

Thanks for linking that. It's going to make my life easier this week, and I had not heard of it. I was weighing setting up something like Graylog for some troubleshooting and kind of dreading it. lnav looks like a perfect middle-ground between that and my wiki page full of grep commands.

[dancek]

This looks like a great resource. The tools you'd like to have for a specific problem are often quite un-googlable. So you either need complex hacks to get inferior tools to work or you spend an hour googling the tools for a tiny problem.

Of course, it would be even better if you could easily tell which of the dozen JSON query tools is the best choice for the task at hand, or which you should code if you only want to ever use one of them.

In fact I'd love if someone would like to share their set of tried-and-true tools. Personally I mostly go with the POSIX tools, plus jq or gawk on occasion (but I have to read their docs every single time...).

[tannhaeuser]

Nit: awk is a POSIX tool, and has multiple implementations that you've probably used (Debian/Ubuntu comes with mawk, and Mac OS with nawk).

[1]: http://pubs.opengroup.org/onlinepubs/009695399/utilities/awk...

[Lio]

This is great.

One thing I could suggest for the XML list is xmllint. It can be really useful for converting xml to canonical format so you can then use diff to compare it.

E.g. something like diff <(xmllint —c14 first.xml) <(xmllint —c14 second.xml)

I’d love to heat about more command line SOAP tools if anyone can recommend some.

[arundelo]

I'll look into xmllint. I currently use HTML Tidy for this:

  tidy -xml -indent -wrap 0

or

  tidy -xml -indent -wrap 0 -quiet

[IvarTJ]

xmllint also supports XPath queries.

[geocar]

kdb+/q is another really good choice for dsv[1] and json[2]. You can certainly create single-file databases (if you really want to e.g. for exchange), but splayed table[3] is faster so you'd usually do that.

[1]: http://code.kx.com/q/ref/filenumbers/#load-csv

[2]: http://code.kx.com/q/ref/dotj/

[3]: http://code.kx.com/q/cookbook/splayed-tables/

[dancek]

The problem with that might be the licensing costs once you use it commercially (eg. at work). IIRC the license prices aren't public, but you're looking at over $10k in any case.

I personally prefer J to K in the APL family of languages. They also have a relatively cheap database, Jd [1]. Individual licenses are $600. Still a bit too much for my data mangling needs. :)

[1] http://code.jsoftware.com/wiki/Jd/Index

[bjoli]

I took the time to learn recutils a long time ago, and it has been the gift that keeps on giving

Sure, it is not as fast as many other formats, but on the other hand it integrates very well into Emacs an org-mode. I manage a large part of my different collections using a combination of both, and the Emacs integration means it is all less than 2 seconds away.

[jwilk]

If you're curious what's "tabular JSON":

https://johnkerl.org/miller/doc/file-formats.html#Tabular_JS...

[netol]

I don't understand why csvkit is listed in the SQL-based utilities section. csvkit is a suite of multiple command-line tools, including csvcut, csvsort, csvgrep, csvjson, csvstat, csvstack, csvjoin, etc. and multiple converters, so is not only csvsql

[netol]

Awesome. But a list like this could grow indefinitely. I wrote two CSV utilities a few years back; a data generator (https://github.com/pereorga/csvfaker) and a column randomizer (https://github.com/pereorga/csvshuf)

[manaktir]

On macOS there's also textutil, a pre-installed utility for working with text in different formats. Manpage: https://developer.apple.com/legacy/library/documentation/Dar...

[AdamJacobMuller]

I'm very glad to see the 'silly' tools there, cut/join/paste/sort/uniq. While I would never build anything 'important' with them, they're an extremely useful tool to have in your toolkit.

[badsectoracula]

Why silly? I use those (especially sort and uniq) all the time, both in my scripts and in command line.

[joakinen]

If it's important, then you should use those POSIX tools

[stevoski]

Can anyone recommend a command line tool for manipulating Excel files, that runs on macOS?

Edit: I’m looking for a command line tool that allows me to open an Excel file, make a few simple changes, and then save again as an Excel file.

[clarkdave]

If you don’t mind converting the Excel file to CSV, csvkit[0], which is mentioned in the list, has a tool to pipe Excel into CSV for further processing by its sibling tools.

It won’t help if you need to retain anything Excel specific, but I find it very useful to deal with any Excel files that come my way.

[0] https://csvkit.readthedocs.io/en/1.0.3/

[gwn7]

Try visidata.

https://github.com/saulpw/homebrew-vd

[geocar]

xlsx files are also zipfiles that contain xml, so you might get away with just unzipping them, then using some xml query, then zipping it back up.

[chrisweekly]

https://github.com/SheetJS/js-xlsx is a solid choice for that kind of thing.

[jwdunne]

Ruby has a great gem too. I used it to migrate a client's membership data from an excel spreadsheet to a SQL data model.

This sheet was formatted like:

MEMBERS ...rows...

ADMINS ...rows...

EXECUTIVE COMMITTEE ...rows...

You could whip up any command line tool you need with that.

[paulsutter]

Python is one easy / flexible way to wrangle Excel files

[nol13]

csvfix, prob some overlap, but i've found this one invaluable.

http://csvfix.byethost5.com/csvfix15/csvfix.html

[zabzonk]

CSVFix author here - please note a better link is https://neilb.bitbucket.io/csvfix/

[forinti]

Thanks for this link! I frequently have to load CSV files into a database and they are invariably full of errors. People think spitting out CSV is easy, but it's because they don't have to use their product. So every time I write a Perl script and go through various iterations before I find all that's wrong with the file.

[zatkin]

This is missing comparison tables.

[jwilk]

What do you want to be compared?