[jpf0]

I cannot share specific projects. J is particularly good for numeric/algorithmic experimentation. It is interactive/REPL-based. Types and boilerplate are minimal. There is no compilation and minimal packages. This is the language I reach for when step 2 is not going to be “install these packages” or deploy to share code. It is most remarkable for data hacking, as it is trivial to manipulate structures and maintain performance. I use python, R, and clojure frequently, but the ability to move quickly in J is without parallel. Weaknesses include namespacing and deployment, although I have seen deployment of substantial codebases both on desktops and servers. Multi-threading and AX512 instructions in _your_ (not package) code, from the REPL are some of what you get with j904.

[fhsm]

It sounds like we do similar sorts of things (based on tool list) but each time I poke at an APL like system I back away clear on learning curve and unclear on value.

At core, my job is arithmetic on 3D arrays of approx 10x1000x100,000,000.

The rub is that for every LOC of written manipulating those structures I’ve got 100 LOCs doing IO (broadly defined) and then 1,000-10,000 doing some form of ETL, QC, normalization (I.e. find and validate the correctness of the magic numbers that go in the cells of the big array).

Do you think J/APL would be of any use to me and if so where in your similar projects’ life cycle does it crop up?

[jpf0]

Possibly so. We do simple IO primarily from parquet or csv data, and I am working on the Apache Arrow/Flight package currently. Our data sets typically fit in RAM but either if you have enough RAM or a file amenable to memory mapping, you shouldn’t have a problem. J has a memory mapping utility. Numerical data is straightforward. We do ETL, QC, and normalization in J. Avoid transposes where you can, but reshapes are trivial and done with metadata, so fast. Overall what you describe sounds like a fun project to try in J.

[BigElephant]

thanks! Can you share any non trivial J codebases on github (not from you or your organization)?

[jpf0]

I suspect the J package Jd is probably the most non-trivial public codebase. I don’t love the coding style (functions are long and scripted) and it doesn’t make use of newer lambda functions (“direct definitions”) which are easier to read. https://github.com/jsoftware/data_jd