Perspective: Streaming pivot visualization via WebAssembly

[dman]

I wrote the C++ part of this codebase, would be happy to answer any questions about it here.

[stupidcar]

Did you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?

[zimablue]

I see it's possible to embed it in jupyterlab, it seems to do a lot:

* grid format * graph format * actual pivoting of data

Is that right?

It's using webassembly so in effect to do the pivot you must have created functions that exist in pandas?

Has anyone ported pandas to webassembly?

Is the grid editable? What are you using to create your output, html or canvas or something?

I realise I could look this stuff up but since you asked..

[jarpineh]

Briefly in the animation I see there's hierachical (or multi-dimensional) data used. Could you tell me about what sort of data (statistics, monitoring) this supports?

Also, what methods are available for getting data streamed [into Perspective]? I'm skimming through the docs, but am not quite getting it...

[edited for clarity]

[fitzoh]

Just stalked you on linkedin with a similar question, but I was wondering if you were involved with the actual process of getting the library open sourced, and how that works at a large security focused company like JPMorgan Chase. (former Chase employee)

[nickthemagicman]

Thanks for taking questions. For a back end developer that's watching the front end landscape change from afar..

Do you feel like web assembly could potentially replace JS in the future with, say, my favorite back end language of choice?

[polskibus]

Does it support:

virtualization (showing only a view of full dataset that is updated from server on scroll)

Pivoting (multiple levels/hierarchies on rows and on columns)

Apart from features what part did you consider was the biggest hurdle? What browsers did you target?

[icefox]

What was the best/worst part about getting it to work in WebAssembly?

[pavlov]

Is Emscripten the only game in town for WebAssembly C/C++ toolchains, or are there other contenders?

[LukeB42]

I see it's possible to convert to Numpy arrays. Is `perspective` also a Python library?

[polskibus]

Can you say how old is this codebase and what was its original business case?

[polskibus]

What's on the roadmap?

[timkpaine]

[]

[mi100hael]

Really cool to see non-software companies putting cool projects like this out as open source

[bpicolo]

This is great. Finance people especially love this sort of thing - they do want to browse through entire datasets vs seeing higher level metrics in a lot of cases.

[ChicagoDave]

I love seeing new UX on Web Assembly. Someone needs to port or create a standard layout engine like WPF to build on this. If the tools were there, C# + VS Code + solid layout engine...I'd dump JS + HTML + CSS in a "flash".

[pjmlp]

There are some attempts on that direction.

https://www.hanselman.com/blog/NETAndWebAssemblyIsThisTheFut...

[h2g3yhogw8]

my question is how do game engines do styling ?

[elwell]

"J.P. Morgan open sources..." -- that's so cool

[polskibus]

Amazing, despite the fact that some heavy-lifting is done by highcharts and hypergriid.

What's missing for this to effectively kill all typical BI tools like PowerBI, Tableau?

[infinite8s]

Expensive sales teams. Also neither of those handles streaming data.

[tgtweak]

The backend.

[andrewmcwatters]

@dman Cool, but I'm curious why WebAssembly was chosen for this project. What was the value of using it here versus just using the Web API?

[dman]

The following features made wasm compelling here

a. Filter / Pivot / Aggregate moving clientside means a lot of load off the servers.

b. Performance in wasm is surprisingly close to same C++ code running natively on the desktop.

c. Ability to reuse existing C++ codebase and use it to build both native and web apps.

[lucozade]

The C++ probably wasn't written for a browser. More likely it was written for a native frontend and has been ported to the browser via WebAssembly.

[infinite8s]

What do you mean when you say the Web API?

[bcherny]

Presumably, the point of using WASM and service workers is for performance, but I see no benchmarks. I also have trouble imagining that this actually improves performance, unless you’re doing all of your compute in the browser and it’s CPU bound (this seems like bad design).

What is the performance of doing it this way vs. without WASM and workers?

Until we see numbers, this smells like a recruiting play.

[adamsea]

First, let me say this looks awesome! I’m curious if your considered React.js for the presentation layer (i.e. the js/html5 parts) - on the surface it seems like it would be a good fit - and if so, what you saw as the benefits or drawbacks of it vs the approach you went with.

[texodus]

We chose to go with a Web Components based interface for compatibility across frameworks, but alot of where we go in the future will be determined by the expertise of the developers we hire to work on it, and the community if there is interest there.

[dman]

Will defer to texodus on that - he is the front end expert here.

[hackcasual]

For those interested, keep in mind the only way to use WebAssembly with CSP in Chrome is by turning on 'unsafe-eval'. FF/Edge/Safari all at least support compilation from URLs with more locked down policies

[texodus]

True - worth pointing out that this library gracefully falls back to ASM.js if your browser does not support WebAssembly, though.

[devscreen]

This is seriously cool. Unfortunately I have no use-case for it in the product I currently work on, but still fascinating to flick through the source code and play with it.

[mjansen]

really cool. would love to see a Rust version.

[flight21]

What are you using for UI, any framework (Polymer, D3) or just plain WebComponent?

[texodus]

Plain web components. The UI is quite light in terms of functionality so far.

[polskibus]

Does the C++ code use vectorization? If not, is it planned?

[baybal2]

So, what there is about visualisation?

I read your docs, and found nothing about how to actually render stuff to the page.

[wsgeek]

I followed the directions to the letter, and everything installed correctly on MacOS. When I tried the same on Linux, though, I got tons of the following errors on the build step (keep in mind I do have boost development libraries installed in /usr/include):

(Sorry about the formatting.... I tried <code>). Anyway, TL/DR it cannot find boost.

$ ./node_modules/.bin/lerna run start --stream lerna info version 2.8.0 @jpmorganchase/perspective: > @jpmorganchase/[email protected] start /home/dj/usr/src/perspective-clone/packages/perspective @jpmorganchase/perspective: > npm run compile && (npm run perspective & npm run compile_test & npm run compile_node & wait) @jpmorganchase/perspective: > @jpmorganchase/[email protected] compile /home/dj/usr/src/perspective-clone/packages/perspective @jpmorganchase/perspective: > mkdir -p build build/wasm_async build/wasm_sync build/asmjs && (cd build/; emcmake cmake ../; emmake make -j8; cd ..) @jpmorganchase/perspective: -- Configuring done @jpmorganchase/perspective: -- Generating done @jpmorganchase/perspective: -- Build files have been written to: /home/dj/usr/src/perspective-clone/packages/perspective/build @jpmorganchase/perspective: Scanning dependencies of target psp @jpmorganchase/perspective: [ 1%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_win.cpp.o @jpmorganchase/perspective: [ 2%] Building CXX object CMakeFiles/psp.dir/src/cpp/arg_sort.cpp.o @jpmorganchase/perspective: [ 4%] Building CXX object CMakeFiles/psp.dir/src/cpp/calc_agg_dtype.cpp.o @jpmorganchase/perspective: [ 5%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggspec.cpp.o @jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggregate.cpp.o @jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/base.cpp.o @jpmorganchase/perspective: [ 9%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_linux.cpp.o @jpmorganchase/perspective: [ 10%] Building CXX object CMakeFiles/psp.dir/src/cpp/build_filter.cpp.o @jpmorganchase/perspective: [ 12%] Building CXX object CMakeFiles/psp.dir/src/cpp/column.cpp.o @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/cpp/calc_agg_dtype.cpp:11: @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/calc_agg_dtype.h:12: @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/schema.h:13: @jpmorganchase/perspective: /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/base.h:29:10: fatal error: 'boost/unordered_map.hpp' file not found @jpmorganchase/perspective: #include <boost/unordered_map.hpp>\n ^~~~~~~~~~~~~~~~~~~~~~~~~ </code>

[texodus]

If anyone finds this project exciting and is interested in learning more about working on Open Source at J.P.Morgan, feel free to send me a message - we are always looking to hire experienced, passionate talent!

[dman]

+1 - Even though I do not work there anymore, I can vouch for the fact that the team has amazing engineers and works on fundamental CS problems.

[feel_the_need]

I would like to work on cakeshop-- how do I get on touch with you?

[mortdeus]

Hey man, two thumbs up on this. Wall street as a contributor to open source. Who do I thank? Trump?

[ChicagoDave]

Poor naming choice. Google Perspective has been around for a year or two I believe.

[timkpaine]

in that case, its actually a good naming choice, as this existed (in a closed source form) for much longer than that.