Does anyone know of data structures better suited to solve the same goal? (ie fast in memory joins with multiple dimensions, star schema OLAP queries, etc.) ?
This is pretty cool. If you don't mind me asking, did you have a specific application in mind when you decided to write this? Is the idea that your data would be transient (user session?) and that you need to manipulate it, do some things, and then toss it when the work's done?
I am working on adding paid subscription to KidsTrack app (https://www.izhforum.info/forum/izhevsk/tracker_live_map.php...): it is going to be a one-page app with map div and a bunch of dialogs, so user could access and manage all his info. There is a bunch of related tables, that are being displayed on different dialogs in different combinations. For example “Transactions” dialog displays data from linked TRANSACTIONS, ACCOUNT, PAYMENTS, DEVICES and CURRENCIES table; “Devices” dialog displays DEVICES, TRANSACTIONS and CURRENCIES table, “Accounts” table displays data from ACCOUNTS and CURRENCIES tables, etc.
It quickly became a mess of API calls that are querying the same tables but with different joinings. Also, to open each dialog it has to query server and receive the same data many times, which leads to wasting traffic and CPU time. Also, the length of column names from queries becoming bigger, which also wastes traffic. These objects with long prefixed column names need later to be converted on the client back into original objects with unprefixed column names, which also adds complexity to the code.
I tried to optimize it and do some of these joins on client side, but quickly got tired of repeating nested loops and excessive usage of find() and indexOf() functions.
So, I put my main project on hold and decided to write something to do joining more easily on a client.
It has following limitations:
* Objects in array have to have unique "id" field. But, since many server-side frameworks already use "id" field for similar purposes, this should not be a problem.
* Joining can be only done into target table with index on target field. Otherwise it will throw an error.
* Joining currently can only be done by direct value matching, no arbitrary expressions or functions.
* IndexedArray does not store copies of the objects, only references. Therefore it is possible to make indexes inconsistent by updating original object after it has been put into IndexedArray.
To answer your questions - it should be handy for one-page apps or for node.js code, as it may help to reduce asynchronous calls.
[+] [-] polskibus|9 years ago|reply
[+] [-] ngould|9 years ago|reply
https://github.com/square/crossfilter
[+] [-] fiatjaf|9 years ago|reply
[+] [-] partycoder|9 years ago|reply
[+] [-] SmellTheGlove|9 years ago|reply
[+] [-] kidstrack|9 years ago|reply
Thanks for your feedback.
I am working on adding paid subscription to KidsTrack app (https://www.izhforum.info/forum/izhevsk/tracker_live_map.php...): it is going to be a one-page app with map div and a bunch of dialogs, so user could access and manage all his info. There is a bunch of related tables, that are being displayed on different dialogs in different combinations. For example “Transactions” dialog displays data from linked TRANSACTIONS, ACCOUNT, PAYMENTS, DEVICES and CURRENCIES table; “Devices” dialog displays DEVICES, TRANSACTIONS and CURRENCIES table, “Accounts” table displays data from ACCOUNTS and CURRENCIES tables, etc.
It quickly became a mess of API calls that are querying the same tables but with different joinings. Also, to open each dialog it has to query server and receive the same data many times, which leads to wasting traffic and CPU time. Also, the length of column names from queries becoming bigger, which also wastes traffic. These objects with long prefixed column names need later to be converted on the client back into original objects with unprefixed column names, which also adds complexity to the code.
I tried to optimize it and do some of these joins on client side, but quickly got tired of repeating nested loops and excessive usage of find() and indexOf() functions.
So, I put my main project on hold and decided to write something to do joining more easily on a client.
It has following limitations:
* Objects in array have to have unique "id" field. But, since many server-side frameworks already use "id" field for similar purposes, this should not be a problem.
* Joining can be only done into target table with index on target field. Otherwise it will throw an error.
* Joining currently can only be done by direct value matching, no arbitrary expressions or functions.
* IndexedArray does not store copies of the objects, only references. Therefore it is possible to make indexes inconsistent by updating original object after it has been put into IndexedArray.
To answer your questions - it should be handy for one-page apps or for node.js code, as it may help to reduce asynchronous calls.