Why IndexedDB is slow and what to use instead

[jcbrand]

I ran into the IndexedDB slowness problem while adding support for it to Converse.js XMPP messenger[1].

I'm using localForage which lets me persist data to either sessionStorage, localStorage or IndexedDB (depending on user configuration), however writes to IndexedDB were excruciating slow.

I didn't want to give up on it however, because localStorage has a storage limit that power users were constantly coming up against, and IndexedDB is also necessary for progressive web apps.

Eventually I solved the problem by batching IndexedDB writes with a utility function I wrote called mergebounce[2]. It combines Lodash's "debounce" and "merge" functions to combine multiple function calls into a single one, while keeping and merging the data that was passed to the individual function calls.

That way, you can call a function to persist data to IndexedDB many times, but it only executes once (after a small delay). I wrote a blog post about this approach[3]. It's generic enough that anyone else who uses localForage for IndexedDB could use it.

1. https://conversejs.org 2. https://github.com/conversejs/mergebounce 3. https://opkode.com/blog/2021-05-05-mergebounce-indexeddb/

[remram]

> IndexedDB is also necessary for progressive web apps

localStorage doesn't work for progressive web apps?

[alextheparrot]

Thanks for the post, just a heads up the localForage link in your blog post leads to a 404 page.

[samwillis]

This mentions Absurd-SQL[0] which I only discovered recently. It’s seriously clever, worth reading the blog post[1] to see how it works!

If you are building an offline enabled web app for the cost of a 1mb wasm download it’s worth it for full sql support and the speed increase.

I think there is a good chance Absurd-SQL will get folded into SQLite.js.

0: https://github.com/jlongster/absurd-sql

1: https://jlongster.com/future-sql-web

[djbusby]

2: https://sql.js.org/

What kind of sorcery is this!? (WASM)

[devit]

Looks like only transactions are "slow", where "slow" means 2ms, which is perfectly acceptable since it needs to wait for disk commit.

So the problem is that the author is using it wrong, since you only need to wait for transaction commit when you need guaranteed durability, and you only need guaranteed durability when you are communicating success to the user or some other system, which only needs to happen every frame, which is 16ms at 60fps which is plenty of time to wait for a 2ms transaction commit (network communication can also be throttled to the framerate).

So in practice all you need to do is batch writes in a single transaction per frame and there is no problem.

And of course if you don't need durability you can just keep the data in the JavaScript heap (and use inter-tab communication if needed).

[eweitz]

I'm more interested in read speed than write speed. I have about 2 MB of data that I fetch, parse and transform into a nested object for easy look-up by various types of keys. It consists of 6 other objects, and I'd guess it's < 50 MB in total size.

In my brief experiment, it was 12% faster to read from the web Cache API [1], re-parse and re-transform that nested object than to read the fully transformed object using IndexedDB via idb-keyval [2]. That surprised me! I went on to learn that IndexedDB does a structured clone as part of such reads, which I suspect is the main cause of slowness in my use case.

Related commits to reproduce that finding are in [3], specifically [4].

[1] https://developer.mozilla.org/en-US/docs/Web/API/Cache

[2] https://github.com/jakearchibald/idb-keyval

[3] https://github.com/eweitz/ideogram/pull/285

[4] https://github.com/eweitz/ideogram/pull/285/commits/90e374a0...

[jaffathecake]

The conclusion of this article is "do not use IndexedDB as a database", but the reasoning is "if you bulk-insert by creating a transaction for every operation, it's slow".

If you were using IndexedDB as a database, you wouldn't bulk-insert data in this way, as it's the slowest way to do it.

The conclusion should be "use IndexedDB as you would use a database".

[stonemetal12]

It isn't how you would do it if you were bulk loading a bunch of pre determined data, but if it were a bunch of random business transactions in an OLTP scenario that is pretty much how it would happen. The fact that IndexedDB scales poorly in the number of transactions it can handle is an important data point when considering what tech to use.

[pythux]

Unless I missed it, I assume the article is talking about the Chrome implementation of IndexedDB. As far as I know the performance issue with their implementation vs. say, Firefox’ one is a known issue[1].

Other comments mention using batch operations to speed-up IDB but maybe Chrome could also fix their implementation in the first place. It seems, though, that the Chrome team had no intention on improving the write performance of IDB as of 2019[2].

Edit: it seems that Chrome team did improve the performance of IDB since 2019 but it is still not at Firefox level of performance.

[1] https://dev.to/skhmt/why-are-indexeddb-operations-significan... [2] https://bugs.chromium.org/p/chromium/issues/detail?id=102545...

[tehjoker]

I don't understand what the big deal is with grouping a bunch of writes into a transaction? Isn't that how you use a DB?

[setr]

I don’t think you should be expecting a local in-memory db to require batching for reasonable performance; the main reason it’s a problem for normal DB’s is the network — you’re constantly waiting for the roundtrip.

Which you obviously still have for an in-memory DB, but you should expect it to require a lot more picked low-hanging fruit before that becomes your bottleneck.

[sieabah]

That's the first thing I noticed when reading, it's complaining that if you don't batch your writes the performance suffers substantially.

It warrants an audible "duh". More work is always going to run slower.

[hutzlibu]

Hm, but what if I have lots of transactions, going to lots of different object stores? Because every object store requires its own transaction.

(grouping there does not seem to help)

The solution would probably be, to not have lots of object stores, but this would have made my db setup much more complicated.

But FileSystem API seems to finally getting broad support.

[littlestymaar]

> When the window fires the beforeunload event we can assume that the JavaScript process is exited any moment and we have to persist the state. After beforeunload there are several seconds time which are sufficient to store all new changes. This has shown to work quite reliable.

On the contrary, I've found onbeforeunload far from reliable (albeit my use-case was a bit different). You should probably not rely on it for anything important.

[shubik22]

This has been my experience as well. At a prior job, we experimented with deferring reporting/tracking functions until the browser was idle, and then firing the remaining tracking events on unload and found that a non-trivial number of events were just dropped (I believe due to this issue).

[dccoolgai]

One thing I don't see called out as an option: CacheStorage (window.caches). Just store your data as Response. Fast, easy, available in Worker scope...

[tehbeard]

Doesn't solve rich data though as neatly as IndexedDB does.

Because IndexedDB uses structured clone, it can store javascript data as is, which includes storing correctly a number of interesting/useful types, including ArrayBuffers, Files/Blobs, CryptoKey instances (useful to avoid needing export flag on them), the cost of this is the performance hit compared to JSON.parse on a string.

Also the cache is less persistent than IndexedDB when storage pressure increases.

[throwanem]

Not apparently all that durable, though, especially when you hit the storage limit.

[mrweasel]

Not to take away from the points made in the article, but isn't inserting, hundreds or thousands of documents into IndexedDB at once kinda of an edge case?

[jaffathecake]

Not only that, it's doing each insert in a different transaction, which isn't what you'd do for bulk-insert in any database.

[joshuanapoli]

I think that developers are constantly surprised and frustrated when writing data to IndexedDB takes much longer than downloading it. High transaction cost turns out to be one of several performance surprises with Chrome IndexedDB. If a developer discovers these one-by-one, then they will probably need to redesign a few times if they want IndexedDB to support a data-intensive or offline application.

[50]

Alternatively, there's localForage[1][2].

1. https://localforage.github.io/localForage/ 2. https://github.com/localForage/localForage

[typingmonkey]

LocalForage is similar slow as IndexedDB

https://pubkey.github.io/client-side-databases/database-comp...

[ttfkam]

I get the reasoning behind the browser makers' decision, but I really wish Web SQL had become the ubiquitous option. SQLite doesn't make a standard, but pragmatically I think it would have been a better option than how IndexedDB turned out.

A declarative storage syntax for a declarative display engine. Seems a good fit.

[Ginden]

IndexedDB is pinnacle of bad API design and I'm not surprised that people will shoot their own feet with it.

[drenvuk]

I wonder why they don't mention dexie with all of the other options. Like they say, batched operations solves this handily and Dexie has great support for them and it's really darn fast.

[tehbeard]

It's kinda docs / PR / Look how great OUR solution is, not an unbiased piece...

[perttir]

I fixed some of the slowness issues in my application by writing a nice little cache adapter between my application and Dexiejs.