SQLite File Format Viewer

jchw|10 months ago

This really would've come in handy when I was debugging my own SQLite parser a couple weeks ago.

One thing that initially confused me was how exactly the pages worked w.r.t. the first page on disk... I misunderstood the SQLite documentation in different ways, but it's really rather simple: the very first page is just treated as containing the file header in it, and it pushes down the rest of the data, making the page shorter than the other pages. You can see that illustrated clearly if you click into the first page of a database using this tool: the database header comes first, then the page header.

This tool will undoubtedly come in handy for anyone who has a reason to be dealing with SQLite data structures directly for whatever reason, especially since the SQLite documentation is a bit terse at times.

burntcaramel|10 months ago

Yes I’m also working on a SQLite parser, mine is in raw WebAssembly. Is yours open source too? This tool will be so useful. I have basic page reading and parsing of the CREATE TABLE schema: https://github.com/RoyalIcing/SilverOrb/blob/9dacad0ce521b0d...

My plan is to create a miniature .wasm module to read .sqlite files that works in the browser. It will be in the tens of kilobytes rather than the 1 megabyte that the official fantastic sqlite.wasm is. The reduced download means even on 3G you ought to be able to load within a few seconds. You can use SQLite files as your network payloads then, and perhaps even as the working mutable state synced between server and clients.

hinkley|10 months ago

I really want a data format that is effectively binary JSON. What is the subset of all of the features of SQLite that makes either a read-only or an updatable data set that is compact. But better searchability than a streaming parser.

invisal|10 months ago

Glad you like it.

latexr|10 months ago

I especially like that it provides a sample database. Would be nice it were downloadable as a file, so we could explore it with our own tools as well and get a better feeling for how the tool works.

TheRealPomax|10 months ago

https://github.com/invisal/sqlite-internal/tree/main/public, or just download it by copying the URL from your network tab when you load it.

roelschroeven|10 months ago

I tried with a simple database, and it just shows me a blank screen. The console in developer tools shows an exception (both in Firefox and Chromium):

    index-CGfYebmt.js:85 Uncaught RangeError: Offset is outside the bounds of the DataView
        at DataView.prototype.getUint32 (<anonymous>)
        at R0 (index-CGfYebmt.js:85:4683)
        at X0 (index-CGfYebmt.js:85:7966)
        at index-CGfYebmt.js:85:8157
        at Array.map (<anonymous>)
        at Q0 (index-CGfYebmt.js:85:8150)
        at index-CGfYebmt.js:85:8240
        at Object.useMemo (index-CGfYebmt.js:48:46087)
        at My.$.useMemo (index-CGfYebmt.js:17:7276)
        at V0 (index-CGfYebmt.js:85:8228)
    R0 @ index-CGfYebmt.js:85
    X0 @ index-CGfYebmt.js:85
    (anonymous) @ index-CGfYebmt.js:85
    Q0 @ index-CGfYebmt.js:85
    (anonymous) @ index-CGfYebmt.js:85
    useMemo @ index-CGfYebmt.js:48
    My.$.useMemo @ index-CGfYebmt.js:17
    V0 @ index-CGfYebmt.js:85
    tc @ index-CGfYebmt.js:48
    bc @ index-CGfYebmt.js:48
    lo @ index-CGfYebmt.js:48
    No @ index-CGfYebmt.js:48
    wm @ index-CGfYebmt.js:48
    Xc @ index-CGfYebmt.js:48
    Ao @ index-CGfYebmt.js:48
    Qo @ index-CGfYebmt.js:48
    me @ index-CGfYebmt.js:25

invisal|10 months ago

Probably I might made mistake somewhere in my parser. Do you have small reproducible database with no sensitive database and add to the https://github.com/invisal/sqlite-internal/issues

I will try to fix it

NoSalt|10 months ago

Really nice, looks great!

However, I am not a fan of uploading databases to "strange" sites on the internet, so I will probably never use this.

senkora|10 months ago

I wish that we had a browser version of “pledge” that would:

1. Permanently restrict the browser tab from accessing the network.

2. Show an indication of this in the browser UI outside of the content area.

It would be perfect for this kind of app.

jchw|10 months ago

AFAICT it's purely frontend, there is no uploading. Aside from running it yourself, another thing you can do is go into devtools and set the network throttling to "offline". (Note that this is not fool-proof, but it does prevent new connections from being established.)

chungy|10 months ago

It's hard to know whether anything is actually uploaded, but thankfully SQLite databases are plentiful that don't contain sensitive data (and this, also, is why there is a sample database included).

I loaded up my Star Trek database into the program to toy with: https://chiselapp.com/user/chungy/repository/startrek-db/uvl...

olejorgenb|10 months ago

You can probably run it yourself: https://github.com/invisal/sqlite-internal

invisal|10 months ago

It is completely frontend. Everyone can review the code or host it themselve. :)

brulard|10 months ago

There is a github link so you can run it yourself

hamstergene|10 months ago

This is super cool and will really help anyone studying how to build their own storage engines.

A note: the table that describes the data structures (Field # | Description | Value) should also list the data type (uint8 uint16 int32 text etc.). If it's an enum ("Table Leaf"), reference other possible enum values. If its length depends on another field, reference that field No.

graemep|10 months ago

Very cool.

It would be nice if it tied the format in a bit more to the schema and data being shown, but that is a very minor gripe given how nice a tool this is.

NathanFlurry|10 months ago

I've been enjoying watching you build this on X – nice work.

invisal|10 months ago

Thanks so much :)

mubou|10 months ago

This is really cool.

nraynaud|10 months ago

this is a wonderful learning tool, thank you.

T3RMINATED|10 months ago

[deleted]

throw0101d|10 months ago

Meta: always found it interesting that .dev was allowed to be a TLD:

* https://en.wikipedia.org/wiki/.dev

41 comments