Show HN: Drop-in SQS replacement based on SQLite

[memset]

Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage.

I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have.

But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific.

So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice.

For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.)

It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot.

It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer.

Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future.

In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.

I'd love your feedback!

[davidthewatson]

The whole idea here is superlative.

+1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost.

I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge.

Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong.

BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging.

I'd argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.

[memset]

Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?)

Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.

[raihansaputra]

can you share more about the BTLE sensors talking with Apple Watch? I'm aware that BLE heartrate sensors are detected as 'Health Devices' if I want to connect to it, but what other sensors are you working with?

[pfix]

rq as in Redis Queue?

[hrisen]

Keeping questions from scale & benchmarks aside, this is a cool thing for functional/unit testing module that uses SQS, instead of dumb mocks.

[memset]

Thank you! Yes, you should definitely use this instead of localstack :)

[dav43]

Tangential but I looked at the codebase - because I like to imagine I can code (I can’t) - and for a laymen, I could follow the code.

Makes me think quite positively of go lang and of the dev for designing in such a way. Can understand why teams like it because of easier maintenance

Quite elegant from my uneducated pov.

[silisili]

That's the beauty of Go, generally, and why I immediately became attracted to it. The code is supremely readable because it enforces a style guide, and doesn't allow much magic.

A number of people don't like it for limiting their expression and abilities, which I understand that feeling too. But as a middle aged programmer I realized that readability trumps conciseness and cleverness in the long run.

[memset]

Thank you for the kind words! I'll be sure to work on making it as complicated as I can :)

[x-complexity]

Honestly shows the capabilities of both:

- The dev for making it both fast & simple to understand

- Golang for making the codebase easy to follow

[jerrygenser]

> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.

are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data

[memset]

Truthfully, I don't know yet - I haven't even built a paid/hosted version at all. It is related to my existing business in the sense that it deals with realtime data.

But I started working this as something I wish existed as opposed to having some big VC strategy and pitch deck behind it.

(Also, I appreciate all of your feedback on this a month ago! It was really helpful to encourage me to keep looking into this and also figuring out the "first" things to launch with!)

[Onavo]

What's the point of AGPL though? The enterprises with the budget for self hosting this sort of software usually have requirements and scale beyond what SQLite can offer out of the box.

[4RealFreedom]

The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq

[cies]

We use ElasticMQ to have an SQS compatible service for local development. We use it with docker-compose locally. In our remote envs we use SQS.

So far I had not problems with ElasticMQ.

I'm much intrigued by the small LOC count of SmoothMQ. When I compare it to ElasticMQ it's much smaller (probably by using sqlite's features).

https://github.com/softwaremill/elasticmq

[memset]

I have looked at elasticmq but not played with it myself. You might also be interested in their benchmarks of all of the existing queues out there: https://softwaremill.com/mqperf/

[yimpolo]

This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services.

I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.

[ahachete]

Congratulations!

I also love writing AWS API-compatible services. That's why I did Dyna53 [1] ;P

(I know, unrelated, but hopefully funny)

[1] https://dyna53.io

[selcuka]

It is fun, indeed. Could be a candidate for the best technology abuse of the year.

[bensmoif]

This is an absolutely unhinged project idea. Great work

[memset]

This is really fun!

[philippta]

One quick suggestion on project structure:

Move all the structs from models/ into the root directory.

This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

[memset]

Thanks for the tip - and for even looking at the code! I always struggle to figure out how to organize things.

[pqb]

Actually, the "model/" (note: singular form) directory (package model) would be preferred in the golang world.

[PanMan]

Correct me if I’m wrong but: SQLite sounds like: runs on one server. While that will work most of the time, it won’t work 100% of the time. I don’t know the specifics, but I’m fairly sure if a queue server crashes, SQS will keep working, as stuff is redundant. So while it can work in a best case, this (probably) won’t have the same reliability as SQS has..

[stuaxo]

Good, we need open implementations of all the AWS stuff.

I swear they reimplement stuff we have just so there are more places to bill us.

[memset]

Curious: if you were going to switch, how would you want to run this? Would you want to deploy it to your own EC2 instances, or would you want a hosted solution (just as SQS itself is?)

[paulddraper]

SQS is cheap.

10m requests for $4.

You will need hundreds of millions of requests per month for it to be noticable.

And can an implementation like this even help you at that point?

[john-tells-all]

Why not use LocalStack? It has SQS and a lot of AWS services for testing/development. Well documented, open source.

https://docs.localstack.cloud/overview/

[crazysim]

I think the vibe might be that this is something someone could use for production.

[mihaitodor]

At the very least, I'm guessing this performs better since it's written in Golang.

[andrewstuart]

Because SIMPLE.

single exectuable binary

golang with sqlite so fast

minimal config so no ridiculous hours or days of config and operations

This blitzes other ideas for use cases that do not require distributed queues - and that is likely many use cases.

[koito17]

I may be asking a naive question, but what is the rationale behind disabling foreign key support and using them anyway in the database schema? See https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sq...

The "TODO: check for errors" comment, combined with what seems like disabling foreign key constraint checks, makes me a bit hesitant to try this out.

[memset]

Good question. It was an evolution. Originally I enforced foreign keys, but inserts/updates were unbearably slow. So I updated the connection string to disable them (but, as you point out, I haven't updated the CREATE TABLE statements.)

In practice, I did not find they were necessary - I was only using foreign keys to automatically CASCADE deletes when messages were removed. But instead of relying on sqlite to do that, I do it myself and wrap the delete statements in transactions.

There are many TODOs and error checkings that I will, over time, clean up and remove. I'm glad you've pointed them out - that's the great thing about open source, you at least know what you're getting into and can help shine a light on things to improve!

[chromanoid]

Could you elaborate why you didn't choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS "compatible" broker. For example in the Java ecosystem there is https://smallrye.io/smallrye-reactive-messaging/latest/ that serves a similar purpose.

[westurner]

Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework.

The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-...

Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol.

RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia

SQLite is written in C and has lots of tests because aerospace IIUC.

There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http

celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport...

Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-12...

[andrewstuart]

I wrote sasquatch which is similar. sasquatch is more simple than SQS and queues are virtual (i.e the queue exists because you put a message in it, and is gone when there's no messages in it).

https://github.com/crowdwave/sasquatch

sasquatch is also a message queue, also written in Golang and also based on sqlite.

sasquatch implements behaviour very similar to SQS but does not attempt to be a dropin replacement.

sqsquatch is not a complete project though, nor even really a prototype, just early code. Likely it does not compile.

HOWEVER - sasquatch is MIT license (versus this project which is AGPL) so you are free to do with it as you choose.

sasquatch is a single file of 700 lines so easy to get your head around: https://raw.githubusercontent.com/crowdwave/sasquatch/main/s...

Just remember as I say it's early code, won't even compile yet but functionally should be complete.

[mannyv]

So I assume it does the back-end as well?

I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers?

As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it's just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know.

The reason you'd use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.

[memset]

Everything here is on the backend. The client does very little except make api calls.

It implements many of these features so far (ie, visibility timeouts) and there are some that are still in progress (long polling.) a compatibility table is a good idea.

[systemz]

Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.

[memset]

Can you tell me more about why you needed this (and what you ended up using?)

Totally agree on adding more metrics and information to the UI - but how much of that should be on the dashboard vs exposed as a prometheus metric for someone to use their dashboard tool of choice?

I am not very good at visual design and have chosen the simplest possible tech to build it (static rendering + fomanticUI). I sometimes wonder if the lack of react or tailwind or truly beautiful elements will hold the project back.

[deskr]

Perhaps do a small example application.

  go  get github.com/poundifdef/SmoothMQ/models
  go: github.com/poundifdef/[email protected] requires go >= 1.22.2; switching to go1.22.4
  go: github.com/poundifdef/[email protected] (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/[email protected]: parsing go.mod:
          module declares its path as: q
                  but was required as: github.com/poundifdef/SmoothMQ

[memset]

Good idea! fyi this is not meant to be used as a library. It runs as a standalone server, and then your application connects to it using an existing AWS sdk.

[neuroscisoft]

This is very interesting! The self-hosted aspect is something I'll have to consider for certain purposes.

My lab also developed an SQS-esque system based on the filesystem, so no dependencies whatsoever and no need for any operational system other than the OS. It doesn't support all SQS commands (because we haven't needed them), but it also supports commands that SQS doesn't have (like release all messages to visible status).

https://github.com/seung-lab/python-task-queue

[encoderer]

I think it’s better to say it’s sqs compatible versus an sqs replacement.

[rmbyrro]

I'd have named it MQLite. Congrats on finishing and delivering a project, this is quite a challenge in itself. I think SQLite can be a great alternative for many kinds of projects.