Whee! My first Show HN. I've only been here, like, seven years and all.
But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a while. I've found Kue and Bull really frustrating to use (no shade intended; they feel like they made a lot more sense in a pre-ES6 universe?) and `node-resque`, while scratching a little more of that itch, just made me miss the comfortable guardrails of Ruby offline job queues. So I made one.
As my day job is devops and devops accessories, making it easy to run and administer TaskBotJS was a big priority for me and I've tried to get the best of both worlds: mostly turn-key to run in simpler cases but less prescriptive and more pluggable at scale. TaskBotJS is pretty extensible and supports job middleware and event listeners; I have a Sentry plugin coming in the near future as an example (and also 'cause I like Sentry).
I also like to write documentation[0]. While it's not yet complete, what's there is pretty exhaustive and I try to explain my thinking as well as just "what it does"; capturing the whys is as important as the whats, to me, to better foster conversation around the project and to make for more constructive issues when systemic, design-focused concerns are raised.
It's running in at least two production environments (only one of which is mine; I dogfood but I wouldn't have released it as a "you should check this out" version if somebody else wasn't checking me) and is being rolled out to at least one more in the near-ish future.
Thanks for taking a peek, and I'd love to hear any actionable feedback that comes to mind.
Congrats on shipping! As someone who's written similar software[1] (for Python), I can appreciate the hard work that went into this.
One piece of feedback regarding the docs: the documentation section in your README has two links, one to the wiki and the other to the website. I had initially scanned the section and followed the link to the website (I don't automatically associate "wiki" with "documentation") and was then unable to find your documentation for a good couple of minutes. You should make the wiki link more prominent.
Nice! We need more work in this space. I'm one of the node-resque authors. I'd love to hear more about what you felt was missing from the ruby version... clearly we copied a lot :D
I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the problems you found with Bull? Writing a queue that atomically processes items is non-trivial, so when I wanted a feature in Bull I just improved it, the devs are very receptive.
Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS.
EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.
I'm planning to use a similar model OSS / commercial as well. Mike and Sidekiq is my inspiration for this. Good luck going further with the project and congrats for the launch!
After a few minutes I gave up on evaluating this tool. It looks interesting but there's no "hello world" or even a simple use-case explanation in the first few pages of docs. Couple that with the no-warning switching between JS & TS in the code examples, and it is too tedious to try to figure out.
I have no objections to TS, but if this is a "JS" library for consumption in JS, please write your example jobs in JS please! Don't make me try to translate TS to JS in my head–I don't care how easy you might consider it to be, I don't want to do it.
I like the idea and will revisit when docs improve. I recommend user testing your docs: put an unfamiliar JS dev in front of your README, set a timer for ~7 minutes, and when the timer dings ask if they think they'll use your library. If they're still confused after that time, your intro docs are not clear enough. That may seem like a small amount of time but there are 10,000 JS libs out there, 7 minutes is far more time than I can devote to developing an initial impression of each one.
Thanks for checking it out and for the feedback! Agreed on intro docs--that's a big reason it's "0.9.0" instead of "1.0". ;) "1.0", to me, is "the API is stable" (though the current API should be) and "it's documented to the best of my ability for novice users." The current userbase is a little more willing to dig, as it's mostly friends and colleagues, and that'll change as it moves towards 1.0.
I'll look into a JS-based example job; I have never found the translation particularly onerous, but horses for courses, I guess. I'm not switching between JS and TS in the documentation, though, save for the definition of a configuration file--it's pretty silly, IMO, to transpile a config file, especially when you get 99% of the way there through something like VSCode's IntelliSense. I didn't make clear enough the distinction, apparently, but I didn't expect it to be a stumbling block. Lessons learned.
I like Mike a lot and I think Sidekiq is awesome--not having it in Node was largely why I wrote TaskBotJS. And I evaluated Faktory before writing it, because hey, this has been a lot of work, buy instead of build when you can!
As it stands, and things could change in the future, but I think a dedicated ecosystem-specific system is a better bet for the 90th, 95th, and probably 98th percentile of products and projects. I don't feel that centralizing background processes makes a lot of sense in a service-oriented architecture and that takes away a lot of Faktory's benefits of agnosticism. The embedded use of RocksDB also makes me itchy; Redis--while by no means perfect--is, IMO, more understood as a backing store, and it's separate from the runner itself. (TaskBotJS is also likely to eventually grow a Postgres backend. I've also got a good chunk of one written using SQS/DynamoDB, but that's further out for sure.)
All that said: Mike is crazy smart and I wouldn't begrudge anybody who wanted to use Faktory. When I write Ruby, I will continue to use Sidekiq. ;)
I was very enthusiastic about Faktory when it was first announced, and still am, but when we tried to use it for a project at work it wasn't really suited to PaaS based infrastructure because of the dependency on a local database. There are a few Faktory as a Service providers in development at the moment, but none of them are in production in the EU yet.
Very nice job! I'll definitely be trying this out in a personal project.
How's your experience been with TypeScript and oao? I've got a very large monorepo (using yarn workspaces) of TypeScript modules and it's been sort of a PITA. For instance, a typical yarn install will unnecessarily duplicate some dependencies across packages causing tsc to complain about duplicate identifiers. Running yarn --check-files fixes this, but it's still annoying. Also, following symbols with yarn workspaces is sometimes annoying since linked packages will use "main" and "types" fields in the package.json, thus following a symbol takes you to the generated type definition. I have a generated tsconfig.json that sets paths to their appropriate package paths to fix that. Again annoying.
Thanks for the kind words. =) And, good question! I...probably use `oao` wrong, to be honest, or at least not as fully as it could be; my main use of it is to power a workspace-wide `yarn watch`. I have had a lot of problems trying to use its more advanced feature set and, tbh, if you look in `packaging` I am certain there are functions in there that `oao` totally does, especially around version bumping.
100% agreed with regards to yarn workspaces and linked packages. Symbol navigation is Not Great, and I also have to be careful using VSCode's quick import feature because sometimes it will just go wild and import "../../../../dist/client/foo" or whatever instead of using the package import. Once done correctly in a given file it's fine, though.
But converting TaskBotJS to TypeScript was really easy, and it catches a lot of issues, so I think the future's bright on the tooling front. I found TypeScript unusable, like, six months ago. I wrote a couple React projects and a fairly large React Native app[0] in ES6 (which I Do Not Recommend Doing...ow) because getting TypeScript working was just way too much work. Still kind of is with regards to React, to be honest; the web panel for TaskBotJS is just a create-react-app app because of it. But--progress.
Have you tried using Lerna for monorepo management? I see some projects using it, I haven't yet (no monorepos) but it claims to solve some of those problems. https://github.com/lerna/lerna
I don't get it. I'm primarily a Python developer, so this looks like Celery to me. I've never understood why I'd want to tie myself to a particular library, rather than encode messages to an agreed encoding, then write workers in the technology most appropriate for the job?
Does it run the tasks in the same or a different process? Why is redis necessary? Can it pick up where it left off if the server restarts? Why so much boilerplate (still) for simple use cases?
I love seeing projects with a docker-compose.yml, so I can check it out quickly. However, yours only spins up redis - wouldn't it make sense to spin up TaskBotJS in a separate container? Yes, I know the proper answer is "PRs accepted", but just curious about the rationale.
Hey, awesome feedback! The root docker-compose file is for development; it stands up Redis and the example config files, unless overridden, look for a Redis on the port specified in that file.
TaskBotJS does have an integration environment, though. Take a look in `packaging`. When I do a release I use the functionality in there to spin up `@taskbotjs/example` and the associated functionality. On that it does some very basic smoke tests to make sure nothing's obviously broken, but it spins up a set of producers, consumers, and the webapi/web panel.
If you run `NO_CLEANUP=1 ./packaging/build-and-integrate.bash`, it'll stand up the integration environment and you can play around with an active producer/consumer/web panel system.
I should definitely break that out and make it easier to play with right off the bat. Thanks for the idea.
Trying to solve something like this always puzzled me: how to make job publishing transactional? There are no simple way to do transaction across redis and, say, postgres. If you wish to make thing reliable, you will have to put job to an sql first and then pull (or subscribe) from database to a worker queue.
Yes, this is a major problem most people simply ignore and deal with in a one-off manner when it fails. The only way to do this safely that is practical is to create a jobs table which is written to in a transaction with the other writes. Another process reads the table and writes the job data into the queueing system.
For example, the process starts up, reads the highest ID it has written from Redis, then starts reading the table for jobs with IDs higher than that. The job itself is written to Redis in a transaction along with the ID from the database. Ignoring the fact that Redis can lose data during failover, you’re pretty safe and duplicates will be minimal. You can add some idempotency on top if desired.
Attempting to write to both a queue system and a relational database at the same time will fail at some point and you will be left to pick up the pieces of inconsistent state.
If the write to the queue is to send an email, maybe you don’t care care. If it is to calculate a very important value that happens to be expensive to compute and needs to be done outside the request... good luck.
Not criticizing, but just curious what was the reasoning behind picking Redis rather than something like sqlite, which would keep the application a little bit more self-contained so to speak?
Hey, great question! SQLite is great for locally-stored databases with relatively light relational requirements, but I chose Redis as a first go-around in part because it's kind of "the standard" for this sort of job queue (Kue, Bull, Resque, Sidekiq, etc.) and also because it was the simplest network-friendly datastore that included the primitives I wanted to use and because its simple nature--when not in clustered mode, which is currently in "you're on your own" territory for TaskBotJS, but I expect that to improve--makes reasoning through operations simpler for me.
As mentioned elsewhere, I've started work on a PostgreSQL-backed backend (which is actually easier than it sounds to get right, modern Postgres includes functionality to work with advisory-locked rows) and have done some noodling on a fully AWS-backed DynamoDB/SQS backend solution, too. The FoundationDB open-source announcement has me doing some reading, too, but I don't want to say much there just because I don't know enough about it yet to know whether it makes sense.
I am in no way affiliated with this project, but my assumption is that since this uses workers(separate servers) to process the background jobs - they could be on completely separate machines. If you use redis for job storage, they can all communicate without having to talk back to the master server.
Not a fan of this sort of "open core" offering, and it's heavy on the marketing. Recurring jobs are an "Enterprise" feature? Hard pass. That's a core feature of Celery and readily available for Resqueue, latter of which seems to be your project inspiration? Pro/Enterprise should be tool offerings that depend on your hosting and support. Arbitrary feature gating isn't encouraging for a project like this.
Celery requires the system administrator to ensure that there's only a single scheduler in the cluster at a time; the documentation specifically calls that out. This additional administrative load is not a requirement of TaskBotJS. Because of that recurring jobs requires underlying, distributed-environment infrastructural features I'm not prepared to open source right now. Past that? TaskBotJS is LGPL. If you'd like to write and maintain your own cron-type scheduler, I would be happy to link to it as a plugin, and you can own the difficulty of maintaining it when users come yelling. ;)
That last bit is important, IMO. The best way to keep software maintained is to create a financial incentive for its maintenance. The pro/enterprise feature set is borne out of actual use, and to signal the very strong intent to maintain this as unsurprising software that will be there and be supported for the long haul.
See https://opensource.guide/starting-a-project/ for GitHub's latest advice on starting an open source project, which includes adding a Code of Conduct. I think single-person projects just starting out can get away with just a LICENSE and README, but it's best to have CONTRIBUTING and CODE_OF_CONDUCT if you expect any contributions.
Hopefully it's rare that it's needed, but it's really easy to add and a nice thing to have proactively instead of reactively, and plenty of people feel more welcome when things like that are explicit.
The Rust programming language had a Code of Conduct from the get-go. In general, having one gives a reasonable idea of the type of community you'd like to build around the project, and avoids the mess of trying to add one once you have a pile of vehemently anti-CoC community members.
[+] [-] eropple|7 years ago|reply
But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a while. I've found Kue and Bull really frustrating to use (no shade intended; they feel like they made a lot more sense in a pre-ES6 universe?) and `node-resque`, while scratching a little more of that itch, just made me miss the comfortable guardrails of Ruby offline job queues. So I made one.
As my day job is devops and devops accessories, making it easy to run and administer TaskBotJS was a big priority for me and I've tried to get the best of both worlds: mostly turn-key to run in simpler cases but less prescriptive and more pluggable at scale. TaskBotJS is pretty extensible and supports job middleware and event listeners; I have a Sentry plugin coming in the near future as an example (and also 'cause I like Sentry).
I also like to write documentation[0]. While it's not yet complete, what's there is pretty exhaustive and I try to explain my thinking as well as just "what it does"; capturing the whys is as important as the whats, to me, to better foster conversation around the project and to make for more constructive issues when systemic, design-focused concerns are raised.
It's running in at least two production environments (only one of which is mine; I dogfood but I wouldn't have released it as a "you should check this out" version if somebody else wasn't checking me) and is being rolled out to at least one more in the near-ish future.
Thanks for taking a peek, and I'd love to hear any actionable feedback that comes to mind.
[0] - https://github.com/eropple/taskbotjs/wiki
[+] [-] Bogdanp|7 years ago|reply
One piece of feedback regarding the docs: the documentation section in your README has two links, one to the wiki and the other to the website. I had initially scanned the section and followed the link to the website (I don't automatically associate "wiki" with "documentation") and was then unable to find your documentation for a good couple of minutes. You should make the wiki link more prominent.
[1]: https://dramatiq.io
[+] [-] evantahler|7 years ago|reply
[+] [-] k_vi|7 years ago|reply
[+] [-] nodesocket|7 years ago|reply
For example:
[+] [-] doublerebel|7 years ago|reply
Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS.
EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.
[+] [-] ra00l|7 years ago|reply
[+] [-] disiplus|7 years ago|reply
[+] [-] sequoia|7 years ago|reply
I have no objections to TS, but if this is a "JS" library for consumption in JS, please write your example jobs in JS please! Don't make me try to translate TS to JS in my head–I don't care how easy you might consider it to be, I don't want to do it.
I like the idea and will revisit when docs improve. I recommend user testing your docs: put an unfamiliar JS dev in front of your README, set a timer for ~7 minutes, and when the timer dings ask if they think they'll use your library. If they're still confused after that time, your intro docs are not clear enough. That may seem like a small amount of time but there are 10,000 JS libs out there, 7 minutes is far more time than I can devote to developing an initial impression of each one.
[+] [-] eropple|7 years ago|reply
I'll look into a JS-based example job; I have never found the translation particularly onerous, but horses for courses, I guess. I'm not switching between JS and TS in the documentation, though, save for the definition of a configuration file--it's pretty silly, IMO, to transpile a config file, especially when you get 99% of the way there through something like VSCode's IntelliSense. I didn't make clear enough the distinction, apparently, but I didn't expect it to be a stumbling block. Lessons learned.
[+] [-] deedubaya|7 years ago|reply
https://github.com/contribsys/faktory
[+] [-] eropple|7 years ago|reply
As it stands, and things could change in the future, but I think a dedicated ecosystem-specific system is a better bet for the 90th, 95th, and probably 98th percentile of products and projects. I don't feel that centralizing background processes makes a lot of sense in a service-oriented architecture and that takes away a lot of Faktory's benefits of agnosticism. The embedded use of RocksDB also makes me itchy; Redis--while by no means perfect--is, IMO, more understood as a backing store, and it's separate from the runner itself. (TaskBotJS is also likely to eventually grow a Postgres backend. I've also got a good chunk of one written using SQS/DynamoDB, but that's further out for sure.)
All that said: Mike is crazy smart and I wouldn't begrudge anybody who wanted to use Faktory. When I write Ruby, I will continue to use Sidekiq. ;)
[+] [-] jon-wood|7 years ago|reply
[+] [-] JohnDotAwesome|7 years ago|reply
How's your experience been with TypeScript and oao? I've got a very large monorepo (using yarn workspaces) of TypeScript modules and it's been sort of a PITA. For instance, a typical yarn install will unnecessarily duplicate some dependencies across packages causing tsc to complain about duplicate identifiers. Running yarn --check-files fixes this, but it's still annoying. Also, following symbols with yarn workspaces is sometimes annoying since linked packages will use "main" and "types" fields in the package.json, thus following a symbol takes you to the generated type definition. I have a generated tsconfig.json that sets paths to their appropriate package paths to fix that. Again annoying.
[+] [-] eropple|7 years ago|reply
100% agreed with regards to yarn workspaces and linked packages. Symbol navigation is Not Great, and I also have to be careful using VSCode's quick import feature because sometimes it will just go wild and import "../../../../dist/client/foo" or whatever instead of using the package import. Once done correctly in a given file it's fine, though.
But converting TaskBotJS to TypeScript was really easy, and it catches a lot of issues, so I think the future's bright on the tooling front. I found TypeScript unusable, like, six months ago. I wrote a couple React projects and a fairly large React Native app[0] in ES6 (which I Do Not Recommend Doing...ow) because getting TypeScript working was just way too much work. Still kind of is with regards to React, to be honest; the web panel for TaskBotJS is just a create-react-app app because of it. But--progress.
[0]: https://bit.ly/buymyapp
[+] [-] thegoleffect|7 years ago|reply
[+] [-] alexchamberlain|7 years ago|reply
[+] [-] ilaksh|7 years ago|reply
[+] [-] bdcravens|7 years ago|reply
[+] [-] eropple|7 years ago|reply
TaskBotJS does have an integration environment, though. Take a look in `packaging`. When I do a release I use the functionality in there to spin up `@taskbotjs/example` and the associated functionality. On that it does some very basic smoke tests to make sure nothing's obviously broken, but it spins up a set of producers, consumers, and the webapi/web panel.
If you run `NO_CLEANUP=1 ./packaging/build-and-integrate.bash`, it'll stand up the integration environment and you can play around with an active producer/consumer/web panel system.
I should definitely break that out and make it easier to play with right off the bat. Thanks for the idea.
[+] [-] moltar|7 years ago|reply
[+] [-] ex3ndr|7 years ago|reply
Am i right or there are another option?
[+] [-] ryanworl|7 years ago|reply
For example, the process starts up, reads the highest ID it has written from Redis, then starts reading the table for jobs with IDs higher than that. The job itself is written to Redis in a transaction along with the ID from the database. Ignoring the fact that Redis can lose data during failover, you’re pretty safe and duplicates will be minimal. You can add some idempotency on top if desired.
Attempting to write to both a queue system and a relational database at the same time will fail at some point and you will be left to pick up the pieces of inconsistent state.
If the write to the queue is to send an email, maybe you don’t care care. If it is to calculate a very important value that happens to be expensive to compute and needs to be done outside the request... good luck.
[+] [-] andreygrehov|7 years ago|reply
[+] [-] eropple|7 years ago|reply
As mentioned elsewhere, I've started work on a PostgreSQL-backed backend (which is actually easier than it sounds to get right, modern Postgres includes functionality to work with advisory-locked rows) and have done some noodling on a fully AWS-backed DynamoDB/SQS backend solution, too. The FoundationDB open-source announcement has me doing some reading, too, but I don't want to say much there just because I don't know enough about it yet to know whether it makes sense.
[+] [-] aclave1|7 years ago|reply
[+] [-] anothergoogler|7 years ago|reply
[+] [-] eropple|7 years ago|reply
That last bit is important, IMO. The best way to keep software maintained is to create a financial incentive for its maintenance. The pro/enterprise feature set is borne out of actual use, and to signal the very strong intent to maintain this as unsurprising software that will be there and be supported for the long haul.
[+] [-] anothergoogler|7 years ago|reply
[+] [-] alangpierce|7 years ago|reply
Hopefully it's rare that it's needed, but it's really easy to add and a nice thing to have proactively instead of reactively, and plenty of people feel more welcome when things like that are explicit.
[+] [-] eropple|7 years ago|reply
Thanks for taking a look.
[+] [-] vertex-four|7 years ago|reply
[+] [-] swyx|7 years ago|reply
[+] [-] Kequc|7 years ago|reply
[+] [-] swebs|7 years ago|reply
[deleted]
[+] [-] waibelp|7 years ago|reply
[+] [-] mjuszczak|7 years ago|reply
[+] [-] vanwalj|7 years ago|reply
[+] [-] eropple|7 years ago|reply
And I'm very, very confident.
[+] [-] swyx|7 years ago|reply
[+] [-] realPubkey|7 years ago|reply