Show HN: Log collector that runs on a $4 VPS

[Nevin1901]

Hey guys, I'm building erlog to try and solve problems with logging. While trying to add logs to my application, I couldn't find any lightweight log platform which was easy to set up without adding tons of dependencies to my code, or configuring 10,000 files.

ErLog is just a simple go web server which batch inserts json logs into an sqlite3 server. Through tuning sqlite3 and batching inserts, I find I can get around 8k log insertions/sec which is fast enough for small projects.

This is just an MVP, and I plan to add more features once I talk to users. If anyone has any problems with logging, feel free to leave a comment and I'd love to help you out.

[Dachande663]

I’ve found the hard part is not so much the collection of logs (especially at this scale), but the eventual querying. If you’ve got an unknown set of fields been logged, queries very quickly devolve into lots of slow table scans or needing materialised views that start hampering your ingest rate.

I settled on a happy/ok midpoint recently whereby I dump logs in a redis queue using filebeat as it’s very simple. Then have a really simple queue consumer that dumps the logs into clickhouse using a schema Uber detailed (split keys and values), so queries can be pretty quick even over arbitrary fields. 30,00 logs an hour and I can normally search for anything in under a second.

[mr-karan]

I've a similar pipeline to yours (for the storage part). I use vector.dev for collecting and aggregating logs, enriching with metadata (cluster, env, AWS tags) and then finally storing them in a Clickhouse table.

Do you use any particular UI/Frontend tool for querying these logs?

[metadat]

What are the hardware requirements/ / resources dedicated to pull this off?

[FrenchTouch42]

Can you share more information about the schema you're mentioning? Thank you!

[folmar]

Sorry, but I don't see the selling point yet. Rsyslog has omlibdbi module that send your data to sqlite. It can consume pretty much any standard protocol on input, is already available and battle proven.

[remram]

Or just keep it in the log file? I am not sure what is the advantage of putting it in SQLite, if all you're going to do is unindexed `json_extract()` queries on it.

[djbusby]

Or syslog-ng ? And syslog is crazy easy to integrate into nearly any code.

[unxdfa]

I see your idea but you could drop the JSON and use rsyslogd + logrotate + grep? You can grep 10 gig files on a $5 VPS easily and quickly! I can't speak for a $4 one ;)

[tiagod]

If you use grep you'll be doing the same expensive operation every time, following files naively will fail after rotation, etc... And if you use something like Loki it's easier to integrate with other tools to react to the logs

[mekster]

Why do people like to stick to inefficient ancient method like grep for log viewing?

Try tools like Metabase and see how it makes your log reading far better.

[Thaxll]

You could have just used Filebeat? It's also in Go and it's pretty easy to use.

https://www.elastic.co/guide/en/beats/filebeat/current/fileb...

[mekster]

I think Vector really shines with its VRL language to parse and enrich data. It's well thought out with buffering for network errors and throwing errors on parsing instead of silently discarding.

https://vector.dev/docs/reference/vrl/

[rsdbdr203]

This is exactly why I build log-store. Can easily handle 60k logs/sec, but I think more importantly is the query interface. Commands to help you extract value from your logs, including custom commands written in Python.

Free through '23 is my motto... Just a solo founder looking for feedback.

[recck]

I came across this a few months ago and have been following pretty closely. Having been using this locally in a Docker container has been painless. The UI is definitely iterating quickly, but the time-to-first-log was impressive! Happy to keep using it.

[spsesk117]

Disclaimer: I am friends with the founder of log-store.

I have been beta testing it for a while for small scale (~50 million non-nested json objects) log aggregation it's working beautifully for this case.

It's a no nonsense solution that is seemless to integrate and operate. On the ops side, it's painless to setup, maintain, and push logs to. On the user side, its extremely fast and straight forward. End users are not fumbling their way through a monster UI like Kibana, access to information they need is straight forward and uncluttered.

I can't speak to it's suitability in a 1TB logs/day situation, but for a small scale straight forward log agg. tool I can't recommend it enough.

[binwiederhier]

log-store [1] is pretty neat. Thanks for making it. It's super powerful and easy to use. There's a learning curve with the query language, but it's super cool once you figure it out.

[1] https://log-store.com/

[remram]

May be more widely applicable for personal servers: lnav, an advanced log file viewer for the terminal: https://lnav.org/

It uses SQLite internally but can parse log files in many formats on the fly. C++, BSD license, discussed 1 month ago: https://news.ycombinator.com/item?id=34243520

[keroro]

If anyones looking for similar services Im using vector.dev to move logs around & it works great & has a ton of sources/destinations pre-configured.

[Hamuko]

I feel like if you're going to use "$4 VPS" as a quantifier, you could at least specify which $4 VPS is being used.

[sgt]

Look at this:

https://www.hetzner.com/cloud

More like $5 but still, 1 vCPU, 2GB RAM, 20GB NVMe storage. Closer to $4 USD if you let go of IPv4 in favor of IPv6 only.

Edit: Looks like that's also a shared vCPU.

[teruakohatu]

DO's 512mb basic VPS starts at $4, so I am guessing it is that.

[withinboredom]

Neat! Have you considered using query params instead of bodies, then just piping the access logs to a spool (no program actually on the server, just return an empty file). Then your program can just read from the spool and dump them into sqlite.

That should tremendously improve throughput, at the expense of some latency.

[Nevin1901]

That's a really good idea, thanks for suggesting it. I'll try implementing it. I'm hoping the main bottleneck is with inserting the logs into SQLite, so using a spool might help

[aninteger]

I'm doing something similar with a $5 VPS, but with fastcgi/c++/sqlite3. I then have a cronjob that then aggregates error logs, generates an summary and posts to a Slack channel. Personally I wish I didn't have to write it, but it works.

[Nevin1901]

One of my eventual goals with erlog is actually doing observability (eg: it'll send you reports if logs/metrics deviate from the norm), so it's really interesting to see you had this problem.

[sgt]

Imagine what we could do with modern hardware if programs were as efficient as your typical C++/SQLite combo!

[andymac4182]

I have been using https://datalust.co/ to handle this. It scales really well down and up to how much you want to spend. It comes with existing integrations with a lot of libraries and formats and a CLI to push data from file based logs to their service.

They have just added a new parser and query engine written in Rust to get the best performance out of your instance. https://news.ycombinator.com/item?id=34758674

[peterpost2]

Second this, seq is incredibly handy and easy to query. Performance could be better though

[cnkk]

I am been using vector.dev for a long time now. It is also easy to setup. And it looks similar to your idea.

[ilyt]

...uh, just rsyslog and files ? I think it can even write to SQLite

[marcrosoft]

Woah cool. I did the same thing. I Made a poor man’s small scale splunk replacement with SQLite json and go. I used the built in json and full text search extensions.

[Weryj]

I run a self hosted version of Sentry.io on a NUC at home and a relay on a VPS, the. Use Tailscale to connect the two.

If you have an old computer at home, using a VPS as the gateway is always a good option.

Edit: you can then use the VPS as a exit node for internet.

[harisamin]

Ah cool! Somewhat related I built a json log query tool recently using rust and SQLite. Didn’t build the server part of it

https://github.com/hamin/jlq

[Nevin1901]

That's really cool. I might rewrite some of your code in go and use it in erlog for searching (and give you credit of course).

How did you come up with the idea for jlq? It seems like it solved a pretty cool use case.

[arjvik]

I'm working on a project where I'm handling simultaneous connections to a bunch of peers. What's the best way to log messages to trace the flow of requests through my system when multiple code paths are running asynchronously (NodeJS, so I can't simply get a thread ID)?

[Groxx]

With tracing libraries, e.g. https://opentracing.io/

[ilyt]

jaeger + opentracing/opentelemetry libs are easy enough. When testing Jaeger can just work as in-memory database, or put it to some other storage like elasticsearch

[int0x2e]

I strongly urge people to try something like Application Insights. It's not dirt cheap, but not that expensive, and lets you collect anything you'd want and query your telemetry/logs retroactively extremely flexibly. It's just great.

[maybesimpler]

You could also not write your own server. Just configure OpenResty, write some simple LUA to push to the redis queue. Then consume the queue via your language of choice to write to your store(clickhouse).

[vbezhenar]

Logs must be stored in S3, it's no-brainer. Disk storage is too expensive. Logging system should be designed for S3 from ground up IMO.

[addandsubtract]

How is S3 the cheapest option? Backblaze is $0.005 per GB and Hetzner sells storage boxes for less than €0.0038 per GB.