[TomasEkeli]

i've consistently been improving performance by removing logging.

feels bad, but getting a 55 hour job down to 12 hours by removing logging was worth it.

yes, we log asynchronously and without blocking. but it is a constant tax on everything that happens and it adds up.

[munk-a]

I am extremely curious how asyncronous logging was multiplying the job execution by 4x - are you able to share a bit more information about what particular portions of the logging were costing time? If it was done asynchronously I assume you're collecting to-be-logged records in memory with occasional batch dumps to disk/other persistence but were you spending time collecting auxiliary information that was costing a lot of cycles? Like fetching relevant information from the DB that was useful only for logging and not for the actual job?

I've seen some time jumps with logging removal but I don't think I've ever seen anything exceed about 20% of execution time (at least on the scale you're talking about, there were some scripts I revised logging on that were constructing huge logging infrastructures for an otherwise sub-milisecond job) 40 hours is a lot of hours.

[TomasEkeli]

oh no - introducing asynchronous logging made it better. but not nearly enough.

it was far, far worse than that.

[jkharrat]

I would do the same if I was you. I will still have a log whenever we catch an error because it will help debugging this cron or job or even identify an issue in the provided data. No matter how divisive the opinions about logging are, I still think that mini logs are better than no logs at all.