Tqdm (Python)

[BiteCode_dev]

I really love the wave of new tools that are gaining traction in the Python world: tqdm, rich (and soon textual), fastpi, typer, pydantic, shiv, toga, doit, diskcache...

With the better error messages in 3.10 and 11, plus the focus on speed, it's a fantastic era for the language and it's a ton of fun.

I didn't expect to find back the feeling of "too-good-to-be-true" I had when starting with 2.4, and yet.

In fact, being a dev is kinda awesome right now, no matter where you look. JS and PHP are getting more ergonomics, you get a load of new low level languages to sharpen your hardware, Java is modern now, C# runs on Unix, the Rust and Go communities are busy with shipping fantastic tools (ripgrep, fdfind, docker, cue, etc), windows has a decent terminal and WSL, my mother is actually using Linux and Apple came up with the M1. IDEs and browsers are incredible, and they do eat a lot of space, but I have a 32 Go RAM + 1TO SSD laptop that's as slim as a sheet of paper.

Not to mention there is a lot of money to be made.

I know it's trendy right now to say everything is bad in IT, but I disagree, overall, we have it SO good.

[dogline]

I had to look these up (just me doing Google searches):

- tqdm: progress bars (https://tqdm.github.io/)

- rich: text formatting (https://github.com/willmcgugan/rich)

- textual: TUI, using rich (https://github.com/willmcgugan/textual)

- fastpi: Rest APIs (https://fastapi.tiangolo.com/)

- typer: CLI Library, uses Click (https://typer.tiangolo.com/)

- pydantic: Custom data types (https://pydantic-docs.helpmanual.io/)

- shiv: Create Python zipapps (https://shiv.readthedocs.io/en/latest/)

- toga: GUI Toolkit (https://toga.readthedocs.io/en/latest/)

- doit: Task runner (https://pydoit.org/)

- diskcache: A disk cache (https://github.com/grantjenks/python-diskcache/)

[trutannus]

TQDM isn't all that new, it's one of the first things I started to play with when learning to write code. That said, it is an amazing tool.

[systemvoltage]

Despite of all the hate Python gets (pkg system), it’s still my go to language. Most fun. Most rewarding. Getting things done fast.

I can go home with Python at 5pm. And have a good time with my family.

[axg11]

Your positivity and enthusiasm is infectious. Thanks for being an optimist.

[rbdixon]

doit [0] is a superb toolkit for building task-oriented tools with dependencies. It isn't too complex to get started and gives you a lot to work with. I've used it for an internal tool used by everyone in the company for 5+ years and it has never given me a headache.

[0]: https://pydoit.org

[MichaelMoser123]

https://github.com/keleshev/schema a library for validating python data structures by schema definitions, gave me a wow moment. (i have a slightly less original one here: https://github.com/MoserMichael/kwchecker )

[ropable]

I'll add BeeWare (https://beeware.org/) as a pretty nice nice to deploy Python applications cross-platform easily. It's more like a suite of tools, though.

[hpen]

I love Python for many things: scripts, data science, prototypes, etc. But I would never use it to build a large backend system. I just can't handle all the warts. Yes it has been used by large companies, but not without their problems.

[iooi]

tqdm has been around for ages now, it's not exactly new.

[smus]

I'd chip in dramatiq as a nice python tasks/message passing library

[jansky]

rich is great - exactly this library i used to solve same problem as tqdm does and it includes estimation (ETA) too!

Rich is just amazing library

[fnord77]

too bad python is the least energy efficient programming language, by a lot. It's like rolling coal of programming.

[make3]

fire is also really nice for easily building cli

[karmasimida]

tqdm is like around for 5 years or something now

[jasonpeacock]

Might as well toss out Plumbum: https://plumbum.readthedocs.io/en/latest/

No more bash! No more subprocess! Write shell-like code with the convenience of Python!

Seriously, it's a great module. A bit of a learning curve, but then it feels natural.

[pedrovhb]

I raise you xonsh: bash in your Python and vice versa, all in the shell! https://xon.sh/

Also have to mention the fantastic Python Prompt Toolkit, which xonsh is based on - https://python-prompt-toolkit.readthedocs.io/en/master/

I mirror another commenter's excitement on this thread about the cool libraries we have at this age, and the tools that are made possible by them.

[MichaelMoser123]

i have a subprocess wrapper, might also be of help: https://github.com/MoserMichael/subb

[mrbonner]

wow, didn't know this existed. I've been using ipython as shell-ish replacement for not-so-serious-thing. I need to take a detail look at this from my initial glance over. thanks

[andrewshadura]

Also sh.

[emehex]

`from tqdm import tqdm; [i for i in tqdm(range(10))]`

...bit of a learning curve?

[ejdyksen]

If you like tqdm, it's worth checking out pqdm, a parallelized version. If you have embarrassingly parallel work to process in a script, it makes it dead simple to parallelize and monitor the progress of something. Highly recommend:

https://github.com/niedakh/pqdm

[tdekok]

+1 for PQDM, I use it a ton and most of the time it just works. I did have some rare cases where PQDM was much slower than a direct joblib implemention, but that could well be a fluke on my end. Either way, amazing package!

[anonymousCar]

My current issue with tqdm is nested progress bars in multi-threading/processing causing dead locks + tqdm.write just being broken in those contexts, either deadlocking or the ui just being wrong. Does pqdm do a better job?

[Nihilartikel]

One library that haa been really useful for me lately is Dataset https://dataset.readthedocs.io/en/latest/

It makes it a lot easier to casually use a database (e.g. SQLite) to persist dictionaries without explicitly building a schema.

Great for CLI apps and ad hoc data crunching.

[999900000999]

Python is such an amazing language, on one hand, it's easy enough where you could probably train a non technical person on it within about two months.

Yet, you can still make 200K writing it, I don't know the next time I'll create a command line application in Python, but I'll keep this little tool in mind. I hope Python eats the world.

[MichaelMoser123]

it may be easy to get going with python, but it takes a non trivial amount of time to understand, what is going on. I have a advanced python3 course https://github.com/MoserMichael/python-obj-system that explains some of the more advanced concepts. One of the things covered are decorators (tqdm is a decorator)

[pak]

Big fan of tqdm in python, so I ported it to ruby 5 years ago:

https://github.com/powerpak/tqdm-ruby

(shameless plug and an invitation for pull requests)

[QuackyTheDuck]

Good stuff, thank you!

[perrygeo]

Check out Will Chrichton's talk "Type-Driven API Design in Rust" where he live-codes a tqdm-like progress bar for Rust iterators. Solid presentation and eye-opening how straightforward it was to extend the core language through traits.

[curiousgal]

Fun fact, tqdm has an official (pre-alpha) C++ port!

https://github.com/tqdm/tqdm.cpp

[teddyh]

See also the shell command pv(1).

[dwater]

tqdm & click make a nice CLI pairing for taking a simple idea and turning into a reusable tool that you can feel good about sharing in minutes.

[minimaxir]

tqdm is one of the very few Python packages that makes it into every script I write. It's a very high ROI for managing and tracking simple loops.

My only complaint is the smoothing parameter; by default it predicts the estimated time remaining based on the most recent updates so it can fluctuate wildly; smoothing=0 predicts based on the total runtime which makes more sense given law of large numbers.

[drexlspivey]

It has no dependencies which is pretty cool

[enriquto]

Is there a way to globally disable all tqdm progress bars?

Something like the NO_COLOR environement variable?

These progress bars are nice when you launched a single loop yourself, but when you are running an automated battery of many things they become annoying and pollute your terminal too much. I know that you can silence each particular program that uses tqdm by setting a 'disable' option. But this requires editing the python source code of the program.

[zbentley]

Something like this should work:

    import os
    import sys
    from tqdm import tqdm as _tqdm

    def tqdm(*args, **kwargs):
        try:
            disable = bool(int(os.environ['NO_PROGRESSBARS']))
        except KeyError:
            disable = not sys.stdout.isatty()
        except (ValueError, TypeError):
            disable = False
        kwargs.setdefault('disable', disable)
        return _tqdm(*args, **kwargs)

Then import that instead of tqdm.tqdm.

sys.stdout.isatty() isn't a perfect answer to what people ask when they want to know "am I running in an automated environment, or is a human user looking at my output?", but it's close. More nuance is available online.

[emehex]

chime [1] is another python package that fans of tqdm might like.

[1] https://github.com/MaxHalford/chime/

[oars]

Fantastic share and subsequent discussion about in this thread. Threads like these are why I love HN.

My Python skills have improved so much thanks to HN.

[avelis]

I always wondered how a post like this a year ago didn't go anywhere but this post makes the front page. Love the Python package. https://news.ycombinator.com/item?id=23160774

[nurettin]

I like how tqdm can manage multiple progressbars in the same terminal. I have used it to track the progress of multi day processes. It also produces a nice animation on jupyter notebooks.

[debo_]

I just had the pleasant experience of importing tqdm into a script, checking HN, and seeing tqdm here :)

[tmabraham]

Tqdm is awesome and I use it all the time!

Another great alternative is fastprogress: https://github.com/fastai/fastprogress

It often works better in Jupyter Notebooks.

[nvalis]

For notebook I am using tqdm.notebook[0]. Works quite well for me.

[0] https://tqdm.github.io/docs/notebook/

[hathym]

Do you know that tqdm is pronounced like the Arabic word تقدم which mean progress?