Color coding OOM (order-of-magnitude) is smart, but the colors chosen seem quite hard to distinguish. Consider picking different default colors; also consider adding a redundant representation of scale, like an integer representing OOM scale.
It is strange that a thoughtful design around OOM would also choose keep so many digits. If the goal is to summarize, then please throw those extra digits away. (e.g. 10.3992s is specifying the time down to the millisecond, but those extra ms are not meaningful at the second time scale.)
What is this about the runtime polling tasks? Does that happen in Tokio? Why? I am only familiar with 2 async runtimes (node, vert.x) and I was under the impression that neither of these "poll their threads" in any meaningful way. Threads (or process or fiber or whatever you want to call it in this context) never initiate action on their own. They are ALWAYS waiting for something to happen to them, and that includes timeouts, which would be triggered by passing a thread a clock tick. The runtime's job is to centrally manage resources that must persist between thread behavior, so at most it is going to be polling external resources, and not it's own threads.
Or maybe I don't understand what polling means here? I would interpret it as meaning "keep checking a well-known place for changes, and then do something if a change is found". But since async threads can't initiate any change on their own, this is nonsensical.
(primary author of the console here) you're right that there are too many digits of precision right now...the reason for that is that it's actually _not_ a thoughtful design at all, though I appreciate you saying that it is; I just picked an arbitrary number when I was writing the format string and didn't really think about it. We probably don't want to display that much precision --- for smaller units, we probably don't want any fractional digits, for larger units like seconds, we probably want two digits of precision maximum.
Regarding the color scheme, glad you like the idea. Because it's a terminal application, the choice of the colors was constrained a bit by the ANSI 256 color palette (https://www.ditig.com/256-colors-cheat-sheet); I wanted it to be obviously a gradient, so I just picked colors that were immediately adjacent to each other in the ANSI palette. It might be better to pick colors that are one step apart from each other, instead, so they're more distinguishable visually...but there's kind of a balancing act between distinguishability and having a clear gradient. We'll keep playing with it!
I'll answer the polling question. The Tokio runtime (and async rust in general) works a bit differently than other async runtimes like node. With node, callbacks are provided and executed when an OS event is received. With Tokio, there are no callbacks. Instead, async logic is organized in terms of tasks (kind of like async green threads). When the task is blocked on external events, it goes into a waiting state. When external OS events are received, the task is scheduled by the runtime and eventually the runtime "polls" it. Because the poll happens in response to an OS event, most times, the poll results in the task making progress. Sometimes there are false positive polls.
This page goes into a bit more depth and shows an example of how one would implement a (very simple) runtime/executor: https://tokio.rs/tokio/tutorial/async
Generally the way async Rust works is that you have a Future trait with a poll method, and if you call it the future will attempt to make progress if it can — e.g. if the task is a timer it will check the time and complete the task if so and otherwise return Pending.
However, async Rust includes an additional concept: Wakers. When your runtime (Tokio) calls poll on a future, it gives the future a waker object, and when the future is ready to continue work, something needs to call wake on the waker. Once this happens, Tokio will poll the task again soon, and Tokio wont poll tasks that have not been woken.
For example, for timers, Tokio includes a timer wheel (a sort of priority queue) with all the registered timers, and the timer wheel calls wake on the appropriate waker whenever a timer expires. Similarly with a message passing channel, the sender calls wake on the receiver's waker when a message is sent.
Have you delved at all into the potential for having a true "rust repl"? This is something I have very much wanted for some time, where instead of just recompiling and re-running everything every time, it just compiles the new line and executes it in the memory context of the already running rust program. I'm just not enough of a low level rust hacker to get it working, but imagine the web frameworks and things that could exist with a good REPL for rapid debugging and prototyping in rust.
I actually spent a whole summer trying to do this in Crystal and I was very nearly successful, however a few low level limitations got in my way at the end of the day. In Crystal it is actually possible to do this kind of REPL if you have a perfect ability to deep marshal/copy any object, and I almost, almost got that working here: https://github.com/sam0x17/marshal
`valuable`[1] was initially written to support tracing, but I see this 0.1 release doesn't use it. Is valuable seen as more of an add-on to this approach, rather than core to it?
What's the intended workflow? Would I run a console an all of my rust services, and then when debugging some prod issue connect to it? Or would I flip a switch? Or is it more for CLIs?
This is such amazing tooling. There's so much best in class engineering going on with this language, Tokio/async runtimes, graphics, web libraries, etc.
Can't wait to try this out! I always wanted the ability to see the tasks that are currently running or waiting.
In the screenshots I can see that tasks have descriptive names. Does anybody know how to set the name for a task? tokio::spawn doesn't take a name parameter. Does it require `tracing`?
Looks like they have added a builder for tokio::task[1] that will allow you to set a name. It's unstable at the moment so you would need to set the tokio_unstable cfg flag.
Very interesting! I can't help but wonder why these technologies (the framework, and accompanying observability) aren't an OS-level feature; wondering if at this point we need an OS at all, and why not ship a minimal kernel + the binary of an application built w/ this to run directly on the VM.
They're not OS-level because there isn't wide consensus on the right way to do things yet.
Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel.
And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be.
But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all?
> wondering if at this point we need an OS at all
Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.
It's been done, check out https://www.nerves-project.org/ for an example of running a minimal kernel + Elixir (BEAM VM) on devices. As I understand it, it's powered by Buildroot and should be possible to do for other VMs/languages. Don't think I've seen anything like this for desktop OS's though!
In Linux there's work being done on a language-agnostic lightweight threading model, User-Mode Concurrency Groups or UMCG: https://lwn.net/Articles/863386/
One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.
Unikernels are a thing e.g. https://mirage.io. AFIACT uptake has been ... slow. You give up a lot to erase most or all of your OS, and only the most performance sensitive applications realize a benefit
That's basically what we did with https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.
Have you considered to also expose this information in an interactive web interface? Using a zoomable timeline view (https://www.tensorflow.org/tensorboard/tensorboard_profiling...), both for after the fact analysis (taking a fixed N second trace and then inspecting it) as well as interactive visualization (automatically scrolling timeline with option to pause and scrub).
Yes, we've designed the overall architecture of the system to be modular so that the telemetry can be consumed by a number of different UIs --- we'd love to see someone write web interfaces and/or native GUIs for the console data. I have basically no web development experience whatsoever, though, so I went with the terminal app, because not having to learn JavaScript first made it a lot easier to get started :)
We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.
this is very exciting b/c it feels like one of the 1st language features centered around runtime debugging.
Most languages don't seem to do a lot for run-time debugging. Being able to `gdb` and step-through on a local binary is a far-cry from detailed metrics / visualizations. We end up resorting to stuff like Honeycomb, but I'm waiting for the days of a programming language built for the ground-up for runtime debugging.
Anyways, this feels like an important step in the right direction and I'm excited to try this out
I've been waiting years for some cooperation between languages and editors to achieve things Bret Victor wanted years ago, like "Just change this constant while the program is running", where you can have very tight iteration loops for things that don't need a full recompile.
I mean, Visual Studio has some kind of hot reloading, but I'm not gonna VS, I want mainstream support for this fast iteration and deep runtime debugging.
The Tokio ecosystem is tremendously important for people considering Rust for professional networking stuff. It not only has great libs / features, but will be well maintained for years to come - this is my firm believe at least and of course what you need for professional projects (as opposed to personal).
Go has pprof (https://github.com/google/pprof), which I've heard good things about --- and, the pprof data model was one of the influences I looked at when designing the Tokio console's wire format. But, I'm not sure if pprof has any similar UIs to the one we've implemented for the Tokio console; and I haven't actually used it all that much.
Will I be able to use console or part of it with the rest of the rust async ecosystem? (for example, async-std or futures-rs), Or is is Tokio specific?
This looks great though it's a shame it only has a TUI mode. The interface is pretty much begging for a proper GUI. I guess we're still a way away from having a good de fact Rust GUI toolkit.
I'm using it for hardware libraries that are mostly I/O bound. We're not C10K, we're barely C10, but it's easier than managing threads manually for me.
At my day job (https://buoyant.io/), we're using it to write a reverse proxy/load balancer --- one of the use-cases where you really, absolutely do need asynchronous concurrency. :)
Having programmed in both Rust and Go: I mostly didn't need what this does in Go because my problem was usually solved by panic'ing, which prints a stack trace of each go routine, allowing me to get enough of the way through figuring out my current problem (usually something was stuck waiting or sending). In tokio, there's no such print-stack behavior, so it's much harder to get a snapshot of what's going on. (I'm relatively new to tokio and Rust, so there's perhaps a handy "print all the tokio tasks and their stack traces" method, but I haven't come across it yet.)
If folks use this console thing for perf reasons and not debug reasons, then yeah, maybe cool to have in Go.
[+] [-] javajosh|4 years ago|reply
Color coding OOM (order-of-magnitude) is smart, but the colors chosen seem quite hard to distinguish. Consider picking different default colors; also consider adding a redundant representation of scale, like an integer representing OOM scale.
It is strange that a thoughtful design around OOM would also choose keep so many digits. If the goal is to summarize, then please throw those extra digits away. (e.g. 10.3992s is specifying the time down to the millisecond, but those extra ms are not meaningful at the second time scale.)
What is this about the runtime polling tasks? Does that happen in Tokio? Why? I am only familiar with 2 async runtimes (node, vert.x) and I was under the impression that neither of these "poll their threads" in any meaningful way. Threads (or process or fiber or whatever you want to call it in this context) never initiate action on their own. They are ALWAYS waiting for something to happen to them, and that includes timeouts, which would be triggered by passing a thread a clock tick. The runtime's job is to centrally manage resources that must persist between thread behavior, so at most it is going to be polling external resources, and not it's own threads.
Or maybe I don't understand what polling means here? I would interpret it as meaning "keep checking a well-known place for changes, and then do something if a change is found". But since async threads can't initiate any change on their own, this is nonsensical.
[+] [-] mycoliza|4 years ago|reply
Regarding the color scheme, glad you like the idea. Because it's a terminal application, the choice of the colors was constrained a bit by the ANSI 256 color palette (https://www.ditig.com/256-colors-cheat-sheet); I wanted it to be obviously a gradient, so I just picked colors that were immediately adjacent to each other in the ANSI palette. It might be better to pick colors that are one step apart from each other, instead, so they're more distinguishable visually...but there's kind of a balancing act between distinguishability and having a clear gradient. We'll keep playing with it!
[+] [-] carllerche|4 years ago|reply
This page goes into a bit more depth and shows an example of how one would implement a (very simple) runtime/executor: https://tokio.rs/tokio/tutorial/async
[+] [-] aliceryhl|4 years ago|reply
However, async Rust includes an additional concept: Wakers. When your runtime (Tokio) calls poll on a future, it gives the future a waker object, and when the future is ready to continue work, something needs to call wake on the waker. Once this happens, Tokio will poll the task again soon, and Tokio wont poll tasks that have not been woken.
For example, for timers, Tokio includes a timer wheel (a sort of priority queue) with all the registered timers, and the timer wheel calls wake on the appropriate waker whenever a timer expires. Similarly with a message passing channel, the sender calls wake on the receiver's waker when a message is sent.
[+] [-] carllerche|4 years ago|reply
[+] [-] mycoliza|4 years ago|reply
[+] [-] sam0x17|4 years ago|reply
I actually spent a whole summer trying to do this in Crystal and I was very nearly successful, however a few low level limitations got in my way at the end of the day. In Crystal it is actually possible to do this kind of REPL if you have a perfect ability to deep marshal/copy any object, and I almost, almost got that working here: https://github.com/sam0x17/marshal
[+] [-] AceJohnny2|4 years ago|reply
[+] [-] kevinmgranger|4 years ago|reply
[1]: https://tokio.rs/blog/2021-05-valuable
[+] [-] staticassertion|4 years ago|reply
Curious to hear, in general, how it's been used.
[+] [-] IMTDb|4 years ago|reply
[+] [-] ninkendo|4 years ago|reply
[+] [-] echelon|4 years ago|reply
This is such amazing tooling. There's so much best in class engineering going on with this language, Tokio/async runtimes, graphics, web libraries, etc.
[+] [-] mycoliza|4 years ago|reply
[+] [-] dthul|4 years ago|reply
In the screenshots I can see that tasks have descriptive names. Does anybody know how to set the name for a task? tokio::spawn doesn't take a name parameter. Does it require `tracing`?
[+] [-] lkt|4 years ago|reply
[1]: https://github.com/tokio-rs/tokio/blob/master/tokio/src/task...
[+] [-] hcarvalhoalves|4 years ago|reply
[+] [-] ReactiveJelly|4 years ago|reply
Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel.
And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be.
But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all?
> wondering if at this point we need an OS at all
Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.
[+] [-] xtagon|4 years ago|reply
[+] [-] pcwalton|4 years ago|reply
One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.
[+] [-] kristjansson|4 years ago|reply
[+] [-] im_down_w_otp|4 years ago|reply
[+] [-] xtagon|4 years ago|reply
[+] [-] Inufu|4 years ago|reply
Have you considered to also expose this information in an interactive web interface? Using a zoomable timeline view (https://www.tensorflow.org/tensorboard/tensorboard_profiling...), both for after the fact analysis (taking a fixed N second trace and then inspecting it) as well as interactive visualization (automatically scrolling timeline with option to pause and scrub).
[+] [-] mycoliza|4 years ago|reply
We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.
[+] [-] carllerche|4 years ago|reply
[+] [-] marmada|4 years ago|reply
Most languages don't seem to do a lot for run-time debugging. Being able to `gdb` and step-through on a local binary is a far-cry from detailed metrics / visualizations. We end up resorting to stuff like Honeycomb, but I'm waiting for the days of a programming language built for the ground-up for runtime debugging.
Anyways, this feels like an important step in the right direction and I'm excited to try this out
[+] [-] w7|4 years ago|reply
These tools also have decent editor integration and can be use hand in hand:
https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic...
https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...
[0] https://github.com/google/pprof
[1] https://github.com/go-delve/delve
[+] [-] ReactiveJelly|4 years ago|reply
I mean, Visual Studio has some kind of hot reloading, but I'm not gonna VS, I want mainstream support for this fast iteration and deep runtime debugging.
[+] [-] pse_pan2233|4 years ago|reply
[+] [-] seanp2k2|4 years ago|reply
[+] [-] mycoliza|4 years ago|reply
[+] [-] pse_pan2233|4 years ago|reply
I've just seen that Goland (the IDE) has some nice integration:
https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...
[+] [-] ReactiveJelly|4 years ago|reply
I've used Tracy for C++, which does similar tracing on OS-level threads, but I don't know much about Python.
[+] [-] zbentley|4 years ago|reply
Java has historically been amazing in this area, it's great to see other languages stepping up as well.
Does anyone know of a similar visualizer for coroutines/threads for async Python?
[+] [-] realcr|4 years ago|reply
Will I be able to use console or part of it with the rest of the rust async ecosystem? (for example, async-std or futures-rs), Or is is Tokio specific?
[+] [-] carllerche|4 years ago|reply
[+] [-] outside1234|4 years ago|reply
So much great stuff!
[+] [-] IshKebab|4 years ago|reply
[+] [-] carllerche|4 years ago|reply
[+] [-] the__alchemist|4 years ago|reply
[+] [-] qdot76367|4 years ago|reply
[+] [-] mycoliza|4 years ago|reply
[+] [-] Dowwie|4 years ago|reply
[+] [-] stjohnswarts|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] iillexial|4 years ago|reply
[+] [-] mjibson|4 years ago|reply
If folks use this console thing for perf reasons and not debug reasons, then yeah, maybe cool to have in Go.
[+] [-] pse_pan2233|4 years ago|reply
Goland (the IDE) has some nice integration:
https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...
[+] [-] teknopurge|4 years ago|reply