brendangregg's comments

Thanks, I never saw the Tech Crunch parodies before. I also didn't use the words "make/making the world a better place".

I actually said "it's about saving the planet" based on all the environmentally worried news articles I've seen.

Given the widely reported issue of AI environmental cost, and being among the top people in the world who can fix it, I don't feel comfortable with _inaction_. I can reduce datacenter sizes, so I'll do that.

As a performance engineer I'm familiar with Jevons paradox, but it does not discourage improving efficiency.

No, that's not what they say.

When did I say I don't think comp is important?

As an engineer I can't comment on future company predictions. It sounds like a question for Sam Altman, as he has discussed risks in the past.

My wife was paying for ChatGPT before I joined. I didn't ask Mia. I probably have three months of hair growth before my next chance to ask.

Again, many comments here saying I only care about the money, and while comp is an important factor I think it characterizes me as someone I'm not, and forgets what I've been doing for the past two decades. I've spent thousands of hours of my life writing textbooks for roughly minimum wage, as I want to help others like me (I came from nothing, with no access tech meetups or conferences, and books were the gateway to a better job). I've published technologies as open source that have allowed others to make millions and are the basis for many startups. I'm also helping pioneer remote work and hoping to set a good example for others to follow (as I've published about before). So I think I'm well known for caring about a lot of things during the past couple of decades.

To answer a few people at once: I did mention compensation as a factor in the post, but I didn't elaborate details, so easy to miss. Comp is important of course, but so are the other factors. It feels like I can't go for a day without reading about the cost of AI datacenters in the news, and I can do something about it.

Funny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed:

https://www.brendangregg.com/sshanalysis.html

The 2023 patch should finally fix that 2004 issue.

Yeah, different numbers, 110 customer meetings, the other post tracked 1-6am meetings. I'm glad I tracked 1-6am meetings since I've shared that number when people think that remote workers aren't making an effort.

Right, laser 300 was called the VZ300 here. I'm out of desk space so I had to put the VZ300 on a stand above my C64C. Maybe AI can finally help me code some C64 and VZ games. :-)

I like to measure things. In real life and on computers. But I also have a couple of work reasons for it:

As a remote worker, I'm under extra pressure to prove that remoteness works.

As a senior employee, I'm also under pressure to regularly report where my time is spent.

I actually only just got the C64C and was surprised I could get it to fit on my desk. C64Cs are massive. Now to connect it to some display...

Right, this kind of job can improve, and should improve. But right now there's talk of ending remote work entirely, without considering that people have been making the effort for years, hence sharing an anecdote. The bigger point is that remote workers are more accommodating than is assumed, and my anecdote is an example of that (even if it is too much).

One other factor I'm curious what people think: If you were offered 7-figures USD to do these kind of hours from anywhere in the world, would that make a difference?

Thanks, glad it's useful!

To everyone building these things: Please add a disclaimer to say something like:

"This is not a vulnerability: eBPF currently requires root access to do this. Also, eBPF makes this easy but does not make it possible, as debuggers, interposers/shims, and other tools can also attach to pre-encryption points, and therefore banning eBPF (as some people want to do after seeing projects like this) would not actually improve security, but it would instead _reduce_ security as it would prevent eBPF-based security solutions from being used."

I could just regenerate these heat maps with 60 rows instead of 50. I'm limited by the sampling rate that was captured in the profile data file. To provide even more resolution (so you had many samples within a game frame) I'd need to re-profile the target with a higher frequency.

When Martin, my colleague at Netflix at the time, built a d3 version of FlameScope, he put a row selector in the UI: https://github.com/Netflix/flamescope

The origin problem for flame graphs was MySQL server performance involving dozens of threads: as a timeline view you need dozens of timelines, one for each thread, since if you render it on one (I know this is probably obvious) then you have samples from different threads from one moment to the next turning the visualization into hair. Flame graphs scale forever and always show the aggregate: any number of threads, servers, microservices, etc.

I think great UI should do both: have a toggle for switching between flame graphs (the summary) and timelines (aka "flame charts") for analyzing time-based patterns. I've encouraged this before and now some do provide that toggle, like Firefox's profiler (Flame Graphs and Stack Charts for timeline view).

As for 20ms, yes, we do want to take it down. A previous HN comment from years ago, when I first published FlameScope, was to put a game frame on the y-axis instead of 1 second, so now each column shows the rendering of a game frame, and you can see time-offset patterns across the frames (better than a time-series timeline). We started work on it and I was hoping to include it in this post. Maybe next one.

Nice tool. Looks like it uses what we now call a "flame chart" (not a flame graph). I don't know if it ever had an original name, but I've seen these in tools from at least the 1990's.

The answer has always been in the source of flamegraph.pl:

  # This was inspired by Neelakanth Nadgir's excellent function_call_graph.rb
  # program, which visualized function entry and return trace events.  As Neel
  # wrote: "The output displayed is inspired by Roch's CallStackAnalyzer which
  # was in turn inspired by the work on vftrace by Jan Boerhout". 

Neel's did upward "flames" and I think CallStackAnalyzer did as well from memory, so based on the ones I was familiar with this was the precedent. We were also dealing with stacks that were commonly less than 30 frames deep, so it tended to fit on the screen. Also in my original flamegraph.pl implementation was an option to invert the flame graph (--inverted), so I've always let people choose. But I guess I had to pick something as the default.

Neel's was the biggest direct inspiration. I changed it to put the alphabet on the x-axis and reduced the color hues, but they look similar.

Microsoft have been driving the work to make eBPF an IETF industry standard.