[Fulgen]

Focusing on the GUI applications: There have been a few GUI automation solutions over the years - since the post's software is from MS, I'll take UI Automation as an example. Works well with Win32 controls, not sure how well it works with the XAML-based toolkits.

But not all software is written with those UI frameworks. Some use different widget frameworks, some immediate GUIs, others just render a webpage and either use HTML or fully render the controls themselves. And without everybody using the same standard, the only standard we have for parsing their output is the pixels they render to.