OmniParser for Pure Vision Based GUI Agent

[deergomoo]

One thing I have found very odd about the current wave of AI tools is that there seems to be an unspoken element of giving up and admitting failure in other areas of computing.

Programming copilots are often sold on how they can automate drudgery and boilerplate, which implies we are incapable of or uninterested in designing programming languages, tools, and patterns which do not require boilerplate or drudgery.

Teaching models to use traditional GUI apps implies we have given up on or are not even bothering to create proper hooks for an automation system to utilise.

Something about it feels wrong to me, because it bakes existing inefficiencies into the system. Can we really not solve the inefficiencies instead of pouring unfathomable amounts of compute into working around them?

[franga2000]

This is not a computer problem, it's a human one. It's not that we don't have APIs and hooks because they're so difficult to implement - we don't have them because software producers don't want or care for us us to have them.

Enabling automation will never be zero effort and anything more than zero effort for something with such a low ROI is a no-go by default. But increasingly, automation is actually seen as a danger to their business models and companies sometimes even go out of their way to prevent it.

Looking at the screen the same way a user does is the only way to win.

[Fulgen]

Focusing on the GUI applications: There have been a few GUI automation solutions over the years - since the post's software is from MS, I'll take UI Automation as an example. Works well with Win32 controls, not sure how well it works with the XAML-based toolkits.

But not all software is written with those UI frameworks. Some use different widget frameworks, some immediate GUIs, others just render a webpage and either use HTML or fully render the controls themselves. And without everybody using the same standard, the only standard we have for parsing their output is the pixels they render to.

[mountainriver]

Computer based agents have no limits, that’s the advantage. Sure a proper automation hook is better if its available, but a lot of the time it isn’t, either due to lack of resources or monopolistic behavior

[Smaug123]

To a considerable extent, we are stuck in the world we live in; but I am reminded of a quote by Guillaume Allais:

> My entire job seems to be repeating variations of "never start by forgetting the user's stated intent only to then attempt to guess it".

[trq_]

This is awesome, can't wait for evals against Claude Computer Use!

[amelius]

Can we first test this with basic sysadmin work in a simple shell?

Can't wait to replace "apt get install" by "gpt get install" and then have it solve all the dependency errors by itself.

[asdev]

how can you write metrics against something that's non deterministic?

[amelius]

Can it detect ads and mask them out?

[bee_rider]

If these sorts of tools kill the ad business, it would be so incredibly cool, and justify Nvidia’s half-of-the-economy-or-whatever market cap.

[dymk]

I’m reminded of Permutation City where your personal AI intercepts ads sent to you, but ad companies of course have their own AI for tricking your AI, so of course you have a countermeasure AI to intercept that, and so on and so forth

[jauntywundrkind]

I have a little bit of a vice of enjoying some "idle" games. I have intended to do some very basic manual screen carving & ocr & computer vision to try to "read" my state in these games, & have multi-actor "play" models for them, just for fun really & to decrease time sunk gaming (by spending significant time coding/learning).

This certainly seems like it has a lot of promise to make that much much much easier. Game UI's are less uniform so maybe this might be harder or not easily be applicable, but hopefully

[_adamb]

As someone who has done this to many games over a few decades, I can definitively say: 100% of the time, it ruins the fun of the game.

I can't say exactly why. Maybe you feel like you haven't earned it. Maybe it's the idle nature of farming that we really enjoy...

[nmstoker]

You might want to look at Serpent AI: granted the repo is now in an archived state, but it did similar things to those you mention.

https://github.com/SerpentAI/SerpentAI

[patrickhogan1]

Since this is a research paper with promising ideas but non-functional code, what are people using as the best-in-class agents for computer automation? For example:

1. Claude for computer use

2. Various startup offerings—if you have recommendations, please list them

3. Established tools like Playwright, Selenium, and WebDriver, combined with screenshots and LLM-based guidance

What tools or approaches are actually working for building useful automation solutions?

[nmstoker]

Are you sure about the non-working code point?

I've yet to try it but my understanding is the repo here has got working code along with installation instructions:

https://github.com/microsoft/OmniParser

[tomatohs]

our agent is available via NPM: http://testdriver.ai

[akshayKMR]

Does it also tell the coordinates (x,y) of the annotated box w.r.t. the screenshot dimensions?

[anonzzzies]

Computer Use, Agent.exe and so on, but nothing actually is useful yet. It's all very terrible. And then to think we had perfection already (and Claude is good at it); emacs... No need for any of this; everything can be scripted.

[s3tt3mbr1n1]

Has anyone gotten this to work?

Copying the repo and downloading the models through HuggingFace or manually does not seem to work, you get errors indicating missing files.

[suriya-ganesh]

I tried as well. Seems like it is a proprietary model.