This is fantastic. I'm using a combination of Chrome and PhantomJS for karma testing right now, for https://github.com/paypal/paypal-checkout and https://github.com/krakenjs/xcomponent. There are hundreds of tests opening up hundreds of iframes and popup windows, and sending a lot of cross-window messages, and that ends up being really memory hungry.
Chrome deals pretty well with garbage collection, so long as I'm careful to de-reference closed windows properly¹, and only uses a maximum of 150mbs. PhantomJS eats up almost 6GB of memory before it's done, which makes it almost unusable on machines with less memory or CI boxes. Travis is a no-go.
I'm hoping running Chrome in headless mode should give a nice speedup for our tests.
-----
¹ Turns out even a closed popup window or iframe keeps a huge amount of memory hanging around. Who knew.
The DevTools Protocol is the primary API for headless Chrome, but we are excited for higher-level abstractions like PhantomJS & NightmareJS's API to manipulate the browser as well. Plenty of details to work out, but hopefully sometime this year you'll get a drop-in solution for some of your testing to upgrade from Phantom's older QTWebKit to the latest Chromium.
We're using xvfb and selenium* for testing and a proper headless support would be a hundred time more stable than the self restarting framebuffer can't wait to move to headless chrome
*Yeah I know phantomjs is more cool these days but phantom doesn't support windows height so there's that.
We also did a POC with PhantomJS and found similar issues, as well as generally flakiness causing too many false negatives. Ended up not using it; I'm hoping this can simplify things are give something more solid to build on.
I've been using post-robot and xcomponent a lot recently as I'm solving a bunch of similar challenges. Just wanted to say thank you for sharing your solutions and expertise.
I've been testing Chrome headless extensively for the past few months, and while it's a good step, but it's not stable for high-volume or even diverse set of webpages.
Memory usage is pretty high, lot of heavy webpages result in crashes/hangs, there are many inconsistencies between features available in full version and headless, their debugging protocol has different APIs that work on headless/non-headless in Linux or Windows, and so on.
Of the bugs I've submitted, some have been fixed in the upcoming M59, so other critical ones may take longer due to their backlog. I suppose for now (maybe until M61-62), Chrome full with xvfb or even PhantomJS are better options. When you realize that Chrome is about the same size (by LoC) as the Linux kernel [1], you can't help but wish for a leaner & faster headless browser.
There seems to be some work going on building Firefox pure headless as well. Great overall, as long as all the browsers try to follow the RemoteDebug initiative [2].
I've been successfully using Chrome headless in a 500MB Docker container for dumping the DOM for https://www.prerender.cloud/ for months (rendering a large variety of sites without restarts for weeks at a time)
Run it with:
--js-flags="--max_old_space_size=500"
to force the VM to keep it GC'd below 500
Chrome v55 was a 30% memory savings, before that I used 1GB containers.
It's not perfect, but I am definitely pushing high volume (multiple tabs, concurrent activity) and I am not having any significant stability issues and I am pushing diverse sets of webpages.
Yes, this exactly. I wrote Crabby [1] a few months back to schedule automated page testing using Chrome and webdriver but doing anything automated with Chrome is really atrocious. You can't expect to load more than one page every 10-15 seconds on the average 8GB instance and it occasionally crashes or otherwise stops working completely.
I ended up writing a simple check using Go's net/http library to do basic performance profiling but it doesn't measure DOM loading like the Chrome checks do. Such a bummer
What I really want is an easy, cross-platform way to collect the network timings for each object like you get in Chrome's dev tools network waterfall graphs.
Thanks for that information. I've been putting off moving over to Chrome headless for https://urlscan.io and haven't had the time to do extensive testing yet. Right now Xvfb works fine for me. Still, I'm running 3 Chrome instances with 16 tabs each in parallel and have to kill the processes every so often because they get "stuck".
Unfortunately, I have to agree, though they have been very helpful and vigilant on the bugs that have been filed. Most of mine have been fixed in a few days.
I've been working on a fully open source Windows/macOS library (via Chromium Embedded Framework) that allows you to render pages to memory (and then of course to bitmaps, textures etc.) as well as inject synthesized mouse/keyboard/JavaScript events. It currently uses (what amounts to) Chrome 57.
It's a phantom js (and other headless browser) web service. Using the site, you can quickly create different tests, scheduled tests, chained tests, keep screen shots, create videos of multi step tests, and have historical information of it all.
Can't say enough good things about the site.
Edit: also there's a great chrome extension that will record your mouse clicks and keyboard commands to make creating a test that much simpler.
+1 to GhostInspector; I used them at a previous company a few years ago, and it was very useful.
They were just starting, but service was rather reliable, and their tech support was excellent (maybe because we were early customers). We used to run a bunch of automated tests for monitoring and compliance, archiving hourly screenshots over different builds for later comparison.
I'm the co-founder of a startup that provides an enterprise solution in this space (https://functionize.com).
I agree with your sentiment regarding these types of tools. It has been a long time coming but this release and the tools available now are things I wish I had years ago when I was building my first company.
Mostly, I'd like to know how the control of the virtual time system would be exposed. Would it be through the C++ API, or could it be made available through the debugging protocol?
If you're using Go, you might want to check out https://github.com/knq/chromedp ... There is also a similar package for NodeJS. Otherwise, you can use Selenium in other languages.
I've been trying to test audio and video with headless browsers (namely PhantomJS) but have experienced extreme difficulty, I wonder if headless Chrome is able to support/supports already HTMLAudioElement or HTMLVideoElement or any media interface that would make, for example, testing YouTube or SoundCloud embeds easier.
Related: I want to take screenshots of a few news websites for a little fake news project of mine, and most approaches return something completely different than what I'm seeing when I open Chrome.
Limited height would be better/ok (something like the first 3000 pixels).
Low volume / can be slow (30 seconds would be ok).
Those news websites many times have infinite scrolling.
I've tried:
- phantomJS (rendering sucked, tried every technique I could find to wait for JS to load)
- wkhtmltopdf (almost ok, generates a huge 30M image with all the height, no antialiasing it seems)
WebGL is supposedly a first class citizen of the browser and is used in a ton of pages, but it gets left out or deferred in many tooling packages. There are many tools that are nearly useless to us because they don't support WebGL. It's disappointing.
Oh my goodness I have been waiting for this day for a while - we ran into PhantomJS problems with keyboard/mouse eventing and the HTMLVideoElement for testing, this sounds like it should be the cure for our woes of having to hack around PhantomJS's deficiencies.
the chrome browser spends a decent amount of time on other steps such as parsing HTML. I wonder how much time could be saved by not rendering pages into pixels.
[+] [-] bluepnume|9 years ago|reply
Chrome deals pretty well with garbage collection, so long as I'm careful to de-reference closed windows properly¹, and only uses a maximum of 150mbs. PhantomJS eats up almost 6GB of memory before it's done, which makes it almost unusable on machines with less memory or CI boxes. Travis is a no-go.
I'm hoping running Chrome in headless mode should give a nice speedup for our tests.
-----
¹ Turns out even a closed popup window or iframe keeps a huge amount of memory hanging around. Who knew.
[+] [-] paulirish|9 years ago|reply
The DevTools Protocol is the primary API for headless Chrome, but we are excited for higher-level abstractions like PhantomJS & NightmareJS's API to manipulate the browser as well. Plenty of details to work out, but hopefully sometime this year you'll get a drop-in solution for some of your testing to upgrade from Phantom's older QTWebKit to the latest Chromium.
[+] [-] LoSboccacc|9 years ago|reply
*Yeah I know phantomjs is more cool these days but phantom doesn't support windows height so there's that.
[+] [-] PatrickAuld|9 years ago|reply
[+] [-] akras14|9 years ago|reply
[+] [-] Rumudiez|9 years ago|reply
[+] [-] sbuccini|9 years ago|reply
[+] [-] jslove|9 years ago|reply
[+] [-] nreece|9 years ago|reply
Memory usage is pretty high, lot of heavy webpages result in crashes/hangs, there are many inconsistencies between features available in full version and headless, their debugging protocol has different APIs that work on headless/non-headless in Linux or Windows, and so on.
Of the bugs I've submitted, some have been fixed in the upcoming M59, so other critical ones may take longer due to their backlog. I suppose for now (maybe until M61-62), Chrome full with xvfb or even PhantomJS are better options. When you realize that Chrome is about the same size (by LoC) as the Linux kernel [1], you can't help but wish for a leaner & faster headless browser.
There seems to be some work going on building Firefox pure headless as well. Great overall, as long as all the browsers try to follow the RemoteDebug initiative [2].
[1] https://josephg.com/blog/electron-is-flash-for-the-desktop/
[2] http://remotedebug.org
[+] [-] jotto|9 years ago|reply
Run it with:
to force the VM to keep it GC'd below 500Chrome v55 was a 30% memory savings, before that I used 1GB containers.
It's not perfect, but I am definitely pushing high volume (multiple tabs, concurrent activity) and I am not having any significant stability issues and I am pushing diverse sets of webpages.
[+] [-] chrissnell|9 years ago|reply
I ended up writing a simple check using Go's net/http library to do basic performance profiling but it doesn't measure DOM loading like the Chrome checks do. Such a bummer
What I really want is an easy, cross-platform way to collect the network timings for each object like you get in Chrome's dev tools network waterfall graphs.
[1] https://github.com/chrissnell/crabby
[+] [-] heipei|9 years ago|reply
[+] [-] fastest963|9 years ago|reply
[+] [-] callumprentice|9 years ago|reply
I've been working on a fully open source Windows/macOS library (via Chromium Embedded Framework) that allows you to render pages to memory (and then of course to bitmaps, textures etc.) as well as inject synthesized mouse/keyboard/JavaScript events. It currently uses (what amounts to) Chrome 57.
Looks like this might make my project obsolete.
[+] [-] keville|9 years ago|reply
[+] [-] MichaelApproved|9 years ago|reply
It's a phantom js (and other headless browser) web service. Using the site, you can quickly create different tests, scheduled tests, chained tests, keep screen shots, create videos of multi step tests, and have historical information of it all.
Can't say enough good things about the site.
Edit: also there's a great chrome extension that will record your mouse clicks and keyboard commands to make creating a test that much simpler.
[+] [-] guiambros|9 years ago|reply
They were just starting, but service was rather reliable, and their tech support was excellent (maybe because we were early customers). We used to run a bunch of automated tests for monitoring and compliance, archiving hourly screenshots over different builds for later comparison.
[+] [-] rgrieselhuber|9 years ago|reply
I agree with your sentiment regarding these types of tools. It has been a long time coming but this release and the tools available now are things I wish I had years ago when I was building my first company.
[+] [-] vvoyer|9 years ago|reply
[+] [-] kenshaw|9 years ago|reply
It's also worth noting that the 57+ series has a nice embedded viewer you can use to view the actual viewport via devtools.
[+] [-] oceanswave|9 years ago|reply
Will generate a project file you can use in your own solution. Generated protocol is customizable via mustache templates too
[+] [-] skibz|9 years ago|reply
Mostly, I'd like to know how the control of the virtual time system would be exposed. Would it be through the C++ API, or could it be made available through the debugging protocol?
[+] [-] ctphipps|9 years ago|reply
[+] [-] vvoyer|9 years ago|reply
But ultimately selenium and webdriver.io will do a better job at this
[+] [-] kenshaw|9 years ago|reply
[+] [-] sandstrom|9 years ago|reply
or wait for this issue: https://github.com/segmentio/nightmare/issues/224
[+] [-] RandomBookmarks|9 years ago|reply
Not headless, but ideally suited for automating complex websites (date controls, etc).
[+] [-] bcherny|9 years ago|reply
[+] [-] fake-name|9 years ago|reply
I'm trying to replace PhantomJS in my infrastructure with chromium. Not having to build my own chromium will be a very nice thing.
[+] [-] vmasto|9 years ago|reply
[+] [-] swah|9 years ago|reply
Limited height would be better/ok (something like the first 3000 pixels).
Low volume / can be slow (30 seconds would be ok).
Those news websites many times have infinite scrolling.
I've tried:
- phantomJS (rendering sucked, tried every technique I could find to wait for JS to load)
- wkhtmltopdf (almost ok, generates a huge 30M image with all the height, no antialiasing it seems)
- https://github.com/gen2brain/url2img (this was the best so far, uses Qt bindings but not the latest version)
- actually run a headless browser in DigitalOcean with xvfb-run and take a screenshot: I failed at this
What I didn't tried was Selenium, because it seemed even harder.
How would you guys do it?
[+] [-] yarp|9 years ago|reply
[+] [-] jevinskie|9 years ago|reply
https://docs.google.com/document/d/1VTcYz4q_x0f1O5IVrvRX4u1D...
[+] [-] colordrops|9 years ago|reply
[+] [-] Bahamut|9 years ago|reply
[+] [-] iAm25626|9 years ago|reply
server side SCTP to client(p2p over SCTP/data channel) would be cool.
[+] [-] arrowgunz|9 years ago|reply
[+] [-] JepZ|9 years ago|reply
[+] [-] sandstrom|9 years ago|reply
[+] [-] ericb|9 years ago|reply
[+] [-] vvoyer|9 years ago|reply
Video: https://youtu.be/GivjumRiZ8c
Slides: https://docs.google.com/presentation/d/1gqK9F4lGAY3TZudAtdcx...
More links: Headless Chrome architecture: https://docs.google.com/document/d/11zIkKkLBocofGgoTeeyibB2T...
Mailing list: https://groups.google.com/a/chromium.org/forum/#!forum/headl...
All of those links are on https://chromium.googlesource.com/chromium/src.git/+/master/...
[+] [-] mstade|9 years ago|reply
[+] [-] est|9 years ago|reply
[+] [-] livoras|9 years ago|reply
[+] [-] tianlins|9 years ago|reply
https://developers.google.com/web/fundamentals/performance/c...
the chrome browser spends a decent amount of time on other steps such as parsing HTML. I wonder how much time could be saved by not rendering pages into pixels.