[at_smith]

Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?

[jardah]

This tool basicaly performs the simplest data loading, it opens the webpage, then waits till most xhr requests are done, wait's a second (tio give JS time to manipulate DOM) and then loads data from the page. This way, it has what user sees when he opens the page in browser. So if the data is visible, or loaded through XHR or hidden in global JS variable it will see it.

For more advanced usage (like clicking, or submiting a search request) it would need to have some kind of scenario like: "Click on this" -> "wait till this loads" -> "type something here" -> "scroll to this" -> load data.

Which is possible with headless chrome, so the trick is to make it general and easy to use (something like recording what user does through chrome plugin). Maybe in future versions :)

[cseelus]

Could be an interesting enhancement. Sounds a little bit like what Capybara, a test framework for Ruby apps can do[1], things like

  click_link('Link Text')
  fill_in('Password', with: 'Seekrit')
  choose('A Radio Button')
  check('A Checkbox')
  uncheck('Another Checkbox')
  select('Option', from: 'Select Box')

1) https://github.com/teamcapybara/capybara#navigating

[razki]

I'd have money on using Horseman with phantomjs in node.

[bdcravens]

No, it says on the page it's using headless Chrome.

[oodavid]

PhantomJS is dead. The arrival of Puppeteer rung the death knell.