Skip to main content

Headless Chrome and the Puppeteer Library for Scraping and Testing the Web

With the advent of Single Page Applications, scraping pages for information as well as running automated user interaction tests has become much harder due to its highly dynamic nature. The solution? Headless Chrome and the Puppeteer library.
While there's always been Selenium, PhantomJS and others, and despite headless Chrome and Puppeteer arriving late to the party, they make for valuable additions to the team of web testing automation tools, which allow developers to simulate interaction of real users with a web site or application.
Headless Chrome is able to run without Puppeteer, as it can be programmatically controlled through the Chrome DevTools Protocol, typically invoked by attaching to a remotely running Chrome instance:
chrome --headless --disable-gpu
                     --remote-debugging-port=9222
Subsequently loading the protocol's sideckick module 'chrome-remote-interface' which provides  a simple abstraction of commands and notifications using a straightforward JavaScript API, one can execute  JavaScript scripts under a local Node.js installation.
From the official documentation, here is an  example that navigates to https://example.com and saves a screenshot as example.png::
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});
  await browser.close();
})();
But since there's 'chrome-remote-interface' already, what does Puppeteer do differently? Puppeteer offers a higher level API to the CDP than the one made available by 'chrome-remote-interface'.

Comments

Popular posts from this blog

Serverless JavaScript

We recently joined in an interesting two-hour long conversation about Serverless JavaScript led by Steve Faulkner of Bustle who answered questions on Bustle, the Shep framework, the mindset behind the AWS Lambda infrastructure, and related topics.

The discussion took place on the Sideway conversation-sharing platform on January 6th. Here we present the best takeaways from the session which really should be taken notice of by anyone working on AWS.

Steve Faulkner:
At Bustle we serve over 50 million unique readers per month through a "serverless" architecture based on AWS Lambda and Node.js.  Of course there are still servers but we don't manage them. This shift has allowed us to develop products faster and decreased the cost of our infrastructure. I'll answer any questions about how we made this transition and how it has worked out. I'll also discuss some of the tools and best practises including our open source framework shep

Eran Hammer:
When would you…

Insider's Guide To Udacity Android Developer Nanodegree Part 3 - Making the Baking App

Continuing to chart my experience of Udacity's Android Developer Nanodegree we step up in level, embarking on the advanced part of the super-course.
Completing project "Popular Movies" (see Part 2 of this series) signaled the end of "Android Developer". Now we are ready to tackle the second element of the program "Advanced Android Developer", a new class with a new syllabus and project. Continuing to chart my experience of Udacity's Android Developer Nanodegree we step up in level, embarking on the advanced part of the super-course.

Completing project "Popular Movies" (see Part 2 of this series) signaled the end of "Android Developer". Now we are ready to tackle the second element of the program "Advanced Android Developer", a new class with a new syllabus and project.

"Advanced Android Developer" is a mixed bag of self contained material and of coding seven different sample apps to learn about the…

Export your Wunderlist tasks with XPath

As brought up in this ProductHunt thread, the news is that Wunderlist is going to be deprecated in favor of the new Microsoft To-Do note taking platform.

This is what Wunderlist support had to say in response to my inquiry on Wunderlist's future:

"Now that the next evolution of Wunderlist is here, in the form of Microsoft To-Do Preview (https://www.wunderlist.com/blog/...), Wunderlist will no longer receive any updates or bug fixes and will eventually be retired. It won’t happen in the next few months and we’ll be sure to give our users plenty of notice beforehand. In the meantime, you can continue to use Wunderlist normally. Of course, we’d also love for you to try To-Do and let us know how you like it – and how we can improve it. While Wunderlist will continue to exist alongside To-Do for the time being, support for Wunderlist will eventually be removed. Not to worry, though! We will inform all Wunderlist users prior to shutting down service. You'll have ample opport…