August 24, 2018
I firmly believe that web automation / scraping is something that should be in every developer’s tool chest. What was once an extremely time-consuming and complicated task (building web crawlers) has become much easier with NPM packages like selenium-webdriver, cheerio, nightwatch, and puppeteer (not to mention many more).
These packages make it easy to do things like making your computer spin up a browser instance to navigate, save screenshots, fill out forms, save data, and much, much more.
I don’t have a huge preference for any particular setup, but knowing the general concepts and having some automation boilerplate on hand can be a huge time-saver when you’re faced with boring, repetitive tasks.
I recently set up some boilerplate code for myself using selenium webdriver - you can check it out on my github here: selenium tasks boilerplate. Here’s the readme for a quick breakdown:
This is a quick-start repo for anyone wanting to play around with selenium webdriver in node. It gives you four files to start with: login.js, get-console-logs.js, take-screenshot.js, and submit-form.js.
Using it is simple - clone the repo to your machine, cd into it, and run npm install.
That will download the necessary dependencies. To test drive the files, just choose one and run it with node from the terminal: node login.js.
These files are meant to give you a starting point for using selenium. You can find all the api documentation at selenium tasks boilerplate
Feel free to download it just to play around, or to use it as a place to start in saving yourself some time!
Remember: life is too short to do boring stuff. (at least boring stuff you can automate!)