I’m planning to buy a laptop before the end of 2022, but there are many laptops out there, and recording the data for each one is a repetitive process, so what I wanted to do was automate it. I decided to use Selenium WebDriver, a tool I’ve wanted to use for a long time, as it can interact with the actual web elements through JavaScript. This is useful when I need to click checkboxes, or in my case, I used it to click a button that loads more laptops.
FEATURES
If you run the program, the program will search Best Buy for every single laptop owned by Best Buy, record information about it (e.g. processor, amount of RAM, battery life) and instead of just printing everything, it will create a spreadsheet with the time and date on the filename. From there on, you can just filter/sort the data yourself to meet specific needs (e.g. if you want a certain colour for your laptop) Although this is specifically programmed to scrape data on laptops, the idea is that this can be expanded to track any pool of data given a set of constraints.
PROGRAMMING DETAILS
I was new to Selenium and using CSV, so I found some resources on them. CSV is easy to use as it comes in as a built-in feature.
What I used was a feature named XPATH, which allows you to search for specific tags with an HTML document. For example, if you wanted to find all links within a <div> tag with class=”ListOfLinks”, you would use “div[@class=’ListOfLinks’]/a”. The cool thing about Selenium is that it doesn’t automatically crash if it can’t find an element. For example, I usually try to find deals for each laptop: (screenshot of a laptop with a deal and one without, side by side) But sometimes laptops don’t have any deals active, so I wrote code that checks if a deal exists. If there is no such deal I mark that down.
(the N/A’s mean no deal is active)
I encountered a few other bugs, such as Selenium not being able to find certain text I wanted because it didn’t wait until the page fully loaded, and I found a function called WebDriverWait() that helps me deal with that problem, although I still get it from time to time.
And some Best Buy laptops do not specify how much storage there is, which leads to some data looking out of place.
An old screenshot where some missing attributes shifted the data.
So I had to manually handle all these discrepancies, and I also formatted some of the input to look better (for example, I changed $599.99 to $600).
UPCOMING FEATURES
Some next steps I had in mind were: (in order)
Version 2.0- Allow the program to scrape sites like Dell and HP and put them in the same spreadsheet.
Version 2.1- Allowing the user to select certain features. (e.g. if the user doesn’t want anything over $1500, they can save time by having the program not check laptops over $150)).
Version 2.2- Have a backup plan in case Selenium can’t find a certain element that I can find using my browser.
Version 3.0- A graphic user interface (GUI), including a progress bar to inform the user how long it is taking so they know what step the web scraper is on, just like when you install a program in Windows. This might make the program look more professional.