Python
Price Scraper Bot - Scan for Bargains
The two versions of code can use BeautifulSoup or Selenium to periodically scrape data from online shops for a given product to determine the current price of that product. The upgrade to the project (that I haven't included yet) is to include pre-defined prices at which you think there is a good deal on the product and send yourself a notification through services like automated email or apps like Pushover. See my Gmail Price Alerts and Trade Finder Bot projects for examples of this.
To start off you need to work out if the price value is returned in the HTML or if it is in Javascript. BeautifulSoup can only work with HTML, and Selenium only with Javascript.
We will us this page as an example for HTML:
Jacket Link - k-way
These are the steps to find the informtion we need:
-Opening the page in your preferred web browser
-Highlight the price, right click, and select 'Inspect' to see how the price is returned
-Alternatively, you can press F12 and Ctrl + F and search for the value in the 'Elements' tab, e.g. search for '250' or 'price' yourself
We can see the following:
We should be able to see where in the HTML the price is returned to tell the code where it should look for the product. See if you can find the relevant section here:
I ended up using a different section of the code to pull the price, which is contained in an attribute called 'data-gtm-price' within the 'div' tag. In this case, we could also have used the attribute 'data-gtm-compare' to correctly find the price.
We can then extract the price using this code below:
import requests from bs4 import BeautifulSoup # Product URL URL = "https://www.k-way.com/products/jackets-woman-aubette-memory-touch-black-pure-k8132pw-usy?_pos=2&_psq=aubette&_ss=e&_v=1.0" # Headers to mimic a real browser (avoid being blocked) HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" } # Fetch the page response = requests.get(URL, headers=HEADERS) # Parse with BeautifulSoup soup = BeautifulSoup(response.text, "html.parser") # Find the div containing the price in data-gtm-price price_tag = soup.find("div", {"data-gtm-price": True}) if price_tag: price = price_tag["data-gtm-price"] print(f"Price: {price}") else: print("Price not found.")
For sites that give you the price within Javascript code, Selenium must be used, which I've found to be slightly slower to use. Take this site for example:
Golf Balls
Using the BeautifulSoup code above will not work in this case, as the price information is sent in Javascript. We must instead use Selenium. You may need to try scraping the price using BeautifulSoup before you determine Selenium is the way to go.
Selenium is more verbose, as you can tell, since it actually loads the site through Chrome to scrape the data. This means you need to have Chrome installed for it to work.
In our example, we are using 'options.add_arguments()' to reduce the amount of output logs without opening a Chrome window. The code gives 10 seconds for the elements to appear, then attempts to extract the price.
Here is the code:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.service import Service from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from webdriver_manager.chrome import ChromeDriverManager # Product URL URL = "https://houseofgolf.com.au/collections/golf-balls/products/srixon-2024-q-star-tour-golf-balls" # Set up Selenium WebDriver (downloads the latest ChromeDriver automatically) options = webdriver.ChromeOptions() options.add_argument("--headless") # Run in background (no browser window) options.add_argument("--disable-gpu") options.add_argument("--no-sandbox") options.add_argument("--log-level=3") # Suppress warnings options.add_experimental_option("excludeSwitches", ["enable-logging"]) # Remove DevTools logs driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options) driver.get(URL) # Give the page time to load (useful for dynamic content) driver.implicitly_wait(10) # Find the price element print(f"Checking: {driver.title}") try: price_element = WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.CLASS_NAME, "price"))) price = price_element.text print(f"Current Price: {price}") except: print("Price not found.")
It should be simple to pre-define a target price for each item, and then compare that price to the current price. If the current price is at or below the target price, send a notification or email to yourself that you have found a great deal.