Need help - Web Scraping - BeautifulSoup

I am working on a web scraping project to get data from website.
I am able to get the name and other information, but unable to scrape the price and the tax fields. It returns None.
Request you to please guide me how to proceed. I have pasted the code I am using below:

import requests
import re
from bs4 import BeautifulSoup

page = requests.get("")

#Create an instance of the BeautifulSoup class to parse our document
parser = BeautifulSoup(page.content, "lxml")

hotels ="#hotellist_inner div.sr_item.sr_item_new")

for hd in hotels:
    price = hd.find('div', {'class' : 'prco-ltr-right-align-helper'})
    print (price)

Just checking… Does your work and you have hotels variable filled with some values?

Yes, #hotellist_inner covers data for all the hotels and it returns those values, I can parse hotel name, location, etc. but it does not show the price value even if I can see it while inspecting the website. Maybe it is hidden, but hence I am not able to retrieve it.

Have you tried with Selenium?

I haven’t tried with Selenium yet, it is quite new to me. Do you know of any good resources to learn?

Selenium can be a good solution for scraping websites that render part of their content with JavaScript on the fly.

This guide might help you get started.

Thank you so much Slavina! I will definitely try it.

1 Like

Let us know if it worked!