Need help - Web Scraping - BeautifulSoup

Q&A DQ Courses
#1

Hello,
I am working on a web scraping project to get data from booking.com website.
I am able to get the name and other information, but unable to scrape the price and the tax fields. It returns None.
Request you to please guide me how to proceed. I have pasted the code I am using below:

import requests
import re
from bs4 import BeautifulSoup

page = requests.get("https://www.booking.com/searchresults.en-us.html?label=gen173nr-1FCAEoggI46AdIM1gEaIkCiAEBmAExuAEXyAEM2AEB6AEB-AECiAIBqAIDuALa0NztBcACAQ&sid=5b86dc8f82a6b02b3062384d5f86d7d3&sb=1&src=index&src_elem=sb&error_url=https%3A%2F%2Fwww.booking.com%2Findex.html%3Flabel%3Dgen173nr-1FCAEoggI46AdIM1gEaIkCiAEBmAExuAEXyAEM2AEB6AEB-AECiAIBqAIDuALa0NztBcACAQ%3Bsid%3D5b86dc8f82a6b02b3062384d5f86d7d3%3Bsb_price_type%3Dtotal%26%3B&ss=Los+Angeles%2C+California%2C+USA&is_ski_area=&checkin_month=11&checkin_monthday=27&checkin_year=2019&checkout_month=12&checkout_monthday=2&checkout_year=2019&group_adults=2&group_children=0&no_rooms=1&b_h4u_keep_filters=&from_sf=1&ss_raw=los+angeles&ac_position=0&ac_langcode=en&ac_click_type=b&dest_id=20014181&dest_type=city&iata=LAX&place_id_lat=34.052051&place_id_lon=-118.243269&search_pageview_id=63ff7c6d7dcb0064&search_selected=true&search_pageview_id=63ff7c6d7dcb0064&ac_suggestion_list_length=5&ac_suggestion_theme_list_length=0")

#Create an instance of the BeautifulSoup class to parse our document
parser = BeautifulSoup(page.content, "lxml")

hotels = parser.select("#hotellist_inner div.sr_item.sr_item_new")

for hd in hotels:
    price = hd.find('div', {'class' : 'prco-ltr-right-align-helper'})
    print (price)
#2

Just checking… Does your parser.select work and you have hotels variable filled with some values?

#3

Yes, #hotellist_inner covers data for all the hotels and it returns those values, I can parse hotel name, location, etc. but it does not show the price value even if I can see it while inspecting the website. Maybe it is hidden, but hence I am not able to retrieve it.

#4

Have you tried with Selenium?

#5

I haven’t tried with Selenium yet, it is quite new to me. Do you know of any good resources to learn?

#6

Selenium can be a good solution for scraping websites that render part of their content with JavaScript on the fly.

This guide might help you get started.

#7

Thank you so much Slavina! I will definitely try it.

1 Like
#8

Let us know if it worked!