Web Scraping with Python

Steps of web scraping
  • BeautifulSoup : BeautifulSoup is a Python tool that allows you to parse HTML and XML texts. It generates parse trees, which are useful for quickly extracting information.
  • Pandas: Pandas is a library that may be used to manipulate and analyse data. It is customary to extract data and save it in the desired format.
  • Requests :The requests module allows you to use Python to send HTTP requests. The response data from an HTTP request is returned as a Response Object (content, encoding, status, etc).
  1. First find the URL or website you want to scrap
import requests
from bs4 import BeautifulSoup
import pandas as pd
products=[]
prices=[]
url="https://www.flipkart.com/search?q=phones&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off"
response = requests.get(url)
htmlcontent = response.content
soup = BeautifulSoup(htmlcontent,"html.parser")
for a in soup.findAll('a', attrs={'class':'_1fQZEK'}):
name=a.find('div',attrs={'class':'_4rR01T'})
price=a.find('div',attrs={'class':'_30jeq3 _1_WHN1'})
products.append(name.text)
prices.append(price.text)
final=[]
for i in prices:
i=i[1:]
final.append(i)
df = pd.DataFrame({'Product Name':products,'Prices':final})
df.head()
Final Output

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Add Video Calling to Your App Easily (Quickblox SDK Overview)

How To Improve Your Data Quality With An API?

Use An API To Get LME Tin Real-Time Rates

How to Deal With the Headache of Managing Multiple Configuration Files

How To Use An IP Location Lookup API

Frame Rate Test — Parsec Streaming Versus Steam-In-Home-Streaming + VPN For Cloud Gaming

Creating my first package in R

13 Non-Technical Ways to Make Your Computer Run Faster

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nidhish Gandhi

Nidhish Gandhi

More from Medium

Learn Concatenation and Interpolation with Python Strings

Create Ping Pong Game Using Python — Turtle

How to Scrape Google Maps in Python?

Tutorial: Write The Sims 4 Script Mod with Python — Part 1: Introduction