Skip to content

[lab-web-scraping-multipages] Karolin Diede #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
KaroD123 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[lab-web-scraping-multipages] Karolin Diede #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
238 changes: 209 additions & 29 deletions 01_Billboard.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
"cells": [
{
"cell_type": "markdown",
"id": "adequate-aurora",
"metadata": {
"toc": true
},
Expand All @@ -13,7 +12,6 @@
},
{
"cell_type": "markdown",
"id": "together-terminology",
"metadata": {},
"source": [
"## Import libraries"
Expand All @@ -22,7 +20,6 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "little-prophet",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -33,16 +30,14 @@
},
{
"cell_type": "markdown",
"id": "political-wright",
"metadata": {},
"source": [
"## Store the hot-100 songs list from billboard url in a variable"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "promotional-algorithm",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -51,92 +46,277 @@
},
{
"cell_type": "markdown",
"id": "indian-fossil",
"metadata": {},
"source": [
"## Download html with a get request"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "civic-broad",
"execution_count": 3,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"200"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 200 status code means OK!"
"# 200 status code means OK!\n",
"\n",
"response = requests.get(url)\n",
"response.status_code"
]
},
{
"cell_type": "markdown",
"id": "arbitrary-patrick",
"metadata": {},
"source": [
"## Parse html (create the 'soup')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "revised-digest",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": []
"source": [
"soup = BeautifulSoup(response.content, \"html.parser\")"
]
},
{
"cell_type": "markdown",
"id": "damaged-watts",
"metadata": {},
"source": [
"## Retrieve/extract the desired info"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "falling-chambers",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# song titles\n",
"\n",
"songs = soup.find_all(\"span\", class_=\"chart-element__information__song text--truncate color--primary\")\n",
"# artists\n",
"\n",
"artists = soup.find_all(\"span\", class_=\"chart-element__information__artist text--truncate color--secondary\")\n",
"# last week\n",
"\n",
"last_week_ranks = soup.find_all(\"span\", class_= \"chart-element__meta text--center color--secondary text--last\")\n",
"# peak rank\n",
"\n",
"# weeks on chart\n"
"peak_ranks = soup.find_all(\"span\", class_= \"chart-element__meta text--center color--secondary text--peak\")\n",
"# weeks on chart\n",
"weeks_on_chart = soup.find_all(\"span\", class_= \"chart-element__meta text--center color--secondary text--week\")\n"
]
},
{
"cell_type": "markdown",
"id": "employed-chemistry",
"metadata": {},
"source": [
"## Get the text"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "amateur-protocol",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": []
"source": [
"for i in [songs, artists, last_week_ranks, peak_ranks, weeks_on_chart]:\n",
" for j in range(len(i)):\n",
" i[j] = i[j].getText()"
]
},
{
"cell_type": "markdown",
"id": "every-degree",
"metadata": {},
"source": [
"## Build a dataframe"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>song</th>\n",
" <th>artist</th>\n",
" <th>last_week_rank</th>\n",
" <th>peak_rank</th>\n",
" <th>weeks_on_chart</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>My Universe</td>\n",
" <td>Coldplay x BTS</td>\n",
" <td>-</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Stay</td>\n",
" <td>The Kid LAROI &amp; Justin Bieber</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>12</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Industry Baby</td>\n",
" <td>Lil Nas X &amp; Jack Harlow</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Way 2 Sexy</td>\n",
" <td>Drake Featuring Future &amp; Young Thug</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Fancy Like</td>\n",
" <td>Walker Hayes</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>15</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>Pipe Down</td>\n",
" <td>Drake</td>\n",
" <td>68</td>\n",
" <td>14</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>Papi's Home</td>\n",
" <td>Drake</td>\n",
" <td>66</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>Chosen</td>\n",
" <td>Blxst &amp; Tyga Featuring Ty Dolla $ign</td>\n",
" <td>-</td>\n",
" <td>98</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>Toxic Punk</td>\n",
" <td>YoungBoy Never Broke Again</td>\n",
" <td>-</td>\n",
" <td>99</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>99</th>\n",
" <td>Moon</td>\n",
" <td>Kanye West</td>\n",
" <td>76</td>\n",
" <td>17</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100 rows × 5 columns</p>\n",
"</div>"
],
"text/plain": [
" song artist last_week_rank \\\n",
"0 My Universe Coldplay x BTS - \n",
"1 Stay The Kid LAROI & Justin Bieber 1 \n",
"2 Industry Baby Lil Nas X & Jack Harlow 2 \n",
"3 Way 2 Sexy Drake Featuring Future & Young Thug 3 \n",
"4 Fancy Like Walker Hayes 5 \n",
".. ... ... ... \n",
"95 Pipe Down Drake 68 \n",
"96 Papi's Home Drake 66 \n",
"97 Chosen Blxst & Tyga Featuring Ty Dolla $ign - \n",
"98 Toxic Punk YoungBoy Never Broke Again - \n",
"99 Moon Kanye West 76 \n",
"\n",
" peak_rank weeks_on_chart \n",
"0 1 1 \n",
"1 1 12 \n",
"2 2 10 \n",
"3 1 4 \n",
"4 5 15 \n",
".. ... ... \n",
"95 14 4 \n",
"96 8 4 \n",
"97 98 1 \n",
"98 99 1 \n",
"99 17 5 \n",
"\n",
"[100 rows x 5 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"billboard = pd.DataFrame(\n",
" {\"song\": songs,\n",
" \"artist\": artists,\n",
" \"last_week_rank\": last_week_ranks,\n",
" \"peak_rank\": peak_ranks,\n",
" \"weeks_on_chart\": weeks_on_chart}\n",
")\n",
"\n",
"billboard"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "external-instrumentation",
"metadata": {},
"outputs": [],
"source": []
Expand Down
Loading