diff --git a/your-code/main (1).ipynb b/your-code/main (1).ipynb new file mode 100644 index 0000000..df4c13e --- /dev/null +++ b/your-code/main (1).ipynb @@ -0,0 +1,1369 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.5" + }, + "colab": { + "name": "main.ipynb", + "provenance": [] + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "04XdaynY2UQX" + }, + "source": [ + "#### 1. Import pandas library" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qdvB-RE12UQc" + }, + "source": [ + "import pandas as pd" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eN5H2eba2UQd" + }, + "source": [ + "#### 2. Import pymysql and sqlalchemy as you have learnt in the lesson of importing/exporting data \n" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "PF9YpOvj2UQe", + "outputId": "a32ac729-2167-4405-b869-526d56509b9a" + }, + "source": [ + "!pip install pymysql" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Collecting pymysql\n", + " Downloading PyMySQL-1.0.2-py3-none-any.whl (43 kB)\n", + "\u001b[?25l\r\u001b[K |███████▌ | 10 kB 28.5 MB/s eta 0:00:01\r\u001b[K |███████████████ | 20 kB 36.9 MB/s eta 0:00:01\r\u001b[K |██████████████████████▍ | 30 kB 27.3 MB/s eta 0:00:01\r\u001b[K |██████████████████████████████ | 40 kB 20.6 MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 43 kB 1.0 MB/s \n", + "\u001b[?25hInstalling collected packages: pymysql\n", + "Successfully installed pymysql-1.0.2\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "SXoRote33wxd" + }, + "source": [ + "import pymysql\n", + "from sqlalchemy import create_engine" + ], + "execution_count": 5, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wUYXOW7P2UQf" + }, + "source": [ + "#### 3. Create a mysql engine to set the connection to the server. Check the connection details in [this link](https://relational.fit.cvut.cz/search?tableCount%5B%5D=0-10&tableCount%5B%5D=10-30&dataType%5B%5D=Numeric&databaseSize%5B%5D=KB&databaseSize%5B%5D=MB)" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "xgwsLgz62UQf" + }, + "source": [ + "engine = create_engine('mysql+pymysql://guest:relational@relational.fit.cvut.cz/stats')" + ], + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UeJ5RVC12UQh" + }, + "source": [ + "#### 4. Import the users table " + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 348 + }, + "id": "LQID8hnV2UQi", + "outputId": "6aa3b4bc-e3f3-499e-ec14-0f0751c2ca56" + }, + "source": [ + "data = pd.read_sql_query('SELECT * FROM stats.users', engine)\n", + "data.head()" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
IdReputationCreationDateDisplayNameLastAccessDateWebsiteUrlLocationAboutMeViewsUpVotesDownVotesAccountIdAgeProfileImageUrl
0-112010-07-19 06:55:26Community2010-07-19 06:55:26http://meta.stackexchange.com/on the server farm<p>Hi, I'm not really a person.</p>\\n\\n<p>I'm ...050071920-1NaNNone
121012010-07-19 14:01:36Geoff Dalgas2013-11-12 22:07:23http://stackoverflow.comCorvallis, OR<p>Developer on the StackOverflow team. Find ...2530237.0None
231012010-07-19 15:34:50Jarrod Dixon2014-08-08 06:42:58http://stackoverflow.comNew York, NY<p><a href=\"http://blog.stackoverflow.com/2009...22190335.0None
341012010-07-19 19:03:27Emmett2014-01-02 09:31:02http://minesweeperonline.comSan Francisco, CA<p>currently at a startup in SF</p>\\n\\n<p>form...1100199828.0http://i.stack.imgur.com/d1oHX.jpg
4567922010-07-19 19:03:57Shane2014-08-13 00:23:47http://www.statalgo.comNew York, NY<p>Quantitative researcher focusing on statist...114566255450335.0None
\n", + "
" + ], + "text/plain": [ + " Id Reputation ... Age ProfileImageUrl\n", + "0 -1 1 ... NaN None\n", + "1 2 101 ... 37.0 None\n", + "2 3 101 ... 35.0 None\n", + "3 4 101 ... 28.0 http://i.stack.imgur.com/d1oHX.jpg\n", + "4 5 6792 ... 35.0 None\n", + "\n", + "[5 rows x 14 columns]" + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VmXB3dn42UQl" + }, + "source": [ + "#### 5. Rename Id column to userId" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 348 + }, + "id": "uNsH_hvM2UQn", + "outputId": "3ab0860a-0faf-4be0-eda0-97fc86ce15ee" + }, + "source": [ + "users = data.rename(columns={'Id': 'userId'})\n", + "users.head()" + ], + "execution_count": 15, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
userIdReputationCreationDateDisplayNameLastAccessDateWebsiteUrlLocationAboutMeViewsUpVotesDownVotesAccountIdAgeProfileImageUrl
0-112010-07-19 06:55:26Community2010-07-19 06:55:26http://meta.stackexchange.com/on the server farm<p>Hi, I'm not really a person.</p>\\n\\n<p>I'm ...050071920-1NaNNone
121012010-07-19 14:01:36Geoff Dalgas2013-11-12 22:07:23http://stackoverflow.comCorvallis, OR<p>Developer on the StackOverflow team. Find ...2530237.0None
231012010-07-19 15:34:50Jarrod Dixon2014-08-08 06:42:58http://stackoverflow.comNew York, NY<p><a href=\"http://blog.stackoverflow.com/2009...22190335.0None
341012010-07-19 19:03:27Emmett2014-01-02 09:31:02http://minesweeperonline.comSan Francisco, CA<p>currently at a startup in SF</p>\\n\\n<p>form...1100199828.0http://i.stack.imgur.com/d1oHX.jpg
4567922010-07-19 19:03:57Shane2014-08-13 00:23:47http://www.statalgo.comNew York, NY<p>Quantitative researcher focusing on statist...114566255450335.0None
\n", + "
" + ], + "text/plain": [ + " userId Reputation ... Age ProfileImageUrl\n", + "0 -1 1 ... NaN None\n", + "1 2 101 ... 37.0 None\n", + "2 3 101 ... 35.0 None\n", + "3 4 101 ... 28.0 http://i.stack.imgur.com/d1oHX.jpg\n", + "4 5 6792 ... 35.0 None\n", + "\n", + "[5 rows x 14 columns]" + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W0o0mBZw2UQo" + }, + "source": [ + "#### 6. Import the posts table. " + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 521 + }, + "id": "JkykFXU92UQo", + "outputId": "dc5d27f5-f769-46e9-9bab-3bcdff324c54" + }, + "source": [ + "posts = pd.read_sql_query('SELECT * FROM stats.posts', engine)\n", + "posts.head()" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
IdPostTypeIdAcceptedAnswerIdCreaionDateScoreViewCountBodyOwnerUserIdLasActivityDateTitleTagsAnswerCountCommentCountFavoriteCountLastEditorUserIdLastEditDateCommunityOwnedDateParentIdClosedDateOwnerDisplayNameLastEditorDisplayName
01115.02010-07-19 19:12:12231278.0<p>How should I elicit prior distributions fro...8.02010-09-15 21:08:26Eliciting priors from experts<bayesian><prior><elicitation>5.0114.0NaNNaTNaTNaNNaTNoneNone
12159.02010-07-19 19:12:57228198.0<p>In many different statistical methods there...24.02012-11-12 09:21:54What is normality?<distributions><normality>7.018.088.02010-08-07 17:56:44NaTNaNNaTNoneNone
2315.02010-07-19 19:13:28543613.0<p>What are some valuable Statistical Analysis...18.02013-05-27 14:48:36What are some valuable Statistical Analysis op...<software><open-source>19.0436.0183.02011-02-12 05:50:032010-07-19 19:13:28NaNNaTNoneNone
341135.02010-07-19 19:13:31135224.0<p>I have two groups of data. Each with a dif...23.02010-09-08 03:00:19Assessing the significance of differences in d...<distributions><statistical-significance>5.022.0NaNNaTNaTNaNNaTNoneNone
452NaN2010-07-19 19:14:4381NaN<p>The R-project</p>\\n\\n<p><a href=\"http://www...23.02010-07-19 19:21:15NoneNoneNaN3NaN23.02010-07-19 19:21:152010-07-19 19:14:433.0NaTNoneNone
\n", + "
" + ], + "text/plain": [ + " Id PostTypeId ... OwnerDisplayName LastEditorDisplayName\n", + "0 1 1 ... None None\n", + "1 2 1 ... None None\n", + "2 3 1 ... None None\n", + "3 4 1 ... None None\n", + "4 5 2 ... None None\n", + "\n", + "[5 rows x 21 columns]" + ] + }, + "metadata": {}, + "execution_count": 11 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "og_YRbks2UQp" + }, + "source": [ + "#### 7. Rename Id column to postId and OwnerUserId to userId" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 521 + }, + "id": "ZxnAMcPp2UQq", + "outputId": "bb17f0dc-76f1-465a-cf73-ec975fc2c25d" + }, + "source": [ + "posts2 = posts.rename(columns={'Id': 'postId', 'OwnerUserId': 'userId'}).head()\n", + "posts2.head()" + ], + "execution_count": 16, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
postIdPostTypeIdAcceptedAnswerIdCreaionDateScoreViewCountBodyuserIdLasActivityDateTitleTagsAnswerCountCommentCountFavoriteCountLastEditorUserIdLastEditDateCommunityOwnedDateParentIdClosedDateOwnerDisplayNameLastEditorDisplayName
01115.02010-07-19 19:12:12231278.0<p>How should I elicit prior distributions fro...8.02010-09-15 21:08:26Eliciting priors from experts<bayesian><prior><elicitation>5.0114.0NaNNaTNaTNaNNaTNoneNone
12159.02010-07-19 19:12:57228198.0<p>In many different statistical methods there...24.02012-11-12 09:21:54What is normality?<distributions><normality>7.018.088.02010-08-07 17:56:44NaTNaNNaTNoneNone
2315.02010-07-19 19:13:28543613.0<p>What are some valuable Statistical Analysis...18.02013-05-27 14:48:36What are some valuable Statistical Analysis op...<software><open-source>19.0436.0183.02011-02-12 05:50:032010-07-19 19:13:28NaNNaTNoneNone
341135.02010-07-19 19:13:31135224.0<p>I have two groups of data. Each with a dif...23.02010-09-08 03:00:19Assessing the significance of differences in d...<distributions><statistical-significance>5.022.0NaNNaTNaTNaNNaTNoneNone
452NaN2010-07-19 19:14:4381NaN<p>The R-project</p>\\n\\n<p><a href=\"http://www...23.02010-07-19 19:21:15NoneNoneNaN3NaN23.02010-07-19 19:21:152010-07-19 19:14:433.0NaTNoneNone
\n", + "
" + ], + "text/plain": [ + " postId PostTypeId ... OwnerDisplayName LastEditorDisplayName\n", + "0 1 1 ... None None\n", + "1 2 1 ... None None\n", + "2 3 1 ... None None\n", + "3 4 1 ... None None\n", + "4 5 2 ... None None\n", + "\n", + "[5 rows x 21 columns]" + ] + }, + "metadata": {}, + "execution_count": 16 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f7KS0pwd2UQq" + }, + "source": [ + "#### 8. Define new dataframes for users and posts with the following selected columns:\n", + " **users columns**: userId, Reputation,Views,UpVotes,DownVotes\n", + " **posts columns**: postId, Score,userID,ViewCount,CommentCount" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "z6wGEl4h2UQr", + "outputId": "ac0c93c8-a232-4eb1-9451-4fa985b2f56d" + }, + "source": [ + "users_filt = users[['userId', 'Reputation', 'Views', 'UpVotes', 'DownVotes']]\n", + "posts_filt = posts2[['postId', 'Score', 'userId', 'ViewCount', 'CommentCount']]\n", + "posts_filt.head()" + ], + "execution_count": 20, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
postIdScoreuserIdViewCountCommentCount
01238.01278.01
122224.08198.01
235418.03613.04
341323.05224.02
458123.0NaN3
\n", + "
" + ], + "text/plain": [ + " postId Score userId ViewCount CommentCount\n", + "0 1 23 8.0 1278.0 1\n", + "1 2 22 24.0 8198.0 1\n", + "2 3 54 18.0 3613.0 4\n", + "3 4 13 23.0 5224.0 2\n", + "4 5 81 23.0 NaN 3" + ] + }, + "metadata": {}, + "execution_count": 20 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2R_15Bkt2UQr" + }, + "source": [ + "#### 8. Merge both dataframes, users and posts. \n", + "You will need to make a [merge](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html) of posts and users dataframes." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "szphPZCK2UQs", + "outputId": "4aa2304e-8dd4-4dd4-9062-08cbaebdbecc" + }, + "source": [ + "merged_df = pd.merge(left=users_filt, right=posts_filt, left_on='userId', right_on='userId')\n", + "merged_df.head()" + ], + "execution_count": 31, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
userIdReputationViewsUpVotesDownVotespostIdScoreViewCountCommentCount
0867641089604251231278.01
11812881603543613.04
223308523414135224.02
32330852341581NaN3
424344483612228198.01
\n", + "
" + ], + "text/plain": [ + " userId Reputation Views UpVotes ... postId Score ViewCount CommentCount\n", + "0 8 6764 1089 604 ... 1 23 1278.0 1\n", + "1 18 128 8 16 ... 3 54 3613.0 4\n", + "2 23 308 52 34 ... 4 13 5224.0 2\n", + "3 23 308 52 34 ... 5 81 NaN 3\n", + "4 24 344 48 36 ... 2 22 8198.0 1\n", + "\n", + "[5 rows x 9 columns]" + ] + }, + "metadata": {}, + "execution_count": 31 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jz7i0OjN2UQs" + }, + "source": [ + "#### 9. How many missing values do you have in your merged dataframe? On which columns?" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "YsRGHRgqHsWs" + }, + "source": [ + "users.info()" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4Xs_xbvd2UQs", + "outputId": "2a3cb280-b23b-4995-81f0-f8f1a40c335b" + }, + "source": [ + "merged_df.info()" + ], + "execution_count": 32, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Int64Index: 5 entries, 0 to 4\n", + "Data columns (total 9 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 userId 5 non-null int64 \n", + " 1 Reputation 5 non-null int64 \n", + " 2 Views 5 non-null int64 \n", + " 3 UpVotes 5 non-null int64 \n", + " 4 DownVotes 5 non-null int64 \n", + " 5 postId 5 non-null int64 \n", + " 6 Score 5 non-null int64 \n", + " 7 ViewCount 4 non-null float64\n", + " 8 CommentCount 5 non-null int64 \n", + "dtypes: float64(1), int64(8)\n", + "memory usage: 400.0 bytes\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ys7NnxD92UQt" + }, + "source": [ + "#### 10. You will need to make something with missing values. Will you clean or filling them? Explain. \n", + "**Remember** to check the results of your code before passing to the next step" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cnv71DTGKNJs" + }, + "source": [ + "Hay que reemplazar el valor nulo (NaN) por un float para que no haya problema al manipular los valores de la columna" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "umGNR1EF2UQt", + "outputId": "f152d2d0-85f4-4b8f-ad78-23c2370432db" + }, + "source": [ + "merged_df[['ViewCount']] = merged_df[['ViewCount']].fillna(0)\n", + "merged_df.head()" + ], + "execution_count": 39, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
userIdReputationViewsUpVotesDownVotespostIdScoreViewCountCommentCount
0867641089604251231278.01
11812881603543613.04
223308523414135224.02
323308523415810.03
424344483612228198.01
\n", + "
" + ], + "text/plain": [ + " userId Reputation Views UpVotes ... postId Score ViewCount CommentCount\n", + "0 8 6764 1089 604 ... 1 23 1278.0 1\n", + "1 18 128 8 16 ... 3 54 3613.0 4\n", + "2 23 308 52 34 ... 4 13 5224.0 2\n", + "3 23 308 52 34 ... 5 81 0.0 3\n", + "4 24 344 48 36 ... 2 22 8198.0 1\n", + "\n", + "[5 rows x 9 columns]" + ] + }, + "metadata": {}, + "execution_count": 39 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "epUVhwyo2UQt" + }, + "source": [ + "#### 11. Adjust the data types in order to avoid future issues. Which ones should be changed? " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "XxTKlS_82UQt" + }, + "source": [ + "" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kJZvR7cc2UQu" + }, + "source": [ + "#### Bonus: Identify extreme values in your merged dataframe as you have learned in class, create a dataframe called outliers with the same columns as our data set and calculate the bounds. The values of the outliers dataframe will be the values of the merged_df that fall outside that bounds. You will need to save your outliers dataframe to a csv file on your-code folder." + ] + } + ] +} \ No newline at end of file diff --git a/your-code/main.ipynb b/your-code/main.ipynb deleted file mode 100644 index 31724c5..0000000 --- a/your-code/main.ipynb +++ /dev/null @@ -1,204 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 1. Import pandas library" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 2. Import pymysql and sqlalchemy as you have learnt in the lesson of importing/exporting data \n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 3. Create a mysql engine to set the connection to the server. Check the connection details in [this link](https://relational.fit.cvut.cz/search?tableCount%5B%5D=0-10&tableCount%5B%5D=10-30&dataType%5B%5D=Numeric&databaseSize%5B%5D=KB&databaseSize%5B%5D=MB)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 4. Import the users table " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 5. Rename Id column to userId" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 6. Import the posts table. " - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 7. Rename Id column to postId and OwnerUserId to userId" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 8. Define new dataframes for users and posts with the following selected columns:\n", - " **users columns**: userId, Reputation,Views,UpVotes,DownVotes\n", - " **posts columns**: postId, Score,userID,ViewCount,CommentCount" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 8. Merge both dataframes, users and posts. \n", - "You will need to make a [merge](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html) of posts and users dataframes." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 9. How many missing values do you have in your merged dataframe? On which columns?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 10. You will need to make something with missing values. Will you clean or filling them? Explain. \n", - "**Remember** to check the results of your code before passing to the next step" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 11. Adjust the data types in order to avoid future issues. Which ones should be changed? " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Bonus: Identify extreme values in your merged dataframe as you have learned in class, create a dataframe called outliers with the same columns as our data set and calculate the bounds. The values of the outliers dataframe will be the values of the merged_df that fall outside that bounds. You will need to save your outliers dataframe to a csv file on your-code folder." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.5" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -}