wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wrangle Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wrangling Data @dog_rates aka. WeRateDogs\n",
    "\n",
    "# Introduction\n",
    "Real-world data rarely come clean. Using Python and its libraries, we will gather data from a variety of sources and in a variety of formats, assess its quality and tidiness, then clean it. This is called data wrangling. We will document our wrangling efforts in a Jupyter Notebook, plus showcase them through analyses and visualizations using Python its libraries.\n",
    "\n",
    "The dataset that we will be wrangling (and analyzing and visualizing) is the tweet archive of Twitter user [@dog_rates](https://twitter.com/dog_rates), also known as [WeRateDogs](https://en.wikipedia.org/wiki/WeRateDogs). WeRateDogs is a Twitter account that rates people's dogs with a humorous comment about the dog. These ratings almost always have a denominator of 10. The numerators, though? Almost always greater than 10. 11/10, 12/10, 13/10, etc. Why? Because \"they're good dogs, Brent\". WeRateDogs has over 4 million followers and has received international media coverage.\n",
    "\n",
    "**Software that we will be used**  \n",
    "Since we work in a local environment, the following libraries should be installed:\n",
    "* pandas\n",
    "* NumPy\n",
    "* requests\n",
    "* tweepy\n",
    "* json\n",
    "\n",
    "**Context**  \n",
    "Goal: wrangle WeRateDogs Twitter data to create interesting and trustworthy analyses and visualizations. \n",
    "\n",
    "**The Data**  \n",
    "* Enhanced Twitter Archive\n",
    ">The WeRateDogs Twitter archive contains basic tweet data for all 2356 of their tweets. Containing one column the archive does contain though: each tweet's text, which Udacity team has extracted the rating, dog name, and dog \"stage\" (i.e. doggo, floofer, pupper, and puppo) to make this Twitter archive \"enhanced\".\n",
    "\n",
    "* Additional Data via the Twitter API\n",
    ">Then we need retweet count and favorite count are two of the notable column omissions. Fortunately, this additional data can be gathered by anyone from Twitter's API. Using this API we can extract needed data to make our dataset more concise.\n",
    "\n",
    "* Image Predictions File\n",
    "> The Udacity team has run every image in the WeRateDogs Twitter archive through a neural network that can classify breeds of dogs. The results are so amazing: a table full of image predictions (the top three only) alongside each tweet ID, image URL, and the image number that corresponded to the most confident prediction.\n",
    "\n",
    "\n",
    "**Project Details** \n",
    "\n",
    "* Data wrangling, which consists of:\n",
    "> Gathering data    \n",
    "> Assessing data  \n",
    "> Cleaning data\n",
    "* Storing, analyzing, and visualizing your wrangled data\n",
    "* Reporting on:\n",
    "> 1) your data wrangling efforts and   \n",
    "> 2) your data analyses and visualizations\n",
    "\n",
    "## Gather Data\n",
    "\n",
    "* The WeRateDogs Twitter archive.\n",
    "> The archive data is downloaded manually from the Udacity lesson's page, then we will be inserted using Pandas libraries.\n",
    "* The tweet image predictions.\n",
    "> This data is hosted on Udacity's servers and should be downloaded programmatically using the Requests library and the following URL: https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv.\n",
    "* Each tweet's retweet count and favorite (\"like\") count at minimum, and any additional data may be interesting.\n",
    "> For this data we will be using TwitterAPI and Tweepy library. Using the tweet IDs in the WeRateDogs Twitter archive, query the Twitter API for each tweet's JSON data using Python's Tweepy library and store each tweet's entire set of JSON data in a file called tweet_json.txt file. Each tweet's JSON data should be written to its line. Then read this .txt file line by line into a pandas DataFrame with (at minimum) tweet ID, retweet count, and favorite count."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessing Data\n",
    "In this step, we will be assessing them visually and programmatically for quality and tidiness issues using two types of assessment. We will be intensively using Pandas and its method, i.e:\n",
    "* `.describe()` to see the summary statistic\n",
    "* `.info()` to see the data types each column and detect missing data\n",
    "* `.duplicates()` to see if there is any duplicated row\n",
    "* we also using some loops to see the weird rating on the archive dataframe\n",
    "\n",
    "\n",
    "**Key Points**  \n",
    "Key points in the data wrangling process for this project:\n",
    "\n",
    "* We want original ratings (no retweets) that have images. \n",
    "* Cleaning includes merging individual pieces of data according to the rules of tidy data.\n",
    "* The fact that the rating numerators are greater than the denominators does not need to be cleaned. This unique rating system is a big part of the popularity of WeRateDogs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the assessment process above, the result is divide into two kinds, quality and tidiness issues.\n",
    "\n",
    "### Quality\n",
    "Quality: issues with content. Low-quality data is also known as dirty data.\n",
    "\n",
    "#### `archive` dataframe:\n",
    "* keep the original tweet except the retweeted\n",
    "* some not useful columns for analysis i.e: in_reply_to_status_id, in_reply_to_user_id, source, expanded_urls, retweeted_status_id, retweeted_status_user_id, and retweeted_status_timestamp\n",
    "* tweet_id in int64 Dtype\n",
    "* timestamp in object Dtype\n",
    "* wrong numerator (decimal value or false detection) in index 516, 1712, 1202, and 763\n",
    "* wrong denominator in index 2335, 342, and 516\n",
    "* 'None' value instead of NaN in name and dog stages colummn\n",
    "\n",
    "\n",
    "#### `image` dataframe:\n",
    "* duplicated image\n",
    "* tweet_id in int64 Dtype\n",
    "* not columns for analysis for analysis\n",
    "\n",
    "#### `tweepy` dataframe:\n",
    "* non original tweet\n",
    "* id column name is not match with other dataframe\n",
    "* id in int64 Dtype  \n",
    "* not useful columns for analysis i.e (id_str, in_reply_to_status_id, in_reply_to_status_id_str, in_reply_to_user_id, in_reply_to_user_id_str, lang, quoted_status_id, and quoted_status_id_str\n",
    "\n",
    "### Tidiness\n",
    "Tidiness: issues with a structure that prevents easy analysis. Untidy data is also known as messy data.\n",
    "\n",
    "#### `archive` dataframe\n",
    "* dog stage columns: doggo, floofer, pupper, and puppo is not good\n",
    "\n",
    "#### `image` dataframe:\n",
    "* p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog\n",
    "\n",
    "#### `tweepy` dataframe:\n",
    "-\n",
    "\n",
    "#### make all dataframes into one whole master dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Data\n",
    "\n",
    "The programmatic data cleaning process:\n",
    "\n",
    "* Define\n",
    "* Code\n",
    "* Test\n",
    "\n",
    "As always, we need to copy our dataframe before do any cleaning process, so we can refer back to the old ones.\n",
    "\n",
    "### Archive Dataframe\n",
    "\n",
    "What we will do for this dataframe are:\n",
    "* remove retweeted row with filtering technique\n",
    "* remove not useful for analysis columns using `.drop()` method\n",
    "* change tweet_id datatype into 'object' using `.astype()` method\n",
    "* change timestamp datatype into datetime using `.astype()` method\n",
    "* with some looping we will fix\n",
    "    * numerator for index 516, 1712, 1202, and 763\n",
    "    * wrong denominator for index 2335, 342, and 516\n",
    "* change 'None' into NaN in name and dog stages colummn using numpy \n",
    "* make dog_stage column, then delete the messy columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image Dataframe\n",
    "\n",
    "What we will do for this dataframe are:\n",
    "* remo duplicated image row\n",
    "* change tweet_id in into object datatype\n",
    "* remove all not useful columns for analysis for analysis\n",
    "* select one of p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tweepy Dataframe\n",
    "\n",
    "What we will do for this dataframe are:\n",
    "* remove non original tweet\n",
    "* change id column name to tweet_id then change the datatype to 'object'\n",
    "* remove not useful columns for analysis i.e (id_str, in_reply_to_status_id, in_reply_to_status_id_str, in_reply_to_user_id, in_reply_to_user_id_str, lang, quoted_status_id, and quoted_status_id_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Join and Store All Three Dataframes\n",
    "All dataframe will be merged based on tweet_id as the primary key. The final dataframe will be inner-joined.\n",
    "Then, after final checking, we will save the dataframe to CSV file, named 'twitter_archive_master.csv'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}