BigData_Overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center><font color=navy>Big Data Economics</font></center>\n",
    "### <center>An Overview</center>\n",
    "#### <center>Ali Habibnia</center>\n",
    "\n",
    "    \n",
    "<center> Assistant Professor, Department of Economics, </center>\n",
    "<center> and Division of Computational Modeling & Data Analytics at Virginia Tech</center>\n",
    "<center> habibnia@vt.edu </center> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "<img src=\"images/tech3.jpg\" alt=\"Drawing\" width=\"350\"/>\n",
    "\n",
    "#### The image exemplifies the intersection and collaborative synergy among three pivotal technological domains:\n",
    "\n",
    "1. **Artificial Intelligence and Machine Learning**: AI and ML can leverage Big Alternative Data for developing more sophisticated models and algorithms, which can lead to more accurate predictions and better decision-making.\n",
    "\n",
    "2. **Big Data**: provides the raw information that allows AI models to make informed decisions. This data can be processed and analyzed using machine learning algorithms to uncover insights that were previously not possible with traditional data sources, enhancing research and development across various fields.\n",
    "\n",
    "3. **High-Performance Computing (HPC)**: HPC provides the necessary computational power to handle the vast amounts of data and complex calculations required by AI and ML, thus speeding up research and enabling more complex simulations. \n",
    "\n",
    "The integration of Big Data, AI, and HPC creates a powerful ecosystem for advanced analytics, enabling the tackling of intricate problems and the extraction of profound insights. This triad fosters the capability for real-time analysis and decision-making, revolutionizing sectors such as finance, healthcare, and transportation by improving efficiency and outcomes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* #### The concept of Big Data is often associated with the (?) V's:\n",
    "\n",
    "<img src=\"images/bigdatavs.jpg\" alt=\"Drawing\" width=\"500\"/>\n",
    "\n",
    "\n",
    "1. **Visualization**: Represents the importance of presenting data in a manner that is easily and immediately understandable.\n",
    "\n",
    "2. **Velocity**: Refers to the speed at which data is generated, processed, and analyzed.\n",
    "\n",
    "3. **Variety**: Indicates the different types of data (structured, unstructured, and semi-structured) that are available for analysis.\n",
    "\n",
    "4. **Variability**: Suggests that data flows can be highly inconsistent with periodic peaks.\n",
    "\n",
    "5. **Volume**: Points to the vast amounts of data generated from various sources.\n",
    "\n",
    "6. **Vulnerability**: Highlights the security concerns and risks associated with managing and storing large quantities of data.\n",
    "\n",
    "7. **Validity**: Concerns the accuracy and correctness of data for the intended use.\n",
    "\n",
    "8. **Volatility**: Describes how long data is valid and how quickly it becomes outdated.\n",
    "\n",
    "9. **Veracity**: Addresses the quality and reliability of data.\n",
    "\n",
    "10. **Value**: Emphasizes the worth of the data being collected and how it can be turned into a valuable resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Big Data?\n",
    "\n",
    "<img src=\"images/model.png\" alt=\"Drawing\"/>\n",
    "\n",
    "Big Data is a term that varies in definition depending on the context and the person you're asking. In the field of econometrics, the concept of Big Data can be framed with respect to the dimensions of the dataset, namely the number of variables and observations.\n",
    "\n",
    "* **Wild data** (unstructured, constract with Census surveys, or twitter)\n",
    "\n",
    "* **Wide data** (a.k.a Larg-P data because p>>N)\n",
    "\n",
    "* **Long data** (a.k.a Large-N data because N is very large and may not even fit onto a single hard drive)\n",
    "\n",
    "* **Complex model** (a.k.a Large-Theta because model/algorithm has many parameters)\n",
    "\n",
    "<img src=\"images/mlp.png\" alt=\"Drawing\" width=\"500\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pillars of Big Data\n",
    "\n",
    "* Foundation of basic calculus, linear algebra, probability analysis, and neumerical optimization)\n",
    "\n",
    "* Programming (for automation of data collection, manipulation, cleaning, visualization, and modeling)\n",
    "\n",
    "* Visualization & exploration\n",
    "\n",
    "* Machine learning (to capture nonlinearity and non normality in data, to compress data, and prediction)\n",
    "\n",
    "* Causal inference (to be able to make policy prescription)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Impact of the Curse of Dimensionality on Regression Analysis\n",
    "\n",
    "Before delving into the specific impacts, let's establish the general equations and notations used in regression analysis:\n",
    "\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<center> $ \\mathbf{y} = \\mathbf{X}\\beta + \\boldsymbol{\\varepsilon} $ </center>\n",
    "<br>\n",
    "<center> $ \\boldsymbol{\\varepsilon} \\sim \\mathcal{N}(\\mathbf{0}, \\sigma^2\\mathbf{I}) $ </center>\n",
    "<br>\n",
    "<center> $ L_{OLS}(\\hat{\\beta}) = \\sum_{i=1}^n (y_i - \\mathbf{x}_i^T \\hat{\\beta})^2 = \\|\\mathbf{y} - \\mathbf{X}\\beta\\|^2 $ </center>\n",
    "<br>\n",
    "<center> $ \\hat{\\beta}_{OLS} = (\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{y} $ </center>\n",
    "<br>\n",
    "<center> $ \\text{Bias} (\\hat{\\beta}_{OLS}) = \\mathbb{E}[\\hat{\\beta}_{OLS}] - \\beta $ </center>\n",
    "<br>\n",
    "<center> $ \\text{Var} (\\hat{\\beta}_{OLS}) = \\sigma^2(\\mathbf{X}^T\\mathbf{X})^{-1} $ </center>\n",
    "<br>\n",
    "<center> $ \\sigma^2 = \\frac{\\boldsymbol{\\varepsilon}^T \\boldsymbol{\\varepsilon}}{n - p - 1} $ </center>\n",
    "<br>\n",
    "\n",
    "### Overfitting:\n",
    "- In high-dimensional settings where $p$ is large, the OLS model $ \\mathbf{y} = \\mathbf{X}\\beta + \\boldsymbol{\\varepsilon} $ can fit the noise in the data as well as the signal.\n",
    "- Overfitting is indicated when $ L_{OLS}(\\hat{\\beta}) $ is small for the training data but large for new data.\n",
    "- When $p$ approaches or exceeds $n$, $(\\mathbf{X}^T\\mathbf{X})^{-1}$ can become unstable or non-invertible, leading to a model that perfectly fits the training data but fails to generalize.\n",
    "\n",
    "\n",
    "### Collinearity:\n",
    "- Collinearity refers to high correlation among predictor variables in matrix $\\mathbf{X}$.\n",
    "- This leads to instability in $(\\mathbf{X}^T\\mathbf{X})^{-1}$, reflected in inflated values in the diagonal of $\\text{Var}(\\hat{\\beta}_{OLS})$, indicating high variance of the coefficient estimates.\n",
    "- Collinearity can make it difficult to discern the individual impact of predictors on $\\mathbf{y}$.\n",
    "\n",
    "\n",
    "### Sample Size and Power Issues:\n",
    "- Adequate sample size ($n$) is essential for reliable estimation. However, as $p$ increases, more data is required to maintain statistical power.\n",
    "- The power of a test is the probability that the test correctly rejects a false null hypothesis (Type II error). In high dimensions, the power to detect true effects is diminished due to the increased variance in coefficient estimates.\n",
    "- Type I error is the incorrect rejection of a true null hypothesis, while Type II error is failing to reject a false null hypothesis. High dimensionality can inflate both Type I and Type II errors, especially when $p$ is large relative to $n$.\n",
    "\n",
    "\n",
    "### Unreliable Estimation of Variance:\n",
    "- In high-dimensional spaces, $\\text{Var}(\\hat{\\beta}_{OLS}) = \\sigma^2(\\mathbf{X}^T\\mathbf{X})^{-1}$ can become unreliable due to the increased variance of the estimator.\n",
    "- When $p$ is close to or greater than $n$, the matrix $(\\mathbf{X}^T\\mathbf{X})$ is often of low rank and thus non-invertible or poorly conditioned, which significantly affects the reliability of the OLS estimator.\n",
    "\n",
    "\n",
    "### Estimation of Error Variance ($\\sigma^2$):\n",
    "- The accurate estimation of $\\sigma^2$ is crucial for inference. The formula $\\sigma^2 = \\frac{\\boldsymbol{\\varepsilon}^T \\boldsymbol{\\varepsilon}}{n - p - 1}$ shows the dependency on $n$ and $p$.\n",
    "- When $p$ is large relative to $n$, the denominator $n-p-1$ becomes small, leading to an overestimation of $\\sigma^2$.\n",
    "\n",
    "\n",
    "The curse of dimensionality significantly impacts regression analysis by increasing the potential for overfitting, causing issues with collinearity, necessitating larger sample sizes for statistical power, and leading to unreliable estimation of the variance of the OLS estimator and error variance $\\sigma^2$. Techniques like regularization, dimensionality reduction, and careful model selection are crucial in high-dimensional settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Big Data: FROM HYPOTHESIS TESTING TO MACHINE LEARNING\n",
    "\n",
    "* Machine learning (ML) allows researchers to analyze data in novel ways. Computers today can process multiple sets of data in little time and, with the correct classification sets, recognize highly complex patterns among them. \n",
    "\n",
    "* Designed to simulate the interactions of biological neurons, “deep learning” uses artificial neural networks to discern features in successive layers of data while iterating on previously recognized trends. \n",
    "\n",
    "\n",
    "### Econometrics vs. Machine Learning\n",
    "\n",
    "#### Goal of econometrics\n",
    "\n",
    "* \"the goal of econometrics is to find β hat\" where here we mean β hat to be the causal impact of X on y\n",
    "\n",
    "* The primary statistical concern of econometrics is sampling error. In other words, the goal is to quantify the uncertainty around β hat due to randomness in the sampling of the population. \n",
    "\n",
    "* The goal is to make counterfactual predictions.\n",
    "\n",
    "        What would happen to Amazon's profits if it changed its website layout?\n",
    "\n",
    "We don't get to observe the world under these alternative policies, so we can't simply find the answers in the data. Knowing the counterfactual requires being able to measure a causal effect. Being able to measure a causal effect requires making assumptions. That's what economics is all about!\n",
    "\n",
    "\n",
    "#### Goal of machine learning\n",
    "\n",
    "* In contrast, the goal of machine learning is to come up with the best possible out-of-sample prediction. \n",
    "\n",
    "* We refer to this as the primary concern of machine learning being \"y hat\"\n",
    "\n",
    "* The goal is to make sure that the best prediction is had by tuning and validating many different kinds of models. This is what cross-validation is all about, and it is what machine learning practitioners obsess about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unlocking New Dimensions: The Pivotal Role of Alternative Data\n",
    "\n",
    "- **Alternative data**: refers to non-traditional data sources that can provide additional insights beyond what's available through conventional data. Its advantages are frequency and near-real-time data, accuracy and objectiveness. Its disadvantages are the fact that the indicators available are merely proxies for what policymakers are interested in and need for policy design. Here are some examples of alternative data and variables:\n",
    "\n",
    "    - **Textual data such as News Headlines**\n",
    "    - **Digital footprints from social media**\n",
    "    - **Mobile phone data**\n",
    "    - **Satellite Imagery** nighttime light measures, or luminosity, as proxies for economic activity and population distribution\n",
    "    - **Search Engine Trends**\n",
    "    - **Environmental, Social, and Governance (ESG) Data**\n",
    "\n",
    "\n",
    "<img src=\"images/text.png\" alt=\"Drawing\" width=\"400\"/>\n",
    "<img src=\"images/satt2.jpg\" alt=\"Drawing\" width=\"400\"/>\n",
    "<img src=\"images/satt.jpg\" alt=\"Drawing\" width=\"400\"/>\n",
    "<img src=\"images/satt3.jpg\" alt=\"Drawing\" width=\"400\"/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}