im_v2/common/notebooks/Master_raw_data_gallery.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "85907be2",
   "metadata": {},
   "source": [
    "# Master raw data gallery"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8a30f92",
   "metadata": {},
   "source": [
    "This notebook showcases locations and basic structure of raw data from:\n",
    "\n",
    "- S3 (parquet datasets)\n",
    "- IM DB (Postgres)\n",
    "\n",
    "The secondary purpose is to provide a guide on how to use `RawDataReader`\n",
    "\n",
    "## Specs\n",
    "- This notebook:\n",
    "  - is a catalog of all the datasets that exist in the our system\n",
    "  - shows how to load data using our low-level functions or specific API for specific datasets\n",
    "  - shows how a snippet of the data looks like (for this we want to load the minimal amount of data)\n",
    "  - doesn't compute any statistics\n",
    "  - should be quickly to execute, like < 1min, so we can run it in the unit tests\n",
    "\n",
    "## Life cycle\n",
    "- Any time a new dataset is added (e.g., in real-time DB, Parquet) we add some information on how to load it and how it looks like\n",
    "- In general we try not to delete any data but we only add data loaders\n",
    "\n",
    "## Monster dataset matrix spreadsheet\n",
    "\n",
    "The gallery should match 1-to-1 with the dataset matrix\n",
    "\n",
    "https://docs.google.com/spreadsheets/d/13Vyrxs9Eg-C6y91XIogLHi4A1_AFK7_KCF2KEnnxYv0/edit#gid=521856766"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102c57e9",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "01579d41",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-09T17:17:21.462657Z",
     "start_time": "2024-07-09T17:17:15.491876Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/core/statistics/random_samples.py:12: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n",
      "\u001b[33mWARNING\u001b[0m: No module named 'gluonts'\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "import logging\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import helpers.hdbg as hdbg\n",
    "import helpers.henv as henv\n",
    "import helpers.hpandas as hpandas\n",
    "import helpers.hprint as hprint\n",
    "import im_v2.common.data.client.im_raw_data_client as imvcdcimrdc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0578f2f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-09T17:17:22.083171Z",
     "start_time": "2024-07-09T17:17:21.468722Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0mWARNING: Running in Jupyter\n",
      "INFO  > cmd='/venv/lib/python3.9/site-packages/ipykernel_launcher.py -f /home/.local/share/jupyter/runtime/kernel-223cb510-05de-4297-83b9-09af6258e474.json'\n",
      "INFO  # Git\n",
      "  branch_name='CmampTask8922_Update_dataset_matrix_and_gallery_for_crypto.com_datasets'\n",
      "  hash='3d9f7c7da'\n",
      "  # Last commits:\n",
      "    * 3d9f7c7da pavolrabatin removed `prod-im-db` from terraform configuration (#8938)         (   5 hours ago) Tue Jul 9 12:15:48 2024  (HEAD -> CmampTask8922_Update_dataset_matrix_and_gallery_for_crypto.com_datasets, origin/master, origin/HEAD, origin/CmampTask8922_Update_dataset_matrix_and_gallery_for_crypto.com_datasets, origin/CmTask8946_Remove_obsolete_code_from_trading_DAGs, origin/CmTask8759_Organize_Broker_TradingOps_doc_oms_order_, master)\n",
      "    * f86b25654 pavolrabatin resource added (#8905)                                            (   7 hours ago) Tue Jul 9 09:50:30 2024           \n",
      "    * 3d921ae5f Shayan   AlertManager optimization (#8912)                                 (  22 hours ago) Mon Jul 8 19:36:26 2024  (origin/CmTask8759_Organize_Broker_TradingOps_doc_oms_child_order_quantity_computer)\n",
      "# Machine info\n",
      "  system=Linux\n",
      "  node name=bc94efe6e371\n",
      "  release=5.15.0-1058-aws\n",
      "  version=#64~20.04.1-Ubuntu SMP Tue Apr 9 11:12:27 UTC 2024\n",
      "  machine=x86_64\n",
      "  processor=x86_64\n",
      "  cpu count=8\n",
      "  cpu freq=scpufreq(current=2499.998, min=0.0, max=0.0)\n",
      "  memory=svmem(total=33280274432, available=14685626368, percent=55.9, used=18115727360, free=2098094080, active=9213751296, inactive=17991057408, buffers=2147794944, cached=10918658048, shared=1253376, slab=3368349696)\n",
      "  disk usage=sdiskusage(total=218506772480, used=93192990720, free=125297004544, percent=42.7)\n",
      "# Packages\n",
      "  python: 3.9.5\n",
      "  cvxopt: 1.3.2\n",
      "  cvxpy: 1.4.2\n",
      "  gluonnlp: ?\n",
      "  gluonts: ?\n",
      "  joblib: 1.3.2\n",
      "  mxnet: ?\n",
      "  numpy: 1.26.0\n",
      "  pandas: 2.1.1\n",
      "  pyarrow: 15.0.0\n",
      "  scipy: 1.11.3\n",
      "  seaborn: 0.13.0\n",
      "  sklearn: 1.3.1\n",
      "  statsmodels: 0.14.0\n"
     ]
    }
   ],
   "source": [
    "hdbg.init_logger(verbosity=logging.INFO)\n",
    "log_level = logging.INFO\n",
    "\n",
    "_LOG = logging.getLogger(__name__)\n",
    "\n",
    "_LOG.info(\"%s\", henv.get_system_signature()[0])\n",
    "\n",
    "hprint.config_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2906354a",
   "metadata": {},
   "source": [
    "# Active datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cec6ac96",
   "metadata": {},
   "source": [
    "## realtime.airflow.resampled_1min.postgres.bid_ask.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0d699cc1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-17T18:56:51.438946Z",
     "start_time": "2024-05-17T18:56:50.793402Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Unable to fetch DB credentials from environment variables: \n",
      "\t'POSTGRES_HOST'\n",
      "\tAttempting env file method.\n",
      "INFO  Unable to fetch DB credentials from env file: \n",
      "\t\n",
      "################################################################################\n",
      "* Failed assertion *\n",
      "File '/app/amp/im_v2/devops/env/preprod.im_db_config.env' doesn't exist\n",
      "################################################################################\n",
      "\n",
      "\tAttempting AWS SecretsManager method.\n",
      "INFO  Fetching secret: preprod.im_data_db\n",
      "INFO  Created preprod DB connection: \n",
      " None\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_resampled_1min` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_resampled_1min` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_futures_resampled_1min WHERE exchange_id = 'binance' LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>ask_price_close</th>\n",
       "      <th>ask_price_high</th>\n",
       "      <th>ask_price_low</th>\n",
       "      <th>ask_price_mean</th>\n",
       "      <th>ask_price_open</th>\n",
       "      <th>ask_size_close</th>\n",
       "      <th>ask_size_max</th>\n",
       "      <th>ask_size_mean</th>\n",
       "      <th>ask_size_min</th>\n",
       "      <th>ask_size_open</th>\n",
       "      <th>bid_price_close</th>\n",
       "      <th>bid_price_high</th>\n",
       "      <th>bid_price_low</th>\n",
       "      <th>bid_price_mean</th>\n",
       "      <th>bid_price_open</th>\n",
       "      <th>bid_size_close</th>\n",
       "      <th>bid_size_max</th>\n",
       "      <th>bid_size_mean</th>\n",
       "      <th>bid_size_min</th>\n",
       "      <th>bid_size_open</th>\n",
       "      <th>bid_ask_midpoint_open</th>\n",
       "      <th>half_spread_open</th>\n",
       "      <th>log_size_imbalance_open</th>\n",
       "      <th>bid_ask_midpoint_close</th>\n",
       "      <th>half_spread_close</th>\n",
       "      <th>log_size_imbalance_close</th>\n",
       "      <th>bid_ask_midpoint_min</th>\n",
       "      <th>half_spread_min</th>\n",
       "      <th>log_size_imbalance_min</th>\n",
       "      <th>bid_ask_midpoint_max</th>\n",
       "      <th>half_spread_max</th>\n",
       "      <th>log_size_imbalance_max</th>\n",
       "      <th>bid_ask_midpoint_mean</th>\n",
       "      <th>half_spread_mean</th>\n",
       "      <th>log_size_imbalance_mean</th>\n",
       "      <th>bid_ask_midpoint_var_100ms</th>\n",
       "      <th>bid_ask_midpoint_autocovar_100ms</th>\n",
       "      <th>log_size_imbalance_var_100ms</th>\n",
       "      <th>log_size_imbalance_autocovar_100ms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>APE_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>1.3705</td>\n",
       "      <td>1.3706</td>\n",
       "      <td>1.3694</td>\n",
       "      <td>1.370104</td>\n",
       "      <td>1.3702</td>\n",
       "      <td>151.0</td>\n",
       "      <td>1365.0</td>\n",
       "      <td>207.435065</td>\n",
       "      <td>4.0</td>\n",
       "      <td>358.0</td>\n",
       "      <td>1.3704</td>\n",
       "      <td>1.3705</td>\n",
       "      <td>1.3693</td>\n",
       "      <td>1.370003</td>\n",
       "      <td>1.3701</td>\n",
       "      <td>769.0</td>\n",
       "      <td>3280.0</td>\n",
       "      <td>667.461039</td>\n",
       "      <td>36.0</td>\n",
       "      <td>354.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>AVAX_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>35.023</td>\n",
       "      <td>35.029</td>\n",
       "      <td>34.974</td>\n",
       "      <td>35.005584</td>\n",
       "      <td>35.015</td>\n",
       "      <td>29.0</td>\n",
       "      <td>402.0</td>\n",
       "      <td>49.987013</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>35.022</td>\n",
       "      <td>35.028</td>\n",
       "      <td>34.973</td>\n",
       "      <td>35.004539</td>\n",
       "      <td>35.014</td>\n",
       "      <td>109.0</td>\n",
       "      <td>427.0</td>\n",
       "      <td>94.857143</td>\n",
       "      <td>1.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>AXS_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>7.081</td>\n",
       "      <td>7.082</td>\n",
       "      <td>7.079</td>\n",
       "      <td>7.080197</td>\n",
       "      <td>7.079</td>\n",
       "      <td>101.0</td>\n",
       "      <td>1558.0</td>\n",
       "      <td>575.223684</td>\n",
       "      <td>10.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>7.08</td>\n",
       "      <td>7.081</td>\n",
       "      <td>7.078</td>\n",
       "      <td>7.079197</td>\n",
       "      <td>7.078</td>\n",
       "      <td>213.0</td>\n",
       "      <td>1584.0</td>\n",
       "      <td>384.493421</td>\n",
       "      <td>69.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>0.6273</td>\n",
       "      <td>0.628</td>\n",
       "      <td>0.627</td>\n",
       "      <td>0.627445</td>\n",
       "      <td>0.6278</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7872.0</td>\n",
       "      <td>2159.821192</td>\n",
       "      <td>16.0</td>\n",
       "      <td>857.0</td>\n",
       "      <td>0.6271</td>\n",
       "      <td>0.6278</td>\n",
       "      <td>0.6269</td>\n",
       "      <td>0.627314</td>\n",
       "      <td>0.6276</td>\n",
       "      <td>101.0</td>\n",
       "      <td>1762.0</td>\n",
       "      <td>397.357616</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1302.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>DOGE_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>0.07946</td>\n",
       "      <td>0.07947</td>\n",
       "      <td>0.07946</td>\n",
       "      <td>0.079466</td>\n",
       "      <td>0.07947</td>\n",
       "      <td>5153503.0</td>\n",
       "      <td>5548917.0</td>\n",
       "      <td>2366407.155844</td>\n",
       "      <td>60133.0</td>\n",
       "      <td>323984.0</td>\n",
       "      <td>0.07945</td>\n",
       "      <td>0.07946</td>\n",
       "      <td>0.07945</td>\n",
       "      <td>0.079456</td>\n",
       "      <td>0.07946</td>\n",
       "      <td>621673.0</td>\n",
       "      <td>738487.0</td>\n",
       "      <td>379849.584416</td>\n",
       "      <td>41907.0</td>\n",
       "      <td>614888.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1707326160000</td>\n",
       "      <td>DOT_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>2024-02-07 17:16:01.044261+00:00</td>\n",
       "      <td>6.87</td>\n",
       "      <td>6.871</td>\n",
       "      <td>6.863</td>\n",
       "      <td>6.867229</td>\n",
       "      <td>6.864</td>\n",
       "      <td>1889.0</td>\n",
       "      <td>3865.9</td>\n",
       "      <td>1477.835294</td>\n",
       "      <td>106.5</td>\n",
       "      <td>1747.8</td>\n",
       "      <td>6.869</td>\n",
       "      <td>6.87</td>\n",
       "      <td>6.862</td>\n",
       "      <td>6.866229</td>\n",
       "      <td>6.863</td>\n",
       "      <td>73.3</td>\n",
       "      <td>3367.5</td>\n",
       "      <td>1395.09085</td>\n",
       "      <td>73.3</td>\n",
       "      <td>1976.3</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp currency_pair exchange_id level end_download_timestamp               knowledge_timestamp ask_price_close ask_price_high ask_price_low ask_price_mean ask_price_open ask_size_close ask_size_max   ask_size_mean ask_size_min ask_size_open bid_price_close bid_price_high bid_price_low bid_price_mean bid_price_open bid_size_close bid_size_max  bid_size_mean bid_size_min bid_size_open bid_ask_midpoint_open half_spread_open log_size_imbalance_open bid_ask_midpoint_close half_spread_close log_size_imbalance_close bid_ask_midpoint_min half_spread_min log_size_imbalance_min bid_ask_midpoint_max half_spread_max log_size_imbalance_max bid_ask_midpoint_mean half_spread_mean log_size_imbalance_mean bid_ask_midpoint_var_100ms bid_ask_midpoint_autocovar_100ms log_size_imbalance_var_100ms log_size_imbalance_autocovar_100ms\n",
       "0  1707326160000      APE_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00          1.3705         1.3706        1.3694       1.370104         1.3702          151.0       1365.0      207.435065          4.0         358.0          1.3704         1.3705        1.3693       1.370003         1.3701          769.0       3280.0     667.461039         36.0         354.0                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None\n",
       "1  1707326160000     AVAX_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00          35.023         35.029        34.974      35.005584         35.015           29.0        402.0       49.987013          1.0           7.0          35.022         35.028        34.973      35.004539         35.014          109.0        427.0      94.857143          1.0          99.0                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None\n",
       "2  1707326160000      AXS_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00           7.081          7.082         7.079       7.080197          7.079          101.0       1558.0      575.223684         10.0         100.0            7.08          7.081         7.078       7.079197          7.078          213.0       1584.0     384.493421         69.0          69.0                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None\n",
       "             ...           ...         ...   ...                    ...                               ...             ...            ...           ...            ...            ...            ...          ...             ...          ...           ...             ...            ...           ...            ...            ...            ...          ...            ...          ...           ...                   ...              ...                     ...                    ...               ...                      ...                  ...             ...                    ...                  ...             ...                    ...                   ...              ...                     ...                        ...                              ...                          ...                                ...\n",
       "7  1707326160000      CTK_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00          0.6273          0.628         0.627       0.627445         0.6278           22.0       7872.0     2159.821192         16.0         857.0          0.6271         0.6278        0.6269       0.627314         0.6276          101.0       1762.0     397.357616         16.0        1302.0                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None\n",
       "8  1707326160000     DOGE_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00         0.07946        0.07947       0.07946       0.079466        0.07947      5153503.0    5548917.0  2366407.155844      60133.0      323984.0         0.07945        0.07946       0.07945       0.079456        0.07946       621673.0     738487.0  379849.584416      41907.0      614888.0                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None\n",
       "9  1707326160000      DOT_USDT     binance     1                   None  2024-02-07 17:16:01.044261+00:00            6.87          6.871         6.863       6.867229          6.864         1889.0       3865.9     1477.835294        106.5        1747.8           6.869           6.87         6.862       6.866229          6.863           73.3       3367.5     1395.09085         73.3        1976.3                  None             None                    None                   None              None                     None                 None            None                   None                 None            None                   None                  None             None                    None                       None                             None                         None                               None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.resampled_1min.postgres.bid_ask.futures.v8.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763c6e8c",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "75b8c429",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-17T18:57:08.886090Z",
     "start_time": "2024-05-17T18:57:05.399488Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_futures_raw WHERE exchange_id = 'binance' LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1711689552058</td>\n",
       "      <td>14.569</td>\n",
       "      <td>3572.63</td>\n",
       "      <td>39.939</td>\n",
       "      <td>3572.64</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.228817+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1711689552012</td>\n",
       "      <td>0.487</td>\n",
       "      <td>70441.1</td>\n",
       "      <td>3.077</td>\n",
       "      <td>70441.2</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.230280+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1711689551961</td>\n",
       "      <td>6459.0</td>\n",
       "      <td>0.703</td>\n",
       "      <td>4362.0</td>\n",
       "      <td>0.7031</td>\n",
       "      <td>SAND_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.231349+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1711689552041</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0383</td>\n",
       "      <td>341.0</td>\n",
       "      <td>2.0384</td>\n",
       "      <td>APE_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.234426+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1711689552061</td>\n",
       "      <td>775.0</td>\n",
       "      <td>0.9971</td>\n",
       "      <td>23973.0</td>\n",
       "      <td>0.9972</td>\n",
       "      <td>MATIC_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.234926+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1711689552002</td>\n",
       "      <td>1210.7</td>\n",
       "      <td>3.504</td>\n",
       "      <td>3219.8</td>\n",
       "      <td>3.505</td>\n",
       "      <td>DYDX_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-03-29 05:19:12.235282+00:00</td>\n",
       "      <td>2024-03-29 05:19:12.681645+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1711689552058   14.569   3572.63   39.939   3572.64      ETH_USDT     binance     1  2024-03-29 05:19:12.228817+00:00  2024-03-29 05:19:12.681645+00:00\n",
       "1  1711689552012    0.487   70441.1    3.077   70441.2      BTC_USDT     binance     1  2024-03-29 05:19:12.230280+00:00  2024-03-29 05:19:12.681645+00:00\n",
       "2  1711689551961   6459.0     0.703   4362.0    0.7031     SAND_USDT     binance     1  2024-03-29 05:19:12.231349+00:00  2024-03-29 05:19:12.681645+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1711689552041      3.0    2.0383    341.0    2.0384      APE_USDT     binance     1  2024-03-29 05:19:12.234426+00:00  2024-03-29 05:19:12.681645+00:00\n",
       "8  1711689552061    775.0    0.9971  23973.0    0.9972    MATIC_USDT     binance     1  2024-03-29 05:19:12.234926+00:00  2024-03-29 05:19:12.681645+00:00\n",
       "9  1711689552002   1210.7     3.504   3219.8     3.505     DYDX_USDT     binance     1  2024-03-29 05:19:12.235282+00:00  2024-03-29 05:19:12.681645+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v8.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8292b4",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v8.binance.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "90005e51",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:15.124846Z",
     "start_time": "2024-05-16T19:43:15.090133Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `binance_bid_ask_futures_raw` DB table\n",
      "INFO  Enabled connection to the `binance_bid_ask_futures_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM binance_bid_ask_futures_raw WHERE exchange_id = 'binance' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888594810</td>\n",
       "      <td>0.109</td>\n",
       "      <td>55.23</td>\n",
       "      <td>7.974</td>\n",
       "      <td>55.24</td>\n",
       "      <td>COMP_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.979093+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888594807</td>\n",
       "      <td>2907.0</td>\n",
       "      <td>0.5492</td>\n",
       "      <td>2561.0</td>\n",
       "      <td>0.5493</td>\n",
       "      <td>SEI_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.977667+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888594807</td>\n",
       "      <td>5.0</td>\n",
       "      <td>159.237</td>\n",
       "      <td>123.0</td>\n",
       "      <td>159.238</td>\n",
       "      <td>SOL_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.978062+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888594798</td>\n",
       "      <td>521.0</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>7665.0</td>\n",
       "      <td>0.3355</td>\n",
       "      <td>CHR_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.978995+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888594791</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.467</td>\n",
       "      <td>797.0</td>\n",
       "      <td>6.468</td>\n",
       "      <td>RUNE_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.977835+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888594786</td>\n",
       "      <td>378.0</td>\n",
       "      <td>0.3793</td>\n",
       "      <td>10202.0</td>\n",
       "      <td>0.3794</td>\n",
       "      <td>1INCH_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:14.979225+00:00</td>\n",
       "      <td>2024-05-16 19:43:14.992051+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888594810    0.109     55.23    7.974     55.24     COMP_USDT     binance     1  2024-05-16 19:43:14.979093+00:00  2024-05-16 19:43:14.992051+00:00\n",
       "1  1715888594807   2907.0    0.5492   2561.0    0.5493      SEI_USDT     binance     1  2024-05-16 19:43:14.977667+00:00  2024-05-16 19:43:14.992051+00:00\n",
       "2  1715888594807      5.0   159.237    123.0   159.238      SOL_USDT     binance     1  2024-05-16 19:43:14.978062+00:00  2024-05-16 19:43:14.992051+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888594798    521.0    0.3354   7665.0    0.3355      CHR_USDT     binance     1  2024-05-16 19:43:14.978995+00:00  2024-05-16 19:43:14.992051+00:00\n",
       "8  1715888594791      2.0     6.467    797.0     6.468     RUNE_USDT     binance     1  2024-05-16 19:43:14.977835+00:00  2024-05-16 19:43:14.992051+00:00\n",
       "9  1715888594786    378.0    0.3793  10202.0    0.3794    1INCH_USDT     binance     1  2024-05-16 19:43:14.979225+00:00  2024-05-16 19:43:14.992051+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v8.binance.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f7c6520",
   "metadata": {},
   "source": [
    "## realtime.airflow.resampled_1min.postgres.bid_ask.futures.v8.binance.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c2cfc05a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:15.195139Z",
     "start_time": "2024-05-16T19:43:15.126795Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `binance_bid_ask_futures_resampled_1min` DB table\n",
      "INFO  Enabled connection to the `binance_bid_ask_futures_resampled_1min` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM binance_bid_ask_futures_resampled_1min WHERE exchange_id = 'binance' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size_open</th>\n",
       "      <th>bid_size_close</th>\n",
       "      <th>bid_size_min</th>\n",
       "      <th>bid_size_max</th>\n",
       "      <th>bid_size_mean</th>\n",
       "      <th>bid_price_open</th>\n",
       "      <th>bid_price_close</th>\n",
       "      <th>bid_price_high</th>\n",
       "      <th>bid_price_low</th>\n",
       "      <th>bid_price_mean</th>\n",
       "      <th>ask_size_open</th>\n",
       "      <th>ask_size_close</th>\n",
       "      <th>ask_size_min</th>\n",
       "      <th>ask_size_max</th>\n",
       "      <th>ask_size_mean</th>\n",
       "      <th>ask_price_open</th>\n",
       "      <th>ask_price_close</th>\n",
       "      <th>ask_price_high</th>\n",
       "      <th>ask_price_low</th>\n",
       "      <th>ask_price_mean</th>\n",
       "      <th>bid_ask_midpoint_open</th>\n",
       "      <th>half_spread_open</th>\n",
       "      <th>log_size_imbalance_open</th>\n",
       "      <th>bid_ask_midpoint_close</th>\n",
       "      <th>half_spread_close</th>\n",
       "      <th>log_size_imbalance_close</th>\n",
       "      <th>bid_ask_midpoint_min</th>\n",
       "      <th>half_spread_min</th>\n",
       "      <th>log_size_imbalance_min</th>\n",
       "      <th>bid_ask_midpoint_max</th>\n",
       "      <th>half_spread_max</th>\n",
       "      <th>log_size_imbalance_max</th>\n",
       "      <th>bid_ask_midpoint_mean</th>\n",
       "      <th>half_spread_mean</th>\n",
       "      <th>log_size_imbalance_mean</th>\n",
       "      <th>bid_ask_midpoint_var_100ms</th>\n",
       "      <th>bid_ask_midpoint_autocovar_100ms</th>\n",
       "      <th>log_size_imbalance_var_100ms</th>\n",
       "      <th>log_size_imbalance_autocovar_100ms</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>646.0</td>\n",
       "      <td>440.4</td>\n",
       "      <td>87.4</td>\n",
       "      <td>13842.4</td>\n",
       "      <td>3303.668447</td>\n",
       "      <td>0.4947</td>\n",
       "      <td>0.495</td>\n",
       "      <td>0.4951</td>\n",
       "      <td>0.4946</td>\n",
       "      <td>0.494913</td>\n",
       "      <td>4635.6</td>\n",
       "      <td>5814.2</td>\n",
       "      <td>87.4</td>\n",
       "      <td>7981.8</td>\n",
       "      <td>4087.821536</td>\n",
       "      <td>0.4948</td>\n",
       "      <td>0.4951</td>\n",
       "      <td>0.4952</td>\n",
       "      <td>0.4947</td>\n",
       "      <td>0.495013</td>\n",
       "      <td>0.49475</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>-1.970721</td>\n",
       "      <td>0.49505</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>-2.580375</td>\n",
       "      <td>0.49465</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>-4.514424</td>\n",
       "      <td>0.49515</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>5.064996</td>\n",
       "      <td>0.494963</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>-1.080507</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3298.716855</td>\n",
       "      <td>3189.009827</td>\n",
       "      <td>ZRX_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>11392.0</td>\n",
       "      <td>169869.0</td>\n",
       "      <td>11392.0</td>\n",
       "      <td>378240.0</td>\n",
       "      <td>209576.008403</td>\n",
       "      <td>0.02309</td>\n",
       "      <td>0.02309</td>\n",
       "      <td>0.0231</td>\n",
       "      <td>0.02308</td>\n",
       "      <td>0.023087</td>\n",
       "      <td>143692.0</td>\n",
       "      <td>29713.0</td>\n",
       "      <td>1342.0</td>\n",
       "      <td>405076.0</td>\n",
       "      <td>45770.117647</td>\n",
       "      <td>0.0231</td>\n",
       "      <td>0.0231</td>\n",
       "      <td>0.02311</td>\n",
       "      <td>0.02309</td>\n",
       "      <td>0.023097</td>\n",
       "      <td>0.023095</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>-2.534761</td>\n",
       "      <td>0.023095</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>1.743443</td>\n",
       "      <td>0.023085</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>-3.571163</td>\n",
       "      <td>0.023105</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>5.419823</td>\n",
       "      <td>0.023092</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>1.340125</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2312.307205</td>\n",
       "      <td>2234.818069</td>\n",
       "      <td>ZIL_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>142.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>315.0</td>\n",
       "      <td>118.144876</td>\n",
       "      <td>1.6222</td>\n",
       "      <td>1.6231</td>\n",
       "      <td>1.6232</td>\n",
       "      <td>1.6216</td>\n",
       "      <td>1.622619</td>\n",
       "      <td>106.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1056.0</td>\n",
       "      <td>159.540636</td>\n",
       "      <td>1.6226</td>\n",
       "      <td>1.6234</td>\n",
       "      <td>1.6238</td>\n",
       "      <td>1.6218</td>\n",
       "      <td>1.623007</td>\n",
       "      <td>1.6224</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.292388</td>\n",
       "      <td>1.62325</td>\n",
       "      <td>0.00015</td>\n",
       "      <td>-0.922935</td>\n",
       "      <td>1.6217</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>-5.016333</td>\n",
       "      <td>1.6235</td>\n",
       "      <td>0.00035</td>\n",
       "      <td>3.520461</td>\n",
       "      <td>1.622813</td>\n",
       "      <td>0.000194</td>\n",
       "      <td>-0.240552</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1279.487642</td>\n",
       "      <td>1221.724396</td>\n",
       "      <td>ZETA_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>0.9</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>80.0</td>\n",
       "      <td>18.323986</td>\n",
       "      <td>9.356</td>\n",
       "      <td>9.355</td>\n",
       "      <td>9.357</td>\n",
       "      <td>9.354</td>\n",
       "      <td>9.355757</td>\n",
       "      <td>12.7</td>\n",
       "      <td>36.7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.7</td>\n",
       "      <td>20.540541</td>\n",
       "      <td>9.358</td>\n",
       "      <td>9.358</td>\n",
       "      <td>9.358</td>\n",
       "      <td>9.355</td>\n",
       "      <td>9.356988</td>\n",
       "      <td>9.357</td>\n",
       "      <td>0.001</td>\n",
       "      <td>-2.646963</td>\n",
       "      <td>9.3565</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>-0.16879</td>\n",
       "      <td>9.3545</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-4.037186</td>\n",
       "      <td>9.3575</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>3.32569</td>\n",
       "      <td>9.356372</td>\n",
       "      <td>0.000616</td>\n",
       "      <td>-0.240438</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1410.622218</td>\n",
       "      <td>1341.699652</td>\n",
       "      <td>XVS_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>302028.0</td>\n",
       "      <td>26761.0</td>\n",
       "      <td>3605.0</td>\n",
       "      <td>421352.0</td>\n",
       "      <td>41696.846416</td>\n",
       "      <td>0.005886</td>\n",
       "      <td>0.005881</td>\n",
       "      <td>0.005886</td>\n",
       "      <td>0.005877</td>\n",
       "      <td>0.005882</td>\n",
       "      <td>135405.0</td>\n",
       "      <td>19169.0</td>\n",
       "      <td>1276.0</td>\n",
       "      <td>291028.0</td>\n",
       "      <td>36328.090444</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>0.005882</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>0.005878</td>\n",
       "      <td>0.005883</td>\n",
       "      <td>0.005886</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.802249</td>\n",
       "      <td>0.005881</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.333651</td>\n",
       "      <td>0.005878</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-2.819428</td>\n",
       "      <td>0.005886</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>3.612037</td>\n",
       "      <td>0.005882</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.082351</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>991.568682</td>\n",
       "      <td>868.036853</td>\n",
       "      <td>XVG_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888580000</td>\n",
       "      <td>16919.6</td>\n",
       "      <td>18563.0</td>\n",
       "      <td>818.6</td>\n",
       "      <td>30852.6</td>\n",
       "      <td>10217.447466</td>\n",
       "      <td>0.917</td>\n",
       "      <td>0.917</td>\n",
       "      <td>0.918</td>\n",
       "      <td>0.916</td>\n",
       "      <td>0.917316</td>\n",
       "      <td>10159.4</td>\n",
       "      <td>1751.5</td>\n",
       "      <td>1291.5</td>\n",
       "      <td>62122.8</td>\n",
       "      <td>22197.377703</td>\n",
       "      <td>0.918</td>\n",
       "      <td>0.918</td>\n",
       "      <td>0.919</td>\n",
       "      <td>0.917</td>\n",
       "      <td>0.918319</td>\n",
       "      <td>0.9175</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.510073</td>\n",
       "      <td>0.9175</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>2.360698</td>\n",
       "      <td>0.9165</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-4.089123</td>\n",
       "      <td>0.9185</td>\n",
       "      <td>0.001</td>\n",
       "      <td>3.173417</td>\n",
       "      <td>0.917818</td>\n",
       "      <td>0.000502</td>\n",
       "      <td>-0.619486</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2333.741311</td>\n",
       "      <td>2280.146302</td>\n",
       "      <td>XTZ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:43:07.323688+00:00</td>\n",
       "      <td>2024-05-16 19:43:07.417691+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size_open bid_size_close bid_size_min bid_size_max  bid_size_mean bid_price_open bid_price_close bid_price_high bid_price_low bid_price_mean ask_size_open ask_size_close ask_size_min ask_size_max ask_size_mean ask_price_open ask_price_close ask_price_high ask_price_low ask_price_mean bid_ask_midpoint_open half_spread_open log_size_imbalance_open bid_ask_midpoint_close half_spread_close log_size_imbalance_close bid_ask_midpoint_min half_spread_min log_size_imbalance_min bid_ask_midpoint_max half_spread_max log_size_imbalance_max bid_ask_midpoint_mean half_spread_mean log_size_imbalance_mean bid_ask_midpoint_var_100ms bid_ask_midpoint_autocovar_100ms log_size_imbalance_var_100ms log_size_imbalance_autocovar_100ms currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888580000         646.0          440.4         87.4      13842.4    3303.668447         0.4947           0.495         0.4951        0.4946       0.494913        4635.6         5814.2         87.4       7981.8   4087.821536         0.4948          0.4951         0.4952        0.4947       0.495013               0.49475          0.00005               -1.970721                0.49505           0.00005                -2.580375              0.49465         0.00005              -4.514424              0.49515         0.00005               5.064996              0.494963          0.00005               -1.080507                        0.0                              0.0                  3298.716855                        3189.009827      ZRX_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00\n",
       "1  1715888580000       11392.0       169869.0      11392.0     378240.0  209576.008403        0.02309         0.02309         0.0231       0.02308       0.023087      143692.0        29713.0       1342.0     405076.0  45770.117647         0.0231          0.0231        0.02311       0.02309       0.023097              0.023095         0.000005               -2.534761               0.023095          0.000005                 1.743443             0.023085        0.000005              -3.571163             0.023105        0.000005               5.419823              0.023092         0.000005                1.340125                        0.0                              0.0                  2312.307205                        2234.818069      ZIL_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00\n",
       "2  1715888580000         142.0           60.0          6.0        315.0     118.144876         1.6222          1.6231         1.6232        1.6216       1.622619         106.0          151.0          4.0       1056.0    159.540636         1.6226          1.6234         1.6238        1.6218       1.623007                1.6224           0.0002                0.292388                1.62325           0.00015                -0.922935               1.6217         0.00005              -5.016333               1.6235         0.00035               3.520461              1.622813         0.000194               -0.240552                   0.000001                              0.0                  1279.487642                        1221.724396     ZETA_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00\n",
       "             ...           ...            ...          ...          ...            ...            ...             ...            ...           ...            ...           ...            ...          ...          ...           ...            ...             ...            ...           ...            ...                   ...              ...                     ...                    ...               ...                      ...                  ...             ...                    ...                  ...             ...                    ...                   ...              ...                     ...                        ...                              ...                          ...                                ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888580000           0.9           31.0          0.6         80.0      18.323986          9.356           9.355          9.357         9.354       9.355757          12.7           36.7          1.0         36.7     20.540541          9.358           9.358          9.358         9.355       9.356988                 9.357            0.001               -2.646963                 9.3565            0.0015                 -0.16879               9.3545          0.0005              -4.037186               9.3575          0.0015                3.32569              9.356372         0.000616               -0.240438                   0.000007                              0.0                  1410.622218                        1341.699652      XVS_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00\n",
       "8  1715888580000      302028.0        26761.0       3605.0     421352.0   41696.846416       0.005886        0.005881       0.005886      0.005877       0.005882      135405.0        19169.0       1276.0     291028.0  36328.090444       0.005887        0.005882       0.005887      0.005878       0.005883              0.005886         0.000001                0.802249               0.005881          0.000001                 0.333651             0.005878             0.0              -2.819428             0.005886        0.000001               3.612037              0.005882         0.000001                0.082351                        0.0                              0.0                   991.568682                         868.036853      XVG_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00\n",
       "9  1715888580000       16919.6        18563.0        818.6      30852.6   10217.447466          0.917           0.917          0.918         0.916       0.917316       10159.4         1751.5       1291.5      62122.8  22197.377703          0.918           0.918          0.919         0.917       0.918319                0.9175           0.0005                0.510073                 0.9175            0.0005                 2.360698               0.9165          0.0005              -4.089123               0.9185           0.001               3.173417              0.917818         0.000502               -0.619486                   0.000005                              0.0                  2333.741311                        2280.146302      XTZ_USDT     binance     1  2024-05-16 19:43:07.323688+00:00  2024-05-16 19:43:07.417691+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.resampled_1min.postgres.bid_ask.futures.v8.binance.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c970fdab",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.parquet.bid_ask.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7b437c67",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:17.355436Z",
     "start_time": "2024-05-16T19:43:17.352901Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO(Juraj): #CmTask8309"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b228c67",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a1f08786",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:17.986756Z",
     "start_time": "2024-05-16T19:43:17.983892Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO(Juraj): #CmTask8309"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9bd7a42",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.parquet.bid_ask.futures.v8.binance.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0ca025bd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:43.754541Z",
     "start_time": "2024-05-16T19:43:19.541265Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Before from_parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:00:00.143000+00:00</th>\n",
       "      <td>1715817600143</td>\n",
       "      <td>4.022</td>\n",
       "      <td>66175.4</td>\n",
       "      <td>3.215</td>\n",
       "      <td>66175.5</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:00:00.337472+00:00</td>\n",
       "      <td>2024-05-16 00:00:00.350773+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:00:00.365000+00:00</th>\n",
       "      <td>1715817600365</td>\n",
       "      <td>2.226</td>\n",
       "      <td>66175.4</td>\n",
       "      <td>3.755</td>\n",
       "      <td>66175.5</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:00:00.538837+00:00</td>\n",
       "      <td>2024-05-16 00:00:00.552395+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:00:00.570000+00:00</th>\n",
       "      <td>1715817600570</td>\n",
       "      <td>2.623</td>\n",
       "      <td>66175.4</td>\n",
       "      <td>3.555</td>\n",
       "      <td>66175.5</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:00:00.740175+00:00</td>\n",
       "      <td>2024-05-16 00:00:00.754108+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:01:56.289000+00:00</th>\n",
       "      <td>1715817716289</td>\n",
       "      <td>54.277</td>\n",
       "      <td>3031.67</td>\n",
       "      <td>41.898</td>\n",
       "      <td>3031.68</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:01:56.450604+00:00</td>\n",
       "      <td>2024-05-16 00:01:56.475475+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:01:56.437000+00:00</th>\n",
       "      <td>1715817716437</td>\n",
       "      <td>54.274</td>\n",
       "      <td>3031.67</td>\n",
       "      <td>42.601</td>\n",
       "      <td>3031.68</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:01:56.651671+00:00</td>\n",
       "      <td>2024-05-16 00:01:56.673926+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-16 00:01:56.544000+00:00</th>\n",
       "      <td>1715817716544</td>\n",
       "      <td>52.984</td>\n",
       "      <td>3031.67</td>\n",
       "      <td>42.731</td>\n",
       "      <td>3031.68</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 00:01:56.853605+00:00</td>\n",
       "      <td>2024-05-16 00:01:56.874993+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      timestamp bid_size bid_price ask_size ask_price exchange_id level            end_download_timestamp               knowledge_timestamp currency_pair  year month  day\n",
       "2024-05-16 00:00:00.143000+00:00  1715817600143    4.022   66175.4    3.215   66175.5     binance     1  2024-05-16 00:00:00.337472+00:00  2024-05-16 00:00:00.350773+00:00      BTC_USDT  2024     5   16\n",
       "2024-05-16 00:00:00.365000+00:00  1715817600365    2.226   66175.4    3.755   66175.5     binance     1  2024-05-16 00:00:00.538837+00:00  2024-05-16 00:00:00.552395+00:00      BTC_USDT  2024     5   16\n",
       "2024-05-16 00:00:00.570000+00:00  1715817600570    2.623   66175.4    3.555   66175.5     binance     1  2024-05-16 00:00:00.740175+00:00  2024-05-16 00:00:00.754108+00:00      BTC_USDT  2024     5   16\n",
       "                                            ...      ...       ...      ...       ...         ...   ...                               ...                               ...           ...   ...   ...  ...\n",
       "2024-05-16 00:01:56.289000+00:00  1715817716289   54.277   3031.67   41.898   3031.68     binance     1  2024-05-16 00:01:56.450604+00:00  2024-05-16 00:01:56.475475+00:00      ETH_USDT  2024     5   16\n",
       "2024-05-16 00:01:56.437000+00:00  1715817716437   54.274   3031.67   42.601   3031.68     binance     1  2024-05-16 00:01:56.651671+00:00  2024-05-16 00:01:56.673926+00:00      ETH_USDT  2024     5   16\n",
       "2024-05-16 00:01:56.544000+00:00  1715817716544   52.984   3031.67   42.731   3031.68     binance     1  2024-05-16 00:01:56.853605+00:00  2024-05-16 00:01:56.874993+00:00      ETH_USDT  2024     5   16"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.parquet.bid_ask.futures.v8.binance.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-05-16T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=2)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\", \"ETH_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4403abe",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_1min.postgres.ohlcv.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "52e7c3b1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:43:43.792416Z",
     "start_time": "2024-05-16T19:43:43.757342Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_ohlcv_futures` DB table\n",
      "INFO  Enabled connection to the `ccxt_ohlcv_futures` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_ohlcv_futures WHERE exchange_id = 'binance' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>0.02167</td>\n",
       "      <td>0.02168</td>\n",
       "      <td>0.02166</td>\n",
       "      <td>0.02168</td>\n",
       "      <td>210396.0</td>\n",
       "      <td>USTC_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.928690+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>0.0868</td>\n",
       "      <td>0.0868</td>\n",
       "      <td>0.08674</td>\n",
       "      <td>0.08677</td>\n",
       "      <td>176463.0</td>\n",
       "      <td>LOOM_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.927322+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>5.038</td>\n",
       "      <td>5.04</td>\n",
       "      <td>5.037</td>\n",
       "      <td>5.04</td>\n",
       "      <td>822.9</td>\n",
       "      <td>GAS_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.925938+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>81.54</td>\n",
       "      <td>81.58</td>\n",
       "      <td>81.52</td>\n",
       "      <td>81.58</td>\n",
       "      <td>42.0</td>\n",
       "      <td>ILV_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.918907+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>1.0688</td>\n",
       "      <td>1.0689</td>\n",
       "      <td>1.0679</td>\n",
       "      <td>1.0688</td>\n",
       "      <td>2629.0</td>\n",
       "      <td>TWT_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.917531+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888520000</td>\n",
       "      <td>4.5453</td>\n",
       "      <td>4.5496</td>\n",
       "      <td>4.5436</td>\n",
       "      <td>4.5467</td>\n",
       "      <td>12282.0</td>\n",
       "      <td>JTO_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 19:43:10.916163+00:00</td>\n",
       "      <td>2024-05-16 19:43:10.992352+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp     open     high      low    close    volume currency_pair exchange_id            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888520000  0.02167  0.02168  0.02166  0.02168  210396.0     USTC_USDT     binance  2024-05-16 19:43:10.928690+00:00  2024-05-16 19:43:10.992352+00:00\n",
       "1  1715888520000   0.0868   0.0868  0.08674  0.08677  176463.0     LOOM_USDT     binance  2024-05-16 19:43:10.927322+00:00  2024-05-16 19:43:10.992352+00:00\n",
       "2  1715888520000    5.038     5.04    5.037     5.04     822.9      GAS_USDT     binance  2024-05-16 19:43:10.925938+00:00  2024-05-16 19:43:10.992352+00:00\n",
       "             ...      ...      ...      ...      ...       ...           ...         ...                               ...                               ...\n",
       "7  1715888520000    81.54    81.58    81.52    81.58      42.0      ILV_USDT     binance  2024-05-16 19:43:10.918907+00:00  2024-05-16 19:43:10.992352+00:00\n",
       "8  1715888520000   1.0688   1.0689   1.0679   1.0688    2629.0      TWT_USDT     binance  2024-05-16 19:43:10.917531+00:00  2024-05-16 19:43:10.992352+00:00\n",
       "9  1715888520000   4.5453   4.5496   4.5436   4.5467   12282.0      JTO_USDT     binance  2024-05-16 19:43:10.916163+00:00  2024-05-16 19:43:10.992352+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.downloaded_1min.postgres.ohlcv.futures.v8.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0b83d9",
   "metadata": {
    "run_control": {
     "marked": false
    }
   },
   "source": [
    "## periodic_daily.airflow.downloaded_all.parquet.trades.futures.v8.binance.binance.v2_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a9345be9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:44:53.492602Z",
     "start_time": "2024-05-16T19:43:43.794985Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Before from_parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>price</th>\n",
       "      <th>amount</th>\n",
       "      <th>is_buyer_maker</th>\n",
       "      <th>id</th>\n",
       "      <th>quote_qty</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.006000+00:00</th>\n",
       "      <td>1704067200006</td>\n",
       "      <td>42314.0</td>\n",
       "      <td>0.033</td>\n",
       "      <td>False</td>\n",
       "      <td>4426785098</td>\n",
       "      <td>1396.362</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-10 15:40:39.807031+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.006000+00:00</th>\n",
       "      <td>1704067200006</td>\n",
       "      <td>42314.0</td>\n",
       "      <td>0.215</td>\n",
       "      <td>False</td>\n",
       "      <td>4426785099</td>\n",
       "      <td>9097.51</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-10 15:40:39.807031+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.022000+00:00</th>\n",
       "      <td>1704067200022</td>\n",
       "      <td>42314.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>False</td>\n",
       "      <td>4426785100</td>\n",
       "      <td>4231.4</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-10 15:40:39.807031+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:01:59.554000+00:00</th>\n",
       "      <td>1704067319554</td>\n",
       "      <td>42350.5</td>\n",
       "      <td>0.006</td>\n",
       "      <td>False</td>\n",
       "      <td>4426790290</td>\n",
       "      <td>254.103</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-18 01:10:16.793311+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:01:59.780000+00:00</th>\n",
       "      <td>1704067319780</td>\n",
       "      <td>42350.5</td>\n",
       "      <td>0.059</td>\n",
       "      <td>False</td>\n",
       "      <td>4426790291</td>\n",
       "      <td>2498.6795</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-18 01:10:16.793311+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:01:59.828000+00:00</th>\n",
       "      <td>1704067319828</td>\n",
       "      <td>42350.4</td>\n",
       "      <td>0.015</td>\n",
       "      <td>True</td>\n",
       "      <td>4426790292</td>\n",
       "      <td>635.256</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-04-18 01:10:16.793311+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      timestamp    price amount is_buyer_maker          id  quote_qty exchange_id               knowledge_timestamp currency_pair  year month  day\n",
       "2024-01-01 00:00:00.006000+00:00  1704067200006  42314.0  0.033          False  4426785098   1396.362     binance  2024-04-10 15:40:39.807031+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:00.006000+00:00  1704067200006  42314.0  0.215          False  4426785099    9097.51     binance  2024-04-10 15:40:39.807031+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:00.022000+00:00  1704067200022  42314.0    0.1          False  4426785100     4231.4     binance  2024-04-10 15:40:39.807031+00:00      BTC_USDT  2024     1    1\n",
       "                                            ...      ...    ...            ...         ...        ...         ...                               ...           ...   ...   ...  ...\n",
       "2024-01-01 00:01:59.554000+00:00  1704067319554  42350.5  0.006          False  4426790290    254.103     binance  2024-04-18 01:10:16.793311+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:01:59.780000+00:00  1704067319780  42350.5  0.059          False  4426790291  2498.6795     binance  2024-04-18 01:10:16.793311+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:01:59.828000+00:00  1704067319828  42350.4  0.015           True  4426790292    635.256     binance  2024-04-18 01:10:16.793311+00:00      BTC_USDT  2024     1    1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_all.parquet.trades.futures.v8.binance.binance.v2_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-01-01T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=2)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "98415afd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:31.509151Z",
     "start_time": "2024-05-16T19:46:31.506833Z"
    }
   },
   "outputs": [],
   "source": [
    "## TODO(Sonaal): Add info about bid/ask historical data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d42db81",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_200ms.postgres.bid_ask.spot.v7_3.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "def705ad",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:32.275689Z",
     "start_time": "2024-05-16T19:46:32.243832Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_spot_raw WHERE exchange_id = 'binance' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>1111.0</td>\n",
       "      <td>0.5261</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.5263</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>2928.0</td>\n",
       "      <td>0.526</td>\n",
       "      <td>441.0</td>\n",
       "      <td>0.5264</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>3898.0</td>\n",
       "      <td>0.5259</td>\n",
       "      <td>8131.0</td>\n",
       "      <td>0.5265</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>21095.0</td>\n",
       "      <td>0.5254</td>\n",
       "      <td>4867.0</td>\n",
       "      <td>0.527</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>8413.0</td>\n",
       "      <td>0.5253</td>\n",
       "      <td>3264.0</td>\n",
       "      <td>0.5271</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>16450.0</td>\n",
       "      <td>0.5252</td>\n",
       "      <td>2099.0</td>\n",
       "      <td>0.5272</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888752320   1111.0    0.5261     39.0    0.5263    STORJ_USDT     binance     1  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "1  1715888752320   2928.0     0.526    441.0    0.5264    STORJ_USDT     binance     2  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "2  1715888752320   3898.0    0.5259   8131.0    0.5265    STORJ_USDT     binance     3  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888752320  21095.0    0.5254   4867.0     0.527    STORJ_USDT     binance     8  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "8  1715888752320   8413.0    0.5253   3264.0    0.5271    STORJ_USDT     binance     9  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "9  1715888752320  16450.0    0.5252   2099.0    0.5272    STORJ_USDT     binance    10  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.downloaded_200ms.postgres.bid_ask.spot.v7_3.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9654a74",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.postgres.bid_ask.spot.v7.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "39d52397",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:32.689194Z",
     "start_time": "2024-05-16T19:46:32.659497Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_spot_raw WHERE exchange_id = 'binance' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>1111.0</td>\n",
       "      <td>0.5261</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.5263</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>2928.0</td>\n",
       "      <td>0.526</td>\n",
       "      <td>441.0</td>\n",
       "      <td>0.5264</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>3898.0</td>\n",
       "      <td>0.5259</td>\n",
       "      <td>8131.0</td>\n",
       "      <td>0.5265</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>21095.0</td>\n",
       "      <td>0.5254</td>\n",
       "      <td>4867.0</td>\n",
       "      <td>0.527</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>8413.0</td>\n",
       "      <td>0.5253</td>\n",
       "      <td>3264.0</td>\n",
       "      <td>0.5271</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888752320</td>\n",
       "      <td>16450.0</td>\n",
       "      <td>0.5252</td>\n",
       "      <td>2099.0</td>\n",
       "      <td>0.5272</td>\n",
       "      <td>STORJ_USDT</td>\n",
       "      <td>binance</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:45:52.457091+00:00</td>\n",
       "      <td>2024-05-16 19:45:52.483667+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888752320   1111.0    0.5261     39.0    0.5263    STORJ_USDT     binance     1  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "1  1715888752320   2928.0     0.526    441.0    0.5264    STORJ_USDT     binance     2  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "2  1715888752320   3898.0    0.5259   8131.0    0.5265    STORJ_USDT     binance     3  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888752320  21095.0    0.5254   4867.0     0.527    STORJ_USDT     binance     8  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "8  1715888752320   8413.0    0.5253   3264.0    0.5271    STORJ_USDT     binance     9  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00\n",
       "9  1715888752320  16450.0    0.5252   2099.0    0.5272    STORJ_USDT     binance    10  2024-05-16 19:45:52.457091+00:00  2024-05-16 19:45:52.483667+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.postgres.bid_ask.spot.v7.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f80b9e3",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_200ms.postgres.bid_ask.spot.v7_3.ccxt.okx.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "66e087be",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:34.713787Z",
     "start_time": "2024-05-16T19:46:34.688264Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_spot_raw WHERE exchange_id = 'okx' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>3011.472</td>\n",
       "      <td>4.863</td>\n",
       "      <td>678.777</td>\n",
       "      <td>4.882</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>900.853</td>\n",
       "      <td>4.864</td>\n",
       "      <td>5162.821</td>\n",
       "      <td>4.881</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>5114.216</td>\n",
       "      <td>4.865</td>\n",
       "      <td>817.85</td>\n",
       "      <td>4.88</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>438.143</td>\n",
       "      <td>4.87</td>\n",
       "      <td>953.924</td>\n",
       "      <td>4.875</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>168.851</td>\n",
       "      <td>4.871</td>\n",
       "      <td>118.686</td>\n",
       "      <td>4.874</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>101.229</td>\n",
       "      <td>4.872</td>\n",
       "      <td>54.147</td>\n",
       "      <td>4.873</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp  bid_size bid_price  ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888790109  3011.472     4.863   678.777     4.882      WLD_USDT         okx    10  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "1  1715888790109   900.853     4.864  5162.821     4.881      WLD_USDT         okx     9  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "2  1715888790109  5114.216     4.865    817.85      4.88      WLD_USDT         okx     8  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "             ...       ...       ...       ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888790109   438.143      4.87   953.924     4.875      WLD_USDT         okx     3  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "8  1715888790109   168.851     4.871   118.686     4.874      WLD_USDT         okx     2  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "9  1715888790109   101.229     4.872    54.147     4.873      WLD_USDT         okx     1  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = (\n",
    "    \"realtime.airflow.downloaded_200ms.postgres.bid_ask.spot.v7_3.ccxt.okx.v1_0_0\"\n",
    ")\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1cde37d",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.postgres.bid_ask.spot.v7_3.ccxt.okx.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1381e04e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:36.436080Z",
     "start_time": "2024-05-16T19:46:36.409422Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_spot_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_spot_raw WHERE exchange_id = 'okx' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>3011.472</td>\n",
       "      <td>4.863</td>\n",
       "      <td>678.777</td>\n",
       "      <td>4.882</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>900.853</td>\n",
       "      <td>4.864</td>\n",
       "      <td>5162.821</td>\n",
       "      <td>4.881</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>5114.216</td>\n",
       "      <td>4.865</td>\n",
       "      <td>817.85</td>\n",
       "      <td>4.88</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>438.143</td>\n",
       "      <td>4.87</td>\n",
       "      <td>953.924</td>\n",
       "      <td>4.875</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>168.851</td>\n",
       "      <td>4.871</td>\n",
       "      <td>118.686</td>\n",
       "      <td>4.874</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888790109</td>\n",
       "      <td>101.229</td>\n",
       "      <td>4.872</td>\n",
       "      <td>54.147</td>\n",
       "      <td>4.873</td>\n",
       "      <td>WLD_USDT</td>\n",
       "      <td>okx</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:46:30.481910+00:00</td>\n",
       "      <td>2024-05-16 19:46:30.585210+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp  bid_size bid_price  ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888790109  3011.472     4.863   678.777     4.882      WLD_USDT         okx    10  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "1  1715888790109   900.853     4.864  5162.821     4.881      WLD_USDT         okx     9  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "2  1715888790109  5114.216     4.865    817.85      4.88      WLD_USDT         okx     8  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "             ...       ...       ...       ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888790109   438.143      4.87   953.924     4.875      WLD_USDT         okx     3  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "8  1715888790109   168.851     4.871   118.686     4.874      WLD_USDT         okx     2  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00\n",
       "9  1715888790109   101.229     4.872    54.147     4.873      WLD_USDT         okx     1  2024-05-16 19:46:30.481910+00:00  2024-05-16 19:46:30.585210+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.postgres.bid_ask.spot.v7_3.ccxt.okx.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d17fe28",
   "metadata": {},
   "source": [
    "## realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v7_4.ccxt.cryptocom.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8a13ed40",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:38.229671Z",
     "start_time": "2024-05-16T19:46:38.202050Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_futures_raw WHERE exchange_id = 'cryptocom' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>9.37</td>\n",
       "      <td>159.39</td>\n",
       "      <td>8.91</td>\n",
       "      <td>159.72</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>0.8</td>\n",
       "      <td>159.4</td>\n",
       "      <td>8.95</td>\n",
       "      <td>159.71</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>83.47</td>\n",
       "      <td>159.41</td>\n",
       "      <td>9.0</td>\n",
       "      <td>159.7</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>1.5</td>\n",
       "      <td>159.47</td>\n",
       "      <td>4.05</td>\n",
       "      <td>159.59</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>2.69</td>\n",
       "      <td>159.48</td>\n",
       "      <td>3.5</td>\n",
       "      <td>159.58</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>2.0</td>\n",
       "      <td>159.52</td>\n",
       "      <td>37.24</td>\n",
       "      <td>159.57</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888797183     9.37    159.39     8.91    159.72       SOL_USD   cryptocom    10  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "1  1715888797183      0.8     159.4     8.95    159.71       SOL_USD   cryptocom     9  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "2  1715888797183    83.47    159.41      9.0     159.7       SOL_USD   cryptocom     8  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888797183      1.5    159.47     4.05    159.59       SOL_USD   cryptocom     3  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "8  1715888797183     2.69    159.48      3.5    159.58       SOL_USD   cryptocom     2  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "9  1715888797183      2.0    159.52    37.24    159.57       SOL_USD   cryptocom     1  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"realtime.airflow.downloaded_200ms.postgres.bid_ask.futures.v7_4.ccxt.cryptocom.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c24f7b4e",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.postgres.bid_ask.futures.v7_4.ccxt.cryptocom.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bed650c8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:39.872458Z",
     "start_time": "2024-05-16T19:46:39.834481Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Enabled connection to the `ccxt_bid_ask_futures_raw` DB table\n",
      "INFO  Executing query: \n",
      "\tSELECT * FROM ccxt_bid_ask_futures_raw WHERE exchange_id = 'cryptocom' ORDER BY timestamp DESC LIMIT 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/app/amp/helpers/hsql_implementation.py:582: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  df = pd.read_sql_query(query, connection)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>9.37</td>\n",
       "      <td>159.39</td>\n",
       "      <td>8.91</td>\n",
       "      <td>159.72</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>10</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>0.8</td>\n",
       "      <td>159.4</td>\n",
       "      <td>8.95</td>\n",
       "      <td>159.71</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>9</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>83.47</td>\n",
       "      <td>159.41</td>\n",
       "      <td>9.0</td>\n",
       "      <td>159.7</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>8</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>1.5</td>\n",
       "      <td>159.47</td>\n",
       "      <td>4.05</td>\n",
       "      <td>159.59</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>3</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>2.69</td>\n",
       "      <td>159.48</td>\n",
       "      <td>3.5</td>\n",
       "      <td>159.58</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1715888797183</td>\n",
       "      <td>2.0</td>\n",
       "      <td>159.52</td>\n",
       "      <td>37.24</td>\n",
       "      <td>159.57</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-05-16 19:46:37.323794+00:00</td>\n",
       "      <td>2024-05-16 19:46:37.378304+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       timestamp bid_size bid_price ask_size ask_price currency_pair exchange_id level            end_download_timestamp               knowledge_timestamp\n",
       "0  1715888797183     9.37    159.39     8.91    159.72       SOL_USD   cryptocom    10  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "1  1715888797183      0.8     159.4     8.95    159.71       SOL_USD   cryptocom     9  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "2  1715888797183    83.47    159.41      9.0     159.7       SOL_USD   cryptocom     8  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "             ...      ...       ...      ...       ...           ...         ...   ...                               ...                               ...\n",
       "7  1715888797183      1.5    159.47     4.05    159.59       SOL_USD   cryptocom     3  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "8  1715888797183     2.69    159.48      3.5    159.58       SOL_USD   cryptocom     2  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00\n",
       "9  1715888797183      2.0    159.52    37.24    159.57       SOL_USD   cryptocom     1  2024-05-16 19:46:37.323794+00:00  2024-05-16 19:46:37.378304+00:00"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.postgres.bid_ask.futures.v7_4.ccxt.cryptocom.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0388d168",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7_3.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "49496dd6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:46.293606Z",
     "start_time": "2024-05-16T19:46:40.129710Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading the data from `s3://cryptokaizen-data.preprod/v3/periodic_daily/airflow/downloaded_1min/parquet/ohlcv/spot/v7_3/ccxt/binance/v1_0_0`\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:00:00+00:00</th>\n",
       "      <td>1715731200000</td>\n",
       "      <td>0.6319</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6315</td>\n",
       "      <td>0.6338</td>\n",
       "      <td>1010.9</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:01:00+00:00</th>\n",
       "      <td>1715731260000</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6345</td>\n",
       "      <td>0.6333</td>\n",
       "      <td>0.6336</td>\n",
       "      <td>1779.8</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:02:00+00:00</th>\n",
       "      <td>1715731320000</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6349</td>\n",
       "      <td>0.6331</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>3290.5</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:07:00+00:00</th>\n",
       "      <td>1715731620000</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.634</td>\n",
       "      <td>914.0</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:08:00+00:00</th>\n",
       "      <td>1715731680000</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>155.8</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:09:00+00:00</th>\n",
       "      <td>1715731740000</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6343</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6343</td>\n",
       "      <td>2469.2</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp    open    high     low   close  volume exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-05-15 00:00:00+00:00  1715731200000  0.6319  0.6342  0.6315  0.6338  1010.9     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:01:00+00:00  1715731260000  0.6337  0.6345  0.6333  0.6336  1779.8     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:02:00+00:00  1715731320000  0.6337  0.6349  0.6331  0.6339  3290.5     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "                                     ...     ...     ...     ...     ...     ...         ...                               ...           ...   ...   ...\n",
       "2024-05-15 00:07:00+00:00  1715731620000   0.634  0.6342  0.6337   0.634   914.0     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:08:00+00:00  1715731680000  0.6339  0.6342  0.6339  0.6342   155.8     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:09:00+00:00  1715731740000  0.6339  0.6343  0.6337  0.6343  2469.2     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7_3.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff4377a",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_3.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "08fa0287",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:46.387747Z",
     "start_time": "2024-05-16T19:46:46.295929Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading the data from `s3://cryptokaizen-data.preprod/v3/periodic_daily/airflow/downloaded_1min/parquet/ohlcv/spot/v7_3/ccxt/binance/v1_0_0`\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:00:00+00:00</th>\n",
       "      <td>1715731200000</td>\n",
       "      <td>0.6319</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6315</td>\n",
       "      <td>0.6338</td>\n",
       "      <td>1010.9</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:01:00+00:00</th>\n",
       "      <td>1715731260000</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6345</td>\n",
       "      <td>0.6333</td>\n",
       "      <td>0.6336</td>\n",
       "      <td>1779.8</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:02:00+00:00</th>\n",
       "      <td>1715731320000</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6349</td>\n",
       "      <td>0.6331</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>3290.5</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:07:00+00:00</th>\n",
       "      <td>1715731620000</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.634</td>\n",
       "      <td>914.0</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:08:00+00:00</th>\n",
       "      <td>1715731680000</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6342</td>\n",
       "      <td>155.8</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:09:00+00:00</th>\n",
       "      <td>1715731740000</td>\n",
       "      <td>0.6339</td>\n",
       "      <td>0.6343</td>\n",
       "      <td>0.6337</td>\n",
       "      <td>0.6343</td>\n",
       "      <td>2469.2</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-05-16 01:24:33.668966+00:00</td>\n",
       "      <td>CTK_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp    open    high     low   close  volume exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-05-15 00:00:00+00:00  1715731200000  0.6319  0.6342  0.6315  0.6338  1010.9     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:01:00+00:00  1715731260000  0.6337  0.6345  0.6333  0.6336  1779.8     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:02:00+00:00  1715731320000  0.6337  0.6349  0.6331  0.6339  3290.5     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "                                     ...     ...     ...     ...     ...     ...         ...                               ...           ...   ...   ...\n",
       "2024-05-15 00:07:00+00:00  1715731620000   0.634  0.6342  0.6337   0.634   914.0     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:08:00+00:00  1715731680000  0.6339  0.6342  0.6339  0.6342   155.8     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5\n",
       "2024-05-15 00:09:00+00:00  1715731740000  0.6339  0.6343  0.6337  0.6343  2469.2     binance  2024-05-16 01:24:33.668966+00:00      CTK_USDT  2024     5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7_3.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "286bd2eb",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v8.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bce13c5e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:46:54.067921Z",
     "start_time": "2024-05-16T19:46:46.389719Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Before from_parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00+00:00</th>\n",
       "      <td>1704067200000</td>\n",
       "      <td>42314.0</td>\n",
       "      <td>42335.8</td>\n",
       "      <td>42289.6</td>\n",
       "      <td>42331.9</td>\n",
       "      <td>289.641</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 20:28:37.170290+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:01:00+00:00</th>\n",
       "      <td>1704067260000</td>\n",
       "      <td>42331.9</td>\n",
       "      <td>42353.1</td>\n",
       "      <td>42331.8</td>\n",
       "      <td>42350.4</td>\n",
       "      <td>202.444</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 20:28:37.170290+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:02:00+00:00</th>\n",
       "      <td>1704067320000</td>\n",
       "      <td>42350.4</td>\n",
       "      <td>42370.8</td>\n",
       "      <td>42349.6</td>\n",
       "      <td>42360.2</td>\n",
       "      <td>271.521</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 20:28:37.170290+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:03:00+00:00</th>\n",
       "      <td>1704067380000</td>\n",
       "      <td>2286.3</td>\n",
       "      <td>2288.44</td>\n",
       "      <td>2286.29</td>\n",
       "      <td>2288.35</td>\n",
       "      <td>2191.911</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 21:23:12.203301+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:04:00+00:00</th>\n",
       "      <td>1704067440000</td>\n",
       "      <td>2288.36</td>\n",
       "      <td>2289.92</td>\n",
       "      <td>2288.36</td>\n",
       "      <td>2289.92</td>\n",
       "      <td>1946.412</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 21:23:12.203301+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:05:00+00:00</th>\n",
       "      <td>1704067500000</td>\n",
       "      <td>2289.92</td>\n",
       "      <td>2291.93</td>\n",
       "      <td>2288.77</td>\n",
       "      <td>2291.45</td>\n",
       "      <td>3691.669</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-02-27 21:23:12.203301+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp     open     high      low    close    volume exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-01-01 00:00:00+00:00  1704067200000  42314.0  42335.8  42289.6  42331.9   289.641     binance  2024-02-27 20:28:37.170290+00:00      BTC_USDT  2024     1\n",
       "2024-01-01 00:01:00+00:00  1704067260000  42331.9  42353.1  42331.8  42350.4   202.444     binance  2024-02-27 20:28:37.170290+00:00      BTC_USDT  2024     1\n",
       "2024-01-01 00:02:00+00:00  1704067320000  42350.4  42370.8  42349.6  42360.2   271.521     binance  2024-02-27 20:28:37.170290+00:00      BTC_USDT  2024     1\n",
       "                                     ...      ...      ...      ...      ...       ...         ...                               ...           ...   ...   ...\n",
       "2024-01-01 00:03:00+00:00  1704067380000   2286.3  2288.44  2286.29  2288.35  2191.911     binance  2024-02-27 21:23:12.203301+00:00      ETH_USDT  2024     1\n",
       "2024-01-01 00:04:00+00:00  1704067440000  2288.36  2289.92  2288.36  2289.92  1946.412     binance  2024-02-27 21:23:12.203301+00:00      ETH_USDT  2024     1\n",
       "2024-01-01 00:05:00+00:00  1704067500000  2289.92  2291.93  2288.77  2291.45  3691.669     binance  2024-02-27 21:23:12.203301+00:00      ETH_USDT  2024     1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v8.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-01-01T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=5)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\", \"ETH_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76d6949",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_4.ccxt.cryptocom.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3f7743b7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:47:07.693404Z",
     "start_time": "2024-05-16T19:46:54.070292Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading the data from `s3://cryptokaizen-data.preprod/v3/periodic_daily/airflow/downloaded_1min/parquet/ohlcv/futures/v7_4/ccxt/cryptocom/v1_0_0`\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:00:00+00:00</th>\n",
       "      <td>1715731200000</td>\n",
       "      <td>141.96</td>\n",
       "      <td>142.08</td>\n",
       "      <td>141.94</td>\n",
       "      <td>142.06</td>\n",
       "      <td>15.93</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:01:00+00:00</th>\n",
       "      <td>1715731260000</td>\n",
       "      <td>142.03</td>\n",
       "      <td>142.11</td>\n",
       "      <td>142.02</td>\n",
       "      <td>142.07</td>\n",
       "      <td>40.14</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:02:00+00:00</th>\n",
       "      <td>1715731320000</td>\n",
       "      <td>142.09</td>\n",
       "      <td>142.26</td>\n",
       "      <td>142.09</td>\n",
       "      <td>142.18</td>\n",
       "      <td>65.22</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:07:00+00:00</th>\n",
       "      <td>1715731620000</td>\n",
       "      <td>142.32</td>\n",
       "      <td>142.35</td>\n",
       "      <td>142.27</td>\n",
       "      <td>142.27</td>\n",
       "      <td>3.6</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:08:00+00:00</th>\n",
       "      <td>1715731680000</td>\n",
       "      <td>142.28</td>\n",
       "      <td>142.4</td>\n",
       "      <td>142.25</td>\n",
       "      <td>142.25</td>\n",
       "      <td>11.34</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-15 00:09:00+00:00</th>\n",
       "      <td>1715731740000</td>\n",
       "      <td>142.21</td>\n",
       "      <td>142.32</td>\n",
       "      <td>142.21</td>\n",
       "      <td>142.32</td>\n",
       "      <td>8.79</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-05-16 01:24:46.466080+00:00</td>\n",
       "      <td>SOL_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp    open    high     low   close volume exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-05-15 00:00:00+00:00  1715731200000  141.96  142.08  141.94  142.06  15.93   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5\n",
       "2024-05-15 00:01:00+00:00  1715731260000  142.03  142.11  142.02  142.07  40.14   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5\n",
       "2024-05-15 00:02:00+00:00  1715731320000  142.09  142.26  142.09  142.18  65.22   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5\n",
       "                                     ...     ...     ...     ...     ...    ...         ...                               ...           ...   ...   ...\n",
       "2024-05-15 00:07:00+00:00  1715731620000  142.32  142.35  142.27  142.27    3.6   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5\n",
       "2024-05-15 00:08:00+00:00  1715731680000  142.28   142.4  142.25  142.25  11.34   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5\n",
       "2024-05-15 00:09:00+00:00  1715731740000  142.21  142.32  142.21  142.32   8.79   cryptocom  2024-05-16 01:24:46.466080+00:00       SOL_USD  2024     5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_4.ccxt.cryptocom.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21ca830",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v8.ccxt.okx.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f8aa8a28",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-12T22:18:55.769999Z",
     "start_time": "2024-06-12T22:18:49.467997Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00+00:00</th>\n",
       "      <td>1704067200000</td>\n",
       "      <td>42297.7</td>\n",
       "      <td>42310.0</td>\n",
       "      <td>42266.1</td>\n",
       "      <td>42308.0</td>\n",
       "      <td>89.86</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:17:19.094084+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:01:00+00:00</th>\n",
       "      <td>1704067260000</td>\n",
       "      <td>42308.0</td>\n",
       "      <td>42329.7</td>\n",
       "      <td>42307.9</td>\n",
       "      <td>42327.5</td>\n",
       "      <td>83.71</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:17:19.094084+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:02:00+00:00</th>\n",
       "      <td>1704067320000</td>\n",
       "      <td>42327.6</td>\n",
       "      <td>42345.1</td>\n",
       "      <td>42325.0</td>\n",
       "      <td>42334.4</td>\n",
       "      <td>82.67</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:17:19.094084+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:03:00+00:00</th>\n",
       "      <td>1704067380000</td>\n",
       "      <td>2285.06</td>\n",
       "      <td>2286.95</td>\n",
       "      <td>2285.06</td>\n",
       "      <td>2286.95</td>\n",
       "      <td>671.3</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:21:00.192712+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:04:00+00:00</th>\n",
       "      <td>1704067440000</td>\n",
       "      <td>2286.95</td>\n",
       "      <td>2288.76</td>\n",
       "      <td>2286.95</td>\n",
       "      <td>2288.69</td>\n",
       "      <td>1308.8</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:21:00.192712+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:05:00+00:00</th>\n",
       "      <td>1704067500000</td>\n",
       "      <td>2288.68</td>\n",
       "      <td>2290.83</td>\n",
       "      <td>2287.52</td>\n",
       "      <td>2290.24</td>\n",
       "      <td>1706.2</td>\n",
       "      <td>okx</td>\n",
       "      <td>2024-02-21 21:21:00.192712+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp     open     high      low    close  volume exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-01-01 00:00:00+00:00  1704067200000  42297.7  42310.0  42266.1  42308.0   89.86         okx  2024-02-21 21:17:19.094084+00:00      BTC_USDT  2024     1\n",
       "2024-01-01 00:01:00+00:00  1704067260000  42308.0  42329.7  42307.9  42327.5   83.71         okx  2024-02-21 21:17:19.094084+00:00      BTC_USDT  2024     1\n",
       "2024-01-01 00:02:00+00:00  1704067320000  42327.6  42345.1  42325.0  42334.4   82.67         okx  2024-02-21 21:17:19.094084+00:00      BTC_USDT  2024     1\n",
       "                                     ...      ...      ...      ...      ...     ...         ...                               ...           ...   ...   ...\n",
       "2024-01-01 00:03:00+00:00  1704067380000  2285.06  2286.95  2285.06  2286.95   671.3         okx  2024-02-21 21:21:00.192712+00:00      ETH_USDT  2024     1\n",
       "2024-01-01 00:04:00+00:00  1704067440000  2286.95  2288.76  2286.95  2288.69  1308.8         okx  2024-02-21 21:21:00.192712+00:00      ETH_USDT  2024     1\n",
       "2024-01-01 00:05:00+00:00  1704067500000  2288.68  2290.83  2287.52  2290.24  1706.2         okx  2024-02-21 21:21:00.192712+00:00      ETH_USDT  2024     1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v8.ccxt.okx.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-01-01T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=5)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\", \"ETH_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4616d014",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v8.ccxt.binance.v2_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "da1c14dd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-12T22:38:12.482249Z",
     "start_time": "2024-06-12T22:38:02.039353Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>level_1.bid_price.open</th>\n",
       "      <th>level_1.bid_size.open</th>\n",
       "      <th>level_1.ask_price.open</th>\n",
       "      <th>level_1.ask_size.open</th>\n",
       "      <th>level_1.bid_ask_midpoint.open</th>\n",
       "      <th>level_1.half_spread.open</th>\n",
       "      <th>level_1.log_size_imbalance.open</th>\n",
       "      <th>level_1.bid_price.close</th>\n",
       "      <th>level_1.bid_size.close</th>\n",
       "      <th>level_1.ask_price.close</th>\n",
       "      <th>level_1.ask_size.close</th>\n",
       "      <th>level_1.bid_ask_midpoint.close</th>\n",
       "      <th>level_1.half_spread.close</th>\n",
       "      <th>level_1.log_size_imbalance.close</th>\n",
       "      <th>level_1.bid_price.high</th>\n",
       "      <th>level_1.bid_size.max</th>\n",
       "      <th>level_1.ask_price.high</th>\n",
       "      <th>level_1.ask_size.max</th>\n",
       "      <th>level_1.bid_ask_midpoint.max</th>\n",
       "      <th>level_1.half_spread.max</th>\n",
       "      <th>level_1.log_size_imbalance.max</th>\n",
       "      <th>level_1.bid_price.low</th>\n",
       "      <th>level_1.bid_size.min</th>\n",
       "      <th>level_1.ask_price.low</th>\n",
       "      <th>level_1.ask_size.min</th>\n",
       "      <th>level_1.bid_ask_midpoint.min</th>\n",
       "      <th>level_1.half_spread.min</th>\n",
       "      <th>level_1.log_size_imbalance.min</th>\n",
       "      <th>level_1.bid_price.mean</th>\n",
       "      <th>level_1.bid_size.mean</th>\n",
       "      <th>level_1.ask_price.mean</th>\n",
       "      <th>level_1.ask_size.mean</th>\n",
       "      <th>level_1.bid_ask_midpoint.mean</th>\n",
       "      <th>level_1.half_spread.mean</th>\n",
       "      <th>level_1.log_size_imbalance.mean</th>\n",
       "      <th>level_1.bid_ask_midpoint_var.100ms</th>\n",
       "      <th>level_1.bid_ask_midpoint_autocovar.100ms</th>\n",
       "      <th>level_1.log_size_imbalance_var.100ms</th>\n",
       "      <th>level_1.log_size_imbalance_autocovar.100ms</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:01:00+00:00</th>\n",
       "      <td>1718064060000</td>\n",
       "      <td>178.50</td>\n",
       "      <td>0.111</td>\n",
       "      <td>178.54</td>\n",
       "      <td>0.512</td>\n",
       "      <td>178.520</td>\n",
       "      <td>0.020</td>\n",
       "      <td>-1.528794</td>\n",
       "      <td>178.87</td>\n",
       "      <td>0.089</td>\n",
       "      <td>178.91</td>\n",
       "      <td>0.098</td>\n",
       "      <td>178.890</td>\n",
       "      <td>0.020</td>\n",
       "      <td>-0.096331</td>\n",
       "      <td>179.01</td>\n",
       "      <td>5.163</td>\n",
       "      <td>179.07</td>\n",
       "      <td>26.815</td>\n",
       "      <td>179.040</td>\n",
       "      <td>0.065</td>\n",
       "      <td>5.027383</td>\n",
       "      <td>178.49</td>\n",
       "      <td>0.002</td>\n",
       "      <td>178.50</td>\n",
       "      <td>0.015</td>\n",
       "      <td>178.495</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-5.833348</td>\n",
       "      <td>178.804365</td>\n",
       "      <td>0.519182</td>\n",
       "      <td>178.846522</td>\n",
       "      <td>1.202100</td>\n",
       "      <td>178.825443</td>\n",
       "      <td>0.021079</td>\n",
       "      <td>-0.231160</td>\n",
       "      <td>0.092650</td>\n",
       "      <td>-0.00015</td>\n",
       "      <td>1915.053525</td>\n",
       "      <td>1637.803137</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-12 04:41:22.098656+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:02:00+00:00</th>\n",
       "      <td>1718064120000</td>\n",
       "      <td>178.87</td>\n",
       "      <td>1.141</td>\n",
       "      <td>178.92</td>\n",
       "      <td>0.158</td>\n",
       "      <td>178.895</td>\n",
       "      <td>0.025</td>\n",
       "      <td>1.977065</td>\n",
       "      <td>178.90</td>\n",
       "      <td>0.839</td>\n",
       "      <td>178.93</td>\n",
       "      <td>0.105</td>\n",
       "      <td>178.915</td>\n",
       "      <td>0.015</td>\n",
       "      <td>2.078250</td>\n",
       "      <td>179.04</td>\n",
       "      <td>33.944</td>\n",
       "      <td>179.09</td>\n",
       "      <td>7.352</td>\n",
       "      <td>179.055</td>\n",
       "      <td>0.035</td>\n",
       "      <td>4.323220</td>\n",
       "      <td>178.87</td>\n",
       "      <td>0.010</td>\n",
       "      <td>178.91</td>\n",
       "      <td>0.028</td>\n",
       "      <td>178.890</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-4.211380</td>\n",
       "      <td>178.929550</td>\n",
       "      <td>0.499678</td>\n",
       "      <td>178.972067</td>\n",
       "      <td>0.475495</td>\n",
       "      <td>178.950808</td>\n",
       "      <td>0.021258</td>\n",
       "      <td>-0.142474</td>\n",
       "      <td>0.033975</td>\n",
       "      <td>0.00010</td>\n",
       "      <td>1849.311905</td>\n",
       "      <td>1612.953513</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-12 04:41:22.098656+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:03:00+00:00</th>\n",
       "      <td>1718064180000</td>\n",
       "      <td>178.90</td>\n",
       "      <td>0.839</td>\n",
       "      <td>178.93</td>\n",
       "      <td>0.105</td>\n",
       "      <td>178.915</td>\n",
       "      <td>0.015</td>\n",
       "      <td>2.078250</td>\n",
       "      <td>179.04</td>\n",
       "      <td>0.032</td>\n",
       "      <td>179.05</td>\n",
       "      <td>0.319</td>\n",
       "      <td>179.045</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-2.299455</td>\n",
       "      <td>179.07</td>\n",
       "      <td>2.589</td>\n",
       "      <td>179.11</td>\n",
       "      <td>14.083</td>\n",
       "      <td>179.085</td>\n",
       "      <td>0.035</td>\n",
       "      <td>4.038994</td>\n",
       "      <td>178.90</td>\n",
       "      <td>0.011</td>\n",
       "      <td>178.93</td>\n",
       "      <td>0.013</td>\n",
       "      <td>178.915</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-4.801783</td>\n",
       "      <td>178.975850</td>\n",
       "      <td>0.474590</td>\n",
       "      <td>179.009217</td>\n",
       "      <td>0.682882</td>\n",
       "      <td>178.992533</td>\n",
       "      <td>0.016683</td>\n",
       "      <td>-0.075147</td>\n",
       "      <td>0.020350</td>\n",
       "      <td>-0.00015</td>\n",
       "      <td>1731.799097</td>\n",
       "      <td>1456.013402</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-12 04:41:22.098656+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:04:00+00:00</th>\n",
       "      <td>1718064240000</td>\n",
       "      <td>179.03</td>\n",
       "      <td>0.126</td>\n",
       "      <td>179.08</td>\n",
       "      <td>5.835</td>\n",
       "      <td>179.055</td>\n",
       "      <td>0.025</td>\n",
       "      <td>-3.835348</td>\n",
       "      <td>178.97</td>\n",
       "      <td>1.144</td>\n",
       "      <td>178.98</td>\n",
       "      <td>0.752</td>\n",
       "      <td>178.975</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.419550</td>\n",
       "      <td>179.07</td>\n",
       "      <td>2.562</td>\n",
       "      <td>179.11</td>\n",
       "      <td>13.094</td>\n",
       "      <td>179.090</td>\n",
       "      <td>0.030</td>\n",
       "      <td>4.046554</td>\n",
       "      <td>178.91</td>\n",
       "      <td>0.018</td>\n",
       "      <td>178.92</td>\n",
       "      <td>0.013</td>\n",
       "      <td>178.915</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-4.795239</td>\n",
       "      <td>179.019200</td>\n",
       "      <td>0.627035</td>\n",
       "      <td>179.048050</td>\n",
       "      <td>2.264520</td>\n",
       "      <td>179.033625</td>\n",
       "      <td>0.014425</td>\n",
       "      <td>-0.395573</td>\n",
       "      <td>0.021650</td>\n",
       "      <td>0.00020</td>\n",
       "      <td>2928.749314</td>\n",
       "      <td>2541.065562</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-12 04:41:22.098656+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:05:00+00:00</th>\n",
       "      <td>1718064300000</td>\n",
       "      <td>178.97</td>\n",
       "      <td>1.144</td>\n",
       "      <td>178.98</td>\n",
       "      <td>0.752</td>\n",
       "      <td>178.975</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.419550</td>\n",
       "      <td>179.10</td>\n",
       "      <td>0.418</td>\n",
       "      <td>179.11</td>\n",
       "      <td>8.643</td>\n",
       "      <td>179.105</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-3.029024</td>\n",
       "      <td>179.10</td>\n",
       "      <td>1.374</td>\n",
       "      <td>179.11</td>\n",
       "      <td>10.873</td>\n",
       "      <td>179.105</td>\n",
       "      <td>0.025</td>\n",
       "      <td>3.171085</td>\n",
       "      <td>178.96</td>\n",
       "      <td>0.001</td>\n",
       "      <td>178.98</td>\n",
       "      <td>0.028</td>\n",
       "      <td>178.975</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-6.436150</td>\n",
       "      <td>179.048683</td>\n",
       "      <td>0.449267</td>\n",
       "      <td>179.064583</td>\n",
       "      <td>2.718245</td>\n",
       "      <td>179.056633</td>\n",
       "      <td>0.007950</td>\n",
       "      <td>-0.982251</td>\n",
       "      <td>0.011850</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>3100.146980</td>\n",
       "      <td>2894.218125</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-12 04:41:22.098656+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-11 00:00:00+00:00</th>\n",
       "      <td>1718064000000</td>\n",
       "      <td>178.44</td>\n",
       "      <td>0.800</td>\n",
       "      <td>178.47</td>\n",
       "      <td>0.101</td>\n",
       "      <td>178.455</td>\n",
       "      <td>0.015</td>\n",
       "      <td>2.069491</td>\n",
       "      <td>178.50</td>\n",
       "      <td>0.111</td>\n",
       "      <td>178.54</td>\n",
       "      <td>0.512</td>\n",
       "      <td>178.520</td>\n",
       "      <td>0.020</td>\n",
       "      <td>-1.528794</td>\n",
       "      <td>178.56</td>\n",
       "      <td>2.209</td>\n",
       "      <td>178.60</td>\n",
       "      <td>1.333</td>\n",
       "      <td>178.580</td>\n",
       "      <td>0.025</td>\n",
       "      <td>4.717382</td>\n",
       "      <td>178.44</td>\n",
       "      <td>0.057</td>\n",
       "      <td>178.47</td>\n",
       "      <td>0.008</td>\n",
       "      <td>178.455</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-3.152136</td>\n",
       "      <td>178.512718</td>\n",
       "      <td>0.476685</td>\n",
       "      <td>178.547517</td>\n",
       "      <td>0.434013</td>\n",
       "      <td>178.530117</td>\n",
       "      <td>0.017399</td>\n",
       "      <td>0.308731</td>\n",
       "      <td>0.006325</td>\n",
       "      <td>-0.00005</td>\n",
       "      <td>2067.222832</td>\n",
       "      <td>1969.132652</td>\n",
       "      <td>binance</td>\n",
       "      <td>2024-06-11 04:41:53.168048+00:00</td>\n",
       "      <td>XMR_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp  level_1.bid_price.open  level_1.bid_size.open  level_1.ask_price.open  level_1.ask_size.open  level_1.bid_ask_midpoint.open  level_1.half_spread.open  level_1.log_size_imbalance.open  level_1.bid_price.close  level_1.bid_size.close  level_1.ask_price.close  level_1.ask_size.close  level_1.bid_ask_midpoint.close  level_1.half_spread.close  level_1.log_size_imbalance.close  level_1.bid_price.high  level_1.bid_size.max  level_1.ask_price.high  level_1.ask_size.max  level_1.bid_ask_midpoint.max  level_1.half_spread.max  level_1.log_size_imbalance.max  level_1.bid_price.low  level_1.bid_size.min  level_1.ask_price.low  level_1.ask_size.min  level_1.bid_ask_midpoint.min  level_1.half_spread.min  level_1.log_size_imbalance.min  level_1.bid_price.mean  level_1.bid_size.mean  level_1.ask_price.mean  level_1.ask_size.mean  level_1.bid_ask_midpoint.mean  level_1.half_spread.mean  level_1.log_size_imbalance.mean  level_1.bid_ask_midpoint_var.100ms  level_1.bid_ask_midpoint_autocovar.100ms  level_1.log_size_imbalance_var.100ms  level_1.log_size_imbalance_autocovar.100ms exchange_id              knowledge_timestamp currency_pair  year  month\n",
       "timestamp                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "2024-06-11 00:01:00+00:00  1718064060000                  178.50                  0.111                  178.54                  0.512                        178.520                     0.020                        -1.528794                   178.87                   0.089                   178.91                   0.098                         178.890                      0.020                         -0.096331                  179.01                 5.163                  179.07                26.815                       179.040                    0.065                        5.027383                 178.49                 0.002                 178.50                 0.015                       178.495                    0.005                       -5.833348              178.804365               0.519182              178.846522               1.202100                     178.825443                  0.021079                        -0.231160                            0.092650                                  -0.00015                           1915.053525                                 1637.803137     binance 2024-06-12 04:41:22.098656+00:00      XMR_USDT  2024      6\n",
       "2024-06-11 00:02:00+00:00  1718064120000                  178.87                  1.141                  178.92                  0.158                        178.895                     0.025                         1.977065                   178.90                   0.839                   178.93                   0.105                         178.915                      0.015                          2.078250                  179.04                33.944                  179.09                 7.352                       179.055                    0.035                        4.323220                 178.87                 0.010                 178.91                 0.028                       178.890                    0.005                       -4.211380              178.929550               0.499678              178.972067               0.475495                     178.950808                  0.021258                        -0.142474                            0.033975                                   0.00010                           1849.311905                                 1612.953513     binance 2024-06-12 04:41:22.098656+00:00      XMR_USDT  2024      6\n",
       "2024-06-11 00:03:00+00:00  1718064180000                  178.90                  0.839                  178.93                  0.105                        178.915                     0.015                         2.078250                   179.04                   0.032                   179.05                   0.319                         179.045                      0.005                         -2.299455                  179.07                 2.589                  179.11                14.083                       179.085                    0.035                        4.038994                 178.90                 0.011                 178.93                 0.013                       178.915                    0.005                       -4.801783              178.975850               0.474590              179.009217               0.682882                     178.992533                  0.016683                        -0.075147                            0.020350                                  -0.00015                           1731.799097                                 1456.013402     binance 2024-06-12 04:41:22.098656+00:00      XMR_USDT  2024      6\n",
       "2024-06-11 00:04:00+00:00  1718064240000                  179.03                  0.126                  179.08                  5.835                        179.055                     0.025                        -3.835348                   178.97                   1.144                   178.98                   0.752                         178.975                      0.005                          0.419550                  179.07                 2.562                  179.11                13.094                       179.090                    0.030                        4.046554                 178.91                 0.018                 178.92                 0.013                       178.915                    0.005                       -4.795239              179.019200               0.627035              179.048050               2.264520                     179.033625                  0.014425                        -0.395573                            0.021650                                   0.00020                           2928.749314                                 2541.065562     binance 2024-06-12 04:41:22.098656+00:00      XMR_USDT  2024      6\n",
       "2024-06-11 00:05:00+00:00  1718064300000                  178.97                  1.144                  178.98                  0.752                        178.975                     0.005                         0.419550                   179.10                   0.418                   179.11                   8.643                         179.105                      0.005                         -3.029024                  179.10                 1.374                  179.11                10.873                       179.105                    0.025                        3.171085                 178.96                 0.001                 178.98                 0.028                       178.975                    0.005                       -6.436150              179.048683               0.449267              179.064583               2.718245                     179.056633                  0.007950                        -0.982251                            0.011850                                   0.00000                           3100.146980                                 2894.218125     binance 2024-06-12 04:41:22.098656+00:00      XMR_USDT  2024      6\n",
       "2024-06-11 00:00:00+00:00  1718064000000                  178.44                  0.800                  178.47                  0.101                        178.455                     0.015                         2.069491                   178.50                   0.111                   178.54                   0.512                         178.520                      0.020                         -1.528794                  178.56                 2.209                  178.60                 1.333                       178.580                    0.025                        4.717382                 178.44                 0.057                 178.47                 0.008                       178.455                    0.005                       -3.152136              178.512718               0.476685              178.547517               0.434013                     178.530117                  0.017399                         0.308731                            0.006325                                  -0.00005                           2067.222832                                 1969.132652     binance 2024-06-11 04:41:53.168048+00:00      XMR_USDT  2024      6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v8.ccxt.binance.v2_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\", add_suffix=\"tokyo\")\n",
    "start_timestamp = pd.Timestamp(\"2024-06-11T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=5)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"XMR_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "422e8575",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ## periodic_daily.airflow.downloaded_200ms.postgres.bid_ask.futures.v7_5.ccxt.cryptocom.v1_0_0\n",
    "# TODO(Juraj): The downloader for this dataset runs in Tokyo, we do not have cross-region RDS access available, should\n",
    "# it be added?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59a42cb",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.parquet.bid_ask.futures.v7_5.ccxt.cryptocom.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6c4ef1ab",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-09T17:17:28.390753Z",
     "start_time": "2024-07-09T17:17:25.030147Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:12.809000+00:00</th>\n",
       "      <td>1720504812809</td>\n",
       "      <td>0.236</td>\n",
       "      <td>57271.4</td>\n",
       "      <td>0.378</td>\n",
       "      <td>57271.5</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:12.901517+00:00</td>\n",
       "      <td>2024-07-09 06:00:12.914780+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:13.011000+00:00</th>\n",
       "      <td>1720504813011</td>\n",
       "      <td>0.236</td>\n",
       "      <td>57271.4</td>\n",
       "      <td>0.8196</td>\n",
       "      <td>57271.5</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:13.103480+00:00</td>\n",
       "      <td>2024-07-09 06:00:13.115856+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:13.313000+00:00</th>\n",
       "      <td>1720504813313</td>\n",
       "      <td>0.236</td>\n",
       "      <td>57271.4</td>\n",
       "      <td>0.789</td>\n",
       "      <td>57271.5</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:13.317016+00:00</td>\n",
       "      <td>2024-07-09 06:00:13.332579+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:12.394000+00:00</th>\n",
       "      <td>1720504812394</td>\n",
       "      <td>1.6024</td>\n",
       "      <td>3067.35</td>\n",
       "      <td>0.0324</td>\n",
       "      <td>3067.61</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:12.408426+00:00</td>\n",
       "      <td>2024-07-09 06:00:12.422987+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:12.596000+00:00</th>\n",
       "      <td>1720504812596</td>\n",
       "      <td>1.6808</td>\n",
       "      <td>3067.27</td>\n",
       "      <td>4.5</td>\n",
       "      <td>3067.5</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:12.611365+00:00</td>\n",
       "      <td>2024-07-09 06:00:12.628991+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-09 06:00:12.898000+00:00</th>\n",
       "      <td>1720504812898</td>\n",
       "      <td>0.4201</td>\n",
       "      <td>3067.39</td>\n",
       "      <td>0.0323</td>\n",
       "      <td>3067.49</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-07-09 06:00:12.901371+00:00</td>\n",
       "      <td>2024-07-09 06:00:12.914780+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      timestamp bid_size bid_price ask_size ask_price exchange_id level            end_download_timestamp               knowledge_timestamp currency_pair  year month  day\n",
       "2024-07-09 06:00:12.809000+00:00  1720504812809    0.236   57271.4    0.378   57271.5   cryptocom     1  2024-07-09 06:00:12.901517+00:00  2024-07-09 06:00:12.914780+00:00       BTC_USD  2024     7    9\n",
       "2024-07-09 06:00:13.011000+00:00  1720504813011    0.236   57271.4   0.8196   57271.5   cryptocom     1  2024-07-09 06:00:13.103480+00:00  2024-07-09 06:00:13.115856+00:00       BTC_USD  2024     7    9\n",
       "2024-07-09 06:00:13.313000+00:00  1720504813313    0.236   57271.4    0.789   57271.5   cryptocom     1  2024-07-09 06:00:13.317016+00:00  2024-07-09 06:00:13.332579+00:00       BTC_USD  2024     7    9\n",
       "                                            ...      ...       ...      ...       ...         ...   ...                               ...                               ...           ...   ...   ...  ...\n",
       "2024-07-09 06:00:12.394000+00:00  1720504812394   1.6024   3067.35   0.0324   3067.61   cryptocom     1  2024-07-09 06:00:12.408426+00:00  2024-07-09 06:00:12.422987+00:00       ETH_USD  2024     7    9\n",
       "2024-07-09 06:00:12.596000+00:00  1720504812596   1.6808   3067.27      4.5    3067.5   cryptocom     1  2024-07-09 06:00:12.611365+00:00  2024-07-09 06:00:12.628991+00:00       ETH_USD  2024     7    9\n",
       "2024-07-09 06:00:12.898000+00:00  1720504812898   0.4201   3067.39   0.0323   3067.49   cryptocom     1  2024-07-09 06:00:12.901371+00:00  2024-07-09 06:00:12.914780+00:00       ETH_USD  2024     7    9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.parquet.bid_ask.futures.v7_5.ccxt.cryptocom.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\", add_suffix=\"tokyo\")\n",
    "start_timestamp = pd.Timestamp(\"2024-07-09T06:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=1)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55df21d0",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v7_5.ccxt.cryptocom.v2_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5f1a673b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-09T18:52:00.207854Z",
     "start_time": "2024-07-09T18:51:53.698434Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>level_1.bid_price.open</th>\n",
       "      <th>level_1.bid_size.open</th>\n",
       "      <th>level_1.ask_price.open</th>\n",
       "      <th>level_1.ask_size.open</th>\n",
       "      <th>level_1.bid_ask_midpoint.open</th>\n",
       "      <th>level_1.half_spread.open</th>\n",
       "      <th>level_1.log_size_imbalance.open</th>\n",
       "      <th>level_1.bid_price.close</th>\n",
       "      <th>level_1.bid_size.close</th>\n",
       "      <th>level_1.ask_price.close</th>\n",
       "      <th>level_1.ask_size.close</th>\n",
       "      <th>level_1.bid_ask_midpoint.close</th>\n",
       "      <th>level_1.half_spread.close</th>\n",
       "      <th>level_1.log_size_imbalance.close</th>\n",
       "      <th>level_1.bid_price.high</th>\n",
       "      <th>level_1.bid_size.max</th>\n",
       "      <th>level_1.ask_price.high</th>\n",
       "      <th>level_1.ask_size.max</th>\n",
       "      <th>level_1.bid_ask_midpoint.max</th>\n",
       "      <th>level_1.half_spread.max</th>\n",
       "      <th>level_1.log_size_imbalance.max</th>\n",
       "      <th>level_1.bid_price.low</th>\n",
       "      <th>level_1.bid_size.min</th>\n",
       "      <th>level_1.ask_price.low</th>\n",
       "      <th>level_1.ask_size.min</th>\n",
       "      <th>level_1.bid_ask_midpoint.min</th>\n",
       "      <th>level_1.half_spread.min</th>\n",
       "      <th>level_1.log_size_imbalance.min</th>\n",
       "      <th>level_1.bid_price.mean</th>\n",
       "      <th>level_1.bid_size.mean</th>\n",
       "      <th>level_1.ask_price.mean</th>\n",
       "      <th>level_1.ask_size.mean</th>\n",
       "      <th>level_1.bid_ask_midpoint.mean</th>\n",
       "      <th>level_1.half_spread.mean</th>\n",
       "      <th>level_1.log_size_imbalance.mean</th>\n",
       "      <th>level_1.bid_ask_midpoint_var.100ms</th>\n",
       "      <th>level_1.bid_ask_midpoint_autocovar.100ms</th>\n",
       "      <th>level_1.log_size_imbalance_var.100ms</th>\n",
       "      <th>level_1.log_size_imbalance_autocovar.100ms</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:00:00+00:00</th>\n",
       "      <td>1720468800000</td>\n",
       "      <td>56475.8</td>\n",
       "      <td>0.101</td>\n",
       "      <td>56483.3</td>\n",
       "      <td>0.4435</td>\n",
       "      <td>56479.55</td>\n",
       "      <td>3.75</td>\n",
       "      <td>-1.479577</td>\n",
       "      <td>56466.0</td>\n",
       "      <td>0.152</td>\n",
       "      <td>56468.4</td>\n",
       "      <td>0.152</td>\n",
       "      <td>56467.2</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>56510.6</td>\n",
       "      <td>0.9557</td>\n",
       "      <td>56516.4</td>\n",
       "      <td>0.6056</td>\n",
       "      <td>56513.5</td>\n",
       "      <td>5.35</td>\n",
       "      <td>3.667353</td>\n",
       "      <td>56450.0</td>\n",
       "      <td>0.0057</td>\n",
       "      <td>56456.9</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>56453.85</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-2.416853</td>\n",
       "      <td>56485.787167</td>\n",
       "      <td>0.125292</td>\n",
       "      <td>56488.508667</td>\n",
       "      <td>0.149978</td>\n",
       "      <td>56487.147917</td>\n",
       "      <td>1.36075</td>\n",
       "      <td>-0.03603</td>\n",
       "      <td>2003.6875</td>\n",
       "      <td>0.0</td>\n",
       "      <td>712.734912</td>\n",
       "      <td>464.278884</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:01:00+00:00</th>\n",
       "      <td>1720468860000</td>\n",
       "      <td>56466.0</td>\n",
       "      <td>0.152</td>\n",
       "      <td>56468.3</td>\n",
       "      <td>0.1418</td>\n",
       "      <td>56467.15</td>\n",
       "      <td>1.15</td>\n",
       "      <td>0.069463</td>\n",
       "      <td>56453.8</td>\n",
       "      <td>0.0735</td>\n",
       "      <td>56461.2</td>\n",
       "      <td>0.0491</td>\n",
       "      <td>56457.5</td>\n",
       "      <td>3.7</td>\n",
       "      <td>0.403426</td>\n",
       "      <td>56487.6</td>\n",
       "      <td>0.1969</td>\n",
       "      <td>56491.9</td>\n",
       "      <td>0.2242</td>\n",
       "      <td>56489.4</td>\n",
       "      <td>6.2</td>\n",
       "      <td>3.809949</td>\n",
       "      <td>56428.1</td>\n",
       "      <td>0.0042</td>\n",
       "      <td>56430.0</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>56429.25</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-3.191276</td>\n",
       "      <td>56453.558833</td>\n",
       "      <td>0.075335</td>\n",
       "      <td>56458.254333</td>\n",
       "      <td>0.070159</td>\n",
       "      <td>56455.906583</td>\n",
       "      <td>2.34775</td>\n",
       "      <td>0.177464</td>\n",
       "      <td>2378.395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>782.276003</td>\n",
       "      <td>606.965937</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:02:00+00:00</th>\n",
       "      <td>1720468920000</td>\n",
       "      <td>56453.8</td>\n",
       "      <td>0.0735</td>\n",
       "      <td>56461.2</td>\n",
       "      <td>0.0491</td>\n",
       "      <td>56457.5</td>\n",
       "      <td>3.7</td>\n",
       "      <td>0.403426</td>\n",
       "      <td>56425.5</td>\n",
       "      <td>0.1057</td>\n",
       "      <td>56430.7</td>\n",
       "      <td>0.1297</td>\n",
       "      <td>56428.1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>-0.204619</td>\n",
       "      <td>56467.3</td>\n",
       "      <td>0.243</td>\n",
       "      <td>56475.1</td>\n",
       "      <td>0.3361</td>\n",
       "      <td>56470.5</td>\n",
       "      <td>5.5</td>\n",
       "      <td>2.571849</td>\n",
       "      <td>56411.1</td>\n",
       "      <td>0.0037</td>\n",
       "      <td>56415.3</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>56413.2</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-3.857024</td>\n",
       "      <td>56430.5495</td>\n",
       "      <td>0.063856</td>\n",
       "      <td>56436.955667</td>\n",
       "      <td>0.07561</td>\n",
       "      <td>56433.752583</td>\n",
       "      <td>3.203083</td>\n",
       "      <td>-0.113241</td>\n",
       "      <td>637.945</td>\n",
       "      <td>0.0</td>\n",
       "      <td>662.704235</td>\n",
       "      <td>470.1478</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>BTC_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:03:00+00:00</th>\n",
       "      <td>1720468980000</td>\n",
       "      <td>3001.62</td>\n",
       "      <td>0.7</td>\n",
       "      <td>3001.8</td>\n",
       "      <td>8.2</td>\n",
       "      <td>3001.71</td>\n",
       "      <td>0.09</td>\n",
       "      <td>-2.460809</td>\n",
       "      <td>3003.47</td>\n",
       "      <td>2.9</td>\n",
       "      <td>3003.48</td>\n",
       "      <td>4.9</td>\n",
       "      <td>3003.475</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-0.524524</td>\n",
       "      <td>3003.96</td>\n",
       "      <td>6.2</td>\n",
       "      <td>3004.08</td>\n",
       "      <td>14.984</td>\n",
       "      <td>3003.99</td>\n",
       "      <td>0.235</td>\n",
       "      <td>2.688848</td>\n",
       "      <td>3001.62</td>\n",
       "      <td>0.0173</td>\n",
       "      <td>3001.74</td>\n",
       "      <td>0.333</td>\n",
       "      <td>3001.68</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-5.646284</td>\n",
       "      <td>3002.765133</td>\n",
       "      <td>2.455578</td>\n",
       "      <td>3002.865383</td>\n",
       "      <td>3.272069</td>\n",
       "      <td>3002.815258</td>\n",
       "      <td>0.050125</td>\n",
       "      <td>-0.377865</td>\n",
       "      <td>4.459375</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1162.261698</td>\n",
       "      <td>923.014995</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:04:00+00:00</th>\n",
       "      <td>1720469040000</td>\n",
       "      <td>3003.47</td>\n",
       "      <td>2.9</td>\n",
       "      <td>3003.48</td>\n",
       "      <td>4.9</td>\n",
       "      <td>3003.475</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-0.524524</td>\n",
       "      <td>2999.43</td>\n",
       "      <td>6.1988</td>\n",
       "      <td>2999.64</td>\n",
       "      <td>7.7</td>\n",
       "      <td>2999.535</td>\n",
       "      <td>0.105</td>\n",
       "      <td>-0.216865</td>\n",
       "      <td>3003.47</td>\n",
       "      <td>10.2</td>\n",
       "      <td>3003.48</td>\n",
       "      <td>14.3</td>\n",
       "      <td>3003.475</td>\n",
       "      <td>0.235</td>\n",
       "      <td>3.578531</td>\n",
       "      <td>2999.12</td>\n",
       "      <td>0.23</td>\n",
       "      <td>2999.37</td>\n",
       "      <td>0.0335</td>\n",
       "      <td>2999.245</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-4.129936</td>\n",
       "      <td>3000.6655</td>\n",
       "      <td>2.496191</td>\n",
       "      <td>3000.81255</td>\n",
       "      <td>4.095613</td>\n",
       "      <td>3000.739025</td>\n",
       "      <td>0.073525</td>\n",
       "      <td>-0.423312</td>\n",
       "      <td>4.24915</td>\n",
       "      <td>-0.0072</td>\n",
       "      <td>1150.490304</td>\n",
       "      <td>978.441419</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-08 20:05:00+00:00</th>\n",
       "      <td>1720469100000</td>\n",
       "      <td>2999.43</td>\n",
       "      <td>6.1988</td>\n",
       "      <td>2999.58</td>\n",
       "      <td>2.2</td>\n",
       "      <td>2999.505</td>\n",
       "      <td>0.075</td>\n",
       "      <td>1.035898</td>\n",
       "      <td>2998.44</td>\n",
       "      <td>0.4</td>\n",
       "      <td>2998.5</td>\n",
       "      <td>1.3</td>\n",
       "      <td>2998.47</td>\n",
       "      <td>0.03</td>\n",
       "      <td>-1.178655</td>\n",
       "      <td>3000.31</td>\n",
       "      <td>9.8273</td>\n",
       "      <td>3000.66</td>\n",
       "      <td>16.214</td>\n",
       "      <td>3000.485</td>\n",
       "      <td>0.33</td>\n",
       "      <td>4.831294</td>\n",
       "      <td>2997.19</td>\n",
       "      <td>0.0026</td>\n",
       "      <td>2997.3</td>\n",
       "      <td>0.0334</td>\n",
       "      <td>2997.245</td>\n",
       "      <td>0.005</td>\n",
       "      <td>-7.18425</td>\n",
       "      <td>2999.193817</td>\n",
       "      <td>2.423717</td>\n",
       "      <td>2999.313817</td>\n",
       "      <td>3.926643</td>\n",
       "      <td>2999.253817</td>\n",
       "      <td>0.06</td>\n",
       "      <td>-0.391764</td>\n",
       "      <td>4.577625</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1813.302223</td>\n",
       "      <td>1414.070233</td>\n",
       "      <td>cryptocom</td>\n",
       "      <td>2024-07-09 04:22:35.212411+00:00</td>\n",
       "      <td>ETH_USD</td>\n",
       "      <td>2024</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               timestamp level_1.bid_price.open level_1.bid_size.open level_1.ask_price.open level_1.ask_size.open level_1.bid_ask_midpoint.open level_1.half_spread.open level_1.log_size_imbalance.open level_1.bid_price.close level_1.bid_size.close level_1.ask_price.close level_1.ask_size.close level_1.bid_ask_midpoint.close level_1.half_spread.close level_1.log_size_imbalance.close level_1.bid_price.high level_1.bid_size.max level_1.ask_price.high level_1.ask_size.max level_1.bid_ask_midpoint.max level_1.half_spread.max level_1.log_size_imbalance.max level_1.bid_price.low level_1.bid_size.min level_1.ask_price.low level_1.ask_size.min level_1.bid_ask_midpoint.min level_1.half_spread.min level_1.log_size_imbalance.min level_1.bid_price.mean level_1.bid_size.mean level_1.ask_price.mean level_1.ask_size.mean level_1.bid_ask_midpoint.mean level_1.half_spread.mean level_1.log_size_imbalance.mean level_1.bid_ask_midpoint_var.100ms level_1.bid_ask_midpoint_autocovar.100ms level_1.log_size_imbalance_var.100ms level_1.log_size_imbalance_autocovar.100ms exchange_id               knowledge_timestamp currency_pair  year month\n",
       "2024-07-08 20:00:00+00:00  1720468800000                56475.8                 0.101                56483.3                0.4435                      56479.55                     3.75                       -1.479577                 56466.0                  0.152                 56468.4                  0.152                        56467.2                       1.2                              0.0                56510.6               0.9557                56516.4               0.6056                      56513.5                    5.35                       3.667353               56450.0               0.0057               56456.9               0.0027                     56453.85                    0.05                      -2.416853           56485.787167              0.125292           56488.508667              0.149978                  56487.147917                  1.36075                        -0.03603                          2003.6875                                      0.0                           712.734912                                 464.278884   cryptocom  2024-07-09 04:22:35.212411+00:00       BTC_USD  2024     7\n",
       "2024-07-08 20:01:00+00:00  1720468860000                56466.0                 0.152                56468.3                0.1418                      56467.15                     1.15                        0.069463                 56453.8                 0.0735                 56461.2                 0.0491                        56457.5                       3.7                         0.403426                56487.6               0.1969                56491.9               0.2242                      56489.4                     6.2                       3.809949               56428.1               0.0042               56430.0               0.0027                     56429.25                    0.05                      -3.191276           56453.558833              0.075335           56458.254333              0.070159                  56455.906583                  2.34775                        0.177464                           2378.395                                      0.0                           782.276003                                 606.965937   cryptocom  2024-07-09 04:22:35.212411+00:00       BTC_USD  2024     7\n",
       "2024-07-08 20:02:00+00:00  1720468920000                56453.8                0.0735                56461.2                0.0491                       56457.5                      3.7                        0.403426                 56425.5                 0.1057                 56430.7                 0.1297                        56428.1                       2.6                        -0.204619                56467.3                0.243                56475.1               0.3361                      56470.5                     5.5                       2.571849               56411.1               0.0037               56415.3               0.0027                      56413.2                    0.05                      -3.857024             56430.5495              0.063856           56436.955667               0.07561                  56433.752583                 3.203083                       -0.113241                            637.945                                      0.0                           662.704235                                   470.1478   cryptocom  2024-07-09 04:22:35.212411+00:00       BTC_USD  2024     7\n",
       "                                     ...                    ...                   ...                    ...                   ...                           ...                      ...                             ...                     ...                    ...                     ...                    ...                            ...                       ...                              ...                    ...                  ...                    ...                  ...                          ...                     ...                            ...                   ...                  ...                   ...                  ...                          ...                     ...                            ...                    ...                   ...                    ...                   ...                           ...                      ...                             ...                                ...                                      ...                                  ...                                        ...         ...                               ...           ...   ...   ...\n",
       "2024-07-08 20:03:00+00:00  1720468980000                3001.62                   0.7                 3001.8                   8.2                       3001.71                     0.09                       -2.460809                 3003.47                    2.9                 3003.48                    4.9                       3003.475                     0.005                        -0.524524                3003.96                  6.2                3004.08               14.984                      3003.99                   0.235                       2.688848               3001.62               0.0173               3001.74                0.333                      3001.68                   0.005                      -5.646284            3002.765133              2.455578            3002.865383              3.272069                   3002.815258                 0.050125                       -0.377865                           4.459375                                      0.0                          1162.261698                                 923.014995   cryptocom  2024-07-09 04:22:35.212411+00:00       ETH_USD  2024     7\n",
       "2024-07-08 20:04:00+00:00  1720469040000                3003.47                   2.9                3003.48                   4.9                      3003.475                    0.005                       -0.524524                 2999.43                 6.1988                 2999.64                    7.7                       2999.535                     0.105                        -0.216865                3003.47                 10.2                3003.48                 14.3                     3003.475                   0.235                       3.578531               2999.12                 0.23               2999.37               0.0335                     2999.245                   0.005                      -4.129936              3000.6655              2.496191             3000.81255              4.095613                   3000.739025                 0.073525                       -0.423312                            4.24915                                  -0.0072                          1150.490304                                 978.441419   cryptocom  2024-07-09 04:22:35.212411+00:00       ETH_USD  2024     7\n",
       "2024-07-08 20:05:00+00:00  1720469100000                2999.43                6.1988                2999.58                   2.2                      2999.505                    0.075                        1.035898                 2998.44                    0.4                  2998.5                    1.3                        2998.47                      0.03                        -1.178655                3000.31               9.8273                3000.66               16.214                     3000.485                    0.33                       4.831294               2997.19               0.0026                2997.3               0.0334                     2997.245                   0.005                       -7.18425            2999.193817              2.423717            2999.313817              3.926643                   2999.253817                     0.06                       -0.391764                           4.577625                                      0.0                          1813.302223                                1414.070233   cryptocom  2024-07-09 04:22:35.212411+00:00       ETH_USD  2024     7"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v7_5.ccxt.cryptocom.v2_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\", add_suffix=\"tokyo\")\n",
    "start_timestamp = pd.Timestamp(\"2024-07-08T20:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=5)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e87caed",
   "metadata": {},
   "source": [
    "# Semi-Active datasets\n",
    "\n",
    " - not currently downloaded but could be revived"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c38165",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.archived_200ms.parquet.bid_ask.spot.v7_4.ccxt.kraken.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7e40dd0a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:47:21.816397Z",
     "start_time": "2024-05-16T19:47:14.040966Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Before from_parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>bid_size</th>\n",
       "      <th>bid_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>ask_price</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>level</th>\n",
       "      <th>end_download_timestamp</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.153000+00:00</th>\n",
       "      <td>1704067200153</td>\n",
       "      <td>0.0712</td>\n",
       "      <td>42289.1</td>\n",
       "      <td>0.036078</td>\n",
       "      <td>42289.2</td>\n",
       "      <td>kraken</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-01-01 00:00:00.252015+00:00</td>\n",
       "      <td>2024-01-01 00:00:01.750584+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.293000+00:00</th>\n",
       "      <td>1704067200293</td>\n",
       "      <td>0.17413</td>\n",
       "      <td>42289.1</td>\n",
       "      <td>0.036078</td>\n",
       "      <td>42289.2</td>\n",
       "      <td>kraken</td>\n",
       "      <td>1</td>\n",
       "      <td>2024-01-01 00:00:00.453643+00:00</td>\n",
       "      <td>2024-01-01 00:00:01.750584+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.293000+00:00</th>\n",
       "      <td>1704067200293</td>\n",
       "      <td>0.0712</td>\n",
       "      <td>42284.8</td>\n",
       "      <td>0.014123</td>\n",
       "      <td>42289.3</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2</td>\n",
       "      <td>2024-01-01 00:00:00.453643+00:00</td>\n",
       "      <td>2024-01-01 00:00:01.750584+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:56.609000+00:00</th>\n",
       "      <td>1704067256609</td>\n",
       "      <td>0.013656</td>\n",
       "      <td>2281.72</td>\n",
       "      <td>0.5906</td>\n",
       "      <td>2282.32</td>\n",
       "      <td>kraken</td>\n",
       "      <td>4</td>\n",
       "      <td>2024-01-01 00:00:56.739928+00:00</td>\n",
       "      <td>2024-01-01 00:01:00.823556+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:56.609000+00:00</th>\n",
       "      <td>1704067256609</td>\n",
       "      <td>1.087427</td>\n",
       "      <td>2281.7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2282.43</td>\n",
       "      <td>kraken</td>\n",
       "      <td>5</td>\n",
       "      <td>2024-01-01 00:00:56.739928+00:00</td>\n",
       "      <td>2024-01-01 00:01:00.823556+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:56.609000+00:00</th>\n",
       "      <td>1704067256609</td>\n",
       "      <td>0.042971</td>\n",
       "      <td>2281.47</td>\n",
       "      <td>0.755004</td>\n",
       "      <td>2282.44</td>\n",
       "      <td>kraken</td>\n",
       "      <td>6</td>\n",
       "      <td>2024-01-01 00:00:56.739928+00:00</td>\n",
       "      <td>2024-01-01 00:01:00.823556+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      timestamp  bid_size bid_price  ask_size ask_price exchange_id level            end_download_timestamp               knowledge_timestamp currency_pair  year month  day\n",
       "2024-01-01 00:00:00.153000+00:00  1704067200153    0.0712   42289.1  0.036078   42289.2      kraken     1  2024-01-01 00:00:00.252015+00:00  2024-01-01 00:00:01.750584+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:00.293000+00:00  1704067200293   0.17413   42289.1  0.036078   42289.2      kraken     1  2024-01-01 00:00:00.453643+00:00  2024-01-01 00:00:01.750584+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:00.293000+00:00  1704067200293    0.0712   42284.8  0.014123   42289.3      kraken     2  2024-01-01 00:00:00.453643+00:00  2024-01-01 00:00:01.750584+00:00      BTC_USDT  2024     1    1\n",
       "                                            ...       ...       ...       ...       ...         ...   ...                               ...                               ...           ...   ...   ...  ...\n",
       "2024-01-01 00:00:56.609000+00:00  1704067256609  0.013656   2281.72    0.5906   2282.32      kraken     4  2024-01-01 00:00:56.739928+00:00  2024-01-01 00:01:00.823556+00:00      ETH_USDT  2024     1    1\n",
       "2024-01-01 00:00:56.609000+00:00  1704067256609  1.087427    2281.7       1.0   2282.43      kraken     5  2024-01-01 00:00:56.739928+00:00  2024-01-01 00:01:00.823556+00:00      ETH_USDT  2024     1    1\n",
       "2024-01-01 00:00:56.609000+00:00  1704067256609  0.042971   2281.47  0.755004   2282.44      kraken     6  2024-01-01 00:00:56.739928+00:00  2024-01-01 00:01:00.823556+00:00      ETH_USDT  2024     1    1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.archived_200ms.parquet.bid_ask.spot.v7_4.ccxt.kraken.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-01-01T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=1)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\", \"ETH_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ffe0b11",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_all.parquet.trades.spot.v7_4.ccxt.kraken.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8cdce9de",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-16T19:47:26.428107Z",
     "start_time": "2024-05-16T19:47:21.819269Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Loading dataset schema file: /app/amp/data_schema/dataset_schema_versions/dataset_schema_v3.json\n",
      "INFO  Loaded dataset schema version v3\n",
      "INFO  Before from_parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>symbol</th>\n",
       "      <th>side</th>\n",
       "      <th>price</th>\n",
       "      <th>amount</th>\n",
       "      <th>exchange_id</th>\n",
       "      <th>knowledge_timestamp</th>\n",
       "      <th>currency_pair</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:00.006000+00:00</th>\n",
       "      <td>1704067200006</td>\n",
       "      <td>BTC/USDT</td>\n",
       "      <td>buy</td>\n",
       "      <td>42289.2</td>\n",
       "      <td>0.013922</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:51.994445+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:02.738000+00:00</th>\n",
       "      <td>1704067202738</td>\n",
       "      <td>BTC/USDT</td>\n",
       "      <td>buy</td>\n",
       "      <td>42289.2</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:51.994445+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:40.338000+00:00</th>\n",
       "      <td>1704067240338</td>\n",
       "      <td>BTC/USDT</td>\n",
       "      <td>buy</td>\n",
       "      <td>42293.3</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:51.994445+00:00</td>\n",
       "      <td>BTC_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:12.209000+00:00</th>\n",
       "      <td>1704067212209</td>\n",
       "      <td>ETH/USDT</td>\n",
       "      <td>sell</td>\n",
       "      <td>2281.4</td>\n",
       "      <td>0.126</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:44.902187+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:56.768000+00:00</th>\n",
       "      <td>1704067256768</td>\n",
       "      <td>ETH/USDT</td>\n",
       "      <td>buy</td>\n",
       "      <td>2282.17</td>\n",
       "      <td>0.056</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:44.902187+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-01-01 00:00:58.088000+00:00</th>\n",
       "      <td>1704067258088</td>\n",
       "      <td>ETH/USDT</td>\n",
       "      <td>buy</td>\n",
       "      <td>2282.19</td>\n",
       "      <td>0.001702</td>\n",
       "      <td>kraken</td>\n",
       "      <td>2024-01-02 02:21:44.902187+00:00</td>\n",
       "      <td>ETH_USDT</td>\n",
       "      <td>2024</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      timestamp    symbol  side    price    amount exchange_id               knowledge_timestamp currency_pair  year month  day\n",
       "2024-01-01 00:00:00.006000+00:00  1704067200006  BTC/USDT   buy  42289.2  0.013922      kraken  2024-01-02 02:21:51.994445+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:02.738000+00:00  1704067202738  BTC/USDT   buy  42289.2    0.0001      kraken  2024-01-02 02:21:51.994445+00:00      BTC_USDT  2024     1    1\n",
       "2024-01-01 00:00:40.338000+00:00  1704067240338  BTC/USDT   buy  42293.3    0.0001      kraken  2024-01-02 02:21:51.994445+00:00      BTC_USDT  2024     1    1\n",
       "                                            ...       ...   ...      ...       ...         ...                               ...           ...   ...   ...  ...\n",
       "2024-01-01 00:00:12.209000+00:00  1704067212209  ETH/USDT  sell   2281.4     0.126      kraken  2024-01-02 02:21:44.902187+00:00      ETH_USDT  2024     1    1\n",
       "2024-01-01 00:00:56.768000+00:00  1704067256768  ETH/USDT   buy  2282.17     0.056      kraken  2024-01-02 02:21:44.902187+00:00      ETH_USDT  2024     1    1\n",
       "2024-01-01 00:00:58.088000+00:00  1704067258088  ETH/USDT   buy  2282.19  0.001702      kraken  2024-01-02 02:21:44.902187+00:00      ETH_USDT  2024     1    1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  None\n"
     ]
    }
   ],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_all.parquet.trades.spot.v7_4.ccxt.kraken.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "start_timestamp = pd.Timestamp(\"2024-01-01T00:00:00+00:00\")\n",
    "end_timestamp = start_timestamp + pd.Timedelta(minutes=1)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=[\"BTC_USDT\", \"ETH_USDT\"]\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "032b6797",
   "metadata": {},
   "source": [
    "# Legacy datasets - datasets not actively maintained"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da242b64",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.csv.ohlcv.futures.v7.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5d2df79",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.051Z"
    }
   },
   "outputs": [],
   "source": [
    "#  the dataset reside under previous, deprecated schema:\n",
    "# s3://cryptokaizen-data/reorg/daily_staged.airflow.pq/\n",
    "\n",
    "# signature = \"periodic_daily.airflow.downloaded_1min.csv.ohlcv.futures.v7.ccxt.binance.v1_0_0\"\n",
    "# reader = imvcdcimrdc.RawDataReader(signature)\n",
    "# data = reader.read_data_head()\n",
    "# _LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00b88afa",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8984f851",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.052Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "741c3ad6",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7.ccxt.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebfe655d",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.053Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded4bb0c",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7.ccxt.binanceus.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "006827a7",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.054Z"
    },
    "run_control": {
     "marked": true
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v7.ccxt.binanceus.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b33825",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "041be0fc",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.055Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v3.crypto_chassis.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "090371c1",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1sec.parquet.bid_ask.futures.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "862cff6b",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.056Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1sec.parquet.bid_ask.futures.v3.crypto_chassis.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95dd001a",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dde315db",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.057Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.resampled_1min.parquet.bid_ask.futures.v3.crypto_chassis.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93e3501c",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1sec.parquet.bid_ask.spot.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08d72270",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.058Z"
    }
   },
   "outputs": [],
   "source": [
    "# The dataset reside under previous schema:\n",
    "# s3://cryptokaizen-data/reorg/daily_staged.airflow.pq/\n",
    "\n",
    "# TODO(Juraj): Spot bid ask spot data are not collected currently\n",
    "# signature = \"periodic_daily.airflow.downloaded_1sec.parquet.bid_ask.spot.v3.crypto_chassis.binance.v1_0_0\"\n",
    "# reader = imvcdcimrdc.RawDataReader(signature)\n",
    "# data = reader.read_data_head()\n",
    "# _LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1c8e42",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.resampled_1min.parquet.bid_ask.spot.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40548bf0",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.059Z"
    }
   },
   "outputs": [],
   "source": [
    "# The dataset reside under previous schema:\n",
    "# s3://cryptokaizen-data/reorg/daily_staged.airflow.pq/\n",
    "\n",
    "# TODO(Juraj): Spot bid ask spot data are not collected currently\n",
    "# signature = \"periodic_daily.airflow.downloaded_1sec.parquet.bid_ask.spot.v3.crypto_chassis.binance.v1_0_0\"\n",
    "# reader = imvcdcimrdc.RawDataReader(signature)\n",
    "# data = reader.read_data_head()\n",
    "# _LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccecad4c",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v3.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a50d187",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.060Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.spot.v3.crypto_chassis.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26364c58",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1sec.parquet.trades.futures.v3_1.crypto_chassis.binance.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb58fe7c",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.061Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1sec.parquet.trades.futures.v3_1.crypto_chassis.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96db3238",
   "metadata": {},
   "source": [
    "## periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_3.ccxt.okx.v1_0_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e9a4c3c",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.062Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_3.ccxt.okx.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data_head()\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22298ad0",
   "metadata": {},
   "source": [
    "## bulk.airflow.downloaded_1min.parquet.ohlcv.futures.v7_5.ccxt.binance.v1_0_0\n",
    "\n",
    "_This dataset is in the test stage only_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf081293",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.063Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = (\n",
    "    \"bulk.airflow.downloaded_1min.parquet.ohlcv.futures.v7_5.ccxt.binance.v1_0_0\"\n",
    ")\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"test\")\n",
    "# 4 months of data is available.\n",
    "start_timestamp = pd.Timestamp(\"2023-02-01T00:00:00+00:00\")\n",
    "end_timestamp = pd.Timestamp(\"2023-06-01T00:00:00+00:00\")\n",
    "binance_ohlcv_data = reader.read_data(start_timestamp, end_timestamp)\n",
    "_LOG.log(\n",
    "    log_level, hpandas.df_to_str(binance_ohlcv_data.head(), log_level=log_level)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "137e94ac",
   "metadata": {},
   "source": [
    "# RawDataReader Guide"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e84b7477",
   "metadata": {},
   "source": [
    "## Loading parquet data with filters\n",
    "\n",
    "TODO(Juraj): Support for filtering by level for parquet bid/ask datasets will be added once\n",
    "#3694 is finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b32dcfd",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.064Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7.ccxt.binance.v1_0_0\"\n",
    "start_timestamp = pd.Timestamp(\n",
    "    datetime.datetime.utcnow() - datetime.timedelta(minutes=10, days=2), tz=\"UTC\"\n",
    ")\n",
    "end_timestamp = start_timestamp + datetime.timedelta(minutes=10)\n",
    "currency_pairs = [\"BTC_USDT\", \"ETH_USDT\"]\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=currency_pairs\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1afbf07",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.065Z"
    }
   },
   "outputs": [],
   "source": [
    "# This works with stage preprod.\n",
    "signature = \"periodic_daily.airflow.downloaded_1min.parquet.ohlcv.futures.v7_6.ccxt.okx.v1_0_0\"\n",
    "start_timestamp = pd.Timestamp(\n",
    "    datetime.datetime.utcnow() - datetime.timedelta(minutes=10, days=1), tz=\"UTC\"\n",
    ")\n",
    "end_timestamp = start_timestamp + datetime.timedelta(minutes=10)\n",
    "currency_pairs = [\"BTC_USDT\", \"ETH_USDT\"]\n",
    "reader = imvcdcimrdc.RawDataReader(signature, stage=\"preprod\")\n",
    "data = reader.read_data(\n",
    "    start_timestamp, end_timestamp, currency_pairs=currency_pairs\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de546067",
   "metadata": {},
   "source": [
    "## Loading postgres data with filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af6d7569",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-16T19:27:57.066Z"
    }
   },
   "outputs": [],
   "source": [
    "signature = \"realtime.airflow.resampled_1min.postgres.bid_ask.futures.v7.ccxt.binance.v1_0_0\"\n",
    "reader = imvcdcimrdc.RawDataReader(signature)\n",
    "start_timestamp = pd.Timestamp(\n",
    "    datetime.datetime.utcnow() - datetime.timedelta(minutes=10, days=2), tz=\"UTC\"\n",
    ")\n",
    "end_timestamp = start_timestamp + datetime.timedelta(minutes=10)\n",
    "currency_pairs = [\"BTC_USDT\", \"ETH_USDT\"]\n",
    "bid_ask_levels = [1, 2]\n",
    "data = reader.read_data(\n",
    "    start_timestamp,\n",
    "    end_timestamp,\n",
    "    currency_pairs=currency_pairs,\n",
    "    bid_ask_levels=bid_ask_levels,\n",
    ")\n",
    "_LOG.log(log_level, hpandas.df_to_str(data, log_level=log_level))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "242px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}