08_SmallerNLIModels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read this!\n",
    "\n",
    "This notebook is mostly a copy of `07_ZeroShotTopicClassificationCORD19.ipynb`.\n",
    "The main difference is that this notebook implements the Zero-Shot NLI method with a smaller pre-trained BERT model, as opposed to the original notebook, that implements the method with a large pre-trained BART model.\n",
    "\n",
    "# Zero Shot Topic Classification on CORD-19\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook we'll build a Zero Shot Topic Classifier on the COVID-19 Open Research Dataset (CORD-19, Wang et al., 2020).\n",
    "Essentially, we aim to build a web application capable of receiving natural language questions, such as \"what do we know about vaccines and therapeutics?\", and then displaying the most relevant research literature regarding the specific question.\n",
    "This dataset has received wide attention in the data mining and natural language processing community in order to develop tools to aid health workers stay up-to-date with the latest and most relevant research about the current pandemic.\n",
    "\n",
    "Recent advances in NLP, such as OpenAI's GPT-3 (Brown et al., 2020), have shown that large language models can achieve competitive performance on downstream tasks with less task-specific data than it'd be required by smaller models.\n",
    "However, GPT-3 is currently difficult to use on real world applications due to its size of ~175 billions of parameters.\n",
    "\n",
    "Recent experiments made at HuggingFace (Davison, 2020) explored the potential of using Sentence-BERT (Reimers and Gurevych, 2020) to separately embed sentences and never-seen-before topic labels.\n",
    "Then, they'd rank the sentence's topics by measuring the cosine distance between both vectors (Veeranna, 2016), obtaining promising results.\n",
    "\n",
    "In another experiment, they use a pre-trained natural languange inference (NLI) sequence-pair classifier as an out of-the-box zero shot text classifier, as proposed by Yin et al. (2020).\n",
    "By using a pre-trained BART model (Lewis et al., 2019) fine-tuned on the Multigenre NLI corpus, they were able to score an F1 score of 53.7 on the Yahoo News dataset.\n",
    "The dataset has 10 classes and the current supervised models state of the art is an accuracy of 77.62."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: CUDA_VISIBLE_DEVICES=6\n"
     ]
    }
   ],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp zero_shot_nli_small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all_flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Inference (NLI) method\n",
    "\n",
    "In this approach we use a BART classifier (Lewis et al., 2019) pre-trained on the Multi-Genre NLI (MultiNLI, Williams et al., 2018) corpus as the base model.\n",
    "\n",
    "Given research interests expressed in natural language, we pose the problem of recovering relevant research from the CORD-19 dataset (Wang et al., 2020) as a Zero Shot Topic Classification task (Yin et al., 2019).\n",
    "Leveraging the Natural Language Inference task framework, we assess each paper relevance by feeding the model with the paper's title and abstract as premise and a research interest as hypothesis.\n",
    "\n",
    "Finally, we use the model's entailment inference values as proxy relevance scores for each paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "\n",
    "import math\n",
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "\n",
    "\n",
    "def get_nli_model(name=\"facebook/bart-large-mnli\"):\n",
    "    model = AutoModelForSequenceClassification.from_pretrained(name)\n",
    "    tokenizer = AutoTokenizer.from_pretrained(name)\n",
    "    return model, tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "061967a3177b4b7db89e3ecb7f059409",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1311.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "513aef763aca46ce8a2132f86d45dac2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=437973677.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6a4085c51e246b99cae31f39a55dea9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3929eab6a144ccd9b8ad45825234caf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=112.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "02f91c01e21348cea7c229ff51a420bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=39.0, style=ProgressStyle(description_w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lmarti/risotto/venv-risotto/lib/python3.7/site-packages/transformers/tokenization_utils.py:831: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.\n",
      "  category=FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "from risotto.artifacts import load_papers_artifact\n",
    "\n",
    "try:\n",
    "    papers = load_papers_artifact()\n",
    "    model, tokenizer = get_nli_model(name=\"huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli\")\n",
    "except FileNotFoundError:\n",
    "    print('Data artifacts not ready.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "\n",
    "import numpy as np\n",
    "from fastprogress.fastprogress import progress_bar\n",
    "\n",
    "\n",
    "def build_tokenized_papers_artifact(papers,\n",
    "                                    tokenizer,\n",
    "                                    should_dump=True,\n",
    "                                    dump_path=None,\n",
    "                                    batch_size=128):\n",
    "    num_batches = math.ceil(len(papers) / batch_size)\n",
    "    tokenized_series = pd.Series([], dtype=object, name=\"tokenized_papers\")\n",
    "\n",
    "    for batch_idx in progress_bar(range(num_batches)):\n",
    "        start_idx = batch_idx * batch_size\n",
    "        end_idx = start_idx + batch_size\n",
    "        papers_batch = papers.iloc[start_idx:end_idx]\n",
    "\n",
    "        title_abstract = (papers_batch.title + \". \" +\n",
    "                          papers_batch.abstract).fillna(\" \").values.tolist()\n",
    "        tokenized_batch = tokenizer.batch_encode_plus(\n",
    "            title_abstract, max_length=tokenizer.model_max_length)\n",
    "\n",
    "        for i, (paper_idx, _) in enumerate(papers_batch.iterrows()):\n",
    "            tokenized_series.at[paper_idx] = tokenized_batch[\"input_ids\"][i]\n",
    "\n",
    "    if should_dump:\n",
    "        tokenized_series.to_hdf(dump_path, key=\"tokenized_papers\")\n",
    "\n",
    "    return tokenized_series\n",
    "\n",
    "\n",
    "def load_tokenized_papers_artifact(artifacts_path):\n",
    "    return pd.read_hdf(artifacts_path, key=\"tokenized_papers\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='604' class='' max='604' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [604/604 12:49<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lmarti/risotto/venv-risotto/lib/python3.7/site-packages/pandas/core/generic.py:2505: PerformanceWarning: \n",
      "your performance may suffer as PyTables will pickle object types that it cannot\n",
      "map directly to c-types [inferred_type->mixed,key->values] [items->None]\n",
      "\n",
      "  encoding=encoding,\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ug7v899j    [101, 6612, 2838, 1997, 3226, 1011, 10003, 202...\n",
       "02tnwd4m    [101, 9152, 12412, 15772, 1024, 1037, 4013, 10...\n",
       "ejv2xln0    [101, 14175, 18908, 4630, 5250, 1011, 1040, 19...\n",
       "2b73a28n    [101, 2535, 1997, 2203, 14573, 18809, 1011, 10...\n",
       "9785vg6d    [101, 4962, 3670, 1999, 4958, 8939, 24587, 444...\n",
       "Name: tokenized_papers, dtype: object"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Build tokenized papers\n",
    "tokenized_papers = build_tokenized_papers_artifact(\n",
    "    papers=papers,\n",
    "    tokenizer=tokenizer,\n",
    "    dump_path=\"artifacts/nli_bert_artifacts.hdf\",\n",
    ")\n",
    "tokenized_papers.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ug7v899j    [0, 20868, 1575, 9, 2040, 12, 32012, 1308, 438...\n",
       "02tnwd4m    [0, 19272, 4063, 30629, 35, 10, 1759, 12, 3382...\n",
       "ejv2xln0    [0, 6544, 24905, 927, 8276, 12, 495, 8, 34049,...\n",
       "2b73a28n    [0, 21888, 9, 253, 15244, 2614, 12, 134, 11, 1...\n",
       "9785vg6d    [0, 13120, 8151, 11, 22201, 44828, 4590, 11, 1...\n",
       "Name: tokenized_papers, dtype: object"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load tokenized papers\n",
    "tokenized_papers = load_tokenized_papers_artifact(\n",
    "    \"artifacts/nli_bert_artifacts.hdf\")\n",
    "tokenized_papers.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "\n",
    "import torch\n",
    "\n",
    "\n",
    "def build_entailments_artifact(tokenized_papers,\n",
    "                               query_tokenized,\n",
    "                               batch_size=64,\n",
    "                               device=\"cuda\",\n",
    "                               should_dump=True,\n",
    "                               dump_path=None):\n",
    "    query_encoded = [*query_tokenized[1:]]\n",
    "\n",
    "    model.eval()\n",
    "    model.to(device)\n",
    "\n",
    "    num_batches = math.ceil(len(tokenized_papers) / batch_size)\n",
    "    entail_probs = []\n",
    "\n",
    "    for batch_idx in progress_bar(range(num_batches)):\n",
    "        start_idx = batch_idx * batch_size\n",
    "        end_idx = start_idx + batch_size\n",
    "        tokenized_papers_batch = tokenized_papers.iloc[\n",
    "            start_idx:end_idx].tolist()\n",
    "        max_length = float(\"-inf\")\n",
    "        for i in range(len(tokenized_papers_batch)):\n",
    "            tokenized_paper = tokenized_papers_batch[i]\n",
    "            # Ugly hack...\n",
    "            tokenized_papers_batch[i] = [\n",
    "                *tokenized_paper[:-1][:(tokenizer.model_max_length -\n",
    "                                        len(query_encoded))], *query_encoded\n",
    "            ]\n",
    "            tokenized_paper = tokenized_papers_batch[i]\n",
    "            if len(tokenized_paper) > max_length:\n",
    "                max_length = len(tokenized_paper)\n",
    "        masks = []\n",
    "        token_type_ids = []\n",
    "        for tokenized_paper in tokenized_papers_batch:\n",
    "            paper_length = len(tokenized_paper)\n",
    "            delta = max_length - paper_length\n",
    "\n",
    "            paper_type_ids = [\n",
    "                0 for _ in range(paper_length - len(query_encoded))\n",
    "            ]\n",
    "            paper_type_ids += [1 for _ in range(len(query_encoded))]\n",
    "            paper_type_ids += [0 for _ in range(delta)]\n",
    "            token_type_ids.append(paper_type_ids)\n",
    "\n",
    "            tokenized_paper += [0 for _ in range(delta)]\n",
    "\n",
    "            mask = [1 for _ in range(paper_length)]\n",
    "            mask += [0 for _ in range(delta)]\n",
    "            masks.append(mask)\n",
    "\n",
    "        input_ids = torch.tensor(tokenized_papers_batch).to(device)\n",
    "        token_type_ids = torch.tensor(token_type_ids).to(device)\n",
    "        attention_mask = torch.tensor(masks).to(device)\n",
    "\n",
    "        with torch.no_grad():\n",
    "            outputs = model(input_ids,\n",
    "                            token_type_ids=token_type_ids,\n",
    "                            attention_mask=attention_mask)[0]\n",
    "\n",
    "        entail_contradiction_logits = outputs[:, [0, 1]]\n",
    "        probs = entail_contradiction_logits.softmax(dim=1)\n",
    "        entail_probs += (probs[:, 1] * 100).tolist()\n",
    "\n",
    "    entail_series = pd.Series(entail_probs,\n",
    "                              index=tokenized_papers.index,\n",
    "                              name=\"entailments\")\n",
    "\n",
    "    if should_dump:\n",
    "        entail_series.to_hdf(dump_path, key=\"entailments\")\n",
    "\n",
    "    return entail_series\n",
    "\n",
    "\n",
    "def load_entailments_artifact(artifacts_path):\n",
    "    return pd.read_hdf(artifacts_path, key=\"entailments\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='302' class='' max='302' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [302/302 18:46<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "ug7v899j    34.904873\n",
       "02tnwd4m     3.709159\n",
       "ejv2xln0    83.782410\n",
       "2b73a28n     0.407605\n",
       "9785vg6d    83.543762\n",
       "              ...    \n",
       "2upc2spn    72.468544\n",
       "48kealmj    21.194389\n",
       "7goz1agp    90.725761\n",
       "twp49jg3    78.526169\n",
       "wtoj53xy    10.045611\n",
       "Name: entailments, Length: 77304, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_tokenized = tokenizer.encode(\n",
    "    \"This paper is about vaccines and therapeutics.\")\n",
    "build_entailments_artifact(batch_size=256,\n",
    "                           tokenized_papers=tokenized_papers,\n",
    "                           query_tokenized=query_tokenized,\n",
    "                           dump_path=\"artifacts/nli_bert_artifacts.hdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. https://arxiv.org/abs/2005.14165\n",
    "\n",
    "Davison, J. (2020). Zero-Shot Learning in Modern NLP. https://joeddav.github.io/blog/2020/05/29/ZSL.html\n",
    "\n",
    "Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. http://arxiv.org/abs/1910.13461\n",
    "\n",
    "Reimers, N., & Gurevych, I. (2020). Sentence-BERT: Sentence embeddings using siamese BERT-networks. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 3982–3992. https://doi.org/10.18653/v1/d19-1410\n",
    "\n",
    "Veeranna, S. P., Nam, J., Mencía, E. L., & Fürnkranz, J. (2016). Using semantic similarity for multi-label zero-shot classification of text documents. ESANN 2016 - 24th European Symposium on Artificial Neural Networks, April, 423–428.\n",
    "\n",
    "Wang, L. L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Eide, D., Funk, K., Kinney, R., Liu, Z., Merrill, W., Mooney, P., Murdick, D., Rishi, D., Sheehan, J., Shen, Z., Stilson, B., Wade, A. D., Wang, K., Wilhelm, C., … Kohlmeier, S. (2020). CORD-19: The Covid-19 Open Research Dataset. https://arxiv.org/abs/2004.10706\n",
    "\n",
    "Williams, A., Nangia, N., & Bowman, S. R. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1112--1122. http://aclweb.org/anthology/N18-1101\n",
    "\n",
    "Yin, W., Hay, J., & Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 3914–3923. https://doi.org/10.18653/v1/d19-1404"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}