notebooks/01_dataset.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering Engine\n",
    "\n",
    "## 01: Dataset Creation\n",
    "\n",
    "In the text files provided in the [SimpleQuestions](https://github.com/askplatypus/wikidata-simplequestions) repository, only the files ending with \"answerable\" contain questions that exist in Wikidata, so for this reason I used these to create my dataset.\n",
    "\n",
    "### Libraries\n",
    "\n",
    "Importing the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Formats for datasets\n",
    "import json\n",
    "import csv\n",
    "\n",
    "# Wikidata api\n",
    "from wikidata.client import Client\n",
    "from wikidata.entity import EntityId\n",
    "\n",
    "# Preprocessing\n",
    "from unidecode import unidecode\n",
    "import difflib\n",
    "import spacy\n",
    "nlp = spacy.load('en_core_web_lg')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Span Entity Recognition\n",
    "\n",
    "Then defining the functions to perform the span entity detection based on the entity retrieved from the Wikidata identifier. I preprocess the data to remove accents, 's endings and question marks so that entities are more easily identifiable. Then I use difflib with each of the start and end words of the span to identify them in the question, since I'll be keeping only those indices for the span.\n",
    "\n",
    "For most questions this process is rather straightforward since the entity appears identically or with a small divergence, for example the entity: jonathan rozen appears as: yonatan rozen in the question. But for some they are not lexically similar, but semantically, for example in the question: **Name a model that died due to a car accident** the entity mapped to it is: **traffic collision**.\n",
    "\n",
    "Initially, I approached this problem by using python libraries with synonyms to compare the words, but this wasn't succesful. As can be observed in the example I provided above, car accident and traffic collision have similar meanings, but they are not synonymous. So after experimenting with various apis that capture the semantic content, I decided to use the spacy library that performed the best at this task. This way I rank all words by similarity and then get the one with the highest score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finds best match based on their similarity\n",
    "def find_best_match(entity, question):\n",
    "\n",
    "    # Find similarity scores between entity and each ngram in question\n",
    "    n = len(entity.split())\n",
    "    ngram_scores = {}\n",
    "    for i in range(len(question)-n+1):\n",
    "        ngram = ' '.join(question[i:i+n])\n",
    "        ngram_scores[ngram] = nlp(ngram).similarity(nlp(entity))  \n",
    "    \n",
    "    # Return the highest similarity score ngram as a list\n",
    "    if ngram_scores == {}:\n",
    "        return None\n",
    "    best_score = max(ngram_scores.values())\n",
    "    return [ngram for ngram, score in ngram_scores.items() if score == best_score][0].split()\n",
    "\n",
    "# Finds the entity span of an entity\n",
    "def entity_span(question, entity):\n",
    "\n",
    "    if entity == '' or question =='':\n",
    "        return None\n",
    "\n",
    "    # To locate the position in the question\n",
    "    question_list = question.split()\n",
    "\n",
    "    # Normalize both names in case there are accents\n",
    "    norm_name_1 = unidecode(entity.split()[0])\n",
    "    norm_name_2 = unidecode(entity.split()[-1])\n",
    "\n",
    "    # Find the word with the closest match (eg yonatan for jonathan)\n",
    "    norm_name_1 = difflib.get_close_matches(norm_name_1, question_list)\n",
    "    norm_name_2 = difflib.get_close_matches(norm_name_2, question_list)\n",
    "\n",
    "    # If no close matches perform similarity search (e.g. car accident for traffic collision)\n",
    "    if (norm_name_2 ==[] or norm_name_1 ==[]):\n",
    "        new_entity = find_best_match(entity, question_list)\n",
    "        if new_entity is None:\n",
    "            return None\n",
    "        norm_name_1 = new_entity[0]\n",
    "        norm_name_2 = new_entity[-1]\n",
    "    else:\n",
    "        norm_name_1 = norm_name_1[0]\n",
    "        norm_name_2 = norm_name_2[0]\n",
    "\n",
    "    # Beginning and end of span\n",
    "    entity_start = question_list.index(norm_name_1)\n",
    "    entity_end = question_list.index(norm_name_2)\n",
    "\n",
    "    return entity_start, entity_end"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Ingestion\n",
    "\n",
    "To perform the dataset ingestion I use the Wikidata client to retrieve the entity and relation from their identifiers. The datasets I create have the following format:\n",
    "\n",
    "|  entity_id | ent_label  |relation_id|  rel_label | inverse  | question  |  answer | start  | end|\n",
    "|---|---|---|---|---|---|---|---|---|\n",
    "|Q7358590|roger marquis|P20|place of death|False|Where did roger marquis die|Q1637790|2|3 |\n",
    "\n",
    "Inverse refers to the Rxxx property identifiers encode the inverse property of the Wikidata property Pxxx. For example R19 encodes the properties \"born here\", i.e. the inverse of P19 (\"birth place\").\n",
    "\n",
    "I chose to keep the span entity instead of a binary by having the start and end indices of the span, since I'll be doing the classification this way in the model. Finally, after extracting all the data from the three sets I'm saving them in csv and json formats, as well as the dictionary for the entity to identifier mapping and relation vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The files ending with \"_answerable\" contain only triples that are also in Wikidata.\n",
    "paths = [[ 'dataset/test_dataset.json','wikidata/annotated_wd_data_test_answerable.txt'],\n",
    "         [ 'dataset/val_dataset.json','wikidata/annotated_wd_data_valid_answerable.txt'],\n",
    "         [ 'dataset/train_dataset.json','wikidata/annotated_wd_data_train_answerable.txt']]\n",
    "\n",
    "# Wikidata client session\n",
    "client = Client()\n",
    "\n",
    "relation_vocab = []\n",
    "entity_dict = {}\n",
    "counter = 0\n",
    "\n",
    "# Loop over the input text files\n",
    "for path in paths:\n",
    "    \n",
    "    # Dataset to keep everything\n",
    "    dataset = []\n",
    "    mydataset = path[0]\n",
    "    wikidata = path[1]\n",
    "    \n",
    "    with open(wikidata, 'r', encoding='utf-8') as f:\n",
    "        for line in f:\n",
    "            counter+=1\n",
    "\n",
    "            # Split the line to its parts\n",
    "            parts = line.strip().split('\\t')\n",
    "            \n",
    "            # Extract the question \n",
    "            answer = parts[2]\n",
    "            question = unidecode(parts[3].replace(\"?\", \"\").replace(\"'s\", \"\"))\n",
    "\n",
    "            # Extract the relation and replace the identifiers that encode the inverse property\n",
    "            inverse = False\n",
    "            relation_id = parts[1]\n",
    "            if (relation_id.startswith(\"P\")):\n",
    "                try:\n",
    "                    rel = client.get(EntityId(relation_id))\n",
    "                    relation_label = rel.label\n",
    "                except Exception as e:\n",
    "                    print(\"Error: \", e, \" at: \", counter)\n",
    "                    # handle exceptions here by simply ignoring this row\n",
    "                    continue\n",
    "            if (relation_id.startswith(\"R\")):\n",
    "                try:\n",
    "                    rel = client.get(EntityId(relation_id).replace(\"R\",\"P\"))\n",
    "                    relation_label = rel.label\n",
    "                except Exception as e:\n",
    "                    print(\"Error: \", e, \" at question: \", counter)\n",
    "                    continue\n",
    "                inverse = True\n",
    "\n",
    "            # Extract the entity\n",
    "            entity_id = parts[0]\n",
    "            try:\n",
    "                ent = client.get(EntityId(entity_id))\n",
    "                entity_label = str(ent.label).lower()\n",
    "            except Exception as e:\n",
    "                print(\"Error: \", e, \" at question: \", counter)\n",
    "                continue\n",
    "\n",
    "            # Extract the entity span\n",
    "            try:\n",
    "                output = entity_span(question, entity_label)\n",
    "                if output != None:\n",
    "                    entity_start, entity_end = output\n",
    "                else:\n",
    "                    continue\n",
    "            except Exception as e:\n",
    "                print(\"Error: \", e, \" at question: \", counter)\n",
    "                continue\n",
    "            \n",
    "            # Save the entity in the dictionary and relation in the vocab\n",
    "            entity_dict[entity_label] = entity_id\n",
    "            if relation_id not in relation_vocab:\n",
    "                relation_vocab.append(relation_id)\n",
    "\n",
    "            # Append the preprocessed data to the list\n",
    "            dataset.append({\n",
    "                'entity_id': entity_id,\n",
    "                'entity_label': entity_label,\n",
    "                'relation_id': relation_id,\n",
    "                'relation_label': str(relation_label),\n",
    "                'inverse': inverse,\n",
    "                'question': question,\n",
    "                'answer_id': answer,\n",
    "                'entity_start': entity_start,\n",
    "                'entity_end': entity_end\n",
    "            })\n",
    "\n",
    "    # Write the data to the corresponding JSON file\n",
    "    with open(mydataset, 'w', encoding='utf-8') as f:\n",
    "        json.dump(dataset, f)\n",
    "\n",
    "    # Define the header row for the CSV file\n",
    "    header = ['entity_id', 'entity_label', 'relation_id', 'relation_label', 'inverse', 'question', 'answer_id', 'entity_start', 'entity_end']\n",
    "\n",
    "    # Create a new file object in write mode and specify the filename and encoding\n",
    "    with open(mydataset.replace(\".json\",\".csv\"), mode='w', newline='', encoding='utf-8') as f:\n",
    "        \n",
    "        # Create a csv.writer object and write the header row\n",
    "        writer = csv.writer(f)\n",
    "        writer.writerow(header)\n",
    "        \n",
    "        # Loop through the dataset and write each dictionary as a row to the CSV file\n",
    "        for data in dataset:\n",
    "            row = [data['entity_id'], data['entity_label'], data['relation_id'], data['relation_label'], data['inverse'], data['question'], data['answer_id'], data['entity_start'], data['entity_end']]\n",
    "            writer.writerow(row)\n",
    "\n",
    "# Finally save the vocabulary to the corresponding JSON file\n",
    "with open('dataset/relation_vocab.json', 'w', encoding='utf-8') as f:\n",
    "    json.dump(relation_vocab, f)\n",
    "\n",
    "# Create a new file object in write mode and specify the filename and encoding\n",
    "with open('dataset/relation_vocab.csv', mode='w', newline='', encoding='utf-8') as f:\n",
    "\n",
    "    # Write the list to a CSV file\n",
    "    writer = csv.writer(f)\n",
    "    writer.writerow(['Relation'])\n",
    "    \n",
    "    # Loop through the relation vocab\n",
    "    for relation in relation_vocab:\n",
    "        writer.writerow([relation])\n",
    "\n",
    "# Write the dictionary to the corresponding JSON file\n",
    "with open('dataset/entity_dict.json', 'w', encoding='utf-8') as f:\n",
    "    json.dump(entity_dict, f)\n",
    "\n",
    "# Create a new file object in write mode and specify the filename and encoding\n",
    "with open('dataset/entity_dict.csv', mode='w', newline='', encoding='utf-8') as f:\n",
    "    \n",
    "    writer = csv.writer(f)\n",
    "    writer.writerow(['Entity', 'Id'])\n",
    "    for key, value in entity_dict.items():\n",
    "        writer.writerow([key, value])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env_tensorflow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "639c2a9cefc4711935c69ef9458e0ee4184e18203870c7a492801c98b906b529"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}