jupyter/InternalDependenciesTypescript.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2f0eabc4",
   "metadata": {},
   "source": [
    "# Internal Dependencies\n",
    "<br>  \n",
    "\n",
    "### References\n",
    "- [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html)\n",
    "- [Calculate metrics](https://101.jqassistant.org/calculate-metrics/index.html)\n",
    "- [Neo4j Python Driver](https://neo4j.com/docs/api/python-driver/current)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4191f259",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plot\n",
    "from neo4j import GraphDatabase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c5dab37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please set the environment variable \"NEO4J_INITIAL_PASSWORD\" in your shell \n",
    "# before starting jupyter notebook to provide the password for the user \"neo4j\". \n",
    "# It is not recommended to hardcode the password into jupyter notebook for security reasons.\n",
    "\n",
    "driver = GraphDatabase.driver(uri=\"bolt://localhost:7687\", auth=(\"neo4j\", os.environ.get(\"NEO4J_INITIAL_PASSWORD\")))\n",
    "driver.verify_connectivity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1db254b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cypher_query_from_file(cypher_file_name : str):\n",
    "    with open(cypher_file_name) as file:\n",
    "        return ' '.join(file.readlines())\n",
    "\n",
    "\n",
    "def query_cypher_to_data_frame(filename : str, limit: int = -1):\n",
    "    \"\"\"\n",
    "    Execute the Cypher query of the given file and returns the result.\n",
    "    filename : str : The name of the file containing the Cypher query\n",
    "    limit : int : The optional limit of rows to optimize the query. Default = -1 = no limit\n",
    "    \"\"\"\n",
    "    cypher_query = get_cypher_query_from_file(filename)\n",
    "    if limit > 0:\n",
    "        cypher_query = \"{query}\\nLIMIT {row_limit}\".format(query = cypher_query, row_limit = limit)\n",
    "    records, summary, keys = driver.execute_query(cypher_query)\n",
    "    return pd.DataFrame([r.values() for r in records], columns=keys)\n",
    "\n",
    "\n",
    "def query_first_non_empty_cypher_to_data_frame(*filenames : str, limit: int = -1):\n",
    "    \"\"\"\n",
    "    Executes the Cypher queries of the given files and returns the first result that is not empty.\n",
    "    If all given file names result in empty results, the last (empty) result will be returned.\n",
    "    By additionally specifying \"limit=\" the \"LIMIT\" keyword will appended to query so that only the first results get returned.\n",
    "    \"\"\"    \n",
    "    result=pd.DataFrame()\n",
    "    for filename in filenames:\n",
    "        result=query_cypher_to_data_frame(filename, limit)\n",
    "        if not result.empty:\n",
    "            return result\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a56670c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The following cell uses the build-in %html \"magic\" to override the CSS style for tables to a much smaller size.\n",
    "#This is especially needed for PDF export of tables with multiple columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "006b9dc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* CSS style for smaller dataframe tables. */\n",
    ".dataframe th {\n",
    "    font-size: 8px;\n",
    "}\n",
    ".dataframe td {\n",
    "    font-size: 8px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ee8a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas DataFrame Display Configuration\n",
    "pd.set_option('display.max_colwidth', 300)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "91d80bf7",
   "metadata": {},
   "source": [
    "## 1 - Modules\n",
    "\n",
    "List the modules this notebook is based on. Different sorting variations help finding modules by their features and support larger code bases where the list of all modules gets very long.\n",
    "\n",
    "Only the top 30 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`List_all_Typescript_modules`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc682db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "internalModules = query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/List_all_Typescript_modules.cypher\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e48ead1b",
   "metadata": {},
   "source": [
    "### Table 1a - Top 30 modules with the highest element count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f822ddca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of modules descending\n",
    "internalModules.sort_values(by=['numberOfElements','moduleName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e508a21",
   "metadata": {},
   "source": [
    "### Table 1b - Top 30 modules with the highest number of incoming dependencies\n",
    "\n",
    "The following table lists the top 30 internal modules that are used the most by other modules (highest count of incoming dependencies, highest in-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107c0a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of incoming dependencies descending\n",
    "internalModules.sort_values(by=['incomingDependencies','moduleName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52593397",
   "metadata": {},
   "source": [
    "### Table 1c - Top 30 modules with the highest number of outgoing dependencies\n",
    "\n",
    "The following table lists the top 30 internal modules that are depending on the highest number of other modules (highest count of outgoing dependencies, highest out-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eee13bd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of outgoing dependencies descending\n",
    "internalModules.sort_values(by=['outgoingDependencies','moduleName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c928800",
   "metadata": {},
   "source": [
    "### Table 1d - Top 30 modules with the lowest element count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d73c15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of elements ascending\n",
    "internalModules.sort_values(by=['numberOfElements','moduleName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f628024",
   "metadata": {},
   "source": [
    "### Table 1e - Top 30 modules with the lowest number of incoming dependencies\n",
    "\n",
    "The following table lists the top 30 internal modules that are used the least by other modules (lowest count of incoming dependencies, lowest in-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6100d597",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of incoming dependencies ascending\n",
    "internalModules.sort_values(by=['incomingDependencies','moduleName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad07b654",
   "metadata": {},
   "source": [
    "### Table 1f - Top 30 modules with the lowest number of outgoing dependencies\n",
    "\n",
    "The following table lists the top 30 internal modules that are depending on the lowest number of other modules (lowest count of outgoing dependencies, lowest out-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd096a38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of outgoing dependencies ascending\n",
    "internalModules.sort_values(by=['outgoingDependencies','moduleName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d7cfd862",
   "metadata": {},
   "source": [
    "## 2 - Cyclic Dependencies\n",
    "\n",
    "Cyclic dependencies occur when one module uses an elements of another module and vice versa. \n",
    "These dependencies can lead to problems when one of these modules needs to be changed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9112f590",
   "metadata": {},
   "source": [
    "### Table 2a - Cyclic Dependencies Overview\n",
    "\n",
    "Show the top 40 cyclic dependencies sorted by the most promising to resolve first. This is done by calculating the number of forward dependencies (first cycle participant to second cycle participant) in relation to backward dependencies (second cycle participant back to first cycle participant). The higher this rate (approaching 1), the easier it should be to resolve the cycle by focussing on the few backward dependencies.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies_for_Typescript`\n",
    "\n",
    "**Columns:**\n",
    "- *projectFileName* identifies the project of the first participant of the cycle\n",
    "- *modulePathName* identifies the module of the first participant of the cycle\n",
    "- *dependentProjectFileName* identifies the project of the second participant of the cycle\n",
    "- *dependentModulePathName* identifies the module of the second participant of the cycle\n",
    "- *forwardToBackwardBalance* is between 0 and 1. High for many forward and few backward dependencies.\n",
    "- *numberForward* contains the number of dependencies from the first participant of the cycle to the second one\n",
    "- *numberBackward* contains the number of dependencies from the second participant of the cycle back to the first one\n",
    "- *someForwardDependencies* lists some forward dependencies in the text format \"type1 -> type2\"\n",
    "- *backwardDependencies* lists the backward dependencies in the format \"type1 <- type2\" that are recommended to get resolved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa634a4e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cyclic_dependencies = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies_for_Typescript.cypher\")\n",
    "cyclic_dependencies.head(40)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "02146c4d",
   "metadata": {},
   "source": [
    "### Table 2b - Cyclic Dependencies Break Down\n",
    "\n",
    "Lists modules with cyclic dependencies with every dependency in a separate row sorted by the most promising dependency first.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies_Breakdown_for_Typescript`\n",
    "\n",
    "**Columns in addition to Table 2a:**\n",
    "- *dependency* shows the cycle dependency in the text format \"type1 -> type2\" (forward) or \"type2<-type1\" (backward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf49a164",
   "metadata": {},
   "outputs": [],
   "source": [
    "cyclic_dependencies_breakdown = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies_Breakdown_for_Typescript.cypher\",limit=40)\n",
    "cyclic_dependencies_breakdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52f46a62",
   "metadata": {},
   "source": [
    "### Table 2c - Cyclic Dependencies Break Down - Backward Dependencies Only\n",
    "\n",
    "Lists modules with cyclic dependencies with every dependency in a separate row sorted by the most promising  dependency first. This table only contains the backward dependencies from the second participant of the cycle back to the first one that are the most promising to resolve.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies_Breakdown_BackwardOnly_for_Typescript`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61521c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "cyclic_dependencies_breakdown_backward = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies_Breakdown_Backward_Only_for_Typescript.cypher\",limit=40)\n",
    "cyclic_dependencies_breakdown_backward"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4be116e8",
   "metadata": {},
   "source": [
    "## 3 - Module Usage"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bb185191",
   "metadata": {},
   "source": [
    "### Table 3a - Elements that are used by multiple modules\n",
    "\n",
    "This table shows the top 40 modules that are used by the highest number of different modules. The whole table can be found in the CSV report `WidelyUsedTypescriptElements`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36a34924",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "elements_used_by_many_modules=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/List_elements_that_are_used_by_many_different_modules_for_Typescript.cypher\", limit=40)\n",
    "elements_used_by_many_modules"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "73dd1b13",
   "metadata": {},
   "source": [
    "### Table 3b - Elements that are used by multiple modules\n",
    "\n",
    "This table shows the top 30 modules that only use a few (compared to all existing) elements of another module.\n",
    "The whole table can be found in the CSV report `ModuleElementsUsageTypescript`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3d43c5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "used_packages_of_dependent_artifact=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/How_many_elements_compared_to_all_existing_are_used_by_dependent_modules_for_Typescript.cypher\",limit=30)\n",
    "used_packages_of_dependent_artifact"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d06d91b7",
   "metadata": {},
   "source": [
    "### Table 3c - Distance distribution between dependent files\n",
    "\n",
    "This table shows the file directory distance distribution between dependent files. Intuitively, the distance is given by the fewest number of change directory commands needed to navigate between a file and a dependency it uses. Those are aggregate to see how many dependent files are in the same directory, how many are just one change directory command apart, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80166282",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_first_non_empty_cypher_to_data_frame(\"../cypher/Internal_Dependencies/Get_file_distance_as_shortest_contains_path_for_dependencies.cypher\",\n",
    "                                           \"../cypher/Internal_Dependencies/Set_file_distance_as_shortest_contains_path_for_dependencies.cypher\", limit=20)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "JohT"
   }
  ],
  "code_graph_analysis_pipeline_data_validation": "ValidateTypescriptModuleDependencies",
  "kernelspec": {
   "display_name": "codegraph",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "title": "Object Oriented Design Quality Metrics for Java with Neo4j"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}