Merge branch 'master' of https://github.com/neurogenomics/RareDisease…

…Prioritisation

gpt_annotations/gpt_hpo_annot_api.py

@@ -4,7 +4,7 @@
 import re
 from IPython.display import display
 
-# Insert openAI key here
+# Initialize OpenAI client
 client = openai.OpenAI(api_key="")
 
 # Paths
@@ -15,34 +15,38 @@
 prompts = pd.read_csv(input_csv)
 outputs_df = pd.read_csv(previous_outputs_file)
 
+# Initialize an empty DataFrame
+df = pd.DataFrame()
+
 # Get the last processed index or start from the beginning if none
 last_processed_index = outputs_df.index.max() + 1 if not outputs_df.empty else 0
-last_processed_index2 = last_processed_index / 2 + 1
-
-for index, row in prompts.iloc[last_processed_index:].iterrows():
-  prompt_content = row['prompt']
-
-try:
-  response = client.chat.completions.create(
-    model='gpt-4',
-    temperature=0,
-    messages=[
-      {'role': 'user', 'content': prompt_content},
-    ]
-  )
-
-gpt_response = response.choices[0].message.content
-extracted_code = re.search(r'```python\n(.*?)\n```', gpt_response, re.DOTALL).group(1)
-exec(extracted_code)
-
-if isinstance(df, pd.DataFrame):
-  if set(outputs_df.columns) == set(df.columns):
-  df_sorted = df[outputs_df.columns]
-outputs_df = pd.concat([outputs_df, df_sorted], ignore_index=True)
-unique_df = outputs_df.drop_duplicates()
-unique_df.to_csv('/Users/kittymurphy/Documents/gpt_hpo_annotations_extra.csv', index=False)
-
-except Exception as e:
-  print(f"An error occurred for prompt at index {index}: {e}. Skipping to the next iteration.")
-
-last_processed_index = index / 2 + 1
+last_processed_index2 = (last_processed_index // 2) + 1
+
+for index, row in prompts.iloc[last_processed_index2:].iterrows():
+    prompt_content = row['prompt']
+
+    try:
+        response = client.chat.completions.create(
+            model='gpt-4',
+            temperature=0,
+            messages=[
+                {'role': 'user', 'content': prompt_content},
+            ]
+        )
+
+        gpt_response = response.choices[0].message.content
+        extracted_code = re.search(r'```python\n(.*?)\n```', gpt_response, re.DOTALL).group(1)
+        exec(extracted_code)
+
+        if isinstance(df, pd.DataFrame):
+            if set(outputs_df.columns) == set(df.columns):
+                df_sorted = df[outputs_df.columns]
+                outputs_df = pd.concat([outputs_df, df_sorted], ignore_index=True)
+                unique_df = outputs_df.drop_duplicates()
+                unique_df.to_csv('/Users/kittymurphy/Documents/gpt_hpo_annotations_extra.csv', index=False)
+
+    except Exception as e:
+        print(f"An error occurred for prompt at index {index}: {e}. Skipping to the next iteration.")
+
+    # Update last_processed_index with the current index + 1
+    last_processed_index = index + 1

manuscript/gpt_hpo_annotations_manuscript.qmd

@@ -1,5 +1,5 @@
 ---
-title: "Harnessing AI to gain insight into human phenotypes: a comprehensive annotation and exploration of the Human Phenotype Ontology "
+title: "Harnessing AI to annotate the severity of all phenotypic abnormalities within the Human Phenotype Ontology"
 ---
 
 ### Notes
@@ -10,14 +10,40 @@ title: "Harnessing AI to gain insight into human phenotypes: a comprehensive ann
 
 ### Abstract
 
-Comprehensive annotation of phenotypic abnormalities is invaluable for defining, diagnosing, prognosing, and treating human disease. Since 2008, the Human Phenotype Ontology (HPO) has served this purpose, by providing a standardised database for the description and analysis of human phenotypes. Through developing open community resources, the depth and breadth of the HPO has continued to expand and there are now x phenotypic abnormalities, corresponding to y rare diseases, documented. To date, the HPO has largely been manually curated by experts including clinicians, clinical geneticists, and researchers. Although this approach ensures the quality and accuracy of the ontology, it is time-consuming and labor-intensive. As artificial intelligence (AI) capabilities advance, there is an opportunity to integrate natural language processing technologies into assisting in the curation process. 
+<<<<<<< HEAD
+The Human Phenotype Ontology (HPO) has played a crucial role in defining, diagnosing, prognosing, and treating human diseases by providing a standardized database for phenotypic abnormalities. With 18,057 abnormalities now corresponding to over 10,000 rare diseases, manual curation by experts is becoming increasingly labor-intensive and time-consuming. Leveraging advances in artificial intelligence, we employed chatGPT to systematically annotate the severity of all 18,057 phenotypic abnormalities in the HPO. Our validated approach demonstrates the potential for natural language processing technologies to automate the curation process. By defining a severity scale for all phenotypes, our resources aims to assist in achieving important objectives for rare diseases, including improved diagnosis and prioritisation of gene therapy trials. We hope that this will be invaluable to the rare disease community and beyond. 
+=======
+Comprehensive annotation of phenotypic abnormalities is invaluable for defining, diagnosing, prognosing, and treating human disease. Since 2008, the Human Phenotype Ontology (HPO) has been instrumental to this, by providing a standardised database for the description and analysis of human phenotypes. Through developing open community resources, the depth and breadth of the HPO has continued to expand and there are now x phenotypic abnormalities, corresponding to y rare diseases, described. To date, the HPO has largely been manually curated by experts including clinicians, clinical geneticists, and researchers. Although this approach ensures the quality and accuracy of the ontology, it is time-consuming and labor-intensive. As artificial intelligence (AI) capabilities advance, there is an opportunity to integrate natural language processing technologies into assisting in the curation process. Here, we have used chatGPT to systematically annotate the severity of all 18,057 phenotypic abnormalities within the HPO. We have validated our approach and provide examples of how it can aid in prioritising gene therapy trials for rare diseases. Ultimately, we hope that our resource will be of utility to those working in rare diseases, as well as the wider rare disease community.  
+>>>>>>> bb3888e542860f13ad812f3359cb5d0716ba6ca7
 
 ### Introduction
 
+Comprehensive annotation of phenotypic abnormalities is invaluable for defining, diagnosing, prognosing, and treating human disease. Since 2008, the Human Phenotype Ontology (HPO) has been instrumental to this, by providing a standardised database for the description and analysis of human phenotypes. Through developing open community resources, the depth and breadth of the HPO has continued to expand and there are now 18,057 phenotypic abnormalities, corresponding to over 10,000 rare diseases, described. In recent years, the HPO has expanded its disease annotations so that each HPO term can have metadata including typical age of onset and frequency. In addition, there are the Clinical modifier (put this in italics) and Clinical course (also italics) subontologies, which contains terms to describe factors including severity and triggers, and mortality and progression, respectively. Describing the severity-related attributes of a disease is crucial for attaining significant objectives in rare diseases. This includes enhancing diagnostic capabilities, increased availibility of information, and prioritising gene therapy trials.
+
+To date, the HPO has largely been manually curated by experts including clinicians, clinical geneticists, and researchers. Although this approach ensures the quality and accuracy of the ontology, it is time-consuming and labor-intensive. As artificial intelligence (AI) capabilities advance, there is an opportunity to integrate natural language processing technologies into assisting in the curation process. Here, we have used chatGPT to systematically annotate the severity of all 18,057 phenotypic abnormalities within the HPO. Our annotation framework was developed based on previously defined criteria for classifying disease severity [Lazarin et al 2014]. Lazarin et al (2014) consulted healthcare professionals to test their proposed algorithm that through combining clinical characteristics of a genetic disease, you can accurately categorise the diseases' severity as profound, severe, moderate, or mild. 
+
+
+
+We have validated our approach and provide examples of how it can aid in prioritising gene therapy trials for rare diseases. Ultimately, we hope that our resource will be of utility to those working in rare diseases, as well as the wider rare disease community. 
+
 ### Results
 
 ### Methods
 
+Phenotypic abnormality annotation using OpenAI GPT-4
+<<<<<<< HEAD
+We employed the OpenAI GPT-4 model with Python to annotate 18,057 terms within the Human Phenotype Ontology (HPO). Our annotation framework was developed based on previously defined criteria for classifying disease severity [Lazarin et al 2014]. We sought to evaluate the impact of various phenotypes on factors including intellectually disability, death, impairmed mobility, physical malformations, blindness, sensory impairments, immunodeficiency, cancer, reduced fertility, and congenital onset. Through prompt design we found that the performance of GPT-4 improved when we incorporated a scale associated with each effect and required a jsutifcation for each response. For each effect, we asked about the likelihood of its occurrence - whether it never, rarely, often, or always occurred. Furthermore, our prompt design revealed that the optimal trade-off between the number of annotations and performance was achieved when inputting no more than two or three phenotypes per prompt. Below is an example prompt: 
+
+"I need to annotate phenotypes as to whether they typically cause: intellectual disability, death, impaired mobility, physical malformations, blindness, sensory impairments, immunodeficiency, cancer, reduced fertility? Do they have congenital onset? To answer, use a severity scale of: never, rarely, often, always. Do not consider indirect effects. You must provide the output in python code as a data frame called df with columns: phenotype, intellectual_disability, death, impaired_mobility, physical_malformations, blindness, sensory_impairments, immunodeficiency, cancer, reduced_fertility, congenital_onset. Also add a separate justification column for each outcome, e.g. death, death_justification. These are the phenotypes: Extracranial internal carotid artery dissection; Pulmonary arteriovenous fistulas."
+=======
+We employed the OpenAI GPT-4 model with Python to annotate 18,057 terms within the Human Phenotype Ontology (HPO). Our annotation framework was developed based on previously defined criteria for classifying disease severity [Lazarin et al 2014]. We sought to evaluate the impact of various phenotypes on factors including intellectually disability, death, impairmed mobility, physical malformations, blindness, sensory impairments, immunodeficiency, cancer, reduced fertility, and congenital onset. Through prompt design we found that the performance of GPT-4 improved when we incorporated a scale associated with each effect and required a jsutifcation for each response. For each effect, we inquired about the likelihood of its occurrence - whether it never, rarely, often, or always occurred. Furthermore, our prompt design revealed that the optimal trade-off between the number of annotations and performance was achieved when inputting no more than three phenotypes per prompt.
+>>>>>>> bb3888e542860f13ad812f3359cb5d0716ba6ca7
+
+
+Validation
+
+
+
 ### Discussion
 
 ### Data availability