Merge branch 'release/3.0.0'

.gitignore

@@ -1,3 +1,4 @@
+notebooks/fhirdata
 # Temporary and binary files
 *~
 *.py[cod]

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## [Unreleased](https://github.com/dermatologist/fhiry/tree/HEAD)
+
+[Full Changelog](https://github.com/dermatologist/fhiry/compare/2.1.0...HEAD)
+
+**Implemented enhancements:**
+
+- import from BigQuery FHIR tables [\#106](https://github.com/dermatologist/fhiry/issues/106)
+
+## [2.1.0](https://github.com/dermatologist/fhiry/tree/2.1.0) (2023-02-06)
+
+[Full Changelog](https://github.com/dermatologist/fhiry/compare/2.0.0...2.1.0)
+
 ## [2.0.0](https://github.com/dermatologist/fhiry/tree/2.0.0) (2022-03-17)
 
 [Full Changelog](https://github.com/dermatologist/fhiry/compare/1.0.0...2.0.0)

README.md

@@ -1,24 +1,33 @@
-# :fire: fhiry - FHIR to pandas dataframe for data analysis, AI and ML
+# :fire: fhiry - FHIR to pandas dataframe for data analytics, AI and ML
+Virtual flattened view of *FHIR Bundle / ndjson / FHIR server / BigQuery!*
 
 ![Libraries.io SourceRank](https://img.shields.io/librariesio/sourcerank/pypi/fhiry)
 [![PyPI download total](https://img.shields.io/pypi/dm/fhiry.svg)](https://pypi.python.org/pypi/fhiry/)
 ![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/dermatologist/fhiry)
 
-## Open Source Python library for import of FHIR resources to pandas dataframe
-
-[Bulk data export using FHIR](https://hl7.org/fhir/uv/bulkdata/export/index.html) may be important if you want to export a cohort for analysis or machine learning.
+[Bulk data export using FHIR](https://hl7.org/fhir/uv/bulkdata/export/index.html) is needed to export a cohort for data analytics or machine learning.
 :fire: **Fhiry** is a [python](https://www.python.org/) package to facilitate this by converting a folder of [FHIR bundles](https://www.hl7.org/fhir/bundle.html)/ndjson into a [pandas](https://pandas.pydata.org/docs/user_guide/index.html) data frame for analysis and importing
-into ML packages such as Tensorflow and PyTorch. Test it with the [synthea sample](https://synthea.mitre.org/downloads) or the downloaded ndjson from the [SMART Bulk data server](https://bulk-data.smarthealthit.org/). Use the 'Discussions' tab above for feature requests.
+into ML packages such as Tensorflow and PyTorch. Fhiry also supports FHIR server search and FHIR tables on BigQuery.
+
+Test this with the [synthea sample](https://synthea.mitre.org/downloads) or the downloaded ndjson from the [SMART Bulk data server](https://bulk-data.smarthealthit.org/). Use the 'Discussions' tab above for feature requests.
+
+:sparkles: Checkout [this template](https://github.com/dermatologist/kedro-multimodal) for Multimodal machine learning in healthcare!
 
 ## Installation
 
+### Stable
 ```shell
 pip install fhiry
 ```
 
+### Latest dev version
+
+```
+pip install git+https://github.com/dermatologist/fhiry.git
+```
 ## Usage
 
-### Import FHIR bundles (JSON) from folder to pandas dataframe
+### 1. Import FHIR bundles (JSON) from folder to pandas dataframe
 
 ```python
 import fhiry.parallel as fp
@@ -30,7 +39,7 @@ Example source data set: [Synthea](https://synthea.mitre.org/downloads)
 
 Jupyter notebook example: [`notebooks/synthea.ipynb`](notebooks/synthea.ipynb)
 
-### Import NDJSON from folder to pandas dataframe
+### 2. Import NDJSON from folder to pandas dataframe
 
 ```python
 import fhiry.parallel as fp
@@ -42,7 +51,7 @@ Example source data set: [SMART Bulk Data Server](https://bulk-data.smarthealthi
 
 Jupyter notebook example: [`notebooks/ndjson.ipynb`](notebooks/ndjson.ipynb)
 
-### Import FHIR Search results to pandas dataframe
+### 3. Import FHIR Search results to pandas dataframe
 
 Fetch and import resources from [FHIR Search API](https://www.hl7.org/fhir/search.html) results to pandas dataframe.
 
@@ -66,6 +75,28 @@ df = fs.search(resource_type = "Condition", search_parameters = my_fhir_search_p
 print(df.info())
 ```
 
+### 4. Import [Google BigQuery](https://cloud.google.com/bigquery) [FHIR dataset](https://console.cloud.google.com/marketplace/details/mitre/synthea-fhir?q=synthea)
+
+```python
+from fhiry.bqsearch import BQsearch
+bqs = BQsearch()
+
+df = bqs.search("SELECT * FROM `bigquery-public-data.fhir_synthea.patient` LIMIT 20") # can be a path to .sql file
+
+```
+
+## Filters
+
+Pass a config json to any of the constructors:
+* config_json can be a path to a json file.
+```
+df = fp.process('/path/to/fhir/resources', config_json='{ "REMOVE": ["resource.text.div"], "RENAME": { "resource.id": "id" }  }')
+
+fs = Fhirsearch(fhir_base_url = "http://fhir-server:8080/fhir", config_json = '{ "REMOVE": ["resource.text.div"], "RENAME": { "resource.id": "id" }  }')
+
+bqs = BQsearch('{ "REMOVE": ["resource.text.div"], "RENAME": { "resource.id": "id" }  }')
+```
+
 ## Columns
 * see df.columns
 
@@ -82,11 +113,15 @@ resource.gender
 ...
 ```
 
+
 ### [Documentation](https://dermatologist.github.io/fhiry/)
 
+## Give us a star ⭐️
+If you find this project useful, give us a star. It helps others discover the project.
+
 ## Contributors
 
 * [Bell Eapen](https://nuchange.ca) | [![Twitter Follow](https://img.shields.io/twitter/follow/beapen?style=social)](https://twitter.com/beapen)
 * [Markus Mandalka](https://github.com/Mandalka)
-* WIP, PR welcome, please see CONTRIBUTING.md
-* [![forthebadge](https://forthebadge.com/images/badges/built-with-love.svg) using CC](https://computecanada.ca)
+* PR welcome, please see [CONTRIBUTING.md](/CONTRIBUTING.md)
+* [![forthebadge](https://forthebadge.com/images/badges/built-with-love.svg) using CC](https://alliancecan.ca/en/services/advanced-research-computing)

dev-requirements.in

@@ -2,8 +2,8 @@
 -c requirements.txt
 pytest-cov
 pytest
-recommonmark
 responses
+recommonmark
 sphinx>=3.2.1
 setuptools
 setuptools_scm

dev-requirements.txt

@@ -12,10 +12,14 @@ babel==2.9.1
     # via sphinx
 backports-entry-points-selectable==1.1.0
     # via virtualenv
-certifi==2021.5.30
-    # via requests
-charset-normalizer==2.0.5
-    # via requests
+certifi==2022.12.7
+    # via
+    #   -c requirements.txt
+    #   requests
+charset-normalizer==3.1.0
+    # via
+    #   -c requirements.txt
+    #   requests
 commonmark==0.9.1
     # via recommonmark
 coverage[toml]==5.5
@@ -30,8 +34,10 @@ filelock==3.0.12
     # via
     #   tox
     #   virtualenv
-idna==3.2
-    # via requests
+idna==3.4
+    # via
+    #   -c requirements.txt
+    #   requests
 imagesize==1.2.0
     # via sphinx
 importlib-metadata==5.1.0
@@ -42,8 +48,9 @@ jinja2==3.0.1
     # via sphinx
 markupsafe==2.0.1
     # via jinja2
-packaging==21.0
+packaging==23.0
     # via
+    #   -c requirements.txt
     #   pytest
     #   setuptools-scm
     #   sphinx
@@ -60,22 +67,21 @@ py==1.10.0
     #   tox
 pygments==2.10.0
     # via sphinx
-pyparsing==2.4.7
-    # via packaging
 pytest==7.1.2
     # via
     #   -r dev-requirements.in
     #   pytest-cov
 pytest-cov==3.0.0
     # via -r dev-requirements.in
-pytz==2022.6
+pytz==2022.7.1
     # via
     #   -c requirements.txt
     #   babel
 recommonmark==0.7.1
     # via -r dev-requirements.in
-requests==2.26.0
+requests==2.28.2
     # via
+    #   -c requirements.txt
     #   responses
     #   sphinx
 responses==0.22.0
@@ -118,8 +124,9 @@ tox==3.25.0
     # via -r dev-requirements.in
 types-toml==0.10.8.1
     # via responses
-urllib3==1.26.6
+urllib3==1.26.14
     # via
+    #   -c requirements.txt
     #   requests
     #   responses
 virtualenv==20.8.0