Merge branch 'release/1.5.0'

Author: KevinHanson

CHANGELOG

@@ -1,5 +1,43 @@
 Changelog
 =========
+## [1.5.0] - 2020-12-08
+### Summary
+
+This version introduces several smaller maintenance-level updates.
+The FTLD builder files now function similar to the UDS3 module builders,
+where NACCulator will first scan the Z1X to see which forms are present
+in the record rather than automatically trying to process all possible
+forms. This will prevent users from having to repeatedly manually delete
+empty optional forms that will never be filled out.
+
+NACCulator's event detection has been generalized to be more compatible
+with more centers. The requirement for the initial visit packet's event
+name to contain "initial_visit" has been changed to "initial", and the
+requirement for the followup visit packet's event event name to contain
+"followup_visit" has been changed to just "follow" (to account for variants
+with either "followup" OR "follow_up").
+
+The README has been updated to include more information about what data
+NACCulator needs from REDCap in order to run correctly (such as the filled
+Z1X form), and updated documentation on running the filters. I corrected the
+example commands for running the individual filters (with an input csv AND an
+output csv) so that they work properly now. All filter commands now require an
+argument specifying the config file so that the "validate" function in
+filters.py works properly. This makes the command structure consistent across
+all filters.
+
+### added
+ * Add note about Z1X to README
+ * Add optional form logic to fvp builder
+ * Add optional form logic to ftld ivp builder
+
+### changed
+ * Fix typo in README
+ * Edit NACCulator capitalization to be consistent within README
+ * Update documentation on filters and adjust example config file to make it easier to comment/uncomment filters
+ * Fixed typo fukid9agd and fusib17pdx to have underscores in uds fvp builder
+ * Update compatible redcap_event_names to be more general (initial, follow)
+
 ## [1.4.0] - 2020-10-21
 ### Summary
 

README.md

@@ -21,16 +21,22 @@ Once the project data is exported from REDCap to the CSV file `data.csv`, run:
     $ redcap2nacc <data.csv >data.txt
 
 This command will work only in the simplest case; UDS3 IVP data only.
-Nacculator will automatically skip PTIDs with errors, so the output `data.txt`
+NACCulator will automatically skip PTIDs with errors, so the output `data.txt`
 file will be ready to submit to NACC.
-In order to properly filter the data in the csv, nacculator is expecting that
+In order to properly filter the data in the csv, NACCulator is expecting that
 REDCap visits (denoted by `redcap_event_name`) contain certain keywords:
-    "initial_visit" for initial visit packets,
-    "followup_visit" for all followups,
+    "initial" for initial visit packets,
+    "follow" for all followups,
     "milestone" for milestone packets,
     "neuropath" for neuropathology packets,
     "telephone" for telephone followup packets
 
+NACCulator collects data from the Z1X form first and uses that to determine the
+presence of other forms in the packet. The Z1X form for that record must be
+marked "Unverified" or "Complete" for NACCulator to recognize the record, and
+each optional form must be marked as submitted within the Z1X for NACCulator to
+find those forms.
+
 _Note: output is written to `STDOUT`; errors are written to `STDERR`; input is
 expected to be from `STDIN` (the command line) unless a file is specified using
 the `-file` flag._
@@ -80,7 +86,7 @@ Both LBD / LBDSV and FTLD forms can have IVP or FVP arguments.
 
 **Example** - Run data through the `cleanPtid` filter:
 
-    $ redcap2nacc -f cleanPtid -meta nacculator_cfg.ini <data.csv >data.txt
+    $ redcap2nacc -f cleanPtid -meta nacculator_cfg.ini <data.csv >filtered_data.csv
 
 
 HOW TO Filter Data Using NACCulator
@@ -96,12 +102,26 @@ remove the `.example` portion, and then fill in your center's information.
 The config file contains sections with in-code filter function name. Each of
 these sections contains elements necessary for the filter to run.
 The filters described below will discuss what is required, if anything.
+
+The filters can be run all at once with your REDCap API token using:
+
+    $ nacculator_filters nacculator_cfg.ini
+
+You can find more details on `nacculator_filters` under the section:
+HOW TO Acquire current-db-subjects.csv for the filters
+
+Or they can be run one at a time on a `.csv` file with the `-f` and `-meta`
+flags.
+For example, to run the fixHeaders filter:
+
+    $ redcap2nacc -f fixHeaders -meta nacculator_cfg.ini <data_input.csv >filtered_output.csv
+
 If the filter requires the config, it must be passed with the `-meta` flag like
 the example above shows.
 
+
 * **cleanPtid**
 
-  **Filter config required**
   This filter requires a section in the config called `filter_clean_ptid`. This
   section will contain a single key `filepath` which will point to a csv 
   (usually called `current-db-subjects.csv`) file of ptids to be removed. All 
@@ -117,16 +137,15 @@ the example above shows.
       110003,I,001,Current
       110003,F,002,Current
 
+
 * **replaceDrugId**
 
   This filter replaces the first character of non empty fields of columns
   `drugid_1` to `drugid_30` with character "**d**".
 
-  This filter does not require any meta data file as of now.
 
 * **fixHeaders**
 
-  **Filter config required**
   This filter requires a section in the config called `filter_fix_headers` with
   as many keys as needed to replace the necessary columns. See example below.
   This filter fixes the column names of any column found in the filter mapping.
@@ -142,6 +161,8 @@ the example above shows.
       fu_otherneur: fu_othneur
       fu_otherneurx: fu_othneurxs
       fu_strokedec: fu_strokdec
+      fukid9agd: fu_kid9agd
+      fusib17pdx: fu_sib17pdx
 
 
 * **fillDefault**
@@ -154,14 +175,15 @@ the example above shows.
 
   *If field is blank, it will be updated to default value.*
 
+
 * **updateField**
 
   This filter is used to update fields that already had a value in the REDCap
   export. Currently, only `adcid` is updated.
 
+
 * **removePtid**
 
-  **Filter config required**
   This filter requires a section in the config called `filter_remove_ptid` with
   a single key called `ptid_format`. The value for that key is a regex string
   to match ptids that are to be kept.
@@ -211,7 +233,7 @@ move it to whatever location you specified in your `nacculator_cfg.ini` file.
 
 The csv is used by the filter_clean_ptid filter to identify and cull all
 packets already in NACC's Current database from your input csv. It is used to
-make nacculator run faster for very large databases.
+make NACCulator run faster for very large databases.
 
 
 Example Workflow

nacc/ftld/fvp/builder.py

@@ -1,14 +1,16 @@
 ###############################################################################
-# Copyright 2015-2019 University of Florida. All rights reserved.
+# Copyright 2015-2020 University of Florida. All rights reserved.
 # This file is part of UF CTS-IT's NACCulator project.
 # Use of this source code is governed by the license found in the LICENSE file.
 ###############################################################################
 
+import sys
+
 from nacc.ftld.fvp import forms as ftld_fvp_forms
 from nacc.uds3 import packet as ftld_fvp_packet
 
 
-def build_ftld_fvp_form(record: dict):
+def build_ftld_fvp_form(record: dict, err=sys.stderr):
     ''' Converts REDCap CSV data into a packet (list of FVP Form objects) '''
     packet = ftld_fvp_packet.Packet()
 
@@ -21,6 +23,35 @@ def build_ftld_fvp_form(record: dict):
                 and int(record['visitday']) >= 1):
         raise ValueError('Form date cannot precede March 1, 2015.')
 
+    add_z1x(record, packet)
+    # Forms B3F, B9F, C1F, C2F, C3F, E2F, and E3F are REQUIRED.
+    # Forms A3A, C4F, C5F, and C6F are OPTIONAL and must be specifically
+    # marked as present for nacculator to process them
+    if record['ivp_z1x_complete'] in ['1', '2']:
+        if record['ftda3afs'] == '1':
+            add_a3a(record, packet)
+        add_b3f(record, packet)
+        add_b9f(record, packet)
+        add_c1f(record, packet)
+        add_c2f(record, packet)
+        add_c3f(record, packet)
+        if record['ftdc4fs'] == '1':
+            add_c4f(record, packet)
+        if record['ftdc5fs'] == '1':
+            add_c5f(record, packet)
+        if record['ftdc6fs'] == '1':
+            add_c6f(record, packet)
+    else:
+        print("ptid " + str(record['ptid']) +
+              ": No Z1X form found.", file=err)
+    add_e2f(record, packet)
+    add_e3f(record, packet)
+    update_header(record, packet)
+
+    return packet
+
+
+def add_z1x(record, packet):
     Z1X = ftld_fvp_forms.FormZ1X()
     Z1X.LANGA1   = record['fu_langa1']
     Z1X.LANGA2   = record['fu_langa2']
@@ -71,8 +102,10 @@ def build_ftld_fvp_form(record: dict):
     Z1X.LANGE3F  = record['fu_lange3f']
     Z1X.LANGCLS  = record['fu_langcls']
     Z1X.CLSSUB   = record['fu_clssub']
-    packet.append(Z1X)
+    packet.insert(0, Z1X)
 
+
+def add_a3a(record, packet):
     A3a = ftld_fvp_forms.FormA3a()
     A3a.FTDRELCO = record['fu_ftdrelco']
     A3a.FTDSIBBY = record['fu_ftdsibby']
@@ -82,6 +115,8 @@ def build_ftld_fvp_form(record: dict):
     A3a.FTDCOMME = record['fu_ftdcomme']
     packet.append(A3a)
 
+
+def add_b3f(record, packet):
     B3F = ftld_fvp_forms.FormB3F()
     B3F.FTDLTFAS = record['fu_ftdltfas']
     B3F.FTDLIMB  = record['fu_ftdlimb']
@@ -93,6 +128,8 @@ def build_ftld_fvp_form(record: dict):
     B3F.FTDGTYPX = record['fu_ftdgtypx']
     packet.append(B3F)
 
+
+def add_b9f(record, packet):
     B9F = ftld_fvp_forms.FormB9F()
     B9F.FTDPPASL = record['fu_ftdppasl']
     B9F.FTDPPAPO = record['fu_ftdppapo']
@@ -122,6 +159,8 @@ def build_ftld_fvp_form(record: dict):
     B9F.FTDPABVF = record['fu_ftdpabvf']
     packet.append(B9F)
 
+
+def add_c1f(record, packet):
     C1F = ftld_fvp_forms.FormC1F()
     C1F.FTDWORRC = record['fu_ftdworrc']
     C1F.FTDWORRS = record['fu_ftdworrs']
@@ -150,6 +189,8 @@ def build_ftld_fvp_form(record: dict):
     C1F.FTDREAPR = record['fu_ftdreapr']
     packet.append(C1F)
 
+
+def add_c2f(record, packet):
     C2F = ftld_fvp_forms.FormC2F()
     C2F.FTDCPC2F = record['fu_ftdcpc2f']
     C2F.FTDhAIRD = record['fu_ftdhaird']
@@ -180,6 +221,8 @@ def build_ftld_fvp_form(record: dict):
     C2F.FTDSNRAT = record['fu_ftdsnrat']
     packet.append(C2F)
 
+
+def add_c3f(record, packet):
     C3F = ftld_fvp_forms.FormC3F()
     C3F.FTDSELF  = record['fu_ftdself']
     C3F.FTDBADLY = record['fu_ftdbadly']
@@ -235,6 +278,8 @@ def build_ftld_fvp_form(record: dict):
     C3F.FTDLENGT = record['fu_ftdlengt']
     packet.append(C3F)
 
+
+def add_c4f(record, packet):
     C4F = ftld_fvp_forms.FormC4F()
     C4F.FTDCPC4F = record['fu_ftdcpc4f']
     C4F.FTDWORKU = record['fu_ftdworku']
@@ -247,6 +292,8 @@ def build_ftld_fvp_form(record: dict):
     C4F.FTDBIST  = record['fu_ftdbist']
     packet.append(C4F)
 
+
+def add_c5f(record, packet):
     C5F = ftld_fvp_forms.FormC5F()
     C5F.FTDCPC5F = record['fu_ftdcpc5f']
     C5F.FTDINSEX = record['fu_ftdinsex']
@@ -271,6 +318,8 @@ def build_ftld_fvp_form(record: dict):
     C5F.FTDIRIPT = record['fu_ftdiript']
     packet.append(C5F)
 
+
+def add_c6f(record, packet):
     C6F = ftld_fvp_forms.FormC6F()
     C6F.FTDCPC6F = record['fu_ftdcpc6f']
     C6F.FTDALTER = record['fu_ftdalter']
@@ -291,6 +340,8 @@ def build_ftld_fvp_form(record: dict):
     C6F.FTDRSMST = record['fu_ftdrsmst']
     packet.append(C6F)
 
+
+def add_e2f(record, packet):
     E2F = ftld_fvp_forms.FormE2F()
     E2F.FTDSMRI  = record['fu_ftdsmri']
     E2F.FTDSMMO  = record['fu_ftdsmmo']
@@ -340,6 +391,8 @@ def build_ftld_fvp_form(record: dict):
     E2F.FTDOTANS = record['fu_ftdotans']
     packet.append(E2F)
 
+
+def add_e3f(record, packet):
     E3F = ftld_fvp_forms.FormE3F()
     E3F.FTDIDIAG = record['fu_ftdidiag']
     E3F.FTDSMRIO = record['fu_ftdsmrio']
@@ -402,9 +455,6 @@ def build_ftld_fvp_form(record: dict):
     E3F.FTDOThIS = record['fu_ftdothis']
     packet.append(E3F)
 
-    update_header(record,packet)
-    return packet
-
 
 def update_header(record, packet):
     for header in packet: