Part 4 Local
Instead of using the CLI/T-Tool/ETL-Tool. The process can be run with a custom python script to give more control over the inputs/outputs and various configurations
import carrot
import glob
inputs = carrot.tools.load_csv(glob.glob('../data/part2/*'))
inputs
2022-06-17 14:49:16 - LocalDataCollection - INFO - DataCollection Object Created
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Blood_Test.csv [<carrot.io.common.DataBrick object at 0x106aba280>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Demographics.csv [<carrot.io.common.DataBrick object at 0x106af5ca0>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering GP_Records.csv [<carrot.io.common.DataBrick object at 0x10a7e3fd0>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Hospital_Visit.csv [<carrot.io.common.DataBrick object at 0x10a7e3fa0>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Serology.csv [<carrot.io.common.DataBrick object at 0x106af5430>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Symptoms.csv [<carrot.io.common.DataBrick object at 0x106af5580>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering Vaccinations.csv [<carrot.io.common.DataBrick object at 0x10a8476d0>]
2022-06-17 14:49:16 - LocalDataCollection - INFO - Registering pks.csv [<carrot.io.common.DataBrick object at 0x10a8476a0>]
<carrot.io.plugins.local.LocalDataCollection at 0x106af5d60>
inputs.keys()
dict_keys(['Blood_Test.csv', 'Demographics.csv', 'GP_Records.csv', 'Hospital_Visit.csv', 'Serology.csv', 'Symptoms.csv', 'Vaccinations.csv', 'pks.csv'])
outputs = carrot.tools.create_csv_store(output_folder="./test_outputs/local/")
outputs
2022-06-17 14:49:16 - LocalDataCollection - INFO - DataCollection Object Created
<carrot.io.plugins.local.LocalDataCollection at 0x10a847a00>
rules = carrot.tools.load_json("../data/rules.json")
cdm = carrot.cdm.CommonDataModel.from_rules(rules,inputs=inputs,outputs=outputs)
cdm.get_tables()
2022-06-17 14:49:16 - CommonDataModel - INFO - CommonDataModel (5.3.1) created with co-connect-tools version 0.0.0
2022-06-17 14:49:16 - CommonDataModel - INFO - Running with an DataCollection object
2022-06-17 14:49:16 - CommonDataModel - INFO - Turning on automatic cdm column filling
2022-06-17 14:49:16 - CommonDataModel - INFO - Added MALE 3025 of type person
2022-06-17 14:49:16 - CommonDataModel - INFO - Added FEMALE 3026 of type person
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Antibody 3027 of type observation
2022-06-17 14:49:16 - CommonDataModel - INFO - Added H/O: heart failure 3043 of type observation
2022-06-17 14:49:16 - CommonDataModel - INFO - Added 2019-nCoV 3044 of type observation
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Cancer 3045 of type observation
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Headache 3028 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Fatigue 3029 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Dizziness 3030 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Cough 3031 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Fever 3032 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Muscle pain 3033 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Pneumonia 3042 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Mental health problem 3046 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Mental disorder 3047 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Type 2 diabetes mellitus 3048 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Ischemic heart disease 3049 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added Hypertensive disorder 3050 of type condition_occurrence
2022-06-17 14:49:16 - CommonDataModel - INFO - Added COVID-19 vaccine 3034 of type drug_exposure
2022-06-17 14:49:16 - CommonDataModel - INFO - Added COVID-19 vaccine 3035 of type drug_exposure
2022-06-17 14:49:16 - CommonDataModel - INFO - Added COVID-19 vaccine 3036 of type drug_exposure
2022-06-17 14:49:16 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 of type drug_exposure
2022-06-17 14:49:16 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 of type drug_exposure
['person', 'observation', 'condition_occurrence', 'drug_exposure']
cdm.get_objects()
{'person': {'MALE 3025': <carrot.cdm.objects.versions.v5_3_1.person.Person at 0x106af5700>,
'FEMALE 3026': <carrot.cdm.objects.versions.v5_3_1.person.Person at 0x106af5850>},
'observation': {'Antibody 3027': <carrot.cdm.objects.versions.v5_3_1.observation.Observation at 0x10a8a6ca0>,
'H/O: heart failure 3043': <carrot.cdm.objects.versions.v5_3_1.observation.Observation at 0x10a8aa730>,
'2019-nCoV 3044': <carrot.cdm.objects.versions.v5_3_1.observation.Observation at 0x10a8a6d30>,
'Cancer 3045': <carrot.cdm.objects.versions.v5_3_1.observation.Observation at 0x10a8aab80>},
'condition_occurrence': {'Headache 3028': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8aab20>,
'Fatigue 3029': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8ad880>,
'Dizziness 3030': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8adf70>,
'Cough 3031': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8ad280>,
'Fever 3032': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8b3dc0>,
'Muscle pain 3033': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x106af5190>,
'Pneumonia 3042': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8b3d00>,
'Mental health problem 3046': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8b8f10>,
'Mental disorder 3047': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8b8b20>,
'Type 2 diabetes mellitus 3048': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8bcdf0>,
'Ischemic heart disease 3049': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8b8bb0>,
'Hypertensive disorder 3050': <carrot.cdm.objects.versions.v5_3_1.condition_occurrence.ConditionOccurrence at 0x10a8bca00>},
'drug_exposure': {'COVID-19 vaccine 3034': <carrot.cdm.objects.versions.v5_3_1.drug_exposure.DrugExposure at 0x10a8c14c0>,
'COVID-19 vaccine 3035': <carrot.cdm.objects.versions.v5_3_1.drug_exposure.DrugExposure at 0x10a8c1fd0>,
'COVID-19 vaccine 3036': <carrot.cdm.objects.versions.v5_3_1.drug_exposure.DrugExposure at 0x10a8c1910>,
'SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040': <carrot.cdm.objects.versions.v5_3_1.drug_exposure.DrugExposure at 0x10a8c1850>,
'SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041': <carrot.cdm.objects.versions.v5_3_1.drug_exposure.DrugExposure at 0x10a8c5d60>}}
cdm.process()
2022-06-17 14:49:17 - CommonDataModel - INFO - Starting processing in order: ['person', 'observation', 'condition_occurrence', 'drug_exposure']
2022-06-17 14:49:17 - CommonDataModel - INFO - Number of objects to process for each table...
{
"person": 2,
"observation": 4,
"condition_occurrence": 12,
"drug_exposure": 5
}
2022-06-17 14:49:17 - CommonDataModel - INFO - for person: found 2 objects
2022-06-17 14:49:17 - CommonDataModel - INFO - working on person
2022-06-17 14:49:17 - CommonDataModel - INFO - starting on MALE 3025
2022-06-17 14:49:17 - Person - INFO - Called apply_rules
2022-06-17 14:49:17 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Demographics.csv' for the first time
2022-06-17 14:49:17 - Person - INFO - Mapped birth_datetime
2022-06-17 14:49:17 - Person - INFO - Mapped gender_concept_id
2022-06-17 14:49:17 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 14:49:17 - Person - INFO - Mapped gender_source_value
2022-06-17 14:49:17 - Person - INFO - Mapped person_id
2022-06-17 14:49:17 - Person - WARNING - Requiring non-null values in gender_concept_id removed 216 rows, leaving 284 rows.
2022-06-17 14:49:17 - Person - INFO - Automatically formatting data columns.
2022-06-17 14:49:17 - Person - INFO - created df (0x10a8f3c70)[MALE_3025]
2022-06-17 14:49:17 - CommonDataModel - INFO - finished MALE 3025 (0x10a8f3c70) ... 1/2 completed, 284 rows
2022-06-17 14:49:17 - LocalDataCollection - INFO - making output folder ./test_outputs/local/
2022-06-17 14:49:17 - LocalDataCollection - INFO - saving person_ids to ./test_outputs/local//person_ids.csv
2022-06-17 14:49:17 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:17 - CommonDataModel - INFO - starting on FEMALE 3026
2022-06-17 14:49:17 - Person - INFO - Called apply_rules
2022-06-17 14:49:17 - Person - INFO - Mapped birth_datetime
2022-06-17 14:49:17 - Person - INFO - Mapped gender_concept_id
2022-06-17 14:49:17 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 14:49:17 - Person - INFO - Mapped gender_source_value
2022-06-17 14:49:17 - Person - INFO - Mapped person_id
2022-06-17 14:49:17 - Person - WARNING - Requiring non-null values in gender_concept_id removed 286 rows, leaving 214 rows.
2022-06-17 14:49:17 - Person - INFO - Automatically formatting data columns.
2022-06-17 14:49:17 - Person - INFO - created df (0x10a919bb0)[FEMALE_3026]
2022-06-17 14:49:17 - CommonDataModel - INFO - finished FEMALE 3026 (0x10a919bb0) ... 2/2 completed, 214 rows
2022-06-17 14:49:17 - LocalDataCollection - INFO - updating person_ids in ./test_outputs/local//person_ids.csv
2022-06-17 14:49:17 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:17 - CommonDataModel - INFO - saving dataframe (0x10a919be0) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:17 - LocalDataCollection - INFO - saving person to ./test_outputs/local//person.csv
2022-06-17 14:49:17 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:17 - CommonDataModel - INFO - finalised person on iteration 0 producing 498 rows from 2 tables
2022-06-17 14:49:17 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:17 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:17 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:17 - CommonDataModel - INFO - for observation: found 4 objects
2022-06-17 14:49:17 - CommonDataModel - INFO - working on observation
2022-06-17 14:49:17 - CommonDataModel - INFO - starting on Antibody 3027
2022-06-17 14:49:17 - Observation - INFO - Called apply_rules
2022-06-17 14:49:17 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Serology.csv' for the first time
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:17 - Observation - INFO - Mapped person_id
2022-06-17 14:49:17 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:17 - Observation - INFO - created df (0x10aba8730)[Antibody_3027]
2022-06-17 14:49:17 - CommonDataModel - INFO - finished Antibody 3027 (0x10aba8730) ... 1/4 completed, 204 rows
2022-06-17 14:49:17 - CommonDataModel - INFO - starting on H/O: heart failure 3043
2022-06-17 14:49:17 - Observation - INFO - Called apply_rules
2022-06-17 14:49:17 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit.csv' for the first time
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:17 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:17 - Observation - INFO - Mapped person_id
2022-06-17 14:49:18 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 487 rows, leaving 113 rows.
2022-06-17 14:49:18 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:18 - Observation - INFO - created df (0x10aba8c70)[H_O_heart_failure_3043]
2022-06-17 14:49:18 - CommonDataModel - INFO - finished H/O: heart failure 3043 (0x10aba8c70) ... 2/4 completed, 113 rows
2022-06-17 14:49:18 - CommonDataModel - INFO - starting on 2019-nCoV 3044
2022-06-17 14:49:18 - Observation - INFO - Called apply_rules
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:18 - Observation - INFO - Mapped person_id
2022-06-17 14:49:18 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 506 rows, leaving 94 rows.
2022-06-17 14:49:18 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:18 - Observation - INFO - created df (0x10abb9cd0)[2019_nCoV_3044]
2022-06-17 14:49:18 - CommonDataModel - INFO - finished 2019-nCoV 3044 (0x10abb9cd0) ... 3/4 completed, 94 rows
2022-06-17 14:49:18 - CommonDataModel - INFO - starting on Cancer 3045
2022-06-17 14:49:18 - Observation - INFO - Called apply_rules
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:18 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:18 - Observation - INFO - Mapped person_id
2022-06-17 14:49:18 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 426 rows, leaving 174 rows.
2022-06-17 14:49:18 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:18 - Observation - INFO - created df (0x10abc4f10)[Cancer_3045]
2022-06-17 14:49:18 - CommonDataModel - INFO - finished Cancer 3045 (0x10abc4f10) ... 4/4 completed, 174 rows
2022-06-17 14:49:18 - CommonDataModel - INFO - saving dataframe (0x10abb9160) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:18 - LocalDataCollection - INFO - saving observation to ./test_outputs/local//observation.csv
2022-06-17 14:49:18 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:18 - CommonDataModel - INFO - finalised observation on iteration 0 producing 585 rows from 4 tables
2022-06-17 14:49:18 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:18 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:18 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:18 - CommonDataModel - INFO - for condition_occurrence: found 12 objects
2022-06-17 14:49:18 - CommonDataModel - INFO - working on condition_occurrence
2022-06-17 14:49:18 - CommonDataModel - INFO - starting on Headache 3028
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:18 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Symptoms.csv' for the first time
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:18 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 39 rows, leaving 126 rows.
2022-06-17 14:49:18 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 1 rows, leaving 125 rows.
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:18 - ConditionOccurrence - INFO - created df (0x10abc49a0)[Headache_3028]
2022-06-17 14:49:18 - CommonDataModel - INFO - finished Headache 3028 (0x10abc49a0) ... 1/12 completed, 125 rows
2022-06-17 14:49:18 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:18 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:18 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:18 - CommonDataModel - ERROR - 123/125 were good, 2 studies are removed.
2022-06-17 14:49:18 - CommonDataModel - INFO - starting on Fatigue 3029
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:18 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 52 rows, leaving 113 rows.
2022-06-17 14:49:18 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 1 rows, leaving 112 rows.
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:18 - ConditionOccurrence - INFO - created df (0x10abc4280)[Fatigue_3029]
2022-06-17 14:49:18 - CommonDataModel - INFO - finished Fatigue 3029 (0x10abc4280) ... 2/12 completed, 112 rows
2022-06-17 14:49:18 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:18 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:18 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:18 - CommonDataModel - ERROR - 110/112 were good, 2 studies are removed.
2022-06-17 14:49:18 - CommonDataModel - INFO - starting on Dizziness 3030
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:18 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:18 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 88 rows, leaving 77 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10ac069d0)[Dizziness_3030]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Dizziness 3030 (0x10ac069d0) ... 3/12 completed, 77 rows
2022-06-17 14:49:19 - CommonDataModel - INFO - starting on Cough 3031
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:19 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 60 rows, leaving 105 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10ac0bf10)[Cough_3031]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Cough 3031 (0x10ac0bf10) ... 4/12 completed, 105 rows
2022-06-17 14:49:19 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:19 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:19 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:19 - CommonDataModel - ERROR - 103/105 were good, 2 studies are removed.
2022-06-17 14:49:19 - CommonDataModel - INFO - starting on Fever 3032
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:19 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 128 rows, leaving 37 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10ac05970)[Fever_3032]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Fever 3032 (0x10ac05970) ... 5/12 completed, 37 rows
2022-06-17 14:49:19 - CommonDataModel - INFO - starting on Muscle pain 3033
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:19 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 147 rows, leaving 18 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10ac2eb20)[Muscle_pain_3033]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Muscle pain 3033 (0x10ac2eb20) ... 6/12 completed, 18 rows
2022-06-17 14:49:19 - CommonDataModel - INFO - starting on Pneumonia 3042
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:19 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit.csv' for the first time
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:19 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 509 rows, leaving 91 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10ac24190)[Pneumonia_3042]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Pneumonia 3042 (0x10ac24190) ... 7/12 completed, 91 rows
2022-06-17 14:49:19 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:19 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:19 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:19 - CommonDataModel - ERROR - 90/91 were good, 1 studies are removed.
2022-06-17 14:49:19 - CommonDataModel - INFO - starting on Mental health problem 3046
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:19 - LocalDataCollection - INFO - Retrieving initial dataframe for 'GP_Records.csv' for the first time
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:19 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 742 rows, leaving 225 rows.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:19 - ConditionOccurrence - INFO - created df (0x10abc4820)[Mental_health_problem_3046]
2022-06-17 14:49:19 - CommonDataModel - INFO - finished Mental health problem 3046 (0x10abc4820) ... 8/12 completed, 225 rows
2022-06-17 14:49:20 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:20 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:20 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:20 - CommonDataModel - ERROR - 224/225 were good, 1 studies are removed.
2022-06-17 14:49:20 - CommonDataModel - INFO - starting on Mental disorder 3047
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:20 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 742 rows, leaving 225 rows.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - created df (0x10ac3fcd0)[Mental_disorder_3047]
2022-06-17 14:49:20 - CommonDataModel - INFO - finished Mental disorder 3047 (0x10ac3fcd0) ... 9/12 completed, 225 rows
2022-06-17 14:49:20 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:20 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:20 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:20 - CommonDataModel - ERROR - 224/225 were good, 1 studies are removed.
2022-06-17 14:49:20 - CommonDataModel - INFO - starting on Type 2 diabetes mellitus 3048
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:20 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 851 rows, leaving 116 rows.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - created df (0x10ac5df40)[Type_2_diabetes_mellitus_3048]
2022-06-17 14:49:20 - CommonDataModel - INFO - finished Type 2 diabetes mellitus 3048 (0x10ac5df40) ... 10/12 completed, 116 rows
2022-06-17 14:49:20 - CommonDataModel - INFO - starting on Ischemic heart disease 3049
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:20 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 872 rows, leaving 95 rows.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - created df (0x10ac71e80)[Ischemic_heart_disease_3049]
2022-06-17 14:49:20 - CommonDataModel - INFO - finished Ischemic heart disease 3049 (0x10ac71e80) ... 11/12 completed, 95 rows
2022-06-17 14:49:20 - CommonDataModel - INFO - starting on Hypertensive disorder 3050
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:20 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 886 rows, leaving 81 rows.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:20 - ConditionOccurrence - INFO - created df (0x10ac88a60)[Hypertensive_disorder_3050]
2022-06-17 14:49:20 - CommonDataModel - INFO - finished Hypertensive disorder 3050 (0x10ac88a60) ... 12/12 completed, 81 rows
2022-06-17 14:49:20 - CommonDataModel - INFO - saving dataframe (0x10abb9820) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:20 - LocalDataCollection - INFO - saving condition_occurrence to ./test_outputs/local//condition_occurrence.csv
2022-06-17 14:49:20 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:21 - CommonDataModel - INFO - finalised condition_occurrence on iteration 0 producing 1298 rows from 12 tables
2022-06-17 14:49:21 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:21 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:21 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:21 - CommonDataModel - INFO - for drug_exposure: found 5 objects
2022-06-17 14:49:21 - CommonDataModel - INFO - working on drug_exposure
2022-06-17 14:49:21 - CommonDataModel - INFO - starting on COVID-19 vaccine 3034
2022-06-17 14:49:21 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:21 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Vaccinations.csv' for the first time
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:21 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 231 rows, leaving 129 rows.
2022-06-17 14:49:21 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:21 - DrugExposure - INFO - created df (0x10aca01f0)[COVID_19_vaccine_3034]
2022-06-17 14:49:21 - CommonDataModel - INFO - finished COVID-19 vaccine 3034 (0x10aca01f0) ... 1/5 completed, 129 rows
2022-06-17 14:49:21 - CommonDataModel - INFO - starting on COVID-19 vaccine 3035
2022-06-17 14:49:21 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:21 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 246 rows, leaving 114 rows.
2022-06-17 14:49:21 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:21 - DrugExposure - INFO - created df (0x10aca0970)[COVID_19_vaccine_3035]
2022-06-17 14:49:21 - CommonDataModel - INFO - finished COVID-19 vaccine 3035 (0x10aca0970) ... 2/5 completed, 114 rows
2022-06-17 14:49:21 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:21 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:21 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:21 - CommonDataModel - ERROR - 112/114 were good, 2 studies are removed.
2022-06-17 14:49:21 - CommonDataModel - INFO - starting on COVID-19 vaccine 3036
2022-06-17 14:49:21 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:21 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 243 rows, leaving 117 rows.
2022-06-17 14:49:21 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:21 - DrugExposure - INFO - created df (0x10acccd30)[COVID_19_vaccine_3036]
2022-06-17 14:49:21 - CommonDataModel - INFO - finished COVID-19 vaccine 3036 (0x10acccd30) ... 3/5 completed, 117 rows
2022-06-17 14:49:21 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040
2022-06-17 14:49:21 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:21 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 231 rows, leaving 129 rows.
2022-06-17 14:49:21 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:21 - DrugExposure - INFO - created df (0x10acd6730)[SARS_CoV_2_COVID_19_vaccine_mRNA_1273_0_2_MG_ML_Injectable_Suspension_3040]
2022-06-17 14:49:21 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 (0x10acd6730) ... 4/5 completed, 129 rows
2022-06-17 14:49:21 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041
2022-06-17 14:49:21 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:21 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:22 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 243 rows, leaving 117 rows.
2022-06-17 14:49:22 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:22 - DrugExposure - INFO - created df (0x10aca0610)[SARS_CoV_2_COVID_19_vaccine_mRNA_BNT162b2_0_1_MG_ML_Injectable_Suspension_3041]
2022-06-17 14:49:22 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 (0x10aca0610) ... 5/5 completed, 117 rows
2022-06-17 14:49:22 - CommonDataModel - INFO - saving dataframe (0x10ac2e6d0) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:22 - LocalDataCollection - INFO - saving drug_exposure to ./test_outputs/local//drug_exposure.csv
2022-06-17 14:49:22 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:22 - CommonDataModel - INFO - finalised drug_exposure on iteration 0 producing 604 rows from 5 tables
2022-06-17 14:49:22 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:22 - LocalDataCollection - INFO - All input files for this object have now been used.
cdm['person'].dropna(axis=1)
| gender_concept_id | year_of_birth | month_of_birth | day_of_birth | birth_datetime | gender_source_value | gender_source_concept_id | |
|---|---|---|---|---|---|---|---|
| person_id | |||||||
| 1 | 8507 | 1975 | 7 | 13 | 1975-07-13 00:00:00.000000 | Male | 8507 |
| 2 | 8507 | 1968 | 7 | 14 | 1968-07-14 00:00:00.000000 | Male | 8507 |
| 3 | 8507 | 1976 | 7 | 12 | 1976-07-12 00:00:00.000000 | Male | 8507 |
| 4 | 8507 | 1942 | 7 | 21 | 1942-07-21 00:00:00.000000 | Male | 8507 |
| 5 | 8507 | 1943 | 7 | 21 | 1943-07-21 00:00:00.000000 | Male | 8507 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 494 | 8532 | 1966 | 7 | 15 | 1966-07-15 00:00:00.000000 | Female | 8532 |
| 495 | 8532 | 1979 | 7 | 12 | 1979-07-12 00:00:00.000000 | Female | 8532 |
| 496 | 8532 | 1994 | 7 | 8 | 1994-07-08 00:00:00.000000 | Female | 8532 |
| 497 | 8532 | 1950 | 7 | 19 | 1950-07-19 00:00:00.000000 | Female | 8532 |
| 498 | 8532 | 1945 | 7 | 20 | 1945-07-20 00:00:00.000000 | Female | 8532 |
498 rows × 7 columns
cdm['observation'].dropna(axis=1)
| person_id | observation_concept_id | observation_date | observation_datetime | observation_source_value | observation_source_concept_id | |
|---|---|---|---|---|---|---|
| observation_id | ||||||
| 1 | 458 | 4288455 | 2020-11-20 | 2020-11-20 00:00:00.000000 | 61.597638175407624 | 4288455 |
| 2 | 330 | 4288455 | 2022-09-12 | 2022-09-12 00:00:00.000000 | 6.413970727863393 | 4288455 |
| 3 | 136 | 4288455 | 2019-02-02 | 2019-02-02 00:00:00.000000 | 4.691778998076871 | 4288455 |
| 4 | 17 | 4288455 | 2020-08-18 | 2020-08-18 00:00:00.000000 | 14.77684965289811 | 4288455 |
| 5 | 55 | 4288455 | 2019-11-23 | 2019-11-23 00:00:00.000000 | 79.61828150487227 | 4288455 |
| ... | ... | ... | ... | ... | ... | ... |
| 581 | 279 | 40757663 | 2019-11-10 | 2019-11-10 00:00:00.000000 | Cancer | 40757663 |
| 582 | 494 | 40757663 | 2020-08-28 | 2020-08-28 00:00:00.000000 | Cancer | 40757663 |
| 583 | 282 | 40757663 | 2020-02-03 | 2020-02-03 00:00:00.000000 | Cancer | 40757663 |
| 584 | 284 | 40757663 | 2020-11-21 | 2020-11-21 00:00:00.000000 | Cancer | 40757663 |
| 585 | 284 | 40757663 | 2019-09-06 | 2019-09-06 00:00:00.000000 | Cancer | 40757663 |
585 rows × 6 columns
cdm.logs
{'meta': {'version': '0.0.0',
'created_by': 'calummacdonald',
'created_at': '2022-06-17T134916',
'dataset': 'CommonDataModel',
'total_data_processed': {'person': 498,
'observation': 585,
'condition_occurrence': 1298,
'drug_exposure': 604}},
'person': {'0x10a8f3c70': {'required_fields': {'gender_concept_id': {'before': 500,
'after': 284,
'after_formatting': 284},
'birth_datetime': {'before': 284, 'after': 284, 'after_formatting': 284}},
'source_files': {'birth_datetime': {'table': 'Demographics.csv',
'field': 'Age'},
'gender_concept_id': {'table': 'Demographics.csv', 'field': 'Sex'},
'gender_source_concept_id': {'table': 'Demographics.csv', 'field': 'Sex'},
'gender_source_value': {'table': 'Demographics.csv', 'field': 'Sex'},
'person_id': {'table': 'Demographics.csv', 'field': 'ID'}},
'original_file': '../data/part2/Demographics.csv',
'name': 'MALE_3025'},
'0x10a919bb0': {'required_fields': {'gender_concept_id': {'before': 500,
'after': 214,
'after_formatting': 214},
'birth_datetime': {'before': 214, 'after': 214, 'after_formatting': 214}},
'source_files': {'birth_datetime': {'table': 'Demographics.csv',
'field': 'Age'},
'gender_concept_id': {'table': 'Demographics.csv', 'field': 'Sex'},
'gender_source_concept_id': {'table': 'Demographics.csv', 'field': 'Sex'},
'gender_source_value': {'table': 'Demographics.csv', 'field': 'Sex'},
'person_id': {'table': 'Demographics.csv', 'field': 'ID'}},
'original_file': '../data/part2/Demographics.csv',
'name': 'FEMALE_3026'}},
'observation': {'0x10aba8730': {'required_fields': {'person_id': {'before': 204,
'after': 204},
'observation_concept_id': {'before': 204,
'after': 204,
'after_formatting': 204},
'observation_datetime': {'before': 204,
'after': 204,
'after_formatting': 204}},
'source_files': {'observation_concept_id': {'table': 'Serology.csv',
'field': 'IgG'},
'observation_datetime': {'table': 'Serology.csv', 'field': 'Date'},
'observation_source_concept_id': {'table': 'Serology.csv', 'field': 'IgG'},
'observation_source_value': {'table': 'Serology.csv', 'field': 'IgG'},
'person_id': {'table': 'Serology.csv', 'field': 'ID'}},
'original_file': '../data/part2/Serology.csv',
'name': 'Antibody_3027',
'valid_person_id': {'before': 204, 'after': 204}},
'0x10aba8c70': {'required_fields': {'person_id': {'before': 600,
'after': 600},
'observation_concept_id': {'before': 600,
'after': 113,
'after_formatting': 113},
'observation_datetime': {'before': 113,
'after': 113,
'after_formatting': 113}},
'source_files': {'observation_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_datetime': {'table': 'Hospital_Visit.csv',
'field': 'admission_date'},
'observation_source_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_source_value': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'person_id': {'table': 'Hospital_Visit.csv', 'field': 'ID'}},
'original_file': '../data/part2/Hospital_Visit.csv',
'name': 'H_O_heart_failure_3043',
'valid_person_id': {'before': 113, 'after': 113}},
'0x10abb9cd0': {'required_fields': {'person_id': {'before': 600,
'after': 600},
'observation_concept_id': {'before': 600,
'after': 94,
'after_formatting': 94},
'observation_datetime': {'before': 94,
'after': 94,
'after_formatting': 94}},
'source_files': {'observation_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_datetime': {'table': 'Hospital_Visit.csv',
'field': 'admission_date'},
'observation_source_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_source_value': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'person_id': {'table': 'Hospital_Visit.csv', 'field': 'ID'}},
'original_file': '../data/part2/Hospital_Visit.csv',
'name': '2019_nCoV_3044',
'valid_person_id': {'before': 94, 'after': 94}},
'0x10abc4f10': {'required_fields': {'person_id': {'before': 600,
'after': 600},
'observation_concept_id': {'before': 600,
'after': 174,
'after_formatting': 174},
'observation_datetime': {'before': 174,
'after': 174,
'after_formatting': 174}},
'source_files': {'observation_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_datetime': {'table': 'Hospital_Visit.csv',
'field': 'admission_date'},
'observation_source_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'observation_source_value': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'person_id': {'table': 'Hospital_Visit.csv', 'field': 'ID'}},
'original_file': '../data/part2/Hospital_Visit.csv',
'name': 'Cancer_3045',
'valid_person_id': {'before': 174, 'after': 174}}},
'condition_occurrence': {'0x10abc49a0': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 126,
'after_formatting': 125},
'condition_start_datetime': {'before': 126,
'after': 125,
'after_formatting': 125}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Headache'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv',
'field': 'Headache'},
'condition_source_value': {'table': 'Symptoms.csv', 'field': 'Headache'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Headache_3028',
'valid_person_id': {'before': 125, 'after': 123}},
'0x10abc4280': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 113,
'after_formatting': 112},
'condition_start_datetime': {'before': 113,
'after': 112,
'after_formatting': 112}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Fatigue'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv',
'field': 'Fatigue'},
'condition_source_value': {'table': 'Symptoms.csv', 'field': 'Fatigue'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Fatigue_3029',
'valid_person_id': {'before': 112, 'after': 110}},
'0x10ac069d0': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 77,
'after_formatting': 77},
'condition_start_datetime': {'before': 77,
'after': 77,
'after_formatting': 77}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Dizzy'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv', 'field': 'Dizzy'},
'condition_source_value': {'table': 'Symptoms.csv', 'field': 'Dizzy'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Dizziness_3030',
'valid_person_id': {'before': 77, 'after': 77}},
'0x10ac0bf10': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 105,
'after_formatting': 105},
'condition_start_datetime': {'before': 105,
'after': 105,
'after_formatting': 105}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Cough'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv', 'field': 'Cough'},
'condition_source_value': {'table': 'Symptoms.csv', 'field': 'Cough'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Cough_3031',
'valid_person_id': {'before': 105, 'after': 103}},
'0x10ac05970': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 37,
'after_formatting': 37},
'condition_start_datetime': {'before': 37,
'after': 37,
'after_formatting': 37}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Fever'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv', 'field': 'Fever'},
'condition_source_value': {'table': 'Symptoms.csv', 'field': 'Fever'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Fever_3032',
'valid_person_id': {'before': 37, 'after': 37}},
'0x10ac2eb20': {'required_fields': {'person_id': {'before': 165,
'after': 165},
'condition_concept_id': {'before': 165,
'after': 18,
'after_formatting': 18},
'condition_start_datetime': {'before': 18,
'after': 18,
'after_formatting': 18}},
'source_files': {'condition_concept_id': {'table': 'Symptoms.csv',
'field': 'Muscle_Pain'},
'condition_end_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'condition_source_concept_id': {'table': 'Symptoms.csv',
'field': 'Muscle_Pain'},
'condition_source_value': {'table': 'Symptoms.csv',
'field': 'Muscle_Pain'},
'condition_start_datetime': {'table': 'Symptoms.csv',
'field': 'date_occurrence'},
'person_id': {'table': 'Symptoms.csv', 'field': 'ID'}},
'original_file': '../data/part2/Symptoms.csv',
'name': 'Muscle_pain_3033',
'valid_person_id': {'before': 18, 'after': 18}},
'0x10ac24190': {'required_fields': {'person_id': {'before': 600,
'after': 600},
'condition_concept_id': {'before': 600,
'after': 91,
'after_formatting': 91},
'condition_start_datetime': {'before': 91,
'after': 91,
'after_formatting': 91}},
'source_files': {'condition_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'condition_end_datetime': {'table': 'Hospital_Visit.csv',
'field': 'admission_date'},
'condition_source_concept_id': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'condition_source_value': {'table': 'Hospital_Visit.csv',
'field': 'reason'},
'condition_start_datetime': {'table': 'Hospital_Visit.csv',
'field': 'admission_date'},
'person_id': {'table': 'Hospital_Visit.csv', 'field': 'ID'}},
'original_file': '../data/part2/Hospital_Visit.csv',
'name': 'Pneumonia_3042',
'valid_person_id': {'before': 91, 'after': 90}},
'0x10abc4820': {'required_fields': {'person_id': {'before': 967,
'after': 967},
'condition_concept_id': {'before': 967,
'after': 225,
'after_formatting': 225},
'condition_start_datetime': {'before': 225,
'after': 225,
'after_formatting': 225}},
'source_files': {'condition_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_end_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'condition_source_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_source_value': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_start_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'person_id': {'table': 'GP_Records.csv', 'field': 'ID'}},
'original_file': '../data/part2/GP_Records.csv',
'name': 'Mental_health_problem_3046',
'valid_person_id': {'before': 225, 'after': 224}},
'0x10ac3fcd0': {'required_fields': {'person_id': {'before': 967,
'after': 967},
'condition_concept_id': {'before': 967,
'after': 225,
'after_formatting': 225},
'condition_start_datetime': {'before': 225,
'after': 225,
'after_formatting': 225}},
'source_files': {'condition_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_end_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'condition_source_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_source_value': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_start_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'person_id': {'table': 'GP_Records.csv', 'field': 'ID'}},
'original_file': '../data/part2/GP_Records.csv',
'name': 'Mental_disorder_3047',
'valid_person_id': {'before': 225, 'after': 224}},
'0x10ac5df40': {'required_fields': {'person_id': {'before': 967,
'after': 967},
'condition_concept_id': {'before': 967,
'after': 116,
'after_formatting': 116},
'condition_start_datetime': {'before': 116,
'after': 116,
'after_formatting': 116}},
'source_files': {'condition_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_end_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'condition_source_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_source_value': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_start_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'person_id': {'table': 'GP_Records.csv', 'field': 'ID'}},
'original_file': '../data/part2/GP_Records.csv',
'name': 'Type_2_diabetes_mellitus_3048',
'valid_person_id': {'before': 116, 'after': 116}},
'0x10ac71e80': {'required_fields': {'person_id': {'before': 967,
'after': 967},
'condition_concept_id': {'before': 967,
'after': 95,
'after_formatting': 95},
'condition_start_datetime': {'before': 95,
'after': 95,
'after_formatting': 95}},
'source_files': {'condition_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_end_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'condition_source_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_source_value': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_start_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'person_id': {'table': 'GP_Records.csv', 'field': 'ID'}},
'original_file': '../data/part2/GP_Records.csv',
'name': 'Ischemic_heart_disease_3049',
'valid_person_id': {'before': 95, 'after': 95}},
'0x10ac88a60': {'required_fields': {'person_id': {'before': 967,
'after': 967},
'condition_concept_id': {'before': 967,
'after': 81,
'after_formatting': 81},
'condition_start_datetime': {'before': 81,
'after': 81,
'after_formatting': 81}},
'source_files': {'condition_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_end_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'condition_source_concept_id': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_source_value': {'table': 'GP_Records.csv',
'field': 'comorbidity'},
'condition_start_datetime': {'table': 'GP_Records.csv',
'field': 'date_of_visit'},
'person_id': {'table': 'GP_Records.csv', 'field': 'ID'}},
'original_file': '../data/part2/GP_Records.csv',
'name': 'Hypertensive_disorder_3050',
'valid_person_id': {'before': 81, 'after': 81}}},
'drug_exposure': {'0x10aca01f0': {'required_fields': {'person_id': {'before': 360,
'after': 360},
'drug_concept_id': {'before': 360, 'after': 129, 'after_formatting': 129},
'drug_exposure_start_datetime': {'before': 129,
'after': 129,
'after_formatting': 129}},
'source_files': {'drug_concept_id': {'table': 'Vaccinations.csv',
'field': 'type'},
'drug_exposure_end_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_source_concept_id': {'table': 'Vaccinations.csv', 'field': 'type'},
'drug_source_value': {'table': 'Vaccinations.csv', 'field': 'type'},
'person_id': {'table': 'Vaccinations.csv', 'field': 'ID'}},
'original_file': '../data/part2/Vaccinations.csv',
'name': 'COVID_19_vaccine_3034',
'valid_person_id': {'before': 129, 'after': 129}},
'0x10aca0970': {'required_fields': {'person_id': {'before': 360,
'after': 360},
'drug_concept_id': {'before': 360, 'after': 114, 'after_formatting': 114},
'drug_exposure_start_datetime': {'before': 114,
'after': 114,
'after_formatting': 114}},
'source_files': {'drug_concept_id': {'table': 'Vaccinations.csv',
'field': 'type'},
'drug_exposure_end_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_source_concept_id': {'table': 'Vaccinations.csv', 'field': 'type'},
'drug_source_value': {'table': 'Vaccinations.csv', 'field': 'type'},
'person_id': {'table': 'Vaccinations.csv', 'field': 'ID'}},
'original_file': '../data/part2/Vaccinations.csv',
'name': 'COVID_19_vaccine_3035',
'valid_person_id': {'before': 114, 'after': 112}},
'0x10acccd30': {'required_fields': {'person_id': {'before': 360,
'after': 360},
'drug_concept_id': {'before': 360, 'after': 117, 'after_formatting': 117},
'drug_exposure_start_datetime': {'before': 117,
'after': 117,
'after_formatting': 117}},
'source_files': {'drug_concept_id': {'table': 'Vaccinations.csv',
'field': 'type'},
'drug_exposure_end_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_source_concept_id': {'table': 'Vaccinations.csv', 'field': 'type'},
'drug_source_value': {'table': 'Vaccinations.csv', 'field': 'type'},
'person_id': {'table': 'Vaccinations.csv', 'field': 'ID'}},
'original_file': '../data/part2/Vaccinations.csv',
'name': 'COVID_19_vaccine_3036',
'valid_person_id': {'before': 117, 'after': 117}},
'0x10acd6730': {'required_fields': {'person_id': {'before': 360,
'after': 360},
'drug_concept_id': {'before': 360, 'after': 129, 'after_formatting': 129},
'drug_exposure_start_datetime': {'before': 129,
'after': 129,
'after_formatting': 129}},
'source_files': {'drug_concept_id': {'table': 'Vaccinations.csv',
'field': 'type'},
'drug_exposure_end_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_source_concept_id': {'table': 'Vaccinations.csv', 'field': 'type'},
'drug_source_value': {'table': 'Vaccinations.csv', 'field': 'type'},
'person_id': {'table': 'Vaccinations.csv', 'field': 'ID'}},
'original_file': '../data/part2/Vaccinations.csv',
'name': 'SARS_CoV_2_COVID_19_vaccine_mRNA_1273_0_2_MG_ML_Injectable_Suspension_3040',
'valid_person_id': {'before': 129, 'after': 129}},
'0x10aca0610': {'required_fields': {'person_id': {'before': 360,
'after': 360},
'drug_concept_id': {'before': 360, 'after': 117, 'after_formatting': 117},
'drug_exposure_start_datetime': {'before': 117,
'after': 117,
'after_formatting': 117}},
'source_files': {'drug_concept_id': {'table': 'Vaccinations.csv',
'field': 'type'},
'drug_exposure_end_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'table': 'Vaccinations.csv',
'field': 'date_of_vaccination'},
'drug_source_concept_id': {'table': 'Vaccinations.csv', 'field': 'type'},
'drug_source_value': {'table': 'Vaccinations.csv', 'field': 'type'},
'person_id': {'table': 'Vaccinations.csv', 'field': 'ID'}},
'original_file': '../data/part2/Vaccinations.csv',
'name': 'SARS_CoV_2_COVID_19_vaccine_mRNA_BNT162b2_0_1_MG_ML_Injectable_Suspension_3041',
'valid_person_id': {'before': 117, 'after': 117}}}}
table = list(cdm.logs.values())[1]
data = [ obj['required_fields'] for obj in table.values()]
data
[{'gender_concept_id': {'before': 500, 'after': 284, 'after_formatting': 284},
'birth_datetime': {'before': 284, 'after': 284, 'after_formatting': 284}},
{'gender_concept_id': {'before': 500, 'after': 214, 'after_formatting': 214},
'birth_datetime': {'before': 214, 'after': 214, 'after_formatting': 214}}]
person_id_map = cdm.person_id_masker
person_id_map
{'pk1002': 1,
'pk1003': 2,
'pk1004': 3,
'pk1006': 4,
'pk1009': 5,
'pk1010': 6,
'pk1011': 7,
'pk1012': 8,
'pk1013': 9,
'pk1014': 10,
'pk1016': 11,
'pk1021': 12,
'pk1022': 13,
'pk1024': 14,
'pk1025': 15,
'pk1026': 16,
'pk1027': 17,
'pk1028': 18,
'pk1029': 19,
'pk1031': 20,
'pk1032': 21,
'pk1033': 22,
'pk1035': 23,
'pk1036': 24,
'pk1038': 25,
'pk1042': 26,
'pk1044': 27,
'pk1048': 28,
'pk1049': 29,
'pk1050': 30,
'pk1051': 31,
'pk1052': 32,
'pk1053': 33,
'pk1055': 34,
'pk1056': 35,
'pk1057': 36,
'pk1059': 37,
'pk1060': 38,
'pk1064': 39,
'pk1069': 40,
'pk1071': 41,
'pk1072': 42,
'pk1073': 43,
'pk1074': 44,
'pk1075': 45,
'pk1077': 46,
'pk1085': 47,
'pk1086': 48,
'pk1088': 49,
'pk1090': 50,
'pk1091': 51,
'pk1092': 52,
'pk1094': 53,
'pk1095': 54,
'pk1096': 55,
'pk1098': 56,
'pk1099': 57,
'pk1100': 58,
'pk1102': 59,
'pk1103': 60,
'pk1104': 61,
'pk1110': 62,
'pk1112': 63,
'pk1115': 64,
'pk1117': 65,
'pk1118': 66,
'pk1119': 67,
'pk1120': 68,
'pk1122': 69,
'pk1123': 70,
'pk1125': 71,
'pk1127': 72,
'pk1128': 73,
'pk1131': 74,
'pk1132': 75,
'pk1138': 76,
'pk1140': 77,
'pk1142': 78,
'pk1145': 79,
'pk1147': 80,
'pk1148': 81,
'pk1149': 82,
'pk1150': 83,
'pk1156': 84,
'pk1157': 85,
'pk1158': 86,
'pk1160': 87,
'pk1161': 88,
'pk1167': 89,
'pk1170': 90,
'pk1172': 91,
'pk1175': 92,
'pk1176': 93,
'pk1177': 94,
'pk1180': 95,
'pk1181': 96,
'pk1185': 97,
'pk1187': 98,
'pk1188': 99,
'pk1192': 100,
'pk1194': 101,
'pk1195': 102,
'pk1196': 103,
'pk1198': 104,
'pk1200': 105,
'pk1201': 106,
'pk1202': 107,
'pk1205': 108,
'pk1207': 109,
'pk1208': 110,
'pk1210': 111,
'pk1211': 112,
'pk1216': 113,
'pk1221': 114,
'pk1222': 115,
'pk1223': 116,
'pk1226': 117,
'pk1227': 118,
'pk1229': 119,
'pk1230': 120,
'pk1231': 121,
'pk1232': 122,
'pk1233': 123,
'pk1235': 124,
'pk1236': 125,
'pk1239': 126,
'pk1240': 127,
'pk1241': 128,
'pk1242': 129,
'pk1243': 130,
'pk1244': 131,
'pk1245': 132,
'pk1246': 133,
'pk1247': 134,
'pk1249': 135,
'pk1251': 136,
'pk1253': 137,
'pk1257': 138,
'pk1258': 139,
'pk1259': 140,
'pk1260': 141,
'pk1262': 142,
'pk1264': 143,
'pk1265': 144,
'pk1266': 145,
'pk1268': 146,
'pk1271': 147,
'pk1273': 148,
'pk1274': 149,
'pk1275': 150,
'pk1277': 151,
'pk1279': 152,
'pk1283': 153,
'pk1284': 154,
'pk1285': 155,
'pk1286': 156,
'pk1287': 157,
'pk1288': 158,
'pk1290': 159,
'pk1293': 160,
'pk1294': 161,
'pk1297': 162,
'pk1299': 163,
'pk1301': 164,
'pk1302': 165,
'pk1305': 166,
'pk1308': 167,
'pk1309': 168,
'pk1310': 169,
'pk1312': 170,
'pk1313': 171,
'pk1315': 172,
'pk1318': 173,
'pk1319': 174,
'pk1320': 175,
'pk1324': 176,
'pk1325': 177,
'pk1328': 178,
'pk1329': 179,
'pk1331': 180,
'pk1332': 181,
'pk1337': 182,
'pk1338': 183,
'pk1339': 184,
'pk1340': 185,
'pk1341': 186,
'pk1343': 187,
'pk1347': 188,
'pk1350': 189,
'pk1351': 190,
'pk1352': 191,
'pk1353': 192,
'pk1357': 193,
'pk1359': 194,
'pk1360': 195,
'pk1362': 196,
'pk1364': 197,
'pk1366': 198,
'pk1367': 199,
'pk1369': 200,
'pk1370': 201,
'pk1374': 202,
'pk1375': 203,
'pk1376': 204,
'pk1378': 205,
'pk1379': 206,
'pk1380': 207,
'pk1381': 208,
'pk1382': 209,
'pk1384': 210,
'pk1385': 211,
'pk1386': 212,
'pk1387': 213,
'pk1388': 214,
'pk1390': 215,
'pk1391': 216,
'pk1392': 217,
'pk1393': 218,
'pk1396': 219,
'pk1397': 220,
'pk1398': 221,
'pk1399': 222,
'pk1400': 223,
'pk1402': 224,
'pk1403': 225,
'pk1404': 226,
'pk1406': 227,
'pk1407': 228,
'pk1409': 229,
'pk1411': 230,
'pk1412': 231,
'pk1413': 232,
'pk1414': 233,
'pk1415': 234,
'pk1417': 235,
'pk1418': 236,
'pk1419': 237,
'pk1420': 238,
'pk1425': 239,
'pk1428': 240,
'pk1430': 241,
'pk1433': 242,
'pk1435': 243,
'pk1437': 244,
'pk1438': 245,
'pk1440': 246,
'pk1441': 247,
'pk1443': 248,
'pk1444': 249,
'pk1445': 250,
'pk1447': 251,
'pk1448': 252,
'pk1449': 253,
'pk1450': 254,
'pk1451': 255,
'pk1453': 256,
'pk1454': 257,
'pk1455': 258,
'pk1456': 259,
'pk1458': 260,
'pk1459': 261,
'pk1464': 262,
'pk1466': 263,
'pk1469': 264,
'pk1471': 265,
'pk1472': 266,
'pk1473': 267,
'pk1474': 268,
'pk1475': 269,
'pk1476': 270,
'pk1477': 271,
'pk1479': 272,
'pk1481': 273,
'pk1484': 274,
'pk1485': 275,
'pk1486': 276,
'pk1488': 277,
'pk1489': 278,
'pk1490': 279,
'pk1491': 280,
'pk1492': 281,
'pk1494': 282,
'pk1496': 283,
'pk1497': 284,
'pk1001': 285,
'pk1005': 286,
'pk1007': 287,
'pk1008': 288,
'pk1015': 289,
'pk1017': 290,
'pk1018': 291,
'pk1019': 292,
'pk1020': 293,
'pk1030': 294,
'pk1034': 295,
'pk1037': 296,
'pk1039': 297,
'pk1040': 298,
'pk1041': 299,
'pk1043': 300,
'pk1045': 301,
'pk1046': 302,
'pk1047': 303,
'pk1054': 304,
'pk1058': 305,
'pk1061': 306,
'pk1062': 307,
'pk1063': 308,
'pk1065': 309,
'pk1066': 310,
'pk1067': 311,
'pk1068': 312,
'pk1070': 313,
'pk1076': 314,
'pk1078': 315,
'pk1079': 316,
'pk1080': 317,
'pk1081': 318,
'pk1082': 319,
'pk1083': 320,
'pk1084': 321,
'pk1087': 322,
'pk1089': 323,
'pk1093': 324,
'pk1097': 325,
'pk1101': 326,
'pk1105': 327,
'pk1106': 328,
'pk1107': 329,
'pk1108': 330,
'pk1109': 331,
'pk1111': 332,
'pk1113': 333,
'pk1114': 334,
'pk1116': 335,
'pk1121': 336,
'pk1124': 337,
'pk1126': 338,
'pk1129': 339,
'pk1130': 340,
'pk1133': 341,
'pk1134': 342,
'pk1135': 343,
'pk1136': 344,
'pk1137': 345,
'pk1139': 346,
'pk1141': 347,
'pk1143': 348,
'pk1144': 349,
'pk1146': 350,
'pk1151': 351,
'pk1152': 352,
'pk1153': 353,
'pk1154': 354,
'pk1155': 355,
'pk1159': 356,
'pk1162': 357,
'pk1163': 358,
'pk1164': 359,
'pk1165': 360,
'pk1166': 361,
'pk1168': 362,
'pk1169': 363,
'pk1171': 364,
'pk1173': 365,
'pk1174': 366,
'pk1178': 367,
'pk1179': 368,
'pk1182': 369,
'pk1183': 370,
'pk1184': 371,
'pk1186': 372,
'pk1189': 373,
'pk1190': 374,
'pk1191': 375,
'pk1193': 376,
'pk1199': 377,
'pk1203': 378,
'pk1204': 379,
'pk1206': 380,
'pk1209': 381,
'pk1212': 382,
'pk1213': 383,
'pk1214': 384,
'pk1215': 385,
'pk1217': 386,
'pk1218': 387,
'pk1219': 388,
'pk1220': 389,
'pk1224': 390,
'pk1225': 391,
'pk1228': 392,
'pk1234': 393,
'pk1237': 394,
'pk1238': 395,
'pk1248': 396,
'pk1250': 397,
'pk1252': 398,
'pk1254': 399,
'pk1255': 400,
'pk1256': 401,
'pk1261': 402,
'pk1263': 403,
'pk1267': 404,
'pk1269': 405,
'pk1270': 406,
'pk1272': 407,
'pk1276': 408,
'pk1278': 409,
'pk1280': 410,
'pk1281': 411,
'pk1282': 412,
'pk1289': 413,
'pk1291': 414,
'pk1292': 415,
'pk1295': 416,
'pk1296': 417,
'pk1298': 418,
'pk1300': 419,
'pk1303': 420,
'pk1304': 421,
'pk1306': 422,
'pk1307': 423,
'pk1311': 424,
'pk1314': 425,
'pk1316': 426,
'pk1317': 427,
'pk1321': 428,
'pk1322': 429,
'pk1323': 430,
'pk1326': 431,
'pk1327': 432,
'pk1330': 433,
'pk1333': 434,
'pk1334': 435,
'pk1335': 436,
'pk1336': 437,
'pk1342': 438,
'pk1344': 439,
'pk1345': 440,
'pk1346': 441,
'pk1348': 442,
'pk1349': 443,
'pk1354': 444,
'pk1355': 445,
'pk1356': 446,
'pk1358': 447,
'pk1361': 448,
'pk1363': 449,
'pk1365': 450,
'pk1368': 451,
'pk1371': 452,
'pk1372': 453,
'pk1373': 454,
'pk1377': 455,
'pk1383': 456,
'pk1389': 457,
'pk1394': 458,
'pk1395': 459,
'pk1401': 460,
'pk1405': 461,
'pk1408': 462,
'pk1410': 463,
'pk1416': 464,
'pk1421': 465,
'pk1422': 466,
'pk1423': 467,
'pk1424': 468,
'pk1426': 469,
'pk1427': 470,
'pk1429': 471,
'pk1431': 472,
'pk1432': 473,
'pk1434': 474,
'pk1436': 475,
'pk1439': 476,
'pk1442': 477,
'pk1446': 478,
'pk1452': 479,
'pk1457': 480,
'pk1460': 481,
'pk1461': 482,
'pk1462': 483,
'pk1463': 484,
'pk1465': 485,
'pk1467': 486,
'pk1468': 487,
'pk1470': 488,
'pk1478': 489,
'pk1480': 490,
'pk1482': 491,
'pk1483': 492,
'pk1487': 493,
'pk1493': 494,
'pk1495': 495,
'pk1498': 496,
'pk1499': 497,
'pk1500': 498}
Reset and change¶
Reset the CDM - remove created dataframes and removing any indexing
cdm.reset()
2022-06-17 14:49:22 - LocalDataCollection - INFO - resetting used bricks
Change the default index start value for observation
cdm.set_indexing_map({'observation':10000000000})
cdm.process()
2022-06-17 14:49:22 - CommonDataModel - INFO - Starting processing in order: ['person', 'observation', 'condition_occurrence', 'drug_exposure']
2022-06-17 14:49:22 - CommonDataModel - INFO - Number of objects to process for each table...
{
"person": 2,
"observation": 4,
"condition_occurrence": 12,
"drug_exposure": 5
}
2022-06-17 14:49:22 - CommonDataModel - INFO - for person: found 2 objects
2022-06-17 14:49:22 - CommonDataModel - INFO - working on person
2022-06-17 14:49:22 - CommonDataModel - INFO - starting on MALE 3025
2022-06-17 14:49:22 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:22 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (person) has not be passed, so starting from 1
2022-06-17 14:49:22 - Person - INFO - Called apply_rules
2022-06-17 14:49:22 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Demographics.csv' for the first time
2022-06-17 14:49:22 - Person - INFO - Mapped birth_datetime
2022-06-17 14:49:22 - Person - INFO - Mapped gender_concept_id
2022-06-17 14:49:22 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 14:49:22 - Person - INFO - Mapped gender_source_value
2022-06-17 14:49:22 - Person - INFO - Mapped person_id
2022-06-17 14:49:22 - Person - WARNING - Requiring non-null values in gender_concept_id removed 216 rows, leaving 284 rows.
2022-06-17 14:49:22 - Person - INFO - Automatically formatting data columns.
2022-06-17 14:49:22 - Person - INFO - created df (0x10ac2ed60)[MALE_3025]
2022-06-17 14:49:22 - CommonDataModel - INFO - finished MALE 3025 (0x10ac2ed60) ... 1/2 completed, 284 rows
2022-06-17 14:49:22 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:22 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (person) has not be passed, so starting from 1
2022-06-17 14:49:22 - LocalDataCollection - INFO - saving person_ids to ./test_outputs/local//person_ids.csv
2022-06-17 14:49:22 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:22 - CommonDataModel - INFO - starting on FEMALE 3026
2022-06-17 14:49:22 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:22 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (person) has not be passed, so starting from 1
2022-06-17 14:49:22 - Person - INFO - Called apply_rules
2022-06-17 14:49:23 - Person - INFO - Mapped birth_datetime
2022-06-17 14:49:23 - Person - INFO - Mapped gender_concept_id
2022-06-17 14:49:23 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 14:49:23 - Person - INFO - Mapped gender_source_value
2022-06-17 14:49:23 - Person - INFO - Mapped person_id
2022-06-17 14:49:23 - Person - WARNING - Requiring non-null values in gender_concept_id removed 286 rows, leaving 214 rows.
2022-06-17 14:49:23 - Person - INFO - Automatically formatting data columns.
2022-06-17 14:49:23 - Person - INFO - created df (0x10ac244f0)[FEMALE_3026]
2022-06-17 14:49:23 - CommonDataModel - INFO - finished FEMALE 3026 (0x10ac244f0) ... 2/2 completed, 214 rows
2022-06-17 14:49:23 - LocalDataCollection - INFO - updating person_ids in ./test_outputs/local//person_ids.csv
2022-06-17 14:49:23 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:23 - CommonDataModel - INFO - saving dataframe (0x10ac24eb0) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:23 - LocalDataCollection - INFO - saving person to ./test_outputs/local//person.csv
2022-06-17 14:49:23 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:23 - CommonDataModel - INFO - finalised person on iteration 0 producing 498 rows from 2 tables
2022-06-17 14:49:23 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:23 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:23 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:23 - CommonDataModel - INFO - for observation: found 4 objects
2022-06-17 14:49:23 - CommonDataModel - INFO - working on observation
2022-06-17 14:49:23 - CommonDataModel - INFO - starting on Antibody 3027
2022-06-17 14:49:23 - Observation - INFO - Called apply_rules
2022-06-17 14:49:23 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Serology.csv' for the first time
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:23 - Observation - INFO - Mapped person_id
2022-06-17 14:49:23 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:23 - Observation - INFO - created df (0x10ac71880)[Antibody_3027]
2022-06-17 14:49:23 - CommonDataModel - INFO - finished Antibody 3027 (0x10ac71880) ... 1/4 completed, 204 rows
2022-06-17 14:49:23 - CommonDataModel - INFO - starting on H/O: heart failure 3043
2022-06-17 14:49:23 - Observation - INFO - Called apply_rules
2022-06-17 14:49:23 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit.csv' for the first time
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:23 - Observation - INFO - Mapped person_id
2022-06-17 14:49:23 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 487 rows, leaving 113 rows.
2022-06-17 14:49:23 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:23 - Observation - INFO - created df (0x10abb9820)[H_O_heart_failure_3043]
2022-06-17 14:49:23 - CommonDataModel - INFO - finished H/O: heart failure 3043 (0x10abb9820) ... 2/4 completed, 113 rows
2022-06-17 14:49:23 - CommonDataModel - INFO - starting on 2019-nCoV 3044
2022-06-17 14:49:23 - Observation - INFO - Called apply_rules
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:23 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:23 - Observation - INFO - Mapped person_id
2022-06-17 14:49:23 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 506 rows, leaving 94 rows.
2022-06-17 14:49:23 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:23 - Observation - INFO - created df (0x10aca08e0)[2019_nCoV_3044]
2022-06-17 14:49:23 - CommonDataModel - INFO - finished 2019-nCoV 3044 (0x10aca08e0) ... 3/4 completed, 94 rows
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Cancer 3045
2022-06-17 14:49:24 - Observation - INFO - Called apply_rules
2022-06-17 14:49:24 - Observation - INFO - Mapped observation_concept_id
2022-06-17 14:49:24 - Observation - INFO - Mapped observation_datetime
2022-06-17 14:49:24 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 14:49:24 - Observation - INFO - Mapped observation_source_value
2022-06-17 14:49:24 - Observation - INFO - Mapped person_id
2022-06-17 14:49:24 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 426 rows, leaving 174 rows.
2022-06-17 14:49:24 - Observation - INFO - Automatically formatting data columns.
2022-06-17 14:49:24 - Observation - INFO - created df (0x10aba89a0)[Cancer_3045]
2022-06-17 14:49:24 - CommonDataModel - INFO - finished Cancer 3045 (0x10aba89a0) ... 4/4 completed, 174 rows
2022-06-17 14:49:24 - CommonDataModel - INFO - saving dataframe (0x10a8f3670) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:24 - LocalDataCollection - INFO - saving observation to ./test_outputs/local//observation.csv
2022-06-17 14:49:24 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:24 - CommonDataModel - INFO - finalised observation on iteration 0 producing 585 rows from 4 tables
2022-06-17 14:49:24 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:24 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:24 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:24 - CommonDataModel - INFO - for condition_occurrence: found 12 objects
2022-06-17 14:49:24 - CommonDataModel - INFO - working on condition_occurrence
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Headache 3028
2022-06-17 14:49:24 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:24 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:24 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Symptoms.csv' for the first time
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 39 rows, leaving 126 rows.
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 1 rows, leaving 125 rows.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - created df (0x10ac2e490)[Headache_3028]
2022-06-17 14:49:24 - CommonDataModel - INFO - finished Headache 3028 (0x10ac2e490) ... 1/12 completed, 125 rows
2022-06-17 14:49:24 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:24 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:24 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:24 - CommonDataModel - ERROR - 123/125 were good, 2 studies are removed.
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Fatigue 3029
2022-06-17 14:49:24 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:24 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 52 rows, leaving 113 rows.
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 1 rows, leaving 112 rows.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - created df (0x10a88fee0)[Fatigue_3029]
2022-06-17 14:49:24 - CommonDataModel - INFO - finished Fatigue 3029 (0x10a88fee0) ... 2/12 completed, 112 rows
2022-06-17 14:49:24 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:24 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:24 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:24 - CommonDataModel - ERROR - 110/112 were good, 2 studies are removed.
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Dizziness 3030
2022-06-17 14:49:24 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:24 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 88 rows, leaving 77 rows.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - created df (0x10ac246d0)[Dizziness_3030]
2022-06-17 14:49:24 - CommonDataModel - INFO - finished Dizziness 3030 (0x10ac246d0) ... 3/12 completed, 77 rows
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Cough 3031
2022-06-17 14:49:24 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:24 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 60 rows, leaving 105 rows.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:24 - ConditionOccurrence - INFO - created df (0x10abaa3d0)[Cough_3031]
2022-06-17 14:49:24 - CommonDataModel - INFO - finished Cough 3031 (0x10abaa3d0) ... 4/12 completed, 105 rows
2022-06-17 14:49:24 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:24 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:24 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:24 - CommonDataModel - ERROR - 103/105 were good, 2 studies are removed.
2022-06-17 14:49:24 - CommonDataModel - INFO - starting on Fever 3032
2022-06-17 14:49:24 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:24 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:24 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:24 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 128 rows, leaving 37 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10a88f700)[Fever_3032]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Fever 3032 (0x10a88f700) ... 5/12 completed, 37 rows
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Muscle pain 3033
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 147 rows, leaving 18 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10abc4be0)[Muscle_pain_3033]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Muscle pain 3033 (0x10abc4be0) ... 6/12 completed, 18 rows
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Pneumonia 3042
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit.csv' for the first time
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 509 rows, leaving 91 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10ac06fd0)[Pneumonia_3042]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Pneumonia 3042 (0x10ac06fd0) ... 7/12 completed, 91 rows
2022-06-17 14:49:25 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:25 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:25 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:25 - CommonDataModel - ERROR - 90/91 were good, 1 studies are removed.
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Mental health problem 3046
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - LocalDataCollection - INFO - Retrieving initial dataframe for 'GP_Records.csv' for the first time
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 742 rows, leaving 225 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10a88f9d0)[Mental_health_problem_3046]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Mental health problem 3046 (0x10a88f9d0) ... 8/12 completed, 225 rows
2022-06-17 14:49:25 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:25 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:25 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:25 - CommonDataModel - ERROR - 224/225 were good, 1 studies are removed.
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Mental disorder 3047
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 742 rows, leaving 225 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10abf67f0)[Mental_disorder_3047]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Mental disorder 3047 (0x10abf67f0) ... 9/12 completed, 225 rows
2022-06-17 14:49:25 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:25 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:25 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:25 - CommonDataModel - ERROR - 224/225 were good, 1 studies are removed.
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Type 2 diabetes mellitus 3048
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 851 rows, leaving 116 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10aba3910)[Type_2_diabetes_mellitus_3048]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Type 2 diabetes mellitus 3048 (0x10aba3910) ... 10/12 completed, 116 rows
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Ischemic heart disease 3049
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 872 rows, leaving 95 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10abc49d0)[Ischemic_heart_disease_3049]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Ischemic heart disease 3049 (0x10abc49d0) ... 11/12 completed, 95 rows
2022-06-17 14:49:25 - CommonDataModel - INFO - starting on Hypertensive disorder 3050
2022-06-17 14:49:25 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:25 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (condition_occurrence) has not be passed, so starting from 1
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 14:49:25 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 886 rows, leaving 81 rows.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 14:49:25 - ConditionOccurrence - INFO - created df (0x10a919040)[Hypertensive_disorder_3050]
2022-06-17 14:49:25 - CommonDataModel - INFO - finished Hypertensive disorder 3050 (0x10a919040) ... 12/12 completed, 81 rows
2022-06-17 14:49:26 - CommonDataModel - INFO - saving dataframe (0x10acbb3d0) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:26 - LocalDataCollection - INFO - saving condition_occurrence to ./test_outputs/local//condition_occurrence.csv
2022-06-17 14:49:26 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:26 - CommonDataModel - INFO - finalised condition_occurrence on iteration 0 producing 1298 rows from 12 tables
2022-06-17 14:49:26 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:26 - LocalDataCollection - INFO - All input files for this object have now been used.
2022-06-17 14:49:26 - LocalDataCollection - INFO - resetting used bricks
2022-06-17 14:49:26 - CommonDataModel - INFO - for drug_exposure: found 5 objects
2022-06-17 14:49:26 - CommonDataModel - INFO - working on drug_exposure
2022-06-17 14:49:26 - CommonDataModel - INFO - starting on COVID-19 vaccine 3034
2022-06-17 14:49:26 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:26 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (drug_exposure) has not be passed, so starting from 1
2022-06-17 14:49:26 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:26 - LocalDataCollection - INFO - Retrieving initial dataframe for 'Vaccinations.csv' for the first time
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:26 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 231 rows, leaving 129 rows.
2022-06-17 14:49:26 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:26 - DrugExposure - INFO - created df (0x10ace3550)[COVID_19_vaccine_3034]
2022-06-17 14:49:26 - CommonDataModel - INFO - finished COVID-19 vaccine 3034 (0x10ace3550) ... 1/5 completed, 129 rows
2022-06-17 14:49:26 - CommonDataModel - INFO - starting on COVID-19 vaccine 3035
2022-06-17 14:49:26 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:26 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (drug_exposure) has not be passed, so starting from 1
2022-06-17 14:49:26 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:26 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 246 rows, leaving 114 rows.
2022-06-17 14:49:26 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:26 - DrugExposure - INFO - created df (0x10ace3eb0)[COVID_19_vaccine_3035]
2022-06-17 14:49:26 - CommonDataModel - INFO - finished COVID-19 vaccine 3035 (0x10ace3eb0) ... 2/5 completed, 114 rows
2022-06-17 14:49:26 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 14:49:26 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 14:49:26 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 14:49:26 - CommonDataModel - ERROR - 112/114 were good, 2 studies are removed.
2022-06-17 14:49:26 - CommonDataModel - INFO - starting on COVID-19 vaccine 3036
2022-06-17 14:49:26 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:26 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (drug_exposure) has not be passed, so starting from 1
2022-06-17 14:49:26 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:26 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 243 rows, leaving 117 rows.
2022-06-17 14:49:26 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:26 - DrugExposure - INFO - created df (0x10accca00)[COVID_19_vaccine_3036]
2022-06-17 14:49:26 - CommonDataModel - INFO - finished COVID-19 vaccine 3036 (0x10accca00) ... 3/5 completed, 117 rows
2022-06-17 14:49:26 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040
2022-06-17 14:49:26 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:26 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (drug_exposure) has not be passed, so starting from 1
2022-06-17 14:49:26 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:26 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 231 rows, leaving 129 rows.
2022-06-17 14:49:26 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:26 - DrugExposure - INFO - created df (0x10acd67c0)[SARS_CoV_2_COVID_19_vaccine_mRNA_1273_0_2_MG_ML_Injectable_Suspension_3040]
2022-06-17 14:49:26 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 (0x10acd67c0) ... 4/5 completed, 129 rows
2022-06-17 14:49:26 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041
2022-06-17 14:49:26 - CommonDataModel - WARNING - {'observation': 10000000000}
2022-06-17 14:49:26 - CommonDataModel - WARNING - indexing configuration has be parsed but this table (drug_exposure) has not be passed, so starting from 1
2022-06-17 14:49:26 - DrugExposure - INFO - Called apply_rules
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 14:49:26 - DrugExposure - INFO - Mapped person_id
2022-06-17 14:49:26 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 243 rows, leaving 117 rows.
2022-06-17 14:49:26 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 14:49:26 - DrugExposure - INFO - created df (0x10ad37bb0)[SARS_CoV_2_COVID_19_vaccine_mRNA_BNT162b2_0_1_MG_ML_Injectable_Suspension_3041]
2022-06-17 14:49:26 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 (0x10ad37bb0) ... 5/5 completed, 117 rows
2022-06-17 14:49:27 - CommonDataModel - INFO - saving dataframe (0x10a8a6130) to <carrot.io.plugins.local.LocalDataCollection object at 0x10a847a00>
2022-06-17 14:49:27 - LocalDataCollection - INFO - saving drug_exposure to ./test_outputs/local//drug_exposure.csv
2022-06-17 14:49:27 - LocalDataCollection - INFO - finished save to file
2022-06-17 14:49:27 - CommonDataModel - INFO - finalised drug_exposure on iteration 0 producing 604 rows from 5 tables
2022-06-17 14:49:27 - LocalDataCollection - INFO - Getting next chunk of data
2022-06-17 14:49:27 - LocalDataCollection - INFO - All input files for this object have now been used.
cdm['observation'].dropna(axis=1)
| person_id | observation_concept_id | observation_date | observation_datetime | observation_source_value | observation_source_concept_id | |
|---|---|---|---|---|---|---|
| observation_id | ||||||
| 10000000585 | 458 | 4288455 | 2020-11-20 | 2020-11-20 00:00:00.000000 | 61.597638175407624 | 4288455 |
| 10000000586 | 330 | 4288455 | 2022-09-12 | 2022-09-12 00:00:00.000000 | 6.413970727863393 | 4288455 |
| 10000000587 | 136 | 4288455 | 2019-02-02 | 2019-02-02 00:00:00.000000 | 4.691778998076871 | 4288455 |
| 10000000588 | 17 | 4288455 | 2020-08-18 | 2020-08-18 00:00:00.000000 | 14.77684965289811 | 4288455 |
| 10000000589 | 55 | 4288455 | 2019-11-23 | 2019-11-23 00:00:00.000000 | 79.61828150487227 | 4288455 |
| ... | ... | ... | ... | ... | ... | ... |
| 10000001165 | 279 | 40757663 | 2019-11-10 | 2019-11-10 00:00:00.000000 | Cancer | 40757663 |
| 10000001166 | 494 | 40757663 | 2020-08-28 | 2020-08-28 00:00:00.000000 | Cancer | 40757663 |
| 10000001167 | 282 | 40757663 | 2020-02-03 | 2020-02-03 00:00:00.000000 | Cancer | 40757663 |
| 10000001168 | 284 | 40757663 | 2020-11-21 | 2020-11-21 00:00:00.000000 | Cancer | 40757663 |
| 10000001169 | 284 | 40757663 | 2019-09-06 | 2019-09-06 00:00:00.000000 | Cancer | 40757663 |
585 rows × 6 columns