Part 5 SQL
Alternatively it's possible to connect directly to an SQL database
import carrot
import glob
inputs = carrot.tools.create_sql_store(connection_string="postgresql://localhost:5432/ExampleCOVID19DataSet")
2022-06-17 15:04:25 - SqlDataCollection - INFO - DataCollection Object Created
2022-06-17 15:04:25 - SqlDataCollection - INFO - Engine(postgresql://localhost:5432/ExampleCOVID19DataSet)
2022-06-17 15:04:26 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x10c04e790>]
2022-06-17 15:04:26 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10c098550>]
2022-06-17 15:04:26 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10c098a60>]
2022-06-17 15:04:26 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10c0b1d30>]
2022-06-17 15:04:27 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c0badf0>]
2022-06-17 15:04:27 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x10c0bad90>]
2022-06-17 15:04:27 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c0d6310>]
inputs.keys()
dict_keys(['Demographics', 'GP_Records', 'Vaccinations', 'Serology', 'Symptoms', 'Hospital_Visit', 'Blood_Test'])
inputs['Symptoms']
2022-06-17 15:04:27 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Symptoms' for the first time
| ID | date_occurrence | Headache | Fatigue | Dizzy | Cough | Fever | Muscle_Pain | |
|---|---|---|---|---|---|---|---|---|
| 0 | pk1 | 2021-01-24 | Yes | Yes | No | Yes | No | Yes |
| 1 | pk1 | 2019-05-30 | Yes | Yes | No | No | Yes | No |
| 2 | pk1 | 2021-05-16 | Yes | No | No | No | Yes | No |
| 3 | pk1 | 2022-06-11 | Yes | Yes | Yes | Yes | No | No |
| 4 | pk1 | 2020-06-18 | Yes | Yes | Yes | Yes | Yes | Yes |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 56668 | pk9992 | 2021-11-22 | No | No | No | No | No | Yes |
| 56669 | pk9992 | 2019-11-07 | Yes | Yes | No | Yes | Yes | Yes |
| 56670 | pk9992 | 2018-09-01 | Yes | Yes | No | No | Yes | Yes |
| 56671 | pk9993 | 2018-12-02 | No | No | No | No | Yes | No |
| 56672 | pk9996 | 2018-08-11 | Yes | Yes | No | Yes | No | Yes |
56673 rows × 8 columns
rules = carrot.tools.load_json("../data/rules_sql.json")
rules
{'metadata': {'date_created': '2022-02-11T12:22:48.465257',
'dataset': 'FAILED: ExampleV4'},
'cdm': {'person': {'MALE 3025': {'birth_datetime': {'source_table': 'Demographics',
'source_field': 'Age',
'operations': ['get_datetime_from_age']},
'gender_concept_id': {'source_table': 'Demographics',
'source_field': 'Sex',
'term_mapping': {'Male': 8507}},
'gender_source_concept_id': {'source_table': 'Demographics',
'source_field': 'Sex',
'term_mapping': {'Male': 8507}},
'gender_source_value': {'source_table': 'Demographics',
'source_field': 'Sex'},
'person_id': {'source_table': 'Demographics', 'source_field': 'ID'}},
'FEMALE 3026': {'birth_datetime': {'source_table': 'Demographics',
'source_field': 'Age',
'operations': ['get_datetime_from_age']},
'gender_concept_id': {'source_table': 'Demographics',
'source_field': 'Sex',
'term_mapping': {'Female': 8532}},
'gender_source_concept_id': {'source_table': 'Demographics',
'source_field': 'Sex',
'term_mapping': {'Female': 8532}},
'gender_source_value': {'source_table': 'Demographics',
'source_field': 'Sex'},
'person_id': {'source_table': 'Demographics', 'source_field': 'ID'}}},
'observation': {'Antibody 3027': {'observation_concept_id': {'source_table': 'Serology',
'source_field': 'IgG',
'term_mapping': 4288455},
'observation_datetime': {'source_table': 'Serology',
'source_field': 'Date'},
'observation_source_concept_id': {'source_table': 'Serology',
'source_field': 'IgG',
'term_mapping': 4288455},
'observation_source_value': {'source_table': 'Serology',
'source_field': 'IgG'},
'person_id': {'source_table': 'Serology', 'source_field': 'ID'}},
'H/O: heart failure 3043': {'observation_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Heart Attack': 4059317}},
'observation_datetime': {'source_table': 'Hospital_Visit',
'source_field': 'admission_date'},
'observation_source_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Heart Attack': 4059317}},
'observation_source_value': {'source_table': 'Hospital_Visit',
'source_field': 'reason'},
'person_id': {'source_table': 'Hospital_Visit', 'source_field': 'ID'}},
'2019-nCoV 3044': {'observation_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'COVID-19': 37311065}},
'observation_datetime': {'source_table': 'Hospital_Visit',
'source_field': 'admission_date'},
'observation_source_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'COVID-19': 37311065}},
'observation_source_value': {'source_table': 'Hospital_Visit',
'source_field': 'reason'},
'person_id': {'source_table': 'Hospital_Visit', 'source_field': 'ID'}},
'Cancer 3045': {'observation_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Cancer': 40757663}},
'observation_datetime': {'source_table': 'Hospital_Visit',
'source_field': 'admission_date'},
'observation_source_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Cancer': 40757663}},
'observation_source_value': {'source_table': 'Hospital_Visit',
'source_field': 'reason'},
'person_id': {'source_table': 'Hospital_Visit', 'source_field': 'ID'}}},
'condition_occurrence': {'Headache 3028': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Headache',
'term_mapping': {'Yes': 378253}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Headache',
'term_mapping': {'Yes': 378253}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Headache'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Fatigue 3029': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Fatigue',
'term_mapping': {'Yes': 4223659}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Fatigue',
'term_mapping': {'Yes': 4223659}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Fatigue'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Dizziness 3030': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Dizzy',
'term_mapping': {'Yes': 4223938}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Dizzy',
'term_mapping': {'Yes': 4223938}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Dizzy'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Cough 3031': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Cough',
'term_mapping': {'Yes': 254761}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Cough',
'term_mapping': {'Yes': 254761}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Cough'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Fever 3032': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Fever',
'term_mapping': {'Yes': 437663}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Fever',
'term_mapping': {'Yes': 437663}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Fever'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Muscle pain 3033': {'condition_concept_id': {'source_table': 'Symptoms',
'source_field': 'Muscle_Pain',
'term_mapping': {'Yes': 442752}},
'condition_end_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'condition_source_concept_id': {'source_table': 'Symptoms',
'source_field': 'Muscle_Pain',
'term_mapping': {'Yes': 442752}},
'condition_source_value': {'source_table': 'Symptoms',
'source_field': 'Muscle_Pain'},
'condition_start_datetime': {'source_table': 'Symptoms',
'source_field': 'date_occurrence'},
'person_id': {'source_table': 'Symptoms', 'source_field': 'ID'}},
'Pneumonia 3042': {'condition_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Pneumonia': 255848}},
'condition_end_datetime': {'source_table': 'Hospital_Visit',
'source_field': 'admission_date'},
'condition_source_concept_id': {'source_table': 'Hospital_Visit',
'source_field': 'reason',
'term_mapping': {'Pneumonia': 255848}},
'condition_source_value': {'source_table': 'Hospital_Visit',
'source_field': 'reason'},
'condition_start_datetime': {'source_table': 'Hospital_Visit',
'source_field': 'admission_date'},
'person_id': {'source_table': 'Hospital_Visit', 'source_field': 'ID'}},
'Mental health problem 3046': {'condition_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Mental Health': 4131548}},
'condition_end_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'condition_source_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Mental Health': 4131548}},
'condition_source_value': {'source_table': 'GP_Records',
'source_field': 'comorbidity'},
'condition_start_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'person_id': {'source_table': 'GP_Records', 'source_field': 'ID'}},
'Mental disorder 3047': {'condition_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Mental Health': 432586}},
'condition_end_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'condition_source_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Mental Health': 432586}},
'condition_source_value': {'source_table': 'GP_Records',
'source_field': 'comorbidity'},
'condition_start_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'person_id': {'source_table': 'GP_Records', 'source_field': 'ID'}},
'Type 2 diabetes mellitus 3048': {'condition_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Diabetes Type-II': 201826}},
'condition_end_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'condition_source_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Diabetes Type-II': 201826}},
'condition_source_value': {'source_table': 'GP_Records',
'source_field': 'comorbidity'},
'condition_start_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'person_id': {'source_table': 'GP_Records', 'source_field': 'ID'}},
'Ischemic heart disease 3049': {'condition_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Heart Condition': 4185932}},
'condition_end_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'condition_source_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'Heart Condition': 4185932}},
'condition_source_value': {'source_table': 'GP_Records',
'source_field': 'comorbidity'},
'condition_start_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'person_id': {'source_table': 'GP_Records', 'source_field': 'ID'}},
'Hypertensive disorder 3050': {'condition_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'High Blood Pressure': 316866}},
'condition_end_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'condition_source_concept_id': {'source_table': 'GP_Records',
'source_field': 'comorbidity',
'term_mapping': {'High Blood Pressure': 316866}},
'condition_source_value': {'source_table': 'GP_Records',
'source_field': 'comorbidity'},
'condition_start_datetime': {'source_table': 'GP_Records',
'source_field': 'date_of_visit'},
'person_id': {'source_table': 'GP_Records', 'source_field': 'ID'}}},
'drug_exposure': {'COVID-19 vaccine 3034': {'drug_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Moderna': 35894915}},
'drug_exposure_end_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_source_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Moderna': 35894915}},
'drug_source_value': {'source_table': 'Vaccinations',
'source_field': 'type'},
'person_id': {'source_table': 'Vaccinations', 'source_field': 'ID'}},
'COVID-19 vaccine 3035': {'drug_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'AstraZenica': 35894915}},
'drug_exposure_end_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_source_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'AstraZenica': 35894915}},
'drug_source_value': {'source_table': 'Vaccinations',
'source_field': 'type'},
'person_id': {'source_table': 'Vaccinations', 'source_field': 'ID'}},
'COVID-19 vaccine 3036': {'drug_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Pfizer': 35894915}},
'drug_exposure_end_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_source_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Pfizer': 35894915}},
'drug_source_value': {'source_table': 'Vaccinations',
'source_field': 'type'},
'person_id': {'source_table': 'Vaccinations', 'source_field': 'ID'}},
'SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040': {'drug_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Moderna': 37003518}},
'drug_exposure_end_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_source_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Moderna': 37003518}},
'drug_source_value': {'source_table': 'Vaccinations',
'source_field': 'type'},
'person_id': {'source_table': 'Vaccinations', 'source_field': 'ID'}},
'SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041': {'drug_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Pfizer': 37003436}},
'drug_exposure_end_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_exposure_start_datetime': {'source_table': 'Vaccinations',
'source_field': 'date_of_vaccination'},
'drug_source_concept_id': {'source_table': 'Vaccinations',
'source_field': 'type',
'term_mapping': {'Pfizer': 37003436}},
'drug_source_value': {'source_table': 'Vaccinations',
'source_field': 'type'},
'person_id': {'source_table': 'Vaccinations', 'source_field': 'ID'}}}}}
cdm = carrot.cdm.CommonDataModel.from_rules(rules,inputs=inputs)
cdm.process()
2022-06-17 15:04:28 - CommonDataModel - INFO - CommonDataModel (5.3.1) created with co-connect-tools version 0.0.0
2022-06-17 15:04:28 - CommonDataModel - INFO - Running with an DataCollection object
2022-06-17 15:04:28 - CommonDataModel - INFO - Turning on automatic cdm column filling
2022-06-17 15:04:28 - CommonDataModel - INFO - Added MALE 3025 of type person
2022-06-17 15:04:28 - CommonDataModel - INFO - Added FEMALE 3026 of type person
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Antibody 3027 of type observation
2022-06-17 15:04:28 - CommonDataModel - INFO - Added H/O: heart failure 3043 of type observation
2022-06-17 15:04:28 - CommonDataModel - INFO - Added 2019-nCoV 3044 of type observation
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Cancer 3045 of type observation
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Headache 3028 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Fatigue 3029 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Dizziness 3030 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Cough 3031 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Fever 3032 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Muscle pain 3033 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Pneumonia 3042 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Mental health problem 3046 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Mental disorder 3047 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Type 2 diabetes mellitus 3048 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Ischemic heart disease 3049 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added Hypertensive disorder 3050 of type condition_occurrence
2022-06-17 15:04:28 - CommonDataModel - INFO - Added COVID-19 vaccine 3034 of type drug_exposure
2022-06-17 15:04:28 - CommonDataModel - INFO - Added COVID-19 vaccine 3035 of type drug_exposure
2022-06-17 15:04:28 - CommonDataModel - INFO - Added COVID-19 vaccine 3036 of type drug_exposure
2022-06-17 15:04:28 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 of type drug_exposure
2022-06-17 15:04:28 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 of type drug_exposure
2022-06-17 15:04:28 - CommonDataModel - INFO - Starting processing in order: ['person', 'observation', 'condition_occurrence', 'drug_exposure']
2022-06-17 15:04:28 - CommonDataModel - INFO - Number of objects to process for each table...
{
"person": 2,
"observation": 4,
"condition_occurrence": 12,
"drug_exposure": 5
}
2022-06-17 15:04:28 - CommonDataModel - INFO - for person: found 2 objects
2022-06-17 15:04:28 - CommonDataModel - INFO - working on person
2022-06-17 15:04:28 - CommonDataModel - INFO - starting on MALE 3025
2022-06-17 15:04:28 - Person - INFO - Called apply_rules
2022-06-17 15:04:28 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Demographics' for the first time
2022-06-17 15:04:28 - Person - INFO - Mapped birth_datetime
2022-06-17 15:04:28 - Person - INFO - Mapped gender_concept_id
2022-06-17 15:04:28 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 15:04:28 - Person - INFO - Mapped gender_source_value
2022-06-17 15:04:28 - Person - INFO - Mapped person_id
2022-06-17 15:04:28 - Person - WARNING - Requiring non-null values in gender_concept_id removed 210 rows, leaving 138 rows.
2022-06-17 15:04:28 - Person - INFO - Automatically formatting data columns.
2022-06-17 15:04:28 - Person - INFO - created df (0x10c5ac310)[MALE_3025]
2022-06-17 15:04:28 - CommonDataModel - INFO - finished MALE 3025 (0x10c5ac310) ... 1/2 completed, 138 rows
2022-06-17 15:04:28 - CommonDataModel - INFO - starting on FEMALE 3026
2022-06-17 15:04:28 - Person - INFO - Called apply_rules
2022-06-17 15:04:28 - Person - INFO - Mapped birth_datetime
2022-06-17 15:04:28 - Person - INFO - Mapped gender_concept_id
2022-06-17 15:04:28 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 15:04:28 - Person - INFO - Mapped gender_source_value
2022-06-17 15:04:28 - Person - INFO - Mapped person_id
2022-06-17 15:04:28 - Person - WARNING - Requiring non-null values in gender_concept_id removed 215 rows, leaving 133 rows.
2022-06-17 15:04:28 - Person - INFO - Automatically formatting data columns.
2022-06-17 15:04:28 - Person - INFO - created df (0x10c5973a0)[FEMALE_3026]
2022-06-17 15:04:28 - CommonDataModel - INFO - finished FEMALE 3026 (0x10c5973a0) ... 2/2 completed, 133 rows
2022-06-17 15:04:28 - CommonDataModel - INFO - called save_dateframe but outputs are not defined. save_files: True
2022-06-17 15:04:28 - CommonDataModel - INFO - finalised person on iteration 0 producing 271 rows from 2 tables
2022-06-17 15:04:28 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:04:28 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Demographics'
2022-06-17 15:04:28 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:28 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Symptoms'
2022-06-17 15:04:29 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:29 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:04:29 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x10c04ef10>]
2022-06-17 15:04:29 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10c04e790>]
2022-06-17 15:04:29 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10c597430>]
2022-06-17 15:04:29 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10c597e80>]
2022-06-17 15:04:30 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c098910>]
2022-06-17 15:04:30 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x10c581760>]
2022-06-17 15:04:30 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c0cd130>]
2022-06-17 15:04:30 - CommonDataModel - INFO - for observation: found 4 objects
2022-06-17 15:04:30 - CommonDataModel - INFO - working on observation
2022-06-17 15:04:30 - CommonDataModel - INFO - starting on Antibody 3027
2022-06-17 15:04:30 - Observation - INFO - Called apply_rules
2022-06-17 15:04:30 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Serology' for the first time
2022-06-17 15:04:30 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:04:30 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:04:30 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:04:30 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:04:30 - Observation - INFO - Mapped person_id
2022-06-17 15:04:30 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:04:31 - Observation - INFO - created df (0x10c4da040)[Antibody_3027]
2022-06-17 15:04:31 - CommonDataModel - INFO - finished Antibody 3027 (0x10c4da040) ... 1/4 completed, 20591 rows
2022-06-17 15:04:31 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:31 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:31 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:31 - CommonDataModel - ERROR - 105/20591 were good, 20486 studies are removed.
2022-06-17 15:04:31 - CommonDataModel - INFO - starting on H/O: heart failure 3043
2022-06-17 15:04:31 - Observation - INFO - Called apply_rules
2022-06-17 15:04:31 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit' for the first time
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:04:31 - Observation - INFO - Mapped person_id
2022-06-17 15:04:31 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7199 rows, leaving 1511 rows.
2022-06-17 15:04:31 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:04:31 - Observation - INFO - created df (0x10c6973d0)[H_O_heart_failure_3043]
2022-06-17 15:04:31 - CommonDataModel - INFO - finished H/O: heart failure 3043 (0x10c6973d0) ... 2/4 completed, 1511 rows
2022-06-17 15:04:31 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:31 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:31 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:31 - CommonDataModel - ERROR - 11/1511 were good, 1500 studies are removed.
2022-06-17 15:04:31 - CommonDataModel - INFO - starting on 2019-nCoV 3044
2022-06-17 15:04:31 - Observation - INFO - Called apply_rules
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:04:31 - Observation - INFO - Mapped person_id
2022-06-17 15:04:31 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7256 rows, leaving 1454 rows.
2022-06-17 15:04:31 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:04:31 - Observation - INFO - created df (0x10c697700)[2019_nCoV_3044]
2022-06-17 15:04:31 - CommonDataModel - INFO - finished 2019-nCoV 3044 (0x10c697700) ... 3/4 completed, 1454 rows
2022-06-17 15:04:31 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:31 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:31 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:31 - CommonDataModel - ERROR - 12/1454 were good, 1442 studies are removed.
2022-06-17 15:04:31 - CommonDataModel - INFO - starting on Cancer 3045
2022-06-17 15:04:31 - Observation - INFO - Called apply_rules
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:04:31 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:04:31 - Observation - INFO - Mapped person_id
2022-06-17 15:04:31 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7097 rows, leaving 1613 rows.
2022-06-17 15:04:31 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:04:32 - Observation - INFO - created df (0x10c7ad430)[Cancer_3045]
2022-06-17 15:04:32 - CommonDataModel - INFO - finished Cancer 3045 (0x10c7ad430) ... 4/4 completed, 1613 rows
2022-06-17 15:04:32 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:32 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:32 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:32 - CommonDataModel - ERROR - 9/1613 were good, 1604 studies are removed.
2022-06-17 15:04:32 - CommonDataModel - INFO - called save_dateframe but outputs are not defined. save_files: True
2022-06-17 15:04:32 - CommonDataModel - INFO - finalised observation on iteration 0 producing 137 rows from 4 tables
2022-06-17 15:04:32 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:04:32 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Serology'
2022-06-17 15:04:32 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:32 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Hospital_Visit'
2022-06-17 15:04:32 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:32 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:04:32 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x10c45a430>]
2022-06-17 15:04:32 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10c4a4490>]
2022-06-17 15:04:32 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10c04e130>]
2022-06-17 15:04:33 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10c4a47f0>]
2022-06-17 15:04:33 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c7ad8b0>]
2022-06-17 15:04:33 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x10c16d880>]
2022-06-17 15:04:33 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c16d8e0>]
2022-06-17 15:04:33 - CommonDataModel - INFO - for condition_occurrence: found 12 objects
2022-06-17 15:04:33 - CommonDataModel - INFO - working on condition_occurrence
2022-06-17 15:04:33 - CommonDataModel - INFO - starting on Headache 3028
2022-06-17 15:04:33 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:33 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Symptoms' for the first time
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:34 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 20289 rows, leaving 36384 rows.
2022-06-17 15:04:34 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 30 rows, leaving 36354 rows.
2022-06-17 15:04:34 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:35 - ConditionOccurrence - INFO - created df (0x10c16d0d0)[Headache_3028]
2022-06-17 15:04:35 - CommonDataModel - INFO - finished Headache 3028 (0x10c16d0d0) ... 1/12 completed, 36354 rows
2022-06-17 15:04:35 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:35 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:35 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:35 - CommonDataModel - ERROR - 224/36354 were good, 36130 studies are removed.
2022-06-17 15:04:35 - CommonDataModel - INFO - starting on Fatigue 3029
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:35 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:36 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 23601 rows, leaving 33072 rows.
2022-06-17 15:04:36 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 28 rows, leaving 33044 rows.
2022-06-17 15:04:36 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:37 - ConditionOccurrence - INFO - created df (0x10e31feb0)[Fatigue_3029]
2022-06-17 15:04:38 - CommonDataModel - INFO - finished Fatigue 3029 (0x10e31feb0) ... 2/12 completed, 33044 rows
2022-06-17 15:04:38 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:38 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:38 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:38 - CommonDataModel - ERROR - 201/33044 were good, 32843 studies are removed.
2022-06-17 15:04:38 - CommonDataModel - INFO - starting on Dizziness 3030
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:38 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 30706 rows, leaving 25967 rows.
2022-06-17 15:04:38 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 25 rows, leaving 25942 rows.
2022-06-17 15:04:38 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:39 - ConditionOccurrence - INFO - created df (0x10e075c10)[Dizziness_3030]
2022-06-17 15:04:39 - CommonDataModel - INFO - finished Dizziness 3030 (0x10e075c10) ... 3/12 completed, 25942 rows
2022-06-17 15:04:39 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:39 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:39 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:39 - CommonDataModel - ERROR - 149/25942 were good, 25793 studies are removed.
2022-06-17 15:04:39 - CommonDataModel - INFO - starting on Cough 3031
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:39 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 23571 rows, leaving 33102 rows.
2022-06-17 15:04:39 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 26 rows, leaving 33076 rows.
2022-06-17 15:04:39 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:41 - ConditionOccurrence - INFO - created df (0x10d06dc40)[Cough_3031]
2022-06-17 15:04:41 - CommonDataModel - INFO - finished Cough 3031 (0x10d06dc40) ... 4/12 completed, 33076 rows
2022-06-17 15:04:41 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:41 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:41 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:41 - CommonDataModel - ERROR - 208/33076 were good, 32868 studies are removed.
2022-06-17 15:04:41 - CommonDataModel - INFO - starting on Fever 3032
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:41 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:42 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 36272 rows, leaving 20401 rows.
2022-06-17 15:04:42 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 22 rows, leaving 20379 rows.
2022-06-17 15:04:42 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:42 - ConditionOccurrence - INFO - created df (0x10c6bc790)[Fever_3032]
2022-06-17 15:04:43 - CommonDataModel - INFO - finished Fever 3032 (0x10c6bc790) ... 5/12 completed, 20379 rows
2022-06-17 15:04:43 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:43 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:43 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:43 - CommonDataModel - ERROR - 118/20379 were good, 20261 studies are removed.
2022-06-17 15:04:43 - CommonDataModel - INFO - starting on Muscle pain 3033
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:43 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 41813 rows, leaving 14860 rows.
2022-06-17 15:04:43 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 21 rows, leaving 14839 rows.
2022-06-17 15:04:43 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:43 - ConditionOccurrence - INFO - created df (0x10e0e9eb0)[Muscle_pain_3033]
2022-06-17 15:04:44 - CommonDataModel - INFO - finished Muscle pain 3033 (0x10e0e9eb0) ... 6/12 completed, 14839 rows
2022-06-17 15:04:44 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:44 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:44 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:44 - CommonDataModel - ERROR - 84/14839 were good, 14755 studies are removed.
2022-06-17 15:04:44 - CommonDataModel - INFO - starting on Pneumonia 3042
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:44 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit' for the first time
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:44 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 7267 rows, leaving 1443 rows.
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:44 - ConditionOccurrence - INFO - created df (0x10cfb5a90)[Pneumonia_3042]
2022-06-17 15:04:44 - CommonDataModel - INFO - finished Pneumonia 3042 (0x10cfb5a90) ... 7/12 completed, 1443 rows
2022-06-17 15:04:44 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:44 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:44 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:44 - CommonDataModel - ERROR - 9/1443 were good, 1434 studies are removed.
2022-06-17 15:04:44 - CommonDataModel - INFO - starting on Mental health problem 3046
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:44 - SqlDataCollection - INFO - Retrieving initial dataframe for 'GP_Records' for the first time
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:44 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53424 rows, leaving 3421 rows.
2022-06-17 15:04:44 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:45 - ConditionOccurrence - INFO - created df (0x1231b9a00)[Mental_health_problem_3046]
2022-06-17 15:04:45 - CommonDataModel - INFO - finished Mental health problem 3046 (0x1231b9a00) ... 8/12 completed, 3421 rows
2022-06-17 15:04:45 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:45 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:45 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:45 - CommonDataModel - ERROR - 23/3421 were good, 3398 studies are removed.
2022-06-17 15:04:45 - CommonDataModel - INFO - starting on Mental disorder 3047
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:45 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53424 rows, leaving 3421 rows.
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:45 - ConditionOccurrence - INFO - created df (0x125c0a7c0)[Mental_disorder_3047]
2022-06-17 15:04:45 - CommonDataModel - INFO - finished Mental disorder 3047 (0x125c0a7c0) ... 9/12 completed, 3421 rows
2022-06-17 15:04:45 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:45 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:45 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:45 - CommonDataModel - ERROR - 23/3421 were good, 3398 studies are removed.
2022-06-17 15:04:45 - CommonDataModel - INFO - starting on Type 2 diabetes mellitus 3048
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:45 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53808 rows, leaving 3037 rows.
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:45 - ConditionOccurrence - INFO - created df (0x1253ceb80)[Type_2_diabetes_mellitus_3048]
2022-06-17 15:04:45 - CommonDataModel - INFO - finished Type 2 diabetes mellitus 3048 (0x1253ceb80) ... 10/12 completed, 3037 rows
2022-06-17 15:04:45 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:45 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:45 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:45 - CommonDataModel - ERROR - 28/3037 were good, 3009 studies are removed.
2022-06-17 15:04:45 - CommonDataModel - INFO - starting on Ischemic heart disease 3049
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:45 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:46 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53941 rows, leaving 2904 rows.
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:46 - ConditionOccurrence - INFO - created df (0x125c0a700)[Ischemic_heart_disease_3049]
2022-06-17 15:04:46 - CommonDataModel - INFO - finished Ischemic heart disease 3049 (0x125c0a700) ... 11/12 completed, 2904 rows
2022-06-17 15:04:46 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:46 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:46 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:46 - CommonDataModel - ERROR - 22/2904 were good, 2882 studies are removed.
2022-06-17 15:04:46 - CommonDataModel - INFO - starting on Hypertensive disorder 3050
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:04:46 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 54362 rows, leaving 2483 rows.
2022-06-17 15:04:46 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:04:46 - ConditionOccurrence - INFO - created df (0x125c07ca0)[Hypertensive_disorder_3050]
2022-06-17 15:04:46 - CommonDataModel - INFO - finished Hypertensive disorder 3050 (0x125c07ca0) ... 12/12 completed, 2483 rows
2022-06-17 15:04:46 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:46 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:46 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:46 - CommonDataModel - ERROR - 16/2483 were good, 2467 studies are removed.
2022-06-17 15:04:46 - CommonDataModel - ERROR - Removed 2 row(s) due to duplicates found when merging condition_occurrence
2022-06-17 15:04:46 - CommonDataModel - WARNING - Example duplicates...
2022-06-17 15:04:46 - CommonDataModel - WARNING - person_id condition_concept_id condition_start_date \
condition_occurrence_id
7813 44.0 4223659 2020-09-13
7815 44.0 4223659 2020-09-13
3628 44.0 437663 2020-09-13
3630 44.0 437663 2020-09-13
condition_start_datetime condition_end_date \
condition_occurrence_id
7813 2020-09-13 00:00:00.000000 2020-09-13
7815 2020-09-13 00:00:00.000000 2020-09-13
3628 2020-09-13 00:00:00.000000 2020-09-13
3630 2020-09-13 00:00:00.000000 2020-09-13
condition_end_datetime condition_source_value \
condition_occurrence_id
7813 2020-09-13 00:00:00.000000 Yes
7815 2020-09-13 00:00:00.000000 Yes
3628 2020-09-13 00:00:00.000000 Yes
3630 2020-09-13 00:00:00.000000 Yes
condition_source_concept_id
condition_occurrence_id
7813 4223659
7815 4223659
3628 437663
3630 437663
2022-06-17 15:04:46 - CommonDataModel - INFO - called save_dateframe but outputs are not defined. save_files: True
2022-06-17 15:04:46 - CommonDataModel - INFO - finalised condition_occurrence on iteration 0 producing 1105 rows from 12 tables
2022-06-17 15:04:46 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:04:46 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'GP_Records'
2022-06-17 15:04:46 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:46 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Symptoms'
2022-06-17 15:04:46 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:46 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Hospital_Visit'
2022-06-17 15:04:46 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:46 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:04:46 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x10c098760>]
2022-06-17 15:04:46 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10c49eeb0>]
2022-06-17 15:04:47 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10c5ac580>]
2022-06-17 15:04:47 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10c444640>]
2022-06-17 15:04:47 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c6c4040>]
2022-06-17 15:04:47 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x123e91670>]
2022-06-17 15:04:47 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c7adcd0>]
2022-06-17 15:04:47 - CommonDataModel - INFO - for drug_exposure: found 5 objects
2022-06-17 15:04:47 - CommonDataModel - INFO - working on drug_exposure
2022-06-17 15:04:47 - CommonDataModel - INFO - starting on COVID-19 vaccine 3034
2022-06-17 15:04:47 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:04:47 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Vaccinations' for the first time
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:04:48 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:04:48 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 53291 rows, leaving 27645 rows.
2022-06-17 15:04:48 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 27643 rows.
2022-06-17 15:04:48 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:04:49 - DrugExposure - INFO - created df (0x125c159d0)[COVID_19_vaccine_3034]
2022-06-17 15:04:49 - CommonDataModel - INFO - finished COVID-19 vaccine 3034 (0x125c159d0) ... 1/5 completed, 27643 rows
2022-06-17 15:04:49 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:49 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:49 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:49 - CommonDataModel - ERROR - 155/27643 were good, 27488 studies are removed.
2022-06-17 15:04:49 - CommonDataModel - INFO - starting on COVID-19 vaccine 3035
2022-06-17 15:04:49 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:04:49 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:04:49 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54293 rows, leaving 26643 rows.
2022-06-17 15:04:49 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26641 rows.
2022-06-17 15:04:49 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:04:50 - DrugExposure - INFO - created df (0x125c3fd00)[COVID_19_vaccine_3035]
2022-06-17 15:04:51 - CommonDataModel - INFO - finished COVID-19 vaccine 3035 (0x125c3fd00) ... 2/5 completed, 26641 rows
2022-06-17 15:04:51 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:51 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:51 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:51 - CommonDataModel - ERROR - 144/26641 were good, 26497 studies are removed.
2022-06-17 15:04:51 - CommonDataModel - INFO - starting on COVID-19 vaccine 3036
2022-06-17 15:04:51 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:04:51 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:04:51 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54288 rows, leaving 26648 rows.
2022-06-17 15:04:51 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26646 rows.
2022-06-17 15:04:51 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:04:52 - DrugExposure - INFO - created df (0x1246dcb80)[COVID_19_vaccine_3036]
2022-06-17 15:04:52 - CommonDataModel - INFO - finished COVID-19 vaccine 3036 (0x1246dcb80) ... 3/5 completed, 26646 rows
2022-06-17 15:04:52 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:52 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:52 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:52 - CommonDataModel - ERROR - 150/26646 were good, 26496 studies are removed.
2022-06-17 15:04:52 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040
2022-06-17 15:04:52 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:04:52 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:04:52 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 53291 rows, leaving 27645 rows.
2022-06-17 15:04:52 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 27643 rows.
2022-06-17 15:04:52 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:04:53 - DrugExposure - INFO - created df (0x10d4298e0)[SARS_CoV_2_COVID_19_vaccine_mRNA_1273_0_2_MG_ML_Injectable_Suspension_3040]
2022-06-17 15:04:54 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 (0x10d4298e0) ... 4/5 completed, 27643 rows
2022-06-17 15:04:54 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:54 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:54 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:54 - CommonDataModel - ERROR - 155/27643 were good, 27488 studies are removed.
2022-06-17 15:04:54 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041
2022-06-17 15:04:54 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:04:54 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:04:54 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54288 rows, leaving 26648 rows.
2022-06-17 15:04:54 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26646 rows.
2022-06-17 15:04:54 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:04:55 - DrugExposure - INFO - created df (0x125276c40)[SARS_CoV_2_COVID_19_vaccine_mRNA_BNT162b2_0_1_MG_ML_Injectable_Suspension_3041]
2022-06-17 15:04:55 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 (0x125276c40) ... 5/5 completed, 26646 rows
2022-06-17 15:04:55 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:04:55 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:04:55 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:04:55 - CommonDataModel - ERROR - 150/26646 were good, 26496 studies are removed.
2022-06-17 15:04:55 - CommonDataModel - INFO - called save_dateframe but outputs are not defined. save_files: True
2022-06-17 15:04:55 - CommonDataModel - INFO - finalised drug_exposure on iteration 0 producing 754 rows from 5 tables
2022-06-17 15:04:55 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:04:55 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Vaccinations'
2022-06-17 15:04:55 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:55 - SqlDataCollection - INFO - All input files for this object have now been used.
Write outputs to SQL¶
outputs = carrot.tools.create_sql_store(connection_string="postgresql://localhost:5432/ExampleCDMDataSet",
drop_existing=True)
2022-06-17 15:04:55 - SqlDataCollection - INFO - DataCollection Object Created
2022-06-17 15:04:57 - SqlDataCollection - INFO - Engine(postgresql://localhost:5432/ExampleCDMDataSet)
cdm = carrot.cdm.CommonDataModel.from_rules(rules,inputs=inputs,outputs=outputs)
cdm.process()
2022-06-17 15:04:57 - CommonDataModel - INFO - CommonDataModel (5.3.1) created with co-connect-tools version 0.0.0
2022-06-17 15:04:57 - CommonDataModel - INFO - Running with an DataCollection object
2022-06-17 15:04:57 - CommonDataModel - INFO - Turning on automatic cdm column filling
2022-06-17 15:04:57 - CommonDataModel - INFO - Added MALE 3025 of type person
2022-06-17 15:04:57 - CommonDataModel - INFO - Added FEMALE 3026 of type person
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Antibody 3027 of type observation
2022-06-17 15:04:57 - CommonDataModel - INFO - Added H/O: heart failure 3043 of type observation
2022-06-17 15:04:57 - CommonDataModel - INFO - Added 2019-nCoV 3044 of type observation
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Cancer 3045 of type observation
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Headache 3028 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Fatigue 3029 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Dizziness 3030 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Cough 3031 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Fever 3032 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Muscle pain 3033 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Pneumonia 3042 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Mental health problem 3046 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Mental disorder 3047 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Type 2 diabetes mellitus 3048 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Ischemic heart disease 3049 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added Hypertensive disorder 3050 of type condition_occurrence
2022-06-17 15:04:57 - CommonDataModel - INFO - Added COVID-19 vaccine 3034 of type drug_exposure
2022-06-17 15:04:57 - CommonDataModel - INFO - Added COVID-19 vaccine 3035 of type drug_exposure
2022-06-17 15:04:57 - CommonDataModel - INFO - Added COVID-19 vaccine 3036 of type drug_exposure
2022-06-17 15:04:57 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 of type drug_exposure
2022-06-17 15:04:57 - CommonDataModel - INFO - Added SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 of type drug_exposure
2022-06-17 15:04:57 - CommonDataModel - INFO - Starting processing in order: ['person', 'observation', 'condition_occurrence', 'drug_exposure']
2022-06-17 15:04:57 - CommonDataModel - INFO - Number of objects to process for each table...
{
"person": 2,
"observation": 4,
"condition_occurrence": 12,
"drug_exposure": 5
}
2022-06-17 15:04:57 - CommonDataModel - INFO - for person: found 2 objects
2022-06-17 15:04:57 - CommonDataModel - INFO - working on person
2022-06-17 15:04:57 - CommonDataModel - INFO - starting on MALE 3025
2022-06-17 15:04:57 - Person - INFO - Called apply_rules
2022-06-17 15:04:57 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Demographics' for the first time
2022-06-17 15:04:57 - Person - INFO - Mapped birth_datetime
2022-06-17 15:04:57 - Person - INFO - Mapped gender_concept_id
2022-06-17 15:04:57 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 15:04:57 - Person - INFO - Mapped gender_source_value
2022-06-17 15:04:57 - Person - INFO - Mapped person_id
2022-06-17 15:04:57 - Person - WARNING - Requiring non-null values in gender_concept_id removed 210 rows, leaving 138 rows.
2022-06-17 15:04:57 - Person - INFO - Automatically formatting data columns.
2022-06-17 15:04:57 - Person - INFO - created df (0x10e5962e0)[MALE_3025]
2022-06-17 15:04:57 - CommonDataModel - INFO - finished MALE 3025 (0x10e5962e0) ... 1/2 completed, 138 rows
2022-06-17 15:04:57 - SqlDataCollection - INFO - updating person_ids in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:04:57 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:04:57 - CommonDataModel - INFO - starting on FEMALE 3026
2022-06-17 15:04:58 - Person - INFO - Called apply_rules
2022-06-17 15:04:58 - Person - INFO - Mapped birth_datetime
2022-06-17 15:04:58 - Person - INFO - Mapped gender_concept_id
2022-06-17 15:04:58 - Person - INFO - Mapped gender_source_concept_id
2022-06-17 15:04:58 - Person - INFO - Mapped gender_source_value
2022-06-17 15:04:58 - Person - INFO - Mapped person_id
2022-06-17 15:04:58 - Person - WARNING - Requiring non-null values in gender_concept_id removed 215 rows, leaving 133 rows.
2022-06-17 15:04:58 - Person - INFO - Automatically formatting data columns.
2022-06-17 15:04:58 - Person - INFO - created df (0x123943490)[FEMALE_3026]
2022-06-17 15:04:58 - CommonDataModel - INFO - finished FEMALE 3026 (0x123943490) ... 2/2 completed, 133 rows
2022-06-17 15:04:58 - SqlDataCollection - INFO - updating person_ids in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:04:58 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:04:58 - CommonDataModel - INFO - saving dataframe (0x123943370) to <carrot.io.plugins.sql.SqlDataCollection object at 0x10d429760>
2022-06-17 15:04:58 - SqlDataCollection - INFO - updating person in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:04:58 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:04:58 - CommonDataModel - INFO - finalised person on iteration 0 producing 271 rows from 2 tables
2022-06-17 15:04:58 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:04:58 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Demographics'
2022-06-17 15:04:58 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:04:58 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:04:58 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x1239436d0>]
2022-06-17 15:04:58 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x1238615e0>]
2022-06-17 15:04:58 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10c0cd910>]
2022-06-17 15:04:58 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10e0e9e80>]
2022-06-17 15:04:59 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c6c4df0>]
2022-06-17 15:04:59 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x10c6c4040>]
2022-06-17 15:04:59 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c098970>]
2022-06-17 15:04:59 - CommonDataModel - INFO - for observation: found 4 objects
2022-06-17 15:04:59 - CommonDataModel - INFO - working on observation
2022-06-17 15:04:59 - CommonDataModel - INFO - starting on Antibody 3027
2022-06-17 15:04:59 - Observation - INFO - Called apply_rules
2022-06-17 15:04:59 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Serology' for the first time
2022-06-17 15:04:59 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:04:59 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:04:59 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:04:59 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:04:59 - Observation - INFO - Mapped person_id
2022-06-17 15:04:59 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:05:00 - Observation - INFO - created df (0x10d1669a0)[Antibody_3027]
2022-06-17 15:05:00 - CommonDataModel - INFO - finished Antibody 3027 (0x10d1669a0) ... 1/4 completed, 20591 rows
2022-06-17 15:05:00 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:00 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:00 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:00 - CommonDataModel - ERROR - 105/20591 were good, 20486 studies are removed.
2022-06-17 15:05:00 - CommonDataModel - INFO - starting on H/O: heart failure 3043
2022-06-17 15:05:00 - Observation - INFO - Called apply_rules
2022-06-17 15:05:00 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit' for the first time
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:05:00 - Observation - INFO - Mapped person_id
2022-06-17 15:05:00 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7199 rows, leaving 1511 rows.
2022-06-17 15:05:00 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:05:00 - Observation - INFO - created df (0x1252761c0)[H_O_heart_failure_3043]
2022-06-17 15:05:00 - CommonDataModel - INFO - finished H/O: heart failure 3043 (0x1252761c0) ... 2/4 completed, 1511 rows
2022-06-17 15:05:00 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:00 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:00 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:00 - CommonDataModel - ERROR - 11/1511 were good, 1500 studies are removed.
2022-06-17 15:05:00 - CommonDataModel - INFO - starting on 2019-nCoV 3044
2022-06-17 15:05:00 - Observation - INFO - Called apply_rules
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:05:00 - Observation - INFO - Mapped person_id
2022-06-17 15:05:00 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7256 rows, leaving 1454 rows.
2022-06-17 15:05:00 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:05:00 - Observation - INFO - created df (0x10d429640)[2019_nCoV_3044]
2022-06-17 15:05:00 - CommonDataModel - INFO - finished 2019-nCoV 3044 (0x10d429640) ... 3/4 completed, 1454 rows
2022-06-17 15:05:00 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:00 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:00 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:00 - CommonDataModel - ERROR - 12/1454 were good, 1442 studies are removed.
2022-06-17 15:05:00 - CommonDataModel - INFO - starting on Cancer 3045
2022-06-17 15:05:00 - Observation - INFO - Called apply_rules
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_datetime
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_concept_id
2022-06-17 15:05:00 - Observation - INFO - Mapped observation_source_value
2022-06-17 15:05:00 - Observation - INFO - Mapped person_id
2022-06-17 15:05:00 - Observation - WARNING - Requiring non-null values in observation_concept_id removed 7097 rows, leaving 1613 rows.
2022-06-17 15:05:00 - Observation - INFO - Automatically formatting data columns.
2022-06-17 15:05:01 - Observation - INFO - created df (0x10d429dc0)[Cancer_3045]
2022-06-17 15:05:01 - CommonDataModel - INFO - finished Cancer 3045 (0x10d429dc0) ... 4/4 completed, 1613 rows
2022-06-17 15:05:01 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:01 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:01 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:01 - CommonDataModel - ERROR - 9/1613 were good, 1604 studies are removed.
2022-06-17 15:05:01 - CommonDataModel - INFO - saving dataframe (0x1242a9670) to <carrot.io.plugins.sql.SqlDataCollection object at 0x10d429760>
2022-06-17 15:05:01 - SqlDataCollection - INFO - updating observation in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:05:01 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:05:01 - CommonDataModel - INFO - finalised observation on iteration 0 producing 137 rows from 4 tables
2022-06-17 15:05:01 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:05:01 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Serology'
2022-06-17 15:05:01 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:01 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Hospital_Visit'
2022-06-17 15:05:01 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:01 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:05:01 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x10dc730a0>]
2022-06-17 15:05:01 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10c831790>]
2022-06-17 15:05:01 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x1253ce0a0>]
2022-06-17 15:05:01 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x10c4943d0>]
2022-06-17 15:05:02 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c494bb0>]
2022-06-17 15:05:02 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x125c07fd0>]
2022-06-17 15:05:02 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c8315e0>]
2022-06-17 15:05:02 - CommonDataModel - INFO - for condition_occurrence: found 12 objects
2022-06-17 15:05:02 - CommonDataModel - INFO - working on condition_occurrence
2022-06-17 15:05:02 - CommonDataModel - INFO - starting on Headache 3028
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:02 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Symptoms' for the first time
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:02 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 20289 rows, leaving 36384 rows.
2022-06-17 15:05:02 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 30 rows, leaving 36354 rows.
2022-06-17 15:05:02 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:03 - ConditionOccurrence - INFO - created df (0x1253ce4f0)[Headache_3028]
2022-06-17 15:05:04 - CommonDataModel - INFO - finished Headache 3028 (0x1253ce4f0) ... 1/12 completed, 36354 rows
2022-06-17 15:05:04 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:04 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:04 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:04 - CommonDataModel - ERROR - 224/36354 were good, 36130 studies are removed.
2022-06-17 15:05:04 - CommonDataModel - INFO - starting on Fatigue 3029
2022-06-17 15:05:04 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:05 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 23601 rows, leaving 33072 rows.
2022-06-17 15:05:05 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 28 rows, leaving 33044 rows.
2022-06-17 15:05:05 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:06 - ConditionOccurrence - INFO - created df (0x10e0e97c0)[Fatigue_3029]
2022-06-17 15:05:07 - CommonDataModel - INFO - finished Fatigue 3029 (0x10e0e97c0) ... 2/12 completed, 33044 rows
2022-06-17 15:05:07 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:07 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:07 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:07 - CommonDataModel - ERROR - 201/33044 were good, 32843 studies are removed.
2022-06-17 15:05:07 - CommonDataModel - INFO - starting on Dizziness 3030
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:07 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 30706 rows, leaving 25967 rows.
2022-06-17 15:05:07 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 25 rows, leaving 25942 rows.
2022-06-17 15:05:07 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:08 - ConditionOccurrence - INFO - created df (0x123e91280)[Dizziness_3030]
2022-06-17 15:05:09 - CommonDataModel - INFO - finished Dizziness 3030 (0x123e91280) ... 3/12 completed, 25942 rows
2022-06-17 15:05:09 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:09 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:09 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:09 - CommonDataModel - ERROR - 149/25942 were good, 25793 studies are removed.
2022-06-17 15:05:09 - CommonDataModel - INFO - starting on Cough 3031
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:09 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 23571 rows, leaving 33102 rows.
2022-06-17 15:05:09 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 26 rows, leaving 33076 rows.
2022-06-17 15:05:09 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:10 - ConditionOccurrence - INFO - created df (0x125c0a0a0)[Cough_3031]
2022-06-17 15:05:11 - CommonDataModel - INFO - finished Cough 3031 (0x125c0a0a0) ... 4/12 completed, 33076 rows
2022-06-17 15:05:11 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:11 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:11 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:11 - CommonDataModel - ERROR - 208/33076 were good, 32868 studies are removed.
2022-06-17 15:05:11 - CommonDataModel - INFO - starting on Fever 3032
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:11 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 36272 rows, leaving 20401 rows.
2022-06-17 15:05:11 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 22 rows, leaving 20379 rows.
2022-06-17 15:05:11 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:12 - ConditionOccurrence - INFO - created df (0x10e0e92b0)[Fever_3032]
2022-06-17 15:05:13 - CommonDataModel - INFO - finished Fever 3032 (0x10e0e92b0) ... 5/12 completed, 20379 rows
2022-06-17 15:05:13 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:13 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:13 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:13 - CommonDataModel - ERROR - 118/20379 were good, 20261 studies are removed.
2022-06-17 15:05:13 - CommonDataModel - INFO - starting on Muscle pain 3033
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:13 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 41813 rows, leaving 14860 rows.
2022-06-17 15:05:13 - ConditionOccurrence - WARNING - Requiring non-null values in condition_start_datetime removed 21 rows, leaving 14839 rows.
2022-06-17 15:05:13 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:13 - ConditionOccurrence - INFO - created df (0x123e91940)[Muscle_pain_3033]
2022-06-17 15:05:14 - CommonDataModel - INFO - finished Muscle pain 3033 (0x123e91940) ... 6/12 completed, 14839 rows
2022-06-17 15:05:14 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:14 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:14 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:14 - CommonDataModel - ERROR - 84/14839 were good, 14755 studies are removed.
2022-06-17 15:05:14 - CommonDataModel - INFO - starting on Pneumonia 3042
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:14 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Hospital_Visit' for the first time
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:14 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 7267 rows, leaving 1443 rows.
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:14 - ConditionOccurrence - INFO - created df (0x10c6bc910)[Pneumonia_3042]
2022-06-17 15:05:14 - CommonDataModel - INFO - finished Pneumonia 3042 (0x10c6bc910) ... 7/12 completed, 1443 rows
2022-06-17 15:05:14 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:14 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:14 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:14 - CommonDataModel - ERROR - 9/1443 were good, 1434 studies are removed.
2022-06-17 15:05:14 - CommonDataModel - INFO - starting on Mental health problem 3046
2022-06-17 15:05:14 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:14 - SqlDataCollection - INFO - Retrieving initial dataframe for 'GP_Records' for the first time
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:15 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53424 rows, leaving 3421 rows.
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:15 - ConditionOccurrence - INFO - created df (0x125bf2dc0)[Mental_health_problem_3046]
2022-06-17 15:05:15 - CommonDataModel - INFO - finished Mental health problem 3046 (0x125bf2dc0) ... 8/12 completed, 3421 rows
2022-06-17 15:05:15 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:15 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:15 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:15 - CommonDataModel - ERROR - 23/3421 were good, 3398 studies are removed.
2022-06-17 15:05:15 - CommonDataModel - INFO - starting on Mental disorder 3047
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:15 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:16 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53424 rows, leaving 3421 rows.
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:16 - ConditionOccurrence - INFO - created df (0x10c415700)[Mental_disorder_3047]
2022-06-17 15:05:16 - CommonDataModel - INFO - finished Mental disorder 3047 (0x10c415700) ... 9/12 completed, 3421 rows
2022-06-17 15:05:16 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:16 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:16 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:16 - CommonDataModel - ERROR - 23/3421 were good, 3398 studies are removed.
2022-06-17 15:05:16 - CommonDataModel - INFO - starting on Type 2 diabetes mellitus 3048
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:16 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53808 rows, leaving 3037 rows.
2022-06-17 15:05:16 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:16 - ConditionOccurrence - INFO - created df (0x10c4dabb0)[Type_2_diabetes_mellitus_3048]
2022-06-17 15:05:17 - CommonDataModel - INFO - finished Type 2 diabetes mellitus 3048 (0x10c4dabb0) ... 10/12 completed, 3037 rows
2022-06-17 15:05:17 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:17 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:17 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:17 - CommonDataModel - ERROR - 28/3037 were good, 3009 studies are removed.
2022-06-17 15:05:17 - CommonDataModel - INFO - starting on Ischemic heart disease 3049
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:17 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 53941 rows, leaving 2904 rows.
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:17 - ConditionOccurrence - INFO - created df (0x10c0dd8e0)[Ischemic_heart_disease_3049]
2022-06-17 15:05:17 - CommonDataModel - INFO - finished Ischemic heart disease 3049 (0x10c0dd8e0) ... 11/12 completed, 2904 rows
2022-06-17 15:05:17 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:17 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:17 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:17 - CommonDataModel - ERROR - 22/2904 were good, 2882 studies are removed.
2022-06-17 15:05:17 - CommonDataModel - INFO - starting on Hypertensive disorder 3050
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Called apply_rules
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_concept_id
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_end_datetime
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_source_concept_id
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_source_value
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped condition_start_datetime
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Mapped person_id
2022-06-17 15:05:17 - ConditionOccurrence - WARNING - Requiring non-null values in condition_concept_id removed 54362 rows, leaving 2483 rows.
2022-06-17 15:05:17 - ConditionOccurrence - INFO - Automatically formatting data columns.
2022-06-17 15:05:18 - ConditionOccurrence - INFO - created df (0x10c04e190)[Hypertensive_disorder_3050]
2022-06-17 15:05:18 - CommonDataModel - INFO - finished Hypertensive disorder 3050 (0x10c04e190) ... 12/12 completed, 2483 rows
2022-06-17 15:05:18 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:18 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:18 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:18 - CommonDataModel - ERROR - 16/2483 were good, 2467 studies are removed.
2022-06-17 15:05:18 - CommonDataModel - ERROR - Removed 2 row(s) due to duplicates found when merging condition_occurrence
2022-06-17 15:05:18 - CommonDataModel - WARNING - Example duplicates...
2022-06-17 15:05:18 - CommonDataModel - WARNING - person_id condition_concept_id condition_start_date \
condition_occurrence_id
7813 44.0 4223659 2020-09-13
7815 44.0 4223659 2020-09-13
3628 44.0 437663 2020-09-13
3630 44.0 437663 2020-09-13
condition_start_datetime condition_end_date \
condition_occurrence_id
7813 2020-09-13 00:00:00.000000 2020-09-13
7815 2020-09-13 00:00:00.000000 2020-09-13
3628 2020-09-13 00:00:00.000000 2020-09-13
3630 2020-09-13 00:00:00.000000 2020-09-13
condition_end_datetime condition_source_value \
condition_occurrence_id
7813 2020-09-13 00:00:00.000000 Yes
7815 2020-09-13 00:00:00.000000 Yes
3628 2020-09-13 00:00:00.000000 Yes
3630 2020-09-13 00:00:00.000000 Yes
condition_source_concept_id
condition_occurrence_id
7813 4223659
7815 4223659
3628 437663
3630 437663
2022-06-17 15:05:18 - CommonDataModel - INFO - saving dataframe (0x10c494970) to <carrot.io.plugins.sql.SqlDataCollection object at 0x10d429760>
2022-06-17 15:05:18 - SqlDataCollection - INFO - updating condition_occurrence in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:05:18 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:05:18 - CommonDataModel - INFO - finalised condition_occurrence on iteration 0 producing 1105 rows from 12 tables
2022-06-17 15:05:18 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:05:18 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'GP_Records'
2022-06-17 15:05:18 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:18 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Symptoms'
2022-06-17 15:05:18 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:18 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Hospital_Visit'
2022-06-17 15:05:18 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:18 - SqlDataCollection - INFO - All input files for this object have now been used.
2022-06-17 15:05:18 - SqlDataCollection - INFO - Registering Demographics [<carrot.io.common.DataBrick object at 0x1242a9640>]
2022-06-17 15:05:18 - SqlDataCollection - INFO - Registering GP_Records [<carrot.io.common.DataBrick object at 0x10d06dca0>]
2022-06-17 15:05:19 - SqlDataCollection - INFO - Registering Vaccinations [<carrot.io.common.DataBrick object at 0x10dc73fd0>]
2022-06-17 15:05:19 - SqlDataCollection - INFO - Registering Serology [<carrot.io.common.DataBrick object at 0x1253ce3a0>]
2022-06-17 15:05:19 - SqlDataCollection - INFO - Registering Symptoms [<carrot.io.common.DataBrick object at 0x10c4943d0>]
2022-06-17 15:05:19 - SqlDataCollection - INFO - Registering Hospital_Visit [<carrot.io.common.DataBrick object at 0x10e075250>]
2022-06-17 15:05:19 - SqlDataCollection - INFO - Registering Blood_Test [<carrot.io.common.DataBrick object at 0x10c469760>]
2022-06-17 15:05:19 - CommonDataModel - INFO - for drug_exposure: found 5 objects
2022-06-17 15:05:19 - CommonDataModel - INFO - working on drug_exposure
2022-06-17 15:05:19 - CommonDataModel - INFO - starting on COVID-19 vaccine 3034
2022-06-17 15:05:19 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:05:19 - SqlDataCollection - INFO - Retrieving initial dataframe for 'Vaccinations' for the first time
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:05:20 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:05:20 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 53291 rows, leaving 27645 rows.
2022-06-17 15:05:20 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 27643 rows.
2022-06-17 15:05:20 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:05:21 - DrugExposure - INFO - created df (0x10e028e20)[COVID_19_vaccine_3034]
2022-06-17 15:05:23 - CommonDataModel - INFO - finished COVID-19 vaccine 3034 (0x10e028e20) ... 1/5 completed, 27643 rows
2022-06-17 15:05:23 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:23 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:23 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:23 - CommonDataModel - ERROR - 155/27643 were good, 27488 studies are removed.
2022-06-17 15:05:23 - CommonDataModel - INFO - starting on COVID-19 vaccine 3035
2022-06-17 15:05:23 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:05:23 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:05:23 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54293 rows, leaving 26643 rows.
2022-06-17 15:05:23 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26641 rows.
2022-06-17 15:05:23 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:05:25 - DrugExposure - INFO - created df (0x10c597a60)[COVID_19_vaccine_3035]
2022-06-17 15:05:25 - CommonDataModel - INFO - finished COVID-19 vaccine 3035 (0x10c597a60) ... 2/5 completed, 26641 rows
2022-06-17 15:05:25 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:25 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:25 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:25 - CommonDataModel - ERROR - 144/26641 were good, 26497 studies are removed.
2022-06-17 15:05:25 - CommonDataModel - INFO - starting on COVID-19 vaccine 3036
2022-06-17 15:05:25 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:05:25 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:05:25 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54288 rows, leaving 26648 rows.
2022-06-17 15:05:25 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26646 rows.
2022-06-17 15:05:25 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:05:26 - DrugExposure - INFO - created df (0x1246d0760)[COVID_19_vaccine_3036]
2022-06-17 15:05:27 - CommonDataModel - INFO - finished COVID-19 vaccine 3036 (0x1246d0760) ... 3/5 completed, 26646 rows
2022-06-17 15:05:27 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:27 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:27 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:27 - CommonDataModel - ERROR - 150/26646 were good, 26496 studies are removed.
2022-06-17 15:05:27 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040
2022-06-17 15:05:27 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:05:27 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:05:27 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 53291 rows, leaving 27645 rows.
2022-06-17 15:05:27 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 27643 rows.
2022-06-17 15:05:27 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:05:28 - DrugExposure - INFO - created df (0x10c5971c0)[SARS_CoV_2_COVID_19_vaccine_mRNA_1273_0_2_MG_ML_Injectable_Suspension_3040]
2022-06-17 15:05:28 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML Injectable Suspension 3040 (0x10c5971c0) ... 4/5 completed, 27643 rows
2022-06-17 15:05:28 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:28 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:28 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:28 - CommonDataModel - ERROR - 155/27643 were good, 27488 studies are removed.
2022-06-17 15:05:28 - CommonDataModel - INFO - starting on SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041
2022-06-17 15:05:28 - DrugExposure - INFO - Called apply_rules
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped drug_concept_id
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped drug_exposure_end_datetime
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped drug_exposure_start_datetime
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped drug_source_concept_id
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped drug_source_value
2022-06-17 15:05:28 - DrugExposure - INFO - Mapped person_id
2022-06-17 15:05:29 - DrugExposure - WARNING - Requiring non-null values in drug_concept_id removed 54288 rows, leaving 26648 rows.
2022-06-17 15:05:29 - DrugExposure - WARNING - Requiring non-null values in drug_exposure_start_datetime removed 2 rows, leaving 26646 rows.
2022-06-17 15:05:29 - DrugExposure - INFO - Automatically formatting data columns.
2022-06-17 15:05:29 - DrugExposure - INFO - created df (0x123e896d0)[SARS_CoV_2_COVID_19_vaccine_mRNA_BNT162b2_0_1_MG_ML_Injectable_Suspension_3041]
2022-06-17 15:05:30 - CommonDataModel - INFO - finished SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML Injectable Suspension 3041 (0x123e896d0) ... 5/5 completed, 26646 rows
2022-06-17 15:05:30 - CommonDataModel - ERROR - There are person_ids in this table that are not in the output person table!
2022-06-17 15:05:30 - CommonDataModel - ERROR - Either they are not in the original data, or while creating the person table,
2022-06-17 15:05:30 - CommonDataModel - ERROR - studies have been removed due to lack of required fields, such as birthdate.
2022-06-17 15:05:30 - CommonDataModel - ERROR - 150/26646 were good, 26496 studies are removed.
2022-06-17 15:05:30 - CommonDataModel - INFO - saving dataframe (0x122e8d2b0) to <carrot.io.plugins.sql.SqlDataCollection object at 0x10d429760>
2022-06-17 15:05:30 - SqlDataCollection - INFO - updating drug_exposure in Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:05:31 - SqlDataCollection - INFO - finished save to psql
2022-06-17 15:05:31 - CommonDataModel - INFO - finalised drug_exposure on iteration 0 producing 754 rows from 5 tables
2022-06-17 15:05:31 - SqlDataCollection - INFO - Getting next chunk of data
2022-06-17 15:05:31 - SqlDataCollection - INFO - Getting the next chunk of size 'None' for 'Vaccinations'
Exception during reset or similar
Traceback (most recent call last):
File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 739, in _finalize_fairy
fairy._reset(pool)
File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 988, in _reset
pool._dialect.do_rollback(self)
File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 682, in do_rollback
dbapi_connection.rollback()
psycopg2.errors.AdminShutdown: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-06-17 15:05:31 - SqlDataCollection - INFO - --> Got 0 rows
2022-06-17 15:05:31 - SqlDataCollection - INFO - All input files for this object have now been used.
Load CDM from SQL¶
inputs_sql = carrot.tools.create_sql_store(connection_string="postgresql://localhost:5432/ExampleCDMDataSet")
2022-06-17 15:05:31 - SqlDataCollection - INFO - DataCollection Object Created
2022-06-17 15:05:31 - SqlDataCollection - INFO - Engine(postgresql://localhost:5432/ExampleCDMDataSet)
2022-06-17 15:05:31 - SqlDataCollection - INFO - Registering person_ids [<carrot.io.common.DataBrick object at 0x10c597af0>]
2022-06-17 15:05:31 - SqlDataCollection - INFO - Registering person [<carrot.io.common.DataBrick object at 0x10c9c2f70>]
2022-06-17 15:05:31 - SqlDataCollection - INFO - Registering observation [<carrot.io.common.DataBrick object at 0x10c49e9d0>]
2022-06-17 15:05:32 - SqlDataCollection - INFO - Registering condition_occurrence [<carrot.io.common.DataBrick object at 0x10c7ad070>]
2022-06-17 15:05:32 - SqlDataCollection - INFO - Registering drug_exposure [<carrot.io.common.DataBrick object at 0x1256f8d30>]
cdm = carrot.cdm.CommonDataModel.load(inputs=inputs_sql,
do_mask_person_id=False,
format_level=0)
2022-06-17 15:05:32 - CommonDataModel - INFO - CommonDataModel (5.3.1) created with co-connect-tools version 0.0.0
2022-06-17 15:05:32 - CommonDataModel - INFO - Turning on automatic cdm column filling
2022-06-17 15:05:32 - CommonDataModel - WARNING - Not loading person_ids, this is not a valid CDM Table
2022-06-17 15:05:32 - SqlDataCollection - INFO - Retrieving initial dataframe for 'person' for the first time
2022-06-17 15:05:32 - SqlDataCollection - INFO - Retrieving initial dataframe for 'observation' for the first time
2022-06-17 15:05:32 - SqlDataCollection - INFO - Retrieving initial dataframe for 'condition_occurrence' for the first time
2022-06-17 15:05:32 - SqlDataCollection - INFO - Retrieving initial dataframe for 'drug_exposure' for the first time
cdm['person'].dropna(axis=1)
| gender_concept_id | year_of_birth | month_of_birth | day_of_birth | birth_datetime | gender_source_value | gender_source_concept_id | |
|---|---|---|---|---|---|---|---|
| person_id | |||||||
| 1 | 8507 | 1958 | 7 | 17 | 1958-07-17 00:00:00.000000 | Male | 8507 |
| 2 | 8507 | 1983 | 7 | 11 | 1983-07-11 00:00:00.000000 | Male | 8507 |
| 3 | 8507 | 1965 | 7 | 15 | 1965-07-15 00:00:00.000000 | Male | 8507 |
| 4 | 8507 | 1925 | 7 | 25 | 1925-07-25 00:00:00.000000 | Male | 8507 |
| 5 | 8507 | 1971 | 7 | 14 | 1971-07-14 00:00:00.000000 | Male | 8507 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 267 | 8532 | 2019 | 7 | 2 | 2019-07-02 00:00:00.000000 | Female | 8532 |
| 268 | 8532 | 1883 | 8 | 4 | 1883-08-04 00:00:00.000000 | Female | 8532 |
| 269 | 8532 | 1892 | 8 | 1 | 1892-08-01 00:00:00.000000 | Female | 8532 |
| 270 | 8532 | 1899 | 7 | 31 | 1899-07-31 00:00:00.000000 | Female | 8532 |
| 271 | 8532 | 2018 | 7 | 2 | 2018-07-02 00:00:00.000000 | Female | 8532 |
271 rows × 7 columns