Rules

Introduction

The Rules Azure Functions refers to a series of Durable functions, that are reponsible for generating the Rules for a given Scan Report Table.

Function	Type	Purpose
RulesTrigger	HttpTrigger	Consumes the HTTP request from the web app, adds to the queue.
RulesQueue	Queue	Reads the message from the queue, passes to the Orchestrator.
RulesOrchestrator	Orchestrator	Orchestrates the activity, generating Concepts, and parallelising Rules Generation.
RulesConceptsActivity	Activity	Generates the Scan Report Concepts.
RulesGenerationActivity	Activity	Generates the Scan Report Rules.

Converting Concept Codes to ConceptIDs

Introduction

Some scan reports (most likely from the larger data providers) will be supplied with concept codes as values. In effect the mapping has already been done for us-- we just need to make sure we return valid and standard OMOP conceptIDs. In instances where concept codes are provided, they will be coded within a given vocabulary e.g. SNOMED or RxNorm. The ProcessQueue Azure function has therefore been updated with the ability to look up concept codes and convert them to standard and valid OMOP codes. This removes the requirement from the data team of manually looking up concept codes and finding the relevant conceptID.

The Data Team have specified a list of vocabularies that are most likely to be used.

Operation

The concept code to conceptID lookup occurs as each scan report value is processed in __init__.py. Immediately after a value description is looked up, the code checks to see whether the dictionary contains any information on a vocabulary coding system. A dictionary may look something like:

csv_file_name	field_name	code	value
questionnaire	sex	0	male
survey	question1	SNOMED

In this example, all of the values within the field question1 will be encoded as SNOMED concept codes. For example, the question might be: "Main symptom" and the answer of 'cough' would be encoded as "49727002" under the SNOMED vocabulary.

For a given value, we determine whether it falls within a field defined as a vocabulary field with:

code = next(
    (
        row["code"]
        for row in data_dictionary
        if str(row["field_name"]) == str(name)
    ),
    None,
)

If the value of code is within our list of vocabularies, we use omop_helpers.get_concept_from_concept_code() to look up the standard and valid OMOP conceptID with the following:

# If 'code' is in our vocab list, try and convert the ScanReportValue (concept code) to conceptID
# If there's a faulty concept code for the vocab, fail gracefully and set concept_id to default (-1)
if code in vocabs:
    try:
        concept_id = omop_helpers.get_concept_from_concept_code(
            concept_code=value,
            vocabulary_id=code,
            no_source_concept=True,
        )
        concept_id = concept_id["concept_id"]
    except:
        concept_id = -1
else:
    concept_id = -1

Where value is the scan report value (i.e. the concept code) in that loop's iteration and code is the name of the vocabulary from the data dictionary's 'code' column (see here for a standard data dictionary template.) If the supplied scan report value (concept code) is incorrect for any reason, we will simply default to setting concept_id to the default of -1.

Here is an example if you wanted to manually look up a concept code:

concept_id = omop_helpers.get_concept_from_concept_code(
                                concept_code=49727002,
                                vocabulary_id="SNOMED,
                                no_source_concept=True,
                            )

The argument no_source_concept tells the function to not return the non-standard conceptID. It will only return the standard and valid conceptID when this is set to True.

With a conceptID in hand, we construct a dictionary called scan_report_value_entry with all the necessary details and append it to a list for a single POST to the database:

data = []

scan_report_value_entry = {
    "created_at": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
    "updated_at": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
    "value": value,
    "frequency": int(frequency),
    "conceptID": concept_id,
    "value_description": val_desc,
    "scan_report_field": names_x_ids[name],
}

data.append(scan_report_value_entry)

Updating ScanReportConcept

By this point, we have not associated conceptIDs with scan report values using the ScanReportConcept model. In effect, any conceptID information from the scan report is--at present--held in the model ScanReportValue. We therefore need to 'move' the conceptID data from the conceptID field and create a record for it in ScanReportConcept.

Important

The reason we have to do this is--prior to POSTing scan report values to ScanReportValue--we have no primary keys for any scan report values. Without the PKs, we can't establish a relationship between ScanReportValue and ScanReportConcept. After the POST request we have the PKs required to associate a ScanReportConcept to a ScanReportValue.

After POSTing the scan report values to the database, we must then return all ScanReportValues where the conceptID != -1. In other words, we want to return all records where we performed a concept code -> conceptID operation.

We created a custom ViewSet called ScanReportValuePKViewSet which takes a Scan Report ID and returns all scan report values where there is a conceptID in the conceptID field. We then iterate over these model objects and create legitimate concept entries in ScanReportConcept