Module data_request_api.content.mapping_table

Mapping Table

The mapping_table dictionary defines how to map the three-base structure to the one-base structure. Each entry in the dictionary represents a table in the one-base structure and includes the information how to obtain it from the three-base structure. Not all source tables are available in all content versions of the data request. For example, "ESM-BCV 1.3" has been replaced with "ESM-BCV 1.4" in newer content versions.

Explanation of the dictionary keys:

Base ("source_base"): The base containing the table to be selected.

Table ("source_table"): The table to be selected from the "source_base".

Internal Mapping of record attributes ("internal_mapping"): Record attributes may point to records of other tables. However, there is no cross-linkage between the three bases, so these links need to be mapped as well. "internal_mapping" is a dictionary with the key corresponding to the record attributes to be mapped and the values containing the actual mapping information.

The mapping information is again a dictionary with the following keys:
- base_copy_of_table:
    If a copy of table corresponding to the record attribute exists in the current base,
    provide the name; otherwise, set to False.
- base:
    The base containing the original table the record attribute points to.
- table:
    The original table the record attribute points to.
- operation:
    The operation to perform on the attribute value (either "split" or "", if it is
    already provided as list or a string without comma separated values).
- map_by_key:
    A list of keys to map by.
- entry_type:
    The type of entry (either "record_id" or "name").

(Internal) Filters of record attributes ("internal_filters"): Not all records of the raw export shall be included since they may be labeled as junk or not be approved by the community. The filters are applied on all records and also internally on links to other records. "internal_filters" is a dictionary with the key corresponding to the record attributes used for filtering and the value another dictionary with the following possible keys: - operator: Can be one of "nonempty", "in", "not in" - values: A list of values, not necessary for "nonempty" operator.

Internal renaming of keys to achieve consistency across content versions ("internal_consistency"): "internal_consistency" is a dictionary with the key corresponding to the record attributes to be renamed and the value containing the new name. This is required as some attributes have been renamed in newer content versions or are renamed when setting up releases with airtable.

Attributes to remove ("drop_keys"): List of record attributes that are not needed in the one-base (=release) structure.

Data type of record attributes ("field_dtypes"): Dictionary of field names and their data type. This is required as some attributes differ between export types, such as integer fields in one export type being stored as strings in the other.

Example Configuration

Suppose we want to map the "CMIP7 Variable Groups" key in the "Variables" table of the "Data Request Variables (Public)" base to a list of record IDs of "Variable Group" records in the "Data Request Opportunities (Public)" base.

We would define the mapping_table as follows: mapping_table = { "Variables": { "base": "Data Request Variables (Public)", "source_table": "Variables", "internal_mapping": { "CMIP7 Variable Groups": { "base_copy_of_table": False, "base": "Data Request Opportunities (Public)", "table": "Variable Group", "operation": "split", "map_by_key": ["Name"], "entry_type": "name", }, }, "internal_filters": { "Status": {"operator": "not in", "values": ["Junk"]}, }, "internal_consistency": {"OldFieldName": "NewFieldName"}, "drop_keys": ["Field1", "Field2"], "field_dtypes": { "Field1": "str", "Field2": "int", "Field3": "float", "Field4": "listofstr", "Field5": "listofint", "Field6": "listoffloat", }, }, }