process_config

data.process_config

srproj_split module-attribute

srproj_split = {'training': 'Training', 'validation': 'Validation'}

process_data_config

process_data_config(cfg_data: dict) -> Tuple[dict, dict]

split training/validation configs from entier data config dict. you can edit cfg by split to override some configs in cfg_data entire config example) data: code: A.srproj validation: <- code: B.srproj <- this will override base config {"code": "A.srproj"}

Parameters:

• cfg_data (dict) –

  entire config

Returns:

• Tuple[dict, dict] –

  Tuple[dict, dict]: training and validation configs

Source code in SaigeToolkit/data/process_config.py

def process_data_config(cfg_data: dict) -> Tuple[dict, dict]:
    """split training/validation configs from entier data config dict.
    you can edit cfg by split to override some configs in cfg_data
    entire config example)
        data:
            code: A.srproj
            validation: <-
                code: B.srproj <- this will override base config {"code": "A.srproj"}

    Args:
        cfg_data (dict): entire config

    Returns:
        Tuple[dict, dict]: training and validation configs
    """
    cfg_data_copy = cfg_data.copy()
    cfg_training = cfg_data_copy.pop("training", {})
    cfg_validation = cfg_data_copy.pop("validation", {})

    # update config for training
    if cfg_training != "NO_TRAINING":
        assert isinstance(
            cfg_training, dict
        ), f"config for training data should be Dict, not {type(cfg_training)}"

        cfg_training = override_dict(deepcopy(cfg_data_copy), cfg_training)  # TODO:
        if "split" not in cfg_training["dataset"]:
            cfg_training["dataset"].update({"split": srproj_split["training"]})

    # update config for validation
    if cfg_validation != "NO_VALIDATION":
        assert isinstance(
            cfg_validation, dict
        ), f"config for validation data should be Dict, not {type(cfg_validation)}"

        cfg_validation = override_dict(deepcopy(cfg_data_copy), cfg_validation)  # TODO:
        if "split" not in cfg_validation["dataset"]:
            cfg_validation["dataset"].update({"split": srproj_split["validation"]})

    return cfg_training, cfg_validation