Skip to content

learning

Module diagram

classDiagram
  class learning {
  }
  class device {
  }
  class meter {
  }
  class optimizer {
  }
  class scheduler {
  }
  class trainer {
  }
  scheduler --> optimizer

learning

Device manager 및 기본적인 metric 관리 클래스와 기본 Trainer 클래스를 제공합니다.

BaseTrainer 클래스는 Saige Vision2 Engine 학습 API의 베이스 인터페이스를 제공하며, 이를 상속받아 사용자 정의 Trainer를 구현할 수 있습니다. 구현 메서드들은 error_handler를 씌워 노출되어야 합니다.

device

"Device Manager, converting modules and data tensors to device

DeviceType module-attribute 

DeviceType = Union[device, str, int]

AttributeDataParallel

Bases: DataParallel

DataParallel class with attribute get method.

You can now use expressions like:

# Possible with origial DataParallel class.
self.network(x)
# Not possible with origial DataParallel class.
self.network.feature_extractor(x)

...where self.network is a AttributeDataParallel class instance.

Author

Jonghyuk Baek

__getattr__ 
__getattr__(name)
Source code in SaigeToolkit/learning/device.py 
31
32
33
34
35
def __getattr__(self, name):
    try:
        return super().__getattr__(name)
    except AttributeError:
        return getattr(self.module, name)

DeviceManager 

DeviceManager(device: DeviceType = 'cpu', multi_gpu: bool = False)

converting torch.nn.modules and torch.tensors to selected device.

Attributes:

  • device (device) –

    name of target device for learning.

  • multi_gpu (bool) –

    whether use multi-gpu learning.

initializing DeviceManager

Parameters:

  • device (Optional[Union[device, str, int]], default: 'cpu' ) –

    user defined device. Defaults to None.

  • multi_gpu (bool, default: False ) –

    set to class attribute. Defaults to False.

Source code in SaigeToolkit/learning/device.py 
46
47
48
49
50
51
52
53
54
def __init__(self, device: DeviceType = "cpu", multi_gpu: bool = False) -> None:
    """initializing DeviceManager

    Args:
        device (Optional[Union[torch.device, str, int]], optional): user defined device. Defaults to None.
        multi_gpu (bool, optional): set to class attribute. Defaults to False.
    """
    self.set_gpu(device)
    self.multi_gpu = multi_gpu
multi_gpu instance-attribute 
multi_gpu = multi_gpu
set_gpu 
set_gpu(device: DeviceType = 'cpu') -> None

If user gives device string, use the defined device.

Parameters:

  • device (Optional[Union[device, str, int]], default: 'cpu' ) –

    user defined device. Defaults to None.

Source code in SaigeToolkit/learning/device.py 
56
57
58
59
60
61
62
def set_gpu(self, device: DeviceType = "cpu") -> None:
    """If user gives device string, use the defined device.

    Args:
        device (Optional[Union[torch.device, str, int]], optional): user defined device. Defaults to None.
    """
    self.device = torch.device(device)
module_to_device 
module_to_device(module: object) -> List[Optional[Module]]

converting torch modules to device

Parameters:

  • module (object) –

    task specific modules # XXX: isinstance of BaseModule?

Returns:

  • List[Optional[Module]] –

    List[torch.nn.Module]: network module list which needs to be reset after iteration step.

Source code in SaigeToolkit/learning/device.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
def module_to_device(self, module: object) -> List[Optional[torch.nn.Module]]:
    """converting torch modules to device

    Args:
        module (object): task specific modules # XXX: isinstance of BaseModule?

    Returns:
        List[torch.nn.Module]: network module list which needs to be reset after iteration step.
    """
    reset_list = []

    # Setup Network
    for name in [attr for attr in list(dir(module)) if "network" in attr]:
        try:
            named_module = getattr(module, name).to(self.device)
            if hasattr(named_module, "reset"):
                reset_list.append(named_module)
            # Check if Using Multi-GPU
            if self.multi_gpu:
                named_module = AttributeDataParallel(named_module, output_device=self.device)
            setattr(module, name, named_module)
        except AttributeError:
            pass
        except Exception as e:
            print(f"uncaught error <{e}> occured: {name}")

    # Setup Sub-Module
    for name in [attr for attr in list(dir(module)) if "module" in attr]:
        sub_reset_list = self.module_to_device(getattr(module, name))
        reset_list = [*reset_list, *sub_reset_list]

    return reset_list
data_to_device 
data_to_device(*dummy: list, **data: dict) -> dict

convetring tensor-type data to device

Parameters:

  • data (dict, default: {} ) –

    data with tensors and also non-tensors.

Raises:

  • AssertionError –

    input data should be dict type.

Returns:

  • dict ( dict ) –

    data dict with tensors converted to device

Source code in SaigeToolkit/learning/device.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
def data_to_device(self, *dummy: list, **data: dict) -> dict:
    """convetring tensor-type data to device

    Args:
        data (dict): data with tensors and also non-tensors.

    Raises:
        AssertionError: input data should be dict type.

    Returns:
        dict: data dict with tensors converted to device
    """

    if dummy:
        raise AssertionError(f"Non dictionary input is not allowed: {type(dummy)}")

    for key, value in data.items():
        if isinstance(value, list):
            value = [v.to(self.device) if torch.is_tensor(v) else v for v in value]
        else:
            value = value.to(self.device) if torch.is_tensor(value) else value

        data.update({key: value})

    return data

get_module_device 

get_module_device(module: Module) -> device
Source code in SaigeToolkit/learning/device.py 
124
125
126
127
128
129
130
131
def get_module_device(module: torch.nn.Module) -> torch.device:
    try:
        return next(module.parameters()).device
    except StopIteration:
        try:
            return next(module.buffers()).device
        except StopIteration:
            return torch.device("cpu")

cuda_empty_cache 

cuda_empty_cache(device: DeviceType) -> None

CUDA device의 torch cache를 비웁니다 Multi-GPU 환경에서 0번이 아닌 device에 대해 empty_cache 호출 시 0번 device에 메모리 올리는 이슈를 해결

Source code in SaigeToolkit/learning/device.py 
134
135
136
137
138
139
140
def cuda_empty_cache(device: DeviceType) -> None:
    """CUDA device의 torch cache를 비웁니다
    Multi-GPU 환경에서 0번이 아닌 device에 대해 empty_cache 호출 시 0번 device에 메모리 올리는 이슈를 해결
    """
    if torch.device(device).type == "cuda":
        with torch.cuda.device(device):
            torch.cuda.empty_cache()

self_device_context 

self_device_context(method)

method에 self.device context 적용

Source code in SaigeToolkit/learning/device.py 
143
144
145
146
147
148
149
150
151
152
153
154
def self_device_context(method):
    """method에 self.device context 적용"""

    @wraps(method)
    def decorator(self, *args, **kwargs):
        if torch.device(self.device).type == "cuda":
            with torch.cuda.device(self.device):
                return method(self, *args, **kwargs)
        else:
            return method(self, *args, **kwargs)

    return decorator

apply_self_device_context 

apply_self_device_context()

class의 모든 method에 self.device context 적용 (staticmethod, classmethod 및 기본으로 정의된 method 제외)

Note: 모든 메소드에 적용하기 때문에 퍼포먼스 이슈가 발생할 수 있습니다

Usage 
@apply_self_device_context()
class InferenceHandler:
    ...
Source code in SaigeToolkit/learning/device.py 
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
def apply_self_device_context():
    """class의 모든 method에 self.device context 적용
    (staticmethod, classmethod 및 기본으로 정의된 method 제외)

    Note: 모든 메소드에 적용하기 때문에 퍼포먼스 이슈가 발생할 수 있습니다

    Usage:
        ```python

        @apply_self_device_context()
        class InferenceHandler:
            ...

        ```
    """

    def decorator_to_class(cls):
        for attr_name, attr_value in vars(cls).items():
            to_apply = (
                callable(attr_value)
                and not isinstance(attr_value, (staticmethod, classmethod))
                and hasattr(cls, attr_name)
                and getattr(cls, attr_name).__class__ == attr_value.__class__
            )
            if to_apply:
                setattr(cls, attr_name, self_device_context(attr_value))
        return cls

    return decorator_to_class

meter

Meter, updates and stores loss values and report average value.

Meter 

Meter()

Computes and stores the average and current losses by keys

Attributes:

  • meters (dict) –

    stored loss information by keys

Source code in SaigeToolkit/learning/meter.py 
13
14
def __init__(self) -> None:
    self.meters: Dict[str, AverageMeter] = {}
meters instance-attribute 
meters: Dict[str, AverageMeter] = {}
reset 
reset() -> None
Source code in SaigeToolkit/learning/meter.py 
16
17
18
def reset(self) -> None:
    for _, mtr in self.meters.items():
        mtr.reset()
update 
update(loss: Union[Dict[str, float], List[float], float]) -> None

update result loss from network

Parameters:

  • loss (Union[Dict[str, float], List[float], float]) –

    loss to be stored. can be three types of data.

Source code in SaigeToolkit/learning/meter.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
def update(self, loss: Union[Dict[str, float], List[float], float]) -> None:
    """update result loss from network

    Args:
        loss (Union[Dict[str, float], List[float], float]):
            loss to be stored. can be three types of data.
    """

    if isinstance(loss, dict):
        keys, loss = list(loss.keys()), list(loss.values())
    elif isinstance(loss, list):
        keys = ["loss" + str(i) for i in range(len(loss))]
    else:
        keys = ["loss"]
        loss = [loss]

    for key, l in zip(keys, loss):
        if key not in self.meters.keys():
            self.meters[key] = AverageMeter()
        self.meters[key].update(l)

AverageMeter 

AverageMeter()

Computes and stores the average and current value Attributes: val (float): last input loss value avg (float): average loss value sum (float): summed loss value min (float): min loss value max (float): max loss value count (int): total number of loss inputs

Source code in SaigeToolkit/learning/meter.py 
53
54
def __init__(self) -> None:
    self.reset()
reset 
reset() -> None
Source code in SaigeToolkit/learning/meter.py 
56
57
58
59
60
61
62
def reset(self) -> None:
    self.val = 0.0
    self.avg = 0.0
    self.sum = 0.0
    self.min = float("inf")
    self.max = -float("inf")
    self.count = 0
update 
update(val: float, n: int = 1) -> None

update single loss value

Parameters:

  • val (float) –

    current loss value

  • n (int, default: 1 ) –

    current number of loss inputs. Defaults to 1.

Source code in SaigeToolkit/learning/meter.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
def update(self, val: float, n: int = 1) -> None:
    """update single loss value

    Args:
        val (float): current loss value
        n (int, optional): current number of loss inputs. Defaults to 1.
    """
    self.val = val
    self.sum += val * n
    self.count += n
    self.avg = self.sum / self.count
    self.min = min(val, self.min)
    self.max = max(val, self.max)

get_meter 

get_meter() -> Tuple[Meter, Meter]

get loss/score average meter for training and validation

Returns:

  • Tuple[Meter, Meter] –

    Tuple[Meter, Meter]: training/validation meters

Source code in SaigeToolkit/learning/meter.py 
79
80
81
82
83
84
85
86
87
88
89
def get_meter() -> Tuple[Meter, Meter]:
    """get loss/score average meter for training and validation

    Returns:
        Tuple[Meter, Meter]: training/validation meters
    """

    train_meter = Meter()
    val_meter = Meter()

    return train_meter, val_meter

optimizer

building torch optimizers with simple configs

logger module-attribute 

logger = getLogger('SaigeResearch')

_types module-attribute 

_types = {__name__: _Zy6ifor _type in _types}

build_optimizer 

build_optimizer(model_params: Iterator[Parameter], _target_: str = 'SGD', **params) -> Optimizer

build optimizer object

Parameters:

  • model_params (Iterator[Parameter]) –

    parameters of model to be trained

  • _target_ (str, default: 'SGD' ) –

    optimizer name. Defaults to "SGD".

Raises:

  • NotImplementedError –

    description

Returns:

  • Optimizer ( Optimizer ) –

    optimizer object

Source code in SaigeToolkit/learning/optimizer.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def build_optimizer(model_params: Iterator[Parameter], _target_: str = "SGD", **params) -> Optimizer:
    """build optimizer object

    Args:
        model_params (Iterator[Parameter]): parameters of model to be trained
        _target_ (str, optional): optimizer name. Defaults to "SGD".

    Raises:
        NotImplementedError: _description_

    Returns:
        Optimizer: optimizer object
    """
    if _target_ not in _types:
        raise NotImplementedError(f"OPTIMIZER {_target_} not implemented")
    logger.info(f"[{'OPTIMIZER'.center(9)}] {_target_} [params] {params}")
    return _types[_target_](model_params, **params)

scheduler

torch learning rate schdulers and Few custom schedulers

logger module-attribute 

logger = getLogger('SaigeResearch')

_types module-attribute 

_types = {__name__: _t86Xfor _type in _types}

model module-attribute 

model = FooModule()

optimizer module-attribute 

optimizer = build_optimizer(parameters(), **{'_target_': 'SGD', 'lr': 0.0, 'momentum': 0.9, 'weight_decay': 0.0001})

scheduler module-attribute 

scheduler = build_scheduler(optimizer, **{'_target_': 'CosineAnnealingWarmUpRestarts', 'T_0': 50000, 'T_mult': 1, 'eta_max': 0.005, 'T_up': 500, 'gamma': 0.5})

FixedLR 

FixedLR(optimizer: Optimizer, last_epoch: int = -1)

Bases: _LRScheduler

constant learning rate

initializing FixedLR

Parameters:

  • optimizer (Optimizer) –

    optimizer

  • last_epoch (int, default: -1 ) –

    index of last epoch. Defaults to -1.

Source code in SaigeToolkit/learning/scheduler.py 
51
52
53
54
55
56
57
58
def __init__(self, optimizer: Optimizer, last_epoch: int = -1) -> None:
    """initializing FixedLR

    Args:
        optimizer (Optimizer): optimizer
        last_epoch (int, optional): index of last epoch. Defaults to -1.
    """
    super(FixedLR, self).__init__(optimizer, last_epoch)
get_lr 
get_lr()
Source code in SaigeToolkit/learning/scheduler.py 
60
61
def get_lr(self):
    return [base_lr for base_lr in self.base_lrs]

PolynomialLR 

PolynomialLR(optimizer: Optimizer, max_iter: int, decay_iter: int = 1, gamma: float = 0.9, last_epoch: int = -1)

Bases: _LRScheduler

polynomial learning rate scheduler

Attributes:

  • decay_iter (int) –

    decaying iteration number

  • max_iter (int) –

    max iteration number for decaying

  • gamma (float) –

    learning rate decay rate (exponents)

initializing PolynomialLR

Parameters:

  • optimizer (Optimizer) –

    optimizer

  • max_iter (int) –

    max iteration number for decaying

  • decay_iter (int, default: 1 ) –

    decaying iteration number. Defaults to 1.

  • gamma (float, default: 0.9 ) –

    learning rate decay rate (exponents). Defaults to 0.9.

  • last_epoch (int, default: -1 ) –

    index of last epoch. Defaults to -1.

Source code in SaigeToolkit/learning/scheduler.py 
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
def __init__(
    self,
    optimizer: Optimizer,
    max_iter: int,
    decay_iter: int = 1,
    gamma: float = 0.9,
    last_epoch: int = -1,
) -> None:
    """initializing PolynomialLR

    Args:
        optimizer (Optimizer): optimizer
        max_iter (int): max iteration number for decaying
        decay_iter (int, optional): decaying iteration number. Defaults to 1.
        gamma (float, optional): learning rate decay rate (exponents). Defaults to 0.9.
        last_epoch (int, optional): index of last epoch. Defaults to -1.
    """
    self.decay_iter = decay_iter
    self.max_iter = max_iter
    self.gamma = gamma
    super(PolynomialLR, self).__init__(optimizer, last_epoch)
decay_iter instance-attribute 
decay_iter = decay_iter
max_iter instance-attribute 
max_iter = max_iter
gamma instance-attribute 
gamma = gamma
get_lr 
get_lr()
Source code in SaigeToolkit/learning/scheduler.py 
 95
 96
 97
 98
 99
100
def get_lr(self):
    if self.last_epoch % self.decay_iter or self.last_epoch % self.max_iter:
        return [base_lr for base_lr in self.base_lrs]
    else:
        factor = (1 - self.last_epoch / float(self.max_iter)) ** self.gamma
        return [base_lr * factor for base_lr in self.base_lrs]

WarmUpLR 

WarmUpLR(optimizer: Optimizer, scheduler: dict, mode: str = 'linear', warmup_iters: int = 100, gamma: float = 0.2)

Bases: _LRScheduler

Wrapper for learning rate scheduler with warm-up stage

Attributes:

  • mode (str) –

    base schduler mode after warm-up stage

  • scheduler (_LRScheduler) –

    schduler for cold_lrs (warm-up stage)

  • warmup_iters (int) –

    number of iterations for warm-up stage

  • gamma (float) –

    learning rate decay rate

Parameters:

  • scheduler (dict) –

    schduler dict for cold_lrs (warm-up stage)

  • mode (str, default: 'linear' ) –

    base schduler mode after warm-up stage. Defaults to "linear".

  • warmup_iters (int, default: 100 ) –

    number of iterations for warm-up stage. Defaults to 100.

  • gamma (float, default: 0.2 ) –

    learning rate decay ratev. Defaults to 0.2.

Source code in SaigeToolkit/learning/scheduler.py 
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def __init__(
    self,
    optimizer: Optimizer,
    scheduler: dict,
    mode: str = "linear",
    warmup_iters: int = 100,
    gamma: float = 0.2,
) -> None:
    """
    Args:
        scheduler (dict): schduler dict for cold_lrs (warm-up stage)
        mode (str, optional): base schduler mode after warm-up stage. Defaults to "linear".
        warmup_iters (int, optional): number of iterations for warm-up stage. Defaults to 100.
        gamma (float, optional): learning rate decay ratev. Defaults to 0.2.
    """
    self._warmup_strategy = mode
    if self._warmup_strategy == "cos":
        self._warmup_func = self._warmup_cos
    elif self._warmup_strategy == "linear":
        self._warmup_func = self._warmup_linear
    elif self._warmup_strategy == "const":
        self._warmup_func = self._warmup_const
    else:
        raise NotImplementedError(f"Warmup type {self._warmup_strategy} not implemented.")

    self.gamma = gamma

    self._scheduler = build_scheduler(optimizer=optimizer, **scheduler)
    self._init_lr = self._scheduler.optimizer.param_groups[0]["lr"] * self.gamma
    self._warmup_iters = warmup_iters
    self._step_count = 0
    self._format_param()
_warmup_strategy instance-attribute 
_warmup_strategy = mode
_warmup_func instance-attribute 
_warmup_func = _warmup_cos
gamma instance-attribute 
gamma = gamma
_scheduler instance-attribute 
_scheduler = build_scheduler(optimizer=optimizer, **scheduler)
_init_lr instance-attribute 
_init_lr = param_groups[0]['lr'] * gamma
_warmup_iters instance-attribute 
_warmup_iters = warmup_iters
_step_count instance-attribute 
_step_count = 0
_format_param 
_format_param()
Source code in SaigeToolkit/learning/scheduler.py 
146
147
148
149
150
151
def _format_param(self):
    # learning rate of each param group will increase
    # from the min_lr to initial_lr
    for group in self._scheduler.optimizer.param_groups:
        group["warmup_max_lr"] = group["lr"]
        group["warmup_initial_lr"] = min(self._init_lr, group["lr"])
__getattr__ 
__getattr__(name)
Source code in SaigeToolkit/learning/scheduler.py 
153
154
def __getattr__(self, name):
    return getattr(self._scheduler, name)
state_dict 
state_dict()

Returns the state of the scheduler as a :class:dict. It contains an entry for every variable in self.dict which is not the optimizer.

Source code in SaigeToolkit/learning/scheduler.py 
156
157
158
159
160
161
162
163
164
165
166
167
def state_dict(self):
    """Returns the state of the scheduler as a :class:`dict`.
    It contains an entry for every variable in self.__dict__ which
    is not the optimizer.
    """
    wrapper_state_dict = {
        key: value for key, value in self.__dict__.items() if key not in ["optimizer", "_scheduler"]
    }
    wrapped_state_dict = {
        key: value for key, value in self._scheduler.__dict__.items() if key != "optimizer"
    }
    return {"wrapped": wrapped_state_dict, "wrapper": wrapper_state_dict}
load_state_dict 
load_state_dict(state_dict: dict)

Loads the schedulers state.

Parameters:

  • state_dict (dict) –

    scheduler state. Should be an object returned from a call to :meth:state_dict.

Source code in SaigeToolkit/learning/scheduler.py 
169
170
171
172
173
174
175
176
177
def load_state_dict(self, state_dict: dict):
    """Loads the schedulers state.

    Args:
        state_dict (dict): scheduler state. Should be an object returned
            from a call to :meth:`state_dict`.
    """
    self.__dict__.update(state_dict["wrapper"])
    self._scheduler.__dict__.update(state_dict["wrapped"])
_warmup_cos 
_warmup_cos(start, end, pct)
Source code in SaigeToolkit/learning/scheduler.py 
179
180
181
def _warmup_cos(self, start, end, pct):
    cos_out = math.cos(math.pi * pct) + 1
    return end + (start - end) / 2.0 * cos_out
_warmup_const 
_warmup_const(start, end, pct)
Source code in SaigeToolkit/learning/scheduler.py 
183
184
def _warmup_const(self, start, end, pct):
    return start if pct < 0.9999 else end
_warmup_linear 
_warmup_linear(start, end, pct)
Source code in SaigeToolkit/learning/scheduler.py 
186
187
def _warmup_linear(self, start, end, pct):
    return (end - start) * pct + start
step 
step(*args)
Source code in SaigeToolkit/learning/scheduler.py 
189
190
191
192
193
194
195
196
197
198
def step(self, *args):
    if self._step_count <= self._warmup_iters:
        values = self.get_lr()
        for param_group, lr in zip(self._scheduler.optimizer.param_groups, values):
            param_group["lr"] = lr
        self._step_count += 1
    else:
        self._scheduler.step(*args)

    self._last_lr = [param_group["lr"] for param_group in self._scheduler.optimizer.param_groups]
get_lr 
get_lr()
Source code in SaigeToolkit/learning/scheduler.py 
200
201
202
203
204
205
206
207
208
209
210
211
212
213
def get_lr(self):
    lrs = []
    # warm up learning rate
    if self._step_count <= self._warmup_iters:
        for group in self._scheduler.optimizer.param_groups:
            computed_lr = self._warmup_func(
                group["warmup_initial_lr"],
                group["warmup_max_lr"],
                self._step_count / self._warmup_iters,
            )
            lrs.append(computed_lr)
    else:
        lrs = self._scheduler.get_lr()
    return lrs

CosineAnnealingWarmUpRestarts 

CosineAnnealingWarmUpRestarts(optimizer: Optimizer, T_0: int, T_mult: int = 1, eta_max: float = 0.1, T_up: int = 0, gamma: float = 1.0, last_epoch: int = -1)

Bases: _LRScheduler

Cosine Anneeling scheduler with Warmup restarts (SGDR).

Modified from https://github.com/pytorch/pytorch/blob/v1.1.0/torch/optim/lr_scheduler.py#L655, implementing initial warmup stage.

Usage: 

optimizer:
    _target_: sgd
    lr: 0.0  # optimizer lr should be zero or very small value!!!
    momentum: 0.9
    weight_decay: 0.0001
scheduler:
    _target_: CosineAnnealingWarmUpRestarts
    T_0: 50000
    T_mult: 1
    eta_max: 0.005
    T_up: 500
    gamma: 0.5

Parameters:

  • optimizer (Optimizer) –

    Pytorch optimizer instance.

  • T_0 (int) –

    Initial annealing cycle length.

  • T_mult (int, default: 1 ) –

    Cycle multiplication scale after the first cycle. Defaults to 1.

  • eta_max (float, default: 0.1 ) –

    Max learning rate. Defaults to 0.1.

  • T_up (int, default: 0 ) –

    Warmup step size. Defaults to 0.

  • gamma (float, default: 1.0 ) –

    eta_max multiplication scale after the first cycle. Defaults to 1..

  • last_epoch (int, default: -1 ) –

    Last epoch of the scheduler. Defaults to -1.

Raises:

  • ValueError –

    Expected positive integer T_0

  • ValueError –

    Expected integer T_mult >= 1

  • ValueError –

    Expected positive integer T_up

Source code in SaigeToolkit/learning/scheduler.py 
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
def __init__(
    self,
    optimizer: Optimizer,
    T_0: int,
    T_mult: int = 1,
    eta_max: float = 0.1,
    T_up: int = 0,
    gamma: float = 1.0,
    last_epoch: int = -1,
):
    """
    Args:
        optimizer (Optimizer): Pytorch optimizer instance.
        T_0 (int): Initial annealing cycle length.
        T_mult (int, optional): Cycle multiplication scale after the first cycle. Defaults to 1.
        eta_max (float, optional): Max learning rate. Defaults to 0.1.
        T_up (int, optional): Warmup step size. Defaults to 0.
        gamma (float, optional): eta_max multiplication scale after the first cycle. Defaults to 1..
        last_epoch (int, optional): Last epoch of the scheduler. Defaults to -1.

    Raises:
        ValueError: Expected positive integer T_0
        ValueError: Expected integer T_mult >= 1
        ValueError: Expected positive integer T_up
    """
    if T_0 <= 0 or not isinstance(T_0, int):
        raise ValueError(f"Expected positive integer T_0, but got {T_0}")
    if T_mult < 1 or not isinstance(T_mult, int):
        raise ValueError(f"Expected integer T_mult >= 1, but got {T_mult}")
    if T_up < 0 or not isinstance(T_up, int):
        raise ValueError(f"Expected positive integer T_up, but got {T_up}")
    self.T_0 = T_0
    self.T_mult = T_mult
    self.base_eta_max = eta_max
    self.eta_max = eta_max
    self.T_up = T_up
    self.T_i = T_0
    self.gamma = gamma
    self.cycle = 0
    self.T_cur = last_epoch
    super(CosineAnnealingWarmUpRestarts, self).__init__(optimizer, last_epoch)
T_0 instance-attribute 
T_0 = T_0
T_mult instance-attribute 
T_mult = T_mult
base_eta_max instance-attribute 
base_eta_max = eta_max
eta_max instance-attribute 
eta_max = eta_max
T_up instance-attribute 
T_up = T_up
T_i instance-attribute 
T_i = T_0
gamma instance-attribute 
gamma = gamma
cycle instance-attribute 
cycle = 0
T_cur instance-attribute 
T_cur = last_epoch
get_lr 
get_lr()
Source code in SaigeToolkit/learning/scheduler.py 
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
def get_lr(self):
    if self.T_cur == -1:
        return self.base_lrs
    elif self.T_cur < self.T_up:
        return [
            (self.eta_max - base_lr) * self.T_cur / self.T_up + base_lr for base_lr in self.base_lrs
        ]
    else:
        return [
            base_lr
            + (self.eta_max - base_lr)
            * (1 + math.cos(math.pi * (self.T_cur - self.T_up) / (self.T_i - self.T_up)))
            / 2
            for base_lr in self.base_lrs
        ]
step 
step(epoch=None)
Source code in SaigeToolkit/learning/scheduler.py 
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
def step(self, epoch=None):
    if epoch is None:
        epoch = self.last_epoch + 1
        self.T_cur = self.T_cur + 1
        if self.T_cur >= self.T_i:
            self.cycle += 1
            self.T_cur = self.T_cur - self.T_i
            self.T_i = (self.T_i - self.T_up) * self.T_mult + self.T_up
    else:
        if epoch >= self.T_0:
            if self.T_mult == 1:
                self.T_cur = epoch % self.T_0
                self.cycle = epoch // self.T_0
            else:
                n = int(math.log((epoch / self.T_0 * (self.T_mult - 1) + 1), self.T_mult))
                self.cycle = n
                self.T_cur = epoch - self.T_0 * (self.T_mult**n - 1) / (self.T_mult - 1)
                self.T_i = self.T_0 * self.T_mult ** (n)
        else:
            self.T_i = self.T_0
            self.T_cur = epoch

    self.eta_max = self.base_eta_max * (self.gamma**self.cycle)
    self.last_epoch = math.floor(epoch)

    for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):
        param_group["lr"] = lr

    self._last_lr = [param_group["lr"] for param_group in self.optimizer.param_groups]

ProportionalMultiStepLR 

ProportionalMultiStepLR(optimizer, iteration: int, milestones: Sequence[float], **kwargs)

Bases: MultiStepLR

전체 iteration에 대한 비율로 milestone을 설정하는 MultiStepLR 스케줄러

Source code in SaigeToolkit/learning/scheduler.py 
331
332
333
def __init__(self, optimizer, iteration: int, milestones: Sequence[float], **kwargs) -> None:
    milestones_ = [int(iteration * proportion) for proportion in sorted(milestones)]
    super().__init__(optimizer=optimizer, milestones=milestones_, **kwargs)

SaigeVision1_5080_LR 

SaigeVision1_5080_LR(optimizer, iteration: int, **kwargs)

Bases: ProportionalMultiStepLR

SaigeVision1 제품에 적용된 스케줄러. 전체 iteration의 50%, 80% 지점에서 learning rate을 0.5배씩 줄입니다.

Source code in SaigeToolkit/learning/scheduler.py 
339
340
def __init__(self, optimizer, iteration: int, **kwargs) -> None:
    super().__init__(optimizer, iteration=iteration, milestones=[0.5, 0.8], gamma=0.5, **kwargs)

FooModule 

FooModule()

Bases: Module

Source code in SaigeToolkit/learning/scheduler.py 
384
385
386
def __init__(self):
    super(FooModule, self).__init__()
    self.conv = torch.nn.Conv2d(3, 3, 3)
conv instance-attribute 
conv = Conv2d(3, 3, 3)
forward 
forward(x)
Source code in SaigeToolkit/learning/scheduler.py 
388
389
def forward(self, x):
    return self.conv(x)

build_scheduler 

build_scheduler(optimizer: Optimizer, _target_: str = 'FixedLR', **params) -> _LRScheduler

builds leraning rate scheduler

Parameters:

  • optimizer (Optimizer) –

    optimizer object

  • _target_ (str, default: 'FixedLR' ) –

    learning rate scheduler configuration dict. Defaults to "FixedLR".

Raises:

  • NotImplementedError –

    description

Returns:

  • _LRScheduler ( _LRScheduler ) –

    learning rate scheduler object

Source code in SaigeToolkit/learning/scheduler.py 
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def build_scheduler(optimizer: Optimizer, _target_: str = "FixedLR", **params) -> _LRScheduler:
    """builds leraning rate scheduler

    Args:
        optimizer (Optimizer): optimizer object
        _target_ (str, optional): learning rate scheduler configuration dict. Defaults to "FixedLR".

    Raises:
        NotImplementedError: _description_

    Returns:
        _LRScheduler: learning rate scheduler object
    """
    if _target_ not in _types:
        raise NotImplementedError(f"SCHEDULER {_target_} not implemented")
    logger.info(f"[{'SCHEDULER'.center(9)}] {_target_} [params] {params}")
    return _types[_target_](optimizer, **params)

visualize_scheduler 

visualize_scheduler(optimizer, scheduler, epochs)
Source code in SaigeToolkit/learning/scheduler.py 
373
374
375
376
377
378
379
380
381
def visualize_scheduler(optimizer, scheduler, epochs):
    lrs = []
    for _ in range(epochs):
        optimizer.step()
        lrs.append(scheduler.get_lr())
        scheduler.step()

    plt.plot(lrs)
    plt.savefig("lr_scheduler_test.png", dpi=300)

trainer

logger module-attribute 

logger = getLogger('SaigeResearch')

BaseTrainer

Bases: ABC

Saige Vision2 Engine 학습 API의 베이스 인터페이스 입니다. 기본적인 규약만 정해져있으며 각 태스크에 맞게 abstractmethod들을 구현하고, error_handler를 씌워서 노출시키면 됩니다.

About Init

init 메소드 작성 시 util.reproducibility.store_config 데코레이터를 활용하면 _config를 쉽게 저장할 수 있습니다.

Example: class Trainer(BaseTrainer): @store_config(attr="_config") # 인스턴스 생성 시 self._config 변수에 생성 파라미터들 저장됨 def init(self, param1, param2): ...

About Checkpoint

체크포인트를 저장/로드 하는 인터페이스는 save_checkpoint, load_checkpoint로 이미 구현되어 있으며, 태스크에 맞게 state_dict, load_state_dict를 구현하면 해당 함수를 이용해 상태를 저장/로드 하는 방식입니다.

_config instance-attribute 
_config: dict
version instance-attribute 
version: Version
config property 
config: dict
to_device abstractmethod 
to_device(device: Union[device, int, str]) -> None
Source code in SaigeToolkit/learning/trainer.py 
42
43
44
@abstractmethod
def to_device(self, device: Union[torch.device, int, str]) -> None:
    pass
train_one_step abstractmethod 
train_one_step() -> dict
Source code in SaigeToolkit/learning/trainer.py 
46
47
48
@abstractmethod
def train_one_step(self) -> dict:
    pass
validation_enabled abstractmethod 
validation_enabled() -> bool
Source code in SaigeToolkit/learning/trainer.py 
50
51
52
@abstractmethod
def validation_enabled(self) -> bool:
    pass
validate abstractmethod 
validate() -> dict
Source code in SaigeToolkit/learning/trainer.py 
54
55
56
@abstractmethod
def validate(self) -> dict:
    pass
state_dict abstractmethod 
state_dict(**options) -> dict
Source code in SaigeToolkit/learning/trainer.py 
58
59
60
@abstractmethod
def state_dict(self, **options) -> dict:
    pass
load_state_dict abstractmethod 
load_state_dict(state_dict: dict, version: Optional[Version] = None, **options) -> None
Source code in SaigeToolkit/learning/trainer.py 
62
63
64
@abstractmethod
def load_state_dict(self, state_dict: dict, version: Optional[Version] = None, **options) -> None:
    pass
save_checkpoint 
save_checkpoint(checkpoint_path: str, metadata: Optional[dict] = None, password: Optional[str] = None, **options)
Source code in SaigeToolkit/learning/trainer.py 
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def save_checkpoint(
    self,
    checkpoint_path: str,
    metadata: Optional[dict] = None,
    password: Optional[str] = None,
    **options,
):
    CheckpointHandler.save(
        checkpoint_path=checkpoint_path,
        password=password,
        version=str(self.version),
        config=self.config,
        state_dict=self.state_dict(**options),
        metadata=metadata,
    )
    logger.info(f"{type(self).__name__} checkpoint saved: {checkpoint_path}")
load_checkpoint 
load_checkpoint(checkpoint_path: str, password: Optional[str] = None, **options) -> Dict
Source code in SaigeToolkit/learning/trainer.py 
83
84
85
86
87
88
89
90
91
92
93
94
95
96
def load_checkpoint(
    self,
    checkpoint_path: str,
    password: Optional[str] = None,
    **options,
) -> Dict:
    checkpoint = CheckpointHandler.load(
        checkpoint_path=checkpoint_path,
        password=password,
    )
    version = Version.from_string(checkpoint.get("version", "0.0.0"))
    self.load_state_dict(checkpoint.get("state_dict"), version=version, **options)
    logger.info(f"{type(self).__name__} loaded from {checkpoint_path}")
    return checkpoint.pop("metadata", {})
build classmethod 
build(config: dict)
Source code in SaigeToolkit/learning/trainer.py 
 98
 99
100
101
@classmethod
def build(cls, config: dict):
    init_cuda_when_available()
    return cls(**config)