base_sampler

data.batch_sampler.base_sampler

Batch sampler 클래스의 기본이 되는 추상 클래스를 정의한 모듈입니다.

BatchSampler 클래스를 상속받아 새로운 batch sampler를 정의할 수 있습니다. Naive sampler는 가장 기본적인 sampler 구현 예시로, 데이터셋의 인덱스를 순서대로 배치로 묶어 반환합니다.

logger `module-attribute`

logger = getLogger('SaigeResearch')

BatchSampler

BatchSampler(dataset: Dataset, batch_size: int = 1, repeat: bool = False, shuffle: bool = False, drop_last: bool = False)

Bases: Registerable, ABC

Baseclass for all batch_sampler classes.

Attributes:

dataset (Dataset) –

dataset to sample from.
batch_size (int) –

batch size.
repeat (bool) –

repeat data when data is not enough for batch size.
shuffle (bool) –

shuffle data.
drop_last (bool) –

drop last data when data is not enough for batch size.

Note

batch_size, shuffle, and drop_last should be popped from dataloader config, then passed to the batch sampler config.
repeat: for SaigeVision2 api, should be set as True.
- if number of data is enough for batch size, then repeat is neglected.
drop_last: for SaigeVision2 api, should be set as True.
- to prevent the last batch from being smaller than the specified batch size.
- if False, even when repeat is set as True, the last batch will be smaller than the specified batch size.
for Training, shuffle should be set as True.
- drop_last and repeat are optional.
for Validation, repeat, shuffle, and drop_last should be set as False.

Examples:

>>> cfg_sampler.update(
...     {
...         "shuffle": cfg_dataloader.pop("shuffle", False),
...         "drop_last": cfg_dataloader.pop("drop_last", False),
...         "batch_size": cfg_dataloader.pop("batch_size", 1),
...     }
... )
... batch_sampler = build_batch_sampler(dataset=dataset, **cfg_sampler)
... dataloader = build_dataloader(
...     dataset,
...     collate_fn=collate_fn,
...     batch_sampler=batch_sampler,
...     **cfg_dataloader,
... )

Source code in SaigeToolkit/data/batch_sampler/base_sampler.py

def __init__(
    self,
    dataset: data.Dataset,
    batch_size: int = 1,
    repeat: bool = False,
    shuffle: bool = False,
    drop_last: bool = False,
) -> None:
    self.dataset = dataset

    self.batch_size = batch_size
    self.repeat = repeat

    self.shuffle = shuffle
    self.drop_last = drop_last

    self._n_repeat = None
    self._buffer_length = None

NaiveSampler

NaiveSampler(dataset: Dataset, batch_size: int = 1, repeat: bool = False, shuffle: bool = False, drop_last: bool = False)

Bases: BatchSampler

Naive sampler implementation with buffer extension feature added.