Batch.h

Go to the source code of this file.

Batch inspection is a feature that processes multiple images at once.

ExampleClsRuntimeBatch.cpp shows how to perform batch inspection. Batch inspection in SAIGE VISION is a feature that processes multiple images at once. This uses more GPU memory. Also, processing more images at once does not always make it faster. Therefore, it is necessary to select an appropriate Batch Size.

Batch Inspection Steps

To perform batch inspection, follow these steps:

• Set the Batch Size when creating a handle
• Load multiple image data into memory
• Call the Batch Infer API SaigeInferFromMultipleByteArray
• Retrieve and utilize the inspection results for each image from each result handle

Setting the Batch Size

Set the SaigeHandleConfig::batchSize value and create a handle by specifying the flag.

// ExampleClsRuntimeBatch.cpp:91

SaigeInferenceHandle handle = nullptr;
SaigeHandleConfig config;
config.batchSize = batchSize;
unsigned int handleFlags = SAIGE_SET_BATCH_SIZE;
if (optimize) {
    handleFlags |= SAIGE_OPTIMIZE_HANDLE;
}
retval = SaigeCreateInferenceHandle(&handle, kProjectTypeCLS, trainedModelPath.c_str(),
                                    deviceIndex, &config, handleFlags);
if (retval < 0) {
    std::cerr << "SaigeCreateInferenceHandle failed - " << SaigeStrError(retval) << std::endl;
    return retval;
}

Load Images

In this example, all .png files under the specified dataDir path are read and stored in imagePaths. Then, the application code bundles and processes these images according to batchSize. That is, in this example, batchSize >= numOfImages.

The following code is the beginning of the loop that inspects the entire image list divided by batchSize. Load the image corresponding to each iteration into ucBatchImageData.

// ExampleClsRuntimeBatch.cpp:224

unsigned char** ucBatchImageData = new unsigned char*[batchSize];
for (int i = 0; i < batchSize; ++i) {
    ucBatchImageData[i] = nullptr;
}

int numIter = (int)std::ceil((float)imagePaths.size() / (float)batchSize);
int numFinalImages = imagePaths.size() % batchSize;
if (numFinalImages == 0) numFinalImages = batchSize;
int numBatch = batchSize;

// start iteration
for (int iter = 0; iter < numIter; ++iter) {
    if (iter == numIter - 1) {
        numBatch = numFinalImages;
    }
    // 이미지 데이터 메모리를 해제합니다. (루프일관성)
    clearImageBatch(ucBatchImageData, batchSize);

    // 이미지 batch를 생성합니다.
    int w_batch = 0, h_batch = 0, c_batch = 0, type_batch = -1;
    bool invalid_batch = false;
    for (int i = 0; i < numBatch; ++i) {
        invalid_batch = false;
        int img_idx = i + iter * batchSize;
        ucBatchImageData[i] =
            (unsigned char*)SaigeGetImageData(imagePaths[img_idx].c_str(), &w, &h, &c, &type);
        if (i == 0) {
            w_batch = w;
            h_batch = h;
            c_batch = c;
            type_batch = type;
        }
        if (ucBatchImageData[i] == nullptr) {
            // 로드에 실패한 이미지가 있으면 다음 배치로 넘어갑니다.
            invalid_batch = true;
            std::cerr << "SaigeGetImageData() failed - " << imagePaths[img_idx] << std::endl;
            break;
        }
        if (w != w_batch || h != h_batch || c != c_batch) {
            // 한 배치 내의 이미지 크기는 모두 같아야 합니다.
            invalid_batch = true;
            std::cerr << "Image sizes in a same batch should be same." << std::endl;
            break;
        }
    }

Call the Batch Infer API

Call the Batch Infer API, SaigeInferFromMultipleByteArray function.

// ExampleClsRuntimeBatch.cpp:287

retval = SaigeInferFromMultipleByteArray(
    handle, numBatch, (const unsigned char**)ucBatchImageData, w, h, imageFlag, results);
if (retval < 0) {
    std::cerr << "SaigeInferFromByteArray() failed - " << SaigeStrError(retval)
              << std::endl;
    continue;
}

Retrieve Inspection Results

After calling the Batch Infer API, retrieve the inspection results for each image from each result handle. Therefore, the number of result handles must be at least numOfImages.

// ExampleClsRuntimeBatch.cpp:300

for (int i = 0; i < numBatch; ++i) {  // DEFAULT
    retval = SaigeGetInferenceResultScore(results[i], scores[i].data(), true);
    if (retval < 0) {
        std::cerr << "SaigeGetInferenceResultScore() failed - " << SaigeStrError(retval)
                  << std::endl;
        clearImageBatch(ucBatchImageData, batchSize);
        delete[] ucBatchImageData;
        SaigeDestroyInferenceHandle(handle);
        clearResults(results, batchSize);
        return retval;
    }

Relationship between Batch Size and numOfImages

The Batch Size set and the actual number of images input to the SaigeInferFromMultipleByteArray function do not have to match.

The Batch Size is the number of images processed simultaneously on the GPU.

Therefore, if the number of images input to the Batch Infer API, numOfImages, is greater than the Batch Size, the operation is divided into multiple operations. Also, if the number of input images is less than the Batch Size, GPU resources are wasted. Therefore, the Batch Size is an important tuning factor that should be set carefully.

That is, the number of input images, numOfImages, can be set differently depending on the characteristics of the application. For example, there are cases where multiple images are collected and processed at once, and cases where the inspection results are needed as soon as

Generally, to increase the throughput of the inspection, it is recommended to set the batchSize as large as possible. Conversely, if the latency of the inspection is important, it is recommended to call the inspection API as soon as possible without waiting for the inspection images to be collected.