Founded batching opportunity

[andreydyadyunov]

Binary text classification.

Solution is implemented by several models combined into an ensemble. With the chosen architecture,
we were able to achieve batch processing.
The architecture is built as follows:

• splitter

input: "text"
data_type: TYPE_STRING
dims: [ 1 ]

As an input first model takes a text.
Texts may be of different length and may contain different number of sentences.
Model splits that text into sentences.
First model output is an array of one text sentences.

output: "sentences"
data_type: TYPE_STRING
dims: [ -1, 1 ]

• batcher

input: "sentences"
data_type: TYPE_STRING
dims: [ 1 ] # takes dimension 1, to let python_backend collect batch by number of sentences

dynamic_batching {
    max_queue_delay_microseconds: 300000
    preferred_batch_size: [64]
}

As as input model takes an array of sentences with batch size 128.
Inside the model, assembles a batch of the optimal length. Batch has a following structure
[[sent1_text1, ... sent_n_text1], [sent1_text2, ... , sent_n_text2] ... , [sent1_text_n, ...]]
The model itself triggers the next stage - classifier

• classifier

  input: "batch"
  data_type: TYPE_INT32
  dims: [ -1, 1 ]
  

  As an input takes an array of sentences with batch size -1, collected in batcher.
  As an output model returns an array of the same length as in input with scores for each sentence.
  The output returns back to batcher model
  

  output: "scores"
  data_type: TYPE_FP32
  dims: [ -1, 1 ]
  

• batcher

  After classifier inference, model takes apart the batch and aggregates scores.
  The batcher model output is aggregated score for each text.
  

  output: "agg_scores"
  data_type: TYPE_FP32
  dims: [ -1, 1 ]
  

Code to collect batch in batcher model:

for request in requests:
    in_0 = pb_utils.get_input_tensor_by_name(request, "sentences").as_numpy()
    requests_lens.append(in_0.shape[0])
    batch_list.append(in_0)
batch_array = np.vstack(batch_list)
decode_utf8 = np.vectorize(lambda text_bytes: text_bytes.decode("utf-8"))
batch_prepared = list(decode_utf8(batch_array).reshape(-1))

Code to take apart batch back to requests:

offset = 0
for rl in requests_lens:
    cls_scores = classifier_response[offset : offset + rl]
    offset += rl
    agg_score = agg_fun(cls_scores)
    out_tensor_0 = pb_utils.Tensor("agg_scores", agg_score.astype(np.float32))
    inference_response = pb_utils.InferenceResponse(output_tensors=[out_tensor_0])
    responses.append(inference_response)

So the question is

Is that batching type a bag or a feature? Do we use it correctly? How do you see the architecture of such model?