Potential memory leak

[prasannagn16]

Description
I am leveraging Triton Inference Server r24.07.
While trying to analyse the code , I see some potential memory leak for the some of the below file ,

1.common.cc

           ` *infer_input = new InferInput(name, dims, datatype); `

2.common.cc

        `*infer_output = new InferRequestedOutput(name, datatype, class_count);` 

3.torchserve_infer_input.cc

	`TorchServeInferInput* local_infer_input = new TorchServeInferInput(name, dims, datatype); `

4.torchserve_http_client.cc

	`*infer_result = reinterpret_cast<InferResult*>(new InferResult(infer_request));`

5.openai_infer_input.cc

	 `OpenAiInferInput* local_infer_input = new OpenAiInferInput(name, dims, datatype); `

6.grpc_client.cc

	 `*infer_result = reinterpret_cast<InferResult*>(new InferResultGrpc(response, request_status));` 

7.grpc_client.cc

	  `*infer_result = reinterpret_cast<InferResult*>(new InferResultGrpc(response));	`

I think here not handle the memory deallocation, it’s directly passing the raw pointer .

Consider one example :
common.cc

Error
InferInput::Create(
    InferInput** infer_input, const std::string& name,
    const std::vector<int64_t>& dims, const std::string& datatype)
{
  *infer_input = new InferInput(name, dims, datatype);
  return Error::Success;
}

InferInput** infer_input is pointer to a pointer (i.e., a raw pointer). It holds the address of a pointer to InferInput

Here its create the “create” method as a static .

common.h

class InferInput {
 public:
  /// Create a InferInput instance that describes a model input.
  /// \param infer_input Returns a new InferInput object.
  /// \param name The name of input whose data will be described by this object.
  /// \param dims The shape of the input.
  /// \param datatype The datatype of the input.
  /// \return Error object indicating success or failure.
  static Error Create(
      InferInput** infer_input, const std::string& name,
      const std::vector<int64_t>& dims, const std::string& datatype);
};

client_backend.h

class InferInput {
 public:
  /// Create a InferInput instance that describes a model input.
  /// \param infer_input Returns a new InferInput object.
  /// \param kind The kind of the associated client backend.
  /// \param name The name of input whose data will be described by this object.
  /// \param dims The shape of the input.
  /// \param datatype The datatype of the input.
  /// \return Error object indicating success or failure.
  static Error Create(
      InferInput** infer_input, const BackendKind kind, const std::string& name,
      const std::vector<int64_t>& dims, const std::string& datatype);

  virtual ~InferInput() = default;
};

For the above code I have understand that ,
this class designed in factory design pattern function but still since passing raw pointer ,if caller's or user as forgot handle the memory deallocation it leads to memory leak.

Solution is:

• Use while creating “create“ method parameter for smart pointer or handle the memory deallocation in the raw pointer .

• Handel the raw pointer deallocation using delete.

Triton Information r24.07
The code designed such way that as per the requirement user need to handle the memory .Is it documented for any user guide?

Expected behaviour

• Handle the memory deallocation for raw pointer using delete or else use smart pointer.

[nthung11cist]

I encountered a memory leak issue when using the Triton gRPC client. I tried using smart pointers to manage the raw pointer inputs and outputs, and I also initialize the gRPC client, input, and outputs only once in the constructor. However, the memory leak issue still persists. After each gRPC request, the memory increases. If you have any solutions, please help me. Thank you!

std::unique_ptrtriton::client::InferenceServerGrpcClient mGrpcClient;
std::shared_ptrtriton::client::InferInput mInput0Ptr;
std::shared_ptrtriton::client::InferRequestedOutput mOutput0Ptr;
std::shared_ptrtriton::client::InferRequestedOutput mOutput1Ptr;
std::shared_ptrtriton::client::InferRequestedOutput mOutput2Ptr;

triton::client::Error err = triton::client::InferenceServerGrpcClient::Create(&mGrpcClient, url, false);
if (!err.IsOk())
{
LOG(INFO) << "Error: unable to create client for inference: " << err;
exit(1);
}

std::vector<int64_t> shape = {mBatchSize, mInputW, mInputH, mChannel};

triton::client::InferInput *input0;
err = triton::client::InferInput::Create(&input0, mInput0Name, shape, "UINT8");
if (!err.IsOk())
{
LOG(INFO) << "unable to get input: " << err;
exit(1);
}
mInput0Ptr.reset(input0);

triton::client::InferRequestedOutput *output0;
err = triton::client::InferRequestedOutput::Create(&output0, mOutput0Name);
if (!err.IsOk())
{
LOG(INFO) << "unable to get output: " << err;
exit(1);
}
mOutput0Ptr.reset(output0);

triton::client::InferRequestedOutput *output1;
err = triton::client::InferRequestedOutput::Create(&output1, mOutput1Name);
if (!err.IsOk())
{
LOG(INFO) << "unable to get output: " << err;
exit(1);
}
mOutput1Ptr.reset(output1);

triton::client::InferRequestedOutput *output2;
err = triton::client::InferRequestedOutput::Create(&output2, mOutput2Name);
if (!err.IsOk())
{
LOG(INFO) << "unable to get output: " << err;
exit(1);
}
mOutput2Ptr.reset(output2);

[varunrao1991]

@nthung11cist, try to narrow down the approach to find memory leak. First only try

triton::client::Error err = triton::client::InferenceServerGrpcClient::Create(&mGrpcClient, url, false);
if (!err.IsOk())
{
LOG(INFO) << "Error: unable to create client for inference: " << err;
exit(1);
}

for every request and see if memory leak is from mGrpcClient. If not then create one input and see if you get memory leak.

triton::client::Error err = triton::client::InferenceServerGrpcClient::Create(&mGrpcClient, url, false);
if (!err.IsOk())
{
LOG(INFO) << "Error: unable to create client for inference: " << err;
exit(1);
}

std::vector<int64_t> shape = {mBatchSize, mInputW, mInputH, mChannel};

triton::client::InferInput *input0;
err = triton::client::InferInput::Create(&input0, mInput0Name, shape, "UINT8");
if (!err.IsOk())
{
LOG(INFO) << "unable to get input: " << err;
exit(1);
}
mInput0Ptr.reset(input0);