What I am learning building a C++ inference server: ONNX and RAII

A model is a saved function

An ONNX file is a function someone saved to disk. Named inputs in, some operations, named outputs out, with the learned weights stored inside the file. ONNX Runtime is the program that executes it.

I do not need to know every operator to serve a model. I need the edge of the graph: what goes in, what comes out. My fixture is tiny on purpose, four numbers in and two out, so I can check the answer on paper.

Decision

batch can change, 4 cannot

The shape [batch, 4] holds two different kinds of number. batch is mine, it is how many requests I stack into one call. The 4 belongs to the model, it is the four specific numbers it was trained on.

The trap: one sample is [1, 4], not [4]. And a shape check only proves the tensor is well formed. Send the right four numbers in the wrong order or wrong units and you get a valid tensor with a garbage prediction, which is why preprocessing goes in the contract too.

• one sample is [1, 4], never [4]
• batching is stacking rows, not padding: output row i answers input row i
• float32 is 4 bytes, so a [8, 4] batch is 128 bytes - this matters once a queue holds them
• record names, types, shapes, preprocessing, and opset before writing code

const bool valid_input_shape =
    input.shape.size() == 2 && input.shape[0] < 0 && input.shape[1] == 4;
const bool valid_output_shape =
    output.shape.size() == 2 && output.shape[0] < 0 && output.shape[1] == 2;
if (!valid_input_shape || !valid_output_shape) {
  throw std::invalid_argument(
      "incompatible model contract: expected shapes [batch, 4] and "
      "[batch, 2]");
}

RAII in one example

An object grabs a resource when it is constructed and releases it when it is destroyed. That is it. The point is that there is no path out of the scope that skips the cleanup, so the safe path becomes the lazy path.

std::vector frees its memory, std::ifstream closes its file, and the ONNX Runtime C++ wrappers release their handles. I do not call a release function anywhere in this project.

{
  std::lock_guard lock(mu);  // constructor locks
  do_work();                 // throws? returns early? does not matter
}                            // destructor unlocks, always

Who owns the model

One class, ModelSession. Ort::Env is the runtime, Ort::SessionOptions is the load config, Ort::Session is the loaded graph and weights. The session is what runs inference.

The detail I would have missed: declaration order. C++ destroys members in reverse, so env_ first and session_ last means the session dies before the runtime it depends on. Flip those three lines and the code looks identical and breaks on shutdown.

private:
  // Members are destroyed in reverse declaration order:
  // session_ -> options_ -> env_. The runtime must outlive the session.
  Ort::Env env_;
  Ort::SessionOptions options_;
  Ort::Session session_;

The part that almost got me: borrowed memory

When you build a tensor from an existing pointer, the tensor does not copy your data and does not own it. It is a view. You still own the bytes and have to keep them alive as long as anything reads through it.

Where I got stuck: I assumed RAII meant I could stop thinking about lifetimes. RAII guarantees the vector gets freed correctly. It has no opinion about whether something else is still pointing at that memory when it happens.

Result

Ort::Value make_input() {
  std::vector<float> data{0.25f, -0.5f, 1.0f, 0.0f};
  return Ort::Value::CreateTensor<float>(
      memory_info, data.data(), data.size(),
      shape.data(), shape.size());
}  // data dies here; the returned tensor points at dead memory

std::vector<float> input = features;   // I own these bytes
const std::vector<int64_t> input_shape{1, 4};

Ort::Value input_tensor = Ort::Value::CreateTensor<float>(
    memory_info, input.data(), input.size(),
    input_shape.data(), input_shape.size());

auto outputs = session_.Run(Ort::RunOptions{nullptr}, input_names,
                            &input_tensor, 1, output_names, 1);

// Copy out before outputs goes out of scope.
const float* scores = outputs[0].GetTensorData<float>();
return std::vector<float>{scores[0], scores[1]};

Why the first version is a CLI

A CLI proves the hard part with nothing else in the way: model loads, contract matches, lifetimes hold, Run succeeds, output matches a number I did by hand. If inference is wrong here, the bug is in my ONNX Runtime integration and nowhere else.

Start at HTTP instead and the same bug can look like JSON, threads, or routing. My first test is one line: feed it 0.25, -0.5, 1.0, 0.0 and it must print 0.25 and 0.5.

M1  offline CLI      load + contract + Run + golden output
M2  HTTP             one request in, one prediction out
M3  queue            accept work, bound the memory
M4  batching         stack rows, route results back by index

if inference breaks at M1  ->  it is the ORT integration
if inference breaks at M2  ->  ORT, or JSON, or threads, or routing

What I got wrong

The corrections are the useful part.

• thought four numbers meant shape [4] - the batch dimension is not optional
• assumed CreateTensor copies my floats - it views them
• assumed RAII removes lifetime bugs - it removes cleanup bugs, which is not the same thing
• thought a dead input buffer would throw - it reads whatever is at that address and returns confident nonsense
• treated the contract as shapes and types - preprocessing and units belong in it too

Next is M2: the same ModelSession behind one HTTP endpoint. Same ownership rules, harder to see once threads show up.