Allocation in the Hot Path

The previous articles followed the memory reads inside the $AAPL update loop — every field fetch and the cache lines it touched. The next question is where the loop's memory comes from.

A line such as recent_updates.push_back(update) usually writes into space the vector already owns. When the vector is full, the same line asks the heap for a larger buffer, moves the existing updates into it, and frees the old one. The source reads identically in both cases; the iteration that triggered the growth did much more work.

What an allocation is

A heap allocation asks a runtime component called the allocator for a block of memory. Code uses the heap when the amount of memory or its lifetime depends on decisions made while the program is running.

Local variables often use automatic storage instead. A function reserves space for them when it begins and releases that space when it returns. The compiler knows how much room the function needs, so this bookkeeping is usually a stack-pointer adjustment.

Copy
void handle_order() {
    Order order;              // automatic storage
    std::int32_t qty = 100;
}                             // storage released on return

Order* create_order() {
    return new Order();       // asks the allocator for heap storage
}

Compare what happens on each side. Returning from handle_order releases its locals with one stack-pointer adjustment. Returning from create_order releases nothing — the Order stays alive until something deletes it, and the allocator has to track which blocks are in use. To satisfy a request, it may hand back memory it already holds for the current thread, or it may have to synchronize with another thread, refill an internal pool, or ask the operating system for more pages. Which of those paths you get depends on the allocator, the request size, and the program's current memory state.

A function manages automatic storage as a group. The allocator manages heap blocks individually.

That variability is the problem in latency-sensitive code. Most calls follow the short path; occasionally one takes the long one. Put an allocation inside the update loop, and the long path can land on an otherwise ordinary market tick.

How push_back reaches the heap

A std::vector keeps three pieces of state: a pointer to its element buffer, a size counting constructed elements, and a capacity recording how many elements fit in that buffer.

Follow one insertion. While size < capacity, push_back constructs the new element in the next slot and never touches the allocator. When size == capacity, the buffer has no next slot, so the vector allocates a larger buffer, moves each existing element into it, frees the old buffer, and only then constructs the new element.

The `push_back` that exceeds capacity has to replace the vector's buffer before it can add the new element.

Amortized O(1) describes the average cost across many insertions. It does not make every insertion equally cheap, and when you review a hot path, the insertion you care about is a specific one: can this iteration be the one that triggers growth?

The same question applies to the other library types in the loop. Inserting a new std::unordered_map entry commonly allocates a node, and once the map crosses its load-factor limit, it also replaces the bucket array. String concatenation may need a larger character buffer. A std::function can allocate when its callable does not fit in the implementation's internal storage.

Searching for new or malloc will not find these calls. You have to know which operations request storage depending on the object's current state.

Keep growth outside the loop

These two functions collect the same orders. The first vector starts empty and grows whenever it runs out of capacity, so the growth lands somewhere inside the loop. The second sizes the buffer before the loop begins, so every push_back finds a slot that already exists.

Copy
// The vector can grow during the loop.
void handle_orders_a(const std::vector<Order>& new_orders) {
    std::vector<Order> log;
    for (const Order& order : new_orders) {
        log.push_back(order);  // may replace the buffer
    }
    apply(log);
}

// The vector allocates its buffer before the loop.
void handle_orders_b(const std::vector<Order>& new_orders) {
    std::vector<Order> log;
    log.reserve(new_orders.size());
    for (const Order& order : new_orders) {
        log.push_back(order);  // copies order without growing the buffer
    }
    apply(log);
}

Each push_back still copies its Order — reserve removes only the reallocations and element moves that growth causes. That is the distinction to account for when you tally the work inside the loop.

Capacity growth adds allocator work to one iteration even though the loop body has not changed.

If the same buffer can serve every batch, keep it alive and call clear() between batches. clear() destroys the elements but keeps the capacity, so the next batch reuses the existing allocation without a call to the allocator. Memory pools and arenas extend the same idea when several objects need controlled storage.

The steady-state goal: acquire the memory before processing begins and reuse it while updates arrive. The loop then does the work visible in its source, instead of occasionally dropping into the allocator to do work no line of the loop shows.

In interviews

• "What can allocate inside this loop?" → "I would check container growth, map insertion and rehashing, string construction, and type-erased callables such as std::function."
• "How would you remove vector growth from the hot path?" → "Reserve the required capacity before the loop and reuse the buffer across batches when its lifetime allows it."
• "Does reserve make each push_back free?" → "No. It prevents capacity growth, but push_back still constructs or copies the inserted element."

The next article follows the copies and moves that remain after allocation has been moved out of the loop.