RidgeRun.ai Shares Memory Optimization Best Practices to Run AI Models on NVIDIA Jetson

Why Memory Optimization Matters on Edge Devices

If you have ever deployed a deep learning model on an embedded system, you have probably felt disappointed at first. The model works great during development on a workstation: it is accurate, runs quickly in PyTorch, and seems ready for deployment. However, moving it to the target edge device is often where the real challenge begins. Achieving real-time performance turns out to be much harder than expected. At that point, it becomes clear that:

• The time it takes to make a prediction is too long. 

• It uses a lot of memory at runtime or it does not fit in the device’s memory at all.

• The frame rate is not good enough for your application.

• The model size makes it difficult to store and manipulate. 

It is not a limitation of the device. NVIDIA Jetson platforms are powerful enough to run some of today’s most demanding AI models. However, deploying AI at the edge requires a different approach, one that makes efficient use of the available resources. This means minimizing the memory footprint while maintaining the real-time performance required by the application.

NVIDIA highlights 5 layers of memory optimization in its latest tech blog. This brings better memory-efficient AI compute to NVIDIA Jetson Platform and makes it easy to overcome those challenges. We want to go deeper into best practices. 

Optimizing models to reduce the memory footprint and execution time is an art, not a trial and error task. You need to know how models work and how they behave on your target device to understand what can be done and what such changes imply for your model’s performance. 

The goal of this tutorial is to guide you through practical model optimization techniques that go beyond academic benchmarks and deliver real value in production applications. Along the way, we'll show how to reduce the memory footprint of a standard YOLO11m model by up to 67% while discussing the trade-offs involved with each optimization technique.

Effective NVIDIA Jetson memory optimization starts with understanding where memory is consumed during inference and which techniques provide the best balance between memory usage, latency, and model accuracy.

Understanding Where GPU Memory Goes

Before digging into the different techniques for model optimization, it is important to understand how models work  from the memory perspective and how they relate to the execution platform such as the NVIDIA Jetson family. Understanding this will help you understand the goal of each model optimization technique and balance every trade-off required to accomplish your goal.

Deep Learning models, from the classic perceptron to the disruptive transformers, are nothing but a bunch of numbers (the weights) and mathematical operations linked together on a specific layout that will process input data such as image, video, text, audio, among others. So, every time you have a model and run it, you have to deal with:

1. A composed file that contains all these numbers and defines the operations – this is what you call “the model”.

2. Grabbing your input data and passing it through this mathematical model.

3. Getting the output of the mathematical operations and interpreting it accordingly, for instance, the output could be a voting among different classes for an image recognition model, or the probabilities of every word sequence for a LLM. 

All these stages require memory from your system. 

The Model’s Memory

Let’s start from the model perspective. There are three main spots of memory retention within a model: the weights, the activation tensors, and the workspace memory, as depicted in the image below. 

Figure 1. GPU Memory Breakdown for Deep Learning Models

Model Weights

This is the first core element of every deep learning model. Every model is composed of millions or billions of weights that need to be stored and used during inference. This is a static memory usage, it loads into the GPU memory once and is used during all the processing operations. 

Activation Tensors

This is typically one of the biggest spots of memory utilization in deep learning models. Activation tensors are nothing more than the intermediate results between layers within a model. Consider the image below. Every time you execute a layer within a model such as a convolution, a batch normalization, a pooling, etc; it stores the result in an intermediate memory commonly known as activation tensors. 

Figure 2. Activation Tensors 

The image in Figure 2 depicts an image-based model to provide a better understanding of why activation tensors require so much memory. If you think about the convolution layers, despite reducing the size of the input image, they generate N-resized versions of it, one for each output channel. At the end of the day, every convolution layer stores a copy of these N-resized images in the activation tensors. Scaling this up across the total number of layers in your model leads to aggressive memory allocation.

Workspace Memory

This is the final big chunk of memory used when running models. So far we have talked about the memory used for the weights and the memory used to store the final results of every internal layer. The missing part is, where do we store the intermediate results for every mathematical operation? For some of the layers, generating the final output requires an internal processing that also requires a scratch memory. This memory is temporal and dynamically allocated on demand depending on the computations.  This is the workspace memory!

The Pre-Processing Stage (The Input)

Now let’s talk about feeding your model with the input data. If we use image models as an example,, the input data corresponds to the input image we want to run the inference on. This image must be stored in memory and the larger the resolution the larger the memory that needs to be reserved. 

Additionally, depending on your camera specifications and your inference pipeline, you might need to run some pre-processing operations such as color space conversion, resizing, binning, among others. Similarly to the internal operations within the model, these pre-processing operations require scratch memory before producing the final input to the model.

Pre-processing memory is typically not as large as a model's memory but it sometimes offers opportunities for optimization through accelerated hardware units. 

The Post-Processing Stage (The Output)

This one is typically equal or smaller than the input memory, depending on the model’s use case. It just stores the output results from the model and performs some minor post-processing to accommodate the data to what is needed for the downstream stages of the application.

At this point, you might already have a good understanding of how memory relates with the model operation. In the next sections we will focus on how to optimize each of these memory allocations in order to reduce the model’s footprint. 

NVIDIA Jetson Memory Optimization Techniques

This section discusses some of the most common optimization techniques for reducing the memory footprint of a model. The techniques are ordered in a preference order, from the most easy-to-apply (but effective) techniques to the more complex ones. The main rule is not to try the most complex optimizations if you can get what you need with simpler but powerful techniques first. 

Our Subject of Study

For the remainder of this document we will use the YOLOv11 model as subject of study and we will go over the complete optimization process to reduce its memory footprint. 

Although YOLO models are not particularly memory-hungry, most AI applications involve the use of two or more models, and this combination is what will eventually become the bottleneck in your system's performance.

Technique 1: Choosing the Right Model Size

This is the rule #1 for every AI application. Your custom application does not always need the almighty model to do what it is supposed to do.  Let’s take as an example the YOLOv11 model. Table 1 lists the typical memory requirements for YOLOv11 running on PyTorch at FP32 precision. If you choose YOLO11x as default for instance, you are already allocating about 0.5GB of memory for every instance of the model in your system which can easily scale up. From our experience with customers requiring object recognition or similar applications, models in the range of the nano or small are good enough for most applications, providing a good balance between detection accuracy and system performance. 

That said, before any further optimizations, make sure to benchmark the different families of the model you are interested in within a subset of your data to evaluate which version meets your requirements.

Table 1. Typical Memory Requirements for Different Versions of the YOLO11 Model (PyTorch FP32)

Technique 2: Inference Framework: TensorRT

This is a mandatory step for almost any AI implementation on an NVIDIA Jetson platform. The inference framework you use to run your model will define the underlying optimizations to take advantage of most of the hardware. 

Moving your models to the TensorRT inference framework optimizes it to be aware of the running hardware and set up the model layers to run as efficiently as possible on such hardware. Some of the main optimizations included when moving to TensorRT are:

• Layer fusion: mixing layers within a single operation leads to faster execution times and a reduction in the activation memory.

• Memory reuse: TensorRT is capable of optimizing the usage of the activation and context memory, so lower memory is used by reusing it on different layers of the model. This requires a high knowledge of the specific platform the model is running on. 

• Hardware resources assignment: contrary to common or generic frameworks such as PyTorch, TensorRT knows better than anyone the intrinsics of the hardware it is optimizing to. This knowledge allows us to optimize the execution of operations and distribute them within the different hardware resources available in the system (Tensor cores, CUDA cores, DLA engines, etc). 

• Other optimizations performed during the TensorRT conversion such as quantization and context memory optimization will be discussed in later sections.  

All these optimizations that TensorRT performs help not only to improve the model’s execution time but also to reduce the required memory for inference. Let’s first take a look at the model itself: running only the model requires some static memory to be allocated as we already saw. Table 2 shows the model’s memory (weights + activation + i/o) for PyTorch and TensorRT.

Table 2. YOLO11 (FP32) Memory Usage Before and After TensorRT Conversion

Although TensorRT is not originally designed for memory reduction, it can be seen how in some cases it can achieve about 22% of memory reduction. In the case of the model’s memory, the reduction comes from how much can TensorRT optimize the activation memory (and the scratch memory not included in the table), so every model will have a baseline size given by the weights allocation. 

Moreover, if we measure and consider the runtime memory requirement for the execution of the models, including not only the model’s memory but all the CUDA allocations, scratch memory, and other, we get a distribution similar to the one on Table 3. Here the memory reductions become more evident. The main reason for this memory optimization by just switching to TensorRT as inference framework is that, as stated earlier, TensorRT knows very well how to optimize the model’s architecture and execution for the target platform, it is capable of merging operations and plan the memory utilization for reuse, while more generic frameworks such as PyTorch are designed to be more generic and omit this kind of optimizations. 

Table 3. Runtime Memory Allocation Delta for YOLO11 Running on PyTorch and TensorRT (FP32) 

Moving a model to TensorRT not only helps with the memory usage but also with the model’s performance. You can easily achieve a significant reduction on your execution time. 

If you are moving your models to an NVIDIA Jetson platform, changing the inference framework to TensorRT when possible is a must. 

Figure 3 shows this improvement for the experiment we just ran.

Figure 3. YOLO11 Model Speedup When Switching Inference Frameworks (FP32) 

At this point it might become obvious that moving your inference framework to TensorRT is a gold rule for NVIDIA-based systems for both performance and memory optimization. 

Technique 3: Quantization

This is the third must-do technique.  After this one, you should be able to reduce your model’s memory significantly with just some simple steps. The remaining sections describe deeper optimizations if you really want to reduce your memory footprint as much as possible. 

Quantization is likely a must when switching to an embedded platform. Most platforms are designed to be very efficient at numerical precisions like FP16 rather than running the inference at FP32. Some others, like most of the high-end NVIDIA Jetson SoM, support mixed precision, which will find the best combination of numerical precision for your model. 

But, what is quantization and why do we want it? Take a look at Figure 4. As discussed earlier, every layer in a neural network contains a set of weights (parameters) and produces activation tensors, which become the inputs to the next layer. Even a relatively small model such as YOLO11n contains approximately 2.62 million parameters. The precision used to store these weights directly determines how much memory the model occupies:

• FP32: 2.62M × 4 bytes ≈ 10.5 MB

• FP16: 2.62M × 2 bytes ≈ 5.2 MB

• INT8: 2.62M × 1 byte ≈ 2.6 MB

Reducing the precision therefore provides an immediate reduction in the memory required to store the model weights. However, the benefits extend well beyond the weights themselves.

During inference, every layer also generates activation tensors, and mathematical operations are performed on both the weights and these activations. If both are represented in lower precision (for example, INT8 instead of FP32), the intermediate activation tensors require less memory, which means less data that needs to be transferred through the memory hierarchy, and thus, hardware that provides native INT8 acceleration, such as NVIDIA Tensor Cores, can execute many more operations per second. The result is lower memory usage, higher throughput, and reduced inference latency

How quantization works, how to achieve it, what does it imply, and how to calibrate your model correctly to achieve a functional model is a whole topic that is out of the scope of this blog. We will talk about this in future publications. 

Figure 4. Model quantization example. 

Figure 5 shows the memory requirements when moving the model to FP16 using TensorRT compared to the FP32 variant. Activation and weights required memory experiments a reduction of about 50% with respect to the FP32 variant, this is expected since we are reducing the total precision to half the original size so we might need half the space for the weights and thus, half of the required activation memory. The table only shows the model’s weights, but other memory spaces such as the workspace memory get reduced also. 

Special attention is paid to the mean latency of the model, which is boosted 2.53x, this is one of the main reasons why quantization is so powerful. This speed up comes primarily from the fact that the calculations are now faster since they get performed at lower precisions and that hardware gets optimized to lower precisions. 

Figure 5. Model's Memory Reduction with Quantization (YOLO11m)

At this point it might be obvious why quantization is almost a MUST for every model deployment in embedded systems, it not only helps to reduce the memory footprint but also provides a boost in performance. This is one of the clearest steps in real-time inference in embedded systems. 

Technique 4: Dynamic Input Shapes

So far we are doing pretty good in reducing our model’s memory footprint. Let’s step into another common topic. Most modern models support different resolutions without needing any retraining. This is the case of the YOLO11 model that is fully convolutional and can fit any input sizes. 

Although supporting a dynamic range of input sizes sounds cool, it comes with an additional memory allocation, which in most cases is unnecessary. 

Take a look at Figure 6 below. It shows the model’s memory and latency for the YOLO11m in FP16 with support for dynamic input shapes and static. By exporting the model to use only one input shape of 640x640 the model’s memory gets dramatically reduced while maintaining almost the same latency. 

Figure 6. Static vs. Dynamic Input Effect on Model Memory Usage for YOLO11m (FP16)

The reason for this to represent realistic memory improvements is mainly because of the activation memory. Despite that the model’= weights keep mostly the same memory (the small difference relies on the addition of constants and algorithms selected), the activation memory shrinks down to about a half since we do not need to allocate memory for all kinds of different image inputs, we already know what size we will get and how much memory we need. 

In practice, this is a valid optimization since for most cases working with a single resolution is enough, and you can even try out slicing techniques like SAHI to overcome high-resolution high-detail detection. 

Technique 5: Smaller Input Resolution

If you can work with a static input resolution, then a new door opens on your path for memory optimization: the input resolution. When you build your TensorRT engine to a specific static resolution, the model’s weights do not change significantly, but the activation memory changes proportionally. Lower input resolution requires less activation memory for calculations and also requires less calculations, all of this leads to less memory requirements and a lower latency. This behavior is depicted in Figure 7. 

Figure 7. Model's Memory and Speed Improvements for Different Static

Input Resolutions (YOLO11m FP16)

You need to be careful though, reducing the input resolution arbitrarily can lead to a severe loss in your model’s inference performance. Make sure to always select a resolution that still works for your use case. 

Technique 6: Structured Pruning

Now we are entering our last two techniques but not the less important ones. First, let’s talk about pruning. When you train a model for your specific use case, a dark truth comes out: a lot of the model’s internal weights are not used or are not critical for the output calculation. In fact, some weights are almost zero and do not account for the final result. 

That said, the basic idea behind pruning is easy to understand: just remove all weights that are close to zero. This reduces the memory required to store them and avoids performing their associated calculations. This pruning method is called structured pruning, and it is the one we will focus on from now on.

Unfortunately, pruning is not as easy as it sounds. Pruning nowadays models require complex tools that surgically removes near-zero or useless weights while altering the model’s architecture to reconnect the cuts. 

Following our example, we took our YOLO11m model and we pruned it. Figure 8, shows the reduction in the number of parameters, which later on will map to a reduction in the weights and activation memory. 

Figure 8. YOLO11m Parameters Before and After Pruning (LAMP Pruning Method, FP16)

Once the model has been pruned, you'll notice that it has lost much of its original accuracy. This is expected, as pruning removes parameters that contributed to the model's predictions. To restore its performance, the model must undergo another round of fine-tuning, allowing the remaining weights to adapt and recover much of the lost accuracy. Figure 9 shows the inference performance after pruning at different epochs. Note that there is not much difference between 50 and 300 epochs, at this point the model is almost limited by the truncation of the original weights. However, the final inference performance is not that far from the original model.

Figure 9. YOLO11m Pruning Inference Performance

Now the real improvement, Figure 10 shows the memory reduction after pruning our model. Since we are reducing the number of weights in the model by removing them, we experience a reduction in both the weights and the activation memory. Moreover, the less weights, the less computations so we also experience a reduction in the model’s latency. 

Figure 10. Model's Memory Reduction After Pruning (YOLO11m, 640×640)

Technique 7: Going Further with Quantization - INT8

A very special case in the quantization topic is the INT8 quantization (and some other less common integer or special quantization variants). Quantizing to INT8  means to move the numeric precision to a totally different numeric representation; the integer representation. When quantizing to INT8 a pre-trained model, it involves two main steps: 

1. Quantizing the weights to INT8

2. Quantizing the activation outputs to INT8 

The basic idea behind quantizing to INT8 is to associate a floating point step (normally called scale) to every integer step. For instance, if our layer has the following weights: 

[-1.24, -0.82, -0.45, 0.01, 0.39, 0.87, 1.12]

Quantizing them to INT8 requires mapping the max and min ranges to the limits of the INT8 representation [-128, 127] and this will define the scale for every INT8 step. Figure 11 shows this process in more detail.

Figure 11. Simple INT8 Quantization Example

The trickier part comes when quantizing the activation outputs. The weights are fixed numbers, but the activation outputs depend on the calculations performed between the weights and the input fed to the model, so its values depend on the input. Since the input to a model is not meant to be fixed – otherwise there wouldn’t be a need for a deep learning model – it is hard to determine the range of values that every activation output will take so we can set the appropriate scale. This is where the calibration process comes through.

During the calibration process, the model is exposed to a subset of data that is representative of the actual data it will face in the intended use case. This representative dataset is used to calculate the approximated variation of the activation outputs and then determine the proper scale for the quantization. It is worth mentioning that if you select a wrong dataset or your model is exposed to a different input when it is deployed, the performance will be affected due to a wrong calibration. 

That said, you might be wondering, is it worth going through all this hassle of calibration for INT8 quantization? The answer is easy by looking at figures 12 and 13 below. INT8 quantization is capable of providing about 86% of memory reduction while speeding up the performance by about 4x. 

Figure 12. Memory Usage After INT8 Quantization (YOLO11m)

Figure 13. Memory Improvement After Using INT8 Quantization (YOLO11m)

Quantizing a model conveys a trade-off with regards to the model’s performance. The following figure shows different performance metrics measured for the YOLO11m at different quantization variants. Despite INT8 quantization being too aggressive, its resulting performance is similar to the one of FP16 and FP32. You will get a smaller memory footprint by sacrificing just some points on your model’s performance. 

Figure 14. Model performance at different numeric precisions (YOLO11m)

Going Beyond These Optimizations

The techniques covered here provide a strong starting point for reducing the memory footprint of AI models on NVIDIA Jetson, but they are not the only options. The right optimization strategy ultimately depends on your model architecture, target NVIDIA Jetson platform, accuracy requirements, and application constraints.

In future articles, we’ll take a deeper look at several of these techniques and explore additional optimization strategies, including NVFP4 quantization for NVIDIA Blackwell architectures and high-resolution inference techniques such as SAHI.

Final Thoughts on Model’s Performance

Throughout this article, we reduced the memory footprint of a YOLO11 model from its original PyTorch implementation to an optimized TensorRT INT8 deployment while simultaneously improving inference latency. More importantly, we learned that there is no single optimization that magically solves memory problems. Instead, efficient deployment is the result of applying several complementary techniques, each addressing a different source of memory consumption.

When starting a new deployment, we recommend following this order:

1. Choose the smallest model that meets your accuracy requirements.

2. Move the model to TensorRT to benefit from graph optimization, memory reuse, and hardware-aware execution.

3. Enable FP16 inference, which provides substantial memory savings and speed improvements with virtually no accuracy loss on modern NVIDIA Jetson devices.

4. Use static input shapes whenever your application allows it.

5. Reduce the input resolution only after validating that your application still meets its accuracy requirements.

6. Apply structured pruning if further optimization is required, followed by fine-tuning to recover accuracy.

7. Finally, evaluate INT8 quantization for the largest memory and latency reductions, validating the resulting accuracy with a representative calibration dataset.

By following this progression, you can often reduce the memory footprint of an embedded AI application by 60–80% while simultaneously increasing inference speed, enabling larger applications, multiple concurrent models, or deployment on smaller Jetson platforms. This can be seen in the results from our example on Figure 15.

Figure 15. Total System Memory Reduction After Model Optimizations (YOLO11m)

The most important lesson, however, is that optimization should never be treated as a blind checklist. Every technique introduces a trade-off between memory, execution time, and model accuracy. Understanding where memory is consumed (weights, activations, workspace, and input/output buffers) allows you to choose the optimizations that provide the highest return with the lowest impact on your application.

Memory optimization is therefore not simply about making a model fit into a device; it is about designing AI systems that are efficient, scalable, and practical for real-world embedded deployment.

Need Help Optimizing Your AI Application on NVIDIA Jetson?

Reducing memory usage without sacrificing accuracy or real-time performance often requires more than applying individual optimization techniques, it requires understanding how the complete inference pipeline behaves on the target hardware.

At RidgeRun.ai, we help companies deploy and optimize AI and computer vision applications on NVIDIA Jetson, from model profiling and TensorRT optimization to quantization, pipeline acceleration, and production deployment.

If your model doesn’t fit your target device, isn’t meeting your latency requirements, or you want to get more performance from your NVIDIA Jetson platform, contact us at contactus@ridgerun.ai. Let’s find the right optimization strategy for your application.