Loading...
Loading...
Loading...
AI5 September 20259 MIN READ

Right Sizing Your Edge AI Hardware

A Guide to Optimizing Computer Vision Deployments
A collection of industrial hardware from OnLogic

Navigating the dynamic landscape of Artificial Intelligence (AI) hardware presents a significant challenge. With a vast array of computing options and intricate software considerations, it can be difficult to confidently pinpoint the ideal hardware for your specific deployment. This often results in AI solutions operating on over-specified hardware that is not fully utilized, leading to unnecessary expenses that far exceed the cost of a right-sized AI hardware solution. This imbalance between cost and performance intensifies with larger deployments, potentially diminishing the overall return on investment for businesses as their AI initiatives grow.

This article explores the essential factors for selecting the precise hardware your AI application needs. You will gain valuable insights into optimizing performance, effectively managing costs, and ensuring the long-term viability of your AI infrastructure. Our goal is to empower you with the knowledge to make informed decisions that align with your project's unique demands, ultimately helping you make it possible to achieve your AI objectives.

Ready to start harnessing AI but not sure where to start? Connect with our dedicated AI Consulting team.

Speak with an Expert

Defining your edge AI vision: Starting with project goals

Before diving into the selection of edge hardware for any vision AI deployment, it's crucial to clearly define the application's goals. Critical performance characteristics typically include the AI model's latency and frame rate throughput. Inputs will generally involve factors like video streams and resolution. Your project objectives will often be measured by specific metrics such as:

  • Frame rate (fps): The number of frames processed per second, vital for real-time applications.

  • Accuracy: Ensuring the AI model reliably identifies its target – for instance, a car detection model accurately distinguishing cars from motorcycles, trucks, bicycles, or even horses.

  • Latency: The speed of processing and the associated delay within the application.

With these goals established, several variables can be adjusted to achieve the desired outcome. In the following sections, we'll specifically explore the impact of AI model size and the type of device on overall performance, all contributing to the effective right-sizing AI hardware for your unique needs.

A diagram of an edge AI model with data inputs, model sizes and inference device.

Edge AI models have deterministic output based on data inputs, model sizes, and the inference device.

Understanding model sizes and performance tradeoffs

When considering pretrained AI models, such as the five different sizes offered by Ultralytics for YOLO11 (You Only Look Once), it's important to understand the key characteristics that influence performance. These typically include:

  • Parameters: This metric indicates the model's overall size and complexity.

  • FLOPs (Floating Point Operations): FLOPs quantify the computational workload required for the model to perform its tasks.

  • mAP (Mean Average Precision): For object detection, mAP serves as a primary metric for accuracy, indicating how precisely the model identifies and localizes objects.

To illustrate these tradeoffs, consider the YOLO11n (nano) and YOLO11x (extra-large) models. The YOLO11n is the smallest, featuring 2.6 million parameters and utilizing 6.5 billion FLOPs. This makes it exceptionally fast for inference and requires minimal computational power. However, this efficiency comes with a compromise in accuracy.

Conversely, the YOLO11x is the largest model, with 56.9 million parameters and a demanding 194.9 billion FLOPs. While this represents a significant increase in computational requirements—nearly 30 times more FLOPs than the nano model—the corresponding improvement in accuracy is relatively modest. For example, the YOLO11n model achieves a mAP of 39.5, whereas the YOLO11x model reaches 54.7, representing an approximately 38% improvement. This demonstrates that while larger models offer higher accuracy, the gains often diminish relative to the considerable increase in computational demand and potential impact on processing speed. This understanding is key to right-sizing AI hardware effectively.

To further illustrate these tradeoffs, the table below details the characteristics and performance of the various YOLO11 model variants:

A table comparing YOLO Model Sizes vs. Speed and Accuracy

A comparison of YOLO model sizes vs. speed and accuracy.

Understanding frame rates with a discrete GPU

For many demanding edge AI deployments, a discrete GPU (dGPU) like NVIDIA RTX 4000 Ada SFF GPU is often considered a benchmark for performance. To evaluate the capabilities of such hardware across different AI model sizes, OnLogic conducted an experiment using various device types.

Our testing focused on inference results for various YOLO11 models, leveraging NVIDIA TensorRT for optimization with a single 640 x 640 video stream. The data revealed impressive frame rates: the nano model achieved 954 frames per second (fps), while even the extra-large model delivered 167 fps. While these results were obtained using a single, low-resolution video stream, this setup accurately represents many common computer vision use cases at the edge.

These compelling figures prompt a critical question for many project managers and engineers: Is a high-performance discrete GPU truly necessary for your specific application, particularly when this single component can add a substantial cost of approximately $1,500-$2,000 to a single node? This inquiry underscores the importance of right-sizing AI hardware to avoid unnecessary expenditure and ensure optimal system design.

NVIDIA RTX 4000 ADA Frame Rate and Latency Performance Across Model Size, all exceeding 100 fps

NVIDIA RTX 4000 ADA frame rate and latency performance across model size, all exceeding 100 fps.

Right-sizing for edge AI: Achieving optimal performance without overspending

Many of our customers aim for approximately 30 frames per second (fps) as their performance target for various edge AI applications. To demonstrate how to achieve this efficiently, OnLogic conducted further tests focusing on YOLO11 small (s), medium (m), and large (l) models, all run on our Helix 520 Series of industrial computers.

Our findings reveal that for a single video stream application, leveraging Intel's Core™ Ultra Series 1 (Meteor Lake) and Core Ultra Series 2 (Arrow Lake) NPU with OpenVINO™ allows the system to comfortably meet or exceed this 30 fps goal without the need for a discrete accelerator. In fact, running the model on the system's integrated GPU (iGPU) can easily exceed this performance requirement. For applications requiring internal CPU device utilization, Intel's Arrow Lake CPUs demonstrate exceptional results, achieving 76 fps with the NPU and an impressive 108 fps with the iGPU.

Chart showing Devices and Models that Achieve between 30 and 240 Frames Per second. For smaller models, integrated hardware achieves the goals without discrete GPUs

Devices and models that achieve between 30 and 240 frames per second. For smaller models, integrated hardware achieves the goals without discrete GPUs.

These results highlight a crucial insight: For many single video stream applications, a discrete GPU is often unnecessary and likely represents an over-provisioned solution. The choice between utilizing the NPU or iGPU then becomes a matter of fine-tuning for specific factors like latency and power draw. Furthermore, our tests indicate that the integrated GPU on the Intel CPU package can even outperform dedicated AI inference accelerators, such as the Hailo-8 we included in our testing.

However, when a larger AI model is required for more demanding or higher-performance applications, discrete GPUs (dGPUs) unequivocally lead the way. For scenarios targeting inference results greater than 120 fps with medium, large, or extra-large models, the power of a dGPU becomes evident. Our data shows that the NVIDIA RTX 2000 performs nearly identically to the A4500 MXM form factor in these high-performance scenarios.

Here, hardware selection becomes critical, necessitating a careful evaluation of tradeoffs between price and performance. A primary distinction of the MXM form factor is its higher temperature rating and ingress isolation, making it more resilient for performance in adverse environmental conditions compared to the RTX 2000. Conversely, a system utilizing the RTX 2000 would typically offer a price advantage. Ultimately, for single video stream applications where performance goals reside around 30 fps, integrated NPUs and iGPUs are generally sufficient, solidifying the importance of right-sizing AI hardware to match your project's precise demands.

Chart showing Discrete GPU Performance: MXM vs. Traditional

Top end MXM style GPUs are quite comparable to RTX 2000 ADA; best utilized in larger inference models in edge AI applications where operating conditions may be adverse.

From prototyping to scale: Achieving cost-effective AI deployment

Considering the wide range of performance outcomes achievable with different AI models, inputs, accuracy targets, and devices, it is paramount to approach AI deployment in a fiscally responsible manner. For this reason, commencing application development with a system featuring a discrete GPU (dGPU) can be advantageous. This setup allows engineers to develop, quantize, and optimize their models, and then confidently experiment with targeting various devices to benchmark performance.

For instance, if an engineer validates their application on a Helix 520 Series system equipped with an NVIDIA RTX 4000 Ada GPU, it becomes essential to understand how the results align with the application's ultimate goals. This evaluation can lead to two primary scenarios:

  • If the frame rate is significantly greater than the goal:

    • Consider deploying the application to a less powerful, more cost-effective device, such as an integrated GPU (iGPU) or Neural Processing Unit (NPU).

    • Alternatively, you might increase the model size to achieve greater accuracy, leveraging the available performance headroom.

  • If the frame rate is significantly lower than the goal:

    • Confirm deployment on a discrete GPU to meet the performance requirements.

    • Evaluate reducing the model size and validating that any resulting change in accuracy remains acceptable for the application.

Generally, leveraging smaller, more optimized models translates directly into less expensive hardware requirements. As AI applications transition from the prototype phase to pilot deployments and eventually full-scale rollout, the escalating system count can lead to dramatically increased costs if the AI hardware is not thoughtfully selected. For example, consider an application aiming for 30 fps that unnecessarily utilizes a discrete GPU when an integrated NPU would suffice. Across a 1,000-device deployment, this seemingly small oversight can result in a staggering $1.5 million difference in hardware expenditure alone, not to mention substantial long-term energy savings.

Table indicating Frame rate averages across devices and Models - In general, smaller models (yolov11m and below) can effectively utilize on-board NPUs or iGPUs where larger models (yolo11l and above) do better with discrete GPUs (either MXM or Traditional.

Frame rate averages across devices and models. In general, smaller models (YOLO11m and below) can effectively utilize on-board NPUs or iGPUs where larger models (YOLO11l and above) do better with discrete GPUs (either MXM or traditional.

In conclusion, the selection of hardware for AI applications is a significant responsibility that warrants careful consideration. Discrete GPUs are powerful tools, but they are not a one-size-fits-all solution for every problem. Instead, an appropriate solution, meticulously aligned with your project's performance goals, should be properly sized and thoroughly assessed. This critical choice can unlock immense hardware savings and prevent projects from being derailed by prohibitively high deployment costs, ultimately helping to make it possible for your AI initiatives to thrive at scale through effective right-sizing AI hardware.

Ready to see how the right hardware can transform your AI implementations, but not sure where to start? Connect with our dedicated AI Consulting team to discuss your project.

Speak with an Expert

Notes on Testing

  • Testing conducted internally by OnLogic utilizing OnLogic HX520 Series and HX600 Series products.

  • Testing conducted at ambient temperature.

  • Testing conducted utilizing Ultralytics YOLO11 detection models utilizing a single 640x640 video input.

  • All benchmarks ran at FP16 precision.

  • Throughput includes pre-processing time. Model throughput ignores it.

  • NPU does not include NMS in each inference pass.

  • All benchmarks ran using synchronous inference (1 frame processed at a time, as soon as the previous one completes).

Hunter Golden
Hunter Golden
Hunter Golden is a Senior Product Manager at OnLogic where he leads the strategic direction and growth of the industrial computing product line. With a passion for solving complex customer and industry challenges, Hunter has been instrumental in shaping the product roadmap, refining pricing strategies, and fostering new product innovations.

Stay in the know.

Get first access to the newest products and Industry 4.0 innovation, OnLogic Live! events and more.