Loading...
Loading...
Loading...
AIMay 6, 20257 MIN READ

Secure Edge AI: The Role of Private AI and On-Premise Infrastructure

An image of an edge server being used on a busy production

As Artificial Intelligence (AI) capabilities grow and enterprise use of AI continues to scale, many organizations are concerned about data privacy and security implications. AI solutions are exposed to confidential data sources such as customer information, operational data, and other intellectual property. Furthermore, these applications will interact with mission-critical systems and networks, making this security challenge a top priority. 

As a part of a large security plan, Private AI and On-Premise AI infrastructure are tools to build secure edge AI at scale. 

Private AI at the edge

Private AI consists of AI models and applications designed for a specific use-case with proprietary training data that tailors it to the end goals of that user. This is different from Public AI such as OpenAI's ChatGPT that is accessible to anyone in a multi-tenant public cloud environment. While users can fine-tune these types of public AI applications, training and inference data is generally retained by the service provider. 

In private AI deployments, the user maintains training and inferencing data along with the supporting infrastructure in their own privileged networks. 

Learn more about how to repatriate workloads from the cloud to the edge with our free white paper “How Cloud Repatriation is Shaping the Future of Business.” 

Get the White Paper

The infrastructure to support private AI can be deployed on-premise or private cloud depending on the application needs. Where and how data is handled will be dictated by data regulations and industry best practices. For example, the Health Insurance Portability Accountability Act (HIPAA) in the United States dictates standards around how healthcare providers can use patient data known as protected health information (PHI).

Private AI does not mean that applications must be completely disconnected from public cloud applications, but any external connections must serve only what is needed. For example, a time-series database could establish an estimated wait-time for traffic monitoring, but only the “estimated current wait” and “road name” fields need to be served to an external API.

On-premise AI infrastructure

To support secure edge AI applications it is important to understand what types of hardware and software will be required. This breaks down into the GPU computing needed to handle model training and inferencing, and the IT infrastructure to support AI applications via model development and orchestration.

High performance computing

OnLogic Axial AX300 Series 3U Edge Server

Training and Inferencing will require high performance systems with large amounts of GPU memory, also known as VRAM. AI model training is an intensive process so usually organizations will deploy GPU edge servers in at least a 3-node edge cluster with multiple GPUs per server, however this will depend on model complexity. Once models are ready from production, the model can be retrained at a slower speed to further improve the model based on inference results. 

Some organizations may choose to use the training cluster to also handle inferencing. However, this can only happen if the inference is happening centrally, for example in a logistics warehouse if all inference cameras are connected through high speed network switches. For non-CV (computer vision) use-cases central inference makes sense as manufacturing process data such as applications such as historian databases and will already have network connectivity.

However, many edge AI applications will distribute inference to industrial edge computing devices with lower compute power. Runtime optimized models such as inferencing an Intel OpenVINO optimized model against a neural processing unit (NPU) equipped industrial PC or a model optimized for Nvidia Jetson systems are a crucial step for edge AI. In that case, the central training infrastructure can be tailored to focus on solely training and model serving. The decision to distribute inferencing may also be dictated by latency or safety requirements for example, autonomous robotics that need to make real-time decisions.

Two diagrams side by side showing architecture differences between a distributed inference model and a central inference model.

IT Infrastructure

For on-premise AI infrastructure some key components are:

  • Secure DevOps: Creating a secure development environment for application developers and data scientists to create and train models. Whether this is hosted on-premise or on a private cloud will depend on the security restrictions for the data and applications. Just like any edge application, a secure infrastructure is paramount. Employing an experienced security advisor might be the best approach. 

  • Model Orchestration: Serving models to the right edge nodes, routing inference results, monitoring model drift, and managing training/retraining are all roles of the AI orchestration platform. Packaging models with their dependencies in a modern container-based architecture is extremely common for edge AI and many platforms have a preferred packaging method.

While some organizations will decide to use their existing IT infrastructure to create and deploy AI applications, there are a number of end-to-end AI orchestration platforms that are tailored with model specific capabilities. OnLogic has built a partner ecosystem for such AI platforms including Guise.Ai EdgeOps edge AI platform, Viso.Ai Viso Suite for computer vision models, and Red Hat Openshift AI for enterprise AI management.

When to integrate private cloud into edge AI applications

While inferencing on-premise is almost always preferential for edge AI models, there are some functions that can be hosted in an enterprise private cloud. The decision between private cloud and on-premise is not a binary choice, depending on the application requirements some functions, such as training and developer environments, may necessitate hosting in a central private cloud. This can be broken down across 5 key areas:

  • Data sovereignty: Are inference results subject to confidentiality or regulations where it can not be transmitted off-site? This would require all infrastructure to be hosted on-site.

  • Site security: Will the AI applications need to be integrated into an air-gapped facility? If the answer is yes, it will make sense to host all AI infrastructure and computational resources on-premise. If no external connections can be made, then DevOps will also need to be hosted on-premise.

  • Training datasets: Will the training data come from one or multiple locations? If the answer is multiple, collecting that training data and developing centrally will make sense.

  • Distributed edge nodes: Will the models be deployed to a large number of geographically separate locations? It may make sense to handle training and model development centrally if data is collected from a large number of locations. Then publishing models to a remote container registry where they can then be served to edge locations for inferencing.

  • Application interconnect: Will this application directly interconnect with mission-critical systems?

These concepts can be illustrated with two examples: 

Use-case: MRI tumor detection

A person having an MRI exam

A medical OEM sells an FDA approved AI-enabled MRI machine for tumor detection. Inference occurs on an AI computer integrated into the MRI machine. They offer this as a managed service for their customers which includes the radiology department of hospitals around the United States.

  • Data sovereignty: Inference results would be attached to the patient electronic health records (EHR) into a medical records platform such as EPIC. It is the hospital's responsibility to maintain patient privacy of these records.

  • Site security: These are not air-gapped facilities.

  • Training datasets: This detection model is trained on large public data about tumors. As long as health information is de-identified of patient identifiers, inference results can be used without restriction for retraining to improve the model.

  • Distributed edge nodes: Models need to be served to many hospitals around the country and updated regularly.

  • Application interconnect: By manufacturer design, the MRI scanner is not interconnected with life-support systems on other areas of the hospital network.

Based on the above assessment, inference would be handled on-premise while the DevOps environment, training, and model orchestration would be hosted on the OEM’s private cloud. For healthcare use-cases in the United States, it is important to understand HIPAA implications for AI.

Use-case: aerospace defense manufacturing

A machine vision defect detection solution inspecting a PCB circuit board

A defense aerospace manufacturer is looking to develop a defect detection model for top secret electro-optical sensors manufacturing. The AI application will directly interact with the manufacturing line control systems to reject parts.

  • Data sovereignty: No training or inference data can not be transmitted off-site in any capacity

  • Site security: The location is air-gapped and requires security clearance for entry to ensure unauthorized access.

  • Training datasets: All training data will come from the five production lines.

  • Distributed edge nodes: There are five manufacturing lines spread across the facility

  • Application interconnect: The model directly interfaces with mission-critical process control systems.

For this company all AI infrastructure will need to be deployed on-site. It is important to understand that in use-cases where DevOps needs to occur on-site, as in this example, this does mean developers will need to be located directly at those facilities whether temporarily or permanently.

Conclusion: Where you place resources matters

When it comes to deploying AI at the edge, mitigating security risks by building and maintaining secure edge AI applications with the right mix of on-premise and private cloud infrastructure is critical. Likewise, it is important to choose the right hardware for edge applications. From high performance training servers to ruggedized inferencing nodes, OnLogic's line of edge AI hardware and expertise can help supercharge your organization's AI initiatives.

Explore OnLogic Edge AI Solutions

Related links:

What is Edge AI

Edge AI Architecture: The Edge is the Place to be for AI

Cloud Repatriation is Shaping the Future of Business

O
OnLogic
OnLogic
OnLogic is a global industrial computer manufacturer that designs highly-configurable, solution-focused computers engineered for reliability for the IoT edge.

Stay in the know.

Get first access to the newest products and Industry 4.0 innovation, OnLogic events and more.