How to Run a Local LLM: A Guide to RAG Using the OnLogic Karbon 804

Generative AI, a branch of machine learning that leverages large language models (LLMs), is quickly becoming one of the most impactful categories of AI tools and has the potential to add tremendous value to organizations of every size and across industries. Popular examples like ChatGPT, based on OpenAI’s GPT models, have demonstrated the power of these models for various tasks, but cloud-based LLM solutions can pose challenges for data security, latency, and offline implementations.

This article demonstrates how to overcome these limitations by implementing an on-device, local LLM using Retrieval-Augmented Generation (RAG). This approach brings the power of AI directly to your edge deployments through local models, enabling practical use cases in areas like manufacturing, field service and remote monitoring. By leveraging a Llama 3.1 model and a local context database, this solution enables secure access to proprietary data and provides quick, accurate responses, even without an internet connection. See how to bring powerful AI capabilities to your edge deployments using on-premise hardware.

Unlocking on-device intelligence: Why local LLMs matter

Most LLM interactions are processed in the cloud, computed in data centers far from the source of the data, and simply retrieved on a local device. This strategy can be useful for AI-generated content requiring generalized knowledge or centralized access, and can be leveraged by most devices with an internet connection. But for organizations interested in leveraging the power of AI using their own proprietary data, these cloud-based solutions fall short when it comes to data security, and can quickly make cloud-computing costs untenable.

To minimize latency and overcome data-sharing limitations like geography or data sensitivity, moving compute to the edge can be an optimal solution for running LLMs. For instance, when an LLM requires access to sensitive, proprietary data, such as health records or legal information, deploying the model on an edge device offers enhanced data security compared to transmitting it over the internet. Edge deployments unlock a variety of use cases where cloud-based LLMs fall short—especially when reliability, latency, or data privacy are critical. Furthermore, LLMs may require deployment in areas without adequate connectivity, like remote monitoring or manufacturing locations that lack reliable connections to cloud providers. For more information about the value and importance of AI at the edge, please refer to our article: Edge AI Architecture: The Edge is the Place to be for AI .

In this article, we describe the process to implement an on-device AI chatbot for answering questions about its supplied data, in this case information about OnLogic products. Since this scenario requires the AI to have specific, specialized knowledge, we’ll use a concept called RAG to ensure that the model has access to accurate contextual information to help answer each question.

Model selection

Running LLMs locally may seem intimidating at first, but there are a variety of free and open-source tools available to streamline the development process. While powerful proprietary models exist, such as those offered by OpenAI or Microsoft's Azure AI, our focus here is on demonstrating a robust, locally hosted solution using open-source tools for enhanced control and data privacy. The first step in developing a custom AI solution is understanding your performance objectives. For the purposes of our product information chatbot our AI model needs to meet the criteria listed below.

The LLM should:

1. Run locally

2. Be a widely available LLM with open or permissive licensing, such as Llama 3.1, Mistral, Gemma, Mixtral, or DeepSeek

3. Respond to questions quickly and accurately, including:

   1. Begin delivering responses in under a second.

   2. Output text faster than the average reading words per minute

4. Not provide information outside of approved data sources, in this case product manuals 

Based on these goals we've chosen to utilize the 8 billion parameter version of Llama 3.1, a state of the art and open-source language model developed by Meta. Llama 3.1 is the latest evolution of Meta’s large language models, offering substantial improvements in reasoning and tool use over earlier versions like Llama 2. Models like Llama 3.1 can be efficiently run on local hardware using lightweight inference libraries such as llama.cpp which is specifically optimized for CPU performance and low memory environments. While powerful proprietary models exist, such as those offered by OpenAI’s GPT-4, our focus here is on demonstrating a robust, locally hosted solution using open-source tools for enhanced control and data privacy. Like many prominent open-source models, Llama 3 is readily available on the Hugging Face Hub, you can download any LLM with the .gguf extension and run it in the terminal, making it easily accessible for local deployment. Llama 3.1 outperforms similarly sized open-source models in a variety of LLM benchmarks, specifically in reasoning and tool use categories. 

We're confident in using a smaller model because our RAG scenario involves an LLM performing a specialized task focused on our own input data, so our model doesn’t need to be as large since we don’t require vast, generalized knowledge. For organizations with highly specific domain knowledge or unique interaction patterns, fine-tuning Llama 3 on proprietary datasets could further enhance its performance and relevance.

Hardware configuration

With the LLM selected, we can move on to configuring an edge computer that can run the model effectively. When it comes to Large Language Models (the key word being "large"), memory capacity can quickly become a bottleneck for execution - so we’ll need to ensure that our configuration can handle the model with some room for memory overhead. To do this, we’ll use this GPU memory equation from Sam Stoelinga.

M = (P * 4B)/(32 / Q)*1.2

Here's a breakdown of each of the elements in the equation.

Our Llama model uses Q4_K_M quantization (4-bit), and has 8 billion parameters, giving:

M = (8 * 4B)/(32 / 4) * 1.2 = 4.8 GB

With that math done, we know we’ll have plenty of memory overhead on most dedicated GPUs, even when accounting for the extra space required for the context database and embedding model. We'll use the OnLogic Karbon 804 with an Nvidia A4000 GPU and a high-performance CPU  to run the model, which gives us plenty of processing power to handle our LLM. It will also provide the reliability to run our model wherever it's needed.  

The OnLogic Karbon 804 was the ideal choice to run our local LLM.

The OnLogic K804 is a premier rugged edge device for AI applications. It can operate in temperatures ranging from -40°C to 70°C, and is equipped to handle a variety of other harsh conditions, like humidity, shock and vibration. Most importantly for us, it has configurations that provide the processing power for LLM computation and the space for up to two single-slot GPUs.  

Retrieval-augmented generation

The next component of our AI solution is to ensure that the chat bot provides only data from its provided sources. Like we mentioned in the introduction, we’ll use an AI concept that is incredibly useful for edge computing called Retrieval-Augmented Generation (RAG).

RAG is important because LLMs have the unfortunate habit of writing text that “sounds” right, but isn’t grounded in truth. If the model does not have specific training on the information the user needs to retrieve, there is a high risk that the user will receive a hallucinated response. The RAG technique ensures that the model informs its response to the user with a database of accepted context sources, which in our case, will be OnLogic product manuals.

Our system utilizes two key models. First, a text-embedding model analyzes the input prompt, comparing it semantically to our context sources. This allows us to identify and retrieve the most relevant sources, which are then provided, along with the original prompt, to the main Large Language Model (LLM) — in our case, Llama. From there, the LLM model (in this case,  Llama 3.1) then generates the final text output.

The flow of data for an LLM using RAG. The user’s query is used to find relevant context from a database, and the query and context are both provided to the model.

Sample solution

The primary tools that power our sample RAG implementation will be Ollama and LangChain. Before diving into the code, it's important to set up the required dependencies.

• Ollama is an easy-to-use, open-source application for loading and running models locally on Linux, macOS, and Windows. For users who prefer a graphical interface, tools like LM Studio offer a desktop-based experience for running local LLMs, including models like Llama 3, Mistral, and DeepSeek, without needing to use the command line. Ollama also supports Docker-based deployment, allowing you to containerize your local LLM runtime for easy distribution across different environments.While the example demonstrates using the command line, the fundamental concepts and the API it exposes are applicable across different operating systems, including Windows, though specific installation and usage steps might vary. From Ollama you can  download the two necessary models by their model name from the command line with:

  • ollama pull llama3.1:8b

  • ollama pull nomic-embed-text

• LangChain is a powerful framework for building LLM pipelines. It provides Python packages that simplify the integration with APIs for various models, including Ollama, making it easier to execute and manage your local LLM workflows.

Full code for the solution can be found on GitHub.

The first step of our solution will be parsing through our PDFs to create our context database to get them into a format an LLM can understand. We use the `langchain_unstructured` library to parse the PDFs, which allows us to collect the text of each document while keeping some of the structure intact.  When Ollama runs a model, it serves it through a local API accessible via localhost, typically at http://localhost:11434. This allows tools like LangChain to connect to the model for prompt execution and chaining.

Our command to load the PDFs looks like this:

Python
for file in pdf_filepaths:
   loader = UnstructuredLoader(
	file_path=file,
	strategy="fast",
	chunking_strategy="by_title"
   )

   for doc in loader.lazy_load():
	docs.append(doc)

We use the “fast” strategy as opposed to "high resolution" to quickly load in the PDFs, and the “by_title” chunking strategy to ensure that our PDFs don’t split into too many sections, which would bloat the context database. Creating a loader for each of our PDFs gives us a final collection of 207 documents,  which contain the raw text content of each section of the PDF, and useful metadata like element category and document source. 

Then, with our collection of documents, we can create an embedding database using Langchain Chroma. Our text-embedding model, nomic-embed-text, will take the text from each document and calculate an embedding, a highly dimensional vector which tries to categorize the text based on its content. Then, when we query the model, nomic-embed-text will create another embedding vector for the query, which it will then compare to the vectors stored in the database. The database will return the most similar vectors, and provide the text for each vector as context to the main LLM. The code for this process is incredibly simple when using Chroma and Ollama together.

Python
local_embeddings = OllamaEmbeddings(model=”nomic-embed-text”)
vectorstore = Chroma.from_documents(docs, local_embeddings) 

That’s it! We can test out our context database with an example query, and see what the most similar document is:

Python
query = “What are the DisplayPort details for the k800 series?”

docs = vectorstore.similarity_search(query)
docs[0].page_content

>>> "2.9 - DisplayPort\n\nTher are two full-size DisplayPorts on the Karbon 800 series. Both ports support DP 1.4 at 4k 50Hz\n\nand support MST (Multi Stream Topology). An MST hub can be used to support up to four\n\nindependent displays. Please refer to Intel documentation for additional Alder Lake-S display output\n\nspecifications: https://ark.intel.com\n\n22"

The database was able to find the correct section on DisplayPorts within the K800 manual! Next we need to put it all together, which is again made very easy through LangChain’s python library. We’ll load the Llama model and then create a chain, which describes our entire AI pipeline.

Python
model = ChatOllama(model="llama3.1:8b")	

RAG_TEMPLATE = """
You are an assistant for tech support question answering tasks. Use the following product manual sections to answer the question. If you don’t know the answer, say that you don’t know. Use under three sentences and keep the answer concise. 

<context>
{context}
</context>

Answer the following question: 

{question}
"""

rag_prompt = ChatPromptTemplate.from_template(RAG_TEMPLATE)
retriever = vectorstore.as_retriever()

chain = (

	{"context": retriever | format_docs, "question": RunnablePassthrough()}
	| rag_prompt
	| model
)

Now, querying the entire model is as easy as calling one function. 

Python
question = "For the K800 series of computers, what are the details of their CE compliance?"
response = chian.invoke(question)
print(response.content)

>>> The K800 series was evaluated for medical, IT equipment, automotive, maritime, and railway EMC standards as a class A device. It complies with the relevant IT equipment directives for the CE mark. Testing includes EN55032, EN55035, and other specified standards. 

Our response also includes some useful metadata to evaluate its speed. We calculated that the time to first token was .27 seconds, and the output tokens per second was 68.9, which translates to about 3,100 WPM. So, our implementation smashes our speed requirements, while being able to provide an accurate answer directly from the product manual.  

Future work

There are many ways to potentially improve our LLM solution beyond our little Jupyter Notebook, and we don’t need to move past open-source tools to get there. Open Web UI could provide a much more visually appealing user interface for querying. These interfaces typically communicate with the API exposed by the local LLM runtime, like Ollama, to send prompts and receive responses. We could also continue to improve the quality and speed of our results through further model selection or optimization (quantizing, pruning, etc.), by exploring techniques like fine-tuning the Llama 3 model on our specific product documentation, and by improving our LangChain pipeline to include model self-correction or web search. 

There’s also room to dial in our hardware selection once the software solution is finalized. Ollama only used around 7.5 GB of GPU memory in total to run our example, about half of the dedicated VRAM from the A4000. Provided we consider the LLM output to be satisfactory, downsizing the GPU could best match the hardware to the workload.

Concluding thoughts

This sample project shows how easy it can be to set up a locally run Large Language Model with Retrieval-Augmented Generation to ensure high quality, context supported AI chatbot responses. All of the tools used are open source and well documented, which lets us tailor our solution to our goals and available resources. Combined with OnLogic’s edge compute hardware, which can be deployed wherever the model needs to be run, even environments that would cause more standard computer hardware to fail (like typical desktop CPUs or consumer-grade GPUs), the use cases of this edge AI framework are vast. Just think of a remote mine with no wireless connectivity that needs to troubleshoot machinery that failed, or a service technician on-site at a customer location who needs to root cause equipment failures, access to this type of LLM would be invaluable.

If you're interested in exploring how on-site AI could benefit your organization, reach out to our AI Consulting team.