Shortly after the release of Meta AI Llama 3, several options for local usage have become available. This article covers three open-source tools that let you run Llama 3 on Mac, Windows, and Linux locally without internet.

• Ollama

• Open WebUI

• LM Studio

Ollama

Platforms: Mac, Linux, Windows (Beta)

Ollama is a free open-source application that lets you use different large language models, including Llama 3, on your own machine, even if it's not the most powerful. Using enhancements from llama.cpp, an open-source library, Ollama allows you to run LLMs locally without needing high-end hardware. Additionally, it features a kind of package manager, making it possible to swiftly and efficiently download and deploy LLMs with just a single command.

To get started with the Ollama CLI, download the application from ollama.ai/download. It works on Mac, and Linux, with the Windows version currently in "preview" (a gentler term for beta).

Once installed, simply open your terminal. The command to run the model is the same across all platforms.

Run this in your terminal:

# download the 7B model (3.8 GB) 
ollama run llama3 

# or for specific versions

Then, you can start chatting with it:

ollama run llama3 
>>> hi Hello! How can I help you today

Open WebUI with Docker

Platforms: Mac, Linux, Windows

Open WebUI offers a flexible, self-hosted user interface that operates fully within Docker. It works with Ollama as well as other OpenAI compatible large language models (LLMs), such as LiteLLM or customized OpenAI APIs.

Docker Desktop makes it easy to install on Mac, Linux, or Windows systems, letting you run apps effortlessly.

If you've already set up Docker and Ollama on your PC, getting started is straightforward.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Then just go to http://localhost:3000, set up an account, and begin chatting!

Note: If this is your first time using Docker, you will need to download the Llama 3 models. To do this, simply click on the settings icon after selecting your name at the bottom left corner of the screen. Then, select "models" from the left side of the pop-up window and enter a model name from the Ollama registry to begin downloading.

You can choose between a wide range of models, including Llama 3, Lama 2, Mistral, and others.

LM Studio

Platforms: Mac, Linux (Beta), Windows

LM Studio uses llama.cpp to handle models like ggml Llama, MPT, and StarCoder from Hugging Face, allowing fine tuning of generative AI models.

Steps:
1. Download LM Studio from its website and install.
2. Download the Llama 3 8B Instruct model.

LM Studio has a chat interface built into it to help users interact better with generative AI.

Each method lets you download Llama 3 and run the model on your PC or Mac locally in different ways. Choose Meta AI, Open WebUI, or LM Studio to run Llama 3 based on your tech skills and needs. Just follow the steps and use the tools provided to start using Meta Llama effectively without an internet connection.

FAQ

• Can I run Llama 2 locally? Yes, besides Llama 3, you can also run Llama 2 locally using similar tools like Ollama or Open WebUI. You will have to download the Llama 2 model and configure it following similar steps. Read the article related to Llama 2.

• What are the benefits of fine-tuning Llama 3 models locally? Fine-tuning Llama 3 models locally ensure that you have control over your data and can tailor the model for specific tasks without relying on an internet connection.

• Is an internet connection necessary to use Meta AI Llama 3 models? Running the models locally doesn't require a continuous internet connection, making it a perfect solution for offline use. However, you will need internet access initially to download the models.

Related articles

– How to run llama 2 locally

– Running Mixtral 8x7 locally with LlamaIndex and Ollama

– How to Run Code Llama (13B/70B) on Mac