# Running Llamafile on RP5

> experiment running Llamafile on Raspberry Pi 5

URL: [Original article](https://mattcool.tech/posts/running-llamafile-on-rp5)

Published: 2024-08-11T21:10:00.000Z

Tags: notes, ai

I wanted to see if I could run [llamafile](https://github.com/Mozilla-Ocho/llamafile), which lets you distribute and run LLMs with a single file, on a Raspberry Pi 5. I hope I can use my own custom front-end to interact with the model in different ways. I have a few ideas but want to see if I can get it running first.

**Is it fast?**

Yes, using a small model is no problem. Generates fast enough to be usable but slower than popular commercial chatbots at this time.

**How good are the responses?**

Not great in my tests but also not bad in some responses. I would like to experiment more to see what can be done with this.

## Notes

Here are the steps I took to get it working, welcome any tips as I am new to many aspects of this:

## 1: Pi OS setup

I used [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to easily install Raspberry Pi OS lite, you can then ssh into, then updated.

Caption:

```bash
sudo apt update; sudo apt full-upgrade -y
```

## 2: Install Required Packages

Couldn't get things running without the virtual env. I installed the following packages:

```bash
build-essential, wget, curl, unzip, and python3-venv
sudo apt install build-essential wget curl unzip python3-venv -y
```

## 3: Create a Virtual Environment

Create and activate a virtual environment.

```bash
python3 -m venv myenv
source myenv/bin/activate
```

## 4: Install Hugging Face CLI

Install the Hugging Face CLI within the virtual environment:

```bash
pip install -U huggingface_hub[cli]
```

## 5: Login to Hugging Face

Log in to Hugging Face using the CLI:

```bash
huggingface-cli login
```

Enter your read access token when prompted.

## 6: Download the TinyLlama Model

Try whatever model you would like. First I tried a larger one and it was too slow. TinyLlama runs quick on RP5.

```bash
wget --header="Authorization: Bearer <YOUR_HUGGINGFACE_TOKEN>" -O TinyLlama-1.1B-Chat-v1.0.Q8_0.llamafile https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-llamafile/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q8_0.llamafile
```

You can get a access token from your Hugging Face account settings. Use the read token for this purpose. You wont need to add to git credentials.

## 7: Grant Execution Permissions
Make the downloaded LlamaFile executable:

```bash
chmod +x TinyLlama-1.1B-Chat-v1.0.Q8_0.llamafile
```

## 8: Run the LlamaFile Server

Start the LlamaFile server

```bash
./TinyLlama-1.1B-Chat-v1.0.Q8_0.llamafile --host 0.0.0.0 --no-browser
```

## 9: Pi should be running a webserver

Open a web browser on another device connected to the same local network and navigate to:

```bash
http://<your_raspberry_pi_ip>:8080
```