AMD Strix Halo Local LLM Setup: Ollama, llama.cpp, and BIOS Configuration

Why This Setup Matters

Running large language models locally requires careful attention to memory allocation. The AMD Strix Halo series integrates GPU and CPU memory into a unified pool. Your BIOS settings determine how much VRAM is available for LLM inference.

This guide covers the configuration needed to run 70B+ parameter models on a Strix Halo system with 128GB RAM.

Hardware Context

Target System:

Why This Configuration: The Strix Halo architecture uses true unified memory — the GPU and CPU share the same physical RAM pool. Unlike discrete GPUs with fixed VRAM, the AMD iGPU can dynamically access the full 128GB as needed.

The BIOS UMA Setting: The “UMA Frame Buffer Size” in BIOS only pre-allocates a portion of RAM for the GPU at boot time. It does not limit the GPU’s maximum accessible memory. With proper kernel parameters (iommu=pt, amd_iommu=aperture), the GPU can dynamically allocate from the full 128GB pool regardless of the BIOS setting.

Practical Impact:

This is why you can run 122B models without “CPU offloading” — the GPU accesses the full unified memory pool dynamically.

Step 1: BIOS Configuration

UMA Frame Buffer Size

The critical setting for LLM workloads is the UMA Frame Buffer Size (sometimes labeled “iGPU Memory” or “Integrated Graphics Memory”).

What to Change:

Why 2GB Works for Testing: A 2GB allocation allows you to run smaller models (7B-14B) entirely on GPU while leaving 126GB+ for system RAM. For larger models that exceed GPU VRAM, llama.cpp automatically offloads layers to system RAM. You still get functional inference—just slower.

Tradeoffs:

Recommendation: Start with 2GB. If you regularly run 70B models and have headroom, increase to 4GB.

Additional BIOS Settings

Disable Power Saving Features:

These settings prevent the CPU from downclocking during inference, which can cause stuttering in real-time applications.

Enable Resizable BAR:

This allows the CPU to access the full GPU memory space at once, improving data transfer speeds for llama.cpp.

Step 2: Boot Parameters (GRUB Configuration)

After BIOS changes, ensure Linux recognizes the memory layout correctly.

Edit GRUB Configuration

sudo nano /etc/default/grub

Add These Kernel Parameters

Find the GRUB_CMDLINE_LINUX_DEFAULT line and add:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt amd_iommu=aperture=256M"

Parameter Breakdown:

For Systems With Memory Hotplug Issues

Some Strix Halo systems report incorrect memory totals without this parameter:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt amd_iommu=aperture=256M memmap=128G$0"

Warning: The memmap parameter forces the kernel to recognize exactly 128GB. Only use this if your system reports less than installed RAM.

Update GRUB and Reboot

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Verify Memory After Boot:

# Check total memory
free -h

# Check GPU memory recognition
radeontop  # Requires root: sudo radeontop

# Check IOMMU status
dmesg | grep -i iommu

Step 3: Install llama.cpp

Clone and Build

# Install dependencies
sudo dnf install -y git cmake gcc gcc-c++ make

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build with GPU acceleration
make clean
LLAMA_CUDA=1 make -j$(nproc)

For AMD ROCm Support (Optional): If you want ROCm acceleration instead of CUDA compatibility layer:

# Install ROCm dependencies first
sudo dnf install -y rocm-dev hipblas rocblas

# Build with ROCm
make clean
LLAMA_HIPBLAS=1 make -j$(nproc)

Note: CUDA support via LLAMA_CUDA=1 works on AMD GPUs through the HIP compatibility layer. ROCm-native builds (LLAMA_HIPBLAS=1) offer better performance but require more setup.

Install System-Wide (Optional)

sudo make install

This installs binaries to /usr/local/bin, making llama-cli, llama-server, and other tools available globally.

Step 4: Install Ollama

Ollama provides a simpler interface for running models with automatic GPU detection and built-in model management.

Download and Install

curl -fsSL https://ollama.com/install.sh | sh

This script:

Enable and Start Service

# Enable service to start on boot
sudo systemctl enable ollama

# Start the service now
sudo systemctl start ollama

# Verify it's running
sudo systemctl status ollama

Configure Ollama (Optional)

Set Environment Variables:

Create or edit /etc/systemd/system/ollama.service.d/environment.conf:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_GPU=999"
Environment="OLLAMA_MAX_LOADED_MODELS=3"

Reload and Restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Parameter Breakdown:

Verify Installation

# Check version
ollama --version

# List installed models
ollama list

# Check service status
sudo systemctl status ollama

Pull and Run Models

# Pull a 7B model for testing
ollama pull llama3.2

# Pull a larger model (requires significant RAM)
ollama pull llama3.1:70b

# Pull Qwen models
ollama pull qwen2.5:7b
ollama pull qwen2.5:32b

Run Inference

Interactive Mode:

ollama run llama3.2

You’ll drop into an interactive REPL where you can chat with the model. Type /bye to exit.

API Mode:

# Generate text via API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "user", "content": "What is Linux?"}
  ],
  "stream": false
}'

Manage Ollama Models

# Show model details
ollama show llama3.2

# Copy a model
ollama cp llama3.2 my-custom-model

# Delete a model
ollama rm llama3.1:70b

# Create a custom Modelfile
echo "FROM llama3.2" > Modelfile
echo "PARAMETER temperature 0.7" >> Modelfile
ollama create my-llama3 -f Modelfile

Ollama vs llama.cpp

FeatureOllamallama.cpp
SetupOne-command installManual build
Model ManagementBuilt-in libraryManual downloads
APIOpenAI-compatiblellama.cpp native
GPU SupportAutomaticManual configuration
CustomizationLimitedFull control
Best ForQuick deploymentProduction tuning

Recommendation: Use Ollama for quick setup and testing. Use llama.cpp with systemd for production deployments requiring fine-tuned performance.

For complete Ollama documentation, see: https://github.com/ollama/ollama/blob/main/docs

Step 5: Configure Model Quantization

Not all models need full precision. Quantization reduces memory requirements with minimal quality loss.

Understanding Quantization Levels

QuantizationMemory (7B model)Memory (70B model)Quality
FP1614GB140GBFull
Q8_08GB80GB~99%
Q6_K6GB60GB~98%
Q4_K_M4GB40GB~95%
Q3_K_S3GB30GB~90%

For 128GB Systems:

Download Quantized Models

# Using ollama (automatic quantization)
ollama pull llama3.2:1b  # Smallest
ollama pull llama3.2:3b
ollama pull llama3.2:7b
ollama pull llama3.1:8b
ollama pull llama3.1:70b

# Using llama.cpp (manual GGUF files)
# Visit huggingface.co/TheBloke for quantized models

Step 6: Running Models with llama.cpp

Download the Model

For the Qwen3.5-122B-A10B model, download the quantized GGUF files:

# Create models directory
mkdir -p ~/models
cd ~/models

# Download the model (3-part GGUF, ~70GB total for Q4_K_XL)
wget https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf
wget https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Qwen3.5-122B-A10B-UD-Q4_K_XL-00002-of-00003.gguf
wget https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Qwen3.5-122B-A10B-UD-Q4_K_XL-00003-of-00003.gguf

# Download the multimodal projector (required for this model)
wget https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/mmproj-F16.gguf

Expected Sizes:

Basic Inference (Testing Only)

# Quick test before setting up systemd service
./llama-cli -m ~/models/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf \
  -p "What is quantum computing?" \
  -n 256 \
  -c 4096

Key Parameters:

GPU Offloading (For Testing)

# Offload all layers to GPU (fastest, for testing only)
# On Strix Halo, the GPU can access full 128GB dynamically
./llama-cli -m ~/models/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf \
  -p "Your prompt here" -ngl 999

# Note: On Strix Halo with unified memory, -ngl 999 uses the full memory pool
# No partial offloading needed - the GPU dynamically allocates from 128GB

How -ngl Works on Strix Halo:

For persistent operation, run llama-server as a systemd service. This is the recommended approach for production use.

Create the Service File:

sudo nano /etc/systemd/system/llama-server.service

Paste This Configuration:

[Unit]
Description=Llama.cpp ROCm Inference Engine (Strix Halo)
After=network.target

[Service]
Type=simple
User=james
WorkingDirectory=/home/james/llama.cpp
ExecStart=/home/james/llama.cpp/build/bin/llama-server \
  -m /home/james/models/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf \
  --mmproj /home/james/models/UD-Q4_K_XL/mmproj-F16.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 131072 \
  -b 2048 \
  -ub 512 \
  --parallel 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn on \
  --timeout 600 \
  --chat-template-kwargs '{"enable_thinking":true}'
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Customize These Paths:

Parameter Breakdown:

Enable and Start the Service:

# Reload systemd to recognize new service
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable llama-server

# Start the service now
sudo systemctl start llama-server

# Check status
sudo systemctl status llama-server

Manage the Service:

# View logs
sudo journalctl -u llama-server -f

# Restart service
sudo systemctl restart llama-server

# Stop service
sudo systemctl stop llama-server

# Check if running
systemctl is-active llama-server

Test the API:

# Simple health check
curl http://localhost:8080/health

# Generate text
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain quantum entanglement simply.",
    "n_predict": 512,
    "temperature": 0.7
  }'

# Chat completion (if supported)
curl http://localhost:8080/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is Linux?"}
    ],
    "max_tokens": 512
  }'

Troubleshooting the Service:

# View full logs
sudo journalctl -u llama-server --no-pager

# Check for OOM errors in logs
sudo journalctl -u llama-server | grep -i "out of memory"

# Verify port is listening
sudo ss -tlnp | grep 8080

# Check process status
ps aux | grep llama-server

Step 7: Performance Tuning

Thread Count Optimization

LLM inference scales with thread count up to a point. Test different values:

# Test with different thread counts
for threads in 8 12 16 20 24 28 32; do
  echo "Testing with $threads threads:"
  ./llama-cli -m model.gguf -p "Benchmark prompt" -t $threads -n 128 --verbose
done

Typical Results:

Memory Mapping

For large models, use memory mapping to avoid loading the entire file into RAM:

./llama-cli -m model.gguf -p "Prompt" --mlock  # Lock in RAM (faster but uses more)
./llama-cli -m model.gguf -p "Prompt" --no-mmap  # Load entirely (slower startup)

Recommendation: Use --mlock if you have enough RAM. It prevents swapping during inference.

Batch Size Tuning

./llama-cli -m model.gguf -p "Prompt" -b 512  # Default batch size
./llama-cli -m model.gguf -p "Prompt" -b 2048  # Larger batch (faster but more VRAM)

Increase -b until you hit VRAM limits, then reduce.

Troubleshooting

OOM Errors

Symptoms: Model fails to load, “out of memory” errors

Fixes:

  1. Reduce -ngl (fewer GPU layers)
  2. Use a more quantized model (Q4 instead of Q8)
  3. Increase UMA Frame Buffer in BIOS
  4. Close other GPU-intensive applications

Slow Inference

Symptoms: Less than 5 tokens/second on 7B model

Fixes:

  1. Verify GPU offloading: ./llama-cli -m model.gguf -p "test" -ngl 999
  2. Check thermal throttling: sensors (look for high temps)
  3. Disable power saving: Set CPU governor to performance
    sudo cpupower frequency-set -g performance
  4. Verify IOMMU settings: dmesg | grep iommu

Model Not Recognizing GPU

Symptoms: “No GPU found” or “falling back to CPU”

Fixes:

  1. Rebuild llama.cpp with correct flags
  2. Check ROCm/CUDA installation: rocminfo or nvidia-smi
  3. Verify kernel parameters: cat /proc/cmdline
  4. Update GPU drivers: sudo dnf upgrade --refresh

Memory Not Recognized

Symptoms: System reports 64GB instead of 128GB

Fixes:

  1. Add memmap=128G$0 to GRUB parameters
  2. Check BIOS memory settings
  3. Reseat RAM modules
  4. Update BIOS to latest version

Example: Running Qwen3.5-122B-A10B

For very large models like Qwen3.5-122B-A10B, use the systemd service configuration above. The model requires:

Optimization Tips:

API Usage Example:

Once the service is running, interact via API:

# Simple completion
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain quantum entanglement in simple terms.",
    "n_predict": 1024,
    "temperature": 0.7,
    "stop": ["</s>"]
  }'

# With thinking enabled (Qwen3.5 feature)
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Solve this math problem step by step.",
    "n_predict": 2048,
    "temperature": 0.5,
    "chat_template_kwargs": {"enable_thinking": true}
  }'

Monitor Resource Usage:

# Watch VRAM usage
watch -n 2 rocm-smi

# Check service logs
sudo journalctl -u llama-server -f

# Monitor memory pressure
htop  # Look for llama-server process

Next Steps

Once your system is configured:

  1. Benchmark different quantizations to find your quality/speed sweet spot
  2. Set up a model registry with your most-used models
  3. Explore Ollama modelfiles for custom model configurations
  4. Consider Docker containers for isolated model environments
  5. Monitor thermal performance during extended inference sessions

Resources

This guide was created for the LivingOnLinux YouTube series. The accompanying video covers unboxing, physical setup, and live demonstrations of these commands.