Ollama.ai on Photon OS on WSL2

[DCasota]

Hi,

Lately I've tinkered around with Ollama.ai on Photon OS, also in a special use case with WSL2 in a Microsoft environment.

The provisioning of the NVidia GPU drivers on the Windows 11 test environment and installed WSL2  is straight forward.

The strange thing is that ollama-runner throws an issue  error while loading shared libraries: libstdc++.so.6 on Photon OS. Any idea?

The logfile is attached.

Photon OS on WSL2 recipe

# Open a Powershell Terminal (Administrator)

dism /Online /Enable-Feature /All /FeatureName:Microsoft-Hyper-V # reboot
dism /Online /Enable-Feature /All /FeatureName:Microsoft-Windows-Subsystem-Linux # reboot
bcdedit /set hypervisorlaunchtype auto # reboot

invoke-webrequest https://desktop.docker.com/win/main/amd64/Docker%20Desktop%20Installer.exe -outfile "$env:userprofile\Docker Desktop Installer.exe"
$oprocess = start-process "$env:userprofile\Docker Desktop Installer.exe" -passthru
# reboot

start-process "$env:ProgramFiles\Docker\Docker\Docker Desktop.exe"
Start-Sleep 30

docker pull photon:latest # docker pull photon release from https://hub.docker.com/_/photon/
docker run -it -d photon:latest # run container in background

# get docker container id
$ID = (docker container list --format "{{json .}}" | convertfrom-json | where-object -Property image -eq "photon:latest" | select ID).ID

docker export -o "$env:userprofile\photon.tar" $ID # docker export
docker container stop $ID # docker container stop and prune
docker container prune --force # docker container stop and prune

wsl --set-default-version 2 # set wsl version
wsl --import Ph5 "$env:userprofile" "$env:userprofile\photon.tar" # wsl import to a target directory e.g. $env:userprofile
wsl --set-default Ph5 # set wsl default
wsl # start

Ollama setup recipe:

tdnf install -y awk libstdc++
curl https://ollama.ai/install.sh | sh
ollama serve &
ollama run llama2

Thanks for good suggestions!
Daniel

[DCasota]

Okay it works now. The package libstdc++ wasn't installed. Added it to the post above.

tdnf install libstdc++

Some observations:

WSL2 is weired. It somehow uses the NVidia GPU drivers installed on Windows and interacts with the WSL2'ified Photon OS. From a security perspective I would say there are concerns.

Ollama speed is more than 20x faster than in CPU-only mode. This is great news.

With Photon OS on Workstation 17 Pro, the resources used are lower than in comparison to WSL2. Especially, the fan doesn't kick-in in idle mode. However actually it cannot make use of the NVidia adapter.

Photon OS on vSphere (or baremetal) provisioned still is the best fit for lowest resource consumption, security and speed.

edited:

Additional devices e.g. camera, microphone and audio are not necessary for Ollama. The following sections are more for tinkering purposes.

Photon OS can make use of an usb camera by configuring WSL2 with the helper tool usbipd.

# installation of usbipd
winget install usbipd --force
# this seems to be necessary for the usbip client tool
winget install --interactive --exact dorssel.usbipd-win

# update wsl
wsl --update

# reopen Powershell Terminal (Administrator)
# In Docker Desktop > Settings > Resources > WSL integration > Enable integration with additional distros: Ph5 > Apply & restart

# find busid
PS C:\Users\youruser> usbipd list
Connected:
BUSID VID:PID DEVICE STATE
2-4 04f2:b7c1 Integrated Camera, Integrated IR Camera, APP Mode Not shared
2-10 8087:0033 Intel(R) Wireless Bluetooth(R) Not shared
3-2 0bda:8153 Realtek USB GbE Family Controller Not shared
4-1 046d:c046 USB-Eingabegerät Not shared
4-3 046d:c316 USB-Eingabegerät Not shared

Persisted:
GUID DEVICE

# bind the preferred device by busid
usbipd bind -b 2-4
wsl
# enter exit

# attach the preferred device
usbipd attach --wsl -b 2-4

# start wsl
wsl

# install Photon OS usb utils
tdnf install -y usbutils

root@Ph5 [ /mnt/c/Users/youruser ]# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux 5.15.133.1-microsoft-standard-WSL2 vhci_hcd USB/IP Virtual Host Controller
Bus 001 Device 002: ID 04f2:b7c1 SunplusIT Inc Integrated Camera
Bus 001 Device 001: ID 1d6b:0002 Linux 5.15.133.1-microsoft-standard-WSL2 vhci_hcd USB/IP Virtual Host Controller

One culprit on WSL2 is the complexity of make use of audio. An audio device is missing.

I've tried to get it run by using a component called AudioPulse, but actually it does not work.

Good suggestions are welcome! I leave the code snippets here if someone is interested.

tdnf install -y pciutils
# WSL2 seems to make use of virtio driver bits.
# no audio device is listed. Hence, the following recipe does not work even if AudioPulse would be available.
lspci -k
00:00.0 3D controller: Microsoft Corporation Basic Render Driver (rev 00) dxgkrnl
00:00.0 System peripheral: Red Hat, Inc. Virtio file system (rev 01) virtio-pci
00:00.0 SCSI storage controller: Red Hat, Inc. Virtio console (rev 01) virtio-pci
00:00.0 3D controller: Microsoft Corporation Basic Render Driver (rev 00) dxgkrnl

# Correct time is needed as prerequisite.
# This snippet however is uncurated. ntpd does not work in WSL2 because systemd isn't the booted init system.
tdnf install -y ntp iptables tzdata
# add your servers to /etc/ntp.conf
iptables -A INPUT -i eth0 -p udp --dport 123 -j ACCEPT
iptables-save >/etc/systemd/scripts/ip4save
ip6tables-save >/etc/systemd/scripts/ip6save
systemctl enable ntpd
set <continent/town> timezone
ln -sf /usr/share/zoneinfo/<continent>/<town> /etc/localtime
ntpdate -s <your ntp server>
hwclock --systohc --localtime

# A few bits have to be compiled and downloaded. Hence, build-essential and pip is installed.
tdnf install -y build-essential python3-pip

# On WSL2 an audio device is missing.
# An option could be to use PulseAudio. PulseAudio uses sndfile, but the installation of sndfile fails.
pip3 install sndfile # THIS FAILS. The Linux distro isn't recognized and the headerfile sndfile.h is missing.
# As alternative, compiling from source https://github.com/libsndfile/libsndfile, fails, too.

# The following recipe does not work without sndfile.

tdnf install -y meson cmake
git clone https://github.com/pulseaudio/pulseaudio
cd pulseaudio/
meson build
meson compile -C build
build/src/daemon/pulseaudio -n -F build/src/daemon/default.pa -p $(pwd)/build/src/modules/
cd ..

# install google_speech. PulseAudio and libsox are prerequisites.
git clone https://github.com/dmkrepo/libsox
cd libsox/
autoreconf -i
./configure
make -s
make install
cd ..
pip3 install google_speech
google_speech "Hello world."