Install vLLM on RHEL 9 Linux

We’re using a derivative of RHEL 9 Linux being AlmaLinux 9, but this should work for RHEL 9 itself and others like Rocky Linux 9 etc.

First we need an up to date Python 3 version installed, anything older than Python 3.11 is ok and up to and including Python 3.13, we’re going for 3.12 here:

dnf install -y python3.12 python3.12-devel

# Create a virtual environment
python3 -m venv ~/venvs/vllm

# Activate the environment
source ~/venvs/vllm/bin/activate

Before you can start the install of vllm you need to ascertain your CUDA version level, which can be found from the output of nvidia-smi | grep CUDA
And you’ll see something like “CUDA UMD Version: 13.4“, so that corresponds to the variable “cuversion” in the following command being set to “134“

pip install --upgrade pip
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu134

This command takes some time to run, does lots of the heavy lifting of the vllm install.

Verify the vLLM install with the command:

python -c "import vllm; print(vllm.__version__)"

Configure vLLM by editing the /etc/systemd/system/vllm.service file to be:

[Unit]
Description=vLLM OpenAI Compatible API Server
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service

[Service]
Type=simple
User=your_username
Group=your_username
Environment variables for CUDA and Cache directories

Environment="HF_HOME=/home/your_username/.cache/huggingface"
Environment="HF_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
Environment="CUDA_VISIBLE_DEVICES=0"
Path to your virtual environment's python binary and the vLLM execution module

ExecStart=/home/your_username/venvs/vllm/bin/python3 -m vllm.entrypoints.openai.api_server
--model Qwen/Qwen2.5-7B-Instruct
--host 0.0.0.0
--port 8000

## Another model known to work is: ornith-ai/Ornith-1.5-9B

Restart=always
RestartSec=10
Security and resource tuning configurations for RHEL/AlmaLinux

LimitNOFILE=65536
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

Start the vllm service and manage and examine it with the following:

systemctl edit vllm
systemctl start vllm
systemctl status vllm
tail -100 /var/log/messages


I’ll update this post shortly as my vLLM efforts continue..