Install vLLM on RHEL 9 Linux

We’re using a derivative of RHEL 9 Linux being AlmaLinux 9, but this should work for RHEL 9 itself and others like Rocky Linux 9 etc.

dnf install -y python3 python3-devel

# Create a virtual environment
python3 -m venv ~/venvs/vllm

# Activate the environment
source ~/venvs/vllm/bin/activate

Before you can start the install of vllm you need to ascertain your CUDA version level, which can be found from the output of nvidia-smi | grep CUDA
And you’ll see something like “CUDA UMD Version: 13.4“, so that corresponds to the variable “cuversion” in the following command being set to “134“

pip install --upgrade pip
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu134

This command takes some time to run, does lots of the heavy lifting of the vllm install.

Verify the vLLM install with the command:

python -c "import vllm; print(vllm.__version__)"

Configure vLLM by editing the /etc/systemd/system/vllm.service file to be:

[Unit]
Description=vLLM OpenAI Compatible API Server
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service

[Service]
Type=simple
User=your_username
Group=your_username
Environment variables for CUDA and Cache directories

Environment="HF_HOME=/home/your_username/.cache/huggingface"
Environment="CUDA_VISIBLE_DEVICES=0"
Path to your virtual environment's python binary and the vLLM execution module

ExecStart=/home/your_username/venvs/vllm/bin/python3 -m vllm.entrypoints.openai.api_server
--model meta-llama/Meta-Llama-3-8B-Instruct
--host 0.0.0.0
--port 8000

Restart=always
RestartSec=10
Security and resource tuning configurations for RHEL/AlmaLinux

LimitNOFILE=65536
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

I’ll update this post shortly as my vLLM efforts continue..