We’re using a derivative of RHEL 9 Linux being AlmaLinux 9, but this should work for RHEL 9 itself and others like Rocky Linux 9 etc.<\/p>\n\n\n\n
dnf install -y python3 python3-devel\n\n# Create a virtual environment\npython3 -m venv ~\/venvs\/vllm\n\n# Activate the environment\nsource ~\/venvs\/vllm\/bin\/activate\n\n\n<\/code><\/pre>\n\n\n\nBefore you can start the install of vllm you need to ascertain your CUDA version level, which can be found from the output of nvidia-smi | grep CUDA<\/strong><\/code>
And you’ll see something like “CUDA UMD Version: 13.4<\/strong>“, so that corresponds to the variable “cuversion” in the following command being set to “134<\/strong>“<\/p>\n\n\n\npip install --upgrade pip\npip install vllm --extra-index-url https:\/\/download.pytorch.org\/whl\/cu134<\/code><\/pre>\n\n\n\nThis command takes some time to run, does lots of the heavy lifting of the vllm install.<\/p>\n\n\n\n
Verify the vLLM install with the command:<\/p>\n\n\n\n
python -c \"import vllm; print(vllm.__version__)\"<\/code><\/p>\n\n\n\nConfigure vLLM by editing the \/etc\/systemd\/system\/vllm.service file to be:<\/p>\n\n\n\n
[Unit]\nDescription=vLLM OpenAI Compatible API Server\nAfter=network.target nvidia-persistenced.service\nWants=nvidia-persistenced.service\n\n[Service]\nType=simple\nUser=your_username\nGroup=your_username\nEnvironment variables for CUDA and Cache directories\n\nEnvironment=\"HF_HOME=\/home\/your_username\/.cache\/huggingface\"\nEnvironment=\"CUDA_VISIBLE_DEVICES=0\"\nPath to your virtual environment's python binary and the vLLM execution module\n\nExecStart=\/home\/your_username\/venvs\/vllm\/bin\/python3 -m vllm.entrypoints.openai.api_server\n--model meta-llama\/Meta-Llama-3-8B-Instruct\n--host 0.0.0.0\n--port 8000\n\nRestart=always\nRestartSec=10\nSecurity and resource tuning configurations for RHEL\/AlmaLinux\n\nLimitNOFILE=65536\nLimitCORE=infinity\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\nI’ll update this post shortly as my vLLM efforts continue..<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
We’re using a derivative of RHEL 9 Linux being AlmaLinux 9, but this should work for RHEL 9 itself and others like Rocky Linux 9 etc. Before you can start the install of vllm you need to ascertain your CUDA … Continue reading