Deploying LLM

You can deploy the BMC-provided LLM by using one of the following methods:

• Using the LLM library (UI)—This is the recommended method. For more information, see LLM library.
• Manual deployment—We do not recommend this method unless automated deployment isn't possible. Manual deployment involves multiple manual steps. 

If you have an existing LLM that you want to integrate into BMC AMI Platform, then use the following method:

• Through BYOLLM: To integrate a large language model (LLM) hosted in your environment into BMC AMI Platform, whether self-hosted via vLLM or provided via the OpenAI service. For more information, see Bring your own LLM.

Before deploying LLM you must configure kubectl access and environment for RKE2 Kubernetes cluster. 

To configure kubectl access and environment for RKE2 Kubernetes cluster

Ensure you are logged in to the Kubernetes master node as a root user and run the following commands in sequence.

To verify the configuration

To verify the configuration is successful, run the following command.

Deploying LLM manually

The platform supports three LLMs that can be deployed independently:

• Llama 3.1
• Mixtral
• Granite 3.1 

You can also deploy a second LLM service on BMC AMI AI Platform. For more information, see To add a second LLM deployment.

To deploy the Llama model on Kubernetes

1. Log in to the primary manager node of Kubernetes and access /<extracted_dir>/BMC-AMI-PLATFORM-2.0.00.
2. Verify that the scripts/llama.sh file is present.
3. Run the llama.sh file and provide the following input:

   Field	Description
   GPU node	Select the GPU from the list
   CPU KVCache space	Specify the memory allocated for the LLM KV cache. Higher values allow more parallel requests. For more information, see CPU KVCache value.
   Number of GPUs	The value can be found by running the following command on GPU node. nvidia-smi --query-gpu=name --format=csv,noheader | wc -l

To verify the deployment

To add a second LLM deployment

The following instructions describe how to deploy a second LLM service on BMC AMI Platform.

Before you begin

• Make sure your machine meets the minimum system requirements for LLM deployment. For more information, see LLM GPU.
• Make sure the NVIDIA GPU operator is installed. For more information, see To manually install the NVIDIA GPU operator.

To add new node to cluster

1. Add your node to the Kubernetes cluster.
   Warning

   Important

   You must set up the cluster, including adding nodes. Make sure you add all required nodes to the cluster according to the deployment prerequisites.

2. Mount the existing NFS on the newly created node.
3. Verify the node exists in the Kubernetes cluster by running the following command: 
   kubectl get nodes

To add a label to the node

Label the node using the following command:

To run the script file to deploy the model

Navigate to the Helm chart directory and run the appropriate script:

Where to go from here

After you deploy LLM you can proceed to the following topics:

• Undeploying LLM
• Deploying additional services