Deployed large language model (LLM)

This topic describes how to view the deployed large language models (LLMs) and their details in one place. You can also add a local LLM.

To view the deployed LLMs

1. Sign in to BMC AMI Platform by using your credentials. 
2. Click the Platform manager tab.
3. From the menu in the left pane, select BMC AMI AI Manager > AI services settings > Deployed LLMs.
   The Deployed Large Language model (LLM) window lists all the deployed LLMs.
   
   

To add a local model

1. On the upper left of the LLM settings window, click Add local model.
2. In the Add local large language model dialog box, select your model type.

3. Select the Public API or the Self-hosted tab, depending on your model type. 

   • Public API
   • Self-hosted

   Follow these steps:

   1. Select the hosted platform. 
   2. Enter an API access key.
   3. Click Test connection.
   4. If a success confirmation message is displayed, click Next>>. (If a failure message is displayed, verify that the provided API access key is valid.)

   Follow these steps:

   1. Complete the following fields:

      Field	Description
      Inference engine	Select the inference engine you want to use. We support vLLM, Triton, and an OpenAI-compatible inference server.
      Base URL	Enter your inference engine base URL. For example:  Triton- http://my-triton-server:4000/v1 vLLM- https://my-vllm-server/v1 OpenAI Compatible-   https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/
      API access key	(Optional) Key to access the LLM server API if configured in the LLM server

   2. Click Test connection.
   3. If a success confirmation message is displayed, click Next>>. (If a failure message is displayed, verify that the provided details are valid.)

   Warning

   Important

   To set up OpenAI-compatible inference AWS Bedrock:

   1. Open the aws-samples/bedrock-access-gateway GitHub repository.
   2. Follow the steps to set up the Llama3.1 model in the OpenAI Compatible API format.  

   
   

A success confirmation message is displayed. If this procedure fails, the user interface displays an error message with the reason for the failure. Once you have added the LLM, see Integrations settings and BMC AMI Assistant chat settings to use the LLM.