Merge pull request #2928 from ealagna/cms/ealagna/hpe-dev-portal/blog/deploying-a-hugging-face-llm-in-hpe-private-cloud-ai

Create Blog “deploying-a-hugging-face-llm-in-hpe-private-cloud-ai”

Author: choukden

content/blog/deploying-a-hugging-face-llm-in-hpe-private-cloud-ai.md

@@ -0,0 +1,135 @@
+---
+title: Deploying a Small Language Model in HPE Private Cloud AI using a Jupyter
+  Notebook
+date: 2025-02-20T20:03:50.971Z
+author: Dave Wright and Elias Alagna
+authorimage: /img/Avatar1.svg
+disable: false
+tags:
+  - AI
+  - PCAI
+  - vllm
+  - SLM
+---
+Deploying new language models for for users to interact with can be challenging for beginners. HPE developed Private Cloud AI to help users set up and implement AI solutions quickly and easily. 
+
+In this post, we will show how to use the HPE Machine Learning Inference Service (MLIS) as a part of HPE Private Cloud AI to add a new packaged model from a Hugging Face repository and create an endpoint to query the model. This is done using a Jupyter Notebook.
+
+### Prerequisites
+
+This tutorial uses the [HPE Private Cloud AI](https://www.hpe.com/us/en/private-cloud-ai.html) (PCAI) platform. A PCAI system is required for these steps to work. It is assumed that the PCAI system is physically installed, patched and running with user accounts provisioned.
+
+### Steps to deploy
+
+First, you will need to choose a model to deploy. In this case, we've chosen a model hosted on Hugging Face called [SmolLM2 1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct). This is a compact model that can solve a wide range of problems even though it is relatively diminutive at 1.7B parameters.
+
+### Launching the interface
+
+![Computer screen showing the HPE Private Cloud AI user interface and the HPE MLIS tile is highlighted.](/img/mlis.png)
+
+Next select "Add new model".
+
+![Computer screen showing packaged AI models and a selection to add a new model.](/img/new-model.png)
+
+This brings up the "Add new packaged model" dialog box. Fill in the the name of the model, storage requirements, and resources. We have reduced the default resources, given that this is a small model.
+
+![Dialog box for defining a new packaged model.](/img/define-parameters.png)
+
+Once the package is set up, you will receive a confirmation.
+
+![Shows running packaged model.](/img/package-running.png)
+
+With the new packaged model complete, you will need to deploy it for use. Select "create new deployment" from the HPE MLIS "Deployments" tab. Select submit when all tabs are filled out as shown below.
+
+This will create an endpoint for use in the notebook and provide an API token.
+
+![New deployment for AI model](/img/new-deployment.png)
+
+When the process is complete, an endpoint will be provided.
+
+![Endpoint provided by MLIS system](/img/endpoint.png)
+
+Next up, let's take the now deployed model that's ready for inference and connect to it and interact with it from a Jupyter Notebook.
+
+### Building the Jupyter Notebook
+
+First, install `openai` if you do not already have it and import.
+
+```python
+# vLLM Chat OpenAI
+# !pip intall openai
+from openai import OpenAI
+```
+
+Then, using the endpoint and key generated by HPE MLIS, enter them into your Jupyter Notebook. Be sure to append /v1 to the URL.
+
+```python
+# Grab endpoint URL and API key from MLIS, remember to include "/v1" for latest version of the OpenAI-compatible API
+model = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
+openai_api_base = "https://smollm2-1-7b-vllm-predictor-dave-wright-hpe-1073f7cd.hpepcai-ingress.pcai.hpecic.net/v1"
+openai_api_key = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE3Mzk5MzgzMzAsImlzcyI6ImFpb2xpQGhwZS5jb20iLCJzdWIiOiI5MjNhM2JhOC1mMGU4LTQxOTQtODNkMS05ZWY4NzNjZGYxOWYiLCJ1c2VyIjoiZGF2ZS53cmlnaHQtaHBlLmNvbSJ9.YwH9gGPxTWxy4RSdjnQA9-U3_u7P0OIcarqw25DV8bOiftU1L4IvvyERHspj2lMGtZWbff1F3uh84wjAePHaHDcDTLoGtq6gJYwo_qRU03xV8Q2lwBetCCLUE4OHqS608gjJ-j1SLyqwxFxlXkqMOtnBY5_nswlAwCzHV28P8u8XxxfWuXFmoJpSA1egCWVVfEoTuK8CTz9kUJJ5opSp6m8qdqJmC2qxH0igcpKmL2H_MZ-62UHfEf240VRtc0DRNlOjeCoDM79aVPs3SjCtGeVkeEHimJwJbfGFIcu3LibX3QjbABUzWb5BPPZjzyEYUVM5ak12_sJ8j1mUW-r0sA"
+```
+
+You will now need to create an OpenAI client interface.
+
+```python
+# create OpenAI client interface
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+```
+
+In order to interact with the model, you will need to create a chat function. For the purposes of our example, let's give it a history feature as well as basic chat.
+
+```python
+# Interactive chat function with message history. 
+def chat():
+    # Initialize conversation history
+    messages = []
+    
+    print("Chat with "+model+"! Type 'quit' to exit.")
+    
+    while True:
+        # Get user input
+        user_input = input("\nYou: ").strip()
+        
+        # Check for quit command
+        if user_input.lower() == 'quit':
+            print("Goodbye!")
+            break
+        
+        # Add user message to history
+        messages.append({"role": "user", "content": user_input})
+        
+        try:
+            # Get model response using chat completion
+            response = client.chat.completions.create(
+                model=model,
+                messages=messages
+            )
+            
+            # Extract assistant's message
+            assistant_message = response.choices[0].message.content
+            
+            # Add assistant's response to history
+            messages.append({"role": "assistant", "content": assistant_message})
+            
+            # Print the response
+            print("\nAssistant:", assistant_message)
+            
+        except Exception as e:
+            print(f"\nError: {str(e)}")
+```
+
+![Jupyter Notebook showing imported model endpoint and API key.](/img/jupyter.png)
+
+Once this is done, you can interact with the model through a simple chat.
+
+![Interaction with the SmolLM2 Small Language Model in a Jupyter Notebook](/img/chat-interface.png)
+
+You can access [this link](https://www.youtube.com/watch?v=oqjc-2c1Vtk) to see a recorded demonstration that shows this process in real time. [](https://www.youtube.com/watch?v=oqjc-2c1Vtk)
+
+### Summary
+
+With HPE Private Cloud AI, loading new models into the system and providing endpoints is just a few simple clicks and easily integrates with popular tools like Jupyter Notebooks. To learn more about HPE Private Cloud AI, please visit: <https://www.hpe.com/us/en/private-cloud-ai.html>