Code Interpreter + Semantic Kernel

[yuichiromukaiyama]

Has anyone implemented a Code Interpreter on an application developed with Semantic Kernel?
Could you share examples of implementation methods?

For example, the ideas I have in mind are as follows:

1. Develop an execution environment similar to dify-sandbox and transfer the code generated by the LLM there.
2. Use Autogen.
3. Use Assistants API.

If there is a better method, I would love to hear about it. What I am particularly struggling with is how to pass files, such as CSV files uploaded by users, to the plugin.

[moonbox3]

Ho @yuichiromukaiyama, we've just pushed the OpenAI Assistants (v2) Agents to Python main. We will get a package out with this early next week. The OpenAIAssistantAgent whether targeting OpenAI or Azure OpenAI, will allow you to easily work with a code interpreter and be able to include files. You can see some of the agent concept samples for different use cases. Here's an example of the code interpreter using a provided file.

[yuichiromukaiyama]

Hi! @moonbox3

I'm thrilled about these fantastic new features. Thank you so much. I'll start exploring the samples right away while eagerly awaiting the package release.

[yuichiromukaiyama]

@moonbox3

I succeeded in making it work correctly! Thank you very much.
https://github.com/yuichiromukaiyama/sk-assistants-api

Excuse me, but could you please advise on one matter? The following is the thread when it was returned:

Example Response Messages

1. code response

{
  "inner_content": null,
  "ai_model_id": null,
  "metadata": {},
  "content_type": "message",
  "role": "assistant",
  "name": "FileManipulation",
  "items": [
    {
      "inner_content": null,
      "ai_model_id": null,
      "metadata": {},
      "content_type": "text",
      "text": "# Let's first read the contents of the uploaded file to understand its structure and extract relevant information.\r\nfile_path = '/mnt/data/assistant-pW3CLB3GWbkQworP4Rt8FsdO'\r\n\r\n# Open and read the file\r\nwith open(file_path, 'r') as file:\r\n    content = file.read()\r\n\r\ncontent[:1000]  # Displaying first 1000 characters for an overview, to understand its structure.",
      "encoding": null
    }
  ],
  "encoding": null,
  "finish_reason": null
}

2. code response

{
  "inner_content": null,
  "ai_model_id": null,
  "metadata": {},
  "content_type": "message",
  "role": "assistant",
  "name": "FileManipulation",
  "items": [
    {
      "inner_content": null,
      "ai_model_id": null,
      "metadata": {},
      "content_type": "text",
      "text": "import pandas as pd\r\nfrom io import StringIO\r\n\r\n# Load the CSV data into a DataFrame\r\ndata = StringIO(content)\r\ndf = pd.read_csv(data)\r\n\r\n# Display the first few rows of the dataframe to ensure it loaded correctly\r\ndf.head()",
      "encoding": null
    }
  ],
  "encoding": null,
  "finish_reason": null
}

3. message

{
  "inner_content": null,
  "ai_model_id": null,
  "metadata": {},
  "content_type": "message",
  "role": "assistant",
  "name": "FileManipulation",
  "items": [
    {
      "inner_content": null,
      "ai_model_id": null,
      "metadata": {},
      "content_type": "text",
      "text": "The file appears to be in CSV format with the following columns:\n\n- **category**\n- **product_name**\n- **dates**\n- **quantities**\n- **sales_totals**\n- **profits_totals**\n- **weather_conditions**\n\nTo identify which category had the most sales, I'll extract the relevant data and compute the total sales for each category. Let's proceed with that calculation.",
      "encoding": null
    }
  ],
  "encoding": null,
  "finish_reason": null
}

4. code response

{
  "inner_content": null,
  "ai_model_id": null,
  "metadata": {},
  "content_type": "message",
  "role": "assistant",
  "name": "FileManipulation",
  "items": [
    {
      "inner_content": null,
      "ai_model_id": null,
      "metadata": {},
      "content_type": "text",
      "text": "# Grouping by category and summing the sales totals to find which category had the most sales\r\nsales_by_category = df.groupby('category')['sales_totals'].sum().reset_index()\r\n\r\n# Identifying the category with the maximum sales\r\nmax_sales_category = sales_by_category.loc[sales_by_category['sales_totals'].idxmax()]\r\n\r\nmax_sales_category",
      "encoding": null
    }
  ],
  "encoding": null,
  "finish_reason": null
}

5. final response

{
  "inner_content": null,
  "ai_model_id": null,
  "metadata": {},
  "content_type": "message",
  "role": "assistant",
  "name": "FileManipulation",
  "items": [
    {
      "inner_content": null,
      "ai_model_id": null,
      "metadata": {},
      "content_type": "text",
      "text": "The category with the most sales is **Home & Kitchen**, with total sales amounting to approximately **6058.62**.",
      "encoding": null
    }
  ],
  "encoding": null,
  "finish_reason": null
}

1. I cannot determine from specific parameters whether the generated result is a message from AI or code. (For example, I want to display code as a code block to the user)
2. (When returning sequentially like a stream) I can't tell if the generated result is the final response or not. (For example, I want to
   temporarily hide the intermediate progress from the user and only return the final answer)

Is it possible for a program to determine 1. whether it's a message or code, and 2. whether it's the final response or not?

[moonbox3]

Hi @yuichiromukaiyama, thanks for your feedback. Being able to know what is generated from code or not, right now, is a gap. Instead of having to introduce a new content type to handle this, we're going to add a key-value pair to the message's metadata dict in the form of {"code": True}, which you can see will be added in this PR here.

[moonbox3]

Hi @yuichiromukaiyama, the assistant agent code is now available in SK Python 1.4.0.