Tell us what you think about the Semantic Kernel v1 proposal

[madsbolaris]

Earlier today we shared the full proposal for v1 on our blog. If you have any thoughts about our proposal, this is the place to share them so we can incorporate the feedback into our final designs.

[eavanvalkenburg]

after implementing a lot of the same concepts in a python branch, I have some notes, questions and suggestions:

• Unclear from the dotnet how the run_async selects a service to use: what happens when no matching one is found, and is the order of the template maintained by all yaml handlers?
• As noted in the earlier discussion thread, I think at least for native functions we need to be able to name the output and support multiple outputs. There are multiple arguments for that:
  • There is no sample yet of a pre-hook and post-hook, but for me those also will require a function to output a dict with multiple objects, for a semantic function, that might be two things: result (a completion string, image, etc) and a full unparsed response object (like the openai response objects or something similar), from there a post-hook can then easily be created for things like token_usage, streaming with callbacks, logging content filters, etc.
  • Another scenario is that you might want to be able to do a call with multiple functions, where the result of function 1 is outputted to a field called 'result1', which is then added back into the context under that name and passed to the next, potentially with a parameter controlling the behaviour.
• Currently the plugin class, is quite ugly, I would expect some way of creating a plugin from a folder that includes parsing semantic functions and native functions in one call, to truly create a Plugin as a object, rather then a container that you just happen to use because the functions are related but there is currently no underlying commonality between the semantic and native functions of a plugin -> in other words, a plugin currently is a list of functions with a name, why would we need another class to host that?
• Let's create a single naming convention for a fully qualified function name, preferably one that can be used by openai function calling as well, since it is very similar to what I now pass to the template handler. The convention there is: a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64 (API Reference - OpenAI API)
• In the current implementation the kernel is still involved in rendering the template, is this necessary? I think one of the key things here is that you can have semantic functions with different template languages working side by side, therefore the render should not rely on the kernel, all necessary info can be passed through the variabels and or kwargs?

[madsbolaris]

For the first bullet, we recently created an ADR so that we can support this functionality: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0016-semantic-function-multiple-model-support.md

I also agree with the last point, all of the rendering should be done by a template factory. We also recently created an ADR for that here: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0017-custom-prompt-template-formats.md

Once these ADRs become real code, I'll update the v1-proposal samples to snap to the proper implementation.

[madsbolaris]

I'll bring the other points to the team and see if they have feedback!

[eavanvalkenburg]

I've now gone ahead and created a hook approach in the python sdk, following the decision from here to initially only do hooks around the kernel invoke function, not yet in other places, this decision defininately needs a dict in order to still get the full response object from the LLM call, and use that to parse, in my sample 03, I've added two prebuilt hooks, one for token usage, and one to stdout a streaming response, which I think show the reason to have multi-output in both cases, the first uses the openai response object to parse the tokens and adds a additional field to the result so that the users can see the outcome, while the latter takes the response and has the code to handle the async generator in it. For the samples look here

[gingters]

This is my feedback here:

Feedback to SK v1 propoals:

This is only relating to the C# variant.

01.01 ContextVariables to Dictionary<string, object>

Seems more flexible at first, but exposing this through the API is difficult.
As a C# developer, you really want to minimize untyped values as much as you can.
Forcing a weak-typed collection on developers doesn't feel good. Better provide a way
to introduce multiple ContextVariables collections, each strongly typed by using generics.

Better:
Use a ContextVariables carrier object to transfer the values:

var contextVars = new ContextVariables();
contextVars.Set<ChatHistory>("messages", chatHistory);

kernel.RunAsync(function, contextVars);

The generic can be omitted here, as the compiler can infere the type from the
chatHistory variables type. You can also get back a strongly typed instance later on to work
with it:

var history = contextVars.Get<ChatHistory>("messages");

// Since it is typed now, you can easily access all properties of the ChatHistory object.

Ideally, also make it an IContextVariables interface. A developer might decide to
implement a different ContextVariables carrier object which additionally has constraints
on the types that might be registered with it, i.e. force them to be serializable, so that
the Context can be safely stored away and retrieved later on.
When just putting objects in there, you might end up with a type that is not serializable
and then it's impossible to persist the context variables where serialization would be needed.

01.02 SKContext to IKernel

Again, please use a ContextVariables carrier object thet provides a type-safe API here
instead of a Dictionary<string, object>.

02.01. Handlebars

.NET already has powerful template engines that a) most .NET developers know and b)
fit perfectly in the ecosystem. These are T4 templates as well as the Razor engine.

Especially when rendering to a HTML/CML-like structure like you did in the example,
Razor would be a much better fit for .NET developers.

02.03. YAML

No YAML, please. Configuration in the .NET world nowadays is almost exclusively done
in JSON, and we have an extremely powerful, layered JSON configuration system already
in place. Besides that, Json tooling is being optimized in .NET for several years now,
while YAML doesn't really have good tooling at all in the .NET space.

If leveraged correctly, System.Extensions.Configuration also allows you to load changes
to the configuration at runtime, making developing / tuning prompts a breeze. You send
a request, see the answer, tune the prompt, save the file, immidiately send a new request
that - at runtime - picks up the change on the IOptions and you
can directly see the result of the adjusted prompt.

Also, the current configuration system allows you to override only specific values for
prod/dev/env so you can test new prompts in your functions easily by only overriding a
specific prompt in your appsettings.test.json file on your test environment.

If you force adoption of YAML onto .NET developers, they will look for alternatives.

04.01. Functions detached - as singletons?

You suggest that functions can (or should) be instanciated once and be re-used by several kernels.
This is honestly a bad advice. For example, think of an AI-powered API that should provide a function
that looks up stuff in a database with Entity Framework Core.

For EF Core, you need to get an instance of your DbContext that is scoped to the current API request,
because EF Core strongly discourages the use of long-lived DbContextes as they carry a DbConnection
from a pool of multiple connections and these should be as short-lived as possible.
So if your function needs a DbContext, you have to jump through a lot of hoops and your function
oftentimes needs to resort to the ServiceLocator antipattern where your function gets a ServiceProvider,
uses that to create a new scope of its own (which is sadly detached from the scope of the current
request), and resolves the DbContext from that scope.

What you'd actually should provide, is that the developer only registers a Type and not an
function instance with your kernel. The Kernel can then use reflection to determine the functions on
the Type and provide these to the model. If a function is called, then the Kernel uses the DI system
to resolve the function, based on the current scope of the request that also provided the Kernel itself.
This way, singletons are still possible while functions that require some scoped dependencies like a
DbContext will also easily work.

[gingters]

After all, this might sound a bit harsh, but I want to give a bit more feedback and also additional context.

I am a C# dev since the first beta of .NET, and my peers consider me a quite experienced .NET dev.
With all of the AI stuff going on lately I dabbled a bit into Langchain, and so far I accumulated about 100 hours of "experience" with Python.

I built some very basic samples / demos / intro application, both with langchain in Python, with Semantic Kernel in C# and additionally with NO library support in C# except for the Azure OpenAI client and a Qdrant connector library.

With my skillset and my experience, I was able to complete that in Python as well as in my own C# approach, but I was not able to finish my demo applications with Semantik Kernel as I was struggling as fighting with the API and the interfaces all the time. And by finished I mean that my code would be in a state that I could show others and that would live up to my own quality expectations. The code that was required to make it work was something I would never dare to hand over to a customer.

For me, as a seasoned C# developer, Semantic Kernel, at least with its current API, completely feels like an alien element / foreign object in the .NET world.

It does not, like at all, integrate with common approaches that pretty much all of the .NET world adopted in the last few years, like Microsoft.Extensions.DependencyInjection, Microsoft.Extensions.Logging or the options pattern in Microsoft.Extensions.Configuration. It does not use the IHttpClientFactory to create new http clients when needed and it cannot use named clients.

Kernels seem to be intended to be used as singletons, but this makes usage of them extremely hard in a multi-user environment like a web API. Database connections (i.e. EF Core database contexts) should only be used short lived (scoped to the length of a single request to your API) and if your kernel needs a DB context for memory, this is extremely difficult to set up and handle. You have to really fight the obviously intended usage of the kernel to fit into a real-world scenario like a complex API. The same goes for HttpClients which should be created from the pool of underlying HttpMessageHandlers in order to be reusable in a reliable way.

I strongly encourage you to ask other people, i.e. Immo Landwerth from the .NET Team to talk about and review the C# APIs before you put a v1 label on it and ship it with an API that from that point in time is fixed and needs to be maintained as backward-compatible and which really does not fit in the current state of the .NET ecosystem.

[madsbolaris]

Thanks for the in-depth feedback! I'll be sharing it with the engineering team so they can respond to the technical elements. I can try to share the business reasoning for the others :)

02.01. Handlebars

At a Microsoft-level, prompts assets are being standardized around Handlebars so that we can provide cross-language support across Python, C#, Java, and TypeScript. Because of this, we want to provide out-of-the-box support for Handlebars templates across our C#, Python, and Java SDKs. We acknowledge, however, that there are many other template engines out there, so as part of V1, we are updating the interface so you can add your own custom template. Post v1, we'd like to explore supporting other templating languages like Liquid, T4, and Razor.

02.03. YAML

Similarly, at a Microsoft-level, prompt assets are being standardized around YAML files. We acknowledge that there are challenges to working with this file format, so we will eventually provide helper methods for other file types (e.g., JSON). For V1, we believe what's most important for us is to align on the interfaces of the objects themselves. Afterwards, we believe we can add additional serializations to these configuration objects.

[madsbolaris]

Since you've provided such great feedback already, I also wanted to dig-in to this to get your feedback.

Kernels seem to be intended to be used as singletons

Today's documentation makes it look like Kernels should be treated as singletons. This is probably a miss on our part since the kernel is being architected so that it can be transient. The goal is that for every request, you can populate the kernel with the required AI services and functions so that things like semantic functions and planners have the tools they need to run correctly.

Using this pattern, we believe that developers can do the following:

1. Use things like DI to get the required AI services, and functions for each request
2. Spin up a kernel with the resolved AI services, functions, and HTTP client
3. Complete the user request
4. Dispose of the kernel

We're hoping this pattern also helps with A/B tests of AI services and functions in offline and online scenarios. The kernel essentially becomes an object that just holds runtime configuration for a single request.

[markwallace-microsoft]

@gingters Thanks for the feedback.

We are working with Stephen Toub on the SK 1.0 API and he is bringing the perspective of what will be idiomatic for .Net developers. So a lot of the issues you ran into should be resolved in v 1.0. Stephen has organised a review with Immo and some other folks, so that will be happening soon also. We are planning to publish our first release candidate next week and it would be great if you could take another look and let us know what you think.

[gingters]

This is awesome news. I'll happily await the RC. Thanks alot for taking that so serious. 👍

[dsgrieve]

What is the deadline for comment?

[madsbolaris]

Our goal is to have most of the changes complete by EOM so we can reach back out to the community to get additional feedback after trying it out. If you’re able to share feedback over the next week, we’ll have better luck incorporating it in the preview version of v1.0.

[acrigney]

Guys love your work but where does https://github.com/microsoft/kernel-memory project fit in?
I want to implement a RAG but from this https://github.com/Azure-Samples/semantic-kernel-rag-chat
But it looks like kernel memory is what I could be using instead and it does not using Azure Open AI at all.

[sandeepvootoori]

@matthewbolanos - Is handlebars fully available for us to try with beta 6? Any examples on how to import a yaml semantic function from folder?

[madsbolaris]

Today, you need to do things inline with code. You can see an example of it here: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example64_MultiplePromptTemplates.cs

This sprint, we're working on adding YAML support so that you'll be able to import it in more easily. Should be available to try next week.

[dsgrieve]

01.01. ContextVariables will become be a Dictionary<string, object>

The main drawback of Map<String, Object> is that we give up compile-time type checking and open ourselves to ClassCastException at runtime.
However, I cannot think of a better alternative.

The Java code is positioned to handle this with little modification. The Semantic Kernel for Java's SKFunctionParameters annotation already
includes the type of parameter (its Class<?>) which can be used to do the cast at runtime. For a semantic function, the input parameter
"type" in the template would need to be a fully qualified classname in order for Java to do the class lookup (via Class.forName).
For example, in SimpleChat.prompt.yaml from the 01-SimpleChat example, Java would need to have:

input_variables:
  - name: messages
    type: com.microsoft.semantickernel.chatcompletion.ChatHistory
    description: The history of the chat.
    is_required: true

Because of this, this change to Dictionary<string,object> has the unforunate consequence that one cannot apply the same template to any language.
One now needs a .NET version and a Java version, at least. Perhaps there is a way to address this in the yaml by adding a package or namespace element.

01.02. SKContext will be replaced with IKernel and the variables dictionary

I would pass the variables dictionary in some other class that allows for possible future expansion without having to change the IKernel or ISKFunction abstractions. At some point in the future there is going to be the need to pass some piece of data other than the variables. Some piece of data that
doesn't belong in the kernel and isn't a variable. Pre/post hooks come to mind.

01.03. Memory will be modeled like any other plugin

What happens to the existing memory connectors that are in the semantic-kernel repo?

02.04. You can configure execution settings for multiple models

I can see setting defaults in the yaml, but there may be many factors to selecting the best service. Letting the kernel make the decision should be a default
behavior. A more powerful approach would be a mux that can be configured or programmed to select the best service at invocation time. One would tell the mux
what kind of service they need and what the parameters of that service need to be. For instance, I need text embedding but I want to pay only this much for tokenization.

04.01. Use a function with multiple kernels

This may be apparent, but this has the implication that functions must be stateless. On the other hand, this will make it easier to cache results from previous computations for reuse.

04.02. Introducing plugins to the kernel

Will existing plugins be repackaged in a separate library?

04.03. Simplifying the creation of a kernel

Java will still have builders which smooth over issues of having multiple constructors or constructors with a plethora of arguments.
Ultimately, the builders call a single constructor. A developer can choose either.

04.04. Stream functions from the kernel

By streaming, you mean server sent events. In the given code example, how are the events dispatched to the client? Is there some kind of while (!result.done()) thing or is streaming is handled completely within the semantic kernel? How can a client take advantage of streaming?

[madsbolaris]

01.01. ContextVariables will become be a Dictionary<string, object>

@markwallace-microsoft, this is an interesting one for you to take a look at while you work on the YAML import funcitonality. I'm curious how you were planning on configuring custom types and if it could scale to Java.

01.02. SKContext will be replaced with IKernel and the variables dictionary

The latest plan is to introduce a class that inherits from Dictionary to help address some of these concerns. @SergeyMenshykh should be able to comment on the latest. Regarding pre/post-hooks. The current plan is to have them live on the kernel (which will be passed everywhere).

01.03. Memory will be modeled like any other plugin

We'll keep them in the short-term, but by early 2024, we'll likely deprecate and remove them. Instead, we'll point folks to OpenAI's retrieval plugins.

02.04. You can configure execution settings for multiple models

@markwallace-microsoft, introduced a service selector class to create custom logic to choose a model.

04.01. Use a function with multiple kernels

Yes, we should provide guidance that functions should be stateless.

04.02. Introducing plugins to the kernel

Yes, the current plugins will not be moving to v1.0.0. Once we're done with v1.0, we'll likely create a new package for the plugins we want to continue supporting.

04.03. Simplifying the creation of a kernel

This makes sense. In the C# side, we'll also likely keep both ways of creation.

04.04. Stream functions from the kernel

In C#, we're able to use IAsyncEnumerable to allow streaming of chat completions. Is there something similar in Java?

[johnoliver]

01.02. SKContext will be replaced with IKernel and the variables dictionary

I think the issue with this is that we are painting ourselves into a corner to some extent as we would need to make a breaking change to the definitions of the invocation if we want to add more "things" to an invocation. I would guess would want to add a class along the lines of InvocationContext, that contains all the components required for performing an invocation, you can then have simpler convenience methods that form that context for you, i.e:

class InvocationContext {
   ContextVariables variables;
   RequestSettings settings;
   PreHook pre;
   ...
}


class SemanticKernelFunction {
  ...
  // convenience method so you dont need to create a InvocationContext yourself
  ContextVariables InvokeAsync(Kernel kernel, ContextVariables variables);

  // Actually does the work and available for users to fully customise an invocation
  ContextVariables InvokeAsync(Kernel kernel, InvocationContext context);
  ...
}

This would then give us flexibility to add features to invocations without making breaking changes. Although the examples may not use the InvokeAsync(Kernel kernel, InvocationContext context); method as it can be considered generally only for use cases where the user requires a high degree of customisation of their invocation.

[johnoliver]

01.01. ContextVariables will become be a Dictionary<string, object>

For Java side I would probably go with something like:

public interface ContextVariables extends Map<String, ContextVariable<?>> {
  String prettyPrint();
  ...etc
}

public abstract class ContextVariable<T> {
  public abstract String toPromptString();
  public abstract T get();
}

So we would not use Map directly but use our own class that implements the interface, and also use ContextVariable rather than Object

[strikene]

I have a little bit of advice, maybe not very perfect. It's all based on my experience.
I have created an AI personal assistant with complex functions using SK. He has dialogue/Azure/M365/other actions.
1.SKFunction is likely to return an incorrect result. This issue is a headache when there are multiple steps. as opposed to complex processes. It's harder for us to control. So we had to write additional prompts for breaking down the task.But that's probably why I didn't write it properly. If it is indeed my reason. Please ignore this item.
2.I'm trying to use Handlebars to generate complex tasks. Very good performance. But from 1.0.0-beta8 it seems that the Plan generated by Handlebars, I can't modify its parameter values. A possible way to do this is to convert to JSON first and replace it yourself. Isn't it possible to use the Handlebars to generate the Plan parameter in the future to allow external changes to be made using the Context.
3.This is a very subjective feeling. I'd prefer to use JSON instead of YAML. As everyone knows. There is a complete Json library on .net.
4.As it stands. It's hard to create apps with complex tasks using SK. SK seems to offer everything you need once you actually use it. You'll find it really awkward. It seems as if you are simply transferring your previous work to SK. I don't know if we can add such an interface to SK in the future.It allows it to automatically switch between multiple "roles" or calling modes at the same time. Similar to PVAs in M365, they have multiple "Titles". Of course, this may conflict with the positioning of PVA.

Again, I would like to reiterate that the above is based on my experience. Maybe it's because my studies aren't in place, after all, SK is very new, and there are a lot of materials that maybe I haven't fully noticed. If there is a problem with my statement. Corrections are welcome.

[evchaki]

Thank you all for the feedback:
API Documentation for 1.0.1 - https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel?view=semantic-kernel-dotnet
Blog Post - https://devblogs.microsoft.com/semantic-kernel/semantic-kernel-v1-0-1-has-arrived-to-help-you-build-agents/