Python: Code Interpreter returning NaNs when there is missing field

[CristinaStn]

Using Semantic Kernel AzureAssistantAgent with access to Code interpreter to provide comprehensive stats and data insights on some tabular data in .csv format.

For some analysis, the Agent is using Code Interpreter tool to answer. It provides a table with samples of data from input which may contain missing values (which is normal). The missing values are displayed to the end user as NaNs. For example:

Test Case ID	Input Value	Expected Output	Actual Output	Notes
TC-001	10	100	100	Pass
TC-002	NaN	0	NaN	Input was missing
TC-003	"abc"	Error	NaN	Invalid input type
TC-004	50	2500	NaN	Calculation failed
TC-005	5	25	25	Pass

We do understand these NaNs are coming by default from pandas library. However, an important business requirement for our integration is not to provide such NaNs cells to the end user. We tried to control this behaviour of the agent via instruction prompt:

When using the code interpreter tool, treat all NaN, null, or missing values the same way Microsoft Excel would treat empty cells — display them as empty strings. This ensures consistency and clarity for end-users who expect Excel-like behaviour.

But this solution does not always provide accurate and consistent answers to the end user: at some point during conversation with the agent, the agent shows NaNs when answering and when generates files with those reports.

Q: Do you have any recommendation to make the Agent to avoid NaNs while using code interpreter?

Note: we are using semantic kernel==1.29.0

[TaoChenOSU]

Hi @CristinaStn,

Thank you for your question!

Depending on your application, you may need to create a pipeline that post processes the agent's output to have 100% guaranteed on the final form of your table.

There is another thing you can try in addition to the instruction message. You can augment the user request to the agent with additional requirement, i.e. the user message. This may be more effective than the instruction message if your context is long. However, this still won't guarantee the table will not have NaNs.

[CristinaStn]

Hi @TaoChenOSU
We are streaming the response from agent to end user and so, it is very difficult and risky to post-process the chunks.

About the second suggestion, one drawback I see is when the we display the entire conversation to the end user, the user will see those augmentations on its own messages. Would developer role messages help?

[moonbox3]

Thanks for the response, @CristinaStn. We are streaming back the code response from the server, so you would need to try and prompt with additional instructions for it to not include NaN, or other unwanted behavior. Currently there is no way to filter the streaming response further.

[moonbox3]

One other thought, @CristinaStn: when we handle the other issue you raised (#12552), by returning code as part of the intermediate message callback, that should give you a way to filter the chunks for NaN, if you need to.