.NET: [Feature]: AG-UI multi-modal input support

[ravijariwala]

Description

We would like to upload attachment as well as user's input to AG-UI endpoint. AG-UI protocol supports multimodal input, however it appears that AGUIMessage class in MAF is only accepting string content.

e.g. if we send following

{
threadId: '9e315809-21ee-4c44-8124-29892b138c4a',
runId: '143bdd12-a905-4687-b7d7-329e536eca20',
state: {},
messages: [
{
id: '5bb98d8f-2459-444a-b156-910ef0087c85',
role: 'user',
content: [
{ type: 'text', text: 'translate this document' },
{ type: 'document', source: [Object], metadata: [Object] }
]
}
],
tools: [
{
name: 'request_approval',
description: 'Request user approval before executing an action',
parameters: [Object]
}
],
context: [],
forwardedProps: {}
}

we are getting BadRequestException.

Error running agent: Error: HTTP 400: Microsoft.AspNetCore.Http.BadHttpRequestException: Failed to read parameter "RunAgentInput? input" from the request body as JSON.
---> System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.content | LineNumber: 0 | BytePositionInLine: 70.
---> System.InvalidOperationException: Cannot get the value of a token type 'StartArray' as a string.
at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ExpectedString(JsonTokenType tokenType)
at System.Text.Json.Utf8JsonReader.GetString()
at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue) at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
--- End of inner exception stack trace ---
at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Deserialize(Utf8JsonReader& reader, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.DeserializeAsObject(Utf8JsonReader& reader, ReadStack& state) at System.Text.Json.JsonSerializer.ReadFromSpanAsObject(ReadOnlySpan1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable1 actualByteCount) at System.Text.Json.JsonSerializer.Deserialize(JsonElement element, JsonTypeInfo jsonTypeInfo) at Microsoft.Agents.AI.Hosting.AGUI.AspNetCore.Shared.AGUIMessageJsonConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.JsonCollectionConverter2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value) at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue) at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.ContinueDeserialize[TReadBufferState,TStream](TReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, T& value) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.DeserializeAsync[TReadBufferState,TStream](TStream utf8Json, TReadBufferState bufferState, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.HttpRequestJsonExtensions.ReadFromJsonAsync[TValue](HttpRequest request, JsonTypeInfo1 jsonTypeInfo, CancellationToken cancellationToken) at Microsoft.AspNetCore.Http.HttpRequestJsonExtensions.ReadFromJsonAsync[TValue](HttpRequest request, JsonTypeInfo1 jsonTypeInfo, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>FB87DEE35F409427C0993BF3A8E11207C78AB7E4167B350548AEB87B222E3F58E__GeneratedRouteBuilderExtensionsCore.TryResolveBodyAsync[T](HttpContext httpContext, <GeneratedRouteBuilderExtensions_g>FB87DEE35F409427C0993BF3A8E11207C78AB7E4167B350548AEB87B222E3F58E__LogOrThrowExceptionHelper logOrThrowExceptionHelper, Boolean allowEmpty, String parameterTypeName, String parameterName, JsonTypeInfo`1 jsonTypeInfo, Boolean isInferred)

Code Sample

Language/SDK

.NET