Stage an MEAI 10.4.1 release#7402
Merged
jeffhandley merged 14 commits intodotnet:release/10.4from Mar 18, 2026
Merged
Conversation
- Fix duplicate test case IDs caused by unused gapBeginningEnd loop variable - Suppress non-serializable data warnings with DisableDiscoveryEnumeration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
…ation (dotnet#7381) * Add ITextToSpeechClient and friends
* Realtime Client Proposal
* Replace [Experimental("MEAI001")] with DiagnosticIds.Experiments.AIRealTime
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Remove AsIRealtimeClient extension method
The extension method on OpenAIClient was not useful because it
completely ignored the OpenAIClient instance - only validating it
for null before creating a new OpenAIRealtimeClient with the
separately provided apiKey and model parameters.
Users can construct OpenAIRealtimeClient directly instead.
* Address PR review comments
- Fix RealtimeSessionExtensions XML doc to reference IRealtimeSession
instead of IChatClient
- Replace non-standard <ref name> tags with <see cref> in
RealtimeServerMessageType.cs for proper IntelliSense/doc rendering
- Fix ResponseDone doc summary to say 'completed' instead of 'created'
- Add missing Throw.IfNull(updates) in LoggingRealtimeSession
.GetStreamingResponseAsync for consistency with other sessions
* Fix multiple issues found during code review
- Split RealtimeServerMessageType enum: add ResponseOutputItemDone
and ResponseOutputItemAdded to distinguish per-item events
(response.output_item.done, conversation.item.done) from
whole-response events (response.done, response.created)
- Fix function result serialization: use JsonSerializer.Serialize()
instead of ToString() to properly serialize complex objects
- Fix OTel streaming duration: start stopwatch at method entry
instead of immediately before recording, so duration histogram
measures actual streaming time
- URL-encode model name in WebSocket URI for defensive safety
- Fix OTel metadata tag ordering: apply user metadata before
standard tags so standard OTel attributes take precedence
if keys collide
* Remove duplicate internal FunctionInvocationStatus enum
* Remove stale CreateToolsMap calls before the loop
* Remove FunctionInvocationResultInternal; use public FunctionInvocationResult directly
* Add IAsyncDisposable to IRealtimeSession; store and await receive task in OpenAIRealtimeSession
* Revert API baseline JSON changes
* Clarify audio/text token counts are subsets of Input/OutputTokenCount
* Replace ToolChoiceMode enum with ChatToolMode for cross-modality consistency
* Unify TranscriptionOptions across Realtime and SpeechToText
- Move TranscriptionOptions from Realtime/ to SpeechToText/ folder
- Change experimental flag from AIRealTime to AISpeechToText
- Make properties nullable with parameterless constructor
- Rename Language to SpeechLanguage, Model to ModelId
- Replace SpeechToTextOptions.ModelId and .SpeechLanguage with Transcription property
- Update all consumers and tests
* Use HasTopLevelMediaType instead of StartsWith for audio check
* Add API compat suppressions for SpeechToTextOptions breaking changes
* Replace string Eagerness with SemanticEagerness struct (ChatRole pattern)
* Remove InternalsVisibleToTest from Microsoft.Extensions.AI.OpenAI
- Make OpenAIRealtimeSession constructor and ConnectAsync public
- Remove InternalsVisibleToTest from csproj
- Remove OpenAIRealtimeSessionSerializationTests (depended on internal ConnectWithWebSocketAsync)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add RawRepresentationFactory to RealtimeSessionOptions
Add Func<IRealtimeSession, object?>? RawRepresentationFactory property following
the same pattern used by ChatOptions, EmbeddingGenerationOptions, and other
abstraction options types. Add note in OpenAIRealtimeSession to consume the
factory when switching to the OpenAI SDK.
* Remove OpenAI-specific tracing properties from RealtimeSessionOptions
Remove EnableAutoTracing, TracingGroupId, TracingWorkflowName, and
TracingMetadata from the abstraction layer. These are OpenAI-specific
and should be configured via RawRepresentationFactory when the OpenAI
SDK dependency is added.
* Remove AIFunction/HostedMcpServerTool properties, use ChatToolMode
Remove redundant AIFunction and HostedMcpServerTool properties from
RealtimeSessionOptions and RealtimeClientResponseCreateMessage. Callers
should use ChatToolMode.RequireSpecific(functionName) instead.
Update OpenAI serialization to emit structured tool_choice JSON object
when RequireSpecific is used. Update OpenTelemetry and tests accordingly.
* Improve XML docs for Usage on response and transcription messages
* Remove leftover temp files
* Improve ConversationId XML docs on RealtimeServerResponseCreatedMessage
* Split Text/Audio properties on RealtimeServerOutputTextAudioMessage
Add separate Audio property for Base64-encoded audio data. Text is now
only used for text and transcript content. Update OpenAI parser,
OpenTelemetry session, and tests accordingly.
* Replace RealtimeServerMessageType enum with readonly struct
Follow the ChatRole smart-enum pattern: readonly struct with string
Value, IEquatable, operators, and JsonConverter. Providers can now
define custom message types by constructing new instances.
Update pattern-matching in OpenTelemetryRealtimeSession to use ==
comparisons instead of constant patterns.
* Move Parameter into ErrorContent.Details on error messages
Remove the Parameter property from RealtimeServerErrorMessage and
map error.param to ErrorContent.Details instead. Improve ErrorEventId
XML docs to clarify it correlates to the originating client event.
* Add RawRepresentation property to RealtimeContentItem
Add object? RawRepresentation to hold the original provider data
structure, following the same pattern as other types in the
abstraction layer (e.g., ChatMessage). Updated tests accordingly.
* Rename Metadata to AdditionalProperties for consistency
Rename the Metadata property to AdditionalProperties on both
RealtimeClientResponseCreateMessage and RealtimeServerResponseCreatedMessage
to be consistent with the established pattern used across the AI
abstractions (ChatMessage, ChatOptions, AIContent, etc.). Updated
XML docs, OpenAI provider, OTel session, and tests accordingly.
* Improve MaxOutputTokens XML docs to clarify modality scope
Clarify that MaxOutputTokens is a total budget across all output
modalities (text, audio) and tool calls, not per-modality.
* Improve XML docs on RealtimeClientResponseCreateMessage properties
Clarify that ExcludeFromConversation creates an out-of-band response
whose output is not added to conversation history. Document that
Instructions, Tools, ToolMode, OutputModalities, OutputAudioOptions,
and OutputVoice are per-response overrides of session configuration.
* Improve RealtimeClientResponseCreateMessage class-level XML doc
Clarify that this message triggers model inference and that its
properties are per-response overrides of session configuration.
* Rename EventId to MessageId for terminology consistency
Rename EventId to MessageId on RealtimeClientMessage and
RealtimeServerMessage, and ErrorEventId to ErrorMessageId on
RealtimeServerErrorMessage. The abstraction uses 'message' terminology
throughout (class names, docs, method signatures), so properties
should match. The OpenAI provider maps MessageId to/from the wire
protocol's event_id field.
* Remove blank line between using directives in audio buffer messages
* Rename RealtimeAudioFormat.Type to MediaType for consistency
Rename to match the established MediaType naming convention used
across the abstractions (DataContent, HostedFileContent, UriContent,
ImageGenerationOptions). Updated OpenAI provider and tests.
* Refactor IRealtimeSession: rename InjectClientMessageAsync to SendClientMessageAsync and remove updates parameter from GetStreamingResponseAsync
- Rename InjectClientMessageAsync -> SendClientMessageAsync across all implementations
- Remove IAsyncEnumerable<RealtimeClientMessage> updates parameter from GetStreamingResponseAsync
- Move per-message telemetry from WrapClientMessagesForTelemetryAsync into SendClientMessageAsync override in OpenTelemetryRealtimeSession
- Delete WrapUpdatesWithLoggingAsync from LoggingRealtimeSession
- Delete WrapClientMessagesForTelemetryAsync from OpenTelemetryRealtimeSession
- Update AnonymousDelegatingRealtimeSession delegate signature
- Update RealtimeSessionBuilder.Use overload signature
- Update all tests to use new API
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make RealtimeSessionOptions and related types immutable with init-only properties and IReadOnlyList
* Fix XML doc param order, add null validation to constructors
* OpenAI Realtime Provider using OpenAI SDK
* Fix typo: add missing space in comment ('IDsince' -> 'ID since')
* Make CreateSessionAsync return non-nullable IRealtimeSession
- Change IRealtimeClient.CreateSessionAsync return type from Task<IRealtimeSession?> to Task<IRealtimeSession>
- Remove exception-swallowing catch blocks in OpenAIRealtimeClient.CreateSessionAsync and OpenAIRealtimeSession.ConnectAsync; let exceptions propagate to callers
- Change ConnectAsync from Task<bool> to Task
- Remove unused System.IO and System.Net.WebSockets usings
- Update tests to expect OperationCanceledException instead of null/false returns
* Remove SemanticEagerness from Abstractions; move eagerness to AdditionalProperties
- Delete SemanticEagerness.cs (OpenAI-specific concept)
- Remove Eagerness property from SemanticVoiceActivityDetection
- Add AdditionalProperties to base VoiceActivityDetection for provider-specific settings
- Update OpenAI provider to read/write eagerness via AdditionalProperties["eagerness"]
* Include CompatibilitySuppressions for SpeechToTextOptions API changes
* Remove MCP-specific types from abstractions; add provider contract docs
* Remove trailing blank line in RealtimeServerErrorMessage
* Clarify ErrorMessageId XML doc to distinguish from base MessageId
* Rename ErrorMessageId to OriginatingMessageId for clarity
* Change ExcludeFromConversation from bool to bool? for provider-default consistency
* Remove blank lines between Experimental attribute and class declaration
* Add null validation to Content property setter
* Make RealtimeAudioFormat.SampleRate non-nullable and remove misleading doc
* Rename IRealtimeSession to IRealtimeClientSession and SendClientMessageAsync to SendAsync
* Fix null DefaultConversationConfiguration when ExcludeFromConversation is unset
* Remove IDisposable from IRealtimeClientSession, keep only IAsyncDisposable
* Remove ConversationId from RealtimeServerResponseCreatedMessage
* Rename ResponseCreateMessage and ConversationItemCreateMessage to CreateResponseMessage and CreateConversationItemMessage
* Rename RealtimeContentItem to RealtimeConversationItem
* Remove VoiceSpeed from RealtimeSessionOptions and use RawRepresentationFactory seed pattern
* Remove UpdateAsync from IRealtimeClientSession, use message-based approach
Replace UpdateAsync method on IRealtimeClientSession with a new
RealtimeClientSessionUpdateMessage type sent via SendAsync. This avoids
requiring all providers to implement session updates (e.g. Gemini does
not support mid-session updates).
- Add RealtimeClientSessionUpdateMessage with Options property
- Remove UpdateAsync from IRealtimeClientSession and DelegatingRealtimeSession
- Move update logic to OpenAIRealtimeSession.SendAsync handler
- Remove UpdateAsync overrides from Logging and OpenTelemetry sessions
- Update all tests to use SendAsync with the new message type
* Rename realtime session classes to include Client in name
Align implementation class names with IRealtimeClientSession interface
naming. All session-related classes now include 'Client':
- DelegatingRealtimeSession -> DelegatingRealtimeClientSession
- OpenAIRealtimeSession -> OpenAIRealtimeClientSession
- LoggingRealtimeSession -> LoggingRealtimeClientSession
- OpenTelemetryRealtimeSession -> OpenTelemetryRealtimeClientSession
- FunctionInvokingRealtimeSession -> FunctionInvokingRealtimeClientSession
- RealtimeSessionBuilder -> RealtimeClientSessionBuilder
- RealtimeSessionExtensions -> RealtimeClientSessionExtensions
- TestRealtimeSession -> TestRealtimeClientSession
- All related builder extension and test classes updated
* Simplify DisposeAsyncCore to return ValueTask directly
Remove unnecessary async/await in DelegatingRealtimeClientSession.DisposeAsyncCore,
returning InnerSession.DisposeAsync() directly instead.
* Convert RealtimeSessionKind from enum to extensible string struct
Convert RealtimeSessionKind from a closed enum to a readonly struct
following the same pattern as RealtimeServerMessageType and
ChatFinishReason. This allows providers to define custom session kinds.
Rename Realtime -> Conversation for clarity, avoiding the redundant
RealtimeSessionKind.Realtime naming.
* Remove NoiseReductionOptions from abstraction
NoiseReductionOptions is OpenAI-specific and not supported by other
providers. Remove it from the abstraction layer; users who need noise
reduction can use provider-specific options.
* Remove VoiceActivityDetection hierarchy from abstraction
VoiceActivityDetection, ServerVoiceActivityDetection, and
SemanticVoiceActivityDetection are too OpenAI-specific. Remove them
from the abstraction; users who need VAD can configure it through
provider-specific options via RawRepresentationFactory.
* Remove AnonymousDelegatingRealtimeClientSession and anonymous Use overload
Remove the anonymous Use overload that only intercepts
GetStreamingResponseAsync without the ability to intercept SendAsync,
making it too limited to be useful in practice.
* Fix JSON injection in MessageId by using JsonSerializer.Serialize
* Remove meaningless duration metric from SendAsync in OTel session
The stopwatch was started and recorded within the same block without
wrapping the actual SendAsync call, measuring nothing useful. Duration
metrics remain in GetStreamingResponseAsync where they measure actual
streaming time.
* Refactor builder to operate on IRealtimeClient instead of IRealtimeClientSession
- Create DelegatingRealtimeClient base class (replaces DelegatingRealtimeClientSession)
- Create RealtimeClientBuilder (replaces RealtimeClientSessionBuilder)
- Create FunctionInvokingRealtimeClient, LoggingRealtimeClient, OpenTelemetryRealtimeClient
- Make session middleware types (FunctionInvokingRealtimeClientSession, etc.) internal
- Rename builder extension classes to remove 'Session' from names
- Add RealtimeClientExtensions and RealtimeClientBuilderRealtimeClientExtensions
- Use DiagnosticIds.Experiments.AIRealTime for Experimental attributes
- Update all tests to use public client APIs only
* Rename message classes: move RealtimeClient/RealtimeServer before Message
Rename pattern: RealtimeClient{Name}Message -> {Name}RealtimeClientMessage
Rename pattern: RealtimeServer{Name}Message -> {Name}RealtimeServerMessage
Base classes RealtimeClientMessage and RealtimeServerMessage unchanged.
* Fix cancellation tests to also accept WebSocketException on net462
* Remove PreviousId from CreateConversationItemRealtimeClientMessage as OpenAI-specific
* Document that ResponseId may be null and response lifecycle events may be synthesized
* Document that CreateResponseRealtimeClientMessage may be a no-op for VAD-driven providers
* Add RealtimeResponseStatus constants and document interruption/barge-in via Status property
* Remove session parameter from RealtimeSessionOptions.RawRepresentationFactory
* Fix AOT compatibility errors in OpenAIRealtimeClientSession by using source-generated JSON serialization
* Fix SpeechToTextOptionsTests for unified TranscriptionOptions after upstream merge
* Revert SpeechToTextOptions changes per review feedback
Restore ModelId and SpeechLanguage as direct properties on
SpeechToTextOptions instead of nesting them under a Transcription
property. TranscriptionOptions class remains for RealtimeSessionOptions.
- Restore SpeechToTextOptions.ModelId and .SpeechLanguage properties
- Remove SpeechToTextOptions.Transcription property
- Update all consumers to use direct property access
- Fix SpeechToTextClientMetadata doc reference
- Regenerate CompatibilitySuppressions.xml
- Update all related tests
* Use JsonInclude/Core pattern for experimental UsageDetails properties
Apply the [JsonIgnore] public + [JsonInclude] internal Core backing
property pattern to the 4 experimental audio/text token properties
on UsageDetails, matching the convention used elsewhere (e.g.
ChatOptions.AllowBackgroundResponses, ChatResponse.ContinuationToken).
This enables JSON serialization of these properties when using the
library's JsonSerializerOptions while keeping the public API gated
behind [Experimental].
- InputAudioTokenCount -> InputAudioTokenCountCore
- InputTextTokenCount -> InputTextTokenCountCore
- OutputAudioTokenCount -> OutputAudioTokenCountCore
- OutputTextTokenCount -> OutputTextTokenCountCore
- Add tests for property roundtrip, Add summation, and JSON
serialization/deserialization of the new properties
* Fix SendAsync to propagate exceptions instead of swallowing them
Replace silent return on cancellation with ThrowIfCancellationRequested,
throw InvalidOperationException when session is not connected, and remove
catch block that swallowed OperationCanceledException, ObjectDisposedException,
and WebSocketException.
Update tests to verify the new exception behavior.
* Replace toolMap dictionary with FindTool delegate pattern
Refactor FunctionInvocationProcessor to accept a Func<string, AITool?>
delegate instead of Dictionary<string, AITool>? to align with main
branch's FindTool pattern and avoid reintroducing the removed toolMap.
* Use IEnumerable<AITool> in FindTool/HasAnyTools to avoid unsafe IList cast
Options.Tools is IReadOnlyList<AITool> which does not implement
IList<AITool>, so the 'as IList<AITool>' cast could silently return
null and ignore tools. Use IEnumerable<AITool> which both IList<T>
and IReadOnlyList<T> implement.
* Document that function invocation blocks message processing loop
Add known limitation note to FunctionInvokingRealtimeClientSession
XML docs explaining that incoming server messages (including user
interruptions) are buffered during function invocation.
* Add distinct ConversationItem message types to prevent double invocation
ConversationItemAdded and ConversationItemDone were incorrectly mapped
to ResponseOutputItemAdded/Done, which could cause the function
invoking session to invoke the same function call twice.
* Remove custom ToolChoice telemetry attribute
Remove gen_ai.request.tool_choice custom attribute that is not part
of the OpenTelemetry GenAI semantic conventions.
* Add null validation to Item property setter
Consistent with constructor which already validates via Throw.IfNull.
* Add <exception> XML doc tags to realtime types
Document ArgumentNullException on constructors and property setters
that validate via Throw.IfNull/Throw.IfNullOrWhitespace.
* Use OrdinalIgnoreCase comparer for MCP headers dictionary
HTTP headers are case-insensitive per RFC 7230.
* Change RealtimeResponseStatus from const fields to static properties
Avoids baking values into consuming assemblies at compile time.
* Add GetRequiredService extension methods for realtime types
Add GetRequiredService and GetRequiredService<T> to both
IRealtimeClient and IRealtimeClientSession extensions, matching
the pattern from IChatClient and IEmbeddingGenerator.
* Remove unused System.Threading.Channels package reference
* Update OTel semantic conventions version reference from v1.39 to v1.40
---------
Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Initial plan * Update PackageValidationBaselineVersion to 10.4.0 and remove CompatibilitySuppressions.xml files Co-authored-by: joperezr <13854455+joperezr@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: joperezr <13854455+joperezr@users.noreply.github.com> Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
…nt` in FunctionInvokingChatClient (dotnet#7382) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com> Co-authored-by: Tarek Mahmoud Sayed <tarekms@microsoft.com> Co-authored-by: Tarek Mahmoud Sayed <10833894+tarekgh@users.noreply.github.com>
…penTelemetry instrumentation classes (dotnet#7379) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com> Co-authored-by: tarekgh <10833894+tarekgh@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com> Co-authored-by: Stephen Toub <stoub@microsoft.com>
* Initial plan * Make HostedFileDownloadStream read-only by default Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com> * Fix HostedFileDownloadStream write overload ordering Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
…file extension (dotnet#7398) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
…otnet#7399) * Add VoiceActivityDetection options to realtime session abstractions Introduce VoiceActivityDetectionOptions with Enabled and AllowInterruption properties to RealtimeSessionOptions. These represent the common VAD options supported across multiple realtime AI models (OpenAI, Gemini, Anthropic Claude, AWS Nova Sonic). - Add VoiceActivityDetectionOptions class with Enabled (default true) and AllowInterruption (default true) properties - Add VoiceActivityDetection property to RealtimeSessionOptions - Map VAD options to OpenAI SDK TurnDetection in both conversation and transcription session paths - Add concurrency guidance for SendAsync in IRealtimeClientSession docs - Add unit tests for the new types * Address PR review feedback - Fix type name in SendAsync concurrency docs: FunctionInvokingRealtimeSession -> FunctionInvokingRealtimeClientSession - Preserve existing TurnDetection from seed options (RawRepresentationFactory) by mutating InterruptResponseEnabled on existing RealtimeServerVadTurnDetection instead of replacing it, in both conversation and transcription paths * Clarify relationship between Enabled and AllowInterruption in VAD docs Document that AllowInterruption only takes effect when Enabled is true, and that disabling VAD fully disables turn detection making interruption not applicable. --------- Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
…dotnet#7400) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
tarekgh
approved these changes
Mar 18, 2026
Member
tarekgh
left a comment
tarekgh
left a comment
There was a problem hiding this comment.
The Realtime stuff looks good to me. Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
invoke_workflowas an equivalent parent span toinvoke_agentin FunctionInvokingChatClient #7382Microsoft Reviewers: Open in CodeFlow