Offline Speech Recognition #2089

[VladislavAntonyuk]

Description of Change

Added 2 new methods for Offline Speech Recognition, Removed ListenAsync method, as it is impossible (with current implementation) to correctly Stop Listening the recognition. I also added a new State, allowing us to see if Listening is Active, but Silience.

Linked Issues

• Fixes Offline SpeechToText #2089 , [BUG] SpeechToText throws Objective-C exception on iOS #1779 , [BUG] SpeechToText on iOS 17 cuts off or fails to recognize many words #1966

PR Checklist

• Has a linked Issue, and the Issue has been approved(bug) or Championed (feature/proposal)
• Has tests (if omitted, state reason in description)
• Has samples (if omitted, state reason in description)
• Rebased on top of main at time of PR
• Changes adhere to coding standard
• Documentation created or updated: Update SpeechToText docs, Add OfflineSpeechToText MicrosoftDocs/CommunityToolkit#489

Additional information

[bijington]

@VladislavAntonyuk thanks for this. I think it looks great! I only had a few comments to get your thoughts on some things.

Finally, this might be outside of the scope of this PR but I wanted to raise it because I think it should be in a follow-up PR; I would love the ability to chain the 2 implementations together when registering with DI, something like:

builder.Services.AddSpeechToText(SpeechToText.Default).WithFallback(OfflineSpeechToText.Default);

And in theory developers could chain the other way round:

builder.Services.AddSpeechToText(OfflineSpeechToText.Default).WithFallback(SpeechToText.Default);

What do you think to the above? We might have to wrap this in another class rather than complicating the flow of the 2 current implementations so it probably is best in a follow-up PR.

[VladislavAntonyuk]

Thank you @bijington.
if user registers SpeechToText like this builder.Services.AddSpeechToText(OfflineSpeechToText.Default).WithFallback(SpeechToText.Default);
he still needs to somehow resolve the implementation.

Also, we need to provide the lifetime of the service. And what is under the hood of AddSpeechToText? We'll still have the same AddSingleton(OfflineSpeechToText.Default) call in that method.

[bijington]

Thank you @bijington. if user registers SpeechToText like this builder.Services.AddSpeechToText(OfflineSpeechToText.Default).WithFallback(SpeechToText.Default); he still needs to somehow resolve the implementation.

Also, we need to provide the lifetime of the service. And what is under the hood of AddSpeechToText? We'll still have the same AddSingleton(OfflineSpeechToText.Default) call in that method.

Yes I agree the developer will need to define the lifetime of the service which increases the complexity.

Perhaps we could move the WithFallback method onto the ISpeechToText interface instead, then the developer could write something like:

builder.Services.AddSingleton(OfflineSpeechToText.Default.WithFallback(SpeechToText.Default));

Then WithFallback could look something like:

public static void ISpeechToTextExtensions
{
    public ISpeechToText WithFallback(this ISpeechToText primary, ISpeechToText secondary)
    {
        return new PriorityBasedSpeechToText(primary, secondary);
    }
}

internal class PriorityBasedSpeechToText : ISpeechToText
{
    readonly ISpeechToText primaryService;
    readonly ISpeechToText secondaryService;

    public PriorityBasedSpeechToText(this ISpeechToText primary, ISpeechToText secondary)
    {
        primaryService = primary;
        secondaryService = secondary;
    }

    public Task<SpeechToTextRecognitionResult> StartListenAsync()
    {
        // attempt primary, if fails fallback to secondary...
    }
}

It could well become more complicated with things like permissions, so we may have to request all permissions for both primary and secondary.

What do you think?

[VladislavAntonyuk]

I see pros and cons of such approach.
As for developers it is much easier registering the service, but the strategy of choosing the right implementation maybe complicated (Users preferences may forbid online recognition, unstable internet connection, etc).
Also as WithFallback still receives interface as a parameter, developers may write such code:
services.AddSpeechToText(SpeechToText.Default).WithFallback(SpeechToText.Default);

We could technically hide the online/offline recognition in the implementation and keep single service. We had such implementation for Windows in our initial release.

The main idea of Offline Speech To Text to allow developers explicitly specify the required implementation.
Also I don't want they inject 2 separate interfaces in the service (MyService(ISpeechToText s1, IOfflineSpeechToText s2)), because there simultaneous usage may be unpredictable.

We can open a discussion for the next month.

[bijington]

@VladislavAntonyuk thanks for this! I have a few comments and I think they are mostly pretty minor

[bijington]

Thanks @VladislavAntonyuk I have added some xml docs improvements but the rest looks good to me

[bijington]

LGTM! Do we just need to write the docs? Is this something you are happy to do? If not I'm happy to write something

[VladislavAntonyuk]

LGTM! Do we just need to write the docs? Is this something you are happy to do? If not I'm happy to write something

Thank you Shaun! I will create docs PR this weekend.

[pictos]

The sample app is crashing for me. Here's the log that I could get.

I ran on iphone 14 pro. And allowed all permissions.

[VladislavAntonyuk]

The sample app is crashing for me. Here's the log that I could get.

I ran on iphone 14 pro. And allowed all permissions.

It’s text to speech. I will remove it from the offline sample

[pictos]

will try again during this week, asap

Copilot reviewed 20 out of 35 changed files in this pull request and generated no suggestions.

Files not reviewed (15)

• samples/CommunityToolkit.Maui.Sample.sln: Language not supported
• samples/CommunityToolkit.Maui.Sample/CommunityToolkit.Maui.Sample.csproj: Language not supported
• samples/CommunityToolkit.Maui.Sample/Pages/Essentials/OfflineSpeechToTextPage.xaml: Language not supported
• samples/CommunityToolkit.Maui.Sample/Pages/Essentials/SpeechToTextPage.xaml: Language not supported
• samples/CommunityToolkit.Maui.Sample/AppShell.xaml.cs: Evaluated as low risk
• src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/Events/SpeechToTextRecognitionResultCompletedEventArgs.cs: Evaluated as low risk
• samples/CommunityToolkit.Maui.Sample/ViewModels/Essentials/EssentialsGalleryViewModel.cs: Evaluated as low risk
• src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/ISpeechToText.shared.cs: Evaluated as low risk
• samples/CommunityToolkit.Maui.Sample/MauiProgram.cs: Evaluated as low risk
• samples/CommunityToolkit.Maui.Sample/ViewModels/Essentials/SpeechToTextViewModel.cs: Evaluated as low risk
• src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToTextImplementation.macos.cs: Evaluated as low risk
• samples/CommunityToolkit.Maui.Sample/Pages/Essentials/OfflineSpeechToTextPage.xaml.cs: Evaluated as low risk
• src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToTextImplementation.tizen.cs: Evaluated as low risk
• samples/CommunityToolkit.Maui.Sample/ViewModels/Essentials/OfflineSpeechToTextViewModel.cs: Evaluated as low risk
• src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToText.shared.cs: Evaluated as low risk

Comments skipped due to low confidence (3)

src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToTextImplementation.ios.cs:19

• [nitpick] The error message could be more informative by including the current iOS version.

throw new NotSupportedException("Offline listening is supported on iOS 13 and later");

src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToTextImplementation.android.cs:85

• [nitpick] The error message 'Failure in speech engine - {error}' might not be very helpful to the end-user. Consider providing a more user-friendly error message.

OnRecognitionResultCompleted(SpeechToTextResult.Failed(new Exception($"Failure in speech engine - {error}")));

src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/OfflineSpeechToTextImplementation.android.cs:156

• In the OnPartialResults and OnResults methods, the matches variable is being accessed without checking for null or empty values. Although the current implementation handles it, adding an explicit check would improve clarity and avoid potential issues.

if (matches is null || matches.Count == 0)