Unity - Google cloud speech-to-text voice recognition, Unity freezes after successful result - unity3d

Question

A friend of mine and I are working on a VR project in Unity at the moment and we are trying to implement voice recognition as a feature. We are using Unity version 2018.3.3f1. The idea is that a user can say a word and the voice recognition will see if they pronounced it correctly. We have chosen to use the Google cloud speech-to-text service for this as it supports the target language (Norwegian). In addition, the application is also multiplayer and so we are trying to use the streaming version of Google cloud speech. Here is a link to their documentation: https://cloud.google.com/speech-to-text/docs/streaming-recognize
What we have done is to have a plugin that essentially runs the speech recognition for us. It is a modification of the example code given in the link above:
public Task<bool> StartSpeechRecognition()
{
return StreamingMicRecognizeAsync(20, "fantastisk");
}
static async Task<bool> StreamingMicRecognizeAsync(int inputTime, string inputWord)
{
bool speechSuccess = false;
Stopwatch timer = new Stopwatch();
Task delay = Task.Delay(TimeSpan.FromSeconds(1));
if (NAudio.Wave.WaveIn.DeviceCount < 1)
{
//Console.WriteLine("No microphone!");
return false;
}
var speech = SpeechClient.Create();
var streamingCall = speech.StreamingRecognize();
// Write the initial request with the config.
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
StreamingConfig = new StreamingRecognitionConfig()
{
Config = new RecognitionConfig()
{
Encoding =
RecognitionConfig.Types.AudioEncoding.Linear16,
SampleRateHertz = 16000,
LanguageCode = "nb",
},
InterimResults = true,
}
});
// Compare speech with the input word, finish if they are the same and speechSuccess becomes true.
Task compareSpeech = Task.Run(async () =>
{
while (await streamingCall.ResponseStream.MoveNext(
default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream
.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
if (alternative.Transcript.Replace(" ", String.Empty).Equals(inputWord, StringComparison.InvariantCultureIgnoreCase))
{
speechSuccess = true;
return;
}
}
}
}
});
// Read from the microphone and stream to API.
object writeLock = new object();
bool writeMore = true;
var waveIn = new NAudio.Wave.WaveInEvent();
waveIn.DeviceNumber = 0;
waveIn.WaveFormat = new NAudio.Wave.WaveFormat(16000, 1);
waveIn.DataAvailable +=
(object sender, NAudio.Wave.WaveInEventArgs args) =>
{
lock (writeLock)
{
if (!writeMore) return;
streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
AudioContent = Google.Protobuf.ByteString
.CopyFrom(args.Buffer, 0, args.BytesRecorded)
}).Wait();
}
};
waveIn.StartRecording();
timer.Start();
//Console.WriteLine("Speak now.");
//Delay continues as long as a match has not been found between speech and inputword or time that has passed since recording is lower than inputTime.
while (!speechSuccess && timer.Elapsed.TotalSeconds <= inputTime)
{
await delay;
}
// Stop recording and shut down.
waveIn.StopRecording();
timer.Stop();
lock (writeLock) writeMore = false;
await streamingCall.WriteCompleteAsync();
await compareSpeech;
//Console.WriteLine("Finished.");
return speechSuccess;
}
We made a small project in Unity to test if this was working with a cube GameObject that had this script:
private CancellationTokenSource tokenSource;
VR_VoiceRecognition.VoiceRecognition voice = new VR_VoiceRecognition.VoiceRecognition();
IDisposable speech;
// Use this for initialization
void Start() {
speech = Observable.FromCoroutine(WaitForSpeech).Subscribe();
}
// Update is called once per frame
void Update() {
}
IEnumerator WaitForSpeech()
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
Debug.Log("Starting up");
Task<bool> t = Task.Run(() => voice.StartSpeechRecognition());
while (!(t.IsCompleted || t.IsCanceled))
{
yield return null;
}
if (t.Status != TaskStatus.RanToCompletion)
{
yield break;
}
else
{
bool result = t.Result;
UnityEngine.Debug.Log(t.Result);
yield return result;
}
}
void OnApplicationQuit()
{
print("Closing application.");
speech.Dispose();
}
We are also using a plugin that was recommended to us by Unity support that they thought might have a workaround called UniRx (https://assetstore.unity.com/packages/tools/integration/unirx-reactive-extensions-for-unity-17276).
At the moment it is working fine when you play it in the editor for the first time. When the voice recognition returns false then everything is fine (two cases when this happens is if it cannot find a microphone or if the user does not say the specific word). However, if it is a success then it still returns true, but if you exit play mode in the editor and try to play again then Unity will freeze. Unity support suspects that it might have something to do with the Google .dll files or Google API. We are not quite sure what to do from here and we hope that someone could point us to the right direction.