MRTK V2.2 - Access Speech Command via Script - unity3d

In my scenario, buttons are created during runtime. These are to be clicked by a voice command. For this reason I try to find out how I can add voice commands during runtime. But I can't find any approach.
What I tried:
I have extended the interface IMixedRealitySpeechSystem with two methods, RefreshRecognition and AddSpeechCommand:
/// <summary>
/// Refresh recognition after adding new commands
/// </summary>
void RefreshRecognition();
/// <summary>
/// Add command to already existing commands[]
/// </summary>
/// <param name="command"></param>
void AddSpeechCommand(SpeechCommands command);
I have implemented these in the class WindowsSpeechInputProvider: MixedRealitySpeechSystem. But there are two problems.
First: I can't get to the WindowsSpeechInputProvider. I thought I could get it by trying this:
private IMixedRealitySpeechSystem SpeechSystem
{
get
{
if(_speechSystem is null)
{
MixedRealityServiceRegistry.TryGetService(out _speechSystem);
}
return _speechSystem;
}
}
public void SomeMethod()
{
SpeechCommands command = new SpeechCommands("TestCommand", default, default, null);
SpeechSystem.AddSpeechCommand(command);
SpeechSystem.RefreshRecognition();
}
But the problem is that MixedRealityServiceRegistry does not contain an instance of that service or to be precise, it is not even a service.
Second: Even if this would work, it is not a good way to go. Because with this I change the MRTK and with another upgrade to a new version, these lines are overwritten.
My Question:
So how can I access and add commands on runtime?

There's an open feature request to allow adding dynamic speech commands in Github: Add keywords dynamically to MRTK speech commands #6369. It is not currently possible.
This thread has some suggestions for alternative ways to approach the overall scenario. In summary, it is recommended that you use a Grammar Recognizer and use an SRGS XML file to define your speech recognition rules. Voice input in Unity and Hologram 212 has an example showing how to use it.

Related

The type or namespace name 'Forms' does not exist in the namespace 'System.Windows' FIX

Can someone please help me with this, am trying to use OpenFileDialog class from System.Windows.Forms to open a file dialog and read the selected file. Then, this error showed up. I've referenced it but still the same, below is the code.
`using UnityEngine
using UnityEngine.UI
using System.Windows.Forms;
public class OpenFileButtonScript : MonoBehaviour
{
public TextFieldScript textFieldScript;
public void OpenFile()
{
OpenFileDialog openFileDialog = new OpenFileDialog();
openFileDialog.Filter = "Text Files (*.txt)|*.txt|All Files (*.*)|*.*";
openFileDialog.FilterIndex = 1;
openFileDialog.Multiselect = false;
if (openFileDialog.ShowDialog() == DialogResult.OK)
{
string filePath = openFileDialog.FileName;
string text = System.IO.File.ReadAllText(filePath);
textFieldScript.inputField.text = text;
}
}
}`
It may look like you have access to all of the native Window system libraries, but it just looks like it. In actuality, a lot of the time you're simply given stubs, or shims, that look like the full Window libraries, because there's a certain element that Unity wants to use from those namespaces. If you think about it, the code you present above, what do you think it should do on Android or Nintendo devices? The simple answer is, it simply won't work.
Generally in cases like this, you have to gain access to the native operating system, and perform those calls directly. For example, there is a file browser asset on the Asset Store, that does this for you. It's not free, because the process isn't trivial.
Depending on how much effort you want to put in, you CAN read files from the local file stores (to varying degrees based on platform). It's possible to read the list of files in a location, and use either uGUI or UIToolkit to create your own File Open Dialogue box. Again, this isn't a trivial task either. So you have to be sure that you'd want to go down that path.

Assign UIElements button.clickable, using Visual Scripting Graph

I am trying to use Unity3D's UIToolkit with Visual Scripting Graph...
Typically, I can Query for the proper element, and then add the necessary function to the callback... However, as I am new to Visual Scripting, I am not sure how to get to this point. I've got the 'clickable' object of the button isolated, but I'm not sure how to assign the follow-up execution in the graph.
Usually in the code, I would do something like
clickable.clicked += ExampleFunction();
What I am messing up in the graph, is how to access the '.clicked' part. I can get the proper button element, and I can isolate it's clickable property, but I can't figure out how to assign some kind of functionality to react to the button getting clicked.
I could use a custom node to make this work, but I am looking for a way to do this with built-in nodes, if possible.
Alright... I had to write a custom node, but I figured this out. Here is the graph for the solution.
You have to grab the UIDocument from whichever GameObject it is attached to... You then need to get the Root Visual Element, do NOT clone or instantiate it. You then need to Query for the desired button, using the name you gave it in the UI Builder. It is easier if you use the U Query Extensions nodes... After that, I just made a custom node to subscribe the functionality. I am not familiar of any nodes that do this.
Here is the 'Subscribe Start Result' node code:
using Unity.VisualScripting;
using UnityEngine;
using UnityEngine.UIElements;
public class SubscribeStartResult : Unit
{
[DoNotSerialize]
[PortLabelHidden]
public ControlInput inputTrigger;
[DoNotSerialize]
[PortLabelHidden]
public ValueInput element;
protected override void Definition()
{
element = ValueInput<Button>("element");
inputTrigger = ControlInput("inputTrigger", (flow) =>
{
flow.GetValue<Button>(element).clicked += () =>
{
Debug.Log("Button clicked");
};
return null;
});
}
}
With this setup, clicking the 'Start Button' in play-mode will log "Button clicked" in the Console.
The 'return null;' line is an artifact of the lambda. It is required to continue the control flow in the event this node has a follow-up... Otherwise, this combination of nodes and code allow you to assign callbacks for the UI Builder elements, using the Visual Scripting Graph.

MRTKv2 'Interactable' script for dynamically added gameobjects

So I want to be able to use the 'Interactable' script in MRTKv2 for some of my gameobjects. I've had success when attaching the 'Interactable' and 'NearInteractionTouchable' script to the corresponding gameobject, but not when attempting to add the same functionality dynamically in the script.
Since the 'Microsoft.MixedReality.Toolkit.UI' isn't currently defined, I know I can't reference the 'Interactable' script in the script directly, but in using the 'IMixedRealityFocusHandler', 'IMixedRealityPointerHandler', 'IMixedRealityTouchHandler', and 'IMixedRealityInputHandler' interfaces instead I still haven't had success in receiving any input.
The namespaces currently defined by MRTKv2
All of the namespaces in MRTKv2
Event mapping from HL1 to HL2
Any ideas on a step that I'm missing to recreate the functionality of the 'Interactable' script that can be attached to gameobjects dynamically?
Interactable does exist in the MRTK namespace under using Microsoft.MixedReality.Toolkit.UI; perhaps you are missing references to your projects? The following code works for me, using the latest mrtk_development:
using Microsoft.MixedReality.Toolkit.Input;
using Microsoft.MixedReality.Toolkit.UI;
using UnityEngine;
public class AddInteractableTest : MonoBehaviour
{
// Start is called before the first frame update
void Start()
{
gameObject.AddComponent<Interactable>();
gameObject.AddComponent<NearInteractionTouchable>();
}
}

Access the TM Pro InputField text in Unity

I've been working on this solidly for hours and can't seem to get anywhere, so your help will be much appreciated.
I have created a few TextMeshPro UI InputFields in Unity and can't seem to access the text after a user has inputted something. I can do it when it's a normal InputField but not with the TM Pro version. I've created a Serializefield of type TMP_InputField which I have hooked up in the inspector (also including using TMPro namespace).
Could somebody please outline the steps required to get the text into a variable because I am stumped! The script is currently sitting on a different GameObject. I have an array of type TMP_InputField called theFields and when I debug theFields[0].name that seems to work fine (using FindObjectsOfType<>) but when I try to just access a specificField.text it throws a null reference objection when trying to edit the field during the game. Any tutorials/support would be very welcome!
Did you try with UnityEvents fields on the Inspector of TMP_InputField?
I make a simple script and it combines with TMP_InputField
using UnityEngine;
public class InputfieldListener : MonoBehaviour
{
public void OnChangedInputField(string input)
{
Debug.Log("[OnChangedInputField] " + input);
}
public void OnEndedInputField(string input)
{
Debug.Log("[OnEndedInputField] " + input);
}
public void OnSelectedInputField(string input)
{
Debug.Log("[OnSelectedInputField] " + input);
}
public void OnDeslectedInputField(string input)
{
Debug.Log("[OnDeslectedInputField] " + input);
}
}
Then you can find the log of each event.
FYI, the creator has a youtube channel and there is a related video tutorial.
using TMPro;
public TMP_InputField TMP_IF;
TMP_IF.interactable = true;
TMP_IF.text = "whatever you want";
For those interested in year 2020 :)
Seems that property interactable is instantly disabled, solution was to enable it so it text could be updated...
I just code it up and then in the editor drag the TMP_InputField object into the corresponding slots in the script.
So, code-wise, I would have script like Milorad Zivanovic shows.
If you want multiple ones you might have:
public TMP_InputField [] myInputs;
In the editor (as described above in my first sentence) you just hit the plus button and drag them in. Or just highlight them all and drag in all at once. If you lock your object in the editor to do this (little lock icon up top right) don't forget to unlock it.
To access in your code you either loop or use an index. Like to get the first one you would do this:
{//... in some method
myInputs[0].text = "something";
}
PS: I did not have to explicitly set the interactable field.
PPS: The answer that HyoJin KIM gave is great and best if you like using Unity's Event. I just gave this answer so you'd have an additional way of doing it that might match your original attempt(s).
Be aware that if you do choose this latter approach it is very easy to make the mistake of choosing the static version of the same method (they are listed twice!) Many times I make this mistake. Choose the top one (non-static, aka Dynamic) for most cases!

CM+MEF: registering MEF exports from external assemblies

this post ideally continues my other post on MEF plugins, but my first post was too full of comments and this sample is more complete. Here I summarize my updated scenario, with all my findings up to this point. Hope this can be useful to other CM newbies like me.
You can download a full repro sample scenario: it's an almost do-nothing dumb skeleton for a CM + MEF plugins-based application:
VS2010 repro solution (updated)
This is a minimal stripped down solution representing my issues with CM+MEF.
There are 3 projects:
the main UI (CmRepro).
a core DLL shared among all the addins (AddinCore), with a couple of interfaces and custom attributes used for MEF metadata.
a sample addin DLL (AlphaAddin), with a view and a viewmodel implementing the interfaces.
The core contains 2 interfaces representing a viewmodel and its view, and 2 attributes to be used for decorating the viewmodels and the views. The viewmodel interface describes a class which should compose a greeting message from some person name, so it exposes a couple of properties and a method for this. The view interface just exposes a property returning its DataContext cast to the viewmodel interface.
The sample addin has an implementation for a viewmodel and a view; both are MEF-exports, decorated by the corresponding attribute. In the real-world solution several properties of these attributes are used for filtering; here I just have a dummy Language property which should allow for other plugins for different languages.
The main UI has a MEF bootstrapper which adds code for retrieving MEF exports from an Addins folder. I modified this code to include exports from MEF directories and get a better understanding of some MEF exceptions, but still I cannot figure out how to properly "register" them with CM.
The main viewmodel has 2 methods: one (A) uses a MEF catalog to retrieve a viewmodel and its view, bind them and show them into a window. Another (B) uses the same catalog to get a viewmodel, and then a CM window manager to locate, create, bind and show the corresponding view according to CM naming conventions. These methods represent two alternative ways I should deal with in my real-world code, i.e. instantiating some crucial objects "by myself" just using MEF but then let them work for CM, or letting CM (with a MEF-bootstrapper) do most of the work starting from a viewmodel.
Anyway, it seems that in both cases I am missing something as for registration with CM. Issues:
(A) how do I wire up VM+V for CM so that the conventions for databinding etc are applied? At this time I can build my MEF parts together, but CM ignores them as it was not used to instantiate none of them.
I answer to myself here:
ViewModelBinder.Bind(viewmodel, (UserControl)view, null);
(B) how do I register the exports from MEF in CM so that the CM window manager can find the view? Currently it does not manage to locate the view from the viewmodel.
Addition (21 jun)
I try to explain better for whom cannot access the repro solution. I use a "standard" MEF bootstrapper, changing the Configure override like:
_container = new CompositionContainer(
new AggregateCatalog(AssemblySource.Instance.Select(
x => new AssemblyCatalog(x)).OfType()
.Union(GetAddinDirectoryCatalogs())));
this creates a MEF composition container which aggregates the catalog from AssemblySource, with CM types like event aggregator or window manager, with a catalog from several addin directories, which contain exports for both V and VM's.
In my sample main viewmodel I create a new VM from a plugin, found in the host application directory among others, and I'd like a CM window manager to locate, instantiate and show its view in a dialog, e.g.:
viewmodel = GetMyViewModelFromAddin();
windowmanager.ShowDialog(viewmodel);
CM anyway cannot locate the view. AFAIK, naming conventions are honored: both V and VM are in the same addin assembly, marked as MEF exports, named like SomethingViewModel / SomethingView. Anyway, as Leaf pointed out in his clarification, AssemblySource.Instance is a static IObservableCollection collection of assemblies and I have not added my addins to it. But this is right the point: I would not like to add all of them in advance, because this means loading ALL the addins without yet knowing which (if any) will be ever used. A robust plugin system is the reason for using MEF, after all. I'm new to CM and not sure if it is possible (and where) to find an extension point for CM in this scenario. The window manager does not call my bootstrapper implementation at all, clearly because there is nothing to be instantiated by IoC, as no match was found in assembly source Instance. So, it seems I'm stuck here, the only solution being loading in advance all the assemblies in the Instance, but this seems defeating the whole purpose of using MEF.
The plugin-based application I'm developing loads tons of V+VM CM "pairs" representing user interface widgets, which in turn often use a window manager to popup other V+VM's pairs as dialogs. I can bypass instantiation by CM and use MEF to retrieve V+VM for each widget, but still I'm facing the same view location issue for each widget requiring a window manager. The other alternative (workaround) I can see is avoding the use of window manager and implement my own mechanism for the purpose of showing dialogs from widgets, but this makes feel me a little wrong about CM...:). Usually when I find myself writing much more code than expected I am inclined to think I'm not using the tool in the right way. Any idea?
Caliburn.Micro goes through 3 stages to find a view from a viewmodel instance.
Does text transforms to transform view type name to viewmodel type name. There are a set of standard conventions for these transformations, e.g. SomeNamespace.ViewModels.CustomerViewModel might map to SomeNamespace.Views.CustomerView.
With the view type name(s), Caliburn.Micro then uses AssemblySource.Instance (a static IObservableCollection<Assembly> collection of assemblies) to find the first matching Type.
Caliburn.Micro attempts to create an instance of that Type via one of the IoC.GetInstance() methods (which delegate to your bootstrapper and hence MEF).
I'm guessing (your file share site is blocked here) that the problem with resolving views from viewmodels is due to the second step and the AssemblySource.Instance collection not containing your dynamic assembly.
One solution might be to add each dynamically loaded addin's assembly to AssemblySource.Instance when they are loaded, or if you know all assemblies at startup then you can override the Bootstrapper.SelectAssemblies method to return list of assemblies you expect to contain views and viewmodels.
Update to show how you could pull assemblies from MEF loaded parts
If you are using DirectoryCatalog to load your parts from other assemblies then you could find the assemblies used like this:
var directoryCatalog = new DirectoryCatalog(#"./");
AssemblySource.Instance.AddRange(
directoryCatalog.Parts
.Select(part => ReflectionModelServices.GetPartType(part).Value.Assembly)
.Where(assembly => !AssemblySource.Instance.Contains(assembly)));
If your addins folder changes during the runtime of the application then you would have to DirectoryCatalog.Refresh() the catalog and run the code to add any new assemblies to AssemblySource.Instance
I found a workaround. It's not that pretty, but it allows CM and its window manager to work. I summarize here my findings, hope this will help others or let someone point me to a better solution.
Given that (a) I do not want to load ALL the assemblies including all their dependencies in my plugins folder, to avoid polluting my app domain with unused stuff; and that (b) the only available CM extension point for this seems SelectAssemblies, my goal is add my assemblies there, but only the plugin assemblies requiring to be registered with CM.
So I started looking for a way of loading all the DLL's into a temporary app domain, scan them for some aspect marking them as plugins, and then load into the current app domain only the plugin ones, passing them to SelectAssemblies. This is far from being the optimal solution, as I cannot use a general purpose scanning way like MEF, and I feel I'm duplicating the effort, but at least it's a working solution.
First, to provide at least a way for loading only the plugins, I decorate my plugin assemblies requiring CM registration with a custom attribute marking them as plugins, and further enumerate their types looking for the ones which should be later used by MEF.
The attribute is as simple as:
[AttributeUsage(AttributeTargets.Assembly)]
public class AssemblyRegisteredWithCMAttribute : Attribute {}
I then found this very good piece of code for scanning assemblies into another temporary AppDomain:
http://sachabarber.net/?p=560
I had to modify it a bit, because it was failing when loading assemblies with dependencies, and it was scanning the assemblies types while in my case I just need to check for the attribute.
Here is my code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Security.Policy;
using System.Reflection;
using System.Diagnostics.CodeAnalysis;
namespace CmRepro
{
///
/// Separate AppDomain assembly loader.
///
/// Modified from http://sachabarber.net/?p=560 .
public class SeparateAppDomainAssemblyLoader
{
///
/// Loads an assembly into a new AppDomain returning the names of the files
/// containing assemblies marked with the assembly attribute name matching
/// the specified name. The new AppDomain is then Unloaded.
///
/// list of files to load
/// assemblies directory
/// matching attribute name
/// list of found namespaces
/// null files, assembly directory or matching attribute
public List LoadAssemblies(string[] aFiles, string sAssemblyDirectory, string sMatchingAttribute)
{
if (aFiles == null) throw new ArgumentNullException("aFiles");
if (sAssemblyDirectory == null) throw new ArgumentNullException("sAssemblyDirectory");
if (sMatchingAttribute == null) throw new ArgumentNullException("sMatchingAttribute");
List<String> namespaces = new List<String>();
AppDomain childDomain = BuildChildDomain(AppDomain.CurrentDomain);
try
{
Type loaderType = typeof(AssemblyLoader);
if (loaderType.Assembly != null)
{
AssemblyLoader loader = (AssemblyLoader)childDomain.
CreateInstanceFrom(
loaderType.Assembly.Location,
loaderType.FullName).Unwrap();
namespaces = loader.LoadAssemblies(aFiles, sAssemblyDirectory, sMatchingAttribute);
} //eif
return namespaces;
}
finally
{
AppDomain.Unload(childDomain);
}
}
/// <summary>
/// Creates a new AppDomain based on the parent AppDomains
/// Evidence and AppDomainSetup.
/// </summary>
/// <param name="parentDomain">The parent AppDomain</param>
/// <returns>A newly created AppDomain</returns>
private AppDomain BuildChildDomain(AppDomain parentDomain)
{
Evidence evidence = new Evidence(parentDomain.Evidence);
AppDomainSetup setup = parentDomain.SetupInformation;
return AppDomain.CreateDomain("DiscoveryRegion", evidence, setup);
}
/// <summary>
/// Remotable AssemblyLoader, this class inherits from <c>MarshalByRefObject</c>
/// to allow the CLR to marshall this object by reference across AppDomain boundaries.
/// </summary>
private class AssemblyLoader : MarshalByRefObject
{
private string _sRootAsmDir;
/// <summary>
/// ReflectionOnlyLoad of single Assembly based on the assemblyPath parameter.
/// </summary>
/// <param name="aFiles">files names</param>
/// <param name="sAssemblyDirectory">assemblies directory</param>
/// <param name="sMatchingAttribute">matching attribute name</param>
[SuppressMessage("Microsoft.Performance", "CA1822:MarkMembersAsStatic")]
internal List<string> LoadAssemblies(string[] aFiles, string sAssemblyDirectory, string sMatchingAttribute)
{
AppDomain.CurrentDomain.ReflectionOnlyAssemblyResolve += OnReflectionOnlyAssemblyResolve;
_sRootAsmDir = sAssemblyDirectory;
List<string> aAssemblies = new List<String>();
try
{
sMatchingAttribute = "." + sMatchingAttribute;
foreach (string sFile in aFiles)
Assembly.ReflectionOnlyLoadFrom(sFile);
aAssemblies.AddRange(from asm in AppDomain.CurrentDomain.ReflectionOnlyGetAssemblies()
let attrs = CustomAttributeData.GetCustomAttributes(asm)
where attrs.Any(a => a.ToString().Contains(sMatchingAttribute))
select asm.FullName);
return aAssemblies;
}
catch (FileNotFoundException)
{
/* Continue loading assemblies even if an assembly
* can not be loaded in the new AppDomain. */
return aAssemblies;
}
}
private Assembly OnReflectionOnlyAssemblyResolve(object sender, ResolveEventArgs e)
{
// http://blogs.msdn.com/b/junfeng/archive/2004/08/24/219691.aspx
System.Diagnostics.Debug.WriteLine(e.Name);
AssemblyName name = new AssemblyName(e.Name);
string sAsmToCheck = Path.GetDirectoryName(_sRootAsmDir) + "\\" + name.Name + ".dll";
return File.Exists(sAsmToCheck)
? Assembly.ReflectionOnlyLoadFrom(sAsmToCheck)
: Assembly.ReflectionOnlyLoad(e.Name);
}
}
}
}
Now in my bootstrapper I override the SelectAssemblies method as follows:
...
protected override IEnumerable SelectAssemblies()
{
string sAddinPath = GetAbsolutePath(ADDIN_PATH);
FileCheckList list = new FileCheckList(sAddinPath);
// check only DLL files which were added or changed since last check
SeparateAppDomainAssemblyLoader loader = new SeparateAppDomainAssemblyLoader();
List<string> aAssembliesToRegister = loader.LoadAssemblies(list.GetFiles(null),
sAddinPath, "AssemblyRegisteredWithCM");
string[] aFilesToRegister = (from s in aAssembliesToRegister
select Path.Combine(sAddinPath, s.Substring(0, s.IndexOf(',')) + ".dll")).ToArray();
// update checklist
foreach (string sFile in aFilesToRegister) list.SetCheck(sFile, true);
list.UncheckAllNull();
list.Save();
// register required files
return (new[]
{
Assembly.GetExecutingAssembly(),
}).Union((from s in list.GetFiles(true)
select Assembly.LoadFrom(s))).ToArray();
}
...
As you can see, I am calling the loader not for all the DLL's in my addins path, but only for the ones which a cached files checklist tells me have been added or modified in that folder since the last full scan. This should speed up things a bit and does not require the checklist file to exist: if not found it will be recreated at startup by scanning all the files, if found only added or changed files will be scanned again (I use a CRC to detect changes). So I get the addins folder, create a files checklist for that folder, get from it a list of new or changed files, and pass it to the assemblies loader. This returns only the names of the DLL files which should be registered (i.e. the ones containing assemblies marked with my attribute); I then update the checklist for the next startup, and register only the required files. This way I can let my addin VM's use window managers and correctly locate the view for each required viewmodel. Somewhat ugly, but working. Thanks again to Leaf, who explained me the working of CM.