Is it possible to get audio data from the users audio device using NAudio to pass to Unity3D for visualization - unity3d

What I am trying to do is grab the raw audio from the users audio output take that data and get an audio spectrum and output array similar to unity3d and pass that data to my visualizer.
So a couple things I need to know:
Can I grab the raw audio from the users device? What I have found so far is yes using waveinstream which by default gets audio data from all devices.
Can I grab the audio spectrum and output from that data similarly to Unity3D GetOutputData() and GetSpectrumData()? the NAudio demo provides similar functionality but not exactly what I want.
I am a newbie with coding naudio, whereas unity's API and extremely thorough documentation make it easier, naudio seems to have a couple tutorials and examples, nothing of what I need, and no API definitions. I will eventually figure it out but what I need to know is if what I'm attempting is possible, other than that any help is appreciated.
I provided my attempt at problem 1 which provides seemingly random data which I am trying to make sense of using a WaveFileReader but that crashes
using UnityEngine;
using System.Collections;
using System;
using NAudio.Wave;
using NAudio.CoreAudioApi;
using NAudio.Utils;
public class myNAudio : MonoBehaviour {
private WaveIn waveInStream;
private WaveFileReader reader;
private WaveStream readerStream;
void waveInStream_DataAvailable(object sender, WaveInEventArgs e){
//reader.Read(e.Buffer,0,e.BytesRecorded);
//Debug.Log(e.Buffer);
float tempDB = 0;
for(int i = 0; i < e.Buffer.Length; i++){
//Debug.Log(i + " = " +e.Buffer[i]);
tempDB += (float)e.Buffer[i]/255;
}
Debug.Log(e.Buffer.Length + ", " +tempDB);
}
void OnApplicationQuit(){
waveInStream.StopRecording();
waveInStream.Dispose();
waveInStream = null;
}
void OnDisable(){
waveInStream.StopRecording();
waveInStream.Dispose();
waveInStream = null;
}
// Use this for initialization
void Start () {
waveInStream = new WaveIn();
waveInStream.DeviceNumber = 0;
waveInStream.DataAvailable += new EventHandler<WaveInEventArgs>(waveInStream_DataAvailable);
waveInStream.StartRecording();
}
// Update is called once per frame
void Update () {
}
}

Related

Connecting Unity client to XMPP server

I'm trying to make a client for chat using unity.
I have found this library https://github.com/bilelmnasser/Unity-3D-Xmpp-Protocol-
so I've imported to my project and being testing the code but I can't seem to get it to work.
using UnityEngine;
using Xmpp;
using Xmpp.protocol.client;
public class NewBehaviourScript : MonoBehaviour
{
// Start is called before the first frame update
void Start()
{
var xmppClient = new XmppClientConnection("localhost");
//xmppClient.ConnectServer = "ipServer";
xmppClient.Server = "localhost";
xmppClient.Port = 5222;
xmppClient.Username = "admin#localhost";
xmppClient.Password = "password";
xmppClient.Open();
xmppClient.OnLogin += delegate(object o) { xmppClient.Send(new Message("user#localhost", MessageType.chat, "Hello, how are you?")); };
}
Here is my code. I have a ejabberd server running using docker.
I've created two users and tested on node server as client and it worked fine, but can't get to work on unity...
any help would be great!

I want to make a P2P networked (thinking about using mirror) game with unity, with no dedicated server, similar to Terraria and Valheim

I have managed to basically connect with my friend over internet, by forwarding my IP address from my router settings... this is not viable because there are few people willing to do what I did to play games with their friends. So how to actually do UDP hole punching (basically what I did manually to my router) in unity using the mirror networking solution...
A common solution to this problem is WebRTC, which takes care of the hole punching under the hood. Unity maintains this package which implements WebRTC. They also provide a great tutorial on how to use it. The nuts and bolts of it are:
using UnityEngine;
using Unity.WebRTC;
public class MyPlayerScript : MonoBehaviour
{
RTCPeerConnection localConnection, remoteConnection;
RTCDataChannel sendChannel, receiveChannel;
private void Awake()
{
// Initialize WebRTC
WebRTC.Initialize();
// Create local peer
localConnection = new RTCPeerConnection();
sendChannel = localConnection.CreateDataChannel("sendChannel");
channel.OnOpen = handleSendChannelStatusChange;
channel.OnClose = handleSendChannelStatusChange;
// Create remote peer
remoteConnection = new RTCPeerConnection();
remoteConnection.OnDataChannel = ReceiveChannelCallback;
// register comms paths
localConnection.OnIceCandidate = e => {
!string.IsNullOrEmpty(e.candidate)
|| remoteConnection.AddIceCandidate(ref e);
}
remoteConnection.OnIceCandidate = e => {
!string.IsNullOrEmpty(e.candidate)
|| localConnection.AddIceCandidate(ref e);
}
localConnection.OnIceConnectionChange = state => {
Debug.Log(state);
}
}
//handle begin
IEnumerator Call(){
var op1 = localConnection.CreateOffer();
yield return op1;
var op2 = localConnection.SetLocalDescription(ref op1.desc);
yield return op2;
var op3 = remoteConnection.SetRemoteDescription(ref op1.desc);
yield return op3;
var op4 = remoteConnection.CreateAnswer();
yield return op4;
var op5 = remoteConnection.setLocalDescription(op4.desc);
yield return op5;
var op6 = localConnection.setRemoteDescription(op4.desc);
yield return op6;
}
//handle send messages
void SendMessage(string message)
{
sendChannel.Send(message);
}
void SendBinary(byte[] bytes)
{
sendChannel.Send(bytes);
}
//handle receive messages
void ReceiveChannelCallback(RTCDataChannel channel)
{
receiveChannel = channel;
receiveChannel.OnMessage = HandleReceiveMessage;
}
void HandleReceiveMessage(byte[] bytes)
{
var message = System.Text.Encoding.UTF8.GetString(bytes);
Debug.Log(message);
}
//handle end
private void OnDestroy()
{
sendChannel.Close();
receiveChannel.Close();
localConnection.Close();
remoteConnection.Close();
WebRTC.Finalize();
}
}
I have also found a useful way to do this using mirror and the epic free relay for this. Thanks so much for the other answers, it really helped understand better what I needed to search and use!
You can use Photon Bolt or Photon Fusion to let players host a game on their local machine like minecraft, etc. Photon provides relay as well as tries to use STUN to establish direct peer to peer connection to the host via UDP. PUN2 is also a good choice, although I like Photon Bolt/Fusion better - it's less of a simple RPC framework and more programmer oriented. Also, PUN does not do any STUN direct peer connection, it will always be relayed. Photon Bolt and Fusion will first attempt a STUN direct peer connection and then fallback to relay if necessary. It's been around for years and is the best choice.
Sure you can develop a Unity game with Mirror (although without a relay built in) but it's not nearly as easy to setup and use as Photon Bolt/Fusion and they don't provide a relay. As someone mentioned, you might be able to hack something together in some way but yeah - not recommended.
Ugh, yeah don't use WebRTC for a Unity game (or probably anything other than streaming music/video like it was made for to be honest).
Unity's MLAPI is "under development" and their last API was suddenly dropped "deprecated", so I wouldn't use that.

Play sounds synchronously using snd_pcm_writei

I need to play sounds upon certain events, and want to minimize
processor load, because some image processing is being done too, and
processor performance is limited.
For the present, I play only one sound at a time, and I do it as
follows:
At program startup, sounds are read from .wav files
and the raw pcm data are loaded into memory
a sound device is opened (snd_pcm_open() in mode SND_PCM_NONBLOCK)
a worker thread is started which continously calls snd_pcm_writei()
as long as it is fed with data (data->remaining > 0).
Somewhat resumed, the worker thread function is
static void *Thread_Func (void *arg)
{
thrdata_t *data = (thrdata_t *)arg;
snd_pcm_sframes_t res;
while (1)
{ pthread_mutex_lock (&lock);
if (data->shall_stop)
{ data->shall_stop = false;
snd_pcm_drop (data->pcm_device);
snd_pcm_prepare (data->pcm_device);
data->remaining = 0;
}
if (data->remaining > 0)
{ res = snd_pcm_writei (data->pcm_device, data->bufptr, data->remaining);
if (res == -EAGAIN) continue;
if (res < 0) // error
{ fprintf (stderr, "snd_pcm_writeX() error: %s\n", snd_strerror(result));
snd_pcm_recover (data->sub_device, res);
}
else // another chunk has been handed over to sound hw
{ data->bufptr += res * bytes_per_frame;
data->remaining -= res;
}
if (data->remaining == 0) snd_pcm_prepare (data->pcm_device);
}
pthread_mutex_unlock (&lock);
usleep (sleep_us); // processor relief
}
} // Thread_Func
Ok, so this works well for one sound at a time. How do I play various?
I found dmix, but it seems a tool on user level, to mix streams coming
from separate programs.
Furthermore, I found the Simple Mixer Interface in the ALSA Project C
Library Interface, without any hint or example or tutorial about how
to use all these function described by one line of text each.
As a last resort I could calculate the mean value of all the buffers
to be played synchronously. So long I've been avoiding that, hoping
that an ALSA solution might use sound hardware resources, thus
relieving the main processor.
I'd be thankful for any hint about how to continue.

Write to HID with Chip Selection with .NET Console App

Hi I am writing a simple console app that needs to write bytes to MCP2210 USB to SPI Master
I found this library over here, seems to do good job with connecting the device and reading the metadata.
I am writing message to the board as below
public static byte[] Talk()
{
var device = DeviceList.Local.GetHidDevices(1240, 222).FirstOrDefault();
if (device == null)
{
Console.WriteLine($"Could not find a device with Vendor Id:1240, Product Id:222 ");
return null;
}
var reportDescriptor = device.GetReportDescriptor();
foreach (var deviceItem in reportDescriptor.DeviceItems)
{
Console.WriteLine("Opening device for 20 seconds...");
if (!device.TryOpen(out var hidStream))
{
Console.WriteLine("Failed to open device.");
continue;
}
Console.WriteLine("Opened device.");
hidStream.ReadTimeout = Timeout.Infinite;
hidStream.Write(new byte[3] {60, 00, 00});
}
Not sure If I am writing it correctly.
While writing I need to do a chip selection as displayed in this other terminal
Any help is greatly appreciated
Here is the MC I am using https://www.microchip.com/wwwproducts/en/MCP2210
I do not see a closing of your stream. This may cause your data to not even being sent (at least not in time).
Consider using blocks with streams.
But with out parameters not possible.

Photon matchmaking - Join or create a room using in Unity with an SQL lobby

I am trying to implement skill based matchmaking using Photon in Unity. It seems
I got most of this code from the documentation and it works but not well.
The problem is that you can't use JoinOrCreate() with the sql lobby type so my logic here is try and find a room, if it fails create one.
void init()
{
_client = Game.Context.GetComponent<SocketConnectionManager>().client;
joinRoom();
}
public void joinRoom()
{
TypedLobby sqlLobby = new TypedLobby("skillLobby", LobbyType.SqlLobby);
string sqlLobbyFilter = "C0 BETWEEN 100 AND 200";
_client.OpJoinRandomRoom(null, MatchMaker.MaxPlayers, MatchmakingMode.FillRoom, sqlLobby, sqlLobbyFilter);
}
public void createRoom()
{
RoomOptions o = new RoomOptions();
o.MaxPlayers = MatchMaker.MaxPlayers;
o.CustomRoomProperties = new Hashtable() { { "C0", Game.Me.getInt("trophies") } };
o.CustomRoomPropertiesForLobby = new string[] { "C0" }; // this makes "C0" available in the lobby
TypedLobby sqlLobby = new TypedLobby("skillLobby", LobbyType.SqlLobby);
_client.OpCreateRoom("", o, sqlLobby);
}
private void onEvent(EventData obj)
{
if (_client.CurrentRoom != null)
{
if (_client.CurrentRoom.PlayerCount >= _client.CurrentRoom.MaxPlayers)
{
// sweet I am good to go.
}
}
else
{
createRoom();
}
}
The problem is this is pretty unreliable. Say two players try to find a game at the same time they will both search fail and then both create. Now I have two players sitting in empty rooms instead of playing each other.
Any ideas on a better system?
Thanks all.
Thank you for choosing Photon!
First of all, there are few things that you should understand about Photon:
you can't use JoinOrCreate() with the sql lobby type
This is not correct.
Where did you read such thing?
Did you test this yourself? What did you test exactly?
onEvent (LoadBalancingClient.OnEventAction) callback cannot be used to be notified of a failed join random room operation. Instead, you should make use of the LoadBalancingClient.OnOpResponseAction callback, as follows:
private void OnOpResponse(OperationResponse operationResponse)
{
switch (operationResponse.Code)
{
case OperationCode.JoinRandomGame:
if (operationResponse.ReturnCode == ErrorCode.NoMatchFound)
{
createRoom();
}
break;
}
}
To detect a join event inside a room (local or remote player entered a room):
private void onEvent(EventData eventData)
{
switch (eventData.Code)
{
case EventCode.Join:
int actorNr = (int)eventData[ParameterCode.ActorNr];
PhotonPlayer originatingPlayer = this.GetPlayerWithId(actorNr);
if (originatingPlayer.IsLocal)
{
}
else
{
}
break;
}
}
To answer your question:
Say two players try to find a game at the same time they will both
search fail and then both create.
Any ideas on a better system?
No.
This issue happens only during the development phase where you use a few clients to run some tests. Once you have enough user base you won't notice this issue.