Does anyone know anything about using Kinect input for Unity3d with the official SDK? I've been assigned a project to try and integrate these two, but my super doesn't want me to use the open Kinect stuff. Last news out of the Unity site was that Kinect SDK requires 4.0 .Net and Unity3D only takes 3.5
Workarounds? Point me toward resources if you know anything about it please.
The OpenNI bindings for Unity are probably the best way to go. The NITE skeleton is more stable than the Microsoft Kinect SDK, but still requires calibration (PrimeSense mentioned that they'll have a calibration-free skeleton soon).
There are bindings to OpenNI from the Kinect SDK, that make the Kinect SDK work like SensorKinect, this module also exposes The KinectSDK calibration-free skeleton as an OpenNI module:
https://www.assembla.com/code/kinect-mssdk-openni-bridge/git/nodes/
Because the KinectSDK also provides ankles and wrists, and OpenNI already supported it (even though NITE didn't support it) all the OpenNI stuff including Unity character rigs that had included the ankles and wrists just all work and without calibration. The KinectSDK bindings for OpenNI also support using NITE's skeleton and hand trackers, with one caveat, it seems like the NITE gesture detection aren't working with the Kinect SDK yet. The work-around when using the KinectSDK with NITE's handGenerator is to use skeleton-free tracking to provide you with a hand point. Unfortunately you lose the ability to just track hands when your body isn't visible to the sensor.
Still, NITE's skeleton seems more stable and more responsive than the KinectSDK.
How much of the raw Kinect data do you need? For a constrained problem, like just getting limb articulation, have you thought about using an agnostic communication schema like a TcpClient. Just create a simple TCP server, in .net 4.0, that links to the Kinect SDK and pumps out packets w/ the info you need every 30ms or something. Then just write a receiving client in Unity. I had a similar problem with a different SDK. I haven't tried the Kinect though so maybe my suggestion is overkill.
If you want real-time depth/color data you might need something a bit faster, perhaps using Pipes?
Related
How does one go about creating a Face Swap mechanism in Flutter?
Can anyone point me in the right direction?
Thank you
You’ll probably need a good plugin to do all the hard work for you. I recommend Google’s ML Kit on Flutter, as it is the most popular way to run on-device ML with Flutter.
The face detection plugin is what you want. You would basically get the face oval shape with face contour detection and swap those shapes. And this can be done real-time with a given video input.
But you should keep in mind that the plugin is on v0.0.1. If you’re aiming for production, you’d better do that with Swift or Kotlin.
There are multiple ways to archive this thing in Flutter. It might be in real-time or with some delay of seconds.
You can use one of these packages.
Open CV
TensorFlow
Google's ML Kit
It might be possible you will not get good support from openCV and TensorFlow in a flutter. But you can integrate the OpenCV/TensorFlow native libs or SDK for both Android and IOS and invoke them through platform channels
There is also one more possible solution but it will definitely have a delay. For this kind of ML project python have great support of library and projects.
You can set up a python project which is responsible for face swapping it takes input from the flutter app (using rest API or socket) and return the output image after face-swapping.
Some great face swap projects are available on GitHub you can look into it.
I'm a bit lost looking through all the various Agora.io modules (and not sure what it means that only some of them have Unity-specific downloads).
I want to make a Unity app where two remote phones exchange data as follows:
Streaming voice in both directions
Streaming video in one direction (recorded from device camera)
Streaming a small amount of continuously-changing custom data in the other direction (specifically, a position + orientation in a virtual world; probably encoded as 7 floats)
The custom data needs to have low latency but does not need reliability (it's fine if some updates get lost; app only cares about the most recent update). Updates basically every frame.
Ideally I want to support both Android and iOS.
I started looking at Agora video (successfully built a test project) and it seems like it will cover the voice and video, but I'm struggling to find a good way to send the custom data (position + orientation). It's probably theoretically possible to encode it as a custom video feed but that sounds complex and inefficient. Is there some out-of-band signalling mechanism I could use to send some extra data alongside/instead of a video?
Agora real-time messaging sounds like it would probably work for this, but I can't seem to find any info about integrating it with Unity (either on Agora's web site or in a general web search). Can I roll this in somehow?
Agora interactive gaming could maybe also be relevant? The overview doesn't seem real clear about how it's different from regular Agora video. I suspect it's overkill but that might be fine if there isn't a large performance cost.
Could anyone point me in the right direction?
I would also consider alternatives to Agora if there's a better plugin for implementing this feature set in Unity.
Agora's Video SDK for Unity supports exporting projects to Android, iOS, MacOS, and Windows (non-UWP).
Regarding your data streaming needs, Agora's RTM SDK is in the process of being ported to work within Unity. At the moment the best way to send data using the Agora SDK is to use CreateDataStream to leverage Agora's ability to open a data stream that is sent along with the frames. Data stream messages are limited to 1kb per frame and 30kb/s so I would be cautious about running it on every frame if you are using a frame-rate above 30fps.
I just started C++ lessons not so long ago because I am interested in game development. Obviously I am nowhere near creating my own game yet but I was reading some guides on game development and it was talking about Unity and how it makes game development a lot easier than having to code 100% of the game.
However, I also watched some tutorials on youtube regarding SFML which can also be used for game developing, what are the differences? and can you use both at the same time?
I have already started learning about SFML but Unity seems so much simpler to use.
Thanks in advance.
In theory yes, but it's not practical, so the real answer is no.
SFML provides access to basic APIs (OpenGL, OpenAL, OS APIs, FreeType 2, etc) which can be used to build complex applications on top, for example like an engine more in the direction of Unity (e.g. xygine, Nero Game Engine).
But since Unity already provides access to all the things SFML would provide, there's really no reason in trying to integrate SFML into Unity. Similar to how it wouldn't make sense to integrate the Unreal Engine into Unity.
If you really want to do it, you'd approach it similar to how #Programmer described it in the comments.
I am a graphics/web designer with basic JS/php coding knowledge and I am interested in learning to make interactive walls.
I would like to know from anyone experienced at this.
What tools, languages do you use?
Unity, Flash, Cinder....etc. which makes it easier?
Thanks
If you just want basic interaction, po-motion.com is a really easy place to start. It tracks motion for simple effects like leaves being brushed away or revealing one image under another. It works using blog detection and can be set up with a mac or pc using a USB camera and any display you can connect your computer to. It also supports some versions of the Kinect on Windows.
This would be quite hard to make with "basic knowledge of JS/php". However how I think you would handle this would be to make the application as you would normally, but have it be controlled by touch input. And then your wall would be a touch/pressure controlled. Im not an engineer, just a programmer, so I dont know how you would make the actual wall, but I do know unity has some good syntax for touch input which I have used. This is very broad question, but I would recommend looking into unity's pre-built touch classes and input interpretation.
as you have alluded to in your tags, one solution is to create a gesture-controlled solution with Kinect.
The best starting point for this would be to download the SDK and get the hardware:
http://www.microsoft.com/en-us/kinectforwindows/
The SDK comes with demos and existing working C# code in Kinect Explorer that creates the 'interactive wall' experience (see Controls Basics, documentation here: https://msdn.microsoft.com/en-us/library/dn188701.aspx).
You can practically run the demo and replace the images to get your experience started. Just make sure you have the right specs on your machine (https://www.microsoft.com/en-us/kinectforwindows/purchase/sensor_setup.aspx ), and you have a good screen.
In terms of programming language, there's no better opportunity to learn C# than from these demos :p
My colleague and I are working on a project where we need to use Kinect to control an application made in Unity. We have familiarized ourselves with Kinect and Unity, but we can't find any examples or tutorials about how to use the two together. Does anybody have any useful resources for this? As my project partner says: the more examples, the merrier.
here is a good site with a lot of kinect projects people have worked on, with a bunch being Unity+KinectSDK
http://channel9.msdn.com/search?term=kinect+unity
Kinect with MSSDK - https://www.assetstore.unity3d.com/#/content/7747
Kinect with OpenNI 1.5 - https://www.assetstore.unity3d.com/#/content/7225
Markerless motion capture with a kinect (or asus xtion live pro or carmine), and face animations in unity: http://www.faceshift.com/unity