Modify WebcamTexture from unity native plugin - unity3d

I just want to pass a WebcamTexture to a native plugin (using GetNativeTexturePtr() for performance), draw something on it and render results into a plane on the screen.
I guess that some Unity thread would be updating the WebcamTexture contents every frame so writing to that texture may produce some flickering effects as there would be 2 threads updating its contents. Because of that I'm using another texture "drawTexture" to render the result.
The algorithm is easy, on every render event:
Read camTexture
Copy camTexture contents to drawTexture
Draw things on drawTexture
Render drawTexture (OpenGL bindTexture)
I'm following the NativeRenderingPlugin example, but every time I try to copy contents from camTexture to drawTexture (even only 1 pixel) the main thread freezes.
I think that may be happening because I'm trying to read camTexture while it is being modified by an external thread.
Here it is some code:
C# plugin source
[DllImport ("plugin")]
private static extern void setTexture (IntPtr cam, IntPtr draw, int width, int height);
[DllImport ("plugin")]
private static extern IntPtr GetRenderEventFunc();
WebCamTexture webcamTexture;
Texture2D drawTexture;
GameObject plane;
bool initialized = false;
IEnumerator Start () {
// PlaneTexture
// Create a texture
drawTexture = new Texture2D(1280,720,TextureFormat.BGRA32,true);
drawTexture.filterMode = FilterMode.Point;
drawTexture.Apply ();
plane = GameObject.Find("Plane");
plane.GetComponent<MeshRenderer> ().material.mainTexture = drawTexture;
// Webcam texture
webcamTexture = new WebCamTexture();
webcamTexture.filterMode = FilterMode.Point;
webcamTexture.Play();
yield return StartCoroutine("CallPluginAtEndOfFrames");
}
private IEnumerator CallPluginAtEndOfFrames()
{
while (true) {
// Wait until all frame rendering is done
yield return new WaitForEndOfFrame();
// wait for webcam and initialize
if (!initialized && webcamTexture.width > 16) {
setTexture (webcamTexture.GetNativeTexturePtr(), drawTexture.GetNativeTexturePtr(), webcamTexture.width, webcamTexture.height);
initialized = true;
} else if (initialized) {
// Issue a plugin event with arbitrary integer identifier.
// The plugin can distinguish between different
// things it needs to do based on this ID.
// For our simple plugin, it does not matter which ID we pass here.
GL.IssuePluginEvent (GetRenderEventFunc (), 1);
}
}
}
C++ plugin (draw texture method)
static void ModifyTexturePixels()
{
void* textureHandle = drawTexturePointer;
int width = textureWidth;
int height = textureHeight;
if (!textureHandle)
return;
int textureRowPitch;
void* newEmptyTexturePointer = (unsigned char*)s_CurrentAPI->BeginModifyTexture(textureHandle, width, height, &textureRowPitch);
memcpy(newEmptyTexturePointer, webcamTexturePointer, width * height * 4);
s_CurrentAPI->EndModifyTexture(textureHandle, width, height, textureRowPitch, newEmptyTexturePointer);
}
Any help would be appreciated.

Related

Unity: Reduce size of Render Texture before executing Texture2D.ReadPixels

I'm working on a code where I basically have to take a low quality screenshot about every 30 milliseconds. The script is attached to a camara.
What I want to do is reduce the render texture size. The way the code is right now changing either W or H basically gets me a SECTION of of all that is being seen by the camara instead of a reduced size version. So my question is how can I resized or downsample what is read into the screenshot (Texture2D) but that it still is a representation of the entire screen.
public class CameraRenderToImage : MonoBehaviour
{
private RemoteRenderServer rrs;
void Start(){
TimeStamp.SetStart();
Camera.onPostRender += OnPostRenderCallback;
}
void OnPostRenderCallback(Camera cam){
if (TimeStamp.HasMoreThanThisEllapsed(30)){
TimeStamp.SetStart();
int W = Screen.width;
int H = Screen.height;
Texture2D screenshot = new Texture2D(W,H, TextureFormat.RGB24, false);
screenshot.ReadPixels( new Rect(0, 0, W,H), 0, 0);
byte[] bytes = screenshot.EncodeToPNG();
System.IO.File.WriteAllBytes("check_me_out.png", bytes);
TimeStamp.Tok("Encode to PNG and Save");
}
}
// Remove the onPostRender callback
void OnDestroy()
{
Camera.onPostRender -= OnPostRenderCallback;
}
}
If you need to resize your render texture from script you can refer to the next code snippet
void Resize(RenderTexture renderTexture, int width, int height) {
if (renderTexture) {
renderTexture.Release();
renderTexture.width = width;
renderTexture.height = height;
}
}
To make it possible to resize the render texture you first need to make sure it is released.
To get Texture2d:
private Texture2D ToCompressedTexture(ref RenderTexture renderTexture)
{
var texture = new Texture2D(renderTexture.width, renderTexture.height, TextureFormat.ARGB32, false);
var previousTarget = RenderTexture.active;
RenderTexture.active = renderTexture;
texture.ReadPixels(new Rect(0, 0, renderTexture.width, renderTexture.height), 0, 0);
RenderTexture.active = previousTarget;
texture.Compress(false);
texture.Apply(false, true);
renderTexture.Release();
renderTexture = null;
return texture;
}

Unity, How To SetPixels32 Properly?

I'm receiving the error SetPixels32 called with invalid number of pixels in the array UnityEngine.Texture2D:SetPixels32(Color32[]) on a line of code that is trying to retrieve a pixel array.
In some instances I receive the error but in others the webcams stream just fine. I'm not sure why this is occurring. This is the line of code that is giving me problems:
streamTexture.SetPixels32(webcamTexture.GetPixels32(pixels))
That isn't much to go on, however, below is the full script. If Anyone can tell me why this error is occurring since the streaming texture is set to the dimensions of the webcam texture. Any help is much appreciated!
using System.Collections;
using UnityEngine;
using Photon.Pun;
using UnityEngine.UI;
public class WebcamStream : MonoBehaviourPun, IPunObservable
{
[SerializeField] private LocalPlayerSettings playerSettings;
[SerializeField] private AvatarHandler parent;
[SerializeField] private RawImage streamRawimage;
private WebCamTexture webcamTexture;
private Texture2D streamTexture;
private Color32[] pixels;
private byte[] data;
private void OnEnable()
{
if (!parent.photonView.IsMine)
return;
InitWebcam();
}
private void OnDisable()
{
if (!parent.photonView.IsMine)
return;
webcamTexture.Stop();
}
private void InitWebcam()
{
//cast dimensions of target UI as ints for new webcam texture
int width = (int)streamRawimage.rectTransform.rect.width;
int height = (int)streamRawimage.rectTransform.rect.height;
//set new dimensions and target device
webcamTexture = new WebCamTexture(width, height)
{
deviceName = playerSettings.Webcam
};
//display webcam texture on the raw image and start camera
streamRawimage.material.mainTexture = webcamTexture;
webcamTexture.Play();
//set pixels and stream texture to match webcamTexture
pixels = new Color32[webcamTexture.width * webcamTexture.height];
streamTexture = new Texture2D(webcamTexture.width, webcamTexture.height, TextureFormat.RGB24, false);
//Begin Streaming Webcam Data
StartCoroutine(StreamWebcam());
}
private IEnumerator StreamWebcam()
{
while (webcamTexture.deviceName == playerSettings.Webcam)
{
if (webcamTexture.isPlaying)
{
//set the target texture pixels to the webcam texture pixels and apply get/set
streamTexture.SetPixels32(webcamTexture.GetPixels32(pixels));
streamTexture.Apply();
//convert image to byte array
data = streamTexture.EncodeToJPG();
}
yield return null;
}
webcamTexture.Stop();
if(WebCamTexture.devices.Length > 0)
{
InitWebcam();
}
}
public void OnPhotonSerializeView(PhotonStream stream, PhotonMessageInfo info)
{
if (stream.IsWriting)
{
//send the byte array through the stream
stream.SendNext(data);
}
else
{
//convert object received into byte array via cast
data = (byte[])stream.ReceiveNext();
//create new texture to load received data into
streamTexture = new Texture2D(1, 1, TextureFormat.RGB24, false);
streamTexture.LoadImage(data);
//set webcam raw image texture to the newly updated texture
streamRawimage.texture = streamTexture;
}
}
}
Solved! There was another script responsible for enabling this scripts game object and needed a yield for about 1 second.
Also a quick edit of streamRawimage.material.mainTexture = webcamTexture; to streamRawimage.texture = webcamTexture; fixed another issue surrounding a missing texture.

Unity code works in the editor but does not work on Build (EXE File) C#

My code load external images and build from them a SkyBox for unity3d.
All the files of the SkyBox are on the right paths (Copied manually).
The code loads 6 external images and then builds a SkyBox And I think there's a problem in the loading when it's in build stat. (But not sure)
Or maybe Unity prevents me from doing it?
And i have no error or warning code. It really drives me crazy!!!
using System.IO;
using UnityEngine;
public class ChangeSkyBox : MonoBehaviour
{
public static Texture2D LoadPNG(string filePath)
{
Texture2D tex = null;
byte[] fileData;
if (File.Exists(filePath))
{
fileData = File.ReadAllBytes(filePath);
tex = new Texture2D(1024, 1024);
tex.LoadImage(fileData);
}
tex.wrapMode = TextureWrapMode.Clamp;
return tex;
}
public static Material CreateSkyboxMaterial(SkyboxManifest manifest)
{
Material result = new Material(Shader.Find("RenderFX/Skybox"));
result.SetTexture("_FrontTex", manifest.textures[0]);
result.SetTexture("_BackTex", manifest.textures[1]);
result.SetTexture("_LeftTex", manifest.textures[2]);
result.SetTexture("_RightTex", manifest.textures[3]);
result.SetTexture("_UpTex", manifest.textures[4]);
result.SetTexture("_DownTex", manifest.textures[5]);
return result;
}
private Texture2D[] textures;
private void Start()
{
Texture2D xt1 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Front.png");
Texture2D xt2 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Back.png");
Texture2D xt3 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Left.png");
Texture2D xt4 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Right.png");
Texture2D xt5 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Top.png");
Texture2D xt6 = LoadPNG(Directory.GetCurrentDirectory() + "\\Assets\\SkyBox\\Bottom.png");
SkyboxManifest manifest = new SkyboxManifest(xt1, xt2, xt3, xt4, xt5, xt6);
Material newMat = new Material(Shader.Find("RenderFX/Skybox"));
newMat = CreateSkyboxMaterial(manifest);
RenderSettings.skybox = newMat;
DynamicGI.UpdateEnvironment();
}
}
public struct SkyboxManifest
{
public Texture2D[] textures;
public SkyboxManifest(Texture2D front, Texture2D back, Texture2D left, Texture2D right, Texture2D up, Texture2D down)
{
textures = new Texture2D[6]
{
front,
back,
left,
right,
up,
down
};
}
}
I believe your problem lies on this Line.
Material result = new Material(Shader.Find("RenderFX/Skybox"));
Unity cannot find it at runtime. To fix it, make the "base" Material by hand in Unity and attach it to your script throurgh the Inspector.

Take photo in unity c#

I'm trying to build a program that takes your photo and places it with a different background, like a monument or so. So far, I was able to turn the camera on when I start the project with this code
webcamTexture = new WebCamTexture();
rawImage.texture = webcamTexture;
rawImage.material.mainTexture = webcamTexture;
webcamTexture.Play();
Texture2D PhotoTaken = new Texture2D (webcamTexture.width, webcamTexture.height);
PhotoTaken.SetPixels (webcamTexture.GetPixels ());
PhotoTaken.Apply ();
However, I can't take a screenshot or photo because it always ends up all black. I've tried different codes but nothing is working. Can someone please help? Thanks
EDIT
After some tries, this is the code I have:
WebCamTexture webcamTexture;
public RawImage rawImage;
void Start () {
webcamTexture = new WebCamTexture();
rawImage.texture = webcamTexture;
rawImage.material.mainTexture = webcamTexture;
webcamTexture.Play();
RenderTexture texture= new RenderTexture(webcamTexture.width, webcamTexture.height,0);
Graphics.Blit(webcamTexture, texture);
Button btn = yourButton.GetComponent<Button>();
btn.onClick.AddListener(OnClick);
}
public IEnumerator Coroutine(){
yield return new WaitForEndOfFrame ();
}
public void OnClick() {
var width = 767;
var height = 575;
Texture2D texture = new Texture2D(width, height);
texture.ReadPixels(new Rect(0, 0, width, height), 0, 0);
texture.Apply();
// Encode texture into PNG
var bytes = texture.EncodeToPNG();
//Destroy(texture);
File.WriteAllBytes (Application.dataPath + "/../SavedScreen.png", bytes);
}
and with this next code the screenshot is taken, but it takes a photo of the whole thing, and not just a bit of the screen.
void Start()
{
// Set the playback framerate!
// (real time doesn't influence time anymore)
Time.captureFramerate = frameRate;
// Find a folder that doesn't exist yet by appending numbers!
realFolder = folder;
int count = 1;
while (System.IO.Directory.Exists(realFolder))
{
realFolder = folder + count;
count++;
}
// Create the folder
System.IO.Directory.CreateDirectory(realFolder);
}
void Update()
{
// name is "realFolder/shot 0005.png"
var name = string.Format("{0}/shot {1:D04}.png", realFolder, Time.frameCount);
// Capture the screenshot
Application.CaptureScreenshot(name, sizeMultiplier);
}
}
You can take a screenshot like this in Unity
Application.CaptureScreenshot("Screenshot.png");
Reference
EDIT 1
To take a screenshot on a specific part of the screen use the following script:
var width = 400;
var height = 300;
var startX = 200;
var startY = 100;
var tex = new Texture2D (width, height, TextureFormat.RGB24, false);
tex.ReadPixels (Rect(startX, startY, width, height), 0, 0);
tex.Apply ();
// Encode texture into PNG
var bytes = tex.EncodeToPNG();
Destroy(tex);
File.WriteAllBytes(Application.dataPath + "/../SavedScreen.png", bytes);
Reference

EmguCV + Unity Open WebCam is Error;

I use EmguCV open webcam in unity.
But it's fps is low much.
this is my code ↓
private Texture2D texture;
private Capture capture;
private Color32[] color = new Color32[640*480];
// Use this for initialization
void Start () {
texture = new Texture2D (640, 480);
capture = new Capture ();
}
// Update is called once per frame
void Update () {
Image<Bgr, Byte> currentFrame = capture.QueryFrame();
Bitmap bitmapCurrentFrame = currentFrame.ToBitmap();
Image<Bgra, Byte> img = new Image<Bgra, Byte> (bitmapCurrentFrame);
for(int y=0; y<480; y++){
for(int x=0; x<640; x++){
int index = y+x*480;
print(index+";"+x+";"+y);
//byte b = img.Data[x,y,0];
color[index].r = img.Data[x,y,2];
color[index].g = img.Data[x,y,1];
color[index].b = img.Data[x,y,0];
color[index].a = 0xff;
}
}
texture.SetPixels32 (color);
texture.Apply (false);
renderer.material.mainTexture = texture;
}
i don't know why fps is so low...
and why my boss like EmguCV with Unity, why he don't use Unity-WebCamTexture...
OKAY,i really thank you for your read.
Hope, I can get some answer.
Look at Texture2D.LoadRawTextureData. It has no proper docs, so here's a snippet:
Texture2D tex = new Texture2D(width, height, format, false, true);
tex.LoadRawTextureData(buffer);
tex.Apply(false, true);
Buffer must be in the correct hardware format. For the format variable look at the list of formats that unity accepts.