I recently tried to develop a flutter plugin with cameraX, but I found that there was no way to simply bind Preview to flutter's Texture.
In the past, I only needed use camera.setPreviewTexture(surfaceTexture.surfaceTexture()) to bind camera and texture, now I can't find the api.
camera.setPreviewTexture(surfaceTexture.surfaceTexture())
val previewConfig = PreviewConfig.Builder().apply {
setTargetAspectRatio(Rational(1, 1))
setTargetResolution(Size(640, 640))
}.build()
// Build the viewfinder use case
val preview = Preview(previewConfig).also{
}
preview.setOnPreviewOutputUpdateListener {
// it.surfaceTexture = this.surfaceTexture.surfaceTexture()
}
// how to bind the CameraX Preview surfaceTexture and flutter surfaceTexture?
I think you can bind texture by Preview.SurfaceProvider.
final CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
final ListenableFuture<ProcessCameraProvider> listenableFuture = ProcessCameraProvider.getInstance(appCompatActivity.getBaseContext());
listenableFuture.addListener(() -> {
try {
ProcessCameraProvider cameraProvider = listenableFuture.get();
Preview preview = new Preview.Builder()
.setTargetResolution(new Size(720, 1280))
.build();
cameraProvider.unbindAll();
Camera camera = cameraProvider.bindToLifecycle(appCompatActivity, cameraSelector, preview);
Preview.SurfaceProvider surfaceProvider = request -> {
Size resolution = request.getResolution();
surfaceTexture.setDefaultBufferSize(resolution.getWidth(), resolution.getHeight());
Surface surface = new Surface(surfaceTexture);
request.provideSurface(surface, ContextCompat.getMainExecutor(appCompatActivity.getBaseContext()), result -> {
});
};
preview.setSurfaceProvider(surfaceProvider);
} catch (Exception e) {
e.printStackTrace();
}
}, ContextCompat.getMainExecutor(appCompatActivity.getBaseContext()));
Update: CameraX has added functionality which will now allow this since this answer was written, but this might still be useful to someone. See this answer for details.
It seems as though using CameraX is difficult to impossible due to it abstracting the more complicated things away and so not exposing things you need like being able to pass in your own SurfaceTexture (which is normally created by Flutter).
So the simple answer is that you can't use CameraX.
That being said, with some work you may be able to get this to work, but I have no idea if it will work for sure. It's ugly and hacky so I wouldn't recommend it. YMMV.
If we're going to do this, let's first look at how the flutter view creates a texture
#Override
public TextureRegistry.SurfaceTextureEntry createSurfaceTexture() {
final SurfaceTexture surfaceTexture = new SurfaceTexture(0);
surfaceTexture.detachFromGLContext();
final SurfaceTextureRegistryEntry entry = new SurfaceTextureRegistryEntry(nextTextureId.getAndIncrement(),
surfaceTexture);
mNativeView.getFlutterJNI().registerTexture(entry.id(), surfaceTexture);
return entry;
}
Most of that is replicable, so we may be able to do it with the surface texture the camera gives us.
You can get ahold of the texture the camera creates this way:
preview.setOnPreviewOutputUpdateListener { previewOutput ->
SurfaceTexture texture = previewOutput.surfaceTexture
}
What you're going to have to do now is to pass a reference to your FlutterView into your plugin (I'll leave that for you to figure out). Then call flutterView.getFlutterNativeView() to get ahold of the FlutterNativeView.
Unfortunately, FlutterNativeView's getFlutterJni is package private. So this is where it gets really hacky - you can create a class in that same package that calls that package-private method in a publicly accesible method. It's super ugly, and you may have to fiddle around with Gradle to get the compilation security settings to allow it, but it should be possible.
After that, it should be simple enough to create a SurfaceTextureRegistryEntry and to register the texture with the flutter jni. I don't think you want to detach from the opengl context, and I really have no idea if this will actually work. But if you want to try it out and report back what you find I would be interested in hearing the result!
Related
Even the official documentation has borderline insane recommendations to solve what is probably one of the most common UI/3D interaction issues:
If I click while the cursor is over a UI button, both the button (via the graphics raycaster) and the 3D world (via the physics raycaster) will receive the event.
The official manual:
https://docs.unity3d.com/Packages/com.unity.inputsystem#1.2/manual/UISupport.html#handling-ambiguities-for-pointer-type-input essentially says "how about you design your game so you don't need 3D and UI at the same time?".
I cannot believe this is not a solved problem. But everything I've tried failed. EventSystem.current.currentSelectedGameObject is sticky, not hover. PointerData is protected and thus not accessible (and one guy offered a workaround via deriving your own class from Standalone Input Module to get around that, but that workaround apparently doesn't work anymore). The old IsPointerOverGameObject throws a warning if you query it in the callback and is always true if you query it in Update().
That's all just mental. Please someone tell me there's a simple, obvious solution to this common, trivial problem that I'm just missing. The graphics raycaster certainly stores somewhere if it's over a UI element, right? Please?
I've looked into this a fair bit and in the end, the easiest solution seems to be to do what the manual says and put it in the Update function.
bool pointerOverUI = false;
void Update()
{
pointerOverUI = EventSystem.current.IsPointerOverGameObject();
}
Your frustration is well founded: there are NO examples of making UI work with NewInput which I've found. I can share a more robust version of the Raycaster workaround, from Youtube:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.InputSystem;
using UnityEngine.UI;
/* Danndx 2021 (youtube.com/danndx)
From video: youtu.be/7h1cnGggY2M
thanks - delete me! :) */
public class SCR_UiInteraction : MonoBehaviour
{
public GameObject ui_canvas;
GraphicRaycaster ui_raycaster;
PointerEventData click_data;
List<RaycastResult> click_results;
void Start()
{
ui_raycaster = ui_canvas.GetComponent<GraphicRaycaster>();
click_data = new PointerEventData(EventSystem.current);
click_results = new List<RaycastResult>();
}
void Update()
{
// use isPressed if you wish to ray cast every frame:
//if(Mouse.current.leftButton.isPressed)
// use wasReleasedThisFrame if you wish to ray cast just once per click:
if(Mouse.current.leftButton.wasReleasedThisFrame)
{
GetUiElementsClicked();
}
}
void GetUiElementsClicked()
{
/** Get all the UI elements clicked, using the current mouse position and raycasting. **/
click_data.position = Mouse.current.position.ReadValue();
click_results.Clear();
ui_raycaster.Raycast(click_data, click_results);
foreach(RaycastResult result in click_results)
{
GameObject ui_element = result.gameObject;
Debug.Log(ui_element.name);
}
}
}
So, just drop into my "Menusscript.cs"?
But as a pattern, this is terrible for separating UI concerns. I'm currently rewiring EVERY separately-concerned PointerEventData click I had already working, and my question is, Why? I can't even find how it's supposed to work: to your point there IS no official guide at all around clicking UI, and it does NOT just drop-on-top.
Anyway, I haven't found anything yet which makes new input work easily on UI, and definitely not found how I'm going to sensibly separate Menuclicks from Activityclicks while keeping game & ui assemblies separate.
Good luck to us all.
Unity documentation for this issue with regard to Unity.InputSystem can be found at https://docs.unity3d.com/Packages/com.unity.inputsystem#1.3/manual/UISupport.html#handling-ambiguities-for-pointer-type-input.
IsPointerOverGameObject() can always return true if the extent of your canvas covers the camera's entire field of view.
For clarity, here is the solution which I found worked best (accumulated from several other posts across the web).
Attach this script to your UI Canvas object:
public class CanvasHitDetector : MonoBehaviour {
private GraphicRaycaster _graphicRaycaster;
private void Start()
{
// This instance is needed to compare between UI interactions and
// game interactions with the mouse.
_graphicRaycaster = GetComponent<GraphicRaycaster>();
}
public bool IsPointerOverUI()
{
// Obtain the current mouse position.
var mousePosition = Mouse.current.position.ReadValue();
// Create a pointer event data structure with the current mouse position.
var pointerEventData = new PointerEventData(EventSystem.current);
pointerEventData.position = mousePosition;
// Use the GraphicRaycaster instance to determine how many UI items
// the pointer event hits. If this value is greater-than zero, skip
// further processing.
var results = new List<RaycastResult>();
_graphicRaycaster.Raycast(pointerEventData, results);
return results.Count > 0;
}
}
In class containing the method which is handling the mouse clicks, obtain the reference to the Canvas UI either using GameObject.Find() or a public exposed variable, and call IsPointerOverUI() to filter clicks when over UI.
Reply to #Milad Qasemi's answer
From the docs you have attached in your answer, I have tried the following to check if the user clicked on a UI element or not.
// gets called in the Update method
if(Input.GetMouseButton(0) {
int layerMask = 1 << 5;
// raycast in the UI layer
RaycastHit2D hit = Physics2D.Raycast(Camera.main.ScreenToWorldPoint(Input.mousePosition), Vector2.zero, Mathf.Infinity, layerMask);
// if the ray hit any UI element, return
// don't handle player movement
if (hit.collider) { return; }
Debug.Log("Touched not on UI");
playerController.HandlePlayerMovement(x);
}
The raycast doesn't seem to detect collisions on UI elements. Below is a picture of the Graphics Raycaster component of the Canvas:
Reply to #Lowelltech
Your solution worked for me except that instead of Mouse I used Touchscreen
// Obtain the current touch position.
var pointerPosition = Touchscreen.current.position.ReadValue();
An InputSytem is a way to receive new inputs provided by Unity. You can't use existing scripts there, and you'll run into problems like the original questioner. Answers with code like "if(Input.GetMouseButton(0)" are invalid because they use the old system.
I have seen that there are more answers to similar questions, but none of these helped me. I hope you can.
I have a Unity2D shooter game and I try to make a menu for UNET. I watched a youtube tutorial from Brackeys to make the menu (https://youtu.be/V4oRs26vAw8?list=PLPV2KyIb3jR5PhGqsO7G4PsbEC_Al-kPZ&t=896) and when he runs the project it shows me the error: "error CS0122: 'UnityEngine.Networking.Match.ListMatchResponse' is inaccessible due to its protection level. I found out that this might happen when you access for example a private variable from another function, but I access a function from Unity and I can not modify its type (I believe). Any ideas? Thank you.
public void OnMatchList(bool success, string extendedInfo, ListMatchResponse matchList)
{
status.text = "";
if (matchList == null)
{
status.text = "Couldn't get room list.";
return;
}
ClearRoomList();
foreach(MatchDesc match in matchList.matches)
{
GameObject _roomListItemGO = Instantiate(roomListItemPrefab);
_roomListItemGO.transform.SetParent(roomListParent);
// Have a component sit on the gameobject
// that will take care of setting up the name / amount of users.
// as well as setting up a callback function that will join the game.
roomList.Add(_roomListItemGO);
}
}
'MatchDesc' has been renamed to 'MatchInfoSnapshot'.
Just replace it and it should work fine.
I am able to avoid a collision between my player and my entire plateform with the use of contactFilter2D.SetLayerMask() + rigidBody2D.Cast(Vector2, contactFilter, ...);
But I don't find a way to avoid the collision only if my player try to acces to the plateform from below it (with a vertical jump).
I'm pretty sure I should use the contactFilter2D.setNormalAngle() (after specify the minAngle and maxAngle) but no matter the size of my angles, I can't pass threw it.
This is how I initialize my contactFilter2D.
protected ContactFilter2D cf;
void Start () {
cf.useTriggers = false;
cf.minNormalAngle = 0;
cf.maxNormalAngle = 180;
cf.SetNormalAngle(cf.minNormalAngle, cf.maxNormalAngle);
cf.useNormalAngle = true;
}
void Update () {
}
I use it with
count = rb.Cast(move, contactFilter, hitBuffer, distance + shellRadius);
Any ideas ? If you want more code, tell me. But I don't think it will be usefull to understand the matter.
unity actualy has a ready made component for this: it is a physics component called "Platform Effector 2D" if you drag and drop it on your platform it will immediately work the way you want, and it has adjustable settings for tweaking the parameters. hope this helps!
I've been working on switching my applications from Swing to JavaFX. I've been working on a room escape game which displays a description of the item on which the user clicks. In Swing, I'd subclass JComponent and override the paintComponent(Graphics) method. I could draw the text there, knowing that the method is constantly called to update the screen. However, using the JavaFX Canvas, there is no method that is called constantly, which makes this task harder. I attempted save()ing the GraphicsContext after I drew the images and called restore() when I wanted to remove the text, but to no avail. Here's the important code:
package me.nrubin29.jescape;
import javafx.application.Platform;
import javafx.scene.canvas.Canvas;
import javafx.scene.canvas.GraphicsContext;
import javafx.scene.shape.Rectangle;
import java.util.Timer;
import java.util.TimerTask;
public class RoomPane extends Canvas {
private Room room;
private Toast toast;
public RoomPane() {
super(640, 480);
setOnMouseClicked(e -> {
for (JObject o : room.getObjects()) {
if (o.getBounds().contains(e.getX(), e.getY())) {
toast = new Toast(o.getDescription());
}
}
});
new Timer().schedule(new TimerTask() {
#Override
public void run() {
if (toast == null) {
return;
}
if (toast.decrement()) { // Decrements the internal counter. If the count is 0, this method returns true.
toast = null;
Platform.runLater(() -> getGraphicsContext2D().restore());
}
else {
Platform.runLater(() -> getGraphicsContext2D().strokeText(toast.getText(), 300, 100));
}
}
}, 0, 1000);
}
public void changeRoom(Room room) {
this.room = room;
GraphicsContext g = getGraphicsContext2D();
g.drawImage(room.getBackground(), 0, 0);
for (JObject o : room.getObjects()) {
g.drawImage(o.getImage(), getCenterX(o.getBounds()), getCenterY(o.getBounds()));
}
g.save();
}
}
I attempted save()ing the GraphicsContext after I drew the images and called restore() when I wanted to remove the text, but to no avail.
save and restore have nothing to with removing things like text, what they do is save in a stack the state of various settings like a stroke or fill to use to draw shapes and allow them to be popped off the stack for application later. Those routines don't effect the pixels drawn on the canvas at all.
To remove something from a GraphicsContext, you can either draw over the of it, or clear it. For your code, what you could do is snapshot the canvas node where you are trying to save it, then draw your snapshot image onto the canvas where you are trying to restore it. It is probably not the most efficient way of handling drawing (a smarter routine which just draws only damaged area where the text is would be better, but probably not required for your simple game).
However, using the JavaFX Canvas, there is no method that is called constantly
Rather than using a timer to trigger canvas calls, use a AnimationTimer or a Timeline. The AnimationTimer has a callback method which is invoked every pulse (60 times a second, or as fast as JavaFX can render frames, whichever is the lesser), so it gives you an efficient hook into the JavaFX pulse based rendering system. The Timeline can have keyframes which are invoked at user specified durations and each keyframe can have an event handler callback which is invoked at that duration.
Using the built-in JavaFX animation framework, you don't have to worry about multi-threading issues and doing things like Platform.runLater which overly complicate your code and can easily lead to subtle and serious errors.
On a kind of unrelated note, for a simple game like this, IMO you are probably better off recoding it completely to use the JavaFX scene graph rather than a canvas. That way you will be working at a higher level of abstraction rather than clip areas and repainting damaged paint components.
I have a problem with posting screenshot to FB. Connection is not the problem because I collect, deserialize and show my fb data. Post score and message to friend work fine but I cannot manage to post a shot from my phone. I am using Eclipse for debugging but I don't get any errors. My code look like this:
`
IEnumerator TakeScreenShot()
{
//coroutine because I want to wait until end of the frame for shot
yield return new WaitForEndOfFrame();
//create new texture that will be my screenshot
Texture2D tex = new Texture2D(Screen.width,Screen.height,TextureFormat.RGB24, false);
// read pixels from screen
tex.ReadPixels(new Rect(0,0,Screen.width,Screen.height),0,0);
//and apply them to texture
tex.Apply();
// now we have screenshot
// now we need to encode texture to array of bytes
byte[] buffer = tex.EncodeToPNG();
//so we can save it to storage
// System.IO.File.WriteAllBytes(Application.dataPath + "/screenshots/screen/screen" + ".png", buffer);
//or to convert it to string for posting to facebook
string s = Convert.ToBase64String(buffer);
//????? maybe my string format is not correct
var query = new Dictionary<string, string>();
query["photos"] = s;
FB.API("me?fields=photos",Facebook.HttpMethod.POST, delegate (string r) { ; }, query);
}
`
when I go to GraphExplorer and paste command "me?fields=photos" I get nothing. It means function did nothing. I forgot to say that i granted permission for user_photos. This is frustrating. I have lost three days on problem that looked trivial, and yet no solution in sight. I will appreciate any suggestions.
Right now there's no easy way to handle this through FB.API since it takes in a Dictionary<string, string>.
However FB.API() is really just a wrapper around Unity's WWW class. You can call the graph endpoints at graph.facebook.com and pass in the byte[] buffer = tex.EncodeToPNG() as a param in WWWForm().
UPDATE: FB.API() has been updated in version 4.3.3 to support WWWForm as well. You can now upload photos via that method now.