I'm trying to make an application that streams video through a gtk draw area. The pipeline I'm currently trying to run is videotestsrc ! ximagesink. My problem is, when I try to run my program, it displays videotestsrc, but only as a still image. This is different from running "gst-launch-1.0 videotestsrc ! ximagesink" through a terminal, where the static in the bottom right moves.
Any ideas on what I'm doing wrong?
int main(int argc, char* argv[])
{
Gst::init(argc, argv);
auto app = Gtk::Application::create(argc, argv, "gtkmm.video.sunshine.test");
Program_Window window;
return app->run(window);
}
class Program_Window : public Gtk::Window
{
public:
Program_Window();
virtual ~Program_Window();
protected:
Gtk::DrawingArea* display;
Glib::RefPtr<Gst::Pipeline> playbin;
gulong window_handler;
GstVideoOverlay* overlay;
void on_display_realize();
};
Program_Window::Program_Window()
{
//initialize variables
display = new Gtk::DrawingArea();
window_handler = 0;
//connect realize callback
display->signal_realize().connect( sigc::mem_fun( *this, &Program_Window::on_display_realize ));
//create playbin
playbin = Gst::PlayBin::create("playbin");
//prepare elements for the pipeline
Glib::RefPtr<Gst::Element> source = Gst::ElementFactory::create_element("videotestsrc", "src");
Glib::RefPtr<Gst::Element> sink = Gst::ElementFactory::create_element("ximagesink", "sink");
//add elements to the pipeline
playbin->add(source)->add(sink);
//link elements
source->link(sink);
//prep video overlay interface
overlay = (GstVideoOverlay*) sink->gobj();
//add drawing area to main window
add(*display);
show_all_children();
}
void Program_Window::on_display_realize()
{
//acquire an xwindow pointer to our draw area
window_handler = GDK_WINDOW_XID( display->get_window()->gobj() );
//give xwindow pointer to our pipeline via video overlay interface
gst_video_overlay_set_window_handle(overlay, window_handler);
//start video
playbin->set_state(Gst::STATE_PLAYING);
}
Could be that in the app it is a tad slower causing following frames to miss their clock times and get discarded. Try setting the sync=false option for the video sink and check if it changes anything. Else use GST_DEBUG to get some logs from the pipeline about what is happening.
P.S. When using Gtk consider using gtksink and gtkglsink to make you life easier.
Fixed it. For whatever reason, the program didn't like the way I used playbin. Changing it to a normal Gst::pipeline worked.
//create playbin //in the above code
//playbin = Gst::PlayBin::create("playbin"); //change this
playbin = Gst::Pipeline::create("pipeline"); //to this
Related
I'm following this tutorial to use Lottie animations library on WatchOS. It's working until I try to change the animation while the app is running. Even though I change it, the animation stays the same until I run the watch app from Xcode. (If I simply close the app and open again it doesn't change.)
What I tried:
clearing the cache from the watch file system
clearing URLCache
clearing SDImageCache memory and disk
You were right that caching causes your problem. However, you didn't go deep enough.
You use SDWebImageLottieCoder to display your animations, which in turn uses rlottie.
According to their readme:
"rlottie is a platform independent standalone c++ library for rendering vector based animations and art in realtime."
The point is that the animation is cached by the rlottie framework, so neither clearing URLCache nor SDImageCache will solve this.
The easiest way to solve this is to modify the code in the rlottie C++ library. In your Xcode project navigate to Pods -> Pods -> librlottie and look for the following file: lottieanimation_capi.cpp
In this file search for the following code:
RLOTTIE_API Lottie_Animation_S *lottie_animation_from_data(const char *data, const char *key, const char *resourcePath)
{
if (auto animation = Animation::loadFromData(data, key, resourcePath) ) {
Lottie_Animation_S *handle = new Lottie_Animation_S();
handle->mAnimation = std::move(animation);
return handle;
} else {
return nullptr;
}
}
This is the code that loads your Lottie animation on your device and is responsible for caching. loadFromData takes an additional fourth parameter called cachePolicy, which is a boolean, and nothing is passed for it in this case.
So if you don't want your animation to be cached then pass false into this call.
Modify the code to this and it should work:
RLOTTIE_API Lottie_Animation_S *lottie_animation_from_data(const char *data, const char *key, const char *resourcePath)
{
if (auto animation = Animation::loadFromData(data, key, resourcePath, false) ) {
Lottie_Animation_S *handle = new Lottie_Animation_S();
handle->mAnimation = std::move(animation);
return handle;
} else {
return nullptr;
}
}
I am using ImageAnalysis library to extract live previews to then barcode scanning and OCR on.
I'm not having any issues with barcode scanning at all, but OCR is resulting in some weak results. I'm sure this could be from a few reasons. My current attempt at working on the solution is to send the frames to GCP - Storage before I run OCR (or barcode) on the frames in order to look at them in bulk. All of them look very similar:
My best guess is the way i'm processing the frames could be causing the pixels to be organized in the buffer incorrectly (i'm inexperienced to Android - sorry). Meaning rather than organizing 0,0 then 0,1.....it's randomly taking pixels and putting them in random areas. I can't figure out where this is happening though. Once I can look at the image quality, then i'll be able to analyze what the issue is with OCR but this is my current blocker unfortunately.
Extra note: I am uploading the image to GCP - Storage prior to even running OCR, so for the sake of looking at this, we can ignore the OCR statement I made - I just wanted to give some background.
Below is the code where I initiate the camera and analyzer then observe the frames
private void startCamera() {
//make sure there isn't another camera instance running before starting
CameraX.unbindAll();
/* start preview */
int aspRatioW = txView.getWidth(); //get width of screen
int aspRatioH = txView.getHeight(); //get height
Rational asp = new Rational (aspRatioW, aspRatioH); //aspect ratio
Size screen = new Size(aspRatioW, aspRatioH); //size of the screen
//config obj for preview/viewfinder thingy.
PreviewConfig pConfig = new PreviewConfig.Builder().setTargetResolution(screen).build();
Preview preview = new Preview(pConfig); //lets build it
preview.setOnPreviewOutputUpdateListener(
new Preview.OnPreviewOutputUpdateListener() {
//to update the surface texture we have to destroy it first, then re-add it
#Override
public void onUpdated(Preview.PreviewOutput output){
ViewGroup parent = (ViewGroup) txView.getParent();
parent.removeView(txView);
parent.addView(txView, 0);
txView.setSurfaceTexture(output.getSurfaceTexture());
updateTransform();
}
});
/* image capture */
//config obj, selected capture mode
ImageCaptureConfig imgCapConfig = new ImageCaptureConfig.Builder().setCaptureMode(ImageCapture.CaptureMode.MAX_QUALITY)
.setTargetRotation(getWindowManager().getDefaultDisplay().getRotation()).build();
final ImageCapture imgCap = new ImageCapture(imgCapConfig);
findViewById(R.id.imgCapture).setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
Log.d("image taken", "image taken");
}
});
/* image analyser */
ImageAnalysisConfig imgAConfig = new ImageAnalysisConfig.Builder().setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE).build();
ImageAnalysis analysis = new ImageAnalysis(imgAConfig);
analysis.setAnalyzer(
Executors.newSingleThreadExecutor(), new ImageAnalysis.Analyzer(){
#Override
public void analyze(ImageProxy imageProxy, int degrees){
Log.d("analyze", "just analyzing");
if (imageProxy == null || imageProxy.getImage() == null) {
return;
}
Image mediaImage = imageProxy.getImage();
int rotation = degreesToFirebaseRotation(degrees);
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(toBitmap(mediaImage));
if (!isMachineLearning){
Log.d("analyze", "isMachineLearning is about to be true");
isMachineLearning = true;
String haha = MediaStore.Images.Media.insertImage(getContentResolver(), toBitmap(mediaImage), "image" , "theImageDescription");
Log.d("uploadingimage: ", haha);
extractBarcode(image, toBitmap(mediaImage));
}
}
});
//bind to lifecycle:
CameraX.bindToLifecycle(this, analysis, imgCap, preview);
}
Below is how I structure my detection (pretty straightforward and simple):
FirebaseVisionBarcodeDetectorOptions options = new FirebaseVisionBarcodeDetectorOptions.Builder()
.setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS)
.build();
FirebaseVisionBarcodeDetector detector = FirebaseVision.getInstance().getVisionBarcodeDetector(options);
detector.detectInImage(firebaseVisionImage)
Finally, when I'm uploading the image to GCP - Storage, this is what it looks like:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.JPEG, 100, baos); //bmp being the image that I ran barcode scanning on - as well as OCR
byte[] data = baos.toByteArray();
UploadTask uploadTask = storageRef.putBytes(data);
Thank you all for your kind help (:
My problem was that I was trying to convert to a bitmap AFTER barcode scanning. The conversion wasn't properly written but I found a way around without having to write my own bitmap conversion function (though I plan on going back to it as I see myself needing it, and genuine curiosity wants me to figure it out)
I'm using Gtk2 to make a small tool, it works like this:
Several Scales and Spinboxes control parameters of an algorithm.
When parameter changes, the algorithm will execute, and the updated result is rendered as a picture, shown in UI.
As the algorithm's workload is heavy, I don't want it run frequently during frequent parameter change. Specifically, during Scales are dragged or Spinbox's arrows buttons are pressed. Instead, I want the algorithm to be run "after" users have determined the parameters.
Currently, I listened the button-release event of the Scales, so the algorithm will run only on Scale dragging is done. However, this not fit for the Spinboxes, as they have separate entry and button sub-area. If I listen to Spinbox's button-release, it would behave weirdly.
So what event (or events) should I listen to obtain the occation that a continuous value update is finished for a Spinbox?
Could I see the code you have? The button-release works great for me. However, depending on your algorithm, you may be getting 'feedback'. Are you sure the rest of the code is not updating your spinbutton in some way?
I'd use a deferred computation, independent from the device you use to modify the data. In this way you can also input the numbers with the keyboard or copy and paste their content and the program will still work as expected.
A way to do this in GTK+ is by leveraging the main loop and using a timeout GSource, e.g.:
#include <gtk/gtk.h>
typedef struct {
guint event;
GSourceFunc callback;
GtkWidget *spin_button;
} Algorithm;
static gboolean your_callback(Algorithm *algorithm)
{
g_print("Your heavy computations go here...\n");
/* ... */
algorithm->event = 0;
return FALSE;
}
static void postpone(Algorithm *algorithm)
{
if (algorithm->event > 0) {
g_source_remove(algorithm->event);
}
/* Default delay is 1 second (1000 milliseconds) */
algorithm->event = g_timeout_add(1000, algorithm->callback, algorithm);
}
int main(int argc, char **argv)
{
GtkWidget *window, *spin_button;
Algorithm algorithm;
gtk_init(&argc, &argv);
spin_button = gtk_spin_button_new_with_range(0, 100, 0.1);
g_signal_connect_swapped(spin_button, "value-changed",
G_CALLBACK(postpone), &algorithm);
window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
gtk_container_add(GTK_CONTAINER(window), spin_button);
algorithm.event = 0;
algorithm.callback = (GSourceFunc) your_callback;
algorithm.spin_button = spin_button;
gtk_widget_show_all(window);
gtk_main();
return 0;
}
I'd like to write my own stereo image viewer, because there are certain features I need which are missing from the one bundled with my NVidia/EVGA GTX 580.
I can't figure out how to program the card to enter "shutterglass" mode where every other frame (at 120 HZ) alternates left and right.
I've looked at the OpenGL, Direct3D, and XNA APIs, as well as information from NVIDIA, and can't figure out how to get started. How do I set separate left and right images, how do I tell the screen to display it, and how to I tell the driver to activate the shutterglass transmitter?
(Another disconcerting thing is that whenever I use the bundled software to view stereo images and video in shutterglass mode, it's in fullscreen, and the screen blinks when entering that mode--even though I run the screen at 120Hz in 2D. Is there a way to have a 3D surface in a window without upsetting the rest of the screen on the NVidia "gamer" cards that are 3D capable (570, 580)?
I'm a bit late to this, but I just got the stereoscopic 3D to work using nothing but a GTX 580 and OpenGL. No need for a quadro card or DirectX.
I have the nVidia 3D Vision driver and IR emitter and simply set the emitter to "Always on" in the nVidia control panel.
In my game engine, I switched to a full screen mode with 120Hz and render the scene twice with a slight frustum offset (as per nVidia's own documentation PDF on the manual implementation "2010_GTC2010.pdf").
No quad buffers or any other tricks needed, it works great. Plus, I am in control of all the settings, like convergence etc.
For the NVidia 3Dvision with the GEForce range you need to write a full screen directX surface twice the width of the display with the left image on the left,right on the right (duh).
Then you need to write a magic value into the bottom left of the image which the NVision driver picks up and turns on the glasses, you don't need the nvapi.dll
With the Nvidia pro glasses and a Quadra card you can use the regular OpenGL stereo API.
ps.I did find some sample code that manages to do this with a normal window.
Edit - it was a low level USB code talking to the xmitter that I could never get to build, I think it eventually became this http://sourceforge.net/projects/libnvstusb/
Here is some sample code for full screen with the NVision glasses.
I'm not a DirectX expert so some of this might be less than optimal.
My app is also based on Qt, there might be some Qt bits left in the code
-----------------------------------------------------------------
// header
void create3D();
void set3D();
IDirect3D9 *_d3d;
IDirect3DDevice9 *_d3ddev;
QSize _size; // full screen size
IDirect3DSurface9 *_imageBuf; //Source stereo image
IDirect3DSurface9 *_backBuf;
--------------------------------------------------------
// the code
#include <windows.h>
#include <windowsx.h>
#include <d3d9.h>
#include <d3dx9.h>
#include <strsafe.h>
#pragma comment (lib, "d3d9.lib")
#define NVSTEREO_IMAGE_SIGNATURE 0x4433564e //NV3D
typedef struct _Nv_Stereo_Image_Header
{
unsigned int dwSignature;
unsigned int dwWidth;
unsigned int dwHeight;
unsigned int dwBPP;
unsigned int dwFlags;
} NVSTEREOIMAGEHEADER, *LPNVSTEREOIMAGEHEADER;
// ORedflags in the dwFlagsfielsof the _Nv_Stereo_Image_Headerstructure above
#define SIH_SWAP_EYES 0x00000001
#define SIH_SCALE_TO_FIT 0x00000002
// call at start to set things up
void DisplayWidget::create3D()
{
_size = QSize(1680,1050); //resolution of my Samsung 2233z
_d3d = Direct3DCreate9(D3D_SDK_VERSION); // create the Direct3D interface
D3DPRESENT_PARAMETERS d3dpp; // create a struct to hold various device information
ZeroMemory(&d3dpp, sizeof(d3dpp)); // clear out the struct for use
d3dpp.Windowed = FALSE; // program fullscreen
d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD; // discard old frames
d3dpp.hDeviceWindow = winId(); // set the window to be used by Direct3D
d3dpp.BackBufferFormat = D3DFMT_A8R8G8B8; // set the back buffer format to 32 bit // or D3DFMT_R8G8B8
d3dpp.BackBufferWidth = _size.width();
d3dpp.BackBufferHeight = _size.height();
d3dpp.PresentationInterval = D3DPRESENT_INTERVAL_ONE;
d3dpp.BackBufferCount = 1;
// create a device class using this information and information from the d3dpp stuct
_d3d->CreateDevice(D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
winId(),
D3DCREATE_SOFTWARE_VERTEXPROCESSING,
&d3dpp,
&_d3ddev);
//3D VISION uses a single surface 2x images wide and image high
// create the surface
_d3ddev->CreateOffscreenPlainSurface(_size.width()*2, _size.height(), D3DFMT_A8R8G8B8, D3DPOOL_DEFAULT, &_imageBuf, NULL);
set3D();
}
// call to put 3d signature in image
void DisplayWidget::set3D()
{
// Lock the stereo image
D3DLOCKED_RECT lock;
_imageBuf->LockRect(&lock,NULL,0);
// write stereo signature in the last raw of the stereo image
LPNVSTEREOIMAGEHEADER pSIH = (LPNVSTEREOIMAGEHEADER)(((unsigned char *) lock.pBits) + (lock.Pitch * (_size.height()-1)));
// Update the signature header values
pSIH->dwSignature = NVSTEREO_IMAGE_SIGNATURE;
pSIH->dwBPP = 32;
//pSIH->dwFlags = SIH_SWAP_EYES; // Src image has left on left and right on right, thats why this flag is not needed.
pSIH->dwFlags = SIH_SCALE_TO_FIT;
pSIH->dwWidth = _size.width() *2;
pSIH->dwHeight = _size.height();
// Unlock surface
_imageBuf->UnlockRect();
}
// call in display loop
void DisplayWidget::paintEvent()
{
// clear the window to a deep blue
//_d3ddev->Clear(0, NULL, D3DCLEAR_TARGET, D3DCOLOR_XRGB(0, 40, 100), 1.0f, 0);
_d3ddev->BeginScene(); // begins the 3D scene
// do 3D rendering on the back buffer here
RECT destRect;
destRect.left = 0;
destRect.top = 0;
destRect.bottom = _size.height();
destRect.right = _size.width();
// Get the Backbuffer then Stretch the Surface on it.
_d3ddev->GetBackBuffer(0, 0, D3DBACKBUFFER_TYPE_MONO, &_backBuf);
_d3ddev->StretchRect(_imageBuf, NULL, _backBuf, &destRect, D3DTEXF_NONE);
_backBuf->Release();
_d3ddev->EndScene(); // ends the 3D scene
_d3ddev->Present(NULL, NULL, NULL, NULL); // displays the created frame
}
// my images come from a camera
// _left and _right are QImages but it should be obvious what the functions do
void DisplayWidget::getImages()
{
RECT srcRect;
srcRect.left = 0;
srcRect.top = 0;
srcRect.bottom = _size.height();
srcRect.right = _size.width();
RECT destRect;
destRect.top = 0;
destRect.bottom = _size.height();
if ( isOdd() ) {
destRect.left = _size.width();
destRect.right = _size.width()*2;
// get camera data for _left here, code not shown
D3DXLoadSurfaceFromMemory(_imageBuf, NULL, &destRect,_right.bits(),D3DFMT_A8R8G8B8,_right.bytesPerLine(),NULL,&srcRect,D3DX_DEFAULT,0);
} else {
destRect.left = 0;
destRect.right = _size.width();
// get camera data for _right here, code not shown
D3DXLoadSurfaceFromMemory(_imageBuf, NULL, &destRect,_left.bits(),D3DFMT_A8R8G8B8,_left.bytesPerLine(),NULL,&srcRect,D3DX_DEFAULT,0);
}
set3D(); // add NVidia signature
}
DisplayWidget::~DisplayWidget()
{
_d3ddev->Release(); // close and release the 3D device
_d3d->Release(); // close and release Direct3D
}
I'm writing my first gtk program, using gtkmm, and glade.
I made a filechooserbutton and it has a signal called file-set
So I set that to what I assume is the function name I want it to call when the file is chosen.
But then I see here:
http://library.gnome.org/devel/gtkmm-tutorial/unstable/sec-builder-accessing-widgets.html.en
That they're manually getting the dialog widget and setting a button signal handler in the code.
Which is the right way to do it?
And while I'm here any links to good examples would be handy, they seem to be few and far between. Thanks.
This is how I did it:
// create the UI
refUI = Gtk::Builder::create();
refUI->add_from_file(grq::GLADE_FILE);
// grab your widget
refUI->get_widget("but_new", but_new); // Gtk::ToolButton *but_new;
but_new->signal_clicked().connect(sigc::mem_fun(*this, &MainWindow::on_new_game));
// your signal handler looks something like this :)
void MainWindow::on_new_game() {}
edit:
Basically the *this is the object on which you will be calling the function your signal handler.
This is what my main looks like:
int main(int argc, char **argv) {
Gtk::Main kit(argc, argv);
MainWindow main_window;
kit.run(*main_window.window);
return 0;
}
MainWindow is basically a class that wraps GtkWindow and defines the widgets, a. la.:
class MainWindow
{
private:
Glib::RefPtr<Gtk::Builder> refUI;
//
// Widgets
//
Gtk::ToolButton *but_about;
public:
// The window. This is public so we can hook into events and
// call kit.run(window) against it, if needed.
Gtk::Window *window;
MainWindow()
{
// Load the data for this window and it's widgets.
refUI = Gtk::Builder::create();
refUI->add_from_file(grq::GLADE_FILE);
// The window
refUI->get_widget("main_window", window);
// Widgets
refUI->get_widget("but_about", but_about);
but_about->signal_clicked().connect(sigc::mem_fun(*this, &MainWindow::on_about));
...
}
virtual ~MainWindow()
{
if (window != NULL)
{
delete window; // Frees all the children for the window, too.
}
}
virtual void on_about()
{
// stuff
}
};
Hope this helps!
I found the answer to my question as an afterthought in another stackoverflow question.
But I don't remember which one it was.
The answer seems to be that you have to programmatically add the signal handler to the widget in your code, the gtkbuilder won't do it for you.