convert array<uchar>^ to std::vector<char> data; - type-conversion

I'm working with managed and unmanaged code and I need to convert
convert array<uchar>^ image to std::vector<char> data.
I've started doing this:
array<uchar> ^image = gcnew array<uchar>(tam);
reader2->GetBytes(0, 0, image, 0, tam);
vector<uchar> data;
for (int idxImage = 0; idxImage < tam; idxImage++)
{
data.push_back(image[idxImage]);
}
It looks like it is working but it is very slow. Any idea about how can be done faster?

A first optimization you could do is to reserve the needed space in the vector, so it doesn't have to resize itself as you insert data into it.
This is as easy as:
data.reserve(tam);
This will improve performance, but not by that much. You can do better, you could use memcpy, which, as cppreference says:
std::memcpy is the fastest library routine for memory-to-memory copy.
So let's use that.
First, you need to resize (not reserve) the vector, so it knows the number of used bytes. Then, you can get a pointer to the raw data it holds with the data() function.
As for the array, it's a managed object, which means you need to pin it so the GC won't move it around. In C++/CLI this is done with pin_ptr.
Together, here's the final code:
data.resize(tam);
pin_ptr<uchar> pinned = &image[0];
std::memcpy(data.data(), pinned, tam);
I've tested it, and it's much faster. Here's a full test program:
#include "stdafx.h"
#include <vector>
typedef unsigned char uchar;
void test1(array<uchar>^ image)
{
std::vector<uchar> data;
int tam = image->Length;
auto sw = System::Diagnostics::Stopwatch::StartNew();
for (int idxImage = 0; idxImage < tam; idxImage++)
{
data.push_back(image[idxImage]);
}
sw->Stop();
System::Console::WriteLine("OP: {0} ms", sw->ElapsedMilliseconds);
}
void test2(array<uchar>^ image)
{
std::vector<uchar> data;
int tam = image->Length;
auto sw = System::Diagnostics::Stopwatch::StartNew();
data.reserve(tam);
for (int idxImage = 0; idxImage < tam; idxImage++)
{
data.push_back(image[idxImage]);
}
sw->Stop();
System::Console::WriteLine("reserve: {0} ms", sw->ElapsedMilliseconds);
}
void test3(array<uchar>^ image)
{
std::vector<uchar> data;
int tam = image->Length;
auto sw = System::Diagnostics::Stopwatch::StartNew();
data.resize(tam);
pin_ptr<uchar> pinned = &image[0];
std::memcpy(data.data(), pinned, tam);
sw->Stop();
System::Console::WriteLine("memcpy: {0} ms", sw->ElapsedMilliseconds);
}
int main(array<System::String ^> ^args)
{
size_t tam = 20 * 1024 * 1024;
array<uchar>^ image = gcnew array<uchar>(tam);
(gcnew System::Random)->NextBytes(image);
test1(image);
test2(image);
test3(image);
return 0;
}
My results are:
OP: 123 ms
reserve: 95 ms
memcpy: 8 ms

Related

Bad address error when comparing Strings within BPF

I have an example program I am running here to see if the substring matches the string and then print them out. So far, I am having trouble running the program due to a bad address. I am wondering if there is a way to fix this problem? I have attached the entire code but my problem is mostly related to isSubstring.
#include <uapi/linux/bpf.h>
#define ARRAYSIZE 64
struct data_t {
char buf[ARRAYSIZE];
};
BPF_ARRAY(lookupTable, struct data_t, ARRAYSIZE);
//char name[20];
//find substring in a string
static bool isSubstring(struct data_t stringVal)
{
char substring[] = "New York";
int M = sizeof(substring);
int N = sizeof(stringVal.buf) - 1;
/* A loop to slide pat[] one by one */
for (int i = 0; i <= N - M; i++) {
int j;
/* For current index i, check for
pattern match */
for (j = 0; j < M; j++)
if (stringVal.buf[i + j] != substring[j])
break;
if (j == M)
return true;
}
return false;
}
int Test(void *ctx)
{
#pragma clang loop unroll(full)
for (int i = 0; i < ARRAYSIZE; i++) {
int k = i;
struct data_t *line = lookupTable.lookup(&k);
if (line) {
// bpf_trace_printk("%s\n", key->buf);
if (isSubstring(*line)) {
bpf_trace_printk("%s\n", line->buf);
}
}
}
return 0;
}
My python code here:
import ctypes
from bcc import BPF
b = BPF(src_file="hello.c")
lookupTable = b["lookupTable"]
#add hello.csv to the lookupTable array
f = open("hello.csv","r")
contents = f.readlines()
for i in range(0,len(contents)):
string = contents[i].encode('utf-8')
print(len(string))
lookupTable[ctypes.c_int(i)] = ctypes.create_string_buffer(string, len(string))
f.close()
b.attach_kprobe(event=b.get_syscall_fnname("clone"), fn_name="Test")
b.trace_print()
Edit: Forgot to add the error: It's really long and can be found here: https://pastebin.com/a7E9L230
I think the most interesting part of the error is near the bottom where it mentions:
The sequence of 8193 jumps is too complex.
And a little bit farther down mentions: Bad Address.
The verifier checks all branches in your program. Each time it sees a jump instruction, it pushes the new branch to its “stack of branches to check”. This stack has a limit (BPF_COMPLEXITY_LIMIT_JMP_SEQ, currently 8192) that you are hitting, as the verifier tells you. “Bad Address” is just the translation of kernel's errno value which is set to -EFAULT in that case.
Not sure how to fix it though, you could try:
With smaller strings, or
On a 5.3+ kernel (which supports bounded loops): without unrolling the loop with clang (I don't know if it would help).

Xilinx Echo Server Data Variable

I want to have my Zedboard return a numeric value using the Xilinx lwIP example as a base but no matter what I do I can't figure out what stores the data received or transmitted.
I have found the void type payload but I don't know what to do with it.
Snapshot of one instance of payload and a list of lwIP files
Below is the closest function to my goal:
err_t recv_callback(void *arg, struct tcp_pcb *tpcb,
struct pbuf *p, err_t err){
/* do not read the packet if we are not in ESTABLISHED state */
if (!p) {
tcp_close(tpcb);
tcp_recv(tpcb, NULL);
return ERR_OK;
}
/* indicate that the packet has been received */
tcp_recved(tpcb, p->len);
/* echo back the payload */
/* in this case, we assume that the payload is < TCP_SND_BUF */
if (tcp_sndbuf(tpcb) > p->len) {
err = tcp_write(tpcb, p->payload, p->len, 1);
//I need to change p->paylod but IDK where it is given a value.
} else
xil_printf("no space in tcp_sndbuf\n\r");
/* free the received pbuf */
pbuf_free(p);
return ERR_OK;
}
Any guidance is appreciated.
Thanks,
Turtlemii
-I cheated and just made sure that the function has access to Global_tpcb from echo.c
-tcp_write() reads in an address and displays each char it seems.
void Print_Code()
{
/* Prepare for TRANSMISSION */
char header[] = "\rSwitch: 1 2 3 4 5 6 7 8\n\r"; //header text
char data_t[] = " \n\r\r"; //area for storing the
data
unsigned char mask = 10000000; //mask to decode switches
swc_value = XGpio_DiscreteRead(&SWCInst, 1); //Save switch values
/* Write switch values to the LEDs for visual. */
XGpio_DiscreteWrite(&LEDInst, LED_CHANNEL, swc_value);
for (int i =0; i<=7; i++) //load data_t with switch values (0/1)
{
data_t[8+2*i] = '0' + ((swc_value & mask)/mask); //convert one bit to 0/1
mask = mask >> 1;//move to next bit
}
int len_header = *(&header + 1) - header; //find the length of the
header string
int len_data = *(&data_t + 1) - data_t; //find the length of the data string
tcp_write(Global_tpcb, &header, len_header, 1); //print the header
tcp_write(Global_tpcb, &data_t, len_data, 1); //print the data
}

What's the best way to save terraindata to file in Runtime?

My game lets the user modify the terrain at runtime, but now I need to save said terrain. I've tried to directly save the terrain's heightmap to a file, but this takes almost up to two minutes to write for this 513x513 heightmap.
What would be a good way to approach this? Is there any way to optimize the writing speed, or am I approaching this the wrong way?
public static void Save(string pathraw, TerrainData terrain)
{
//Get full directory to save to
System.IO.FileInfo path = new System.IO.FileInfo(Application.persistentDataPath + "/" + pathraw);
path.Directory.Create();
System.IO.File.Delete(path.FullName);
Debug.Log(path);
//Get the width and height of the heightmap, and the heights of the terrain
int w = terrain.heightmapWidth;
int h = terrain.heightmapHeight;
float[,] tData = terrain.GetHeights(0, 0, w, h);
//Write the heights of the terrain to a file
for (int y = 0; y < h; y++)
{
for (int x = 0; x < w; x++)
{
//Mathf.Round is to round up the floats to decrease file size, where something like 5.2362534 becomes 5.24
System.IO.File.AppendAllText(path.FullName, (Mathf.Round(tData[x, y] * 100) / 100) + ";");
}
}
}
As a sidenote, the Mathf.Round doesn't seem to influence the saving time too much, if at all.
You are making a lot of small individual File IO calls. File IO is always time consuming and expensive as it contains opening the file, writing to it, saving the file and closing the file.
Instead I would rather generate the complete string using e.g. a StringBuilder which is also more efficient than using something like
var someString
for(...)
{
someString += "xyz"
}
because the latter always allocates a new string.
Then use e.g. a FileStream and StringWriter.WriteAsync(string) for writing async.
Also rather use Path.Combine instead of directly concatenating string via /. Path.Combine automatically uses the correct connectors according to the OS it is used on.
And instead of FileInfo.Directory.Create rather use Directory.CreateDirectory which doesn't throw an exception if the directory already exists.
Something like
using System.IO;
...
public static void Save(string pathraw, TerrainData terrain)
{
//Get full directory to save to
var filePath = Path.Combine(Application.persistentDataPath, pathraw);
var path = new FileInfo(filePath);
Directory.CreateDirectory(path.DirectoryName);
// makes no sense to delete
// ... rather simply overwrite the file if exists
//File.Delete(path.FullName);
Debug.Log(path);
//Get the width and height of the heightmap, and the heights of the terrain
var w = terrain.heightmapWidth;
var h = terrain.heightmapHeight;
var tData = terrain.GetHeights(0, 0, w, h);
// put the string together
// StringBuilder is more efficient then using
// someString += "xyz" because latter always allocates a new string
var stringBuilder = new StringBuilder();
for (var y = 0; y < h; y++)
{
for (var x = 0; x < w; x++)
{
// also add the linebreak if needed
stringBuilder.Append(Mathf.Round(tData[x, y] * 100) / 100).Append(';').Append('\n');
}
}
using (var file = File.Open(filePath, FileMode.OpenOrCreate, FileAccess.Write))
{
using (var streamWriter = new StreamWriter(file, Encoding.UTF8))
{
streamWriter.WriteAsync(stringBuilder.ToString());
}
}
}
You might want to specify how exactly the numbers shall be printed with a certain precision like e.g.
(Mathf.Round(tData[x, y] * 100) / 100).ToString("0.00000000");

Unity: Converting Texture2D to YUV420P using FFmpeg

I'm trying to create a game in Unity where each frame is rendered into a texture and then put together into a video using FFmpeg. The output created by FFmpeg should eventually be sent over the network to a client UI. However, I'm struggling mainly with the part where a frame is caught, and passed to an unsafe method as a byte array where it should be processed further by FFmpeg. The wrapper I'm using is FFmpeg.AutoGen.
The render to texture method:
private IEnumerator CaptureFrame()
{
yield return new WaitForEndOfFrame();
RenderTexture.active = rt;
frame.ReadPixels(rect, 0, 0);
frame.Apply();
bytes = frame.GetRawTextureData();
EncodeAndWrite(bytes, bytes.Length);
}
The unsafe encoding method so far:
private unsafe void EncodeAndWrite(byte[] bytes, int size)
{
GCHandle pinned = GCHandle.Alloc(bytes, GCHandleType.Pinned);
IntPtr address = pinned.AddrOfPinnedObject();
sbyte** inData = (sbyte**)address;
fixed(int* lineSize = new int[1])
{
lineSize[0] = 4 * textureWidth;
// Convert RGBA to YUV420P
ffmpeg.sws_scale(sws, inData, lineSize, 0, codecContext->width, inputFrame->extended_data, inputFrame->linesize);
}
inputFrame->pts = frameCounter++;
if(ffmpeg.avcodec_send_frame(codecContext, inputFrame) < 0)
throw new ApplicationException("Error sending a frame for encoding!");
pkt = new AVPacket();
fixed(AVPacket* packet = &pkt)
ffmpeg.av_init_packet(packet);
pkt.data = null;
pkt.size = 0;
pinned.Free();
...
}
sws_scale takes a sbyte** as the second parameter, therefore I'm trying to convert the input byte array to sbyte** by first pinning it with GCHandle and doing an explicit type conversion afterwards. I don't know if that's the correct way, though.
Moreover, the condition if(ffmpeg.avcodec_send_frame(codecContext, inputFrame) < 0) alwasy throws an ApplicationException, where I also really don't know why this happens. codecContext and inputFrame are my AVCodecContext and AVFrame objects, respectively, and the fields are defined as the following:
codecContext
codecContext = ffmpeg.avcodec_alloc_context3(codec);
codecContext->bit_rate = 400000;
codecContext->width = textureWidth;
codecContext->height = textureHeight;
AVRational timeBase = new AVRational();
timeBase.num = 1;
timeBase.den = (int)fps;
codecContext->time_base = timeBase;
videoAVStream->time_base = timeBase;
AVRational frameRate = new AVRational();
frameRate.num = (int)fps;
frameRate.den = 1;
codecContext->framerate = frameRate;
codecContext->gop_size = 10;
codecContext->max_b_frames = 1;
codecContext->pix_fmt = AVPixelFormat.AV_PIX_FMT_YUV420P;
inputFrame
inputFrame = ffmpeg.av_frame_alloc();
inputFrame->format = (int)codecContext->pix_fmt;
inputFrame->width = textureWidth;
inputFrame->height = textureHeight;
inputFrame->linesize[0] = inputFrame->width;
Any help in fixing the issue would be greatly appreciated :)
Check examples on here: https://github.com/FFmpeg/FFmpeg/tree/master/doc/examples
Especially scaling_video.c. In FFmpeg scaling and pixel format conversion is same operation (keep the size parameters same for just pixel format conversion).
These examples very easy to follow. Give it a try.
I think your casting is incorrect sbyte** inData = (sbyte**)address;
because address is IntPtr object, so the correct casting probably should be
sbyte* pinData = (sbyte *)address.ToPointer(); sbyte** ppInData = &pinData;

Embedded fonts in RTF

According to the rtf specs, we can embed a font in an rtf file using the \fontemb and \fontfile control words. Can someone give me a working example of that? I'd like the rtf file to use the font that's located in a separate file (i.e. .ttf file)
You should use TTEmbedFont function to create embedded font data. http://msdn.microsoft.com/en-us/library/windows/desktop/dd145145(v=vs.85).aspx
Like this.
//WRITEEMBEDPROC
unsigned long WriteEmbedProc(void *lpvWriteStream, const void *lpvBuffer, const unsigned long cbBuffer)
{
BYTE *rgByte = new BYTE[cbBuffer];
memcpy(rgByte, lpvBuffer, cbBuffer);
//stream to store your font information
std::ofstream *ofs = static_cast<std::ofstream*>(lpvWriteStream);
//convert binary data to hexadeciaml, that rtf uses
std::string byte_string = BinToHex(rgByte, cbBuffer);
//Write formated data to your file (stream)
for (int i = 0; i < byte_string.size(); ++i)
{
*ofs << byte_string[i];
if((i + 1) % 128 == 0)
{
*ofs << "\n";
}
}
delete rgByte;
return cbBuffer;
}
void EmbedFontWrap(HDC hdc)
{
ULONG ulPrivStatus = 0;
ULONG ulStatus = 0;
std::ofstream *lpvWriteStream = new std::ofstream("D:\\out.txt", std::ios::binary);
USHORT *pusCharCodeSet;
USHORT usCharCodeCount;
USHORT usLanguage;
LONG ret = TTEmbedFont(
hdc,
TTEMBED_RAW | TTEMBED_EMBEDEUDC,
CHARSET_UNICODE,
&ulPrivStatus,
&ulStatus,
WriteEmbedProc,
lpvWriteStream,
nullptr,
0,
0,
nullptr);
lpvWriteStream->close();
delete lpvWriteStream;
}
Font you want to embed should be set as current for you device context by SelectObject function.