Trying to determine h.264 profile & level pragmatically - encoding

Ideally the solution would be in python and cross platform, but that's probably not too likely, so all I require is it work in linux, and I can use a c extension to interface w/python if necessary. I see there is a python binding for ffmpeg which I was thinking about using, however I can't figure out how to determine the profile and level as it is, with fmmpeg or anything else, much less do it pragmatically. Google is not much help on the matter either.
I've been able to determine what features I'd be looking for if I needed to determine the profile and levels manually then I can do that, but then that leads to the question, can ffmpeg then determine if the video was encoded with that feature set? I guess what I'm wondering to that effect is, is it perhaps not possible to fully determine the level and specific profile after encoding? I would think you'd have to know in order to decode it, but maybe not; that would explain why I can't find any information on it. I've been toying with this on and off for awhile, but recently decided to consider a project I'd been thinking about, but this is one of this big things holding me back.

Here is a small program I wrote. It prints the profile and level of MP4 files that use h264 as the video codec.
You can compile it with the following command line:
gcc -std=c99 printProfileAndLevel.c -o printProfileAndLevel
Here is the C source :
#include <stdio.h>
#include <stdlib.h>
void printProfile(int profile_idc, int profile_iop, int level_idc) {
switch(profile_idc) {
case 0x42: printf("Baseline Profile"); break;
case 0x4D: printf("Main Profile"); break;
case 0x58: printf("Extended Profile"); break;
case 0x64: printf("High Profile"); break;
default: printf("Unknown profile (%x)", profile_idc);
}
switch(level_idc) {
case 0x15: printf(" # Level 2.1\n"); break;
case 0x1F: printf(" # Level 3.1\n"); break;
case 0x29: printf(" # Level 4.1\n"); break;
case 0x33: printf(" # Level 5.1\n"); break;
default: printf(" # unknown level (%x)", level_idc);
}
}
int main(int argc, char* argv[])
{
if(argc < 2) {
printf("syntax: %s <files>\n", argv[0]);
exit(-1);
}
int buffsize = 1024;
char *buffer = malloc(buffsize + 1);
for(int nArg = 1; nArg < argc; nArg++) {
printf("File %s:\n", argv[nArg]);
FILE *file = fopen(argv[nArg], "r+");
if(file == NULL) {
printf("Cannot open input file %s\n", argv[nArg]);
continue;
}
int nRead = 0;
nRead = fread(buffer, 1, buffsize, file);
for(int i = 0; i < nRead - 7; i++) {
if(buffer[i] == 0x61 && buffer[i+1] == 0x76 && buffer[i+2] == 0x63 && buffer[i+3] == 0x43) {
printProfile(buffer[i+5], buffer[i+6], buffer[i+7]);
}
}
fclose(file);
}
free(buffer);
return 0;
}

Basically you need to identify SPS (Sequence Parameter Set) in the bitstream and decode a couple of its leading bytes.
See H.264 stream header and links there.

Related

How to write a local branch predictor?

I am trying to use runspec test my local branch predictor, but only find a disappointing result.
By now I have tried use a 64 terms LHT, and when the LHT is full, I use FIFO tactics replace a terms in LHT.I don't know if I use a tiny LHT or my improper replacement tactics makes it a terrible precision, anyway it's only 60.9095.
for (int i = 0; i < 1 << HL; i++)
{
if (tag_lht[i] == (addr&(1-(1<<HL))))
{
addr = addr ^ LHT[i].getVal();
goto here;
break;
}
}
index_lht = index_lht%(1<<HL);
tag_lht[index_lht] = (addr&(1-(1<<HL)));
LHT[index_lht] = ShiftReg<2>();
addr = addr ^ LHT[index_lht].getVal();
index_lht++;
here:
for (int i = 0; i < 1 << L; i++)
{
if (tag[i] == (addr))
{
return bhist[i].isTaken();
}
}
index = index % (1 << L);
tag[index] = (addr);
bhist[index].reset();
return bhist[index++].isTaken();
Here I make some explain about the code. bhist is a table store 2-bit status about each branch instructions when the table is full, use FIFO replacement tactics. tag is where the table store address of each instruction. Besides, likely I use tag_lht to store address of each instruction that stored in LHT. Function isTaken() can easily get the predict result.
Thank you all guys, I find that stupid mistake I make, and the code above is correct, but may not seem work prefect. The mistake bellow:
for (int i = 0; i < (1 << L); i++)
{
if (tag[i] == (addr))
{
if (takenActually)
{
LHT[j].shiftIn(1);
bhist[i].increase();
}
else
{
LHT[j].shiftIn(0);
bhist[i].decrease();
}
}
break;
}
But it should be like this:
for (int i = 0; i < (1 << L); i++)
{
if (tag[i] == (addr))
{
if (takenActually)
{
LHT[j].shiftIn(1);
bhist[i].increase();
}
else
{
LHT[j].shiftIn(0);
bhist[i].decrease();
}
break;
}
}
I am so stupid that I waste you helpful people' s time, I spent so much time to figure out why it don't work, at first I thought that wrong variable or argument are used, now I just think I am a careless man.
Again I thank all you ardent fellows. Then I will answer the question with my full code.
PS. wish that my terrible English have not confuse anyone.:)

Bad address error when comparing Strings within BPF

I have an example program I am running here to see if the substring matches the string and then print them out. So far, I am having trouble running the program due to a bad address. I am wondering if there is a way to fix this problem? I have attached the entire code but my problem is mostly related to isSubstring.
#include <uapi/linux/bpf.h>
#define ARRAYSIZE 64
struct data_t {
char buf[ARRAYSIZE];
};
BPF_ARRAY(lookupTable, struct data_t, ARRAYSIZE);
//char name[20];
//find substring in a string
static bool isSubstring(struct data_t stringVal)
{
char substring[] = "New York";
int M = sizeof(substring);
int N = sizeof(stringVal.buf) - 1;
/* A loop to slide pat[] one by one */
for (int i = 0; i <= N - M; i++) {
int j;
/* For current index i, check for
pattern match */
for (j = 0; j < M; j++)
if (stringVal.buf[i + j] != substring[j])
break;
if (j == M)
return true;
}
return false;
}
int Test(void *ctx)
{
#pragma clang loop unroll(full)
for (int i = 0; i < ARRAYSIZE; i++) {
int k = i;
struct data_t *line = lookupTable.lookup(&k);
if (line) {
// bpf_trace_printk("%s\n", key->buf);
if (isSubstring(*line)) {
bpf_trace_printk("%s\n", line->buf);
}
}
}
return 0;
}
My python code here:
import ctypes
from bcc import BPF
b = BPF(src_file="hello.c")
lookupTable = b["lookupTable"]
#add hello.csv to the lookupTable array
f = open("hello.csv","r")
contents = f.readlines()
for i in range(0,len(contents)):
string = contents[i].encode('utf-8')
print(len(string))
lookupTable[ctypes.c_int(i)] = ctypes.create_string_buffer(string, len(string))
f.close()
b.attach_kprobe(event=b.get_syscall_fnname("clone"), fn_name="Test")
b.trace_print()
Edit: Forgot to add the error: It's really long and can be found here: https://pastebin.com/a7E9L230
I think the most interesting part of the error is near the bottom where it mentions:
The sequence of 8193 jumps is too complex.
And a little bit farther down mentions: Bad Address.
The verifier checks all branches in your program. Each time it sees a jump instruction, it pushes the new branch to its “stack of branches to check”. This stack has a limit (BPF_COMPLEXITY_LIMIT_JMP_SEQ, currently 8192) that you are hitting, as the verifier tells you. “Bad Address” is just the translation of kernel's errno value which is set to -EFAULT in that case.
Not sure how to fix it though, you could try:
With smaller strings, or
On a 5.3+ kernel (which supports bounded loops): without unrolling the loop with clang (I don't know if it would help).

What does it mean when CreateNamedPipe returns of 0xFFFFFFFF perror() says "NO ERROR'?

I am using CreateNamedPipe. It returns 0XFFFFFFFF but when I call GetLastError and perror I get "NO ERROR".
I have checked https://learn.microsoft.com/en-us/windows/win32/ipc/multithreaded-pipe-server and I heve coded very similar.
I coded this using an example provided here: https://stackoverflow.com/questions/47731784/c-createnamedpipe-error-path-not-found-3#= and he says it means ERROR_PATH_NOT_FOUND (3). But my address is "\\.\pipe\pipe_com1. Note that StackOverflow seems to remove the extra slashes but you will see them in the paste of my code.
I followed the example here: Create Named Pipe C++ Windows but I still get the error. Here is my code:
// Create a named pipe
// It is used to test TcpToNamedPipe to be sore it it is addressing the named pipe
#include <windows.h>
#include <stdio.h>
#include <process.h>
char ch;
int main(int nargs, char** argv)
{
if (nargs != 2)
{
printf("Usage pipe name is first arg\n");
printf("press any key to exit ");
scanf("%c", &ch);
return -1;
}
char buffer[1024];
HANDLE hPipe;
DWORD dwRead;
sprintf(buffer, "\\\\.\\pipe\\%s", argv[1]);
hPipe = CreateNamedPipe((LPCWSTR)buffer,
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT, // FILE_FLAG_FIRST_PIPE_INSTANCE is not needed but forces CreateNamedPipe(..) to fail if the pipe already exists...
1,
1024*16,
1024*16,
NMPWAIT_USE_DEFAULT_WAIT,
NULL);
if (hPipe == INVALID_HANDLE_VALUE)
{
//int errorno = GetLastError();
//printf("error creating pipe %d\n", errorno);
perror("");
printf("press any key to exit ");
scanf("%c", &ch);
return -1;
}
while (hPipe != INVALID_HANDLE_VALUE)
{
if (ConnectNamedPipe(hPipe, NULL) != FALSE) // wait for someone to connect to the pipe
{
while (ReadFile(hPipe, buffer, sizeof(buffer) - 1, &dwRead, NULL) != FALSE)
{
/* add terminating zero */
buffer[dwRead] = '\0';
/* do something with data in buffer */
printf("%s", buffer);
}
}
DisconnectNamedPipe(hPipe);
}
return 0;
}
I'm guessing that the pointer to the address may be wrong and CreateNamedPipe is not seeing the name of the pipe properly. So I used disassembly and notice that the address is in fact a far pointer. Here is that disassembly:
00CA1A45 mov esi,esp
00CA1A47 push 0
00CA1A49 push 0
00CA1A4B push 4000h
00CA1A50 push 4000h
00CA1A55 push 1
00CA1A57 push 0
00CA1A59 push 3
00CA1A5B lea eax,[buffer]
00CA1A61 push eax
00CA1A62 call dword ptr [__imp__CreateNamedPipeW#32 (0CAB00Ch)]
Can someone spot my problem?

GPS Data Parsing

I got my hands on a USB DeLORME Earthmate GPA LT-20, I want to use it as part of a mobile GPS ratification unit, Raspberry Pi based. I have been able to access the raw serial data but am at odds with an effective means of parsing the data into a usable format. the current plan is just to have it printed on screen in a meaningful way. just looking at ideas. Bellow is a sampling of the data, i have altered the GPS location data to remove the particular location of testing. Perfer to code in C
I have read the following refrence sites:
http://www.gpsinformation.org/dale/nmea.htm
https://en.wikipedia.org/wiki/List_of_GPS_satellites
$GPRMC,050229.000,A,3XX8.647,N,11XX1.282,W,0.1,0.0,140518,11.7,E*4B
$GPGGA,050229.000,3XX8.64662,N,11XX1.28205,W,1,06,1.5,725.48,M,-28.4,M,,*5D
$GPVTG,0.0,T,11.7,M,0.1,N,0.1,K*79
$GPGSV,3,1,09,10,34,240,34,13,24,054,00,15,47,086,26,16,25,292,30*77
$GPGSV,3,2,09,20,79,310,31,21,65,345,37,26,25,260,00,27,11,320,00*78
$GPGSV,3,3,09,29,46,147,34,,,,,,,,,,,,*4C
$PSTMECH,21,7,20,7,15,7,29,7,10,7,00,0,16,7,00,0,00,0,00,0,00,0,00,0*5C
Looking at this information:
`"$GPRMC,050229.000,A,3008.647,N,11001.282,W,0.1,0.0,140518,11.7,E*4B"`
Use strtok for parsing:
int main(void)
{
FILE *fp = fopen("test.txt", "r");
char buf[256];
char *array[20];
while(fgets(buf, sizeof(buf), fp))
{
if(strstr(buf, "$GPRMC"))
{
int count = 0;
char *token;
token = strtok(buf, ",");
while(token != NULL)
{
array[count++] = token;
token = strtok(NULL, ",");
if(count == 20)
break;
}
printf("Latitude : %s %s\n", array[3], array[4]);
printf("Longitude : %s %s\n", array[5], array[6]);
}
}
return 0;
}
Result:
Latitude : 3008.647 N
Longitude : 11001.282 W

GPS output being incorrectly written to file on SD card- Arduino

I have a sketch to take information (Lat, Long) from an EM-406a GPS receiver and write the information to an SD card on an Arduino shield.
The program is as follows:
#include <TinyGPS++.h>
#include <SoftwareSerial.h>
#include <SD.h>
TinyGPSPlus gps;
SoftwareSerial ss(4, 3); //pins for the GPS
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
void setup()
{
Serial.begin(115200); //for the serial output
ss.begin(4800); //start ss at 4800 baud
Serial.println("gpsLogger by Aaron McRuer");
Serial.println("based on code by Mikal Hart");
Serial.println();
//initialize the SD card
if(!card.init(SPI_FULL_SPEED, 9))
{
Serial.println("card.init failed");
}
//initialize a FAT volume
if(!volume.init(&card)){
Serial.println("volume.init failed");
}
//open the root directory
if(!root.openRoot(&volume)){
Serial.println("openRoot failed");
}
//create new file
char name[] = "WRITE00.TXT";
for (uint8_t i = 0; i < 100; i++){
name[5] = i/10 + '0';
name[6] = i%10 + '0';
if(file.open(&root, name, O_CREAT | O_EXCL | O_WRITE)){
break;
}
}
if(!file.isOpen())
{
Serial.println("file.create");
}
file.print("Ready...\n");
}
void loop()
{
bool newData = false;
//For one second we parse GPS data and report some key values
for (unsigned long start = millis(); millis() - start < 1000;)
{
while (ss.available())
{
char c = ss.read();
//Serial.write(c); //uncomment this line if you want to see the GPS data flowing
if(gps.encode(c)) //did a new valid sentence come in?
newData = true;
}
}
if(newData)
{
file.write(gps.location.lat());
file.write("\n");
file.write(gps.location.lng());
file.write("\n");
}
file.close();
}
When I open up the file on the SD card when the program is finished executing, I get a message that it has an encoding error.
I'm currently inside (and unable to get a GPS signal, thus the 0), but the encoding problem needs to be tackled, and there should be as many lines as there are seconds that the device has been on. There's only that one. What do I need to do to make things work correctly here?
Closing the file in the loop, and never reopening it, is the reason there's only one set of data in your file.
Are you sure gps.location.lat() and gps.location.lng() return strings, not an integer or float? That would explain the binary data and the "encoding error" you see.