I've written a couple production BPF agents, but my approach is very iterative until I please the verifier and can move on. I've reached my limit again.
Here's a program that works if I have one fewer && condition -- and breaks otherwise. The confusing part is that the warning implies that 103 insns is greater-than at most 4096 insns. There's obviously something I'm misunderstanding about how this is all strung together.
My ultimate goal is to do logging based on a process' environment -- so alternative approaches are welcome. :)
Error:
$ sudo python foo.py
bpf: Argument list too long. Program too large (103 insns), at most 4096 insns
Failed to load BPF program b'tracepoint__sched__sched_process_exec': Argument list too long
BPF Source:
#include <linux/mm_types.h>
#include <linux/sched.h>
#include <linux/version.h>
int tracepoint__sched__sched_process_exec(
struct tracepoint__sched__sched_process_exec* args
) {
struct task_struct* task = (typeof(task))bpf_get_current_task();
const struct mm_struct* mm = task->mm;
unsigned long env_start = mm->env_start;
unsigned long env_end = mm->env_end;
// Read up to 512 environment variables -- only way I could find to "limit"
// the loop to satisfy the verifier.
char var[12];
for (int n = 0; n < 512; n++) {
int result = bpf_probe_read_str(&var, sizeof var, (void*)env_start);
if (result <= 0) {
break;
}
env_start += result;
if (
var[0] == 'H' &&
var[1] == 'I' &&
var[2] == 'S' &&
var[3] == 'T' &&
var[4] == 'S' &&
var[5] == 'I' &&
var[6] == 'Z' &&
var[7] == 'E'
) {
bpf_trace_printk("Got it: %s\n", var);
break;
}
}
return 0;
}
Basic loader program for reproducing:
#!/usr/bin/env python3
import sys
from bcc import BPF
if __name__ == '__main__':
source = open("./foo.c").read()
try:
BPF(text=source.encode("utf-8")).trace_print()
except Exception as e:
error = str(e)
sys.exit(error)
bpf: Argument list too long. Program too large (103 insns), at most 4096 insns
Looking at the error message, my guess would be that your program has 103 instructions and it's rejected because it's too complex. That is, the verifier gave up before analyzing all instructions on all paths.
On Linux 5.15 with a privileged user, the verifier gives up after reading 1 million instructions (the complexity limit). Since it has to analyze all paths through the program, a program with a small number of instructions can have a very high complexity. That's particularly the case when you have loops and many conditions, as is your case.
Why is the error message confusing? This error message is coming from libbpf.c:
if (ret < 0 && errno == E2BIG) {
fprintf(stderr,
"bpf: %s. Program %s too large (%u insns), at most %d insns\n\n",
strerror(errno), attr->name, insns_cnt, BPF_MAXINSNS);
return -1;
}
Since the bpf(2) syscall returns E2BIG both when the program is too large and when its complexity is too high, libbpf prints the same error message for both cases, always with at most 4096 instructions. I'm confident upstream would accept a patch to improve that error message.
I was wondering if it's possible to display the current source line in windbg?
Right now I am able to step through the code and display 5 lines before and after the current line every time I step through the code. This is great, but it would be nice if there was a command to print the current source line on demand instead of having to step to the next line.
if you want to print a single source line use lsp
The default value is 20 (0x14)
> lsp -a 1
WARNING: Source line display is disabled
At the prompt, display 0 source lines before and 1 after
For lsa commands, display 0 source lines before
For ls and lsa commands, display 1 source lines
now use lsa where .denotes the current Eip/Rip
0:000:x86> lsa .
> 28: void main(int argc, char *argv[]) {
or provide an address
0:000:x86> lsa #$ip+42
> 30: SymInitialize(hProcess, NULL, FALSE);
you can also provide the source line to lsa
0:000> lsa `symtype!symtype.cpp:16`
> 16: if (maxcmplen == pSymInfo->NameLen) {
0:000> lsa `symtype!symtype.cpp:28`
> 28: void main(int argc, char *argv[]) {
I am trying to connect to PostgreSQL database using ecpg program and I am getting below error.
cc testecpg.c
/tmp/ccSzqgA7.o: In function `main':
testecpg.c:(.text+0x5d): undefined reference to `ECPGconnect'
testecpg.c:(.text+0x62): undefined reference to `ECPGget_sqlca'
testecpg.c:(.text+0x70): undefined reference to `sqlprint'
collect2: error: ld returned 1 exit status
testecpg.c file generated after executing ecpg testecpg.pgc
/* Processed by ecpg (4.11.0) */
/* These include files are added by the preprocessor */
#include "/opt/rh/rh-postgresql95/root/usr/include/ecpglib.h"
#include "/opt/rh/rh-postgresql95/root/usr/include/ecpgerrno.h"
#include "/opt/rh/rh-postgresql95/root/usr/include/sqlca.h"
/* End of automatic include section */
#line 1 "testecpg.pgc"
#include <stdio.h>
#include "/opt/rh/rh-postgresql95/root/usr/include/libpq-fe.h"
int main(void)
{
/* exec sql begin declare section */
#line 6 "testecpg.pgc"
char * dbname = "dbname" ;
#line 7 "testecpg.pgc"
char * db = "dbname#hostname:5432" ;
#line 8 "testecpg.pgc"
char * user = "user" ;
#line 9 "testecpg.pgc"
char * passwd = "password" ;
#line 10 "testecpg.pgc"
const char * target = "dbname#hostname:5432" ;
/* exec sql end declare section */
#line 11 "testecpg.pgc"
/* exec sql whenever sqlerror sqlprint ; */
#line 12 "testecpg.pgc"
{ ECPGconnect(0, 0, target , user , passwd , NULL, 0);
#line 13 "testecpg.pgc"
if (sqlca.sqlcode < 0) sqlprint();}
#line 13 "testecpg.pgc"
printf("connection succssfull");
}
Is there any library to be included or any step I have missed?
You forgot to link with the ECPG library.
On Unix systems, that would look somewhat like
cc -o testecpg testecpg.c -lecpg
You have to add the appropriate -I and -L options so that the compiler can find the include files and the libraries.
I am trying to learn more about how to detect memory leaks in Perl.
I have this program:
p.pl:
#! /usr/bin/env perl
use Devel::LeakTrace;
my $foo;
$foo = \$foo;
Output:
leaked SV(0xac2df8e0) from ./p.pl line 5
leaked SV(0xac2df288) from ./p.pl line 5
Why is this leaking two scalars (and not just a single)?
Then I run it through valgrind. First I created a debugging version of perl:
$ perlbrew install perl-5.30.0 --as=5.30.0-D3L -DDEBUGGING \
-Doptimize=-g3 -Accflags="-DDEBUG_LEAKING_SCALARS"
$ perlbrew use 5.30.0-D3L
$ cpanm Devel::LeakTrace
Then I ran valgrind setting PERL_DESTRUCT_LEVEL=2 as recommended in perlhacktips:
$ PERL_DESTRUCT_LEVEL=2 valgrind --leak-check=yes perl p.pl
==12479== Memcheck, a memory error detector
==12479== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12479== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==12479== Command: perl p.pl
==12479==
leaked SV(0x4c27320) from p.pl line 5
leaked SV(0x4c26cc8) from p.pl line 5
==12479==
==12479== HEAP SUMMARY:
==12479== in use at exit: 105,396 bytes in 26 blocks
==12479== total heap usage: 14,005 allocs, 13,979 frees, 3,011,508 bytes allocated
==12479==
==12479== 16 bytes in 1 blocks are definitely lost in loss record 5 of 21
==12479== at 0x483874F: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12479== by 0x484851A: note_changes (LeakTrace.xs:80)
==12479== by 0x48488E3: XS_Devel__LeakTrace_hook_runops (LeakTrace.xs:126)
==12479== by 0x32F0A2: Perl_pp_entersub (pp_hot.c:5237)
==12479== by 0x2C0C50: Perl_runops_debug (dump.c:2537)
==12479== by 0x1A2FD9: Perl_call_sv (perl.c:3043)
==12479== by 0x1ACEE3: Perl_call_list (perl.c:5084)
==12479== by 0x181233: S_process_special_blocks (op.c:10471)
==12479== by 0x180989: Perl_newATTRSUB_x (op.c:10397)
==12479== by 0x220D6C: Perl_yyparse (perly.y:295)
==12479== by 0x3EE46B: S_doeval_compile (pp_ctl.c:3502)
==12479== by 0x3F4F87: S_require_file (pp_ctl.c:4322)
==12479==
==12479== LEAK SUMMARY:
==12479== definitely lost: 16 bytes in 1 blocks
==12479== indirectly lost: 0 bytes in 0 blocks
==12479== possibly lost: 0 bytes in 0 blocks
==12479== still reachable: 105,380 bytes in 25 blocks
==12479== suppressed: 0 bytes in 0 blocks
==12479== Reachable blocks (those to which a pointer was found) are not shown.
==12479== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==12479==
==12479== For counts of detected and suppressed errors, rerun with: -v
==12479== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
so 16 bytes are lost. However, if I comment out the line use Devel::LeakTrace in p.pl and run valgrind again, the output is:
==12880== Memcheck, a memory error detector
==12880== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12880== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==12880== Command: perl p.pl
==12880==
==12880==
==12880== HEAP SUMMARY:
==12880== in use at exit: 0 bytes in 0 blocks
==12880== total heap usage: 1,770 allocs, 1,770 frees, 244,188 bytes allocated
==12880==
==12880== All heap blocks were freed -- no leaks are possible
==12880==
==12880== For counts of detected and suppressed errors, rerun with: -v
==12880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
So the question is: Why is Devel::LeakTrace causing a memory leak?
It seems like there are even more memory leaks than valgrind reported.
Each time a new SV is created, Devel::LeakTrace records the current file name and line number in a 16 bytes structure called when:
typedef struct {
char *file;
int line;
} when;
These blocks are allocated at line #80 with malloc() but it seems it never frees these blocks. So the more scalars are created, the more memory will leak.
Some background information
The module tries to determine leaked SVs from the END{} phaser. At this point all allocated SVs should have gone out of scope from the main program and had their reference count decreased to zero, which should destroy them. However, if for some reason the reference count is not decremented to zero, the scalar will not be destroyed and freed
from perl's internal memory management pool. In this case the scalar is considered as leaked by the module.
Note that this is not the same as leaked memory as seen from the operating
systems memory pool handled by e.g. malloc(). When perl exits it will still
free any leaked scalars (from its internal memory pool) back to the systems memory pool.
This means that the module is not meant to detect leaked system memory. For this, we can use e.g. valgrind.
The module hooks into the perl runops loop and for each OP that is of type OP_NEXTSTATE it will scan all arenas and all SVs in those for new SVs (that is: SVs that has been introduced since the previous OP_NEXTSTATE).
For this sample program p.pl in my question I counted 31 arenas, and each arena contained space for 71 SVs. Almost all of these SVs were in use during run time (approximately 2150 of them). The module keeps each of these SVs in a hash used with key equal to the address of the SV and value equal to the when block (see above) where the scalar was allocated. For each OP_NEXTSTATE, it can then scan all SVs and check if there are some that are not present in the used hash.
The used hash is not a perl hash ( I guess this was to avoid any conflicts with
the allocated SVs that the module tries to keep track of), instead the module uses GLib hash tables.
Patch
In order to keep track of the allocated when blocks, I used a new glib hash called when_hash. Then after the module had printed the leaked scalars, the when blocks could be freed by looking up all keys in the when_hash.
I also found that the module did not free the used-hash. As far as I can see it should be calling the glib g_hash_table_destroy() to release it from the END{} block. Here is the patch:
LeakTrace.xs (patched):
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include <glib.h>
typedef struct {
char *file;
int line;
} when;
/* a few globals, never mind the mess for now */
GHashTable *used = NULL;
GHashTable *new_used = NULL;
/* cargo from Devel::Leak - wander the arena, see what SVs live */
typedef long used_proc _((void *,SV *,long));
/* PATCH: fix memory leaks */
/***************************/
GHashTable *when_hash = NULL; /* store the allocated when blocks here */
static int have_run_end_hook = 0; /* indicator to runops that we are done */
static runops_proc_t save_orig_run_ops; /* original runops function */
/* Called from END{}, i.e. from show_used() after having printed the leaks.
* Free memory allocated for the when blocks */
static
void
free_when_block(gpointer key, gpointer value, gpointer user_data) {
free(key);
}
static
void
do_cleanup() {
/* this line was missing from the original show_used() */
if (used) g_hash_table_destroy( used );
if (when_hash) g_hash_table_foreach( when_hash, free_when_block, NULL );
g_hash_table_destroy( when_hash );
PL_runops = save_orig_run_ops;
have_run_end_hook = 1;
}
/* END PATCH: fix memory leaks */
/*******************************/
static
long int
sv_apply_to_used(void *p, used_proc *proc, long n) {
SV *sva;
for (sva = PL_sv_arenaroot; sva; sva = (SV *) SvANY(sva)) {
SV *sv = sva + 1;
SV *svend = &sva[SvREFCNT(sva)];
while (sv < svend) {
if (SvTYPE(sv) != SVTYPEMASK) {
n = (*proc) (p, sv, n);
}
++sv;
}
}
return n;
}
/* end Devel::Leak cargo */
static
long
note_used(void *p, SV* sv, long n) {
when *old = NULL;
if (used && (old = g_hash_table_lookup( used, sv ))) {
g_hash_table_insert(new_used, sv, old);
return n;
}
g_hash_table_insert(new_used, sv, p);
return 1;
}
static
void
print_me(gpointer key, gpointer value, gpointer user_data) {
when *w = value;
char *type;
switch SvTYPE((SV*)key) {
case SVt_PVAV: type = "AV"; break;
case SVt_PVHV: type = "HV"; break;
case SVt_PVCV: type = "CV"; break;
case SVt_RV: type = "RV"; break;
case SVt_PVGV: type = "GV"; break;
default: type = "SV";
}
if (w->file) {
fprintf(stderr, "leaked %s(0x%x) from %s line %d\n",
type, key, w->file, w->line);
}
}
static
int
note_changes( char *file, int line ) {
static when *w = NULL;
int ret;
/* PATCH */
if (have_run_end_hook) return 0; /* do not enter after clean up is complete */
/* if (!w) w = malloc(sizeof(when)); */
if (!w) {
w = malloc(sizeof(when));
if (!when_hash) {
/* store pointer to allocated blocks here */
when_hash = g_hash_table_new( NULL, NULL );
}
g_hash_table_insert(when_hash, w, NULL); /* store address to w */
}
/* END PATCH */
w->line = line;
w->file = file;
new_used = g_hash_table_new( NULL, NULL );
if (sv_apply_to_used( w, note_used, 0 )) w = NULL;
if (used) g_hash_table_destroy( used );
used = new_used;
return ret;
}
/* Now this bit of cargo is a derived from Devel::Caller */
static
int
runops_leakcheck(pTHX) {
char *lastfile = 0;
int lastline = 0;
IV last_count = 0;
while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
PERL_ASYNC_CHECK();
if (PL_op->op_type == OP_NEXTSTATE) {
if (PL_sv_count != last_count) {
note_changes( lastfile, lastline );
last_count = PL_sv_count;
}
lastfile = CopFILE(cCOP);
lastline = CopLINE(cCOP);
}
}
note_changes( lastfile, lastline );
TAINT_NOT;
return 0;
}
MODULE = Devel::LeakTrace PACKAGE = Devel::LeakTrace
PROTOTYPES: ENABLE
void
hook_runops()
PPCODE:
{
note_changes(NULL, 0);
PL_runops = runops_leakcheck;
}
void
reset_counters()
PPCODE:
{
if (used) g_hash_table_destroy( used );
used = NULL;
note_changes(NULL, 0);
}
void
show_used()
CODE:
{
if (used) g_hash_table_foreach( used, print_me, NULL );
/* PATCH */
do_cleanup(); /* released allocated memory, restore original runops */
/* END PATCH */
}
Testing the patch
$ wget https://www.cpan.org/modules/by-module/Devel/Devel-LeakTrace-0.06.tar.gz
$ tar zxvf Devel-LeakTrace-0.06.tar.gz
$ cd Devel-LeakTrace-0.06
$ perlbrew use 5.30.0-D3L
# replace lib/Devel/LeakTrace.xs with my patch
$ perl Makefile.PL
$ make
$ make install # <- installs the patch
# cd to test folder, then
$ PERL_DESTRUCT_LEVEL=2 valgrind --leak-check=yes perl p.pl
==25019== Memcheck, a memory error detector
==25019== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25019== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==25019== Command: perl p.pl
==25019==
leaked SV(0x4c26cd8) from p.pl line 5
leaked SV(0x4c27330) from p.pl line 5
==25019==
==25019== HEAP SUMMARY:
==25019== in use at exit: 23,324 bytes in 18 blocks
==25019== total heap usage: 13,968 allocs, 13,950 frees, 2,847,004 bytes allocated
==25019==
==25019== LEAK SUMMARY:
==25019== definitely lost: 0 bytes in 0 blocks
==25019== indirectly lost: 0 bytes in 0 blocks
==25019== possibly lost: 0 bytes in 0 blocks
==25019== still reachable: 23,324 bytes in 18 blocks
==25019== suppressed: 0 bytes in 0 blocks
==25019== Reachable blocks (those to which a pointer was found) are not shown.
==25019== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==25019==
==25019== For counts of detected and suppressed errors, rerun with: -v
==25019== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
First, valgrind reports 16 bytes of leaked memory in a script containing only up to use Devel::LeakTrace. The possible leak is independent of the fourth and fifth lines. From your link,
NOTE 3: There are known memory leaks when there are compile-time errors
within eval or require, seeing S_doeval in the call stack is a good sign
of these. Fixing these leaks is non-trivial, unfortunately, but they must be fixed
eventually.
Since I see the line by 0x3F18E5: S_doeval_compile (pp_ctl.c:3502), and a similar line in your example, I would say that this is why Devel::LeakTrace causes an apparent memory leak.
Second, regarding the original script, Devel::LeakTrace is simply reporting the leak caused by (at least) a circular reference at the fifth line. You can see this by using weaken from Scalar::Util:
#! /usr/bin/env perl
use Devel::LeakTrace;
use Scalar::Util;
my $foo;
$foo = \$foo;
Scalar::Util::weaken($foo);
Then, perl p.pl will not report any leak. My guess is that the first scripts reports two leaks because, in addition to creating a circular reference, perl is losing a pointer at $foo = \$foo. There is some magic I cannot understand that occurs when you weaken $foo that apparently fixes both issues. You can see this by tweaking the original script:
#! /usr/bin/env perl
use Devel::LeakTrace;
my $foo;
my $bar = \$foo;
$foo = $bar;
The resulting $foo should be identical, we have just created $bar to hold the reference. However, in this case the script only reports one leak.
So, in summary, I would say that 1)Devel::LeakTrace has a bug that shows as a memory leak in valgrind independently of the code; 2) perl is creating a circular reference and losing a pointer in the original script, which is why Devel::LeakTrace reports two leaks.
My program writenotes keeps seg faulting when I try to write a note that is too long.
./writenotes lolololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololo
[ * ] Writing notes
Segmentation fault
Anyways, I was trying to write a python script that calls the program and curiously enough, calling it from a python script doesn't bring a seg fault, which I thought was rather peculiar.
Heres this code:
#!/usr/bin/python
from subprocess import call
call(["./writenotes", "lolololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololololo"])
Which returns
[ * ] Writing notes
Is this because of parent processing or something like such? How would calling a program through subprocess save a program from a segfault though? Are there other ways to call programs from a script that suffer seg faults?
As a note, the writenotes program was written in C. The other script is python.
You'll almost certainly find your C program is crashing but that Python is hiding that from you. Try instead with:
print call(["./writenotes", "lolololol..."])
and see what you get as a return value.
For example, this program tries to modify a string literal and, when run normally dumps core:
int main (void) {
*"xyzzy" = 'X';
return 0;
}
However, when run from the following script:
from subprocess import call
print call(["./testprog"])
I get the output -11, indicating that signal 11 (usually SIGSEGV) was raised, as per the documentation discussing Popen.returncode which subprocess.call() uses under the covers:
A negative value -N indicates that the child was terminated by signal N (Unix only).
An alternative to checking the return code is to import check_call and CalledProcessError instead of call and then use that function. It will raise an exception if the return code is non-zero.
That's probably not so important if you're only calling one executable (just get the return value in that case) but, if you're doing a lot in sequence, catching an exception from the entire group may be more readable.
Changing the C program to only crash when the first argument is 3:
#include <stdio.h>
#include <string.h>
int main (int argc, char *argv[]) {
if (argc > 1) {
printf ("hello there %s\n", argv[1]);
if (strcmp (argv[1], "3") == 0)
*"xyzzy" = 'X';
}
return 0;
}
and the script to call it with several different arguments:
from subprocess import check_call, CalledProcessError
try:
check_call(["./testprog", "0"])
check_call(["./testprog", "1"])
check_call(["./testprog", "2"])
check_call(["./testprog", "3"])
check_call(["./testprog", "4"])
check_call(["./testprog", "5"])
check_call(["./testprog", "6"])
check_call(["./testprog", "7"])
check_call(["./testprog", "8"])
check_call(["./testprog", "9"])
except CalledProcessError as e:
print e.cmd, "failed with", e.returncode
else:
print "Everything went well"
shows that in action:
hello there 0
hello there 1
hello there 2
hello there 3
['./testprog', '3'] failed with -11