UNIX 'diff' returns improperly when called using popen in main.c - diff

I have a series of tests for a school project that involves building an assembler.
The tests use popen to catch the output of "make" and "./assembler" -- this is just so the output of these commands don't crowd the output of the test suite.
The problem is that the call to popen that executes "diff" returns a string inside the automated test suite, but not when I call it manually.
Here's the test suite code:
char buf1[256];
FILE* make;
FILE* assemble;
FILE *diff;
make = popen("make", "r");
assemble = popen("./assembler input/simple.s inter.txt out.txt", "r");
diff = popen("diff inter.txt out/ref/simple_ref.int", "r");
fgets(buf1, sizeof(buf1), diff);
printf("\nafter fgets simple -- strlen(buf) is %d\t buf is %s\n", strlen(buf1), buf1);
Here's the segment of the test suite output that corresponds with the above code:
after fgets simple -- strlen(buf) is 8 buf is 1,7c1,6
Here's the series of commands when called manually:
aweeeezy  ⋯  make
rm -f *.o assembler test-assembler core
gcc -g -std=gnu99 -Wall -o assembler assembler.c src/utils.c src/tables.c src/translate_utils.c src/translate.c
aweeeezy  ⋯  ./assembler input/simple.s inter.txt out.txt
Running pass one: input/simple.s -> inter.txt
Running pass two: inter.txt -> out.txt
Assembly operation completed successfully.
aweeeezy  ⋯  diff inter.txt out/ref/simple_ref.int
aweeeezy  ⋯  

When you use popen(), you're launching subprocesses which run at the same time as your program. In the code fragment you showed, the subprocesses will all be running around the same time without coordination.
To replicate the command line behavior, you want to wait until each subprocess is done; which is to say, call pclose() before moving to the next stages of your process. For examples, check something like "Pipes the easy way!" at http://www.tldp.org/LDP/lpg/node12.html

Related

SCP command not working in karate project - it throws command error:cannot run program scp.exe: CreateProcess error=2 [duplicate]

I'm trying to execute bash script using karate. I'm able to execute the script from karate-config.js and also from .feature file. I'm also able to pass the arguments to the script.
The problem is, that if the script fails (exits with something else than 0) the test execution continues and finishes as succesfull.
I found out that when the script echo-es something then i can access it as a result of the script so I could possibly echo the exit value and do assertion on it (in some re-usable feature), but this seems like a workaround rather than a valid clean solution. Is there some clean way of accessing the exit code without echo-ing it? Am I missing on something?
script
#!/bin/bash
#possible solution
#echo 3
exit 3;
karate-config.js
var result = karate.exec('script.sh arg1')
feture file
def result = karate.exec('script.sh arg1')
Great timing. We very recently did some work for CLI testing which I am sure you can use effectively. Here is a thread on Twitter: https://twitter.com/maxandersen/status/1276431309276151814
And we have just released version 0.9.6.RC4 and new we have a new karate.fork() option that returns an instance of Command on which you can call exitCode
Here's an example:
* def proc = karate.fork('script.sh arg1')
* proc.waitSync()
* match proc.exitCode == 0
You can get more ideas here: https://github.com/intuit/karate/issues/1191#issuecomment-650087023
Note that the argument to karate.fork() can take multiple forms. If you are using karate.exec() (which will block until the process completes) the same arguments work.
string - full command line as seen above
string array - e.g. ['script.sh', 'arg1']
json where the keys can be
line - string (OR)
args - string array
env - optional environment properties (as JSON)
redirectErrorStream - boolean, true by default which means Sys.err appears in Sys.out
workingDir - working directory
useShell - default false, auto-prepend cmd /c or sh -c depending on OS
And since karate.fork() is async, you need to call waitSync() if needed as in the example above.
Do provide feedback and we can tweak further if needed.
EDIT: here's a very advanced example that shows how to listen to the process output / log, collect the log, and conditionally exit: fork-listener.feature
Another answer which can be a useful reference: Conditional match based on OS
And here's how to use cURL for advanced HTTP tests ! https://stackoverflow.com/a/73230200/143475
In case you need to do a lot of local file manipulation, you can use the karate.toJavaFile() utility so you can convert a relative path or a "prefixed" path to an absolute path.
* def file = karate.toJavaFile('classpath:some/file.txt')
* def path = file.getPath()

OpenOCD exit on breakpoint

I'm developing an application on an STM32F042.
I drive everything from a makefile, including my unit tests.
I use OpenOCD and ST-LINK to flash the target.
My unit tests run on the host and on the target.
The host unit test driver returns 0 from main() on success and non-zero on failure, so the makefile knows if the tests pass.
The makefile flashes and starts tests on the target, but doesn't know if they succeed or fail.
The embedded test application turns on a red LED for fail and green for pass, so I know--now I want to automate this step.
I want to set two breakpoints in the code, one in the failure handler and one at the end of main, and tell OpenOCD to exit with zero or non-zero status if it hits one or the other breakpoint.
So my question boils down to two specific ones:
To set a breakpoint, I need to know the PC value at a specific line of code. How do I get that value from the arm-gcc toolchain?
Can I configure OpenOCD to exit on specific breakpoints, and with specific status?
Here's what I ended up with. For each target unit test, I start an OpenOCD server and connect to it with gdb. Gdb runs a script that sets two breakpoints, one for success, one for failure. If it hits either breakpoint, it shuts down the OCD server and exits with a code that communicates success and failure to the shell. To run the same tests on the host, I simply compile them as regular executables.
Makefile:
# target unit test binaries
foo_tests.elf bar_tests.elf baz_tests.elf bop_tests.elf: unit_test_runner.ao
# disable optimization for target unit test driver to avoid optimizing
# away functions that serve as breakpoint labels
unit_test_runner.ao: CFLAGS += -O0 -g
# link target unit test binaries for semihosting
%_tests.elf: ARM_LDLIBS += -specs=rdimon.specs -lrdimon
# host unit test binaries
foo_tests bar_time_tests baz_tests bop_tests: unit_test_runner.o
# run target unit test binaries through gdb and OpenOCD; redirecting stderr
# leaves printf output from `assert()' clearly visible on the console
%.tut: %.elf
openocd -f interface/stlink-v2-1.cfg -f target/stm32f0x.cfg 2> $#.log &
gdb-multiarc -batch-silent -x tut.gdb $< 2> $#-gdb.log
# run host binary
%.run: %
./$*
tests: foo_tests.run bar_time_tests.run baz_tests.run bop_tests.run \
foo_tests.tut bar_time_tests.tut baz_tests.tut bop_tests.tut
tut.gdb:
target remote localhost:3333
monitor arm semihosting enable # let assert()'s printf() through
monitor reset halt
load
monitor reset init
break success # set breakpoint on function `sucess()'
commands # on hitting this bp, execute the following:
monitor shutdown # shutdown OpenOCD server
quit 0 # exit GDB with success code
end
break failure # set breakpoint on function `sucess()'
commands
monitor shutdown
quit 1 # exit GDB with failure code
end
continue
unit_test_runner.c:
#include <stdlib.h>
/* These two functions serve as labels where gdb can place
breakpoints. */
void success() {}
void failure() {}
/* Implementation detail for `assert()' macro */
void assertion_failure(const char *file,
int line,
const char *function,
const char *expression)
{
printf("assertion failure in %s:%d (%s): `%s'\n",
file, line, function, expression);
failure();
exit(1);
}
/* This function is necessary for ARM semihosting */
extern void initialise_monitor_handles(void);
int main(int argc, char* argv[])
{
#ifdef __arm__
initialise_monitor_handles();
#endif
tests(); /* client code implements this function */
success();
return 0;
}

Setting the output directory of mex files in the compiler command

My Project has the following structure:
MainFolder:
>>InitToolbox.m //Here addpaths are executed
>>Compile.m //Here mex compilations calls are made
AlgorithmsFolder //MATLAB code
UtilitiesFolder //MATLAB code
MexFolder // C++ files
>>test1.cpp
>>test2.cu
Whever I run (either in compile.m or directly in the command line) the following compiler call:
mex -v -largeArrayDims ./MexFolder/test1.cpp ./MexFolder/test2.cu
The output test1.mexw64 is saved in MainFolder.
Is there any way to modify the compiler call to create the .mexw64 file either on the original location of the files or in an specific user defined location?
You want to specify the output directory using the outdir option for mex
mex -v -largeArrayDims ./MexFolder/test1.cpp ./MexFolder/test2.cu -outdir output_directory
"output_directory" above can be any path you want.
It can also be a variable, but that would require you to update the way you're calling mex
outputFolder = 'path/to/folder';
mex('-v', '-largeArrayDims', 'MexFolder/test1.cpp', ...
'MexFolder/test2.cu', '-outputdir', outputFolder);

Finding which file defines extern value

I am using raspberry PI to run some assembler code on GPU.
It works like this: you assemble the code into binary file. Then you include it into a C code which pushes data into the GPU. This binary file is defined as
extern uint32_t arrayOfCode[];
However I can not find where is this code included(it is not in any other include files).
The whole code can be found here
Upon running makefile it works.
Where the problem comes is when I am trying to build it as MEX function in Matlab.
Matlab cannot find where is the assembler binary code defined. Thus I have a suspicion that it must be linked somehow in Makefile since that's the only difference in building it.
Does anyone has an idea how to find where is this extern value defined ?
==EDIT 1==
I posted my one solution how to build this into a Matlab library.
But the question remains more or less the same. How this makefile
CXX=g++
ASMSRCS := gemm_float.asm
ASMOBJS := $(subst .asm,.do,$(ASMSRCS))
CPPSRCS := $(shell find . -name '*.cpp' -not -name '._*')
CPPOBJS := $(subst .cpp,.o,$(CPPSRCS))
CPPFLAGS=-Ofast -DTARGET_PI -march=armv6 \
-mfloat-abi=hard \
-ftree-vectorize \
-funroll-all-loops \
-mfpu=vfp \
%.cdat: %.asm helpers.asm
m4 $< | qpu-asm -o $(basename $#).cdat -c g_$(basename $#)Code
%.do: %.cdat
$(CXX) $(CPPFLAGS) -x c -c $< -o $(basename $#).do
%.o: %.cpp
$(CXX) $(CPPFLAGS) -fPIC -c $< -o $(basename $#).o
gemm: $(CPPOBJS) $(ASMOBJS)
g++ -g -O3 -o gemm $(CPPOBJS) $(ASMOBJS) -lblas
Could include "gemm_float.asm" assembled code into the C array defined with the keyword extern. I read that these "%." rules in makefile are rules for dependencies. Okay, that would mean that "gemm" would build anew if I changed something in files that "gemm" depends on. Or I might not understand makefiles well enough.
I was able to fix the issue by separately building C code and assembler code. This is for anyone who cannot use an original makefile for some reason(mine was trying to build this into a mex Matlab library).
First, we build code.asm into binary file
m4 code.asm | ./qpu-asm -o code.bin
Now we have to adjust our C code in such a way that it loads .bin code in it's main function. I used a function from eman's tutorial which simply reads the assembler code from a bin file and saves it into an array for later use.
int loadQPUCode(const char *fname, unsigned int* buffer, int len)
{
FILE *in = fopen(fname, "r");
if (!in) {
fprintf(stderr, "Failed to open %s.\n", fname);
return -1;
}
size_t items = fread(buffer, sizeof(unsigned int), len, in);
fclose(in);
return items * sizeof(unsigned int);
}
Then it is enough to call call this before using any gpu related function.
loadQPUCode("asmCode.bin",arrayOfCode,CONSTANT_RELATIVE_TO_ASMCODE);
With this adjusted C code and binary assembler code I was able to build Matlab library using matlab mex function.
mex main.cpp include1.cpp include2.cpp

Calling SED for a source in Makefile.am

I've got c++ code that needs a sed done to it prior to compilation. How do I place this into Makefile.am?
I tried the typical makefile setup and the target appears to not exist:
gentest.cc:
$(SED) -i "s|FIND|REPLACE|" gentest.cc
If you are interested as to why I want to do this, it's because I wrote my program (slider3.py) in python and my partner wrote his in c++ (gentest.cc) and his needs to call mine. I'm accomplishing this by editing the argv and then using execv().
... {
char **argv2 = new char*[argc];
memset(argv2,0,sizeof(argv2));
argv2[0] = "__PREFIX__/bin/slider3.py";
memcpy(argv2 + 1, argv + 2, sizeof(char *) * (argc - 2));
int oranges = execv(argv2[0], argv2);
printf("%s\n", strerror(oranges));
return oranges;
} ...
I've already handled getting the #! added to slider3.py and chmod +x by using the method that was not working for gentest.cc. I've also handled adding slider3.py to the list of files that get installed.
EXTRA_DIST=testite.sh slider3_base.py
bin_SCRIPTS = slider3.py
CLEANFILES = $(bin_SCRIPTS)
slider3.py: slider3_base.py
rm -f slider3.py
echo "#! " $(PYTHON) > slider3.py
cat slider3_base.py >> slider3.py
chmod +x slider3.py
gentest is defined this way in Makefile.am:
bin_PROGRAMS = gentest
gentest_SOURCES = gentest.cc
gentest_LDADD = libgen.a #../libsbsat.la $(LIBM)
And this fails to be run during make (note the # pattern is successfully expanded in Makefile):
gentest.cc:
$(SED) -i "s|__PREFIX__|#prefix#|" gentest.cc
Any ideas on how to get sed to run before compiling gentest.cc?
Don't use in-place sed.
Instead:
gentest_SOURCES = gentest-seded.cc
gentest-seded.cc : gentest.cc
$(SED) "s|__PREFIX__|#prefix#|" $< >$#
Have you ever considered #define-ing it in config.h (you're using autotools, right?) or passing it using -D when compiling? This is really not the case for sed.
The details from Andrew Y's answer:
in your C++ source, specify:
argv2[0] = SCRIPTPREFIX "/bin/slider3.py";
then compile with
-DSCRIPTPREFIX='"/your/script/prefix"'
Have you considered calling the Python code directly from the C++? Here is a tutorial on using boost to call python functions from C++. The method you are describing here seems very brittle.