Finding which file defines extern value - matlab

I am using raspberry PI to run some assembler code on GPU.
It works like this: you assemble the code into binary file. Then you include it into a C code which pushes data into the GPU. This binary file is defined as
extern uint32_t arrayOfCode[];
However I can not find where is this code included(it is not in any other include files).
The whole code can be found here
Upon running makefile it works.
Where the problem comes is when I am trying to build it as MEX function in Matlab.
Matlab cannot find where is the assembler binary code defined. Thus I have a suspicion that it must be linked somehow in Makefile since that's the only difference in building it.
Does anyone has an idea how to find where is this extern value defined ?
==EDIT 1==
I posted my one solution how to build this into a Matlab library.
But the question remains more or less the same. How this makefile
CXX=g++
ASMSRCS := gemm_float.asm
ASMOBJS := $(subst .asm,.do,$(ASMSRCS))
CPPSRCS := $(shell find . -name '*.cpp' -not -name '._*')
CPPOBJS := $(subst .cpp,.o,$(CPPSRCS))
CPPFLAGS=-Ofast -DTARGET_PI -march=armv6 \
-mfloat-abi=hard \
-ftree-vectorize \
-funroll-all-loops \
-mfpu=vfp \
%.cdat: %.asm helpers.asm
m4 $< | qpu-asm -o $(basename $#).cdat -c g_$(basename $#)Code
%.do: %.cdat
$(CXX) $(CPPFLAGS) -x c -c $< -o $(basename $#).do
%.o: %.cpp
$(CXX) $(CPPFLAGS) -fPIC -c $< -o $(basename $#).o
gemm: $(CPPOBJS) $(ASMOBJS)
g++ -g -O3 -o gemm $(CPPOBJS) $(ASMOBJS) -lblas
Could include "gemm_float.asm" assembled code into the C array defined with the keyword extern. I read that these "%." rules in makefile are rules for dependencies. Okay, that would mean that "gemm" would build anew if I changed something in files that "gemm" depends on. Or I might not understand makefiles well enough.

I was able to fix the issue by separately building C code and assembler code. This is for anyone who cannot use an original makefile for some reason(mine was trying to build this into a mex Matlab library).
First, we build code.asm into binary file
m4 code.asm | ./qpu-asm -o code.bin
Now we have to adjust our C code in such a way that it loads .bin code in it's main function. I used a function from eman's tutorial which simply reads the assembler code from a bin file and saves it into an array for later use.
int loadQPUCode(const char *fname, unsigned int* buffer, int len)
{
FILE *in = fopen(fname, "r");
if (!in) {
fprintf(stderr, "Failed to open %s.\n", fname);
return -1;
}
size_t items = fread(buffer, sizeof(unsigned int), len, in);
fclose(in);
return items * sizeof(unsigned int);
}
Then it is enough to call call this before using any gpu related function.
loadQPUCode("asmCode.bin",arrayOfCode,CONSTANT_RELATIVE_TO_ASMCODE);
With this adjusted C code and binary assembler code I was able to build Matlab library using matlab mex function.
mex main.cpp include1.cpp include2.cpp

Related

Yocto- how to get a link map for the kernel?

I'd like to modify my Bitbake .bbappend file (for linux-imx) to have it generate a linker map file. I've tried things like the following with no success (though they do seem to work correctly for non-kernel recipes):
LDFLAGS += "-Map mymap.map"
LDFLAGS += "-Map=mymap.map"
LDFLAGS_append = " -Map mymap.map"
The end goal is to verify whether or not the on-chip RAM (which is at a fixed address) is being used or not.

Control mex link options with g++ from command line

I'm trying to link a library using mex from command line, or more exactly, from a makefile. I do this from a Makefile which I post here:
BDDM_MATLAB = #matlabhome#
MEXCC = $(BDDM_MATLAB)/bin/mex
MEXFLAGS = -v -largeArrayDims -O
MEXEXT = mexa64
TDIR = $(abs_top_srcdir)/test
IDIR = $(abs_top_srcdir)/src
LDIR = $(abs_top_srcdir)/lib
LOP1 = $(CUDA_LDFLAGS) $(LIBS)
SOURCES := $(wildcard *.cpp)
OBJS = $(SOURCES:.cpp=.o)
mTESTS = $(addprefix $(TDIR)/, $(SOURCES:.cpp=.$(MEXEXT)))
all: $(TDIR) $(mTESTS)
$(OBJS) : %.o : %.cpp
$(MEXCC) $(MEXFLAGS) -c -outdir ./ -output $# $(CUDA_CFLAGS) -I$(IDIR) CFLAGS="\$$CFLAGS -std=c99" $^
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ $(LOP1) -lmpdcm LDFLAGS="-lcudart -lcuda"
.PHONY = $(TDIR)
$(TDIR):
$(MKDIR_P) $#
clean:
$(RM) *.o
libmpdcm is a static library that includes calls to two shared libraries libcuda and libcudart. My environment has
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH:
My make rule produces
/usr/local/MATLAB/R2014a/bin/mex -v -largeArrayDims -O -L/home/eaponte/projects/test_cpp/lib -outdir /home/eaponte/projects/test_cpp/test test_LayeredEEG.o -L/usr/local/cuda/lib64 -lcudart -lcuda -lmpdcm LDFLAGS="-lcudart -lcuda"
This produces the following g++ command:
/usr/bin/gcc -lcudart -lcuda -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o -lcudart -lcuda -lmpdcm -L/home/eaponte/projects/test_cpp/lib -L/usr/local/cuda/lib64 -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64
The problem is that afterwards I get a linking error in Matlab:
Invalid MEX-file '/home/eaponte/projects/test_cpp/test/test_Fmri.mexa64': /home/eaponte/projects/test_cpp/test/test_Fmri.mexa64: undefined symbol: cudaFree
I know that the solution is simply to put the cuda libraries at the end of the g++ command
/usr/bin/gcc -lcudart -lcuda -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o -lmpdcm -L/home/eaponte/projects/test_cpp/lib -L/usr/local/cuda/lib64 -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -lcudart -lcuda -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64
How can achieve that running mex from command line (or from a Makefile)?
Just to illuminate the problem and solution and offer some help in avoiding the like:
The fundamental rule of linkage with the GNU linker
that your problem makefile transgressed is: In the commandline sequence of entities to be linked, ones that need symbol definitions
must appear before the ones that provide the definitions.
An object file (.o) in the linkage sequence will be incorporated entire in the output executable,
regardless of whether or not it defines any symbols that the executable uses. A library
on the other hand, is merely examined to see if it provides any definitions for symbols
that are thus-far undefined, and only such definitions as it provides are linked into in
the executable (I am simplifying somewhat). Thus, linkage doesn't get started until some object file is seen,
and any library must appear after everything that needs definitions from it.
Breaches of this principle usually arise from inept bundling of some linker flag-options
and some library-options together into a make-variable and its placement in the linkage recipe,
with the result that the bundled options are interpolated at a position that is valid for
the flags but not valid for libraries. This was so in your problem makefile, with LOP1 the
bad bundle.
In the typical case, the bundling causes all of the libraries to be placed before all the object files,
and never mentioned again. So the object files yield undefined symbol errors, because the libraries
they require were seen by the linker before it had discovered any undefined symbols, and were ignored.
In your untypical case, it resulted in libcudart and libcuda being seen later than your only
object file test_LayeredEEG.o - which however required no symbols from them - but earlier than
the only thing that did require symbols from them, the library libmpdcm. So they were ignored,
and you built a .mex64 shared library that had not been linked with them.
Long ago - pre-GCC 4.5 - shared libraries (like libcudart and libcuda) were exempt
from the requirement that they should be needed, at the point when the linker sees them,
in order to be linked. They were linked regardless, like object files, and the belief that
this is so has not entirely died out. It is not so. By default, shared libraries and
static libraries alike are linked if and only if needed-when-seen.
To avoid such traps it is vastly helpful to understand the canonical nomenclature of
the make variables involved in compilation and linkage and their semantics, and
their canonical use in compilation and linkage recipes for make. Mex is a
manipulator of C/C++/Fortran compilers that adds some commandline options of its own:
for make purposes, it is another compiler. For the options that it inherits from and
passes to the underlying compiler, you want to adhere to the usage for that compiler in make recipes.
These are the make variables most likely to matter to you and their meanings:
CC = Your C compiler, e.g. gcc
FC = Your Fortran compiler, e.g. gfortran
CXX = Your C++ compiler, e.g. g++.
LD = Your linker, e.g. ld. But you should know that only for specialized uses
should the linker be directly invoked. Normally, the real linker is invoked on your
behalf by the compiler. It can deduce from the options that you pass it whether you
want compiling done or linking done, and will invoke the appropriate tool. When you
want linking done, it will quietly augment the linker options that you pass with
additional ones that it would be very tiresome to specify, but which ensure
that the linkage acquires all the the correct flags and libraries for the language of the
program you are linking. Consequently almost always, specify your compiler as your
linker.
AR = Your archiving tool (static library builder)
CFLAGS = Options for C compilation
FFLAGS = Options for Fortran compilation
CXXFLAGS = Options for C++ compilation
CPPFLAGS = Options for the C preprocessor, for any compiler that uses it. Avoid the common mistake of writing CPPFLAGS when you mean CXXFLAGS
LDFLAGS = Options for linkage, N.B. excluding library options, -l<name>
LDLIBS = Library options for linkage, -l<name>
And the canonical make rules for compiling and linking:
C source file $< to object file $#:
$(CC) $(CPPFLAGS) $(CFLAGS) -c $# $<
Free-from Fortran file $< to object file $#, with preprocessing:
$(FC) $(CPPFLAGS) $(FFLAGS) -c $# $<
(without preprocessing, remove $(CPPFLAGS))
C++ source file $< to object file $#:
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $# $<
Linking object files $^ into an executable $#:
$(<compiler>) $(LDFLAGS) -o $# $^ $(LDLIBS)
If you can as much as possible write makefiles so that a) you have assigned the right options to the right variables from
this glossary, and b) used the canonical make recipes, then your path will be much smoother.
And BTW...
Your makefile has the following bug:
.PHONY = $(TDIR)
This is apparently an attempt to make $(TDIR) a phony target,
but the syntax is wrong. It should be:
.PHONY: $(TDIR)
what the assignment does is simply create a make variable called, .PHONY with the value of $(TDIR),
and does not make $(TDIR) a phony target.
Which is fortunate, because $(TDIR) is your output directory and not a phony
target.
You wish to ensure that make creates $(TDIR) before you need to output anything into
it, but you do not want it to a normal prequisite of those artefacts, which would oblige
make to rebuild them whenever the timestamp of $(TDIR) was touched. That is presumably
why you thought to make it a phony target.
What you actually want $(TDIR) to be is an order-only prerequsite
of the $(mTESTS) that will be output there. The way to do that is to amend the $(mTESTS) rule to be:
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o | $(TDIR)
This will cause $(TDIR) to be made, if needed, before $(mTESTS) is made, but
nevertheless $(TDIR) will not be considered in determining whether $(mTESTS) does
need to be made.
On the other hand, the targets all and clean are phony targets: no such artefacts
are to be made, so you should tell make so with:
.PHONY: all clean
As pointed out in the comments, the problem was in the order of the dynamic libraries in the compilation flags. After searching the reason for this I found in SO that static libraries need to be linked taking into account the order of dependency. In my case, the library libmpdc had dependencies on libcuda and libcudart but was on the left. The solution is to swap the order in the makefile from:
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ $(LOP1) -lmpdcm LDFLAGS="-lcudart -lcuda"
to
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ -lmpdcm $(LOP1)

UNIX 'diff' returns improperly when called using popen in main.c

I have a series of tests for a school project that involves building an assembler.
The tests use popen to catch the output of "make" and "./assembler" -- this is just so the output of these commands don't crowd the output of the test suite.
The problem is that the call to popen that executes "diff" returns a string inside the automated test suite, but not when I call it manually.
Here's the test suite code:
char buf1[256];
FILE* make;
FILE* assemble;
FILE *diff;
make = popen("make", "r");
assemble = popen("./assembler input/simple.s inter.txt out.txt", "r");
diff = popen("diff inter.txt out/ref/simple_ref.int", "r");
fgets(buf1, sizeof(buf1), diff);
printf("\nafter fgets simple -- strlen(buf) is %d\t buf is %s\n", strlen(buf1), buf1);
Here's the segment of the test suite output that corresponds with the above code:
after fgets simple -- strlen(buf) is 8 buf is 1,7c1,6
Here's the series of commands when called manually:
aweeeezy  ⋯  make
rm -f *.o assembler test-assembler core
gcc -g -std=gnu99 -Wall -o assembler assembler.c src/utils.c src/tables.c src/translate_utils.c src/translate.c
aweeeezy  ⋯  ./assembler input/simple.s inter.txt out.txt
Running pass one: input/simple.s -> inter.txt
Running pass two: inter.txt -> out.txt
Assembly operation completed successfully.
aweeeezy  ⋯  diff inter.txt out/ref/simple_ref.int
aweeeezy  ⋯  
When you use popen(), you're launching subprocesses which run at the same time as your program. In the code fragment you showed, the subprocesses will all be running around the same time without coordination.
To replicate the command line behavior, you want to wait until each subprocess is done; which is to say, call pclose() before moving to the next stages of your process. For examples, check something like "Pipes the easy way!" at http://www.tldp.org/LDP/lpg/node12.html

boost_python import error: module does not define init function

First off: I looked at the related questions, but they are not very helpful unfortunately. I'm trying to wrap an enum and a class from an external library.
#include <Python.h>
#include <boost/python.hpp>
using namespace boost::python;
#include <libvpsc/rectangle.h>
using vpsc::Rectangle;
using vpsc::Dim;
BOOST_PYTHON_MODULE(adaptagrams)
{
enum_<Dim>("dim")
.value("x", vpsc::XDIM)
.value("y", vpsc::YDIM)
.value("unset", vpsc::UNSET)
;
class_<Rectangle>("Rectangle",
init<double, double, double, double, optional<bool> >())
.add_property("centerX", &Rectangle::getCentreX)
.add_property("centerY", &Rectangle::getCentreY)
.add_property("width", &Rectangle::width, &Rectangle::set_width)
.add_property("height", &Rectangle::height, &Rectangle::set_height)
;
}
and compile with:
g++ -fPIC -I/usr/include/python2.7 -c adaptagrams.cpp -o adaptagrams.o
g++ -shared -Wl,-soname,adaptagrams.so -o adaptagrams.so adaptagrams.o -lpython2.7 -lboost_python -lvpsc
However, when I try to import the .so module, I get an error:
ImportError: dynamic module does not define init function (PyInit_adaptagrams)
Any ideas?
Update: When I restart Python and try the import, the first error I get is:
ImportError: ./adaptagrams.so: undefined symbol: _ZN8topology13computeStressERKSt6vectorIPNS_4EdgeESaIS2_EE
When I try it again, the 2nd one is the dynamic import from above (2.7) and a segfault (3.2). Boost is compiled against both 2.7 and 3.2 and I am linking the right ones on each approach.
Update 2: The tutorial code from the boost_python page works:
#include <Python.h>
#include <boost/python.hpp>
using namespace boost::python;
struct Hello
{
Hello(std::string msg): msg(msg) {}
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(constructor)
{
class_<Hello>("Hello", init<std::string>())
.def("greet", &Hello::greet)
.def("set", &Hello::set)
;
}
Same compilation:
g++ -fPIC -I/usr/include/python2.7 -c constructor.cpp -o constructor.o
g++ -shared -Wl,-soname,constructor.so -o constructor.so constructor.o -lpython2.7 -lboost_python
The name used in BOOST_PYTHON_MODULE must match the name of the .so library you generate and import into python.
I have seen this exception before. I got it using Visual Studio on windows, so things might be a little different over in unix-oid land but:
Two Possibilities:
Debug/Release miss-match:
You are trying to import a debug build of your module into a release build of python (or vice-versa). The solution is to include boost/python/detail/wrap_python.hpp instead of Python.h. This will fix some includes and defines to make it possible to do what you want.
Python/Boost.Python version miss-match:
Boost.Python is compiled against one specific version of python. You are using it with a different version. For example: you seem to be using python 2.7. Your boost_python library might be compiled against python 2.6. Yes, this means that your module can only work with one version of python at a time.
In addition to the other answers (in case another unfortunate soul runs unto this), make sure you're not accidentally compiling with the -fvisibility=hidden flag.
Doing so seems to strip the init function from the binary in both g++ / clang++.
Background info
In my case I had some trouble integrating some wrappers made with Boost.Python into a project. When built with the project's build-system I'd get the same runtime-error as OP (as opposed to building it with my proof-of-concept Makefile where it worked just fine).
Comparing the symbol tables with nm -g foo.so | grep Py showed me that in the non-working case the PyInit_* function was completely absent. Doing some comparison between compilation flags led me to -fvisibilty=hidden being the culprit.

make error when cross compile Perl for ARM

I have successfully configure perl for cross comile by using configure options:
./Configure -des -Dusecrosscompile \
-Dtargethost=172.17.185.91 \
-Dtargetdir=/home/perl/ \
-Dtargetuser=root \
-Dtargetarch=arm-linux \
-Dcc=arm-linux-gcc \
-Dusrinc=/opt/Mozart_Toolchain/arm-eabi-uclibc/include/ \
-Dincpth=/opt/Mozart_Toolchain/arm-eabi-uclibc/include/ \
-Dlibpth=/opt/Mozart_Toolchain/arm-eabi-uclibc/lib/
And the configure script tell me "Now you must run 'make'." But I encounter such as error when I make:
`sh cflags "optimize='-O2'" miniperlmain.o` miniperlmain.c
CCCMD = arm-linux-gcc -DPERL_CORE -c -DOVR_DBL_DIG=14 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -Wall
In file included from perl.h:38,
from miniperlmain.c:40:
config.h:4425:12: error: operator '==' has no left operand
In file included from miniperlmain.c:40:
perl.h:713:14: error: operator '>=' has no left operand
... ...
In config.h, some macro is left blank, for example:
#define INTSIZE /**/
#define LONGSIZE /**/
#define SHORTSIZE /**/
... much more ...
And I think it is the undefined macro result in the make error. I have no idea how to fix it. Why the macro is blank even if successfully configure?
Are there some guides to cross compile Perl?
There is a Cross directory that features a README file that includes the following instructions for arm-linux:
1) You should be reading me (README) in perl-5.x.y/Cross
2) Make sure you are in the Cross directory.
3) Edit the file 'config' to contain your target platform information.
4) make patch ## This will patch the existing source-tree.
5) make perl ## Will make perl
(Read the whole thing.)
I got the easiest way to cross compile Perl for arm-linux.Please refer to Cross-compiling perl. It's a great work! It saved my life.
Just according to instructions that give, you can get what you want. You may encounter such error when 'make':
pp_sys.c:78: error: non-thread-local declaration of 'h_errno' follows thread-local declaration
Simply comment that line.
Enjoy it!