Control mex link options with g++ from command line - matlab

Question

I'm trying to link a library using mex from command line, or more exactly, from a makefile. I do this from a Makefile which I post here:
BDDM_MATLAB = #matlabhome#
MEXCC = $(BDDM_MATLAB)/bin/mex
MEXFLAGS = -v -largeArrayDims -O
MEXEXT = mexa64
TDIR = $(abs_top_srcdir)/test
IDIR = $(abs_top_srcdir)/src
LDIR = $(abs_top_srcdir)/lib
LOP1 = $(CUDA_LDFLAGS) $(LIBS)
SOURCES := $(wildcard *.cpp)
OBJS = $(SOURCES:.cpp=.o)
mTESTS = $(addprefix $(TDIR)/, $(SOURCES:.cpp=.$(MEXEXT)))
all: $(TDIR) $(mTESTS)
$(OBJS) : %.o : %.cpp
$(MEXCC) $(MEXFLAGS) -c -outdir ./ -output $# $(CUDA_CFLAGS) -I$(IDIR) CFLAGS="\$$CFLAGS -std=c99" $^
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ $(LOP1) -lmpdcm LDFLAGS="-lcudart -lcuda"
.PHONY = $(TDIR)
$(TDIR):
$(MKDIR_P) $#
clean:
$(RM) *.o
libmpdcm is a static library that includes calls to two shared libraries libcuda and libcudart. My environment has
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH:
My make rule produces
/usr/local/MATLAB/R2014a/bin/mex -v -largeArrayDims -O -L/home/eaponte/projects/test_cpp/lib -outdir /home/eaponte/projects/test_cpp/test test_LayeredEEG.o -L/usr/local/cuda/lib64 -lcudart -lcuda -lmpdcm LDFLAGS="-lcudart -lcuda"
This produces the following g++ command:
/usr/bin/gcc -lcudart -lcuda -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o -lcudart -lcuda -lmpdcm -L/home/eaponte/projects/test_cpp/lib -L/usr/local/cuda/lib64 -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64
The problem is that afterwards I get a linking error in Matlab:
Invalid MEX-file '/home/eaponte/projects/test_cpp/test/test_Fmri.mexa64': /home/eaponte/projects/test_cpp/test/test_Fmri.mexa64: undefined symbol: cudaFree
I know that the solution is simply to put the cuda libraries at the end of the g++ command
/usr/bin/gcc -lcudart -lcuda -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o -lmpdcm -L/home/eaponte/projects/test_cpp/lib -L/usr/local/cuda/lib64 -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -lcudart -lcuda -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64
How can achieve that running mex from command line (or from a Makefile)?

Answer 1

Just to illuminate the problem and solution and offer some help in avoiding the like:
The fundamental rule of linkage with the GNU linker
that your problem makefile transgressed is: In the commandline sequence of entities to be linked, ones that need symbol definitions
must appear before the ones that provide the definitions.
An object file (.o) in the linkage sequence will be incorporated entire in the output executable,
regardless of whether or not it defines any symbols that the executable uses. A library
on the other hand, is merely examined to see if it provides any definitions for symbols
that are thus-far undefined, and only such definitions as it provides are linked into in
the executable (I am simplifying somewhat). Thus, linkage doesn't get started until some object file is seen,
and any library must appear after everything that needs definitions from it.
Breaches of this principle usually arise from inept bundling of some linker flag-options
and some library-options together into a make-variable and its placement in the linkage recipe,
with the result that the bundled options are interpolated at a position that is valid for
the flags but not valid for libraries. This was so in your problem makefile, with LOP1 the
bad bundle.
In the typical case, the bundling causes all of the libraries to be placed before all the object files,
and never mentioned again. So the object files yield undefined symbol errors, because the libraries
they require were seen by the linker before it had discovered any undefined symbols, and were ignored.
In your untypical case, it resulted in libcudart and libcuda being seen later than your only
object file test_LayeredEEG.o - which however required no symbols from them - but earlier than
the only thing that did require symbols from them, the library libmpdcm. So they were ignored,
and you built a .mex64 shared library that had not been linked with them.
Long ago - pre-GCC 4.5 - shared libraries (like libcudart and libcuda) were exempt
from the requirement that they should be needed, at the point when the linker sees them,
in order to be linked. They were linked regardless, like object files, and the belief that
this is so has not entirely died out. It is not so. By default, shared libraries and
static libraries alike are linked if and only if needed-when-seen.
To avoid such traps it is vastly helpful to understand the canonical nomenclature of
the make variables involved in compilation and linkage and their semantics, and
their canonical use in compilation and linkage recipes for make. Mex is a
manipulator of C/C++/Fortran compilers that adds some commandline options of its own:
for make purposes, it is another compiler. For the options that it inherits from and
passes to the underlying compiler, you want to adhere to the usage for that compiler in make recipes.
These are the make variables most likely to matter to you and their meanings:
CC = Your C compiler, e.g. gcc
FC = Your Fortran compiler, e.g. gfortran
CXX = Your C++ compiler, e.g. g++.
LD = Your linker, e.g. ld. But you should know that only for specialized uses
should the linker be directly invoked. Normally, the real linker is invoked on your
behalf by the compiler. It can deduce from the options that you pass it whether you
want compiling done or linking done, and will invoke the appropriate tool. When you
want linking done, it will quietly augment the linker options that you pass with
additional ones that it would be very tiresome to specify, but which ensure
that the linkage acquires all the the correct flags and libraries for the language of the
program you are linking. Consequently almost always, specify your compiler as your
linker.
AR = Your archiving tool (static library builder)
CFLAGS = Options for C compilation
FFLAGS = Options for Fortran compilation
CXXFLAGS = Options for C++ compilation
CPPFLAGS = Options for the C preprocessor, for any compiler that uses it. Avoid the common mistake of writing CPPFLAGS when you mean CXXFLAGS
LDFLAGS = Options for linkage, N.B. excluding library options, -l<name>
LDLIBS = Library options for linkage, -l<name>
And the canonical make rules for compiling and linking:
C source file $< to object file $#:
$(CC) $(CPPFLAGS) $(CFLAGS) -c $# $<
Free-from Fortran file $< to object file $#, with preprocessing:
$(FC) $(CPPFLAGS) $(FFLAGS) -c $# $<
(without preprocessing, remove $(CPPFLAGS))
C++ source file $< to object file $#:
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $# $<
Linking object files $^ into an executable $#:
$(<compiler>) $(LDFLAGS) -o $# $^ $(LDLIBS)
If you can as much as possible write makefiles so that a) you have assigned the right options to the right variables from
this glossary, and b) used the canonical make recipes, then your path will be much smoother.
And BTW...
Your makefile has the following bug:
.PHONY = $(TDIR)
This is apparently an attempt to make $(TDIR) a phony target,
but the syntax is wrong. It should be:
.PHONY: $(TDIR)
what the assignment does is simply create a make variable called, .PHONY with the value of $(TDIR),
and does not make $(TDIR) a phony target.
Which is fortunate, because $(TDIR) is your output directory and not a phony
target.
You wish to ensure that make creates $(TDIR) before you need to output anything into
it, but you do not want it to a normal prequisite of those artefacts, which would oblige
make to rebuild them whenever the timestamp of $(TDIR) was touched. That is presumably
why you thought to make it a phony target.
What you actually want $(TDIR) to be is an order-only prerequsite
of the $(mTESTS) that will be output there. The way to do that is to amend the $(mTESTS) rule to be:
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o | $(TDIR)
This will cause $(TDIR) to be made, if needed, before $(mTESTS) is made, but
nevertheless $(TDIR) will not be considered in determining whether $(mTESTS) does
need to be made.
On the other hand, the targets all and clean are phony targets: no such artefacts
are to be made, so you should tell make so with:
.PHONY: all clean

Answer 2

As pointed out in the comments, the problem was in the order of the dynamic libraries in the compilation flags. After searching the reason for this I found in SO that static libraries need to be linked taking into account the order of dependency. In my case, the library libmpdc had dependencies on libcuda and libcudart but was on the left. The solution is to swap the order in the makefile from:
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ $(LOP1) -lmpdcm LDFLAGS="-lcudart -lcuda"
to
$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
$(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ -lmpdcm $(LOP1)