In a Java project I am using matrix-toolkits-java (MTJ) for efficient matrix multiplication. This relies on netlib-java, which in turn relies on an optimized implementation of BLAS and LAPACK installed on the machine. It specifically looks for /usr/lib64/libblas.so.3 and /usr/lib64/liblapack.so.3 to find these libraries.
When installing blas and lapack via Yum, we get symbolic links /usr/lib64/libblas.so.3 and /usr/lib64/liblapack.so.3 pointing to the .so files from the reference blas and lapack installed via Yum.
Now we want to use implementations that are faster than the reference ones, in my case OpenBLAS. Irrelevant of whether I compile that myself, or install it via Yum, I end up with /usr/lib64/libopenblas-r0.2.18.so.
Now, according to any guide on the internet I am supposed to replace the symlinks to the reference implementation with symlinks to the OpenBLAS implementation, ending up with something like this:
libblas.so.3 -> libopenblas-r0.2.18.so
liblapack.so.3 -> libopenblas-r0.2.18.so
Okay, I can do that! I can do that with ln, or via alternatives. And if I do that, my code is happily using the fast OpenBLAS.
However, when ldconfig runs my awesome symbolic links are gone, they are overwritten by the reference BLAS and LAPACK installations. And then my software is sad and slow again.
So my question is, how to install OpenBLAS on CentOS/Fedora in such a way that running ldconfig will not destroy it? I can not remove the blas and lapack packages, as other clients of the host might rely on it. Rather I would somehow make the OS understand that OpenBLAS is an drop-in alternative to blas and lapack.
This article proposes installing openblas-compat via Yum. That fixes the issue for me. After installing the default blas package my software was still using OpenBLAS.
i don't think this behaviour is something that can be avoided without removing the reference implementations.
if you need to manually rebuild the ld.so.cache, you can run ldconfig -X to avoid updating links.
apart from that you will most likely have to create a custom script to restore the symbolic links to openblas after ldconfig updated the links
Related
I need to install SUMO 0.30.0 to be used with the VEINS_INET subproject in veins 4.6. I have tried following the instructions here and suggestions from forums but haven't had any luck being able to install sumo. I run ./configure (trying various tool/library options) then run sudo make but all I get is target marouter failed or nothing to be done for 'install-exec-am' 'install-data-am'.
Does anyone know how to install sumo-0.30.0 from source and/or make the veins_inet subproject work with the latest version of sumo-0.32.0?
Don't run sudo make.
Don't run sudo make.
Your problem is probably related to a dependency/packaging change in 16.04, which is explicitly pointed out in the veins tutorial:
Note that Ubuntu 16.04 no longer includes libproj0; this can be worked around by temporarily adding the packet repository of, e.g., Ubuntu Vivid when installing this package.
Short answer: Unfortunately this means that long-term, you're going to either have to package SUMO yourself, use the versions someone else compiled (see this launchpad for example) or rely on an old version.
Long answer:
In general, I would recommend building SUMO from source by building its' dependencies from source, since I've encountered this problem on various distributions. In particular, the fox, proj and gdal libraries tend to be packaged in different versions, and along with changes in the SUMO source code. I currently use this script (with the package versions downloaded) to compile SUMO -- but this is for 0.30.0, and it breaks if any of the referenced source packages are moved (which happens quite often). My general recommendation would be to either use a completely isolated version of SUMO (i.e., compiling by hand as much as possible) or relying on a pre-packaged version (see above), as long as that version is recent enough to work with VEINS.
I have created a relocatable RPM using the instructions in this website:
http://www.cyberciti.biz/faq/rpm-relocatable-packages/
This means that I can install the package into its default location, /opt/app, using
rpm -ivh mypackage.rpm
However, if I decide that I weant the package to be installed into /usr/local/bin instead, I can install it using:
rpm -ivh --prefix=/usr/local/bin mypackage.rpm
All of this works perfectly. However, I need to install the package via Yum. How do I pass the --prefix argument to Yum?
I don't believe this is possible and a quick online search seems to concur. Including this mailing list thread from 2007. The point that Seth Vidal makes in his reply is, I think, the main one. Relocations cause problems for file-based dependency tracking which, at least at the time, nothing bothered to handle.
That being said I think the utility of relocatable RPMs is likely not very high as building one that functions correctly is difficult as not being able to depend on file locations makes many (normally trivial) programmatic operations quite difficult.
Today I noticed this fairly old (and highly voted) question and it made me wonder, if pip is generally preferred over easy_install in the Python community, why is the official ipython.org documentation instructing newcomers like me to use easy_install as shown here?
Perhaps there is an implicit assumption that readers are using Python 2? Although the highly voted question and several answers all seem to indicate that pip is widely preferred over easy_install for both py2 and py3, so that seems like it's not the answer...
The command:
easy_install ipython[zmq,qtconsole,notebook,test]
will install IPython and some dependencies, including pyzmq, which has compiled parts.
easy_install handles binary packages, so it can install precompiled versions of things like pyzmq. pip, by contrast, only works with source packages. So for a package with compiled parts, it downloads the source code and tries to compile it. There are two problems with that:
The user needs to have a C compiler installed, plus whatever development headers the code being compiled relies on. The user often doesn't.
It's slow, especially for larger packages.
So the scientific Python world, which uses a lot of compiled packages, doesn't use pip as much as web developers. Actually, we don't much like easy_install either - we have a variety of more powerful installation methods.
I'm trying to build DBD::Pg on a linux host via the Makefile.PL; my requirements are such that I must be able to dynamically link against perl, but statically link against libpq.so (since it may not be available on all boxes).
Is there an easy way to do this? I have tried changing the link options in the LIBS directive of the Makefile.PL but MakeMaker ignores my options.
IMO you've mis-specified your requirements.
You do not need to statically link to libpq just because it might not be available on all systems.
What you should generally do instead is dynamically link to libq and either set LD_LIBRARY_PATH in a wrapper script or use rpath linking so that libpq can be found.
Be aware though that whether statically or dynamically linking, if some other module loads a libpq into the same Perl you'll either get two incompatible libpqs linked into the same executable (boom) or one of the modules using a libpq other than the one it was compiled against (also boom). If you use rpath linking, ld.so's awareness of link scopes might let you get away with it, but setting LD_LIBRARY_PATH will almost certainly cause issues.
You might want to look into using rpath with $ORIGIN.
Unfortunately, trying to do static linking of libpq is not likely to solve your problem at large.
libpq itself is likely to depend on libc (glibc). If you link it statically, but other modules dynamically, it means you will have 2 copies of libc: one inside libpq, another referenced by Perl itself and loaded dynamically. This is very dangerous situation, especially if some procedure allocates memory using malloc and passes pointer back to caller. If you have memory allocated by malloc from one copy of libc, but freeed by another copy, your program (and Perl) will definitely crash.
In other words, if you want to go static, you must go all way through - everything, 100% must be compiled statically, so only one copy of libc used by your application. And the opposite is true - if you are dynamic, everything should be dynamic, as to only ever use one copy of libc. These rules don't apply only if your libraries do not use anything from libc (not even sprintf).
Even if you succeed at static libpq compilation and it will work (not very likely), what if DBI is not installed? I have seen enough Linux boxes where DBI is not present by default. Do you compile DBI statically then as well? What if Perl is not present (as unlikely this is on Linux), or if it is very old?
Proper solution is to install it using native OS package manager:
sudo apt-get install libdbd-pg-perl # Ubuntu/Debian
sudo yum install perl-DBD-Pg # Redhat/Fedora
If you do not have root on hosts in question, maybe you should consider using perlbrew - install your own Perl in home directory. With this, you should be able to compile your own copy of libpq and link it with Perl provided by perlbrew dynamically.
I'm trying to setup a Perl development environment on my Mac laptop and have been having a really hard time getting it working. I thought I had everything configured correctly but when I try to run a sample script it is reporting errors with the DBI module and can't access the DB.
Here is what is reported in the Apache error logs:
[Fri Apr 30 23:11:33 2010] [error] [client 127.0.0.1] Can't locate DBI.pm in #INC (#INC contains: /Library/Perl/Updates/5.10.0/darwin-thread-multi-2level /Library/Perl/Updates/5.10.0 /System/Library/Perl/5.10.0/darwin-thread-multi-2level /System/Library/Perl/5.10.0 /Library/Perl/5.10.0/darwin-thread-multi-2level /Library/Perl/5.10.0 /Network/Library/Perl/5.10.0/darwin-thread-multi-2level /Network/Library/Perl/5.10.0 /Network/Library/Perl /System/Library/Perl/Extras/5.10.0/darwin-thread-multi-2level /System/Library/Perl/Extras/5.10.0 .) at main.pm line 5.
I downloaded and installed both modules manually to work with MAMP using the following commands as specified in this forum post:
For DBI
1. cd /Library/Perl/DBI-1.611
2. sudo Perl Makefile.PL
3. sudo make
4. sudo make install
For DBD
1. cd /Library/Perl/DBD-mysql-4.014
2. sudo Perl Makefile.PL --mysql_config=/Applications/MAMP/Library/bin/mysql_config
3. sudo make
4. sudo make install
What I noticed while running the above commands is that the files seems to be getting installed in the '/opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/' directory which doesn't seem to be one of the search directories that Apache mentions in the error at the beginning of this post. Here is what I'm seeing during the install:
$ sudo make install
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/DBI.bundle
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/dbipport.h
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/DBIXS.h
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/dbixs_rev.h
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/Driver.xst
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/Driver_xst.h
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/TASKS.pod
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBD/DBM.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBD/File.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBD/Gofer.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI/Changes.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI/DBD.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI/Profile.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI/ProxyServer.pm
Installing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/DBI/PurePerl.pm
Installing /opt/local/share/man/man3/DBD::DBM.3pm
Installing /opt/local/share/man/man3/DBD::File.3pm
Installing /opt/local/share/man/man3/DBD::Gofer.3pm
Installing /opt/local/share/man/man3/DBI.3pm
Installing /opt/local/share/man/man3/DBI::DBD.3pm
Installing /opt/local/share/man/man3/DBI::Profile.3pm
Installing /opt/local/share/man/man3/DBI::ProxyServer.3pm
Installing /opt/local/share/man/man3/DBI::PurePerl.3pm
Installing /opt/local/share/man/man3/TASKS.3pm
Installing /opt/local/bin/dbiprof
Installing /opt/local/bin/dbiproxy
Writing /opt/local/lib/perl5/site_perl/5.8.9/darwin-2level/auto/DBI/.packlist
Appending installation info to /opt/local/lib/perl5/5.8.9/darwin-2level/perllocal.pod
My question is, what am I doing wrong and how can I either 1) Get Apache to look in the right directory where the DBD & DBI modules are installed or 2) Update the way I'm installing the module to install them into one of the search directories. I honestly don't know what option makes more sense and could use guidance on that as well.
As you can probably tell I'm pretty lost at the moment. Please help!!! Thanks in advance.
It looks like you've already installed another Perl via macports (/opt/local is where all macports installations go), and /opt/local/bin is earlier in your $PATH than the system Perl in /usr/bin. That's fine, if you are happy running Perl 5.8.9 rather than Perl 5.10.0 (hint: if you aren't sure of the differences, then the differences don't matter).
It's usually advised to not make extra installations to the system Perl. Apple may upgrade components through regular system updates, which could interfere with any modifications you have made, and if you make a mistake with an installation, it's difficult to remedy it without doing a full system reinstallation or having some serious understanding of the operating system guts. So, since you've already got another Perl installation ready, I would strongly encourage you to stick with that one.
However, you probably shouldn't be manually installing libraries if there is already a distribution available on macports. I used port search dbi and port search dbd to find them: the distributions are named p5-dbi and p5-dbd-mysql. You can install those like any other macports module: with sudo port install <distroname>. (You may need to install mod_perl itself, too.)
After that, you simply need to tell Apache/mod_perl to use that Perl installation rather than the system perl. I've never done that, so I can't advise on the best way to do it. However, quick searches on http://superuser.com suggest that the macports version of apache will run by default (via the same $PATH ordering), so I'd just Try It And See :).
Great answer, Ether. Having done this far too many times to count, I can give you a few pieces of advice:
Note: I am apparently limited to a single link in the post, so I had to remove all of my annotations. Thankfully, there is Delicious where I've stored them all with a stackoverflowmacports tag. Any place below where I removed a link to fit under Stack Overflow's ridiculous anti-spam measure, I've marked it with (*).
If having a reliably-working development environment at all times is important to you, rely on as LITTLE Mac OS X bundled software as possible. I love Apple but they have absolutely no qualms about breaking custom setups of their software as often as possible.
If #1 sounds like what you need to do, Macports is an EXCELLENT choice. I used to use Fink but they got left in the dust ages ago in terms of ease of use and spectrum of available software. The easiest route to installing macports is via the binary package install method (*)
As Ether mentions, when you have everything set up correctly, the Macports-provided MySQL, PHP and Apache all work together well without the system-installed analogs interfering. Most of that has to do with your PATH setting but all of those details are handled by the package installers post-flight script (*)
Once you're on the Macports train, it should become the very first place you look for any software. port search and port info are constant companions. They've got 6863 ports (*) currently which covers MOST of your bases.
When you do need to go outside of the Macports realm to find something, install it in /usr/local. That part of the file system hierarchy is yours to play with. Don't be lulled into a false sense of security and think that because Macports doesn't have what you're installing, it's okay to put it in /opt/local because invariably that software will install some dependency that will ALSO be a dependency for some piece of Macports software down the line and Macports will not allow a port to be installed if any one of its files would overwrite an existing file not managed by Macports (unless you force it which is always bad manners)
If you do any work with Perl and you use Macports' version, you will absolutely find yourself in a situation Macports doesn't have the one CPAN module you're looking for. (And, really, given that there are two and a half billion CPAN modules, who can blame them?). This will happen often enough that you will most likely tire of the manual installation method (*) (perl Makefile.PL; make; make test; sudo make install; cha; cha; cha) and long for the ease of use you've grown accustomed to with Macports.
If so, you can absolutely use the cpan (*) utility, CPANPLUS (*) or cpanminus (*) for all of your installing needs. Just make sure to make the necessary adjustments in the configuration of your tool of choice to instruct it to install your modules into /usr/local/lib/perl5, ignoring /opt/local/bin/perl's insistence that modules go into /opt/local/lib/perl5. You can set the PERL5LIB environment variable in your shell's init scripts to additionally look in /usr/local/lib/perl5 for modules. Just grab the #INC output from perl -V and tack it on the end...
And finally... Leveraging the system's daily init scripts or third-party software like Anacron (*) or MacPorts Notifier (*) (both available through MacPorts), make sure to update your software frequently. You don't have Mother Apple protecting you with Software Updates for the Macports installed software which have just as many bugs and security exploits as the very same software Apple bundles.
By updating frequently, you'll stay ahead of the baddies and by automating it, the upgrades will actually happen and you won't end up as I have in the past with a full weekend blown because you had a mountain of outdated ports to upgrade. Note: Macports stages its updates and if it fails at any point, your current version continues to work. Apple could learn a thing or two from them, I tell ya...
So, that's all I can think of a the moment. Hopefully the lessons above will get you going quickly and save all of the time, effort and stress I've experienced in past years in learning it. I would argue it's still far better than the alternative: Hating Apple because they break all of your nice things... :-)
I had a similar problem with Apache using the wrong Perl. I fixed it by appending the following lines to my httpd.conf file:
SetEnv PATH [colon-separated list of directories]
SetEnv PERL5LIB [colon-sep'd list of directories]
In my case, it looked like this:
SetEnv PATH /opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/local/mysql/bin:$PATH
SetEnv PERL5LIB /opt/local/lib:/usr/local/lib