subprocess not calling multiqc in pycharm - subprocess

I think it is an environment issue, but I am new to linux.
I try to run subprocess.run('multiqc .') in pycharm
but it shows me the file is not found
FileNotFoundError: [Errno 2] No such file or directory: 'multiqc .'
and echo $PATH shows that anaconda path is exposed
$ echo $PATH
/media/SSD1T/Software/anaconda3/bin:
I tried subprocess.run('multiqc', shell = True), it returns error: /bin/sh: 1: multiqc: not found
while in terminal, I can call multiqc no problem
$which multiqc
/media/SSD1T/Software/anaconda3/bin/multiqc
tried run other commands in pycharm, no problem:
subprocess.run('ls')
In[11]: subprocess.run("ls")
loadSamples.py
venv
xFastqc.py
xTrim.py
Out[11]: CompletedProcess(args='ls', returncode=0)
The only way I can make it work is to add the whole path to subprocess.run
In[12]: subprocess.run("/media/SSD1T/Software/anaconda3/bin/multiqc")
Usage: multiqc [OPTIONS] <analysis directory>
Error: Missing argument "analysis_dir".
This is MultiQC v1.5
For more help, run 'multiqc --help' or visit http://multiqc.info
Out[12]: CompletedProcess(args='/media/SSD1T/Software/anaconda3/bin/multiqc', returncode=2)
My question is: how can I just call multiqc in subprocess.run without put in the whole path?
Thank you

So I bypassed this problem by adding a link to /usr/bin. If you have a better solution,
please let me know.
Thanks

This is an old question, but for the sake of future googlers I can try to help. I think the problem is that using subprocess opens up a new shell where the conda environment with MultiQC is not activated. Because of this, multiqc is not available on the PATH, and you have to specify the full absolute path to the binary.
If I'm right about this, the problem is not so much about MultiQC, but more about using conda within a subprocess call. There are several other Stack Overflow questions on this topic. The quick and dirty method is to just activate the relevant conda environment within the same subprocess call:
subprocess.run('source activate base && multiqc .', shell = True)
But, I have good news! As of MultiQC v1.8 (which should be released in a few days) and pull-request #1042, you can now import MultiQC into python scripts and run it as a function. This is much easier and more portable. You can see an example here: https://github.com/MultiQC/MultiQC_Notebook
Basic script:
import IPython
import multiqc
multiqc.run('./path/to/data/')
IPython.display.IFrame('./multiqc_report.html', '100%', 600)

Related

BiqQuery detected on windows command prompt but not on Bash

I've installed Cloud SDK on my windows 10 machine and I'm able to run commands like "gcloud", "gsutil" and "bq" on my command prompt. However, when I run "gsutil" or "bq" on Bash, this is the error I'm getting.
$ bq
bash: bq: command not found
Then I added this location to PATH C:\Users\User\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin, and afterwards I seem to be able to call gsutil and bq using gsutil.cmd and bq.cmd. But of course, I'd prefer calling bq / gsutil directly.
Has anyone has this problem before? Thank you!
I found one possible solution!
In Bash, go to the root bash folder by typing cd
Type touch .bashrc
write alias bq="bq.cmd" or alias gsutil="gsutil.cmd" or both
Press Esc and type :qa and Enter to save and exit
This tells Bash to remember bq as invoking bq.cmd, hence now I can invoke bq anywhere -- as far as I know.
If anyone has a better suggestion, please let me know! Thank you!
In the bash session itself, check what echo $PATH returns. It should inherit the path that was defined on Windows.
So open a new CMD session, and:
make sure the %PATH% does include C:\Users\User\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin,
then call C:\path\to\Git\bin\bash.exe to open a bash session
bq should work in such a shell session.

second line on my system or python terminal now saying: “ -bash: zzzzz#: command not found“

I have been trying to pip install psycopg2 for some time now
I have just updated to python 3.7.4, before this problem started.
To set my path to a specific python version I used the code below.
nano .bash_profile
I thought that it would now be easy for my system to identify the path of the newly installed python, as to enable it to install psycopg2. Then the below started happening.
The second line of system terminal or python terminal is now always showing:
-bash: zzzzz#: command not found on my terminal
No matter what I type on my terminal, I am always getting command not found
This would mean you literally have "zzzzz" somewhere in the bash_profile. Bash is seeing "zzzzz" as just another command to run at startup like the rest of the profile script. As there is nothing in your PATH matching that string, bash reports the issue back to you.
Either remove the extra line from your .bash_profile. OR use a terribly wasteful work-around!
ln -s /bin/true /bin/zzzzz
This will create a symbolic link to the "true" binary (all it ever does is return true) from zzzzz. Now bash can find zzzzz and run it during start up, which does nothing. No more error and an absurd work around. You should fix the file.

Executing terminal commands in Jupyter notebook

I am trying to run the following in Jupyter notebook (with Python 2 if it makes a difference):
!head xyz.txt
and I get the following error:
'head' is not recognized as an internal or external command, operable
program or batch file.
Is there anything I need to import to be able to do this?
Might be useful for others.
Use ! followed by terminal command you want to execute. To run a shell command. E.g.,
! pip install some_package
to install the some_package.
An easier way to invoke terminal using jupyter-notebooks is to use magic function %%bash and use the jupyter cell as a terminal:
%%bash
head xyz.txt
pip install keras
git add model.h5.dvc data.dvc metrics.json
git commit -m "Second model, trained with 2000 images"
For Windows it would be %%cmd.
Write it at the beginning of the cell like this :
%%cmd
where python
myprogram "blabla" -x -y -z
You can start the cell with the magic % bash before the rest of your code. There is an example in this blog post, together with a list of some of the most useful magics.
Make sure you run your command in linux shell because there is non such command in windows.
Another option nowadays is the Jupyter kernel for Bash.
I had the same issue. Solved by running
!bash -c "head xyz.txt"

pycuda -- 'CUDA_ROOT not set, and nvcc not in path.'

Although i had installed pycuda and using it ok,it started (without doing sth) not to work.So,i i tried to do the install again ,but when i am doing
python configure.py --cuda-root=/usr/local/cuda/bin
it gives me the error in the title.
The nvcc file is in the above directory.
pycuda is not finding nvcc. Did you try adding /usr/local/cuda/bin to your env PATH variable? That's the way I have this setup.
Edit:
As far as I can tell the configure.py doesn't call nvcc compiler it just creates the the makefile. I take that this problem happens when you run sudo -c "make install" which calls setup.py.
A couple of things to try. Make sure that you have CUDA_ROOT set:
echo $CUDA_ROOT
If it's empty, set it with:
export CUDA_ROOT=/usr/local/cuda/bin
Try running the make command again. Now with the -E to preserve your env:
sudo -E sh -c "make install"
I encountered the same issue on a Slackware64 13.37.
Install command su -c "make install" switches to root (0bv10u5Ly) thus CUDA_ROOT should be set in the root's profile. CUDA_ROOT is not an environment variable, it's used by the setup.py. Add /usr/local/cuda/bin to PATH and define CUDA_ROOT=/usr/local/cuda/bin then try to install again.
This is the quick and dirty way but if none of above worked out for you like me, below will definitely work. (:
Remove
nvcc_path = search_on_path(["nvcc", "nvcc.exe"])
if nvcc_path is None:
print("*** CUDA_ROOT not set, and nvcc not in path. Giving up.")
sys.exit(1)
and set
cuda_root_default = "/usr/local/cuda/bin"
in setup.py file. Then try su -c "make install".
In my case, I had to set CUDA_ROOT=/usr/local/cuda because with /usr/local/cuda/bin path, it was not able find include folder and it was failing with error didn't find cuda.h.

`GLIBCXX_3.4.11' not found, run system call from MATLAB that links to glibc different than what's in matlab bin path

I'm trying to circumvent using MEX to link to MATLAB and just call a binary using "!" as in:
>> !template_image_rigid -args ....
template_image_rigid: /opt/MatlabR2010a/sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /usr/lib/libboost_program_options.so.1.40.0)
template_image_rigid: /opt/MatlabR2010a/sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /usr/lib/libdirac_encoder.so.0)
Is there a way to easily fix this dynamic link issue from within MATLAB? I know I can recompile the source with MATLAB and use a MEX call, but since it takes a while to run the solver I just want to run it as shell command and import text data later into MATLAB.
If it helps, the source & CMakeLists.txt can be found here: https://github.com/pkarasev3/nlmagick/tree/master/samples
Grr, community = fail.
Diagnoising: do !gnome-terminal from within matlab and look at "env":
env | grep Matlab
which gives
XKEYSYMDB=/opt/MatlabR2010a/X11/app-defaults/XKeysymDB
MATLABPATH=/opt/MatlabR2010a/toolbox/local
XAPPLRESDIR=/opt/MatlabR2010a/X11/app-defaults
LD_LIBRARY_PATH=/opt/MatlabR2010a/sys/os/glnxa64:/opt/MatlabR2010a/bin/glnxa64:/opt/MatlabR2010a/extern/lib/glnxa64:/opt/MatlabR2010a/runtime/glnxa64:/opt/MatlabR2010a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/opt/MatlabR2010a/sys/java/jre/glnxa64/jre/lib/a md64/server:/opt/MatlabR2010a/sys/java/jre/glnxa64/jre/lib/amd64
OSG_LD_LIBRARY_PATH=/opt/MatlabR2010a/sys/openscenegraph/lib/glnxa64
TOOLBOX=/opt/MatlabR2010a/toolbox
XFILESEARCHPATH=/opt/MatlabR2010a/sys/java/jre/glnxa64/jre/lib/locale/%L/%T/%N%S::/usr/dt/app-defaults/%L/Dt
MATLAB=/opt/MatlabR2010a
Ok so the LD_LIBRARY_PATH is bad.
Trick: write a poltergeist script and run it from gnome-terminal, Launch it from Matlab with:
!./hack.sh RunStuffThatLinksElsewhere
where hack.sh is a script with something like:
#!/bin/bash
source ~/.bashrc
export LD_LIBRARY_PATH=''
gnome-terminal --command="${1}"
so an easy test is to try it with "eog", this hack gets around the link issue and lets you run it from within matlab...
Simpler:
setenv('foo',num2str(some_value) )
!LD_LIBRARY_PATH="" && ./my_binary -f $foo
disp('done with external program!')
I solved this problem by replacing the version of libstdc++.so.6 with a newer version from my system (I use ubuntu 12.04).
First find the system version of libstdc++.so.6.
From the command line type:
locate libstdc++.so.6
My system version of libstdc was
/usr/lib/i386-linux-gnu/libstdc++.so.6
Then replace the matlab libstdc version with a link to the system libstdc.
From the command line type (replace [....] with you settings):
cd [matlab_dir]/sys/os/glnx86
mv libstdc++.so.6 libstdc++.so.6-OLD
ln -s [your_system_version_of_libstdc] libstdc++.so.6
I recently ran into the same problem. My solution also uses a poltergeist script like other answers. The script is as follows (poltergeist.sh):
#!/bin/bash
export LD_LIBRARY_PATH=''
eval "$#"
It basically resets the library path and subsequently evaluates the call given by the arguments to the script. From within matlab I then call in this manner:
system([pwd,'/poltergeist.sh echo hello world!']);
The advantage to this approach is that you can dynamically modify the call command within matlab. As far as I know this is not possible using the bang syntax in the currently provided answers.