How to run shell script file on ipython at Google Colab - ipython

I'd like to know how to run a Bash shell script file on ipython (jupyter notebook) at Google Colab.
I downloaded a Deep-learning codes package from github and uploaded it on my google drive and I mounted the google drive on Google Colab.
The code package includes '*.py' python codes and 'fn.sh' script file.
By executing the script file the python codes can be executed.
I tried os.system('fn.sh') and subprocess.call('fn.sh') on the ipython prompt at Google Colab but they doesn't work like below.
1)
import os
os.system('drive/DL/denet-master/examples/simple-cifar10.sh')
32256
import subprocess
subprocess.call('drive/DL/denet-master/examples/simple-cifar10.sh')
OSError: [Errno 8] Exec format error: 'drive/DL/denet-master/examples/simple-cifar10.sh'

In Colab, you can invoke shell commands using either ! or %%shell.
Your invocation above would be:
!drive/DL/denet-master/examples/simple-cifar10.sh
Here's an example notebook:
https://colab.research.google.com/drive/1N7p0B-7QWEQ9TIWRgYLueW03uJgJLmka

Option 1
Use ! as other answers mentioned.
!ls -la
!echo "Hello"
!bash path/to/script.sh
Options 2
Use python to write a script, then execute it with !bash script.sh. Paste the following snippet to a cell to run a speed test example.
sh = """
curl ipinfo.io; echo
if ! hash ping &>/dev/null; then
echo "Installing ping tools ..."
apt-get install iputils-ping -y &>/dev/null
fi
curl ninh.js.org/speed.sh -sL | bash
"""
with open('script.sh', 'w') as file:
file.write(sh)
!bash script.sh
It should show something like this

!bash drive/DL/denet-master/examples/simple-cifar10.sh

here is way
first run and save the output to a text file like this
import os
os.system("pip list > file.txt")
then read the output from the file
import os
with open("file.txt","r") as file:
print(file.read())

For google colab use
!/content/folder/file.sh
you can copy path of the .sh file. Make sure you use a '/' before the content. This worked for me.
I didn't use !bash or !sh

Related

Bun Not Found After Running Installation Script

I have ran the installation script by pasting this code:
$ curl https://bun.sh/install | bash
However, when I try to get the version of bun, it says it could not find it:
$ bun --version
Command 'bun' not found, did you mean:
command 'ben' from deb ben (0.9.0ubuntu2)
command 'bus' from deb atm-tools (1:2.5.1-4)
command 'zun' from deb python3-zunclient (4.0.0-0ubuntu1)
Try: sudo apt install <deb name>
I had the same issue running on Windows 10 WSL2 Ubuntu-22.04 with Bun v0.1.5.
The solution (and more detail just in case anyone needs it) below:
The executable for bun is in the directory "/home/username/.bun". You need to add this to your $PATH so that this can be found when typing bun commands such as "bun --help".
The bun setup process does not add this path so you need to do it yourself.
Two ways to do this :
(1) Manual method
Type in the terminal:
export BUN_INSTALL="/home/YOUR_USERNAME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
Replacing YOUR_USERNAME with your real username (the current username can be found by typing 'whoami' in the terminal).
Note: This process will have to be REPEATED for every new shell you open.
(2) Automatic method
Edit the .bashrc file :
nano ~/.bashrc
at the end of this file add
BUN_INSTALL="/home/YOUR_USERNAME/.bun"
PATH="$BUN_INSTALL/bin:$PATH"
Replacing YOUR_USERNAME with your real username (the current username can be found by typing 'whoami' in the terminal).
(Remember to save your changes with Ctrl-O)
Note: You will NEED TO OPEN A NEW SHELL for this to work OR type 'source ~/.bashrc' to use in the current terminal.
You should now be able to run bun commands in any new shell.
The installation script says a message at the end telling you how to add bun to your PATH manually. Here is that output:
Manually add the directory to your $HOME/.bashrc (or similar)
BUN_INSTALL="/home/sno2/.bun"
PATH="$BUN_INSTALL/bin:$PATH"
I advise you re-run the installation command and copy the environment variables and add them to your PATH.
export BUN_INSTALL="/Users/manendra/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
add these to your .bashrc, .zshrc or you can use export command to use for current session.
Note: Change your username place of (manendra) "/Users/manendra/.bun"
Manually add the directory to ~/.bashrc (or similar):
export BUN_INSTALL="$HOME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
From the installer, last message is:
To get started, run
exec /bin/zsh
bun --help

Why does the WSL2 import command in PowerShell output "Access is denied"?

On Windows 10 Pro and 11 Pro I have installed and activated Ubuntu-20.04 and Debian. Using the documentation from MS on switching those distros to a secondary drive, everything seemed to work fine. Until the WSL import command. It outputs "Access is denied". I've tried Windows Terminal, PowerShell, and even WebStorm; I get the same output.
I am running with elevated privileges but to no avail. The export works fine, I use a different name as the source file to ensure I restore the name to its original name. The wsl.conf editing looks good, everything lines up... until the import command.
I am at a loss. I've exhausted all research. Can anyone help me resolve this so I can run these from my F: drive?
Cheers,
RN
You just have to put a filename in the end, like:
wsl --export Ubuntu C:\Users\Desktop\OneDrive\Documents\ubuntu.tar
Suppose you want to import an exported distribution "ubuntu.tar".
Try to cd at the location of the .tar file before executing the wsl --import command in PowerShell (running as standard user), for example:
PS X:\> cd D:\
PS D:\> wsl --import Ubuntu_copy .\Ubuntu_copy ubuntu.tar
Executing the wsl --import command with an absolute path didn't work for me, but the above mentioned method did.
Just in case this is an ongoing issue for anyone, you need to run wsl --import not just from an Administrator account, but you need to run Powershell/cmd as Administrator, for example by right-clicking a pwsh.exe icon/shortcut and clicking "Run as administrator". If you're running as a standard user and "Run as administrator", the import will install the distro for the admin user you've chosen to run as.
The full syntax is:
wsl --import <Distro name> <Install folder> <Source .tar file>
The import syntax is as following, you should be carefull about install dir and imported tar file's arguments order:
--import <Distro> <InstallLocation> <FileName> [Options]
Imports the specified tar file as a new distribution.
The filename can be - for standard input.
Options:
--version <Version>
Specifies the version to use for the new distribution.

Can't run ibm_db in Jupyter Notebook

I am trying to run ibm_db in a jupyter notebook. When I run ibm_db I get the below error.
ImportError Traceback (most recent call last)
in ()
----> 1 import ibm_db
ImportError: dlopen(/Users/myName/anaconda/envs/householding/lib/python3.6/site-packages/ibm_db.cpython-36m-darwin.so, 2): Library not loaded: libdb2.dylib
Referenced from: /Users/myName/anaconda/envs/householding/lib/python3.6/site-packages/ibm_db.cpython-36m-darwin.so
Reason: image not found
When i run os.getcwd() I get '/Users/myName'
What I think is happening is that because my current directory is to levels down from the start of the path dlopen is looking for, it is failing. I've done some looking around but can't find a way to change where dlopen is looking
You have to actually update your environment variable
DYLD_LIBRARY_PATH
to include
/ibm_db-2.0.8-py3.6-macosx-10.6-intel.egg/clidriver/lib
If you have installed ibm_db-2.0.8 on python3.6,
On terminal write
export DYLD_LIBRARY_PATH=/Users/myName/anaconda/envs/householding/lib/python3.6/site-packages/ibm_db-2.0.8-py3.6-macosx-10.6-intel.egg/clidriver/lib
It should work like a charm after this.
For reference checkout this:Issues with MAC OS X
I was having the same error and found that the installDSDriver script creates a file at /Applications/dsdriver/db2profile stating the below:
# NAME: db2profile
#
# FUNCTION: This script sets up a default database environment for
# Bourne shell or Korn shell users.
#
# This file is tuned for IBM Data Server Driver Package only.
#
# USAGE: . db2profile
# This script can either be invoked directly as above or
# it can be added to the user's .profile file so that the
# database environment is established during login.
#
so I just added on my ~/.bash_profile the line below:
source /Applications/dsdriver/db2profile
Open a new terminal window or restart and should work.
This file exports all the environment variables needed for the db2cli command to work.

Executing terminal commands in Jupyter notebook

I am trying to run the following in Jupyter notebook (with Python 2 if it makes a difference):
!head xyz.txt
and I get the following error:
'head' is not recognized as an internal or external command, operable
program or batch file.
Is there anything I need to import to be able to do this?
Might be useful for others.
Use ! followed by terminal command you want to execute. To run a shell command. E.g.,
! pip install some_package
to install the some_package.
An easier way to invoke terminal using jupyter-notebooks is to use magic function %%bash and use the jupyter cell as a terminal:
%%bash
head xyz.txt
pip install keras
git add model.h5.dvc data.dvc metrics.json
git commit -m "Second model, trained with 2000 images"
For Windows it would be %%cmd.
Write it at the beginning of the cell like this :
%%cmd
where python
myprogram "blabla" -x -y -z
You can start the cell with the magic % bash before the rest of your code. There is an example in this blog post, together with a list of some of the most useful magics.
Make sure you run your command in linux shell because there is non such command in windows.
Another option nowadays is the Jupyter kernel for Bash.
I had the same issue. Solved by running
!bash -c "head xyz.txt"

Command line foward engineering using a .mwb file?

I need to be able to perform a forward engineering from a model located in a .mwb file. All of this from the command line as I would like to automate the process.
Can anyone please let me know if this is possible and if so how?
This is the output in the command line once you call WB with --help:
mysql-workbench [<options>] [<model file>]
Options:
--force-sw-render Force Xlib rendering
--force-opengl-render Force OpenGL rendering
--query <connection> Open a query tab to the named connection
--admin <instance> Open a administration tab to the named instance
--model <model file> Open the given EER model file
--script <script file> Execute the given Python or Lua script file
--run <script> Execute the given code in default language for GRT shell
--run-python <script> Execute the given code in Python
--run-lua <script> Execute the given code in Lua
--quit-when-done Quit Workbench when the script is done
--help, -h Show command line options and exit
--log-level=<level> Valid levels are: error, warning, info, debug1, debug2, debug3
--verbose Enable diagnostics output
--version Show Workbench version number and exit
I guess you can load your model using the --model option and then create an script that will perform the forward engineering and run it using the --run option and then instruct WB to exit once it finishes with the --quit-when-done option.
You can consult WB help to learn more about creating scripts as well as this guide.
You can actually automate this task with Python (or Lua) script - MySQL Workbench already has an interpreter under Scripting menu. Create a new script and use the stub:
# -*- coding: utf-8 -*-
import os
import grt
from grt.modules import DbMySQLFE
c = grt.root.wb.doc.physicalModels[0].catalog
DbMySQLFE.generateSQLCreateStatements(c, c.version, {
'GenerateDrops' : 1,
'GenerateSchemaDrops' : 1,
'OmitSchemata' : 1,
'GenerateUse' : 1
})
DbMySQLFE.generateSQLCreateStatements(c, c.version, {
DbMySQLFE.createScriptForCatalogObjects(os.path.dirname(grt.root.wb.docPath) + 'ddl.sql', c, {})
It does not actully run from command line, but I beleive you can run it with --run-script option.
This question is too old, but I found a project on github that make this, below the commands in cmd windows, and here the github repository and more linux sh file version.
Windows
#echo off
REM generate sql from mwb
REM usage: mwb2sql.bat {.mwb file} {output file}
SET WORKBENCH="C:\Program Files (x86)\MySQL\MySQL Workbench 6.0 CE\MySQLWorkbench.exe"
SET OUTPUT=%~f2
%WORKBENCH% ^
-open %~f1 ^
-run-python "import os;import grt;from grt.modules import DbMySQLFE as fe;c = grt.root.wb.doc.physicalModels[0].catalog;fe.generateSQLCreateStatements(c, c.version, {});fe.createScriptForCatalogObjects(os.getenv('OUTPUT'), c, {})" ^
-quit-when-done