use double quotes inside sed - sed

I have a sed command to change something in a file of python:
sed -i 's|encoding = "ascii"|encoding = "utf-8"|g' /usr/local/lib/python2.7/site.py
so I want to replace "ascii" by "utf-8" but this does not work.
Someone who can explain why it isn't working and what do I need to change?
>> import sys
>> reload(sys)
>> sys.getdefaultencoding();
--> "ascii"
Run my command
>> import sys
>> reload(sys)
>> sys.getdefaultencoding();
--> "ascii"

Are you doing it after installing Python, aren't you? Could you post your dockerfile to try to reproduce your problem? Did you try to execute an bash shell to test if that file was changed as expected?
This is my working Dockerfile example:
FROM ubuntu:14.04
MAINTAINER github/ojgarciab
# Use the "noninteractive" debconf frontend
ENV DEBIAN_FRONTEND noninteractive
# Add python
RUN apt-get update && apt-get install -y python && \
apt-get clean
RUN sed -i 's|encoding = "ascii"|encoding = "utf-8"|g' /usr/lib/python2.7/site.py
ADD ejemplo.py /ejemplo.py
CMD [ "python", "/ejemplo.py" ]
Before using sed command the output was:
redstar#nvidiastar:~$ docker run -t python
ascii
And now the output is:
redstar#nvidiastar:~$ docker run -t python
utf-8
The content of ejemplo.py script:
import sys
reload(sys)
print sys.getdefaultencoding();
Note that the path used was /usr/lib/python2.7/site.py and not /usr/local/lib/python2.7/site.py.

Related

Which profile does sh loads

I am trying to load a specific tool ( nvm ) from within sh.
Installing it as explained in the page for bash it works perfectly and testing it returns the following.
$ bash
$ nvm --version
+ X.XX.X
but if I type
$ sh
$ nvm --version
+ sh: 1: nvm: not found
but still its expected as the default installation modifies the .bashrc.
now i have included the same .bashrc code in my /etc/profile
export NVM_DIR="/opt/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
and tried again.
$ sh
$ nvm --version
+ sh: 1: nvm: not found
$ echo $NVM_DIR
+ /dir/to/nvm
$ [ -s "$NVM_DIR/nvm.sh" ] && echo "it works?"
+ it works?
$ [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
$ nvm --version
X.XX.X
which got me extremely confused. What exactly is happening ? Isn't sh loading the /etc/profile or am I doing something really wrong?
--edit after comments--
also tried to include it in the local profile
$ cat ~/.profile
+ export NVM_DIR="/opt/nvm"
+ [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
$ sh
$ nvm --version
+ sh: 1: nvm: not found
The problem was described in this article with the sh blue line.
The solution to the problem was to set the /etc/profile/ in the ENV variable.
for example ENV=/etc/profile sh would load the profile when the sh opens. That solved the problem

Spark-submit AWS EMR with anaconda installed python libraries

I launch an EMR cluster with boto3 from a separate ec2 instance and use a bootstrapping script that looks like this:
#!/bin/bash
############################################################################
#For all nodes including master #########
############################################################################
wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /mnt1/anaconda3
export PATH=/mnt1/anaconda3/bin:$PATH
echo "export PATH="/mnt1/anaconda3/bin:$PATH"" >> ~/.bash_profile
sudo sed -i -e '$a\export PYSPARK_PYTHON=/mnt1/anaconda3/bin/python' /etc/spark/conf/spark-env.sh
echo "export PYSPARK_PYTHON="/mnt1/anaconda3/bin/python3"" >> ~/.bash_profile
conda install -c conda-forge -y shap
conda install -c conda-forge -y lightgbm
conda install -c anaconda -y numpy
conda install -c anaconda -y pandas
conda install -c conda-forge -y pyarrow
conda install -c anaconda -y boto3
############################################################################
#For master #########
############################################################################
if [ `grep 'isMaster' /mnt/var/lib/info/instance.json | awk -F ':' '{print $2}' | awk -F ',' '{print $1}'` = 'true' ]; then
sudo sed -i -e '$a\export PYSPARK_PYTHON=/mnt1/anaconda3/bin/python' /etc/spark/conf/spark-env.sh
echo "export PYSPARK_PYTHON="/mnt1/anaconda3/bin/python3"" >> ~/.bash_profile
sudo yum -y install git-core
conda install -c conda-forge -y jupyterlab
conda install -y jupyter
conda install -c conda-forge -y s3fs
conda install -c conda-forge -y nodejs
pip install spark-df-profiling
jupyter labextension install jupyterlab_filetree
jupyter labextension install #jupyterlab/toc
fi
Then I add a step programatically to the running cluster using add_job_flow_steps
action = conn.add_job_flow_steps(JobFlowId=curr_cluster_id, Steps=layer_function_steps)
The step is a spark-submit that is perfectly formed.
In one of the imported python files I import boto3. The error I get is
ImportError: No module named boto3
Clearly I am installing this library. If I SSH into the master node and run
python
import boto3
it works fine.Is there some kind of issue with spark-submit using the installed libraries since I am doing a conda install?
AWS has a project (AWS Data Wrangler) that helps with EMR launching.
This snippet should work to launch your cluster with Python 3 enabled:
import awswrangler as wr
cluster_id = wr.emr.create_cluster(
cluster_name="wrangler_cluster",
logging_s3_path=f"s3://BUCKET_NAME/emr-logs/",
emr_release="emr-5.28.0",
subnet_id="SUBNET_ID",
emr_ec2_role="EMR_EC2_DefaultRole",
emr_role="EMR_DefaultRole",
instance_type_master="m5.xlarge",
instance_type_core="m5.xlarge",
instance_type_task="m5.xlarge",
instance_ebs_size_master=50,
instance_ebs_size_core=50,
instance_ebs_size_task=50,
instance_num_on_demand_master=1,
instance_num_on_demand_core=1,
instance_num_on_demand_task=1,
instance_num_spot_master=0,
instance_num_spot_core=1,
instance_num_spot_task=1,
spot_bid_percentage_of_on_demand_master=100,
spot_bid_percentage_of_on_demand_core=100,
spot_bid_percentage_of_on_demand_task=100,
spot_provisioning_timeout_master=5,
spot_provisioning_timeout_core=5,
spot_provisioning_timeout_task=5,
spot_timeout_to_on_demand_master=True,
spot_timeout_to_on_demand_core=True,
spot_timeout_to_on_demand_task=True,
python3=True, # Relevant argument
spark_glue_catalog=True,
hive_glue_catalog=True,
presto_glue_catalog=True,
bootstraps_paths=["s3://BUCKET_NAME/bootstrap.sh"], # Relevant argument
debugging=True,
applications=["Hadoop", "Spark", "Ganglia", "Hive"],
visible_to_all_users=True,
key_pair_name=None,
spark_jars_path=[f"s3://...jar"],
maximize_resource_allocation=True,
keep_cluster_alive_when_no_steps=True,
termination_protected=False,
spark_pyarrow=True, # Relevant argument
tags={
"foo": "boo"
}
)
bootstrap.sh content:
#!/usr/bin/env bash
set -e
echo "Installing Python libraries..."
sudo pip-3.6 install -U awswrangler
sudo pip-3.6 install -U LIBRARY1
sudo pip-3.6 install -U LIBRARY2
...

Running last command with sudo in fish only works if it has no arguments?

I'm trying to write a function that does the equivalent of sudo !! in Bash. It works, but only when the last command has no arguments.
So far the function is:
function s --description "Run last command (or specified command) using sudo"
if test $argv
switch $argv[1]
case '!!'
command sudo (echo $history[1])
case '*'
command sudo $argv
end
else
command sudo fish
end
end
Testing the relevant line:
$ command sudo whoami
root
$ whoami
nick
$ command sudo (echo $history[1])
root
So far so good, now lets try a command with a few args:
$ echo hi >> /etc/motd
An error occurred while redirecting file '/etc/motd'
open: Permission denied
$ command sudo (echo $history[1])
sudo: echo hi >> /etc/motd: command not found
Hmm, strange.
Got it working using eval.
function sudo --description 'Run command using sudo (use !! for last command)'
if test (count $argv) -gt 0
switch $argv[1]
case '!!'
if test (count $argv) -gt 1
set cmd "command sudo $history[1] $argv[2..-1]"
else
set cmd "command sudo $history[1]"
end
case '*'
set cmd "command sudo $argv"
end
else
set cmd "command sudo fish"
end
eval $cmd
end
I had the same problem as you, and I fixed it by using oh-my-fish
(it's a plugin manager for fish shell) https://github.com/oh-my-fish/oh-my-fish. You can install it with this command :
curl -L https://get.oh-my.fish | fish
Then install the plugin bang-bang (to allow !! and !$) with this command :
omf install bang-bang

How modify /etc/network/interfaces without gedit?

I'm doing a script and I would like to add the following line
pre-up iptables-restore < /etc/iptables.rules
to the file interfaces which is located on /etc/network/interfaces,but although I have enabled the permissions to write in this file(I work in Ubuntu), I'm not able to do it... I'm trying to use the following command in my bash script
sudo echo "pre-up iptables-restore < /etc/iptables.rules" >> /etc/network/interfaces
Any suggestion of how to do it without using gedit o vi?
Thanks in advance!
You need to tell bash not to use redirection before it starts sudo:
sudo bash -c 'echo "pre-up iptables-restore < /etc/iptables.rules" >> /etc/network/interfaces'
this way the complete command will be executed with root access, not only the echo "pre-up iptables-restore < /etc/iptables.rules" part

python and bpython using different PYTHONPATHs in Virtualenv

Something strange and unexpected is happening with the sys.path of any virtual environment I set. For example, a clean env:
$ virtualenv test
$ source test/bin/activate
(test) $
This is the expected PYTHONPATH:
(test) $ python
>>> import sys
>>> print '\n'.join(sys.path)
/home/user/test/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7
/home/user/test/lib/python2.7/plat-linux2
/home/user/test/lib/python2.7/lib-tk
/home/user/test/lib/python2.7/lib-old
/home/user/test/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/home/user/test/local/lib/python2.7/site-packages
/home/user/test/lib/python2.7/site-packages
But this is the one I really get:
(test) $ bpython
>>> import sys
>>> print '\n'.join(sys.path)
/usr/bin
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
I can't figure out the reason of the two different sys.paths.
Because of that, no pip installation works!
I'm using Virtualenv 1.7.2, Ubuntu 12.04, Python 2.7.3.
Any help will be appreciated.
Rather than installing one copy of bpython per virtualenv, I've added this function to my shell profile (for example ~/.bashrc or ~/.zshrc). It wraps the bpython command with some logic to load the virtual environment's python path (if you have an active virtual environment).
bpython() {
if test -n "$VIRTUAL_ENV"
then
PYTHONPATH="$(python -c 'import sys; print ":".join(sys.path)')" \
command bpython "$#"
else
command bpython "$#"
fi
}
I found that I needed to deactivate and reactivate my virtualenv after installing bpython for it to work.
pip install bpython
deactivate
. bin/activate # or your equivalent activation command
My hypothesis is that you have not installed bpython after you have activated the new virtualenv.
I followed it up exactly like you mentioned:
mkvirtualenv bpython
(bpython)~ $ pip install bpython
(bpython)~ $bpython
and then ran the commands:
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/bin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
and did the same thing again by running python under the activated virtualenv
(bpython)~ $ python
.....
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
I saw no difference in the two results
I also discovered that if you have bpython installed locally, you need to create your virtualenv with --no-site-packages for it to work properly. If you created your virtualenv without that flag, you can create an empty file named no-global-site-packages.txt in ~/.virtualenvs/<env-name>/lib/python2.7/ as noted in this Stack Exchange answer.