Python Code
def main():
print("test")
if __name__ == "__main__":
# python my_test_test.py -t ./this_is_a_important_test_model.model.tfrecord -m 2 -o ./logs -r 1000
# python my_test_test.py -t ./this_is_a_important_test_model.model.tfrecord -m 1
main()
Command Line and Configuration
[pep8]
Command Line
$ autopep8 tmp.py
Your Environment
Python version: Python 3.6.13 :: Anaconda, Inc.
autopep8 version: autopep8 1.6.0 (pycodestyle: 2.8.0)
Platform: linux
Output
def main():
print("test")
if __name__ == "__main__":
# python my_test_test.py -t ./this_is_a_important_test_model.model.tfrecord -m 2 -o ./logs -r 1000
# python my_test_test.py -t
# ./this_is_a_important_test_model.model.tfrecord -m 1
main()
Question
The output of this code still do NOT satisfy the line limit.
How can I solve it?
Related
I am using Buildroot to build an initramfs image for my IMX Board. On the board, I run a custom Linux 4.19.35 Linux kernel (4.19.35-gexxxxxx) and a custom U-Boot bootloader. Hence, do not require these from Buildroot. My use case is only the rootfs.cpio (initramfs) image that gets built.
I am able to load the above initramfs into memory and execute my custom init and post-init scripts. However, I am unable to spawn an interactive shell. On reaching the command /bin/sh in the init script, I am greeted with a shell prompt but it seems that the serial console is not registering any keyboard inputs. Note that all other shell utilities and commands are executed just fine, but only when they are run in a script. Since one of my objectives is to have a minimal image, I am using busybox (1.32.0).
This gets even more confusing when I run the same initramfs along with the kernel image that is generated by buildroot. In this case, I do get an interactive shell prompt and I am able to enter my input like in a regular terminal.
I am suspecting that this might happen because of the different kernels. The buildroot kernel image is 4.19.35 but the kernel I use is 4.19.35-gexxxxx. However, I am not sure how the initramfs might be dependent on the kernel version string.
Any directions on what might be going wrong would be very helpful.
Edit 1: Below is my init code:
#!/bin/sh
/bin/mount -t devtmpfs devtmpfs /dev
export PATH=/sbin:/usr/sbin:/bin:/usr/bin
[ -d /dev ] || mkdir -m 0755 /dev
[ -d /root ] || mkdir -m 0700 /root
[ -d /sys ] || mkdir /sys
[ -d /proc ] || mkdir /proc
[ -d /tmp ] || mkdir /tmp
[ -d /run ] || mkdir /run
mkdir -p /dev/pts
mkdir -p /var/lock
/bin/mount -t sysfs -o nodev,noexec,nosuid sysfs /sys
/bin/mount -t proc -o nodev,noexec,nosuid proc /proc
/bin/mknod -m 666 /dev/ttyS0 c 4 64
/bin/mknod -m 666 /dev/ttyS0 c 4 64
/bin/mknod -m 622 /dev/console c 5 1
/bin/mknod -m 666 /dev/null c 1 3
/bin/mknod -m 666 /dev/tty c 5 0
/bin/mknod -m 666 /dev/zero c 1 5
/bin/mknod -m 666 /dev/ttymxc3 c 5 1
/bin/sh # --------------------> Spawning a shell
Try using a 5v serial adapter instead of 3.3v - with the lower voltage, you can still see what it's sending, but your adapter doesn't get heard by the device.
The three commands are :
Start the Virtual Environment by this command - source Detect_Outliers_In_Dataset/bin/activate
Change the directory by this command - cd Outlier_Detection
Start running Django Server by this command - python3 manage.py runserver
import subprocess
import pdb
p1 = subprocess.run("source Detect_Outliers_In_Dataset/bin/activate | cd Outlier_Detection | python3 manage.py runserver", capture_output=True, text=True, shell=True)
This is the error :
ompletedProcess(args='source Detect_Outliers_In_Dataset/bin/activate | cd Outlier_Detection | python3 manage.py runserver', returncode=2, stdout='', stderr="/bin/sh: 1: source: not found\n/bin/sh: 1: cd: can't cd to Outlier_Detection\npython3: can't open file 'manage.py': [Errno 2] No such file or directory\n")
I launch an EMR cluster with boto3 from a separate ec2 instance and use a bootstrapping script that looks like this:
#!/bin/bash
############################################################################
#For all nodes including master #########
############################################################################
wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /mnt1/anaconda3
export PATH=/mnt1/anaconda3/bin:$PATH
echo "export PATH="/mnt1/anaconda3/bin:$PATH"" >> ~/.bash_profile
sudo sed -i -e '$a\export PYSPARK_PYTHON=/mnt1/anaconda3/bin/python' /etc/spark/conf/spark-env.sh
echo "export PYSPARK_PYTHON="/mnt1/anaconda3/bin/python3"" >> ~/.bash_profile
conda install -c conda-forge -y shap
conda install -c conda-forge -y lightgbm
conda install -c anaconda -y numpy
conda install -c anaconda -y pandas
conda install -c conda-forge -y pyarrow
conda install -c anaconda -y boto3
############################################################################
#For master #########
############################################################################
if [ `grep 'isMaster' /mnt/var/lib/info/instance.json | awk -F ':' '{print $2}' | awk -F ',' '{print $1}'` = 'true' ]; then
sudo sed -i -e '$a\export PYSPARK_PYTHON=/mnt1/anaconda3/bin/python' /etc/spark/conf/spark-env.sh
echo "export PYSPARK_PYTHON="/mnt1/anaconda3/bin/python3"" >> ~/.bash_profile
sudo yum -y install git-core
conda install -c conda-forge -y jupyterlab
conda install -y jupyter
conda install -c conda-forge -y s3fs
conda install -c conda-forge -y nodejs
pip install spark-df-profiling
jupyter labextension install jupyterlab_filetree
jupyter labextension install #jupyterlab/toc
fi
Then I add a step programatically to the running cluster using add_job_flow_steps
action = conn.add_job_flow_steps(JobFlowId=curr_cluster_id, Steps=layer_function_steps)
The step is a spark-submit that is perfectly formed.
In one of the imported python files I import boto3. The error I get is
ImportError: No module named boto3
Clearly I am installing this library. If I SSH into the master node and run
python
import boto3
it works fine.Is there some kind of issue with spark-submit using the installed libraries since I am doing a conda install?
AWS has a project (AWS Data Wrangler) that helps with EMR launching.
This snippet should work to launch your cluster with Python 3 enabled:
import awswrangler as wr
cluster_id = wr.emr.create_cluster(
cluster_name="wrangler_cluster",
logging_s3_path=f"s3://BUCKET_NAME/emr-logs/",
emr_release="emr-5.28.0",
subnet_id="SUBNET_ID",
emr_ec2_role="EMR_EC2_DefaultRole",
emr_role="EMR_DefaultRole",
instance_type_master="m5.xlarge",
instance_type_core="m5.xlarge",
instance_type_task="m5.xlarge",
instance_ebs_size_master=50,
instance_ebs_size_core=50,
instance_ebs_size_task=50,
instance_num_on_demand_master=1,
instance_num_on_demand_core=1,
instance_num_on_demand_task=1,
instance_num_spot_master=0,
instance_num_spot_core=1,
instance_num_spot_task=1,
spot_bid_percentage_of_on_demand_master=100,
spot_bid_percentage_of_on_demand_core=100,
spot_bid_percentage_of_on_demand_task=100,
spot_provisioning_timeout_master=5,
spot_provisioning_timeout_core=5,
spot_provisioning_timeout_task=5,
spot_timeout_to_on_demand_master=True,
spot_timeout_to_on_demand_core=True,
spot_timeout_to_on_demand_task=True,
python3=True, # Relevant argument
spark_glue_catalog=True,
hive_glue_catalog=True,
presto_glue_catalog=True,
bootstraps_paths=["s3://BUCKET_NAME/bootstrap.sh"], # Relevant argument
debugging=True,
applications=["Hadoop", "Spark", "Ganglia", "Hive"],
visible_to_all_users=True,
key_pair_name=None,
spark_jars_path=[f"s3://...jar"],
maximize_resource_allocation=True,
keep_cluster_alive_when_no_steps=True,
termination_protected=False,
spark_pyarrow=True, # Relevant argument
tags={
"foo": "boo"
}
)
bootstrap.sh content:
#!/usr/bin/env bash
set -e
echo "Installing Python libraries..."
sudo pip-3.6 install -U awswrangler
sudo pip-3.6 install -U LIBRARY1
sudo pip-3.6 install -U LIBRARY2
...
I have a sed command to change something in a file of python:
sed -i 's|encoding = "ascii"|encoding = "utf-8"|g' /usr/local/lib/python2.7/site.py
so I want to replace "ascii" by "utf-8" but this does not work.
Someone who can explain why it isn't working and what do I need to change?
>> import sys
>> reload(sys)
>> sys.getdefaultencoding();
--> "ascii"
Run my command
>> import sys
>> reload(sys)
>> sys.getdefaultencoding();
--> "ascii"
Are you doing it after installing Python, aren't you? Could you post your dockerfile to try to reproduce your problem? Did you try to execute an bash shell to test if that file was changed as expected?
This is my working Dockerfile example:
FROM ubuntu:14.04
MAINTAINER github/ojgarciab
# Use the "noninteractive" debconf frontend
ENV DEBIAN_FRONTEND noninteractive
# Add python
RUN apt-get update && apt-get install -y python && \
apt-get clean
RUN sed -i 's|encoding = "ascii"|encoding = "utf-8"|g' /usr/lib/python2.7/site.py
ADD ejemplo.py /ejemplo.py
CMD [ "python", "/ejemplo.py" ]
Before using sed command the output was:
redstar#nvidiastar:~$ docker run -t python
ascii
And now the output is:
redstar#nvidiastar:~$ docker run -t python
utf-8
The content of ejemplo.py script:
import sys
reload(sys)
print sys.getdefaultencoding();
Note that the path used was /usr/lib/python2.7/site.py and not /usr/local/lib/python2.7/site.py.
Something strange and unexpected is happening with the sys.path of any virtual environment I set. For example, a clean env:
$ virtualenv test
$ source test/bin/activate
(test) $
This is the expected PYTHONPATH:
(test) $ python
>>> import sys
>>> print '\n'.join(sys.path)
/home/user/test/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7
/home/user/test/lib/python2.7/plat-linux2
/home/user/test/lib/python2.7/lib-tk
/home/user/test/lib/python2.7/lib-old
/home/user/test/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/home/user/test/local/lib/python2.7/site-packages
/home/user/test/lib/python2.7/site-packages
But this is the one I really get:
(test) $ bpython
>>> import sys
>>> print '\n'.join(sys.path)
/usr/bin
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
I can't figure out the reason of the two different sys.paths.
Because of that, no pip installation works!
I'm using Virtualenv 1.7.2, Ubuntu 12.04, Python 2.7.3.
Any help will be appreciated.
Rather than installing one copy of bpython per virtualenv, I've added this function to my shell profile (for example ~/.bashrc or ~/.zshrc). It wraps the bpython command with some logic to load the virtual environment's python path (if you have an active virtual environment).
bpython() {
if test -n "$VIRTUAL_ENV"
then
PYTHONPATH="$(python -c 'import sys; print ":".join(sys.path)')" \
command bpython "$#"
else
command bpython "$#"
fi
}
I found that I needed to deactivate and reactivate my virtualenv after installing bpython for it to work.
pip install bpython
deactivate
. bin/activate # or your equivalent activation command
My hypothesis is that you have not installed bpython after you have activated the new virtualenv.
I followed it up exactly like you mentioned:
mkvirtualenv bpython
(bpython)~ $ pip install bpython
(bpython)~ $bpython
and then ran the commands:
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/bin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
and did the same thing again by running python under the activated virtualenv
(bpython)~ $ python
.....
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
I saw no difference in the two results
I also discovered that if you have bpython installed locally, you need to create your virtualenv with --no-site-packages for it to work properly. If you created your virtualenv without that flag, you can create an empty file named no-global-site-packages.txt in ~/.virtualenvs/<env-name>/lib/python2.7/ as noted in this Stack Exchange answer.