Setting Up Windows System Variables with Python - operating-system

I am looking to set up two separate system variables through python, so that both gdal_calc and gdal_translate will work properly on my computer. However, it the pathway's that I have appended and the variables that I have added don't seem to be working properly. Any suggestions?
#!/usr/bin/env python
import subprocess
from subprocess import call
import sys
import os
# make dictionary of environmental variables to set up for gdal_calc and gdal_translate
gdal_env = os.environ.copy()
# modify and add variables for environment so that GDAL runs properly
gdal_env["GDAL_DATA"] = "C:\\Program Files (x86)\\GDAL\\gdal-data"
gdal_env["GDAL_DRIVER_PATH"] = "C:\\Program Files (x86)\\GDAL\\gdalplugins"
gdal_env["PATH"] = gdal_env["PATH"] + ";C:\\Program Files (x86)\\GDAL\\bin"
# Set constants
# The pathway to the images files are nested within the '--outfile=' command
inHVFile = os.path.expanduser('~\\Desktop\\Components\\Float32\\newHV32.img')
outPlacement = os.path.expanduser('~\\Desktop\\Components\\Zeros\\newHVZeros_1.img')
outVFile = '--outfile=' + outPlacement
#calc_cmd_HV = ['gdal_calc.py', '-A', inHVFile, outVFile, '--calc=A+1']
inVHFile = os.path.expanduser('~\\Desktop\\Components\\Float32\\newVH32.img')
outPlacement_1 = os.path.expanduser('~\\Desktop\\Components\\Zeros\\newVHZeros_1.img')
outVFile_1 = '--outfile=' + outPlacement_1
#calc_cmd_VH = ['gdal_calc.py', '-A', inVHFile, outVFile_1, '--calc=A+1']
subprocess.call([sys.executable,'C:\Program Files (x86)\GDAL\gdal_calc.py', inHVFile, outVFile, '--calc=A+1'], env=gdal_env)
subprocess.call([sys.executable,'C:\Program Files (x86)\GDAL\gdal_calc.py', inVHFile, outVFile_1, '--calc=A+1'], env=gdal_env)
#subprocess.call([sys.executable, 'C:\\Program Files (x86)\\GDAL\\gdal_calc.py','-A', inHVFile, outVFile, '--calc=A+1'])
#subprocess.call([sys.executable, 'C:\\Program Files (x86)\\GDAL\\gdal_calc.py','-A', inVHFile, outVFile_1, '--calc=A+1'])

Environmental variables hold information about where files and programs can be found. When using Python to call commandline programs via subprocess.call or subprocess.Popen, you can specify a set of environmental variables at the time the subprocess is spawned. This is done by passing a dictionary to the env kwarg of call or Popen. If env is not specified, the default environmental variables will be used.
Modifications to the environmental variables stored in os.environ will not persist after the Python session has ended.
To call GDAL programs via subprocess.call, do the following:
import os
import subprocess
import sys
# make dictionary of environmental variables
gdal_env = os.environ.copy()
# modify and add variables
gdal_env["GDAL_DATA"] = "C:\\Program Files (x86)\\GDAL\gdal-data"
gdal_env["GDAL_DRIVER_PATH"] = "C:\\Program Files (x86)\\GDAL\\gdalplugins"
gdal_env["PATH"] = gdal_env["PATH"] + ";C:\\Program Files (x86)\\GDAL\\bin"
# ... do preparation ...
a = "a.tif"
b = "b.tif"
output = "output.tif"
calc_cmd = [sys.executable, 'gdal_calc.py', '-A', a, '-B', b, '--outfile='+output, '--calc=A+B']
# spawn a new subprocess
subprocess.call(calc_cmd, env=gdal_env)

Related

Subprocess no output in windows

I need to run an exe file with different parameters and write the output to a file. I tried the following script. After the program is running, the output file is empty. Tell me how to solve this problem
import subprocess
output_f = open('output.txt', 'a')
for i in "abcdefghijklmnopqrstuvwxyz{}_":
for j in "abcdefghijklmnopqrstuvwxyz{}_":
program = 'C:/Users/PC/Desktop/task2 (1).exe ' + i +" " +j;
process = subprocess.Popen(program,stdout=output_f)
code = process.wait()
output_f.close()
print(code)

How to efficiently rename many files or partially select the names of these files during import?

how to rename the files efficiently by the number in the name (see picture)? I did not succeed with Windows PowerToys and I dont wana click each file and rename to the number (e.g. 290)
or how to read the files in this order and define a name? If I try it with a script (see below) the following output occurs:
[![ValueError: invalid literal for int() with base 10: '211001_164357_P_Scripted_Powermeasurement_Wavelength_automatic_Powermeter1_0'][1]][1]
or how to select only the numbers (290 to 230 - see picture) within the name when reading?
Script:
#import libraries
import pandas as pd
import os
import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path
data_location = r'C:\Users\...\Characterization_OPO\Data'
data_folder = Path(data_location)
data = {}
allist = list(data_folder.glob('*'))
for i, file in enumerate(allist):
file = str(file)
file_name = file.split('\\')[-1]
wavelength = int(file_name.split('.')[0])
tmp = pd.read_csv(file, skiprows=20, skipfooter=59, index_col="PixelNo")
data[f'{wavelength} nm'] = tmp;
#data.plot(x='Wavelength',y='CCD_1', label=f"{wavelength} nm")
Picture:
I removed all words with windows power rename and than took the last three digits:
for i, file in enumerate(allist):
file = str(file)
file_name = file.split('\\')[-1]
wavelength = int(file_name.split('.')[0])
tmp = pd.read_csv(file, skiprows=26, skipfooter=5)
data[f'{wavelength % 1000} nm'] = tmp;
#data.plot(x='Wavelength',y='CCD_1', label=f"{wavelength} nm")

reading a s3 file with s3a format using pyspark - High

#
# Some constants
#
aws_profile = "your_profile"
aws_region = "your_region"
s3_bucket = "your_bucket"
#
# Reading environment variables from aws credential file
#
import os
import configparser
config = configparser.ConfigParser()
config.read(os.path.expanduser("~/.aws/credentials"))
access_id = config.get(aws_profile, "aws_access_key_id")
access_key = config.get(aws_profile, "aws_secret_access_key")
#
# Configuring pyspark
#
# see https://github.com/jupyter/docker-stacks/issues/127#issuecomment-214594895
# and https://github.com/radanalyticsio/pyspark-s3-notebook/blob/master/s3-source-example.ipynb
os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages=org.apache.hadoop:hadoop-aws:2.7.3 pyspark-shell"
# If this doesn't work you might have to delete your ~/.ivy2 directory to reset your package cache.
# (see https://github.com/databricks/spark-redshift/issues/244#issuecomment-239950148)
import pyspark
sc=pyspark.SparkContext()
# see https://github.com/databricks/spark-redshift/issues/298#issuecomment-271834485
sc.setSystemProperty("com.amazonaws.services.s3.enableV4", "true")
# see https://stackoverflow.com/questions/28844631/how-to-set-hadoop-configuration-values-from-pyspark
hadoop_conf=sc._jsc.hadoopConfiguration()
# see https://stackoverflow.com/questions/43454117/how-do-you-use-s3a-with-spark-2-1-0-on-aws-us-east-2
hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
hadoop_conf.set("com.amazonaws.services.s3.enableV4", "true")
hadoop_conf.set("fs.s3a.access.key", access_id)
hadoop_conf.set("fs.s3a.secret.key", access_key)
# see http://blog.encomiabile.it/2015/10/29/apache-spark-amazon-s3-and-apache-mesos/
hadoop_conf.set("fs.s3a.connection.maximum", "100000")
# see https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
hadoop_conf.set("fs.s3a.endpoint", "s3." + aws_region + ".amazonaws.com")
#
# Downloading the parquet file
#
sql=pyspark.sql.SparkSession(sc)
path = s3_bucket + "your_path"
dataS3=sql.read.parquet("s3://" + path)
Even have tried to write a file thinking that my directory pointing was not correct and if the file write is successful, could pin point the path where it is pointing now but still no progress and say no path exists.
If you please could guide us in this regard, it would really be helpful. Thanks in advance.

Is possible to add a python path with pipenv?

I have some old python projects that are NOT packaged (no setup.py) I need to access them.
Till now I was using buildout where I can specify a python path in buildout.cfg.
I would like to switch to pipenv : how can I specify an arbitrary python path with pipenv so I can import my unpackaged projects ?
I finally wrote a script that creates a .pth file with all the pythonpath I wanted to add in the virtualenv:
#!/usr/bin/env python
from distutils import sysconfig
import os
site_packages_path = sysconfig.get_python_lib()
PTH_FILENAME = 'MyApp.pth'
# Change here to your project home dir
PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
relative_paths = [
'.',
'plugins',
'src/achilterm',
'src/addicted',
'src/naghelp',
'src/textops',
'src',
]
absolute_paths = [ os.path.join(PROJECT_DIR,p) for p in relative_paths ]
with open(os.path.join(site_packages_path,PTH_FILENAME),'w') as fh:
for p in absolute_paths:
print 'Installing path : %s ...' % p
fh.write('%s\n' % p)

How to use numba in Colaboratory

Anybody tried to use numba in google collaboratory? I just can not figure out how to set it up in this environment.
At the moment, I'm stuck with the error library nvvm not found.
Copy this code into cell. It works for me.
!apt-get install nvidia-cuda-toolkit
!pip3 install numba
import os
os.environ['NUMBAPRO_LIBDEVICE'] = "/usr/lib/nvidia-cuda-toolkit/libdevice"
os.environ['NUMBAPRO_NVVM'] = "/usr/lib/x86_64-linux-gnu/libnvvm.so"
from numba import cuda
import numpy as np
import time
#cuda.jit
def hello(data):
data[cuda.blockIdx.x, cuda.threadIdx.x] = cuda.blockIdx.x
numBlocks = 5
threadsPerBlock = 10
data = np.ones((numBlocks, threadsPerBlock), dtype=np.uint8)
hello[numBlocks, threadsPerBlock](data)
print(data)
I didn't have to install the packages #Algis suggested, but the paths to the drivers were different. So I had to do the following.
First determine the correct paths for the drivers
!find / -iname 'libdevice'
!find / -iname 'libnvvm.so'
# Output:
# /usr/local/cuda-9.2/nvvm/lib64/libnvvm.so
# /usr/local/cuda-9.2/nvvm/libdevice
Then set the paths as #Algis described
import os
os.environ['NUMBAPRO_LIBDEVICE'] = "/usr/local/cuda-9.2/nvvm/libdevice"
os.environ['NUMBAPRO_NVVM'] = "/usr/local/cuda-9.2/nvvm/lib64/libnvvm.so"
You can do #Stan's work in one simple sweep if you have this block at the beginning of your colab notebook (this also automatically updates as CUDA gets updated)
import os
dev_lib_path = !find / -iname 'libdevice'
nvvm_lib_path = !find / -iname 'libnvvm.so'
assert len(dev_lib_path)>0, "Device Lib Missing"
assert len(nvvm_lib_path)>0, "NVVM Missing"
os.environ['NUMBAPRO_LIBDEVICE'] = dev_lib_path[0]
os.environ['NUMBAPRO_NVVM'] = nvvm_lib_path[0]