cloud-init: Can write_files be merged? - cloud-init

Can write_files be merged? I can't seem to get the merge_how syntax correct. Only files in the last write_files are being created.
Thanks

The answer is yes.
When working with multi-part MIME, you must add the following to the header of each part.
Merge-Type: list(append)+dict(no_replace,recurse_list)+str()
Below is a modified version of the helper script provided in the cloud-init documentation.
#!/usr/bin/env python3
import click
import sys
import gzip as gz
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
#click.command()
#click.option('--gzip', is_flag=True,
help='gzip MIME message to current directory')
#click.argument('message', nargs=-1)
def main(message: str, gzip: bool) -> None:
"""This script creates a multi part MIME suitable for cloud-init
A MESSAGE has following format <filename>:<type> where <filename> is the
file that contains the data to add to the MIME message. <type> is the
MIME content-type.
MIME content-type may be on of the following:
text/cloud-boothook
text/cloud-config
text/cloud-config-archive
text/jinja2
text/part-handler
text/upstart-job
text/x-include-once-url
text/x-include-url
text/x-shellscript
EXAMPLE:
write-mime-multipart afile.cfg:cloud-config bfile.sh:x-shellscript
"""
combined_message = MIMEMultipart()
for message_part in message:
(filename, format_type) = message_part.split(':', 1)
with open(filename) as fh:
contents = fh.read()
sub_message = MIMEText(contents, format_type, sys.getdefaultencoding())
sub_message.add_header('Content-Disposition',
'attachment; filename="{}"'.format(filename))
sub_message.add_header('Merge-Type',
'list(append)+dict(no_replace,recurse_list)+str()')
combined_message.attach(sub_message)
if gzip:
with gz.open('combined-userdata.txt.gz', 'wb') as fd:
fd.write(bytes(combined_message))
else:
print(combined_message)
if __name__ == '__main__':
main()

Related

Libre office configuration file bootstrap.ini corrupt

I am using the libreoffice-convert package to convert a Word file into PDF. But when I try to convert I get this error. How do I fix this? I have installed LibreOffice 7.0.5
Console log is showing this error:
'C:\\Program Files\\LibreOffice\\program\\soffice.exe -env:UserInstallation=file://C:\\Users\\thesa\\AppData\\Local\\Temp\\soffice-10416-bQefydwUfs2F --headless --convert-to .pdf --outdir C:\\Users\\thesa\\AppData\\Local\\Temp\\libreofficeConvert_-10416-v46bO7ljGHRe C:\\Users\\thesa\\AppData\\Local\\Temp\\libreofficeConvert_-10416-v46bO7ljGHRe\\source'
Try this, you will probably need administrator rights.
Locate and copy your orignal file, for example to "bootstrap.ini.org", in the same directory1.
Open the file and replace its contents with this, which is a copy of my file:
[Bootstrap]
InstallMode=<installmode>
ProductKey=LibreOffice 7.0
UserInstallation=$SYSUSERCONFIG/LibreOffice/4
Another option is to re-install or repair the installation.
Note 1: This copy is just in case you need to revert.
//let data = await fs.promises.readFile(path_to_excel_file);
let data = fs.readFileSync(path_to_excel_file)
let pdfFile = await libreConvert(data, '.pdf', undefined);
await fs.promises.writeFile(`${__dirname}/${docName}.pdf`, pdfFile);
res.download(`${__dirname}/${docName}.pdf`)
It will works File.
Even If don't work then follow
below step:
Add this line 14 after instalDir
const installDir = tmp.dirSync({prefix: 'soffice', unsafeCleanup: true, ...tmpOptions});
const posixInstallDir = installDir.name.split(path.sep).join(path.posix.sep);
then Replace commant
let command = `${results.soffice} --headless --convert-to ${format}`;
That's it..
-env:UserInstallation needs to be in URI form.
Note: also that you using file:// this should be file:/// see File URI scheme
-env:UserInstallation=file:///C:/Users/thesa/AppData/Local/Temp/soffice-10416-bQefydwUfs2F
>>> from pathlib import Path
>>> def get_posix(mypath) -> str:
>>> p = Path(mypath)
>>> return p.as_posix()
>>> tmp = "C:\\Users\\thesa\\AppData\\Local\\Temp\\soffice-10416-bQefydwUfs2F"
>>> get_posix(tmp)
'C:/Users/thesa/AppData/Local/Temp/soffice-10416-bQefydwUfs2F'
or
>>> from pathlib import Path
>>> tmp = Path("C:\\Users\\thesa\\AppData\\Local\\Temp\\soffice-10416-bQefydwUfs2F")
>>> tmp.as_uri()
'file:///C:/Users/thesa/AppData/Local/Temp/soffice-10416-bQefydwUfs2F'

Webscraper not giving the right results with bs4

I'm trying to scrape the live billionaire networth table here > https://www.bloomberg.com/billionaires/
This is my code so far. All I get is [] as result on the python shell.
Something has to be wrong with the "findAll", I don't think I'm using the correct tag lines.
Tried to use just "find"
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import csv
#Open page and grab html
my_url = ('https://www.bloomberg.com/billionaires/')
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
#HTML Parser.
page_soup = soup(page_html, 'html.parser')
table = []
#Find table.
ele_table = page_soup.findAll('div',{'class':'dvz-content'})
print(ele_table)
I'm expecting for the table to be printed out so I can get it into a CSV file.
Data is dynamically loaded. You can pull from script tag provided you supply the right headers. Regex out the required info and parse with json library. Hand this off to pandas to write to csv
from bs4 import BeautifulSoup as bs
import requests, re, json
import pandas as pd
headers = {
'user-agent': 'Mozilla/5.0',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'if-none-match': 'W/^\\^5dbb59e6-91b10^\\^',
'if-modified-since': 'Thu, 31 Oct 2019 22:02:14 GMT' # this may be safeguard for caching. Consider if add dynamically.
}
p = re.compile(r'window.top500 = (.*);')
r = requests.get('https://www.bloomberg.com/billionaires/', headers = headers)
data = json.loads(p.findall(r.text)[0])
df = pd.DataFrame(data)
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False)
Example output:

An error in my code to be a simple ftp

I met an error when running codes at the bottom. It's like a simple ftp.
I use python2.6.6 and CentOS release 6.8
In most linux server, it gets right results like this:(I'm very sorry that I have just sign up and couldn't )
Clinet:
[root#Test ftp]# python client.py
path:put|/home/aaa.txt
Server:
[root#Test ftp]# python server.py
connected...
pre_data:put|aaa.txt|4
cmd: put
file_name: aaa.txt
file_size: 4
upload successed.
But I get errors in some server(such as my own VM in my PC). I have done lots of tests(python2.6/python2.7, Centos6.5/Centos6.7) and found this error is not because them. Here is the error imformation:
[root#Lewis-VM ftp]# python server.py
connected...
pre_data:put|aaa.txt|7sdfsdf ###Here gets the wrong result, "sdfsdf" is the content of /home/aaa.txt and it shouldn't be sent here to 'file_size' and so it cause the "ValueError" below
cmd: put
file_name: aaa.txt
file_size: 7sdfsdf
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 10699)
Traceback (most recent call last):
File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__
self.handle()
File "server.py", line 30, in handle
if int(file_size)>recv_size:
ValueError: invalid literal for int() with base 10: '7sdfsdf\n'
What's more, I found that if I insert a time.sleep(1) between sk.send(cmd+"|"+file_name+'|'+str(file_size)) and sk.send(data) in client.py, the error will disappear. I have said that I did tests in different system and python versions and the error is not because them. So I guess that is it because of some system configs? I have check about socket.send() and socket.recv() in python.org but fount nothing helpful. So could somebody help me to explain why this happend?
The code are here:
#!/usr/bin/env python
#coding:utf-8
################
#This is server#
################
import SocketServer
import os
class MyServer(SocketServer.BaseRequestHandler):
def handle(self):
base_path = '/home/ftp/file'
conn = self.request
print 'connected...'
while True:
#####receive pre_data: we should get data like 'put|/home/aaa|7'
pre_data = conn.recv(1024)
print 'pre_data:' + pre_data
cmd,file_name,file_size = pre_data.split('|')
print 'cmd: ' + cmd
print 'file_name: '+ file_name
print 'file_size: '+ file_size
recv_size = 0
file_dir = os.path.join(base_path,file_name)
f = file(file_dir,'wb')
Flag = True
####receive 1024bytes each time
while Flag:
if int(file_size)>recv_size:
data = conn.recv(1024)
recv_size+=len(data)
else:
recv_size = 0
Flag = False
continue
f.write(data)
print 'upload successed.'
f.close()
instance = SocketServer.ThreadingTCPServer(('127.0.0.1',9999),MyServer)
instance.serve_forever()
#!/usr/bin/env python
#coding:utf-8
################
#This is client#
################
import socket
import sys
import os
ip_port = ('127.0.0.1',9999)
sk = socket.socket()
sk.connect(ip_port)
while True:
input = raw_input('path:')
#####we should input like 'put|/home/aaa.txt'
cmd,path = input.split('|')
file_name = os.path.basename(path)
file_size=os.stat(path).st_size
sk.send(cmd+"|"+file_name+'|'+str(file_size))
send_size = 0
f= file(path,'rb')
Flag = True
#####read 1024 bytes and send it to server each time
while Flag:
if send_size + 1024 >file_size:
data = f.read(file_size-send_size)
Flag = False
else:
data = f.read(1024)
send_size+=1024
sk.send(data)
f.close()
sk.close()
The TCP is a stream of data. That is the problem. TCP do not need to keep message boundaries. So when a client calls something like
connection.send("0123456789")
connection.send("ABCDEFGHIJ")
then a naive server like
while True;
data = conn.recv(1024)
print data + "_"
may print any of:
0123456789_ABCDEFGHIJ_
0123456789ABCDEFGHIJ_
0_1_2_3_4_5_6_7_8_9_A_B_C_D_E_F_G_H_I_J_
The server has no chance to recognize how many sends client called because the TCP stack at client side just inserted data to a stream and the server must be able to process the data received in different number of buffers than the client used.
Your server must contain a logic to separate the header and the data. All of application protocols based on TCP use a mechanism to identify application level boundaries. For example HTTP separates headers and body by an empty line and it informs about the body length in a separate header.
Your program works correctly when server receives a header with the command, name and size in a separate buffer it it fails when client is fast enough and push the data into stream quickly and the server reads header and data in one chunk.

environment variables using subprocess.check_output Python

I'm trying to do some basic module setups on my server using Python. Its a bit difficult as I have no access to the internet.
This is my code
import sys
import os
from subprocess import CalledProcessError, STDOUT, check_output
def run_in_path(command, dir_path, env_var=''):
env_var = os.environ["PATH"] = os.environ["PATH"] + env_var
print(env_var)
try:
p = check_output(command, cwd=dir_path, stderr=STDOUT)
except CalledProcessError as e:
sys.stderr.write(e.output.decode("utf-8"))
sys.stderr.flush()
return e.returncode
else:
return 0
def main():
requests_install = run_in_path('python setup.py build',
'D:\installed_software\python modules\kennethreitz-requests-e95e173')
SQL_install = run_in_path('python setup.py install', # install SQL module pypyodbc
'D:\installed_software\python modules\pypyodbc-1.3.3\pypyodbc-1.3.3')
setup_tools = run_in_path('python setup.py install', # install setup tools
'D:\installed_software\python modules\setuptools-17.1.1')
psycopg2 = run_in_path('easy_install psycopg2-2.6.1.win-amd64-py3.3-pg9.4.4-release', # install setup tools
'D:\installed_software\python modules', ';C:\srv_apps\Python33\Scripts\easy_install.exe')
print('setup complete')
if __name__ == "__main__":
sys.exit(main())
now it gets tricky when i start trying to use easy install. It appears my env variables are not being used by my subprocess.check_output call
File "C:\srv_apps\Python33\lib\subprocess.py", line 1110, in _execute_child
raise WindowsError(*e.args)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I don't want to have to upgrade to 3.4 where easy install is installed by default because my other modules are not supported on 3.4. My main challenge is the subprocess.check_call method does not take environment variables as an input and im wary of trying to use Popen() as I have never really got it to work successfully in the past. Any help would be greatly appreciated.
PATH should contain directories e.g., r'C:\Python33\Scripts', not files such as: r'C:\Python33\Scripts\easy_install.exe'
Don't hardcode utf-8 for an arbitrary command, you could enable text mode using universal_newlines parameter (not tested):
#!/usr/bin/env python3
import locale
import sys
from subprocess import CalledProcessError, STDOUT, check_output
def run(command, *, cwd=None, env=None):
try:
ignored = check_output(command, cwd=cwd, env=env,
stderr=STDOUT,
universal_newlines=True)
except CalledProcessError as e:
sys.stderr.write(e.output)
sys.stderr.flush()
return e.returncode
else:
return 0
Example:
import os
path_var = os.pathsep.join(os.environ.get('PATH', os.defpath), some_dir)
env = dict(os.environ, PATH=path_var)
run("some_command", cwd=some_path, env=env)
run("another_command", cwd=another_path, env=env)

web automation - auto check link

I'm new to web app and I want to check when there's a new version of dota map, I'll check links in getdota.com.
How can I do this and which language, I want it checks every time you start warcraft, and auto download new map to specific folder.
My question is : Can you give a link to a specific article about web automation or something like that.
Thanks first :)
Below is an example in Python.
It parses getdota.com page, reads parameters for POST request for downloading a map, gets the file and saves it in configured directory (by default current directory).
#!/usr/bin/env python
import urllib
import urllib2
import sgmllib
from pprint import pprint
import os.path
import sys
url = 'http://www.getdota.com/'
download_url = 'http://www.getdota.com/app/getmap/'
chunk = 10000
directory = '' #directory where file should be saved, if empty uses current dir
class DotaParser(sgmllib.SGMLParser):
def parse(self, s):
self.feed(s)
self.close()
def __init__(self, verbose=0):
sgmllib.SGMLParser.__init__(self, verbose)
self.URL = ''
self.post_args = {}
def getArgs(self):
return self.post_args
def start_input(self, attributes):
d = dict(attributes)
if d.get('id', None) == None:
return
if d['id'] in ["input_mirror2", "input_file_name2", "input_map_id2", "input_language2", "input_language_id2"]:
self.post_args[d['name']] = d['value']
if __name__ == '__main__':
dotap = DotaParser()
data = urllib2.urlopen(urllib2.Request('http://www.getdota.com/')).read()
dotap.parse(data)
data = urllib.urlencode(dotap.getArgs())
request = urllib2.Request(download_url, data)
response = urllib2.urlopen(request)
page = response.read()
#download file
fname = directory + page.split('/')[-1]
if os.path.isfile(fname):
print "No newer file available"
sys.exit(0)
f = open(fname, 'w')
print "New file available. Saving in: %s" % fname
webFile = urllib.urlopen(page)
c = webFile.read(chunk)
while(c):
f.write(c)
c = webFile.read(chunk)
f.close()
webFile.close()