I Am using Salesforce Bulk Package to login and query data(bulk)
job = bulk.create_query_job("User", contentType='CSV')
batch = bulk.query(job, "SELECT id, name FROM User limit 2")
bulk.close_job(job)
while not bulk.is_batch_done(batch):
sleep(10)
for result in bulk.get_all_results_for_query_batch(batch):
reader = csv.DictReader(result, encoding='utf-8')
for row in reader:
print(row)
The output for the following is
OrderedDict([('Id', 'XXXXXXXXXXXXXX'), ('Name', Test1')])
OrderedDict([('Id', 'XXXXXXXXXXXXXX'), ('Name', 'Walter Paul')])
How to print the result into CSV File ?Please Help
Related
Having the following binary file (mp3) that send audio to a service in Azure to be trascripted.
The following code works in Databricks.
import os
import requests
url = "https://endpoint_service"
headers = {
'Ocp-Apim-Subscription-Key': 'MyKey',
'Content-Type': 'audio/mpeg'
}
def send_audio_transcript(url, payload, header):
"""Send audio.mp3 to a Azure service to be transcripted to text."""
response = requests.request("POST", url, headers=headers, data=payload)
return response.json()
full_path = <my_path>file.mp3
with open(full_path, mode='rb') as file: # b is important -> binary
fileContent = file.read()
send_audio_transcript(url, fileContent, headers) # a POST request its works
But my audio files are in a sensitive storage in Data lake and the only way to access them is by spark read.
looking for the documentation the way to read a binary file is.
df = spark.read.format("binaryFile").load(full_path)
display(df)
path || modificationTime || length || content
path || sometime || some_lenght || 2dnUwAC
first try:
content = df.content
test_service = send_audio_transcript(url, content , headers)
ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
Second try(convert spark to pandas):
pandas_df = df.toPandas()
content = pandas_df["content"]
test_service = send_audio_transcript(url, content , headers)
Valuerror:ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
What is the exactly translate in python-pyspark to:
with open(full_path, mode='rb') as file: # b is important -> binary
fileContent = file.read()
Your content data comming from Spark is not the same as the content data comming from open file.
From spark and later pandas you have a pandas series but from open the file you will have a class bytes
with open(full_path, mode='rb') as file: # b is important -> binary
fileContent = file.read()
print(type(fileContent)) # will return <class 'bytes'>
but from Spark
input_df = spark.read.format("binaryFile").load(full_path)
pandas_df = input_df.toPandas()
content = pandas_df['content']
print(type(content)) # return <class 'pandas.core.series.Series'>
In your case to fix your problem you need to take just the first element of the series.
content_good = content[0]
print(content_good) # you have your <class 'bytes'> wich is what you need
Need help, i receive a file with new line.
name,age
"Maria",28
"Kevin",30
"Joseph",31
"Faith",20
"Arnel
",21
"Kate",40
How can I identify that line and remove it from the list?
output should be
name,age
"Maria",28
"Kevin",30
"Joseph",31
"Faith",20
"Kate",40
This is one approach
import csv
data = []
with open(filename) as infile:
reader = csv.reader(infile)
for line in reader:
if not line[0].endswith("\n"):
data.append(line)
with open(filename, "w") as outfile:
writer = csv.writer(outfile)
writer.writerows(data)
You can also correct the entry using str.strip().
Ex:
import csv
data = []
with open(filename) as infile:
reader = csv.reader(infile)
for line in reader:
if line[0].endswith("\n"):
line[0] = line[0].strip()
data.append(line)
with open(filename, "w") as outfile:
writer = csv.writer(outfile)
writer.writerows(data)
I'm using pg.8000 (Postgres) and trying to run the following SELECT query
cursor.execute(
"""
SELECT orders.name, orders.order_price, orders.selling_price, orders.earnings
FROM member, orders
WHERE member.id=orders.member_id
AND member.name = %s
""",
member_username
)
Where member.username is a String.
But I am getting the following error.
pg8000.core.ProgrammingError: ('ERROR', 'ERROR', '42P18', 'could not determine data type of parameter $2', 'postgres.c', '1350', 'exec_parse_message', '', '')
However, when I run same query using GUI tool, everything runs fine and I get the results. What is the problem?
You passed the parameter wrong, you should give a tuple, a list or a dictionary
Example with a tuple:
cursor.execute(
"""
SELECT orders.name, orders.order_price, orders.selling_price, orders.earnings
FROM member, orders
WHERE member.id=orders.member_id
AND member.name = %s
""",
(member_username,)
)
Example with a list:
cursor.execute(
"""
SELECT orders.name, orders.order_price, orders.selling_price, orders.earnings
FROM member, orders
WHERE member.id=orders.member_id
AND member.name = %s
""",
[member_username]
)
Example with a dictionary:
cursor.execute(
"""
SELECT orders.name, orders.order_price, orders.selling_price, orders.earnings
FROM member, orders
WHERE member.id=orders.member_id
AND member.name = %(mname)s
""",
{'mname' : member_username}
)
http://initd.org/psycopg/docs/usage.html#query-parameters
I have to read 3 different lines from log files based on some text and then output the fields in a csv file.
sample log data:-
20110607 095826 [.] !! Begin test. Script filename/text.txt
20110607 095826 [.] Full path: filename/test/text.txt
20110607 095828 [.] FAILED: Test Failed()..
i have to read file name after !!Begin test. Script. this is my conf file
filter{
grok
{
match => {"message" => "%{BASE10NUM:Date}%{SPACE:pat}{BASE10NUM:Number}%
{SPACE:pat}[.]%{SPACE:pat}%{SPACE:pat}!! Begin test. Script%
{SPACE:pat}%{GREEDYDATA:file}"
}
overwrite => ["message"]
}
if "_grokparserfailure" in [tags]
{
drop{}
}
}
but its not giving me single record, its parsing full log file in json format no parsed field.
I need to provide a button in Sale Order Form View to export order lines to CSV with a specific format. I have searched a lot but I only have found custom modules that don't satisfy the solution because users shouldn't have to select fields.
UPDATE: Solution
I ended up doing the following solution, thanks to #phillip-stack and his answer:
Model
# -*- coding: utf-8 -*-
import csv, sys
from openerp import api, fields, models, _
class sale_order_export_line(models.Model):
_inherit = 'sale.order'
_auto = False
#api.multi
def export_lines_to_csv(self):
return {
'type' : 'ir.actions.act_url',
'url': '/csv/download/sale_order/%s/supplier_name/%s'%(self.id,'American'),
'target': 'blank',
}
#api.model
def _csv_download(self,vals):
order_id = vals.get('order_id')
supplier_name = vals.get('supplier_name')
so = self.env['sale.order'].browse(order_id)
lines = so.order_line.search([('order_id','=',order_id),('supplier_name','ilike',supplier_name)])
columns = [u'Número pedido Dentaltix',u'Nombre de cliente',u'Dirección', u'Código postal', u'Población',
u'Provincia', u'País', u'Teléfono', u'Horario de entrega', u'Referencia', u'Cantidad', u'Envío']
csv = u','.join(columns)
csv += "\n"
if len(lines) > 0:
for ol in lines:
drupal_order_name = so.drupal_order_name if so.drupal_order_name else ''
client_notes = so.client_notes if so.client_notes else ''
supplier_ref = ol.supplier_ref if ol.supplier_ref else ''
picking_policy = DELIVERY_METHODS[so.picking_policy] if so.picking_policy else 'Directo'
product_uos_qty = str(int(ol.product_uos_qty)) if ol.product_uos_qty else '0'
csv_row = u'","'.join(data)
csv += u"\"{}\"\n".format(csv_row)
return csv
sale_order_export_line()
Controller
# -*- coding: utf-8 -*-
from openerp import http
from openerp.http import request
from openerp.addons.web.controllers.main import serialize_exception,content_disposition
class SaleOrderController(http.Controller):
#http.route('/csv/download/sale_order/<int:order_id>/supplier_name/<string:supplier_name>', auth='user')
def sale_order_lines_csv_download(self, order_id, supplier_name, **kw):
if supplier_name:
csv = http.request.env['sale.order']._csv_download({'order_id': order_id, 'supplier_name':supplier_name})
else:
csv = http.request.env['sale.order']._csv_download({'order_id': order_id, 'supplier_name': False})
filename = 'order_lines_%s_%s.csv'%(order_id,supplier_name)
return request.make_response(csv,
[('Content-Type', 'application/octet-stream'),
('Content-Disposition', 'attachment; filename="%s"'%(filename))])
I probably should not be admitting to this. But I use a technique with controllers and adding a link to a form. The controller can be modified to conduct custom security checks of your choosing then you can use sudo() to bypass field restrictions on the models in question. Then just return the csv with the format of your choosing.
And of course... An Example!
CONTROLLER
#http.route('/csv/download/<int:rec_id>/', auth='user', website=True)
def csvdownload(self, rec_id, **kw):
return http.request.env['your_addon.your_model']._csv_download({'rec_id': rec_id})
MODEL METHOD
def _get_csv_url(self):
self.csv_url = "/csv/download/{}/".format(self.id)
csv_url = fields.Char(compute=_get_csv_url)
#api.model
def _csv_download(self,vals):
sql = """SELECT
quote_nullable(field_1),
quote_nullable(field_2),
quote_nullable(field_3),
quote_nullable(field_4)
FROM
table_name
WHERE id={}""".format(vals.get(rec_id))
self.env.cr.execute(sql)
rows = self.env.cr.fetchall()
csv = """'Field 1','Field 2','Field 3','Field 4'\n"""
if rows:
for row in rows:
csv_row = ""
for item in row:
csv_row+= "{},".format(item)
csv+="{}\n".format(csv_row[:-1])
return csv
In your form have a link which points to your controller
<a id="csv_download" href="#" target="_blank" download="file.csv"/>
<div id="csv_url_div" style="display:none"><field name="csv_url"/></div>
<script>
$(document).ready(function(){
var csv_url = $("#csv_url_div").text();
$("#csv_download").attr("href", csv_url);
});
</script>
I acknowledge the level of hackyness that is going on here. I am sure if I spent more time on it I could do something with a nice Odoo widget that would be quite nice. But it has worked for me.