Download Sharepoint Online file using Python - python-3.7

I am trying to automate some work processes by using Python to download an Excel file from our SharePoint site. Some background is that I'm fairly new to Python and am mostly self-taught. I have read over many examples here using sharepy, office365, requests, etc. and none seem to work. Below is what I have tried (basically copied from another example) with the error below that:
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext
from office365.sharepoint.file import File
#Inputs
url_shrpt = "https://company.sharepoint.com/sites/teamsite"
username_shrpt = "user#company.com"
password_shrpt = "password"
folder_url_shrpt = "sites/teamsite/library"
#Authentication
ctx_auth = AuthenticationContext(url_shrpt)
if ctx_auth.acquire_token_for_user(username_shrpt, password_shrpt):
ctx = ClientContext(url_shrpt, ctx_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()
print("Authenticated into sharepoint as: ",web.properties['Title'])
else:
print(ctx_auth.get_last_error())
#Function for extracting the file names from a folder
global print_folder_contents
def print_folder_contents(ctx, folder_url):
try:
folder = ctx.web.get_folder_by_server_relative_url(folder_url)
fold_names = []
sub_folders = folder.files
ctx.load(sub_folders)
ctx.execute_query()
for s_folder in sub_folders:
fold_names.append(s_folder.properties["Name"])
return fold_names
except Exception as e:
print('Problem printing out library contents: ', e)
######################################################
# Call the function by giving your folder URL as input
filelist_shrpt=print_folder_contents(ctx,folder_url_shrpt)
#Print the list of files present in the folder
print(filelist_shrpt)
Here is the error I receive:
Error: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /extSTS.srf (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
After collaborating with some more knowledgeable coworkers, I was told it might be a permissions issue; but I'm not sure how to verify that. Any help and/or advice would be appreciated.

Related

Apache Spark in Azure Synapse Analytics - HTTP request in notebook

I use a Notebook in Synapse where I run my Python code.
I would like to make an API request from this Notebook to Microsoft Purview to send the entities.
I added the pyapacheatlas library to spark.
On my local computer, this code works fine in Visual Studio.
I need to sign in with Microsoft. I created Purview client connections using a service principal. Here is the code that I am running:
from pyapacheatlas.auth import ServicePrincipalAuthentication
from pyapacheatlas.core import PurviewClient
from pyapacheatlas.core import AtlasEntity
auth = ServicePrincipalAuthentication(
tenant_id="...",
client_id="...",
client_secret="..."
)
# Create a client to connect to your service.
client = PurviewClient(
account_name = "...",
authentication = auth
)
# Get All Type Defs
all_type_defs = client.get_all_typedefs()
I am getting an error after running for a long time:
"ConnectionError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /.../oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f00effc5250>: Failed to establish a new connection: [Errno 110] Connection timed out'))"
It turned out that I can't make any HTTP requests in Notebook.
Please advise, maybe this is not provided by the functionality or is it possible to solve it?
Thank you.
Even an ordinary GET like this:
import json
import requests
r = requests.get("http://echo.jsontest.com/insert-key-here/insert-value-here/key/value")
df = sqlContext.createDataFrame([json.loads(line) for line in r.iter_lines()])
As a result I get:ConnectionError:
"HTTPConnectionPool(host='echo.jsontest.com', port=80): Max retries exceeded with url: /insert-key-here/insert-value-here/key/value (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f90b0111940>: Failed to establish a new connection: [Errno 110] Connection timed out'))"

Record Level Data truncation on Mainframe server while doing SFTP from spark server

Please read this fully.
I am working on sending a csv file via SFTP from spark application developed in scala to the mainframe server. I am using jsch (java secure channel) package version 0.1.53 version to accomplish the SFTP connection from spark server to mainframe server. I am facing issue that on the mainframe server, the csv file gets truncated to 1024 bytes per record line.
After research, I found that on the mainframe, we have options like using "lrecl" and "recfm" to control the length of each record in the file and the format of that record. But I am unable to integrate these options on scala. I found this answer on stackoverflow which was meant for implementation in Java. When I use the same logic on scala, I am getting the below error:
EDC5129I No such file or directory., file: /+recfm=fb,lrecl=3000 at
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2198)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2215)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1565)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1526)
Scala code block using the jsch library for establishing the SFTP connection and transferring file is as below:
session = jsch.getSession(username, host, port)
session.setConfig("PreferredAuthentication","publickey")
session.setConfig("MaxAuthTries",2)
System.out.println("Created SFTP Session")
val sftpSessionConfig: Properties = new Properties()
sftpSessionConfig.put("StrictHostKeyChecking","no")
session.setConfig(sftpSessionConfig)
session.connect() //Connect to session
System.out.println("Connected to SFTP Session")
val channel = session.openChannel("sftp")
channel.connect()
val sftpChannel = channel.asInstanceOf[ChannelSftp]
sftpChannel.ls("/+recfm=fb,lrecl=3000") //set lrecl and recfm ---> THROWING ERROR HERE
sftpChannel.put(sourceFile, destinationPath,ChannelSftp.APPEND) //Push file from local to mainframe
Is there any way where we can set these options as the configuration in my scala code using the jsch library? I also tried using the spring-ml's spark-sftp package. But this package also has the problem of data truncation on the mainframe server.
Please help as this issue has become very critical blocker to my project.
EDIT: Updated question with scala code block
From this presentation Dovetail SFTP Webinar on slide 21:
ls /+recfm=fb,lrecl=80
it seems to me there is one '/' too many in your code.
From the error message, I think the SFTP server has the current path in the UNIX file system. You do not set the data set high level qualifier (HLQ) for the data set, do you? I can't see it in the code. Again from the above presentation, do a cd before the ls:
cd //your-hlq-of-choice
This will do two things:
Change the current working directory to the MVS data set side.
Set the HLQ to be used.
Sorry I cannot test myself; I do not know scala.
First, what SFTP server is running on z/OS? If it is the one provided with z/OS (not Dovetail) the command you are executing isn't supported and you will receive a message like Can't ls: "/+recfm=fb,lrecl=80" not found. Which would be valid because that is not valid file. Everything to the right of the / would be considered part of the filename.
I converted your code to Java as I'm not familiar with Scala and didn't have time to learn it. Here was my code sample I used.
import com.jcraft.jsch.JSch;
import java.util.Properties;
import java.util.Vector;
class sftptest {
static public void main(String[] args) {
String username = "ibmuser";
String host = "localhost";
int port = 10022; // Note, my z/OS is running in a docker container so I map 10022 to 22
JSch jsch = new JSch();
String sourceFile = "/";
String destinationPath ="/";
String privateKey = "myPrivateKey";
try {
jsch.addIdentity(privateKey); //add private key path and file
com.jcraft.jsch.Session session = jsch.getSession(username, host, port);
session.setConfig("PreferredAuthentication","password");
session.setConfig("MaxAuthTries", "2");
System.out.println("Created SFTP Session");
Properties sftpSessionConfig = new Properties();
sftpSessionConfig.put("StrictHostKeyChecking","no");
session.setConfig(sftpSessionConfig);
session.connect(); //Connect to session
System.out.println("Connected to SFTP Session");
com.jcraft.jsch.ChannelSftp channel = (com.jcraft.jsch.ChannelSftp) session.openChannel("sftp");
channel.connect();
// com.jcraft.jsch.Channel sftpChannel = (ChannelSftp) channel;
// channel.ls("/+recfm=fb,lrecl=3000"); //set lrecl and recfm ---> THROWING ERROR HERE
// channel.ls("/"); //set lrecl and recfm ---> THROWING ERROR HERE
Vector filelist = channel.ls("/");
for(int i=0; i<filelist.size();i++){
System.out.println(filelist.get(i).toString());
}
// channel.put(sourceFile, destinationPath, com.jcraft.jsch.ChannelSftp.APPEND); //Push file from local to mainframe
} catch (Exception e) {
System.out.println("Exception "+e.getMessage());
}
}
}
For my case I did use an ssh key and not a password. The output with your ls method is:
Created SFTP Session
Connected to SFTP Session
Exception No such file
dropping the + and everything to the right you get:
Created SFTP Session
Connected to SFTP Session
drwxr-xr-x 2 OMVSKERN SYS1 8192 May 13 01:18 .
drwxr-xr-x 7 OMVSKERN SYS1 8192 May 13 01:18 ..
-rw-r--r-- 1 OMVSKERN SYS1 0 May 13 01:18 file 1
-rw-r--r-- 1 OMVSKERN SYS1 0 May 13 01:18 file 2
The main issue is that the z/OS appears to not support the syntax you are using which is provided by a specific SFTP implementation by Dovetail.
If you do not have Dovetail I recommend that since you are sending CSV files that are generally variable in length that you send them as a USS file so that the lines will be properly translated and will be of variable length. Transfer them to USS (regular Unix on z/OS) and then copy them to an MVS file that has a RECFM of VB. Assuming the file is already allocated you could do a cp myuploadedFile.csv "//'MY.MVS.FILE'"

Text to speech - not getting audio in the .wav (connection refused)

I run a flask server in which this function is called whenever an specific action occurs on the page :
def generate_audio(text, target):
# create the path
tmp_dir = os.path.join(os.getcwd(), "app/data/audio")
if not os.path.exists(tmp_dir):
os.mkdir(tmp_dir)
path = os.path.join(tmp_dir, f'{target}-{text}.wav')
# query the API
speech_config = SpeechConfig(
subscription=cfg['speech']['key'], region=cfg['speech']['location'])
audio_config = AudioOutputConfig(filename=path)
synthesizer = SpeechSynthesizer(
speech_config=speech_config, audio_config=audio_config)
synthesizer.speak_text("A simple test")
At the end of the execution, the file containing the audio is just an empty 0B file. I literally copy pasted the quick start guide, so I do not know what is wrong.
What I did try is to change the subscription key to something random and no error was raised. In the logs from the azure service webpage nothing comes up either.
Here's the cancellation details
SpeechSynthesisCancellationDetails(reason=CancellationReason.Error, error_details="Connection failed (no connection to the remote host). Internal error: 11. Error details: Code: 0. USP state: 2. Received audio size: 0 bytes.")
Here's the log
https://pastebin.com/aapsMXYc
I was also facing the same issue and found that I was entering incorrect location name. E.g. in resource, you will see a location name like Central India but in SDK it should be entered as centralindia (this name I found under key management). Hope this will help in resolving this issue.
Thanks

Flask response setting is not working in Bluemix Flask Buildpack

I am trying deploying my flask application to the Bluemix where it is configured by Bluemix Python Flask BuildPack and it is not a custom build back.
#app.route('/run', methods=['GET','POST'])
def stockModuleRunner():
stockCode = request.args.get("stock_code","")
# Store the reference, in case you want to show things again in standard output
old_stdout = sys.stdout
# This variable will store everything that is sent to the standard output
result = StringIO()
sys.stdout = result
# Here we can call anything
# Here we can call anything we like, like external modules, and everything that they will send to standard output will be stored on "result"
r = Resilience.Resilience()
r.main(stockCode)
# Redirect again the std output to screen
sys.stdout = old_stdout
# Then, get the stdout like a string and process it!
result_string = result.getvalue()
print(result_string)
#r = Resilience.Resilience()
#r.main(stockCode)
resp = Response("<html><pre>"+result_string+"</pre></html>")
resp.headers['content-type'] = 'text/html; charset=utf-8'
return
I set the content type of my response as a text/html. But it didn't work in the Bluemix, while it worked in my local environment. In the Bluemix, it changed to text/xml automatically. I confirmed it through the network tab of chrome developer tool. I got the error like the followings. As per the result of a google search, I found out that it is a sort of xml parsing error. Do you have any idea of this problem?
This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.

volttron.platform.vip.agent.core ERROR: Possible conflicting identity

I have been working towards building my agent development skills in Volttron. I am completely new to the platform and trying to understand how to create basic agents that publish and subscribe to the Volttron bus. I'm not alone in this venture and get help from a few other people with experience, but even they are stumped. We are using the same agent files we share through GitHub, but the agent works on their computers and not on mine.
The publishing agent reads from a CSV file that is in the same directory as the agent and is suppose to publish information from the file. I have been careful to map the file directory in my source code to match my setup. I receive the following messages when I start running my publishing agent using eclipse "mars" that's running on Linux Mint 18.1 Serena:
2017-02-02 14:27:22,290 volttron.platform.agent.utils DEBUG: missing file /home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json
2017-02-02 14:27:22,290 volttron.platform.agent.utils INFO: creating file /home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: address: ipc://#/home/edward/.volttron/run/vip.socket
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: identity: None
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: agent_uuid: None
2017-02-02 14:27:22,292 volttron.platform.vip.agent.core DEBUG: severkey: None
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: No response to hello message after 10 seconds.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: A common reason for this is a conflicting VIP IDENTITY.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: Shutting down agent.
2017-02-02 14:27:32,324 volttron.platform.vip.agent.core ERROR: Possible conflicting identity is: f9d18589-d62b-42b7-bac8-3498a0c37220
I have done the following:
Created missing file "/home/edward/.volttron/keystores/f9d18589-d62b-42b7-bac8-3498a0c37220/keystore.json". The only thing that happens when I run the agent again is it gives me the same DEBUG message but with a different file name.
I looked into the "volttron.platform.vip.agent.core" file and have no idea what to do in there. I dont want to create more problems for myself.
I have been using the "Volttron's Documentation" to try and trouble shoot, but I always get the same message when ever I try to run any agent. I have had success when testing the platform and running "make-listener" through the terminal, but that's all.
I have been searching the web for the last couple of days and seen similar issues, but when attempting to follow the advise posted to remedy the situation, I have no luck. Error: volttron.platform.web INFO: Web server not started
Reinstalled Volttron, Mint, and Eclipse on my VM a few times to overcome any compatibility issues...
The source code for the agent is as follows:
#testcodeisforpublishingandprinting
import logging
import sys
#import json
from volttron.platform.vip.agent import Agent, Core, PubSub, compat
#from volttron.platform.vip.agent import *
#from volttron.platform.vip.agent import compat
from volttron.platform.agent import utils
from volttron.platform.messaging import headers as headers_mod
from datetime import datetime
#import numpy as NP
#from numpy import linalg as LA
import csv
outdata=open("/home/edward/volttron/testagent/Agent/PredictionfileP.csv","rb")
Pdata=csv.DictReader(outdata)
Price=[]
for row in Pdata:
Price.append(float(row['Price'])*0.01)
#from volttron.platform.agent import BaseAgent, PublishMixin, periodic, matching, utils
#from volttron.platform.agent import BaseAgent, PublishMixin, periodic
utils.setup_logging()
_log = logging.getLogger(__name__)
class testagent1(Agent):
def __init__(self, config_path, **kwargs):
self.config = utils.load_config(config_path)
super(testagent1, self).__init__(**kwargs)
self.step=0
#print('TestAgent example agent start-up function')
#Core.receiver('onsetup')
def onsetup(self, sender, **kwargs):
self._agent_id = self.config['agentid']
#Core.receiver('onstart')
def onstart(self, sender, **kwargs):
pass
#Core.receiver('onstop')
def onstop(self, sender, **kwargs):
pass
#Core.receiver('onfinish')
def onfinish(self, sender, **kwargs):
pass
#Core.periodic(5)
def simulate(self):
self.step=self.step+1#timestep increase
print('Simulationrunning')
now = datetime.utcnow().isoformat(' ')#time now
headers = {
'AgentID': self._agent_id,
headers_mod.CONTENT_TYPE: headers_mod.CONTENT_TYPE.PLAIN_TEXT,
headers_mod.DATE: now,
}
print(self.step)
self.vip.pubsub.publish('pubsub', 'testcase1/Step', headers, self.step)
print('Simulationupdatingloopingindex')
def main(argv=sys.argv):
'''Main method called by the eggsecutable.'''
try:
utils.vip_main(testagent1)
except Exception as e:
_log.exception('unhandled exception')
if __name__ == '__main__':
# Entry point for script
sys.exit(main())
I installed my version of Volttron using the 3.5RC1 manual published Jan. 2017.
I am assuming that you are running this from eclipse and not through the installation process. The installation of the agent would specify an identity that would remain for the life of the agent.
The remaining answer is specific to running within the eclipse environment.
def main(argv=sys.argv):
'''Main method called by the eggsecutable.'''
try:
# This is where the change is.
utils.vip_main(testagent1, identity='Thisidentity')
except Exception as e:
_log.exception('unhandled exception')
You will have to authorize the agent to be able to connect to the message bus by adding the agent publickey through the auth mechanism. Or you can add the wildcard /.*/ to the credentials of an entry through volttron-ctl auth add.
Thank you for asking this question. We will are updating the documentation to highlight this.
You will need to do the following at the command line:
volttron-ctl auth add
domain []:
address []:
user_id []:
capabilities (delimit multiple entries with comma) []:
roles (delimit multiple entries with comma) []:
groups (delimit multiple entries with comma) []:
mechanism [CURVE]:
credentials []: /.*/
comments []:
enabled [True]:
added entry domain=None, address=None, mechanism='CURVE', credentials=u'/.*/', user_id='ff6fea8e-53bd-4506-8237-fbb718aca70d'