I'm downloading a CSV file from a website. I need to download this file while being logged in. The CSV file gives player projections for fantasy sports. When you download the file it will give you five players. However, if you purchase the premium service you get all player projections. I purchased the premium service, so, I'm trying to download this file while being signed into my account.
The code below downloads the CSV file with only five players. How do I sign into my account and then download this file?
guard let url = URL(string: "https://rotogrinders.com/projected-stats/nba-player.csv?site=fanduel") else { return }
let config = URLSessionConfiguration.default
// I don't know what I'm doing here. Also, the user name and password is not correct
let credential = URLCredential(user: "joe", password: "12345", persistence: .forSession)
let protectionSpace = URLProtectionSpace(host: "rotogrinders.com", port: 443, protocol: "https", realm: "Restricted", authenticationMethod: NSURLAuthenticationMethodHTTPBasic)
// I don't know what I'm doing here either.
let credentialStorage = URLCredentialStorage()
credentialStorage.set(credential, for: protectionSpace)
config.urlCredentialStorage = credentialStorage
let task = URLSession(configuration: config).dataTask(with: url) { data, response, error in
guard data != nil else { return }
guard let rows = NSString(data: data!, encoding: String.Encoding.utf8.rawValue)?.components(separatedBy: "\n") else { return }
print(rows)
}
task.resume()
The problem is that this site doesn't have an API. They specifically don't allow the use case you are trying to do.
https://rotogrinders.com/threads/site-with-api-597932
However, there are great tools in Python that may let you do what you are trying to do. Take a look at scrapy:
https://www.edureka.co/blog/web-scraping-with-python/
Related
My app has bugs that happen randomly and only happen on real device, so I made a func to log important things as a txt file on Apple Watch, I planned to email the log file to myself when needed, but just found out the watchOS doesn't support the MessageUI framework.
Is there any way to send my log file to Mac from Apple Watch? Or What's the best way to read the log file?
Thanks.
This is code for logging, in case it's needed:
let logFileName: String = "log.txt"
func myLog(dataToWrite: String) {
do {
let dir: URL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).last! as URL
let url = dir.appendingPathComponent(logFileName)
try "Test \(Date())".appendLineToURL(fileURL: url as URL)
let result = try String(contentsOf: url as URL, encoding: String.Encoding.utf8)
}
catch {
print("Could not write to file")
}
}
is there a good function to download images from AWS S3 bucket? I have an access key and a secret key for permisson. The URL is thru a different database accessible. I also already imported AWSS3 and AWSCore.
I have already found a upload function:
func uploadFile(withImage image: UIImage) {
let access = "access_key"
let secret = "secret_key"
let credentials = AWSStaticCredentialsProvider(accessKey: access, secretKey: secret)
let configuration = AWSServiceConfiguration(region: AWSRegionType.EUCentral1, credentialsProvider: credentials)
AWSServiceManager.default().defaultServiceConfiguration = configuration
let s3BucketName = "bucket_name"
let compressedImage = image.resizedImage(newSize: CGSize(width: 80, height: 80))
let data: Data = compressedImage.pngData()!
let remoteName = generateRandomStringWithLength(length: 12)+"."+data.format
print("REMOTE NAME : ",remoteName)
let expression = AWSS3TransferUtilityUploadExpression()
expression.progressBlock = { (task, progress) in
DispatchQueue.main.async(execute: {
// Update a progress bar
})
}
var completionHandler: AWSS3TransferUtilityUploadCompletionHandlerBlock?
completionHandler = { (task, error) -> Void in
DispatchQueue.main.async(execute: {
// Do something e.g. Alert a user for transfer completion.
// On failed uploads, `error` contains the error object.
})
}
let transferUtility = AWSS3TransferUtility.default()
transferUtility.uploadData(data, bucket: s3BucketName, key: remoteName, contentType: "image/"+data.format, expression: expression, completionHandler: completionHandler).continueWith { (task) -> Any? in
if let error = task.error {
print("Error : \(error.localizedDescription)")
}
if task.result != nil {
let url = AWSS3.default().configuration.endpoint.url
let publicURL = url?.appendingPathComponent(s3BucketName).appendingPathComponent(remoteName)
if let absoluteString = publicURL?.absoluteString {
// Set image with URL
print("Image URL : ",absoluteString)
}
}
return nil
}
}
I would not recommend to download files directly from S3 using an access and secret key.
I'd propose you do the following:
Make sure the bucket is as "private" as can be.
Have an API with authentication and authorisation (AWS API Gateway) that checks if the user is authenticated and permitted to download the S3 object.
Generate a pre-signed download URL with that is only valid for a short period of time (15-60 minutes).
Return that pre-signed download URL to your app through the API.
Use the URL within your app to download the S3 object.
This way you don't have to ship username and password in your app and the bucket is closed off to the "outside" reducing the risk of accidental information leakage.
Why I wouldn't recommend using the access key and secret key:
This is a potential security issue. People that reverse engineer the app could gain access to those "static" keys and depending on the underlying IAM role do all sorts of harm. But even if you have proper IAM roles with very limited access, essentially shipping a username and password with your app is not a good idea under any circumstance. How would you "rotate" the secret if something bad happens etc.
I have a server hosted with webfaction that I would like to be able to send a csv file to from my app with FTP or SFTP. I have found many libraries that should help like ConnectionKit, NMSSH, DLSFPT, and LxFTPRequest. However, all of them are in objective-c and not swift which makes them hard to read, understand, and implement in Swift 4. I have tried to implement LXFTPRequest since I found a swift implementation for the upload and here is my code:
let fileName = "user-data.csv"
guard let path = FileManager.default.urls(for: .documentDirectory, in:.userDomainMask).first else {fatalError(ErrorMessageStrings.couldntAccessDocs.rawValue)}
let fileURL = path.appendingPathComponent(fileName)
let folderLocation = "/home/path/"
let uploadUrl = URL(string: "ftp://server-name.webfaction.com" + folderLocation)
let request = LxFTPRequest.upload()
request?.serverURL = uploadUrl
request?.localFileURL = fileURL
request?.username = "username"
request?.password = "password"
request?.successAction = { (resultClass, result) in
print("File uploaded")
}
request?.failAction = { (domain, error, errorMessage) in
print(error)
print(errorMessage?.description)
fatalError("Connection could not be made. Action was not completed.")
}
request?.progressAction = {(_ totalSize: Int, _ finishedSize: Int, _ finishedPercent: CGFloat) -> Void in
print(finishedPercent)
}
request?.start()`
Using this I almost get it to work but I end up with a 550 error "Requested action not taken. File unavailable (e.g., file not found, no access)." Looking through webfaction documentation I get the feeling that I can only send files through SFTP, which this framework doesnt support.
The doc says "To connect with FTP (for shell users only), substitute the connection type with FTP and the port number with 21." I am assuming since I am sending data from my app it does not count as a shell user and so FTP doesn't grant me access (I may be wrong here). If that is the case how would I go about using the other libraries to send my file over SFTP using Swift and not objective-c?
I ended up using NMSSH and using it in Swift it wasn't as complicated as I thought.
let session = NMSSHSession.init(host: serverHost, port: xx, andUsername: serverUsername)
session.connect()
if session.isConnected{
session.authenticate(byPassword: serverPasswordString)
if session.isAuthorized == true {
let sftpsession = NMSFTP(session: session)
sftpsession.connect()
if sftpsession.isConnected {
sftpsession.writeFile(atPath: csvFileURL.path, toFileAtPath: folderLocation)
}
}
}
Powerschool is a site that keeps track of letter days to indicate lab days for my high school. It also keeps track of people's grades, but I just want to scrape the letter day.
So I am trying to access info contained on html off the site https://pschool.princetonk12.org/guardian/home
To do that I must login through this portal https://pschool.princetonk12.org/public/
which uses oauth.
I know I have to use an authentication token and I see it printed in the output. How do I make a session on https://pschool.princetonk12.org/guardian/home to ultimately parse through it? How do I get the token and use it? Thanks so much!
Here is what I have so far:
import Foundation
import PlaygroundSupport
PlaygroundPage.current.needsIndefiniteExecution = true
let credential = URLCredential(user: "username", password: "password", persistence: URLCredential.Persistence.forSession)
let protectionSpace = URLProtectionSpace(host: "pschool.princetonk12.org", port: 443, protocol: "https", realm: "Restricted", authenticationMethod: NSURLAuthenticationMethodHTTPBasic)
URLCredentialStorage.shared.setDefaultCredential(credential, for: protectionSpace)
let config = URLSessionConfiguration.default
let session = URLSession(configuration: config)
let url = URL(string: "https://pschool.princetonk12.org/public/.json")!
let task = session.dataTask(with: url) { (data, response, error) in
guard error == nil else {
print(error?.localizedDescription ?? "")
return
}
print(String(data: data!, encoding: .utf8))
}
task.resume()
This question is related to: Swift Core Data Sync With Web Server.
I have followed the steps that have been mentioned in the question above however I am unable to apply the third step to my current project.
I currently have a class called Records
class Records {
static let shared = Records()
var records = [Record]()
let context = PersistenceServce.context
let request = NSFetchRequest<Record>(entityName: "Record")
func recordData() -> [Record] {
do {
records = try context.fetch(Record.fetchRequest())
}catch {
print("Error fetching data from CoreData")
}
return records
}
}
and here is how I display the data on my tableViewController.
func getData() {
records = Records.shared.recordData()
self.tableView.reloadData()
}
I do know how save data to a web server as this tutorial explains: https://www.simplifiedios.net/swift-php-mysql-tutorial/ as well as check for internet connection. However I am unsure how to apply it to the CoreData where there are multiple data involved.
If anyone could direct me to a solution or an explain how this can be achieved I'd very much appreciate it.
The question that you have linked is not trying to explain how to communicate with a web server. It is explaining how to store data in core data and tag/mark it in a way that you know which records have been sent to the web server or not.
So the Predicate will fetch all records that have not been sent to the web server and allow you to send them when you have an internet connection available.
Communicating with a web server can be a broad topic and will depend on your web server and API setup, so it is too much to explain here fully. I refer you to some free online resources that will help you understand networking in Swift.
Udacity - Networking with Swift
Ray Wenderlich - Alamofire Tutorial
Stack Overflow - Swift POST Request
Here is an example of a POST Request from the StackOverflow answer above
var request = URLRequest(url: URL(string: "http://test.tranzporthub.com/street45/customer_login.php")!)
request.httpMethod = "POST"
let postString = "user_id=chaitanya3191#gmail.com&password=123"
request.httpBody = postString.data(using: .utf8)
let task = URLSession.shared.dataTask(with: request) { data, response, error in
guard let data = data, error == nil else { // check for fundamental networking error
print("error=\(error)")
return
}
if let httpStatus = response as? HTTPURLResponse, httpStatus.statusCode != 200 { // check for http errors
print("statusCode should be 200, but is \(httpStatus.statusCode)")
print("response = \(response)")
}
let responseString = String(data: data, encoding: .utf8)
print("responseString = \(responseString)")
}
task.resume()
Using code similar to this, you should be able to send data to your web server, then your web server can do whatever it likes with it.
UPDATE:
To encode your parameters to JSON you can use the following code as a guide
var dictionary = [
"username": "Test User",
"password": "Password"
]
if let jsonData = try? JSONSerialization.data(withJSONObject: dictionary, options: []) {
// jsonData is a byte sequence, to view it you would need to convert to string
print(String(bytes: jsonData, encoding: String.Encoding.utf8))
}
Which would output:
Optional("{\"username\":\"Test User\",\"password\":\"Password\"}")
Note: you would send it as data, not the string version. so your code might look like this:
request.httpBody = jsonData