I am trying yo click the display button on a website with no success and I really had no idea how to do it but I came across something that might work, splash:mouse_press. Will that work with scrapy-splash and if so how do I implement it?
import scrapy
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import NameItem
class LoginSpider(scrapy.Spider):
name = "LoginSpider"
start_urls = ["http://www.starcitygames.com/buylist/"]
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formcss='#existing_users form',
formdata={'ex_usr_email': 'email#example.com', 'ex_usr_pass': 'password'},
callback=self.after_login
)
def after_login(self, response):
item = NameItem()
item["Name"] = response.css("div.bl-result-title::text").get()
return item
Welcome here (:
No need to overcomplicate things.
All you need is GET request to http://www.starcitygames.com/buylist/search?search-type=name&name=game
Well... of course you can vary params. But anyway you'll get all you want right in json format by this link.
It looks like even login isn't needed in fact.
Good luck.
Related
I have a REST client that returns me a list of systems.
I need this list to be as a parameter for a jenkins job.
I think I need Actice Choices Parameter plugin with Groovy and HTTPBuilder in order to do that.
What do you guys think?
I did not find a way to install HTTPBuilder into Jenkins.
Is there any other way that you guys think it is possible?
I have run into the same problem trying to parse parameters via groovy script. Arun's answer did not work for me. However, I have managed to get it to work using the following:
import java.io.BufferedReader
import java.io.InputStreamReader
import java.io.OutputStreamWriter
import java.net.URL
import java.net.URLConnection
import groovy.json.JsonSlurper
def choices = []
def url = new URL("some.data.url")
def conn = url.openConnection()
conn.setDoOutput(true)
def reader = new BufferedReader(new InputStreamReader(conn.getInputStream()))
def results = new JsonSlurper().parseText(reader.getText());
reader.close()
results.each { data -> choices.push(data.field) }
return choices.sort()
First paste the JSON body snapshot output -or whatever your REST client is going to return. That'll help.
For ex: if it'll return a JSON object then you can use Active Choice Parameter's Groovy script step - OR Scriptler script (within the Active Choice Parameter plugin). PS: Scriptler script runs in the same JVM of Jenkins process so it has access to Jenkins/etc object for free. You don't need HTTPBuilder or anything. See the code sample below.
Assuming if your REST client is returning a JSON object and from that object if you want to list hostname of the system or some field name then replace the following variable with that and you'll get it listed while doing "Build with parameters" from Jenkins job's dashboard.
import groovy.json.JsonSlurper
//this will be your URL which will return something, tweak it if you want to pass parameters or username/password acc.
def SOME_URL = "https://koba.baby.com/some_url"
// now connect to the URL and create a connection variable 'conn'
def conn = SOME_URL.toURL().openConnection()
// create a list variable 'servernames'
def servernames = []
// if connection response was successful i.e. http protocol return code was 200, then do the following
if( conn.responseCode == 200 ) {
// get the results / output of the URL connection in a variable 'results'
def results = new JsonSlurper().parseText(conn.content.text)
// to see results variable output uncomment the next line
//println results
// now read each element in the 'results' variable and pick servername/somefield variable into the list variable 'servernames'
results.each { id, data -> servernames.push(data.someField_or_HostName) }
}
return servernames.sort().unique()
// return servernames.sort()
we have a Project in Jira which we use as an inbox for Email. Not all people sending emails are users in JIRA (and they shall not be). Nevertheless, we would like to inform then on having received the Email. The emailaddress is part of the Issue description.
I am aware of some plugins out there but instead of replacing the Mailhandlers, I am trying to write a groovy script for JIRA adapting this code which I want to post into a Post Function on the CREATE transition of a workflow.
The following code works fine when I grab an existing Test-Issue and run the script in the console:
import com.atlassian.jira.ComponentManager
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.issue.IssueManager
import com.atlassian.mail.Email
import com.atlassian.mail.server.MailServerManager
import com.atlassian.mail.server.SMTPMailServer
ComponentManager componentManager = ComponentManager.getInstance()
MailServerManager mailServerManager = componentManager.getMailServerManager()
SMTPMailServer mailServer = mailServerManager.getDefaultSMTPMailServer()
if (mailServer) {
if (true) {
IssueManager issueManager = componentManager.getIssueManager()
Issue issue = issueManager.getIssueObject("IN-376")
def grabEmail = {
(((it.split( "\\[Created via e-mail received from:")[1]).split("<")[1]).split(">")[0])
}
String senderAddress = grabEmail("${issue.description}")
Email email = new Email(senderAddress)
email.setSubject("JIRA Ticket erstellt: ${issue.summary}")
String content = "Content ----> by Issue2 ${issue.description}"
email.setBody(content)
mailServer.send(email)
}
}
Alas, it will not run in the Post Function like this:
import com.atlassian.jira.ComponentManager
import com.atlassian.jira.issue.Issue
//import com.atlassian.jira.issue.IssueManager
import com.atlassian.mail.Email
import com.atlassian.mail.server.MailServerManager
import com.atlassian.mail.server.SMTPMailServer
ComponentManager componentManager = ComponentManager.getInstance()
MailServerManager mailServerManager = componentManager.getMailServerManager()
SMTPMailServer mailServer = mailServerManager.getDefaultSMTPMailServer()
if (mailServer) {
if (true) {
//IssueManager issueManager = componentManager.getIssueManager()
//Issue issue = issueManager.getIssueObject("IN-376")
def grabEmail = {
(((it.split( "\\[Created via e-mail received from:")[1]).split("<")[1]).split(">")[0])
}
String senderAddress = grabEmail("${issue.description}")
Email email = new Email(senderAddress)
email.setSubject("JIRA Ticket erstellt: ${issue.summary}")
String content = "Content ----> by Issue2 ${issue.description}"
email.setBody(content)
mailServer.send(email)
}
}
I have no idea why the second code breaks since the code this is based on uses issue also as if it is implicitly defined. This Post function is the last to run.
I would also find hints as to debugging this problem helpful.
Thank you!
I'll post my comment as an answer: I did not find an error in any of the logs, then. Maybe I oversaw it, sorry, but I have changed a lot of config by now (installing JEMH trial) so I can not reproduce. Strangely enough, the message gets sent right now, so I have that ill feeling that I had some config in the Notifications Scheme wrong.
Thanks everyone for the help and time.
How do I remove a cookie after processing the request and building the response?
I have tried the following code, but it does not seem to work:
get("/login") { request =>
val message = request.cookies.get("flash-message").map(_.value)
request.removeCookie("flash-message")
render.view(LoginView(message)).toFuture
}
I could not find any methods on ResponseBuilder that would remove a cookie, either.
It turns out, the way to do it, is the usual "JavaScript" way. Just create an expired cookie and send back, like this:
import com.twitter.finagle.http.Cookie
import com.twitter.util.Duration
import java.util.concurrent.TimeUnit
get("/login") { request =>
val message = request.cookies.get("flash-message").map(_.value)
val c = Cookie("flash-message", "")
c.maxAge = Duration(-10, TimeUnit.DAYS)
render.view(LoginView(message)).cookie(c).toFuture
}
Of course 10 days is just an arbitrary "duration" in the past.
I have code like:
https://gist.github.com/daaatz/7665224
but dont know how to test request.
Trying mydomain/secured?user=John&password=p4ssw0rd etc but nothing works.
Can some one tell me or show example in js+html how to check is it working fine ?
Thanks
I've never used BasicAuth in Spray, so i'm not sure if this would be the complete answer, but i hope this will help you.
At first. in spray there is a great spray-testkit written on top of akka testkit. You should definitely check out SecurityDirectives test on github, this will show you how to test basic authentication. A little example, to make this simpler:
As for your route example, i would better edit to the following one:
val myRoute =
(path("secured") & get) {
authenticate(BasicAuth(myUserPassAuthenticator _, realm = "secure site")) {
userName => complete(s"The user is '$userName'")
}
}
}
Adding get directive will specify that this route expects a Get request and sealRoute is obsolete cause RejectionHandler and ExceptionHandler are provided implicitly with runRoute. It is used only in tests, if you want wo check exceptions/rejections.
Now in your tests you should construct auth entities, similar to the test one:
val challenge = `WWW-Authenticate`(HttpChallenge("Basic", "Realm"))
val doAuth = BasicAuth(UserPassAuthenticator[BasicUserContext] { userPassOption ⇒
Future.successful(Some(BasicUserContext(userPassOption.get.user)))
}, "Realm")
And you test case:
"simple auth test" in {
Get("security") ~> Authorization(BasicHttpCredentials("Alice", "")) ~> {
authenticate(doAuth) { echoComplete }
} ~> check { responseAs[String] === "BasicUserContext(Alice)" }
}
In Get("security") specifies that your test will send a Get request on "/security", then add Authorization to the test request, some action and the check part to test the request.
I've never tried to test BasicAuth, so there could be some mistakes.
I would look into CURL for testing routes in web applications, but I've also used the chrome extension Postman with great results as well.
I'm trying to serve an image from a play-mini application.
object App extends Application {
def route = {
case GET(Path("/image")) => Action { request =>
Ok( Source.fromInputStream(getClass.getResourceAsStream("image.gif")).toArray ).as("image/gif")
}
}
}
Unfortunately, this does noe work :) I get the following error
Cannot write an instance of Array[Char] to HTTP response. Try to define a Writeable[Array[Char]]
Don't know about play-mini, but in play20 there is predefined Writeable[Array[Byte]], so you need to provide Array[Byte] for file handling. Also, there is a bit of documentation about serving files in play20.
I had the same problem and kept scratching my head for almost a week. Turned out the solution that worked for me was the following piece of code in my controller class:
def getPhoto(name: String) = Action {
val strPath = Paths.get(".").toAbsolutePath.toString() + "/public/photos/" + name
val file1: File = strPath
.toFile
val fileContent: Enumerator[Array[Byte]] = Enumerator.fromFile(new java.io.File(file1.path.toString))
Ok.stream(fileContent).as("image/jpeg")
}
And the route was defined as below:
GET /photos/:name controllers.myController.getPhoto(name)
Hence typing the URL with the photos extension displayed the photo on the browser like so: http://localhost:9000/photos/2018_11_26_131035.jpg
The image is saved in a folder "public/photos" in the root folder of the application and not necessarily the assets folder. Hope this helps someone :-)