val url = "http://api.hostip.info/get_json.php?ip=12.215.42.19"
val result = scala.io.Source.fromURL(url).mkString
println(result)
This gives me the complete HTML page. I want to access individual html elements of this web-page now to do some data analysis. In C# we did it using DYNAMIC variable and then putting html data into some class (json object).
How can we format this URL result to some classes for analysis?
Problem is to download HTML URL page, access its individual elements through Scala code.
Give a look to scala-scraper: https://github.com/ruippeixotog/scala-scraper
Related
I am importing a Smartsheet Report through Python, using an API. One of the columns in this report contains a hyperlink that works in Smartsheet, however when importing the report with Python I only receive the words of this column, and not the link behind them. Is it possible to get the URLs of the sheets that these hyperlinks are referring to in any other way? I was thinking maybe based on SheetID (which I can find using the title of the indepentent sheets), but all other suggestions are very welcome!
I've been unable to reproduce the problem you've described.
The report I'm testing with contains the following data. The Google link in the first row is a normal URL that points to https://www.google.com and the Contacts List link in the second row is a sheet hyperlink that points to another sheet in Smartsheet.
First, I use the Python SDK to get the report and then print out the contents of the second cell of the first row (i.e., the one that contains the Google hyperlink):
reportID = 6667768033503108
report = smartsheet_client.Reports.get_report(reportID)
print(report.rows[0].cells[1])
The result of this code showed the following output (JSON formatted here for readability):
{
"columnId": 5228827298293636,
"displayValue": "Google",
"hyperlink": {
"url": "https://www.google.com"
},
"value": "Google",
"virtualColumnId": 2581703205119876
}
So, accessing the URL of the hyperlink can be accomplished with the following code:
url = report.rows[0].cells[1].hyperlink.url
print(url) #shows output: https://www.google.com
The same approach works for getting the URL of the sheet hyperlink in the second row. i.e., running the following code:
reportID = 6667768033503108
report = smartsheet_client.Reports.get_report(reportID)
url = report.rows[1].cells[1].hyperlink.url
print(url) #shows output: https://app.smartsheet.com/sheets/[ID]
This approach should work for you, but if for some reason you're seeing that the cell object in the JSON response (when using the Python SDK) for the cell that contains the link doesn't actually contain a hyperlink object with a url property -- that might indicate a bug either with the Python SDK or with the underlying API. In that case, you might try getting the URL string by using a dictionary, as shown in the following code. (Note: you'll need to import json for this code to work).
reportID = 6667768033503108
# get the report
report = smartsheet_client.Reports.get_report(reportID)
# load the contents of the second cell in the first row
resp_dict = json.loads(str(report.rows[0].cells[1]))
# read the url property from the dictionary
url = resp_dict['hyperlink']['url']
print(url) #shows output: https://www.google.com
I have a GWT Application where i am getting some data from RPC.
So i have
String data = "someData";
Now I have a JSP page where I like to send this "data".
I am currently opening my JSP Page like this from within my GWT Client side.
Window.open("Test.jsp","","");
In my JSP Page I have
<%= new String("Hello!") %>
Now Is it possible to send that "data" from GWT to this Jsp Page .
So my Jsp will say
Hello! someData
You can use
Window.open("Test.jsp?data=someData","","");
to pass parameters to your JSP (in this example parameter name data and value someData.
To fetch the parameter inside a JSP you would use
${param.data}
For a servlet you can use
String value = request.getParameter("data");
to do the same.
Note that URL parameter values (like someData in the example above) need to be properly escaped. GWT has the com.google.gwt.http.client.URL class for that:
String value = "someData?with#disallowed&chars";
String jsplink = "Test.jsp?data="+URL.encodeQueryString(value);
Window.open(jsplink, "", "");
-- EDIT --
Stuff after the ? on the URL is called the "query string". Usually it contain named parameters like name1=value1&name2=value2&name3=.... but that's not a requirement. You can put anything after the ? and handle it yourself. For that you need to get the complete query string as a whole using ((HttpServletRequest)request).getQueryString(). You then need to decode and interpret it yourself.
I am building a REST API and facing this issue: How can REST API pass very large JSON?
Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.
What methods we can use to solve this problem?
DBTraining trainingdata = new DBTraining();
#GET
#Produces("application/json")
#Path("/{cat_id}")
public Response getAllDataById(#PathParam("cat_id") String cat_id) {
List<TrainingData> list = new ArrayList<TrainingData>();
try {
list = trainingdata.getAllDataById(cat_id);
Gson gson = new Gson();
Type dataListType = new TypeToken<List<TrainingData>>() {
}.getType();
String jsonString = gson.toJson(list, dataListType);
return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();
} catch (SQLException e) {
logger.warn(e.getMessage());
}
return null;
}
The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.
By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.
The GitHub API provides a good example of this.
My suggestion is no to pass the data as a JSON but as a file using multipart/form-data. In your file, each line could be a JSON representing a data record. Then, it would be easy to use a FileOutputStream to receive te file. Then, you can process the file line by line to avoid memory problems.
A Grails example:
if(params.myFile){
if(params.myFile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
def fileName = "/tmp/myReceivedFile.txt"
new FileOutputStream(fileName).leftShift(params.myFile.getInputStream())
}
else
//print or signal error
}
You can use curl to pass your file:
curl -F "myFile=#/mySendigFile.txt" http://acme.com/my-service
More details on a similar solution on https://stackoverflow.com/a/13076550/2476435
HTTP has the notion of chunked encoding that allows you send a HTTP response body in smaller pieces to prevent the server from having to hold the entire response in memory. You need to find out how your server framework supports chunked encoding.
i am confused on how to combine the json library in dispatch and lift to parse my json response.
I am apparently a scala newbie.
I have written this code :
val status = {
val httpPackage = http(Status(screenName).timeline)
val json1 = httpPackage
json1
}
Now i am stuck on how to parse the twitter json response
I've tried to use the JsonParser:
val status1 = JsonParser.parse(status)
but got this error:
<console>:38: error: overloaded method value parse with alternatives:
(s: java.io.Reader)net.liftweb.json.JsonAST.JValue<and>
(s: String)net.liftweb.json.JsonAST.JValue
cannot be applied to (http.HttpPackage[List[dispatch.json.JsObject]])
val status1 = JsonParser.parse(status1)
I unsure and can't figure out what to do next in order to iterate through the data, extract it and render it to my web page.
Here's another way to use Dispatch HTTP with Lift-JSON. This example fetches JSON document from google, parses all "titles" from it and prints them.
import dispatch._
import net.liftweb.json.JsonParser
import net.liftweb.json.JsonAST._
object App extends Application {
val http = new Http
val req = :/("www.google.com") / "base" / "feeds" / "snippets" <<? Map("bq" -> "scala", "alt" -> "json")
val json = http(req >- JsonParser.parse)
val titles = for {
JField("title", title) <- json
JField("$t", JString(name)) <- title
} yield name
titles.foreach(println)
}
The error that you are getting back is letting your know that the type of status is neither a String or java.io.Reader. Instead, what you have is a List of already parsed JSON responses as Dispatch has already done all of the hard work in parsing the response into a JSON response. Dispatch has a very compact syntax which is nice when you are used to it but it can be very obtuse initially, especially when you are first approaching Scala. Often times, you'll find that you have to dive into the source code of the library when you are first learning to see what is going on. For instance, if you look into the dispatch-twitter source code, you can see that the timeline method actually performs a JSON extraction on the response:
def timeline = this ># (list ! obj)
What this method is defining is a Dispatch Handler which converts the Response object into a JsonResponse object, and then parses the response into a list of JSON Objects. That's quite a bit going on in one line. You can see the definition for the operand ># in the JsHttp.scala file in the http+json Dispatch module. Dispatch defines lots of Handlers that do a conversion behind the scenes into different types of data which you can then pass to block to work with. Check out the StdOut Walkthrough and the Common Tasks pages for some of the handlers but you'll need to dive into the various modules source code or Scaladoc to see what else is there.
All of this is a long way to get to what you want, which I believe is essentially this:
val statuses = http(Status(screenName).timeline)
statuses.map(Status.text).foreach(println _)
Only instead of doing a println, you can push it out to your web page in whatever way you want. Check out the Status object for some of the various pre-built extractors to pull information out of the status response.
If some one come to my url suppose /get it should give back a xml/application format in response in bottle framework. How can i do this? i am using elementree as xml generator.
Look on the official page for the cookie example and do it like this:
#route('/xml')
def xml():
response.headers['Content-Type'] = 'xml/application'
....(create the xml here)......
return xml_content_whatever