While learning, I'm trying to read a web log and extract few fields from it. The web log will be like below
147.172.225.10 - 16401 [16/Sep/2013:23:52:35 +0100] "GET /KBDOC-00057.html HTTP/1.0" 200 11761 "http://www.newbie.com" "test F20L"
147.172.225.10 - 16401 [16/Sep/2013:23:52:35 +0100] "GET /theme.css HTTP/1.0" 200 12353 "http://www.newbie.com" "test Mobile Browser Sorrento F20L"
23.53.29.101 - 32693 [16/Sep/2013:23:49:50 +0100] "GET /KBDOC-00035.html HTTP/1.0" 200 9337 "http://www.newbie.com" "test Mobile Browser i3"
And I need to extract just the IP address and User id(3rd field) from the logs and print as
147.172.225.10/16401
147.172.225.10/16401
23.53.29.101/32693
If I have to use map or flatMap, could someone help me how and let me know if there is a better way to accomplish it. Thanks in advance !!
For each line(as string) in the weblog, you can use split method on the basis of a blank character, and after splitting it will return an array of string, with that, you can extract the required values.
lines map { line =>
val lineArray = line.split(" ")
lineArray(0), lineArray(2)
}
Thanks everyone, I used below to get the required results.
map(line => line.split(' ')).map(fields => (fields(0) + "/" + fields(2)))
Related
In Apiary, the cURL call to production by default is :
https://example.com/v1/findBrandCat?matchstring=&interestType=
I have to make a call in following structure:
https://example.com/v1/findBrandCat/matchstringVALUE/interestTypeVALUE
How to make it?
A URI template for the API resource can be defined as follows:
# GET /v1/findBrandCat/{matchstringValue}/{interestTypeValue}
+ Parameters
+ matchstringValue: (required, string)
+ interestTypeValue: (required, string)
+ Response 200 (application/json)
I am attempting to make RESTful POST request using the WifiESP library (https://github.com/bportaluri/WiFiEsp). I'm able to successfully make the request with curl, but consistently get an error using the Arduino and ESP. I suspect the problem is related to the manual formatting of the POST request the library requires, but I don't see anything wrong. Here my sanitized code:
if (client.connect(server, 80)) {
Serial.println("Connected to server");
// Make a HTTP request
String content = "{'JSON_key': 2.5}"; // some arbitrary JSON
client.println("POST /some/uri HTTP/1.1");
client.println("Host: http://things.ubidots.com");
client.println("Accept: */*");
client.println("Content-Length: " + sizeof(content));
client.println("Content-Type: application/json");
client.println();
client.println(content);
}
The error I get (via serial monitor) is this:
Connected to server
[WiFiEsp] Data packet send error (2)
[WiFiEsp] Failed to write to socket 3
[WiFiEsp] Disconnecting 3
My successful curl requests looks like this:
curl -X POST -H "Content-Type: application/json" -d 'Some JSON' http://things.ubidots.com/some/uri
After some experimentation, here is the solution to the multiple problems.
The JSON object was not correctly formatted. Single quotes were not accepted, so I needed to escape the double quotes.
The host does not need "http://" in a POST request; POST is a HTTP method.
The sizeof() method returns the size, in bytes, of the variable in memory rather than the length of the string. It needs to be replaced by .length().
Appending an integer to a string requires a cast.
This is the corrected code:
if (client.connect(server, 80)) {
Serial.println("Connected to server");
// Make the HTTP request
int value = 2.5; // an arbitrary value for testing
String content = "{\"JSON_key\": " + String(value) + "}";
client.println("POST /some/uri HTTP/1.1");
client.println("Host: things.ubidots.com");
client.println("Accept: */*");
client.println("Content-Length: " + String(content.length()));
client.println("Content-Type: application/json");
client.println();
client.println(content);
}
The code explained by Troy D is right and it's working .I think the error in posting the data to the server is due to this line
client.println("Content-Length: " + sizeof(content));
and the correct way is
client.println("Content-Length: " + String(content.length()));
Now coming to this error
Connected to server
[WiFiEsp] Data packet send error (2)
[WiFiEsp] Failed to write to socket 3
[WiFiEsp] Disconnecting 3
This is the error of library you can ignore it.
The problem with "Data packet send error (2)", "Failed to write to socket 3" and "Disconnecting 3" is not a problem within the WifiEsp library as far as I can see, believe it's more likely to be within the AT firmware. By default the http headers contain a "Connection: close" parameter which in normal cases should be correct. However with this bug the server will get disconnected before the reply is received on the client side and any response from the server will be identified as garbage data. Using the value "Connection: keep-alive" as a workaround will make it possible to receive the acceptance from the server in a proper way.
I'm running my Arduino + ESP8266-07 against a MVC based Web Api that I created on one of my servers and in the controllers Post-method I use a single string as return value, the value I return if everything is ok is simply one of the strings that WifiEsp keeps track of (It will still include the http status code in the response header that it returns)
public async Task<string> Post([FromBody]JObject payload)
{
//Code to handle the data received, in my case I log unit ip, macaddress, datetime and sensordata into a db with entity framework
return "SEND OK";
}
So in your Arduino code try following instead:
String PostHeader = "POST http://" + server + ":" + String(port) + "/api/values HTTP/1.1\r\n";
PostHeader += "Connection: keep-alive\r\n";
PostHeader += "Content-Type: application/json; charset=utf-8\r\n";
PostHeader += "Host: " + server + ":" + String(port) + "\r\n";
PostHeader += "Content-Length: " + String(jsonString.length()) + "\r\n\r\n";
PostHeader += jsonString;
client.connect(server.c_str(), port);
client.println(PostHeader);
client.stop();
In the file debug.h located in the library source code you could alter a define and get more output to your serial console. Open the file and change
#define _ESPLOGLEVEL_ 3
to
#define _ESPLOGLEVEL_ 4
Save the file and recompile/deploy your source code to your Arduino and you will get extensive information about all AT commands the library sends and what the library receives in return.
Is there a standard way to get all possible (excluding wildcards of course) routes / paths valid within a play application ?
I can do it with
Play.current.routes.map( _.documentation.map(_._2))
which gives me all available routes but it looks a bit hacky to me.
Just for elaboration, what I do is this ( exactly what you did ):
val myroutes = Play.current.routes map (routes => routes.documentation) getOrElse (Nil)
myroutes foreach { r =>
println("%-10s %-50s %s".format(r._1, r._2, r._3))
}
And I also get:
GET / controllers.Application.index
GET /apis controllers.Application.apis
GET /docs controllers.Application.docs
GET /sampleapi/status/$id<[^/]+> controllers.SampleAPI.status(id:String)
GET /city/$id<[^/]+> controllers.CityAPI.findById(id:Long)
GET /city1/$id<[^/]+> controllers.CityAPI.findById1(id:Long)
GET /apps/authorize/ controllers.Apps.authorize
POST /apps/send_auth/ controllers.Apps.send_auth
GET /clients/ controllers.Clients.list
GET /client/new controllers.Clients.create
POST /client/add controllers.Clients.add
POST /client/update controllers.Clients.update
GET /client/edit/$id<[^/]+> controllers.Clients.edit(id:String)
PUT /client/ controllers.Clients.update
GET /client/$id<[^/]+> controllers.Clients.get(id:String)
DELETE /client/$id<[^/]+> controllers.Clients.delete(id:String)
GET /login controllers.Auth.login
POST /authenticate controllers.Auth.authenticate
GET /logout controllers.Auth.logout
POST /oauth2/access_token controllers.OAuth2Controller.accessToken
GET /webjars/$file<.+> controllers.WebJarAssets.at(file:String)
GET /assets/$file<.+> controllers.Assets.at(path:String = "/public", file:String)
This also includes webjar and assets path, so you may want to filter them out.
I am testing CherryPy with _cp_dispatch.
However, when I send 1 single post, _cp_dispatch is called twice, not once. First for the expected post then a second time with a get: Why?
The code:
import os
import cherrypy
class WebServerApp:
def __init__(self):
self.index_count = 0
self.cpdispatch_count = 0
def __del__(self):
self.exit()
def _cp_dispatch(self, vpath):
self.cpdispatch_count += 1
cherrypy.log.error('_cp_dispatch: ' + str(vpath) + ' - index count:' + str(self.cpdispatch_count))
if len(vpath) == 0:
return self
if len(vpath) == 2:
vpath.pop(0)
cherrypy.request.params['id'] = vpath.pop(0)
return self
return vpath
#cherrypy.expose
def index(self, **params):
try:
self.index_count += 1
cherrypy.log.error('Index: received params' + str(params) + ' - index count:' + str(self.index_count))
except Exception as e:
cherrypy.log.error(e.message)
def exit(self):
cherrypy.log.error('Exiting')
exit.exposed = True
ws_conf = os.path.join(os.path.dirname(__file__), 'verybasicwebserver.conf')
if __name__ == '__main__':
cherrypy.quickstart(WebServerApp(), config=ws_conf)
The config file:
[global]
server.socket_host = "127.0.0.1"
server.socket_port = 1025
server.thread_pool = 10
log.screen = True
log.access_file = "/Users/antoinebrunel/src/Rankings/log/cherrypy_access.log"
log.error_file = "/Users/antoinebrunel/src/Rankings/log/cherrypy_error.log"
The post with request:
r = requests.post("http://127.0.0.1:1025/id/12345")
The log results showing that cp_dispatch is called 3 times: 1 at startup and twice for the post
pydev debugger: starting (pid: 5744)
[30/Sep/2014:19:16:29] ENGINE Listening for SIGUSR1.
[30/Sep/2014:19:16:29] ENGINE Listening for SIGHUP.
[30/Sep/2014:19:16:29] ENGINE Listening for SIGTERM.
[30/Sep/2014:19:16:29] ENGINE Bus STARTING
[30/Sep/2014:19:16:29] _cp_dispatch: ['global', 'dummy.html'] - _cp_dispatch count:1
[30/Sep/2014:19:16:29] ENGINE Started monitor thread '_TimeoutMonitor'.
[30/Sep/2014:19:16:29] ENGINE Started monitor thread 'Autoreloader'.
[30/Sep/2014:19:16:29] ENGINE Serving on http://127.0.0.1:1025
[30/Sep/2014:19:16:29] ENGINE Bus STARTED
[30/Sep/2014:19:16:34] _cp_dispatch: ['id', '12345'] - _cp_dispatch count:2
127.0.0.1 - - [30/Sep/2014:19:16:34] "POST /id/12345 HTTP/1.1" 301 117 "" "python-requests/2.4.0 CPython/3.4.1 Darwin/13.3.0"
[30/Sep/2014:19:16:34] _cp_dispatch: ['id', '12345'] - _cp_dispatch count:3
[30/Sep/2014:19:16:34] Index: received params{'id': '12345'} - index count:1
127.0.0.1 - - [30/Sep/2014:19:16:34] "GET /id/12345/ HTTP/1.1" 200 - "" "python-requests/2.4.0 CPython/3.4.1 Darwin/13.3.0"
Any idea of why _cp_dispatch is called twice for a single post?
-- EDIT
I'm suspecting some 301 redirection going on internally since it appears in the log.
In cherrypy, an internal redirection occurs when the url does not end with a slash.
https://cherrypy.readthedocs.org/en/3.3.0/refman/_cprequest.html#cherrypy._cprequest.Request.is_index
There are 2 ways to resolve the "problem":
First is obviously posting to http://example.com/id/12345/
Second is adding the following to the configuration file:
tools.trailing_slash.on = False
https://cherrypy.readthedocs.org/en/3.2.6/concepts/config.html
Im trying to connect to Facebook using R. I tried to use the codes from
https://github.com/datamgmt/facebook-data-mining
Which gets the user information with the following code:
access.token<- "AAACE________n6bcITRIMCZCKhMySOV" #my access token
FacebookFetch <- function(access.path = "me", access.token, access.options) {
if (missing(access.options)) {
access.options <- ""
}
else {
access.options <- sprintf("?%s", paste(names(access.options), "=",
unlist(access.options), collapse = "&", sep = ""))
}
data <- getURL(sprintf("https://graph.facebook.com/%s%s&access_token=%s",
access.path, access.options, access.token),cainfo="cacert.pem")
fromJSON(data)
}
individual.id <- "100__72" # my ID
individual <- FacebookFetch(access.path=paste(individual.id),access.token=access.token )
if (length(individual$id)) {
cat ("Working with individual: ",individual$name," (",individual$id,")\n", sep="")
} else{
cat("Message: ", unlist(individual)[1], "\n")
cat("Type: ", unlist(individual)[2], "\n")
cat("Code: ", unlist(individual)[3], "\n")
stop(" Cannot continue")
}
where cainfo="cacert.pem" is added to work around the SSL certificate verification error.
Although now I get the following error and I have no idea how to solve this.
Message: (#803) Some of the aliases you requested do not exist: 100__72&access_token=AAACE________n6bcITRIMCZCKhMySOV
Type: OAuthException
Code: 803
I found some posts where people got the same problem, only not using R but other programs, the problem was fixed by making the token and id a character. However, in R, these are already of class character.
When im recreating the url in the browser it returns the same error, while the graph explorer returns all the correct information. I have tried to refresh the token multiple times, but that didn't work.
The API reference book doesn't state what error code 803 entails.
does anybody have an idea how to solve this?
Im using r 2.15.2 in Rstudio on a windows 7 pro (also tried it on windows vista)