I'm wrapping an upstream API with a Scalatra application and using Dispatch to make async requests. However, I'm having trouble turning the upstream XML into xml.Elems using Dispatch's built-in XML processing support.
I'm trying to do something fairly similar to what's in the Dispatch docs, namely retrieve the upstream XML and do some reprocessing. The functions in question look something like:
def facilitiesSvc = {
val myhost = host("upstream.api.co.uk") / "organisations" / "foo" / "123" / "bar" / "core.xml"
myhost.addQueryParameter("apikey", "123456")
myhost
}
def facilitiesXml: Future[Either[String, xml.Elem]] = {
val res: Future[Either[Throwable, xml.Elem]] = Http((facilitiesSvc) OK as.xml.Elem).either
for(exc <- res.left)
yield "Can't connect to facilities service: \n" +
exc.getMessage
}
This results in:
Left(Can't connect to facilities service: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.)
The upstream API isn't sending back a charset, and when retrieving it, Dispatch is showing it with a Byte Order Mark before the XML begins: <?xml version="1.0" encoding="utf-8"?>.
I can see that earlier versions of Dispatch solved this problem in the following way:
new Http apply(url(uri.toString).copy(defaultCharset = "iso-8859-1") as_str)
However I can't currently see a way to make this work with Dispatch 0.10. Does anybody have any tips for setting the charset on this response, so I can parse what's returned?
Related
I'm trying to write my very first liquidsoap program. It goes something like this:
sounds_path = "../var/sounds"
# Log file
set("log.file.path","var/log/liquidsoap.log")
set("harbor.bind_addr", "127.0.0.1")
set("harbor.timeout", 5)
set("harbor.verbose", true)
set("harbor.reverse_dns", false)
silence = blank()
queue = request.queue()
def play(~protocol, ~data, ~headers, uri) =
request.push("#{sounds_path}#{uri}")
http_response(protocol=protocol, code=20000)
end
harbor.http.register(port=8080, method="POST", "^/(?!\0)+", play)
stream = fallback(track_sensitive=false, [queue, silence])
...output.whatever...
And I was wondering if there is any way to push to the queue from the harbor callback.
Else, how should I proceed about making requests originate from HTTP calls? I really want to avoid telnet. My final objective is having an endpoint that I can call to make my stream play a file on demand and be silent the rest of the time.
give this a go its liquidsoap so its tricky to understand but it should do the trick
########### functions ##############
def playnow(source,~action="override", ~protocol, ~data, ~headers, uri) =
queue_count = list.length(server.execute("playnow.primary_queue"))
arr = of_json(default=[("key","value")], data)
track = arr["track"];
log("adding playnow track '#{track}'")
if queue_count != 0 and action == "override" then
server.execute("playnow.insert 0 #{track}")
source.skip(source)
print("skipping playnow queue")
else
server.execute("playnow.push #{track}")
print("no skip required")
end
http_response(
protocol=protocol,
code=200,
headers=[("Content-Type","application/json; charset=utf-8")],
data='{"status":"success", "track": "#{track}", "action": "#{action}"}'
)
end
######## live stuff below #######
playlist= playlist(reload=1, reload_mode="watch", "/etc/liquidsoap/playlist.xspf")
requested = crossfade(request.equeue(id="playnow"))
live= fallback(track_sensitive=false,transitions=[crossfade, crossfade],[requested, playlist])
output.harbor(%mp3,id="live",mount="live_radio", radio)
harbor.http.register(port=MY_HARBOR_PORT, method="POST","/playnow", playnow(live))
to use the above you need to send a post request with json data like so:
{"track":"http://mydomain/mysong.mp3"}
this is also with the assumption you have the harbor running which you should be able to find out using the liquidsoap docs
there are multiple methods of sending into the queue, there is telnet, you can create a http input, or a metadata request to playnow via the harbor, let me know which one you opt for and i can provide you with a code example
I am using Akka HTTP and would like to log every incoming request and outgoing result. I know, that it exists a logRequestResult directive, but how to use it? Or is it the right for my purpose?
Yes, this is the directive you are looking for, and I agree - the official documentation is a bit hard to grasp on.
Here is how an endpoint with logRequestResult would look like:
val requestHandler: Route = logRequestResult("req/resp", Logging.InfoLevel) {
handleExceptions(errorHandler) {
endpointRoutes
}
}
def start()(implicit actorSystem: ActorSystem,
actorMaterializer: ActorMaterializer): Future[Http.ServerBinding] =
Http().bindAndHandle(
handler = requestHandler,
interface = host,
port = port)
Notice you can choose a generic prefix for each request-response entry, i.e, req/resp, as well as the logging level on which the request-response log is available, i.e. Logging.InfoLevel.
The above example produces log lines similar to the one below:
[your-actor-system-akka.actor.default-dispatcher-19] INFO akka.actor.ActorSystemImpl - req/resp: Response for
Request : HttpRequest(HttpMethod(GET),http://<host>/<path>,List(Host: <host>, Connection: close: <function1>),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))
Response: Complete(HttpResponse(200 OK,List(),HttpEntity.Strict(text/plain; charset=UTF-8,OK),HttpProtocol(HTTP/1.1)))
Happy hakking :)
What is Trigger Event here ?
How to plug this to the EsperEngine for getting events ?
What URI should be passed ? how should engineURI look like ?
Is it the remote location of the esper engine ?
ConfigurationHTTPAdapter adapterConfig = new ConfigurationHTTPAdapter();
// add additional configuration
Request request = new Request();
request.setStream("TriggerEvent");
request.setUri("http://localhost:8077/root");
adapterConfig.getRequests().add(request);
// start adapter
EsperIOHTTPAdapter httpAdapter = new EsperIOHTTPAdapter(adapterConfig, "engineURI");
httpAdapter.start();
// destroy the adapter when done
httpAdapter.destroy();
Changed the stream from TriggerEvents to HttpEvents and I get this exception given below
ConfigurationException: Event type by name 'HttpEvents' not found
The "engineURI" is a name for the CEP engine instance and has nothing to do with the EsperIO http transport. Its a name for looking up what engines exists and finding the engine by name. So any text can be used here and the default CEP engine is named "default" when you allocate the default one.
You should define the event type of the event you expect to receive via http. A sample code is in http://svn.codehaus.org/esper/esper/trunk/esperio-socket/src/test/java/com/espertech/esperio/socket/TestSocketAdapterCSV.java
You need to declare your event type(s) in either Java, or through Esper's EPL statements.
The reason why you are getting exception is because your type is not defined.
Then you can start sending events by specifying type you are sending in HTTP request. For example, here is a bit of code in python:
import urllib
cepurl = "http://localhost:8084"
param = urllib.urlencode({'stream':'DataEvent',
'date': datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"),
'src':data["ipsrc"],
'dst':data["ipdst"],
'type':data["type"]})
# sending event:
f = urllib.urlopen(cepurl + "/sendevent?" + param);
rez = f.read()
in java this probably would be something like this:
SupportHTTPClient client = new SupportHTTPClient();
client.request(8084, "sendevent", "stream", "DataEvent", "date", "mydate");
Suppose I were reading from an InputStream.
How I would normally do it:
val inputStream = ...
try {
doStuff(inputStream)
} finally {
inputStream.close()
}
Whether or not doStuff throws an exception, we will close the InputStream.
How I would do it with iteratees:
val inputStream ...
Enumerator.fromStream(inputStream)(Iteratee.foreach(doStuff))
Will the InputStream be closed (even if doStuff throws an exception)?
A little test:
val inputStream = new InputStream() { // returns 10, 9, ... 0, -1
private var i = 10
def read() = {
i = math.max(0, i) - 1
i
}
override def close() = println("closed") // looking for this
}
Enumerator.fromStream(inputStream)(Iteratee.foreach(a => 1 / 0)).onComplete(println)
We only see:
Failure(java.lang.ArithmeticException: / by zero)
The stream was never closed. Replace 1 / 0 with 1 / 1 and you'll see that the stream closes.
Of course, I could maintain a reference to the original stream and close it in case of failure, but AFAIK the idea of using iteratees is creating composable iteration without having to do that.
Is this expected behavior?
Is there a way to use iteratees so the resources are always disposed correctly?
Iteratees were specially designed for safe resource management. See the first sentence of the Iteratee IO: safe, practical, declarative input processing:
Iteratee IO is a style of incremental input processing with precise resource control.
The idea is that when your resource is only accessed through iteratees then the resource-owning code can tell exactly when the iteratee is finished with the resource and close it immediately. On the other hand, when iteration is managed manually (as with a traditional InputStream) the user of the resource is responsible for closing it. This can lead to leaks.
Saying that, there was a bug in Play 2.1 where fromStream didn't manage the closing of its underlying InputStream! This bug was fixed in Play 2.2.
You can see the fromStream code to see how the Enumerator was fixed by using onDoneEnumerating to close the resource when the iteratee is finished.
I want to forbid the full access to the Solr core from outside, and let it be used only for querying. Thus I am launching secondary server w/ connector instance inside Jetty servlet container (besides, the main webapp) on the port, that is not accessible from the WWW.
When there is incoming HTTP request to the liftweb application, I hook with RestHelper:
object Dispatcher extends RestHelper {
serve {
case List("api", a # _*) JsonGet _ => JString("API is not implemented yet. rest: " + a)
}
}
Targeting my browser to http://localhost/api/solr/select?q=region I get a response "API is not implemented yet. rest: List(solr, select)", so it seems to work. Now I want to do a connection on internal port (where Solr resides) in order to pass the query using the post-api part of the URL (i.e. http://localhost:8080/solr/select?q=region). I am catching the trailing REST-part of the URL (by means of a # _*), but how can I access URL parameters? It would be ideal to pass a raw string (after api path element) to the Solr instance, just to prevent redundant parse/build steps. So applies to the Solr's response: I would like to avoid parsing building JsonResponse.
This seems to be a good example on doing some HTTP-redirection, but then I would have to open the hidden Solr's port, as far as I can understand.
What is the most effective way to cope with this task?
EDIT:
Well, I missed that after JsonGet comes Req value, which has all the needed info. But is there still a way to avoid unwanted parsing/composing URL to hidden port and JSON-response?
SOLUTION:
This is what I've got consdering Dave's suggestion:
import net.liftweb.common.Full
import net.liftweb.http.{JsonResponse, InMemoryResponse}
import net.liftweb.http.provider.HTTPRequest
import net.liftweb.http.rest.RestHelper
import dispatch.{Http, url}
object ApiDispatcher extends RestHelper {
private val SOLR_PORT = 8080
serve { "api" :: Nil prefix {
case JsonGet(path # List("solr", "select"), r) =>
val u = localApiUrl(SOLR_PORT, path, r.request)
Http(url(u) >> { is =>
val bytes = Stream.continually(is.read).takeWhile(-1 !=).map(_.toByte).toArray
val headers = ("Content-Length", bytes.length.toString) ::
("Content-Type", "application/json; charset=utf-8") :: JsonResponse.headers
Full(InMemoryResponse(bytes, headers, JsonResponse.cookies, 200))
})
}}
private def localApiUrl(port: Int, path: List[String], r: HTTPRequest) =
"%s://localhost:%d/%s%s".format(r.scheme, port, path mkString "/", r.queryString.map("?" + _).openOr(""))
}
I'm not sure that I understand your question, but if you want to return the JSON you receive from solr without parsing it you could use a net.liftweb.http.InMemoryResponse that contains a byte[] representation of the JSON.