I am trying to find a high-level Clojure library for making HTTP and HTTPS requests, parsing out forms and links from responses and then POST-ing updated forms or following links. Ideally something that would automatically handle redirects and cookies (i.e. sessions). That is, I'd like to find something whereby my code can as closely as possible mimic a user driving a webapp from a browser, without the browser.
A number of years ago we used Hpricot and Ruby for a similar task but I'm prefer to do this in Clojure if at all possible. From memory - and I haven't used Hpricot for years - we were able to do all this with minimal effort: we were able to concentrate on the 'what' of driving the application, not the 'how'.
I found clj-http https://github.com/dakrone/clj-http but this seems to be one step lower-level than I'm looking for (no form parsing) - although it is based on Apache HttpComponents http://hc.apache.org/httpcomponents-client-ga/ which does seem to expose a nice, fluent, API for forms http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html.
Screen scraping in clojure asks about screen-scraping in Clojure, and there are several good suggestions for that, but nothing that really addresses the above.
HTTP Kit http://www.http-kit.org/client.html looks like it would be a great foundation for the above but doesn't do form parsing or session management (as far as I can see).
Currently I've veering toward using the Apache HttpComponents Java library directly from Clojure. Can anyone suggest any better - perhaps more Clojure idiomatic - alternative? Or anything that they found worked well in similar circumstances? My goal is to write the minimal amount of code quickly to investigate a problem with a web service. This is not production code. Saving time, rather than getting an 'ideal' solution is my main concern.
[The background is that I am trying to mimic certain forms of user behaviour in order to first reproduce and then try and track down an intermitent bug in a large body of legacy Java/EJB code. However the problem only seems to occur one time per several thousand POSTs. (The suspicious is of some form of caching issue.) The existence of the problem, after the fact, is easy to detect however.]
Have you looked at the Enlive library yet? Here is a good tutorial on it.
You seem to really have 2 parts here. The first part is (1) a Selenium-like client, which drives (2) a webserver.
For part (1), either Selenium, Enlive, or something similar will allow you to simulate a browser to submit data, read the responses, and respond from there. For part (2), it seems you just need a regular Clojure web framework such as Ring/Compojure (older & simpler) or Pedestal (newer & more powerful).
Related
So, my task is to develop web application using the following technologies:
Angular on front end,
On back end, I will use civetweb as stand-alone web server (https://github.com/civetweb/civetweb) and Lua (or C if Lua doesn't fit well),
Sqlite3 (http://lua.sqlite.org/index.cgi/doc/tip/doc/lsqlite3.wiki)
After a certain amount of time spend on the research about this topic (back end side), I am not sure what are the best practices for this, due to the fact that I couldn't find useful examples, except for the official documentation on civetweb github repository.
Specifically, I am not sure what is the best way: to handle requests/responses, structure the code (the main server logic, the part of the system which is in the charge of working with database, etc.)
I would appreciate any suggestions, links to examples or useful documentation, etc.
P.S. I am used to object-oriented world, and that is the main cause of my problems.
We have narrow down our search between Silex and Slim PHP frameworks for routing our REST APIs on our Apache/PHP/MySQL Server.
Both seem to have good reviews. Silex has probably a bigger community because it came from Symfony. But the documentation seems to be better in Slim.
What do you guys suggests? Any real world experience from production environments?
Sathish
I had the same choice to make, and I choose Silex, here is why :
Silex seems to have a bigger community than Slim, maybe it's just my point of view
It's based on Symfony components, so with a bit of reflection, you can use tips and workarounds which work for Symfony and apply them to Silex.
As it's based on Symfony it has a better integration with other bundle (for example Twig, which is for me necessary)
This Symfony base also garanty that it will be more "Long Term Support" that Slim, which is independent.
To conclude, the main argument is that is based on Symfony which has many advantage.
The Symfony Debug tools are the best thing on earth !!
Now I have two sites made with Twig and I'm really happy !
You can also see that, it's a technical comparison of these both frameworks :
https://michalzuber.wordpress.com/2015/04/02/silex-vs-slim-php-microframework-comparison/
Slim 3 is very light weight and well suited to APIs.
You can optionally inject a container (Pimple by default but any Container-Interop will work) when constructing your Slim app. The Silex app extends Pimple so is a container.
Require slim/twig-view if you need Twig.
Slim's Request and Response support PSR-7 HTTP message implementation.
On January 12th 2018, Fabien Potencier, main author of this micro-framework, wrote that Sensiolabs stops supporting Silex.
https://symfony.com/blog/the-end-of-silex
Quote from Silex official site:
Silex is in maintenance mode. Ends of life is set to June 2018. Use Symfony 4 instead. Read more on Symfony's blog.
Slim is better if you want to create apis as its light and fast. Hence slim provides you the DI and routes it is much more flexible to use your own libraries or plugins from laravel or symfony or any third party.
For example you can use sentinel from laravel for auth
TL;DR Choose Slim 4. "At its core, Slim is a dispatcher that receives an HTTP request, invokes an appropriate callback routine, and returns an HTTP response. That’s it.". That should be it.
Late to the party but nonetheless here's my 2 cents(!) on the topic especially for future-googlers about this topic (even though Silex has reached end of life); I'd go for Slim. Which I already did on my previous 2 projects and it never let me down. Before the "here's why" part I'd like to share my background in a nutshell; I've worked with CakePHP and Laravel (especially >=5.4, <5.8) (and also with Lumen) years and years ago, right before I introduced myself with Node.js. Since then almost all my work runs on Node.js in the cloud. Now there's Golang on the table... but that's another story. Due to the requirements (or I should say constraints) of the aforementioned 2 projects, I have to write PHP. Since all those years past with Node.js I suddenly realized that I've forgotten about PHP almost completely. But there were 2 things I remember crystal clear; I've suffered. A lot. Because of those frameworks.
Don't get me wrong former is the eldest one and the latter is the hipster of the pack, I respect them, their authors and their community. Everything has a reason to exist. But one is behemoth and the other one is magical (both not in a good way - I mean did you ever try to debug the Laravel with those facades and static methods and stuff?). OK I hear you, they are frameworks and they should do things for you (have you heard of Hollywood Principle), not everything for you. What this does mean to me is that they always whispering to your ear "our way or highway" (sometimes even yells). "Surely you can do this (say push a job to a queue, query your database, send email, etc.) but first do this, and then that" (maybe couple of more steps but this is not that bad). The bad thing about this is (not accusing anyone) I don't know what do they mean to the framework. They must do something meaningful, obviously though, I don't know what it is. Should I know? Yes, of course. I'm writing this application with that framework for God sake, I should know what it does. Version X was easy to grasp, but what about version Y, Z, T... Have you seen the version dropdown on the Laravel's documentation site? I took those steps because I've been told so. Again that's OK, fair is fair, those steps makes me achieve something far greater. But gradually I'm in their (respective authors') control. After that, even small change takes lots of searches on SO, GitHub issues, Google... Sometimes ends up with a success and most of the times don't. Either way the war against the framework must be declared.
In my point of view this is not how an open-source framework should be. I am vendor locked-in in some sense. Maybe I want to adhere PSR (not that because I'm an advisor recommendation freak, because PHP-FIG is a well-known group with standards almost for every aspect of the language - this is important; for the language, not for any framework). Let me ask you something; do you use Composer? If so why? Because it's standard (I'm not sure if it is)? Because everyone uses it? Because the needed package of yours recommends this way of installing? Actually the answer doesn't all that matter, at the end of the day you use it. And you can use it practically for every PHP framework/project. The Composer requires you to have PHP installed on the system at bare-minimum and coincidentally it's the only requirement. This is freedom, and I want that. I want to pick and choose a router or a container for my taste. Today this package, later something else.
Slim gave me that freedom, especially on version 4. It's community is small, it's foreseeable; it does so much less than the other, full-blown frameworks. Actually it's a micro-framework (although I wrote an MVC application and REST API server with it). Other packages' communities are what matters right now. You need a container, composer require php-di/php-di. Now think about it's community, since "your" application (hence the framework) is a part of it. You have a problem? Ask for help specifically for that package. Maybe someone using another framework (or someone using no framework at all - if there's any left nowadays) might help you with your problem. So you've gone framework-agnostic thanks to your setup. Don't you like PHP DI, well find yourself another PSR-11 compliant package. Same applies for almost every part of Slim - except the router (Nikic's FastRouter should ring a bell), although it's already a foundation for other routers I see so far.
Before I finish I should say that too, Lumen and Silex has big brothers. I've been through a handful of frustration moments with Lumen; when I say "I can't do [fill here with a not-so-common problem] with Lumen version X.Y" they say "Use Laravel instead, it is really easy to upgrade". It should be for God sake! They share the same blood. I wasn't ask for that. If I were like to use Laravel I would choose it in the beginning and not use the Lumen in the first place. There were some reasons I did choose Lumen, like there were some reasons the author write it too (which I don't know why, but still...). Lumen should be a lighter weight, micro-framework alternative to Laravel, not a stepping stone.
Choosing Slim has it's disadvantages too but I think it's about the perception. I want to know what's happening on my application given the circumstances. Isn't reasoning about actually that? Even if I'm going to this rocky road at least I know at the end my application will do exactly as I commanded, nothing more nothing less.
Thank you for your time and please excuse me for formatting.
The fact that Silex didn't support PSR-7 (at the time of writing this) a huge let down. It has so many good point already mentioned above. There is a plugin/extension that let you do that, however I don't see the point in adding this overhead when you are looking for a lightweight framework
I have been developing a plain JSP/Servlet web application, which focuses mainly in collecting large sets of data through JSPs, processing them and finally commiting them to a MySQL database. Imagine something like a declaration of assets for a pretty big number of people.
While it works pretty good now, the code is really jumbled up and since I now have the time, I am thinking of completely redesigning the whole application in a more sophisticated and reusable way. My main problem is that forms are built mainly through jQuery (my form_build.js and form_validate.js files span over 600 lines each at the moment), and the back-end java code is jumbled up since there is massive use of name[counter] input fields.
I.e. The user can add 5 assets in the form, where he has to submit something like 30 fields of information on each one. The form sends them in the form of attribute[asset_counter].
If I managed to make myself clear, I have been looking into the Google web toolkit - which I had no idea of before - and from what I gather mastering it will take some time for me. Is it worth the trouble, or is it aimed at something completely different? And is there a framework or technique that can handle efficiently what I need?
Frankly, I am a senior java developer and I used a lot of jsp/jsf web applications accompanied by EJB, webservice, simple spring web apps, and I recently came up to use GWT.
It's a very nice platform, it has the following advantages:
All the code is compiled to html/JS which is better in terms speed
and browser understanding agilty
It uses built in JSON xml transer (nice approach)
It use built in AJAX (nice approach)
It use it's own cleint server platform (no need to use other
libraries for each one)
It will adopt to any browser (strong competetor)
drowbacks:
Sometimes it takes time to load in the browser (specially when using
FlexTables)
I advise you to use it!
GWT is a very powerful toolkit, and from what I understand it might help you to better organize you client side code. But you also have to consider some high-level JS framework like backbone.js+require.js or Google Closure. Time spent to learn this frameworks this bascially the same as for GWT, the choice basically depends on what your team prefer, JS or Java. In most cases when logic on client side gets very complicated, I will choose GWT.
I am looking into something lightweight, that, at a minimum should support the following features:
Support for easy definition of actions through metadata
Wrapper that extracts parameters from request into clojure map, or as function parameters
Support for multiple forms of authentication (basic, form, cookie)
basic authorization based of api method metadata
session object wrapped in clojure map
live coding from REPL (no need to restart server)
automatic serialization of return value to json and xml
have nice (pluggable) url parameter handling (eg /action/par1/par2 instead of /action?par1=val1&par2=val2)
I know it is relatively easy to roll my own micro-framework for each one of these options, but why reinvent the wheel if something like that already exists? Especially if it is:
Active project with rising number of contributors/users
Have at least basic documentation and tutorial online.
First of all, I think that you are unlikely to find a single shrinkwrapped solution to do all this in Clojure (except in the form of a Java library to be used through interop). What is becoming Clojure's standard Web stack comprises a number of libraries which people mix and match in all sorts of ways (since they happily tend to be perfectly compatible).1
Here's a list of some building blocks which you might find useful:
Ring -- Clojure's basic HTTP request handling library; all the other webby libraries (for writing routes &c.) that I know of are compatible with Ring. Ring is being actively developed, has a robust community, is very well-written and has a nice SPEC document detailing its design philosophy. This blog post provides a nice example of how it might be used (reacting to GitHub commits).
Sandbar -- currently an authentication library, more types of functionality planned; under development.
Compojure -- a mature and robust library which provides a nice DSL for writing routes to be used on top of Ring. This will give you the nice URL parameter handling.
Compojure-rest -- "a library for building RESTful applications on top of Compojure". Compojure-rest is, as far as I can tell, in its early stages of development; perhaps you might see this as an opportunity to influence its design. :-)
For dealing with XML, there's clojure.contrib.lazy-xml (and the helper library clojure.contrib.zip-filter.xml) and Enlive (the built-in clojure.xml namespace is currently not very usable); these would be used in tandem (though for your purposes the former might suffice).
For JSON, there a library in contrib and clojure-json (and I think there was at least one other lib I seem to be forgetting now...); pick the one you like best.
All of will be perfectly happy with a REPL-driven development style (see the accepted answer to this SO question for a Ring trick which is very much to the purpose here). I suppose the above collection of links does leave a few blind spots (in particular, the authentication story is still being ironed out, as far as I can tell), but hopefully it's a good start.
1The only single-package solution for building webapps in Clojure that I know of is Conjure, inspired by Rails; unfortunately I have to admit that I don't know much about it, so if you feel interested, follow the link and look around the sources, wiki &c.
While building my first Clojure rest service I found myself asking often the same question. The Clojure Toolbox helped me a lot: http://www.clojure-toolbox.com/
If you are looking for some sample, real-world, illustrative code to get you started, then you could study this clojure-news-feed on github project which demonstrates how to implement a non-trivial RESTful web service with compojure/ring that wraps both SQL (postgresql or mysql) and NoSQL (cassandra), search (solr), caching (redis), event logging (kafka), connection pooling (c3po), and real-time metrics via JMX.
This blog about Building a Scalable News Feed Web Service in Clojure provides a good introduction. I ran some load tests against this service on a humble AWS deployment and got about eighty transactions per second with less than a half second average latency per transaction.
Take a look at liberator library http://clojure-liberator.github.io/liberator/ It's noy a standalone solution, buy very good for rest service definition.
Just to provide an updated answer to this old question, currently (in 2018) I think Luminus provides an excellent starting point. It's using many of the libraries (ring, compojure, etc.) mentioned in previous answers, is modular and as close to "single package" as you can get with Clojure. Specifically for REST, take a look at compojure-api. Luminus recommends buddy for authentication, I've had good success using it both for traditional session-based auth as well as Oauth and stateless JWTs.
I've been trying to find a good form of RPC to standardize on, but so far I've ran into a ton of walls and was wondering what the stackoverflow communities view was.
My ideal RPC would provide the following:
Somewhat wide support in other languages, in that people shouldn't have to write a custom stack to use our server
Input validation
Ideally, some way to turn the above input validation into some sort of automated documentation to distribute
Clean and maintainable code
I am a fan of the catalyst framework and would prefer to stick to it, but if there is a clearly better alternative for RPC servers I'd be open to that as well.
So far I have looked at the following:
Catalyst::Controller::SOAP
Doesn't appear to support returning of complex data structures, only string('literals'). I could probably serialize data on top of that, but that seems very hacky. It also lets you return a pre-formed XML object, but I couldn't get that to work and it looks like you'd need to re-create a lot of SOAP data structure for it to work.
I do like the idea of WSDLs, but the complexity of the overall spec also makes me wonder how well support for communicating with other languages will be.
Custom POSTing XML based controller
We tried to roll our own by hand in a similar way to how we've seen two other projects do it recently where there is a dispatch url that you post XML to. This lets you have XSD validation/documentation, but required creating a lot more code than we want to maintain at this point.
Catalyst::Plugin::Server::XMLRPC
Gave a warning about using a deprecated method that will be removed in a future version of Catalyst.
No input validation or doc creation, but otherwise the best I've found
JSONRPC
Looks pretty similar to XMLRPC only the module is actually updated. I'll probably go with this next unless someone suggests something better
There also appears to be two different modules for catalyst that do JSONRPC
I realize that REST isn't pure RPC (only a subset), but...
I would recommend the Catalyst::Controller::REST and Catalyst::Action::REST modules. They're frequently updated and the documentation is fairly good. There is also a good (but rather dated) example of using Catalyst::Controller::Rest in the 2006 Catalyst Advent calendar titled Day 9 - Web Services with Catalyst::Action::REST.
FWIW, Catalyst::Controller::SOAP does support returning complex data. Take a look at the documentation http://search.cpan.org/~druoso/Catalyst-Controller-SOAP-1.23/lib/Catalyst/Controller/SOAP.pm, which will show you that you can use a WSDL to describe your service. Also, see http://daniel.ruoso.com/categoria/perl/soap-today.html.en for a more detailed step-by-step process.