Related
I am trying to find a high-level Clojure library for making HTTP and HTTPS requests, parsing out forms and links from responses and then POST-ing updated forms or following links. Ideally something that would automatically handle redirects and cookies (i.e. sessions). That is, I'd like to find something whereby my code can as closely as possible mimic a user driving a webapp from a browser, without the browser.
A number of years ago we used Hpricot and Ruby for a similar task but I'm prefer to do this in Clojure if at all possible. From memory - and I haven't used Hpricot for years - we were able to do all this with minimal effort: we were able to concentrate on the 'what' of driving the application, not the 'how'.
I found clj-http https://github.com/dakrone/clj-http but this seems to be one step lower-level than I'm looking for (no form parsing) - although it is based on Apache HttpComponents http://hc.apache.org/httpcomponents-client-ga/ which does seem to expose a nice, fluent, API for forms http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html.
Screen scraping in clojure asks about screen-scraping in Clojure, and there are several good suggestions for that, but nothing that really addresses the above.
HTTP Kit http://www.http-kit.org/client.html looks like it would be a great foundation for the above but doesn't do form parsing or session management (as far as I can see).
Currently I've veering toward using the Apache HttpComponents Java library directly from Clojure. Can anyone suggest any better - perhaps more Clojure idiomatic - alternative? Or anything that they found worked well in similar circumstances? My goal is to write the minimal amount of code quickly to investigate a problem with a web service. This is not production code. Saving time, rather than getting an 'ideal' solution is my main concern.
[The background is that I am trying to mimic certain forms of user behaviour in order to first reproduce and then try and track down an intermitent bug in a large body of legacy Java/EJB code. However the problem only seems to occur one time per several thousand POSTs. (The suspicious is of some form of caching issue.) The existence of the problem, after the fact, is easy to detect however.]
Have you looked at the Enlive library yet? Here is a good tutorial on it.
You seem to really have 2 parts here. The first part is (1) a Selenium-like client, which drives (2) a webserver.
For part (1), either Selenium, Enlive, or something similar will allow you to simulate a browser to submit data, read the responses, and respond from there. For part (2), it seems you just need a regular Clojure web framework such as Ring/Compojure (older & simpler) or Pedestal (newer & more powerful).
There are must be a lot of apps that are designed to communicate with server. My question is only about App installed on iOS device + Server side service interaction. Web app is not what I am talking about, and there should be no webpage involved in this discussion at all. Typical examples are Apps like Instagram and Twitter, in which most of the information exchanged between the App and the server is just data like String, Image and Integers(wrapped in JSON or XML), no webpage presentation needed.
My question will be: if you are an independent app developer, and you are designing such an app from scratch without any existing website API, database structure or application(so you are not limited by any existing API or database structure or application protocol), What will be the most efficient approach?
What the sever side need to do are:
receive data send by the App;
process the data with designed logic;
interact with database(like MySQL);
do necessary data mining and analysis---this could be a constantly running service or one time task requested by the App client;
send the data back to the App upon request or spontaneously;
exchange or broadcast the data between/among different App clients (i.e.: group chatroom and peer to peer message);
As far I as know there are 3 obvious options to implement the server side:
PHP
Python
Ruby on Rails
(please feel free to add more options)
My questions are:
which one is the most appropriate choice to implement the server side?
If the App is focusing intensively on natural human language/text searching, analyzing and data mining, which one is the best choice? I heard Python is doing pretty good in this area.
Any advice on the database choices? I am using MySQL for now, and I found it's quite powerful for my purposes, I heard Twitter is switching to Cassandra. Will that be too difficult to start with?
For the server end, if you need to build a Server management interface, for you as an admin to manage and monitor the community, membership, data and such, is there any existing solution, or framework or tool for that? what will be the most efficient approach?
If a new programmer has no experience in non of them, which one you suggest he/she to start with?
Is there any good reference material or sample code on the server side in such context we can learn from?
I know there are a lot of very experienced experts on these areas on Stackoverflow, but I saw more newbies who just entered the iOS developing area without much knowledge in server/database programming experience. And I hope this thread can help these who are thinking to design an App with server/client structure but have no idea where to start with.
ps: I will keep updating this question thread and adding my findings on this topic, to help all other users at stackoverflow. :-) Please try to make your answer informative, easy to understand, and constructive. I guess most of readers for this thread will be new members of this great community.
Are you sure you want to spend time & money to develop your own Server & develop your own API?
There are lots of mBaaS (mobile Backend as a Service) providers today such QuickBlox, Parse,StackMob, which are ready to use and they have great Custom Objects API and some of predefined modules. They have great free plans with big quota. Some of them such QuickBlox has Enterprise plan - so you can buy license and they server team update server for you purpose.
So, i recommend not develop your server and think about mBaaS market.
Just about your issue - I can recommend look at QuickBlox Custom Objects code sample and also Custom Objects API. Custom Objects module provides flexibility to define any data structure(schema) you need. Schema is defined in Administration Panel. The schema is called Class and contains field names and their type. I think it's what you need.
which one is the most appropriate choice to implement the server
side?
Well that depends on what you know, there is reason to choice one of the other
If the App is focusing intensively on natural human language/text
searching, analyzing and data mining, which one is the best choice? I
heard Python is doing pretty good in this area.
This would reflect on your first question, you pick the language on you needs. Thus if python makes it easier then pick that one.
Any advice on the database choices? I am using MySQL for now, and I
found it's quite powerful for my purposes, I heard Twitter is
switching to Cassandra. Will that be too difficult to start with?
Again not one that is easy to answer, since it all has to do with requirements. But any SQL server will do. Cassandra is meant for "scalability and high availability without compromising performance" accourding to there website. Do you think you webservice will get many request then it might be a choice to consider.
For the server end, if you need to build a Server management
interface, for you as an admin to manage and monitor the community,
membership, data and such, is there any existing solution, or
framework or tool for that? what will be the most efficient approach?
This again is only going to be answered when you pick the a SQL server and server language.
If a new programmer has no experience in non of them, which one you
suggest he/she to start with?
Start with something simpler, you are really going out on a limb here.
Is there any good reference material or sample code on the server
side in such context we can learn from?
Propably there is some, but you should really start small and work from there.
Twitter started out as a Rub on Rails app and is working on scalability and availability which ruby is not really good ar (that is my person opinion). or Look at facebook they have written a php to c compiler to make php run faster.
The only thing I can say to start code, when you app does take off then tackle the some of the performance issues.
And since you state that you are new to programming do not bite of more then you can chew.
This is a huge question and I don't think there is a best answer. It most depends on what you care about, such as how quickly the development process, how easily the implementation, etc.
And which one is popular, which one is cool, I don't think it make really sense.
In my personal opinion, I'm good at ASP.NET and I can get Windows server easily, so I'll start with an ASP.NET service to provides data.
And, to be continued.
I am looking into something lightweight, that, at a minimum should support the following features:
Support for easy definition of actions through metadata
Wrapper that extracts parameters from request into clojure map, or as function parameters
Support for multiple forms of authentication (basic, form, cookie)
basic authorization based of api method metadata
session object wrapped in clojure map
live coding from REPL (no need to restart server)
automatic serialization of return value to json and xml
have nice (pluggable) url parameter handling (eg /action/par1/par2 instead of /action?par1=val1&par2=val2)
I know it is relatively easy to roll my own micro-framework for each one of these options, but why reinvent the wheel if something like that already exists? Especially if it is:
Active project with rising number of contributors/users
Have at least basic documentation and tutorial online.
First of all, I think that you are unlikely to find a single shrinkwrapped solution to do all this in Clojure (except in the form of a Java library to be used through interop). What is becoming Clojure's standard Web stack comprises a number of libraries which people mix and match in all sorts of ways (since they happily tend to be perfectly compatible).1
Here's a list of some building blocks which you might find useful:
Ring -- Clojure's basic HTTP request handling library; all the other webby libraries (for writing routes &c.) that I know of are compatible with Ring. Ring is being actively developed, has a robust community, is very well-written and has a nice SPEC document detailing its design philosophy. This blog post provides a nice example of how it might be used (reacting to GitHub commits).
Sandbar -- currently an authentication library, more types of functionality planned; under development.
Compojure -- a mature and robust library which provides a nice DSL for writing routes to be used on top of Ring. This will give you the nice URL parameter handling.
Compojure-rest -- "a library for building RESTful applications on top of Compojure". Compojure-rest is, as far as I can tell, in its early stages of development; perhaps you might see this as an opportunity to influence its design. :-)
For dealing with XML, there's clojure.contrib.lazy-xml (and the helper library clojure.contrib.zip-filter.xml) and Enlive (the built-in clojure.xml namespace is currently not very usable); these would be used in tandem (though for your purposes the former might suffice).
For JSON, there a library in contrib and clojure-json (and I think there was at least one other lib I seem to be forgetting now...); pick the one you like best.
All of will be perfectly happy with a REPL-driven development style (see the accepted answer to this SO question for a Ring trick which is very much to the purpose here). I suppose the above collection of links does leave a few blind spots (in particular, the authentication story is still being ironed out, as far as I can tell), but hopefully it's a good start.
1The only single-package solution for building webapps in Clojure that I know of is Conjure, inspired by Rails; unfortunately I have to admit that I don't know much about it, so if you feel interested, follow the link and look around the sources, wiki &c.
While building my first Clojure rest service I found myself asking often the same question. The Clojure Toolbox helped me a lot: http://www.clojure-toolbox.com/
If you are looking for some sample, real-world, illustrative code to get you started, then you could study this clojure-news-feed on github project which demonstrates how to implement a non-trivial RESTful web service with compojure/ring that wraps both SQL (postgresql or mysql) and NoSQL (cassandra), search (solr), caching (redis), event logging (kafka), connection pooling (c3po), and real-time metrics via JMX.
This blog about Building a Scalable News Feed Web Service in Clojure provides a good introduction. I ran some load tests against this service on a humble AWS deployment and got about eighty transactions per second with less than a half second average latency per transaction.
Take a look at liberator library http://clojure-liberator.github.io/liberator/ It's noy a standalone solution, buy very good for rest service definition.
Just to provide an updated answer to this old question, currently (in 2018) I think Luminus provides an excellent starting point. It's using many of the libraries (ring, compojure, etc.) mentioned in previous answers, is modular and as close to "single package" as you can get with Clojure. Specifically for REST, take a look at compojure-api. Luminus recommends buddy for authentication, I've had good success using it both for traditional session-based auth as well as Oauth and stateless JWTs.
I was today trying to figure out on working with WebService and found many articles really gospel over the Web Service and its effectiveness in the Market share.
My Questions are:
For a Complex project of critical data, is it better to opt for WebService?
What Makes WebService different from other way of fetching the data?
The answer is... it depends. Web services are not really the next Big thing, they have been a Huge Thing for years now. In business applications, web services allow a big level of interoperability and capabilities never seen before.
They help integration with legacy systems, cooperation between distinct departments, defining loosely coupled interfaces and such. You should read some about Service-oriented architecture.
If all you need is a PHP application that handles data from a single database, you might not need web services at all. If you are designing a solution that revolves around multiple data sources, with complex security involved, multiple languages and/or multiple applications, then web services become essential.
SOAP is a protocol; if working with PHP, you'll need to check out the PHP: SOAP guide to understand how it works. For every language (almost), there are existing APIs to develop web services. Anyhow you might want to check RESTful web services instead of SOAP-based ones, they are generally simpler to implement/understand. But that's another debate ;-).
Cheers.
That mostly depends on the definition of "big thing".
My experience with the WS stack and SOAP and all the acronym soup is that it takes an awful lot of workforce to deploy it. The status of the frameworks is complex, and definitely not something a hobbyist can put to work in a couple of afternoons. We have seen how many things on the net became the next big thing just because they were easy. Easy to understand, easy to interact with, easy in technology. Wikipedia, twitter, digg, youtube are internet big things, and they are, from the interaction point of view, light years away from SOAP/WS based interaction. They are KISS: simple and stupid. A whole horizontal market was opened just because of their simplicity. Even multiprocessing platforms like BOINC don't use anything near the WS stack, but they are the core of many high-throughput efforts.
Now, if you have to deal with complex multi-host transactions, authentication, credential delegation, caching... WS is there. It's the target that makes the need: banks, flight reservation, stuff like this. but they won't impact the common programmer. They require too much energy and too many different competences at once to become something usable for a horizontal market of developers.
Also, I am a REST person. I never advocated SOAP with much emphasis, but there was nothing else and it was a better evolution over XMLRPC (which, if you have to perform dumb RPC, IMHO it's still a good choice). Now I changed my mind. You mostly have resources on the web, and you interact with them with HTTP methods. SOAP is nothing but RPC on hypersteroids. No, REST is not the solution that replaces WS. At all. it's simply easier to use and to debug, albeit more difficult to design (you have to think in terms of resources instead of method calls). It's KISS. That's why it has more chances for success on the horizontal market.
It depends.
Web services can be useful if you need to expose the data across security boundaries, where a direct connection to an RDBMS would be a bad idea.
Popular method for implementing web services nowdays is to use RESTful API (eg. via Ajax/JSON). It's already "next big thing" – almost every major player has been offering it for years. Google, Flickr, Twitter, you name it.
The big advantage is that they help to implement an API layer.
If you implement your solution using a "bus" where the web services sit, it opens up your product to a far greater range of users and moves away from being a proprietary product.
It also enables people to interface using a wide range of solutions e.g web service clients can be implemented using command line, Jsp, Java, Asp, .NET, PHP etc.
They also enable code re-use e.g. if you implement GetClientDetails (ID) as a web service for one user, when the next group comes along wanting the same thing, all you have to do is give them the WSDL and they are away.
I realize that this may be subjective but I truly need an answer to this and I can't seem to find anything close enough to it in the rest of the Forum. I have read some folks say that the framework (any MVC framework) can obscure too many things while others say that it can promote good practices. I realize that frameworks are great for a certain level of programmer but what about individuals starting out? Should one just focus on the language or learn them together?
I think web development is way more than anyone grasps when they first start getting into it! Read this and know that it is all optional...but required to be really good at what you do.
I suggest that you spend time learning your language first. I would suggest learning C# simply because it is vastly more marketable and it is usually directly supported in most of MS products. By learning C# - programming in ASP.NET, console apps, servers, services, desktop apps, etc. will all be within your reach. You can program for most of the MS products as well as on many Linux type platforms.
Once you have this down then you can move to programming for the web as programming for the web has some intricacies that most other environments don't have. Concepts such as sessions, caching, state management, cross site scripting, styling, client side vs server side programming, browser support, how HTTP works, get vs post, how a form works, cookies, etc. are all at the top of the list of things to learn separately not to mention learning the ASP.NET base frameworks and namespaces.
Once you have the programming language down and then the concepts of web programming I suggest that you pause and learn database design. Don't worry about performance just yet...try to first learn good design. Performance will come next. A good start for you is Access (blasphemy I know). It is easy for a beginner to work with. And it translates into a more robust platform such as SQL Server easily. Learn at the very least some SQL...but I suggest that you learn as much as your stomach can handle. I heard someone say that SQL is like the assembly language of the database. The number one thing that slows an application to a halt is piss poor database design and poor queries. Once you have this knowledge - stuff it away in the back of your mind and take a look at a good ORM. NHybernate is probably best at the moment but is more complex that the basic learner needs. For that reason I currently suggest getting LINQ to SQL up and running as it is SUPER EASY to work with. Then look at Entity Framework (although I still think it sucks...and you should wait till EF 2.0...ERRRRR...now 4.0 released with .net 4.0). Then NHybernate.
Now is the time to start to understand the infrastructure that is required by web development. You may bump your head against this as you learn some of the web programming stuff. But you need to understand the basics of DNS, IIS, load balancers, sticky routing, round robin, clustering, fault tolerance, server hardware setup, web farms, cache farms (MemCached Win32, Velocity), SMTP, MSMQ, database mail queuing, etc. Many people may say you don't need this. That there will be some knowledgeable network admin to help you out here. However they generally know things that impact them...not you. The more you know here the more valuable you will be to the company that hires you.
Now you can get into the details of best practices and design patterns. Learn about the basics such as repository pattern, factory pattern, facade pattern, model view presenter pattern, model view controller pattern, observer pattern, and various other things. Follow Martin Fowler and others for suggestions here. Take a look at concepts such as inversion of control, dependency injection, SOLID principle, DRY, FIT, test driven design, and domain driven design, etc. Learn as much as you can here before moving to the next step.
NOW you can think about frameworks! Start by creating a basic application with ASP Classic (comes with IIS for free!). This will give you a flavor of a no frills web development environment. Take a look at ASP.NET web forms (briefly) to see how MS attempted to make things easier by hiding all the complex stuff (which you now know how to manage on your own from your readings of the above materials!!!). Now you no longer need ASP.NET Web Forms. Move immediately to ASP.NET MVC. The MVC framwork gives you all the power you need to create a good easily manageable web application. If you build something really big no framework for pure web development may be able to deal with what you need. However MVC is way more extensible for such UBER custom scenarios.
Now that you have made it through the journey to ASP.NET MVC you can take a look at things such as Microsofts Enterprise Application Blocks (such as they use at MySpace). Take a look at Elmah error logging (a must have). Look at how to build a custom SiteMapProvider for your MVC site. If you need to get into searching stuff understand Lucene.NET.
And if you made it this far...you are ready to figure out the rest on your own as it comes up! Have fun. There is a lot of room in this space for a person with some understanding of all of the above concepts.
You'll be using SOME sort of framework. The question is, what level do you want to learn at?
You'll probably not care to learn about asynchronous I/O and mutlithreaded vs. select/poll styles of web servers.
So then, your language of choice is going to provide a layer atop this, the languages preferred "web interface" API. For Java it's Servlets, the lowest level you'd typically code at for server side web applications.
You should find what this "lower level" layer is in your language and learn the API at least. You should know basic HTTP like status codes, cookies, redirects, POST vs GET, URL encoding, and possibly what some of the more important headers do.
You'll then come to appreciate what these higher level frameworks bring to the table, and be better able to evaluate what is the appropriate level of abstraction for your needs/project.
Web development requires a certain degree of organization, since it relies so much on separation of concerns. The browser, for example, is designed to display data and interact with the user. It is not designed to lookup data from a database, or perform analysis. Consequently, a web development framework can help provide services that are needed to make the browser experience a practical one.
The nice thing about employing a platform is that it will provide core components essential to the making of any web application that you won't (and shouldn't) have to think about, such as user membership, for example. Many of the design decisions and deep thinking about how to implement these services has already been done for you, freeing you to focus on what you actually want you application to do.
Of the available frameworks, I find that frameworks that implement the MVC (model-view-controller) pattern are very practical. They clearly organize different functions of web development, while giving you full control over the markup presented to the browser.
All that said, you will need some fundamental skills to fully realize web development, such as HTML, CSS, and a core programming language for the actual underlying program, whether you use a platform or not.
I don't think I agree with the Andrew. I don't think learning C is a pre requisite for web development. In fact, learning something like Javascript, Action-script or PHP is often easier due in large part to the vast numbers of sites and tutorials available, and are enough to expose you to the fundamentals of pretty much every programing language. Variable, Conditions, Loops and OOP. I just think learning C# introduces a lot of learning that isn't really relevant to web development such as pointers and memory management.
As for wether you should learn a framework first? Definitely not. Never ever. You need to be able to stand on your own two feet first and be comfortable with HTML/CSS, Server Side Scripting (PHP/ASP/Python/Ruby whatever) and love it or loathe it, but you're going to have to have a decent understanding of Flash and Action-script.
The order in which you learn these is entirely up to you. But my learning plan would go like this...
Start with HTML. It takes about half an hour to get the basics (it's made up of tags with attributes, end of lesson 1) and it's good to get it out of the way first.
Then start leaning CSS. You'll get the basics again, very quickly. But CSS is a minefield so expect to spend the rest of your life figuring it out.
Next up Action-script. Most people wouldn't agree with me, but bear with me. HTML and CSS aren't programming languages. Action-script is. And learning a programing language for the first time is difficult and tedious. The advantage Action-script has over most other languages is that the results are very visual. It's enjoyable to work with and you can sit back and take pride in your accomplishments at regular intervals. This isn't possible with server-side scripting languages or Javascript and there's a whole host of stuff you need to learn to get server side scripting up and running. You can't build space invaders in with PHP for example.
I've changed my thinking here. I would encourage beginners to ignore ActionScript and focus on Javascript. I still believe that being able to see stuff on screen quickly is a good motivator, but I would encourage people to look at canvas tag tutorials and frameworks. Javascript has come a long way since 2009, and is now the lingua franca of programming, so it's incredibly useful. My initial point about HTML and CSS not being programming languages still stands.
Then, you can start with your server side language. At the same time, you're going to have to figure out the database stuff. I recommend PHP and MySQL because it's free.
Again, I've changed my thinking here. I would encourage beginners to use Javascript on the backend (Node.js), and split their database learning between relational databases and noSql solutions such as Mongo.
Then.... learn your framework. Or better yet, roll your own. That's what I've been doing and it's supercharged my learning.
If you're getting into web development, You HAVE to know how those building blocks work. You don't have to be an expert in all the areas, but you should try to become an expert in at least one of them. If you start learning a framework before you get the fundamentals you'll be in a sticky middle ground where you don't understand why things don't work which will infuriate you, and anyone who has to work with you.
you should learn how to use framework because it would be helpfull for u in the future also it is easier to learn.
MVC will help you a lot .. trust me ... i was developing web project not using mvc and it is like mess ... (in the past there are no well know mvc and i never heard about it)
Short version: yes, and then some.
FWIW : This more generic answer may be of use to someone out there.
What: Frameworks take out tedium of using boiler-plate code again and again. They hide complexity and design issues under wizards and conventions. They also use special libraries, design patterns etc. in ways that are far from obvious to a beginner.
So using a framework is good for getting things done without knowing exactly how - like using an ATM without knowing the internals. You just add your code bits in certain places and things 'just work'.
HTML > CSS > Ruby > SQL > Rails/Javascript framework > Libraries would make for a good learning track. Rest you learn as you go along by being curious, hanging out on forums or as extended learning as need arises.
HOW: The problem starts the minute you step outside simple text-book examples (i.e. when you try to get it to do something even a bit different).
Decoding cryptic error messages when it seems like you've done everything right but things still don't work. Searching on error strings in forums may help out. Or just re-starting from scratch.
Reading up articles and books, videos, trial-and-error, hard-work, search-engines, stackoverflow/forums, local gurus, design articles, using libraries, source-code browsing are a good way to climb the learning curve gently and on a requirement basis.
Working-against-the-framework is the number one problem for beginners. Understanding what the framework expects is key to avoiding white-hair in this phase. Having enough insight to manually do what the framework automates may help reduce this second-guessing effort.
WHY: For more advanced debugging/design, it's good to know what the framework is doing under the hood esp. when things don't work as you planned. Initially you can take the help of local-gurus or forum gurus who've already done the hard work. Later as you go deeper you can take on more of that role. For example there's a "rebuilding rails" book which looks under the hood of Ruby on Rails.
Note: Some of the tips are oriented towards Ruby/Rails but you can easily substitute your favourite language/framework instead.