Webautomation and proxies - webautomation

very new to the idea of using web automation with rotating proxies and headers here. I have some basic questions that I can't seem to totally have the answers confirmed from the docs.. Here are some of the questions I most want the answers to for now, please excuse if they are too basic:)
First of all, if you're using web automation for example with Selenium or Playwright, with proxies will it mean that the Selenium or Playwright scripts will be sent to the proxy servers to be used and fetch the data? Thus far, I have only been using paid proxy services that use python requests and do not offer web automation, the problem here is that sometime requests don't return full diplayed data.
Second question would be about TorPy a tool I recently found. As I understand from the manuals it is a layered process in fetching data via multiple free proxies for each request, would this be correct assumption? Many thanks.

Related

How to provide a REST API into 3rd Party data?

I use OmniFocus a ton and I'd really like to be able to connect my data there to other things (Zapier, IFFFT, Beeminder, etc). There's a lot of support for putting data into OmniFocus through these services, but I can't find any support for getting data out of OmniFocus.
In thinking about this, I realized my question isn't really about OmniFocus but rather about building a connector to a service that I don't own. So this is my scenario:
I have data on some publicly accessible web service (in the case of OF, it's Dropbox)
I want to build and host some sort of application that accesses that data and parses it and then provides a REST API that other servers can then query.
Ideally I'd like to make this service available to others - this seems tricky because they have to somehow enable my application to read their data.
I'm a fairly experienced software dev but I have zero experience with web applications or cloud applications. I'm not looking for a super in-depth answer here, but more of a general sketch of how this would work (or a confirmation that this really isn't feasible).

Is Meteor an option, if i need an additional REST API?

I'm, going to write a web app, which should be CRUD accessible from both, the web and native mobile device apps. For the latter i'm definitely committed to a REST API. Is it possible to realize that with Meteor.com ? Would it be an option to use Meteor for just the web and a second REST interface to directly talk to the mongo? Since the meteor client listens for changes in the mongodb this should not cause conflicts, does it?
As of 2015, look at Gadi's answer for the Meteorpedia entry on REST APIs, and at krose's answer comparing REST API packages. Discussion for folding REST APIs into core is on Hackpad. This question is a duplicate of How to expose a RESTful service with Meteor, which has much better answers. -- Dan Dascalescu
Old answer (2012) below.
For adding RESTful methods on top of your data, look into the Collection API written for Meteor:
https://github.com/crazytoad/meteor-collectionapi
As for authentication for accessing the database, take a look at this project:
https://github.com/meteor/meteor/wiki/Getting-started-with-Auth
Both are definitely infantile in development, but you can create a RESTful API and integrate it with a mobile native client pretty easily.
There are a lot of duplicates of this question. I did a full write-on on this in Meteorpedia which I believe covers all issues:
http://www.meteorpedia.com/read/REST_API
The post reviews all 6 options for creating REST interfaces, from highest level (e.g. smart packages that handle everything for you) to lowest level (e.g. writing your own connectHandler).
Additionally the post covers when using a REST interface is the right or wrong thing to do in Meteor, references Meteor REST testing tools, and explains common pitfalls like CORS security issues.
If you are planning to develop a production application, then Meteor is not an option right now. Its under constant change, and there are still many common features it has to support before its ready to use, which will be quite some time.
For your Question, Somebody has already asked and answered the question about support for file uploading in meteor(also contains HTTP handing related information).
How would one handle a file upload with Meteor?

RESTful web services and delta updates

I'm writing client code (desktop/mobile apps) that interact with RESTful web services for a while. I wonder that such services don't allow you to get delta updates.
I currently write an app that notifies you about new issues added to your Redmine. So I need to download all issues again and then compare them with that I downloaded before. That's very bad solution sir, since there may be dozens of issues.
I'd like to know why RESTful web services don't give you an option to download delta updates. Does it contradict the basic idea of RESTful? Or probably the solution is too obvious to document it?
Too domain-specific to document. Any RESTful application would not find it hard to add new resources that you can GET to see deltas, if only they knew that's what their clients wanted. Have you asked the Redmine group for this feature?

hosted rest load testing solution

can anyone recommend a hosted solution for load testing a rest interface (including all verbs, signed requests and multiple sets of simulated tasks)?
we don't want to set up a server with the tools and we only need this occasionally.
we'd like to test it at about 500 requests per second and see how the system responds.
would be awsome if anyone could share their experience with hosted solutions providing this service.
thanks!
Not a hosted solution but a quick and easy way to load test a HTTP endpoint is to use Fiddler and a plugin called Stress Stimulus http://stresstimulus.stimulustechnology.com/ It's free too!

Alternative to building a proper web service for iPhone app to consume

I am in the process of scoping the development of an iPhone app for a client. Among other things, the app will allow users to browse through and place orders on specific (tangible) products.
The client has a website that currently does a similar thing and due to their limited budget and the fact that the website runs on a third-party proprietary platform which they have no control over, we are investigating possible alternatives to building a web service.
On the website, user registration and authentication, as well as order placing is done through POST requests via secure HTTP. The response is always a formatted HTML page which will contain strings indicating whether the request was successful or not, and if there was an error, what the error is etc.
So provided I can replicate the POST requests on the phone, and parse the HTML responses to read the results of each request, do you think this is an acceptable alternative to building a web service to handle this?
Apart from the possibility of pages changing (which we can manage) and the fact that I will probably have to download and parse a relatively large HTML response, are there any other drawbacks to this solution and is there anything else that I might be missing?
Many thanks in advance for your thoughts.
Cheers,
Rog
You could create an intermediary server that will communicate with the client server, and on it expose some REST web services with json (small overhead and easy to handle) responses that will be consumed by the iPhone app.
So, you're going to parse HTML and formulate POSTs off a third-party server, and pray that they don't even so much as rename a form field.
Your question is in two parts:
Do I think that a miracle is an acceptable solution? I don't.
Do I think that aside from the fact a miracle is required, are there any other drawbacks? None that I can think of.
You didn't ask, but this is a terrible course of action. Two suggestions.
I spy an assumption that the providers of the third-party platform aren't interested in enabling third-party applications by providing an API. They have a very good business reason for this, which is that it promotes platform lock-in. Reach out to their support department and have a talk with them.
You have to sell the client on building an intermediary web service. To at least try to mitigate the damage that changes on this third-party platform can do to your app, I recommend that you build and operate a proxy that receives requests from your applications, and proxies them over to the third-party platform. You should build into this client-server protocol a means for returning "we are in maintenance mode, go away" messages to apps, for that inevitable day when the third-party server changes something that breaks your app (they swapped the billing and shipping address pages, for instance) and you have to rush through an update through Apple to deal with it.
The proxy could be written in something more flexible and easy to bash stuff out in, such as PHP, Python, Perl, or Ruby. It could be hosted at Amazon in a micro instance.
p.s. This question is inappropriately tagged objective C.
HTML is the worst because of parsing (1-2secs per page), memory, and changes, but you already know that. Check in advance that ALL the data you need is exposed on the HTML.
If you use an intermediary server you are moving work elsewhere and you have another server to maintain. I would only do that if memory is an issue. Check How To Choose The Best XML Parser for Your iPhone Project for memory/performance/xpath support. libxml2 is a good option, but it depends on your needs. And maybe you'll want to check ASIHTTPRequest features before using the SDK.
I think utilising the web language of JSON would contribute to the diminishing of the parsing time. By building a REST service that, when sent a GET request, returns the correct information for easy sorting, you could then display the output a lot faster than that of parsing straight HTML.
I prefer JSON over XML, but everyone has their personal preference. You should look at a few very good libraries that are built specifically for parsing purposes of both XML and JSON.
For XML I recommend using the inbuilt libxml parser. Albeit, this can sometimes deem very difficult to use. A simple Google search will bring up a heap of results that relate specifically to what parser should be used depending on what task is to be completed.
As for a JSON parser, I recommend SBJSON. I am currently using it one of the biggest projects I have undertaken and it is definitely working perfectly for my use.
If you need a good way to connect to a RESTful web service, you should try LRResty.
Don't go for a parsing solution on the iPhone for 4 reasons:
Server can change their design and break your application (AppStore submition is long) + They can also detect that the request are sent from an application based on user agent which you have to update the application to change it.
Some of the requests might be made thru Javascript so you not only have to parse (X)HTML but also Javascript request (which can be in the form of XMLHttpRequest, but don't have to)
Long term evolution of the mobile market : maybe your client want (or will want) an application for android, Blackberry, Bada OS (Samsung), Symbian (Nokia/ OVIStore), Java Mobile or Windows Phone 7?
Of course network traffic, Memory and CPU needed to parse HTML (look the time it takes to the browser to do it?)
Regarding the traffic, if the application will not have a huge traffic you can home-host your proxy. Or you can find some provider to host it for you. I guess you won't need more than a couple of Megabytes of storage but maybe traffic. For less than 100€/year you can find some with unlimited traffic (like OVH Pro plan or Infomaniak). But if you want to go Java have a look at Google App Engine : you pay only if your traffic is important and if your application generate many CPU Cycles. If not : you don't have to pay. And it's hosted on Google server : reliable.
If the client is open, you could consider the paypal API.