Could someone provide ideas how HTTP::Proxy module is compared to others proxies like paros and burp proxy and if someone use it during his work specifically if it used by the pen testing community for real job
HTTP::Proxy is a perl module that you can use to build your own intercepting (and other) proxies. Since it has the entire Perl language at its disposal, it can be very powerful, but it requires you to know what you're doing and develop a lot of code.
The other proxies you mention (Burp, Paros, WebScarab, etc.) have the advantage of being built for the specific purpose of intercepting and tampering with data flows in HTTP sessions. Many of them can man-in-the-middle SSL/TLS sessions, which HTTP::Proxy won't do out of the box. They are by far the preferable tools for pentesting work.
Though a Perl program using HTTP::Proxy could be very extensible, several of the pre-built proxies offer scripting capabilities. WebScarab, for instance, allows scripting with BeanShell.
In short, these are really different tools for different purposes.
Related
What I want to do: I want to add communication capabilities to a couple of applications (soon to be jar libraries for Java) in Scala, and I want to do it in the most painless way, with no Tomcat, wars, paths for GET requests, RPC servers, etc.
What I have done: I've been checking a number of libraries, like Jetty, JAX-RS, Jackson, etc. But then I see the examples and they usually involve many different folders for configuration, WSDL files, etc. Most of the examples lack a main method and I don't have a clear picture about how many additional requirements may they have (e.g. Tomcat).
What I am planning to do: I'm considering to simply open a socket on the "server" to listen, then connect with the "client" and transfer some JSON, in both directions. This should be fairly standard so that I can use other programming languages in compatible ways (e.g. Python).
What I am asking: I would like to know whether there is some library that makes this easier. Not necessarily using raw sockets, but setting up some process communication in just a few lines, maybe not as simple as Node.js, but something similar.
Bonus: It would be cool to
be able to use other programming languages (e.g. Python) by using open standards
have authentication
But I don't really need any of those at this point.
I think you need RPC client/server system, I would suggest to take one of these two:
Finagle - super flexible and powerful RPC client/server from Finagle. You can define your service with Thrift, and it will generate stubs for client/server in scala. With Thrift it should be straightforward to add Python support.
Spray - much smaller library, focused on creating REST services. It's not so powerful as Finagle, however much easier. And REST allows you to use any other clients
Remotely - an elegant RPC system for reasonable people. Interesting and very promising project, however maybe difficult to start with because of extensive Scalaz+Shapeless+Macro usage
Honestly if you want something that is cross-language compatible, simple, straightforward, and concise then you do not want to use plain old sockets!
Check out dropwizard. It is amazing and I use it for small and large projects alike! It is usually configured by no more than a single configuration file. It supports authentication too!
Out of the box it gives you really great inter-process communication over JSON (using Jackson) and much much more. There is also pretty decent Scala support for dropwizard.
If you must roll your own then I'd recommend using Jackson for JSON parsing. It's super simple to use and also has great scala support.
If you've got a "controlled" use case where the client and server are on the same LAN and deployed in tandem, I'd (controversially) recommend Java RMI; it's dumb and JVM-specific (and uses a Java-specific protocol), but it's very simple to use.
If you need something more robust and cross-language, I'd recommend Apache Thrift. You write your interfaces in a platform-independent interface definition language, and it's very clear which changes are compatible and which are not; the thrift compiler generates skeleton interfaces for you to use, and then you just write an implementation of that interface and a couple of lines to start the server (as you can see from the example on the homepage). It's also got good support for async implementations if you need the performance. Thrift itself is reasonably standard and cross-platform, with its own binary protocol, or you can use JSON as a transport if you really want to (I'd recommend against that though).
RabbitMQ provides one easy way to do what you want without writing a server and implementing your own persistence, flow control, authentication, etc. You can brew or apt-get install it.
You start up a broker daemon process (i.e. manages message queues)
In the Scala producer, you can use Maven-provided Java API to send JSON strings without any fuss (e.g. no definition languages) to specified queues
Then in your other Scala program, connect to the broker, and listen for messages on the queue, and parse the incoming JSON
Because it is so popular, there are many tutorials online for different patterns you may want to use to distribute the messages, e.g. pub/sub, one-to-one, exactly-once delivery, etc.
I've built a set of tools I use in my day-to-day work and I would like to make them look a bit more "professional" in order to sell them to financial institutions.
At the moment these tools are written in Perl and are executed from a DOS command line, it's extremely efficient but it doesn't look very attractive.
So, I would like to add a user interface to it but I don't really know what to use for language knowning that :
A Perl CGI interface hosted on the web is not an option since the information to be given as input is quite sensitive.
It would be ideal to sell it as a package/executable.
I don't really like the Perl/Tk interface.
I'm ok with rewritting the application in another language but I would prefere to reuse the main modules in Perl since it's very powerful with regular expressions and lists/arrays.
What would you advise me to do ?
Thanks,
Lory
If you want a non-web-based GUI, and don't like Tk, there's also Wx, which is a wrapper for the wxWidgets GUI toolkit.
However, web applications nowadays can be really easy to create (using a modern framework). Take a Mojolicious application, for example: Mojolicious carries no dependencies other than Perl 5.12.x, and provides its own web server (Hypnotoad). You can start by generating a "Lite::App", which is a simple self-contained single-file application, and then grow it to a bigger distribution later on as the need arises. It even comes with tools to convert your application to a conveniently packaged distribution that can be installed as easily as any CPAN module.
So that leaves the issue of security. User authentication, IP whitelisting, local network only... there are many ways to make a web application "for internal use only" if that's what you need.
You might just throw together a web-application prototype, and once you determine customer interest in your product, invest the substantial time in writing it as a Windows GUI application.
Continuing on from DavidO's answer, because the current web microframeworks for Perl (I prefer Dancer over his suggestion of Mojolicious, but both are good and largely equivalent) contain their own bundled web mini-servers, they also allow for the app to easily run entirely on the local machine.
Since these mini-servers default to a non-standard port (usually 3000 or 5000) by default and you can easily set them to a different port, they can be isolated by firewalls relatively easily, ensuring that nobody can connect to them remotely. You can also, of course, include and IP address check in the app and have it reject any requests that don't originate from localhost.
My guess is that the target system will be Windows. Use a RAD (Rapid Application Development) platform to develop a GUI. Examples for such a platform are Delphi or .NET with C# or VB. For bundling the Perl part, consider using a tool called perl2exe.
It doesn't sound like your scripts should require a web server. Also, consider the installation hassle. Only guessing as you're not giving much information about what your scripts are doing.
I am using the Cava packager to deploy my Perl-written tools. You can even generate an installer executable with just some mouse-clicks. It works pretty well with strawberry Perl and wxPerl on Windows.
I am looking to start Web Programming in Perl (Perl is the only language I know). The problem is, I have no prior knowledge of anything to do with the web, except surfing it. I have no idea where to start.
So my question(s) is...
Where do I start learning Web Programming? What should I know? What should I use?
I thank everybody in advance for answering and helping.
The key things to understand are:
What you can send to browsers
… or rather, the things you intend to send to browsers, but having an awareness of what else is out there is useful (since, in complex web applications in particular, you will need to select appropriate data formats).
e.g.
HTML
CSS
JavaScript
Images
JSON
XML
PDFs
When you are generating data dynamically, you should also understand the available tools (e.g. the Perl community has a strong preference for TT for generating HTML, but there are other options such as Mason, while JSON::Any tends to be my goto for JSON).
Transport mechanisms
HTTP (including what status codes to use and when, how to do redirects, what methods (POST, GET, PUT, etc) to use and when).
HTTPS (HTTP with SSL encryption)
How to get a webserver to talk to your Perl
PSGI/Plack if you want modern and efficient
CGI for very simple
mod_perl if you want crazy levels of power (I've seen someone turn then Apache HTTPD into an SMTP spam filter using it).
Security
How to guard against malicious input (which basically comes down to knowing how to take data in one format (such as submitted form data) and convert it to another (such as HTML or SQL).
Web Frameworks
You can push a lot of work off to frameworks, which provide structured ways to organise a web applications.
Web::Simple is simple
Dancer seems to be holding the middle ground (although I have to confess that I haven't had a chance to use it yet)
Catalyst probably has the steepest learning curve but comes with a lot of power and plugins.
Dependent on complexity of your project, you could have a look at Catalyst MVC. This is a good framework, messing up with the most request stuff, but gives you enough in deep view whats going on.
There is a good tutorial in CPAN
If you want to start with mod_perl or CGI, there are also some Tutorials :
mod_perl
CGI Doc
If you're looking to try some web programming in Perl, you could try hosting a Dancer app for free on OpenShift Express.
There's even a "Dancer on OpenShift Express" repo to get you started: https://github.com/openshift/dancer-example
I'm going to design an open-source web service which should collect ("web-scrape") some data from multiple - currently three - web sites.
The web sites do not expose any web service nor any API, they just publish web pages.
Data will be collected 'live' on any client's request from all the web sites in parallel, and will then be parsed to XML to be returned to the client.
The server operating system will be Linux.
The clients will initially be just an Android application of mine.
The concurrent clients will possibly be about 100 or more, if the project will be successful... ;-).
Currently my preferencese go to the adoption of:
perl (for the service laguage)
mod_perl2 with ModPerl::Registry (for an Apache embedded fast perl interpreter)
perl module CHI::Driver::FastMmap (for a modern and fast cache handler)
perl module Coro (for an async event loop to place many requests in parallel)
Since I suppose the specifications on the project can be of general use and interest, and since I am getting many problems with the combined use of Coro with mod_perl2, I ask:
Are my adoption preferences well matched?
Do you see any incompatibilities or potential problems?
Do you have any suggestion to enhance (in this order):
compatibility among components
neatness of the implementation
ease of maintainability
performances
You probably don't want to develop using mod_perl for any new project anymore. You really want to use something Plack based, or maybe even Plack itself. If you want to use Coro, using a AnyEvent such as Twiggy based backend may make most sense (though you may want to put a reverse proxy in front of it).
Are you happy sticking with apache?
If so, forget Coro and let apache handle concurrency; preload your modules and configuration, and write a super-efficient apache RequestHandler. (That's the way I go whenever apache2+modperl2 is available.)
If not, start learning Plack which is server-agnostic.
If you choose the first route, I'd recommend avoiding traditional CGI and instead adopting CGI::Application, which gives almost the lightness and speed of CGI but with a much much nicer/modern development environment and framework (and is Plack-compatible).
So, there are several languages which will allow you to create a website, as long as you configure the server(s) well enough.
To my knowledge, there is:
PHP
ASP.NET
Ruby(on rails, what is
that all about?)
And thusly, my knowledge is limited. Ruby and ASP, I've only heard of, never worked with. If there are other languages, I suppose they have some way to make files containing the needed html. It would then suffice to add a line to the Apache config to associate the file-extension.
And if other languages: are there any notable characteristics about the one(s) you mention?
ANY language can be use to make a dynamic website - you could do it in COBOL or FORTRAN if you were twisted enough. Back in the olden days (about 10 years ago) most dynamic websites were done with CGI scripts - all you needed was a program that could read data from standard input and write data (usually HTML) to standard output.
Most modern languages have libraries and frameworks to make it easier. As well as the languages you have already mentioned, Java, C# and Python are probably the most common in use today.
Typically a web framework will have:
a way of mapping URLs to a class or function to handle the request
a mechanism for extracting data from a request and converting it into an easy to use form
a template system to easily create HTML by filling in the blanks
an easy way to access a database, such as an ORM
mechanisms to handle caching, redirections, errors etc
You can find a comparison of popular web frameworks on wikipedia.
How can you forget Java ? :)
Python
It runs on Windows, Linux/Unix, Mac OS X, and has been ported to the Java and .NET virtual machines.
Python is a perfect scripting language for web applications, e.g. via mod_python for the Apache web server. With Web Server Gateway Interface a standard API has been developed to facilitate these applications. Web application frameworks or application servers like Django, Pylons, TurboGears, web2py and Zope support developers in the design and maintenance of complex applications. Around libraries like NumPy, Scipy and Matplotlib, Python is a standard in scientific computing.
Among the users of Python are YouTube and the original BitTorrent client. Large organizations that make use of Python include Google, Yahoo!, CERN, NASA,and ITA.
This could be for your interest.
Virtually thru CGI all programming languages that produce output may use for web page generation.
Basically, you can use any language (if you are hosting your own server)
Very closely related and very interesting is this article where LISP has been used to build a very succesfull website.
Python has a 3rd party module CherryPy which can be used with or without a http server.
Amongst others: Erlang (YAWS, Mochiweb), Python
JSP has the advantage that it automatically wraps your code in a servlet, compiles that to bytecodes, then uses the just-in-time Java compiler to recompile critical sections into native object code. Not aware of any alternative which allows optimizes your work automatically in this way.
Also allows you to develop and deploy on any combination of Windows, Mac OS X, or Linux.
If you'd like to choose one for the beginning, you should check out PHP first. It gives you the basic clues about how dynamic sites work in general.
After you've become familiar with the basics, I recommend ASP.NET.
Fist off, you should know that ASP.NET is a technology and not a language. (It actually supports any language that can be used on the .NET platform.) Also it is not to be confused with classic ASP. (The old ASP was much more like PHP.)
ASP.NET is very easy to begin with, and after you have some clues about its concepts, you can always dig deeper and customize everything in it. The http://asp.net site is a very good starting point, if you are to learn it. I think it is really worth the effort, because even if you choose not to stick to it, it will give you some interesting ideas and concepts.
I tell you its most important advantages:
The code is compiled (and NOT interpreted like PHP), and it has a very good performance. (In a performace comparsion, it is 10-15 times faster. http://www.misfitgeek.com/pages/Perf_Stat_0809.htm)
It can be run on Windows without effort, and on Linux / Mac / etc using the Mono project.
It implements the Separation of Concerns principle very well.
It has most of the general functionality you'll need built-in. (Such as membership, roles, database management, and so on.)