I am developing a prototype web application. I am using a toy http server which I have developed myself and it is written in C. The server needs to obtain text data from html form submitted and for furthur processing should pass it to a script which I have written in python. I know the following methods to achieve the same. Also once the processing is done the script has to return the data to server.
Save the data in a file and let the python script access it.
Calling the system or exec family of functions from within the Server code.
I have heard of CGI where a socket is used on both sides. I know how to do the same in my python script. But I am not able to see how the same is done in C server side.
Normally this is abstracted out as most people use apache or nginx as theor service. How can CGI be achieved in Server code?
CGI is covered here: https://www.rfc-editor.org/rfc/rfc3875
By 'CGI over sockets' you probably mean FastCGI which is a bit different (and involves a long running process listening on a Unix socket and sending back responses). Classical CGI involves spawning a short lived process which then receives the request parameters via environment variables (set this from your server via setenv().
Request data is sent to the spawned process via stdin (e.g. in Python, sys.stdin). The output response is then written (with headers and all) via stdout.
CGI is very simple and this is why it was so quick to be adopted by a high number of languages -- the learning curve to implementing CGI was rather quick.
FastCGI is similar to CGI only at the script language layer (i.e. the programming interface remains somewhat similar); however the actual mechanics are rather different (you need to serialize the information over Unix sockets).
Related
I want to write a Forth program for my Raspberry Pi. This program shall read some sensor data from an existing file and send it to a website where the data gets stored.
Searching the web there was plenty of documentation about how easy and fast Forth is and how to calculate Fibonacci numbers. But how can I request a URL? How can I send data to a website and process the result?
If there's no socket support: is it possible to start an external program like curl/wget to do the request?
It is not obvious from the question what the Forth implementation is used.
For example, Gforth (that is available on Raspberry Pi)
has some support of the sockets and also it allows to start an external program using system word. See my answer on the similar question: How do I read raw code from a website in Gforth?
Example of creating curl child process in Gforth:
S" curl https://example.com/" system
In any case,
it seems that the most promising approach for the given problem is to develop a binding to libcurl (if it doesn't exist yet for the used Forth implementation).
I'm writing an application that needs to send data to a socket and receive data from the socket.
I know how to do this; I've done it several times before it other apps.
Unfortunately, the only documentation I have for the server I need to communicate with in the current project is in the form of a JavaScript file that uses some library called "Socket.IO". I am not familiar with this library (or JavaScript for that matter).
The JavaScript code contains statements such as "io.connect" with a parameter that looks like a web site URL, as well as multiple "emit" statements that have at least two parameters each. I need to know exactly what these statements are doing in terms of: what host and port is it connecting to? What bytes is it sending to the socket?
Where can I find this information?
Thanks,
Frank
I want to create a script that the only parameter that I will give him is url. The output of the script will be the status code of the url. Any idea how to do that?
LWP is commonly used to make HTTP requests from Perl.
There's also the very fast Net::Curl::Easy, which supports parallel requests using Net::Curl::Multi.
Of course, event-based frameworks have their own clients. For example, AnyEvent has AnyEvent::HTTP. These also allow parallel requests.
Is there a conventional way to write a program such that commands can be issued to the program from the command line without a repl? For example, how you can send commands to a running nginx server using sudo /etc/init.d/nginx restart (or any other valid command besides restart)
One idea I had was having the long-running program create and monitor a unix socket that other programs can write to to send it commands. Another was to create a local server with a REST interface that can be sent commands that way, though that seems a bit gross.
What's the right way to do this?
Both ways are ok, and you could even consider using some RPC machinery, such as making your application serve JSONRPC on some unix(7) socket. Or use a fifo(7). Or use D-Bus.
A common habit on Unix is to have applications reload their configuration files on e.g. SIGHUP signal, and save some persistent state (before terminating) on SIGTERM. Read signal(7) (notice that only async-signal-safe routines can be called fro signal handlers; a good way is to only set some volatile sig_atomic_t variable inside the handler and test it outside). See also POSIX signal.h documentation.
You might make your application become a specialized HTTP server (e.g. using some HTTP server library like libonion) and give it some Web interface (or REST, or SOAP ...); the user (or sysadmin) will then use his browser to interact with your application.
You could make your server systemd compatible. (I don't know exactly what that requires, it is perhaps D-bus related).
You could embed some command interpreter (like Guile and Lua) in your app and have some limited kind of REPL loop running on some IPC like a socket or a fifo. Beware of nasty code injection.
I had a similar issue where I have a plethora of services running on any number of machines and each is in need of communicating with several others.
My main problem was not so much the communication between the services. That can be done with a simple message sent over a connection (as Basile mentioned, it can be TCP, UDP, Unix sockets, FIFOs...). However, when you have over 20 services, many of which need to communicate with several other services, you start having a headache on how to get all the connections right (I have such a system, although it has a relatively limited number of services, like just 10 and that's already very complicated).
So I created a process (yet another service) called Communicator. All services connect to the Communicator service and when they need to send a message, they include the name of the service they want to reach. The Communicator service is in charge of sending the message to the right place—i.e. it could be to another Communicator service running on a different computer. Communicator has a graph of all the services available on your network and knows how to send messages to them without your service having to know anything about all of that. Computing a graph can be really complex.
For the purpose, I created the eventdispatcher project. It is in C++, which may not be what you're interested in, although you could use it in other languages that interface with C/C++. The structure of the messages are "proprietary" (specific to the Communicator), but you can create any message you want. A message includes a name and parameters (param-name=value). The first version has a simple one line text communication system. The newer version accepts JSON as well (still must be one line of text per message).
The system supports TCP, UDP, Unix sockets, FIFO, and between threads, you can have thread safe fifos. It also understand signals (like SIGHUP, SIGTERM, etc.) It has a specific connection to listen for the death of a thread. It supports encryption over TCP via OpenSSL. The messages can automatically be dispatched (hence the current name of the library). Connections are assigned a timer. And there are CUI and GUI (Qt) extensions as well.
The one main point here is that all your connections can be polled (see poll()) and thus you can implement a system that reacts to events instead of a system which sleeps and checks for events, sleeps and check, etc. or worth, you have a single blocking connection and everything has to happen on that one connection or your service gets stuck. This is one reason Unix has been using signals since early version of Unix did not have select() nor poll().
I know this sound like a question for ServerFault but I know that developers often get the blame when servers are struggling so I thought a post here might be useful to people who still use Perl on the web.
THE STORY:
We had serious issues with defunct processes on our old Apache server so we decided to move to Apache 2. The new server performs much better, no denying that. Tests reveal however, that under a heavy load (~100 users per minute) defunct processes start quickly ramping up on the server and using SSH it is clear that these processes are using the CPU. To overcome these issues we decided to implement CGI::Fast which is a type of FastCGI in Perl. Having that in place the zombies are gone, however performance wise the server is not coping any better.
The results led me to assume that there isn't really a point implementing CGI::Fast if Apache 2 will efficiently reclaim the resources anyway.
Does any of you have come to a different conlusion?
In my opinion it’s not worth moving to anything but a PSGI/Plack-based solution in 2011, independent of any other details you mentioned.
The Plack ecosystem is so much greater than FastCGI’s. (advantage)
You can deploy to many more servers than FastCGI supports. (advantage)
You may run CGI scripts unmodified with e.g. Plack::App::CGIBin whereas CGI::Fast requires a rewrite. (advantage)
Rewriting a CGI program to conform to the PSGInterface requires slightly more effort than rewriting it to CGI::Fast. (disadvantage)
FastCGI is faster than plain CGI because Apache doesn't have to load perl for each new request. However, the scripts need to be reworked to remove the assumption that they are executed once for each request. A FastCGI script at its core typically represents some type of event loop, processing the requests as they come in.
You can use CGI::Fast for plain CGI scripts without reworking the script around an event loop, but you lose the "Fast" part of FastCGI this way, as perl still needs to be run once for every script.
FastCGI also only provides a large benefit if the biggest part of your CGI script is loading perl or executing one-time code. For many web applications, this is true. However, if your script needs to do a lot of work for each request, such that the overhead of loading perl is small, then you won't see a big performance benefit by using FastCGI.
CGI
It was too inefficient for anything but small sites. CGI spawns a new process for every incoming request to execute a script, a very resource intensive and inefficient way of doing things. No wonder it faded away over time as web applications became more complex.
FastCGI was introduced to avoid some of the issues with running languages inside the Apache process, as well as avoiding the inefficiency of CGI.
A FastCGI application is executed outside of the web server (Apache or other wise), and waits for requests from the web server using a socket. The web server and the FastCGI application can even be on separate physical machines and communicate over the network.
Because the web server adn the application processes are separate better isolation is possible.