Here is my problem:
I have a closed-source third-party Win32 application, which acts as a server for other programs via named pipes, i.e. it expects its clients to do smth like this:
HANDLE h = CreateFile("\\\\.\\pipe\\$pipe_name$", GENERIC_READ | GENERIC_WRITE, etc...);
// ...
TransactNamedPipe(h, buf, etc...);
// ...
CloseHandle(h);
This app runs perfectly in WINE, except that I can't communicate with it. So here is my question:
What exactly does WINE do when it is requested to open a pipe? Does it, say, map it to some FIFO file in ~/.wine/ or wherever? Is there any way to communicate with such program from a Linux application? Google doesn't know anything about it.
Thank you.
Named pipes are hosted by the WINE server process. Requests are sent to this process by the WINE clients. For example, CreateNamedPipe uses a request like:
SERVER_START_REQ( open_file_object )
{
req->access = access;
req->attributes = attr->Attributes;
req->rootdir = wine_server_obj_handle( attr->RootDirectory );
req->sharing = sharing;
req->options = options;
wine_server_add_data( req, attr->ObjectName->Buffer, attr->ObjectName->Length );
io->u.Status = wine_server_call( req );
*handle = wine_server_ptr_handle( reply->handle );
}
The server manages connecting the named pipe. Once a client and server have connected, the WINE server gets out of the way by sending an fd to the client. I think this fd is just an anonymous pipe created by the WINE server, one end being sent to the pipe server and one end to the pipe client.
Named Pipes in a wine official wiki
this article could help too: http://lkcl.net/namedpipes/namedpipes-emulation.txt
Related
I am trying to make SSL_read() block until data is received from the server. I tried setting the socket to blocking in my "connect" code like so
{
u_long iMode = 1;
i = ioctlsocket(sock, FIONBIO, &iMode);
}
but strangely I get a stack overflow exception every time.
Is there another more correct way to do this?
(Note: the app works just fine if I omit this code).
I've searched SO on this issue but everywhere people seem to have just the opposite problem, namely blocking when they want non-blocking.
Code Synopsis:
Get method: TLS_client_method()
Get CTX: SSL_CTX_new(method)
Create socket 'sock'
set socket to blocking (code above)
Connect sock to host on port 443
Create SSL*: ssl=SSL_new(ctx)
SSL_set_fd(ssl, sock)
Do SSL_reads and writes
I achieved what I wanted by switching from SSL_* calls to BIO_* calls for connect, read, write, etc..
The BIO family includes function 'BIO_set_nbio()' which sets blocking/nonblocking mode. Worked great.
Sample Code Synopsis (e.g., for google.com):
Get method: method = TLS_client_method()
Get CTX: ctx = SSL_CTX_new(method)
Create BIO object: bio = BIO_new_ssl_connect( ctx )
Create socket 'sock'
Connect 'sock' to google.com, port 443
Create SSL*: ssl = SSL_new(ctx)
SSL_set_mode( ssl, SSL_MODE_AUTO_RETRY )
BIO_set_conn_hostname( bio, "google.com:443" )
BIO_do_connect( bio )
BIO_set_nbio( bio, mode ) (with mode=0 for blocking)
Now can do blocking BIO_reads and writes (with BIO_read, BIO_write)
Is there a way to get a Clients IP in Context of a write?
I want to get the IP of an Client that writes to my Milo-OPCUA-Server, so I can handle these writes differently based on the Clients IP (local Clients should be able to write directly on the Server, whilst other writes should get forwarded to another Server)
Okay, this is not part of any official API right now, so it almost certainly will break in the future, but:
With the OperationContext you get when implementing AttributeManager#write(WriteContext, List<WriteValue>):
context.getSession().ifPresent(session -> {
UaStackServer stackServer = context.getServer().getServer();
if (stackServer instanceof UaTcpStackServer) {
ServerSecureChannel secureChannel = ((UaTcpStackServer) stackServer)
.getSecureChannel(session.getSecureChannelId());
Channel channel = secureChannel.attr(UaTcpStackServer.BoundChannelKey).get();
SocketAddress remoteAddress = channel.remoteAddress();
}
});
I'll have to add some official API to do this, probably something hanging off the Session object.
I've been battling this question for a while, as surely many Catalyst programmers did as well. Now we see some examples from John about nonblocking applications running with Twiggy.
But I think Twiggy is not the best option to run my whole application. So clearly I want to decouple it and run my app on nginx for example and forward my comet traffic to the Twiggy.
The main problem I see is the authentication. There are several possible options here, that I see:
move authentication to a front-end server
move authentication to a separate catalyst app
use session ids when communicationg with a Twiggy
?? Using Plack sessions ??
First option is not really good, because it does not give me flexibility when changing a front-end server. The second one has also considerable expenses. And the third one I think is the easiest one, taking into account that the Catalyst::Plugin::Session::Store::DBI is used as a session backend.
These are the options that came to my mind. Surely I miss something. So maybe someone encountered the same problems? I would be grateful to anyone who can give me a hint or expand my view on this problematic. It will be also helpful to see pros and contras about each option, as well as some hints about them.
Catalyst and long-polling (comet) applications
Overview
When in the middle of 2013 I decided to incorporate support of reverse AJAX ( further Comet ) functionality into my Catalyst application I found surprisingly little information about it on the Web. So I started to collect the information piece by piece, which forced me to plunge down deeper into the understanding of the Catalyst framework. Then there appeared some good examples of non-blocking code from John Napiorkowski (current maintaner of Catalyst), which have clarified a lot about this topic. So I wrote a simple server, running on Twiggy and providing long-living websockets connections for clients. This solution does not pretend to be the best or even a good one. It is just a working solution. The code has not been refactored and is provided as is. It can be used as a basis for building more robust and reliable comet applications. It can be also improved in many ways. So if you see some mistakes or suggestions for improvement, please let me know.
Introduction
In this section I would like to outline the background for this code.
I have a Catalyst application implementing social networking functionality. It uses mostly AJAX-requests to get data from server. Every minute it makes an AJAX-request to get data updates for a logged-in user. If session has expired, user will be logged out and redirected to the log-in page. For a logged in user there is a part of my application, where a user needs to get periodic updates of data. It is not critical to use comet on this page, I could easily use AJAX (an easier option, but related with network latency and bandwidth and with unnecessary requests sent), but I decided to experiment a bit.
If you run your Catalyst app on a preforking server, you will have a number of servers which serve your clients. If you want to have a long-living connection in your Catalyst app, it means that you will block one instance of your app while keeping this connection open. If you are going to have only a few clients and a lot of hardware resources you may be able to prefork your app. However, if you want to have hundreds, let alone thousands of concurrent connections, this solution may not be suitable for you, because you run out of resorces very fast. This means that either your Catalyst application must run on a nonblocking server, or your client (browser) should communicate with another server, which does not consume much hardware resources and can allocate an instance of itself for each client, while keeping the connection open. Or the client can be connected to a server which runs in an event loop and responds to new data for user in a asynchronous way.
The only nonblocking server for my Catalyst app I was able to find was Twiggy. This server is based on AnyEvent. AnyEvent is a framework for event-driven programming model in Perl and it makes it possible for Twiggy server to serve clients in a nonblocking asynchronous way. It is good for example for requests where it takes some time to get data for user. The server does not block and wait for data to be available for user, but instead it continues to listen to new incoming requests and as soon as data for some user is ready it will be sent to appropriate user.It is probably not the best idea to run your whole Catalyst app on Twiggy. One may want to use some robust, well-tested server (nginx or Apache or whatever) and run your Catalyst app behind those front-end server as FastCGI processes, for instance (the option I chose). So I decided to run Twiggy instance and direct Comet traffic to it (I tried to use Twiggy behind nginx proxy for websockets connection but it didn't work somehow, so I dropped it without further investigation). The first problem was authentication. Because it is done in the Catalyst app and not on the front-end server, it means that my Comet app must somehow know if user is authenticated.
Authentication
Catalyst's module for authentication is Catalyst::Plugin::Authentication. It takes care of authentication and authorization of a user in your app. You most certainly use session module Catalyst::Plugin::Session together with it. This allows to preserve application data across HTTP-requests (including that the user is authenticated). This session module is split into two parts: state and store. The first one allows to choose HOW you preserve app's state across different HTTP-requests (most probably with cookies). The second one allows you to choose WHERE you want to store user's data for his session.So I use Session::State::Cookie for the state and Session::Store::FastMmap for the store. This means that when user is being authenticated, he gets a session id, a secret string, which he will send in a HTTP-header in every HTTP-request to a server. This session id will be valid for some time and as long as it is valid, it is uniquely assigned to some user. Then on every incoming request user's data will be restored from a mmap'ed file through Session::Store::FastMmap. This file acts as a shared memory interprocess cache. This solution (FastMap) is good if your whole app runs on a single server, but if you do load-balancing, for example, you may want to use another solution ( like Catalyst::Plugin::Session::Store::DBI ).So I decided to hack on this session data. In my Comet app I can access this session data and check if user is authenticated. This is done in the following sub.
sub _check_session {
my ($sid, $this_user_id) = #_;
my $return = 0;
my $user_session = $session->get("session:$sid");
if ( $user_session ) {
## Check user realm existence
return $return unless ( $user_session->{__user_realm} );
## Check user presence
return $return unless ( $user_session->{__user} );
## Check session expiration time
my $session_expires_time = $session->get("expires:$sid");
my $now = time();
if ( $now > $session_expires_time ) {
return $return;
}
## Check if it is still the same user
if ( $this_user_id && ($this_user_id ne $user_session->{__user}->{id} ) ) {
return $return;
}
else {
$return = $user_session->{__user};
}
}
return $return;
}
Looking through Catalyst::Plugin::Session and Catalyst::Plugin::Authentication I concluded that it is necessary to check at least the following keys in the session data:
__user_realm: if user is authenticated in at least one realm, this key is present in the session hash
__user: if user is authenticated, this key represents user data ( which comes from the ::Store part of the Authentication module, most probably from DBIx)
"expires:$sid" represents a timestamp when the session expires
$session is an object allowing access to our mmap'ed file:
my $session = Cache::FastMmap->new( raw_values => 0, share_file => ('/tmp/myapp/session_data') );
We are interested in two pieces of data which can be looked up in the session file: "session:$sid" is a key for the session data and "expires:$sid" is a timestamp of session expiration.So now, when a browser tries to establish a websocket connection with our Comet app, we have to call this sub. If a user is authenticated, a websocket connection will be established with the server. While my application automatically closes websocket connection when user logs out or navigates away from the Comet app in his browser, I nevertheless decide to check session id every $interval seconds. So if a malicious user opens a websocket connection on his own, he will get no use of this. For the case when user A logs out and user B logs in using the same session id as user A and all this happens before the next session check, the session will be still active but will relate to another user. In this case we have to check if the session corresponds to the user who initially established the websocket connection:
if ( $this_user_id && ($this_user_id ne $user_session->{__user}->{id} ) ) {
return $return;
}
else {
$return = $user_session->{__user};
}
PSGI, Plack::Builder, Plack::Request
It was a natural choice to implement my Comet app as a PSGI application. I assume you are familiar with this specification.Say you want your app to map different URLs to different applications, for example when you have several areas in your website each requiring it's own comet logic. You can achieve this by using Plack::Builder:
use Plack::Builder;
...
## 1st app entrance point
my $psgi_app = sub {
my $env = shift;
...
}
...
builder {
## mount 1st app
mount "/comet/first_app" => $psgi_app;
}
Now you can mount as many applications as you want each corresponding to a different path (URL).As you know first argument to a PSGI app is $env, which is an environment variable, a hash containing different keys pertaining to a HTTP-request and keys which have to do with the PSGI specification. Using it we can create a Plack request object, which allows us to access different request data and cookies. One of the cookies will contain a session id, which is a starting point for authentication check.
## Request object
my $req = Plack::Request->new($env);
## session id
my $sid = $req->cookies->{myapp_session};
## HTTP origin header
my $req_base = $env->{HTTP_ORIGIN};
Delayed and streaming response
As you know a PSGI app should return a tuple (HTTP-status, HTTP-header and HTTP Body Data). But to enable a server push, an app should return a callback as its response. This callback will then be executed by the underlying server. You can now utilize an event loop in your app to stream data to client.To be able to implement a websocket server one has to use a PSGI extension psgix.io which gives access to a raw internet socket, so that one has a full access over streaming data. Because in the websocket specification one has to do an upgrade from HTTP to the ws protocol during an initial handshake connection, the low-level access to the socket is required.
my $psgi_app = sub {
my $env = shift;
my $fh = $env->{'psgix.io'} or return [500, [], []];
## Create websocket handshake
my $hs = Protocol::WebSocket::Handshake::Server->new_from_psgi($env);
$hs->parse($fh) or return [400, [], [$hs->error]];
return sub {
my $responder = shift;
...
}
}
So we create an object which takes care of data format of messages for the websocket protocol, which are exchanged between client and server. This object is initialized with our raw internet socket so that it can fulfil the HTTP upgrade. And afterwards we return a callback which will be our delayed response.
The comet app, server-side
So here is the whole comet psgi app:
use Plack::Builder;
use Plack::Request;
use AnyEvent;
use Protocol::WebSocket::Handshake::Server;
use Cache::FastMmap;
use JSON;
use Template;
use Log::Dispatch;
use Data::Dumper;
use DateTime;
use FindBin qw($Bin);
use lib "$Bin/../lib";
use myapp::Schema;
use warnings;
use strict;
## Session data
my $interval = 3;
my $session = Cache::FastMmap->new( raw_values => 0, share_file => ('/tmp/myapp/session_data') );
## Database connection, for example with a Postgres DB
my $db_schema = myapp::Schema->connect( {
dsn => 'dbi:Pg:dbname=myapp_test',
user => 'my_login',
password => 'my_passwd',
pg_enable_utf8 => 1
} );
## Logging object
my $log = Log::Dispatch->new( outputs => [ [ 'File', min_level => 'debug', filename => '/var/log/myapp_test/comet' ] ] );
## Adjust this sub correspondingly if Session::Store has been changed.
sub _check_session {
my ($sid, $this_user_id) = #_;
my $return = 0;
my $user_session = $session->get("session:$sid");
## Check if the sid and the user email match
if ( $user_session ) {
## Check user realm existence
return $return unless ( $user_session->{__user_realm} );
## Check user presence
return $return unless ( $user_session->{__user} );
## Check session expiration time
my $session_expires_time = $session->get("expires:$sid");
my $now = time();
if ( $now > $session_expires_time ) {
return $return;
}
## Check if it is still the same user
if ( $this_user_id && ($this_user_id ne $user_session->{__user}->{id} ) ) {
return $return;
}
else {
$return = $user_session->{__user};
}
}
return $return;
}
## 1st app entrance point
my $psgi_app = sub {
my $env = shift;
my $fh = $env->{'psgix.io'} or return [500, [], []];
## Create websocket handshake
my $hs = Protocol::WebSocket::Handshake::Server->new_from_psgi($env);
$hs->parse($fh) or return [400, [], [$hs->error]];
return sub {
my $responder = shift;
## App data
my ($w, $hd, $input_params, $req, $sid, $user_id, $ret, $time_lapsed, $req_base);
## Clean up the websocket local environment
my $clean_up = sub {
$log->debug("\nCleaning up...\n");
## Destroy websocket
$hd->destroy;
## Remove timer from event loop
undef $w;
};
$hd = AnyEvent::Handle->new(
fh => $fh,
on_error => sub {
my ($hd, $fatal, $msg) = #_;
$clean_up->();
}
);
## Send server websocket handshake
$hd->push_write($hs->to_string);
## Websockets connection is initialized and is ready for data to be sent
#$hd->push_write( $hs->build_frame( buffer => encode_json( { 'status' => "Connection init..." } ) )->to_bytes );
## Get request data
$req = Plack::Request->new($env);
$sid = $req->cookies->{myapp_session};
$req_base = $env->{HTTP_ORIGIN};
## Check if user is authenticated
unless ( $ret = _check_session($sid, undef) ) {
$clean_up->();
}
else {
$user_id = $ret->{id};
}
$time_lapsed = 0;
## Template toolkit
my $template = Template->new({
INCLUDE_PATH => "$Bin/../root/templates",
VARIABLES => {
req_base => $req_base,
user_id => $user_id,
user_lang => $ret->{language}
},
ENCODING => 'utf8',
});
## Input parameters and recieve user's data.
$hd->on_read(sub {
(my $frame = $hs->build_frame)->append($_[0]->rbuf);
while (my $message = $frame->next) {
my $decoded_data = eval { decode_json $message };
## If it's not a valid json - exit
if ($#) {
$clean_up->();
}
else {
## New connection
if ( $decoded_data->{is_new} ) {
$input_params = $decoded_data;
$stash = {
template_data => "some data"
};
my $tt_output;
$template->process( "template_path", $stash, \$tt_output );
$hd->push_write( $hs->build_frame( buffer => encode_json( { 'init_data' => $tt_output } ), max_payload_size => 200000 )->to_bytes );
}
## Else - additional data are sent from the client
else {
}
}
}
});
## THIS APP'S MAIN LOGIC
## As an example, let's track if user has changed his/her name and return a message to the browser
my $app_logic = sub {
my $this_params = shift;
if ( $user_id ) {
my $rs = $db_schema->resultset('User')->search( { id => $user_id } )->single;
if ( $rs->first_name ne $ret->{first_name}) {
$hd->push_write( $hs->build_frame( buffer => encode_json( { 'data' => "User changed his name!" } ) )->to_bytes );
}
}
};
## Any event logic
$w = AnyEvent->timer (
interval => $interval,
after => $interval,
cb => sub {
## Check every half a minute if the user is still authenticated
if ( $time_lapsed > 30 ) {
$time_lapsed = 0;
unless ( $ret = _check_session($sid, $user_id) ) {
$clean_up->();
}
else {
## Check if user' object has been changed (e.g. his language etc.)
}
}
## Execute main logic
$app_logic->($input_params);
$time_lapsed += $interval;
}
);
};
};
builder {
## mount 1st app
mount "/comet/myapp" => $psgi_app;
}
So we start the program with initializing some common objects like database handle and session object.When a websocket connection is terminated we don't want to respond to events pertaining to it, so we remove them from the event loop. That is what is done in the sub reference $clean_up. Then we define an AnyEvent::Handle object and listen to it's on_read() callback which is fired up every time new data arrives from the client.Because I want to be able to use the same template for generating HTML both for my Catalyst app and for my comet app, I create a Template object and initialize it with variables which must be the same in the Catalyst counterpart. First time the on_read() callback is called is when a client opens a websocket connection. In the javascript part we define a special key for this and send the client initial data on new request (in my case it ís the data for which later I get comet updates).Additionally, we create an AnyEvent timer object which will periodically execute our main logic app $app_logic. It will also check if the user is still authenticated and is granted to get the data update from the server.Don't forget, if you change some user data in your database through a Catalyst controller and this change must be reflected in the session hash, you have to persist it by calling
$c->persist_user();
The comet app, client-side
I use module pattern for javascript modules to create a separate namespace for every javascript module. Here is one to handle communication with the comet server.
var myapp = (function() {
// Context data, private
var data_loaded = false;
var this_page = true;
return {
init: function() {
data_loaded = false;
this_page = true;
// No websockets in safari, somehow they don't work there
if ( navigator.userAgent.indexOf('Safari') != -1 && navigator.userAgent.indexOf('Chrome') == -1 ) {
myapp.myapp_load_ajax();
}
else {
// Create a websocket
websockets["myapp_socket"] = new WebSocket('ws://my-domain-name:5000/comet/myapp');
var input_hash = {};
input_hash["is_new"] = 1;
websockets["myapp_socket"].addEventListener("open", function(e) {
websockets["myapp_socket"].send(JSON.stringify(input_hash));
data_loaded = true;
});
websockets["myapp_socket"].addEventListener("message", function(e) {
var this_obj = JSON.parse(e.data);
// Connection is initialized
if ( this_obj.init_data ) {
// Make necessary initializations
myapp.init_after_loading();
}
// Websockets data update from server
else if ( this_obj.data ) {
// Do something meaningful on data update
}
});
websockets["myapp_socket"].addEventListener("close", function(e) {
//Connection has been closed
});
// In case when a websocket cannot be created, fall back to an AJAX request
websockets["myapp_socket"].addEventListener("error", function(e) {
// Unless the data have already been loaded, load it here for the first time, because
// this method will be also invoked when connection is dropped.
if ( !data_loaded && this_page ) {
myapp.myapp_load_ajax();
}
});
}
},
myapp_load_ajax: function() {
jQuery.ajax({
type: "POST",
url: "my_catalyst_app_load_ajax_data_path",
dataType: "json",
cache: false,
complete: function (xhr, textStatus) {
if ( xhr.responseText ) {
var this_data = jQuery.parseJSON(xhr.responseText);
if ( !this_data ) {
// An error happened
}
else {
// Make necessary initializations after your data has been inserted into the DOM
myapp.init_after_loading();
}
}
}
});
},
init_after_loading: function() {
// If you insert some data into DOM, initialize it here
},
close_socket: function() {
if ( websockets["myapp_socket"] ) {
websockets["myapp_socket"].close();
delete websockets["myapp_socket"];
}
},
};
})();
You can read about module pattern in details elsewhere on Internet. In short, it makes an object out of your javascript module and your methods will be accessible as this object's properties. This allows you to create a separate namespace for your module and to define private variables.In order to be able to access my websocket from another module, which may be necessary, I declare an object which holds it as a global one.What we have to do is to define event handlers for our websocket which include "open", "close" etc. If for some reason we cannot establish a websocket connection to our comet server (server is down, it does not accept new connections etc.), we fall back to AJAX. Additionally, if safari tries to create a websocket-connection with a dead server, it doesn't handle this case in an "error" event, so we just prohibit websockets in safari.So, we start by creating a new websocket connection for the URL that we have mounted in the plack builder in the comet server. Then we use websocket's event "open" to take care of new connection, signifying the server about new client connection. Event "message" is used to send messages to the comet server; "close" is called whenever a connection with the server is closed; "error" is called in case of problems with connection, for instance it cannot be established or it has been broken or the server has closed the connection or died. And that's it. Now we will get data updates from our comet server.
Starting comet server
Now what is left is to start our server. The current server's code assumes that it will run on the same machine as your Catalyst app. Some other possibilities will be discussed in the final notes section.We use command line utility plackup to start our server:
TWIGGY_DEBUG=1 plackup -s Twiggy comet.psgi
I use TWIGGY_DEBUG env var to see debug info from the Twiggy server.
Final notes
First thing to remember is that Twiggy server will exit as soon as a die statement will be executed. It means that you have to program it safely and to intercept every statement that can lead to this with an eval block.The premise for current comet server is that it runs on the same machine as your Catalyst app. If you are planning to do load balancing and to run it on another machine you have to take care of some things. First, you have to change your session plugin to Session::Store::DBI or something suited for distributing across several machines (and afterwards to adjust _check_session() to get data not from a file but from the database). Then change the dsn for the database connection to include a hostname and a port number.Another thing to note is that our main logic in the server checks every N seconds if user's name has been changed. So, it is inefficient to query a database so often if your server has many clients. Instead, there are some better solutions for this. First option is that if you run a Catalyt app and a comet server on a single machine, you can use the Cache::FastMmap file as a mediator between your Catalyst app and your comet server for getting notifications that some new data is available for the server and only then querying database to get the data update. In this case you make database queries only to get the new data. It means that in your Catalyst controller you have to write into the cache file to inform the comet server to check data every time when you make changes through you controller to the data that you get updates to in your comet server. For example you have a User controller and a User model. Whenever a user changes his name, User controller is called which in turn calls User model to change user'd data. So in the controller User you additionally write into the cache file that this is the case. Then the comet server will know when to get data updates from the database. The similar approach you can use if you do load balancing and your Catalyst and comet app run on different machines. But in this case you have to use your database as a mediator. You can, for example, create a new table which will be periodically queried by the comet server. Each column of the table could correspond to some application domain of your comet server. The column can be of type timestamp and to mark the time of last change of the data that you trace. In you Catalyst controller you write into corresponding column every time whenever the data in question has been changed and then you check this column in the comet server and you know then whether to query the database for the data updates or not. Thus you will avoid a lot of unnecessary data queires. But the better choice, however more complicated, would involve sockets. In this case when a user logs in we create a new socket for him and we write all data updates that we want to track directly into the socket. In the comet app instead of using an Anyevent timer, we define another AnyEvent::Handle which we initialize with the user's socket. And by using the on_read() method we get updates when they come and then return them immediately to the user. In this case we bypass data queries and it should work really fast. But this solution will require a lot of additional work in the Catalyst controller.Another thing to note is that current comet server does not support secure websockets (wss) protocol while Twiggy does not support TLS/SSL. The solution would be to use a SSL tunnel in front of your server which will transparently encrypt/decrypt messages (take a look at https://github.com/vti/app-tlsme).And the final note: I have tried to run a front-end proxy nginx in front of my comet server. But somehow nginx could not propagate messages to the Twiggy. So the client's browser communicates directly with the comet server. If you plan to have thousands and thousands of users then a websocket load balancing is a topic to think about.If you find any mistakes or have any improvements ideas please comment or write me an email (dhyana1981#yahoo.de).
I think the better option is 3 or 4.
Setup your nginx to server distinct location points for Catalyst-app and Twiggy-app.
And, if you are using Twiggy you may want speed, so instead of using DBI, I suggest to you save/restore/check sessions via a memory-based application, like Memcached or Redis, so you can scale up this later if you go to AWS or something like that.
You can do the job with Catalyst::Plugin::Session::Store::Memcached::Fast or Catalyst::Plugin::Session::Store::Cache and others, but if you known how to create a secure session token and how to keep/restore, you can do this by yourself, so you will known how to restore in Twiggy, Catalyst or anything else (even other languages)
Have a good day!
Because of shared hosting, my redis server on the target host does not run on a port, but on a very specific socket, which can be connected to via the socket file, only accessible to my user.
However, I have not found how I can specify connection via a socket in the node_redis and connect-redis packages, the ones I want to use.
Anyone know how to do it?
Update: My answer below is not really correct. It turns out that the solution in the issue I mention below actually still works. It's more of a coincidence, IMO, but you can do something like this, and it should work:
var redis = require('redis'),
client = redis.createClient('/tmp/redis.sock');
As you see from the code snippet below, this will get passed to net.createConnection which will connect to the unix socket /tmp/redis.sock.
Old answer:
There is a closed issue about this node_redis/issues/204. It seems, thought, that the underlying node.js net.createConnection API has since changed. It looks as though it would be a quite small fix in node_redis' exports.createClient function:
exports.createClient = function (port_arg, host_arg, options) {
var port = port_arg || default_port,
host = host_arg || default_host,
redis_client, net_client;
net_client = net.createConnection(port, host);
redis_client = new RedisClient(net_client, options);
redis_client.port = port;
redis_client.host = host;
return redis_client;
};
It seems as though net.createConnection will attempt to connect to a unix socket if it's called with one argument, that looks like a path. I suggest you implement a fix and send a pull request, since this seems like something worth supporting.
There is no longer a connect string...
var client = redis.createClient(9000); // Open a port on localhost
var client = redis.createClient('/tmp/redis.sock'); // Open a unix socket
var client = redis.createClient(9000, 'example.com');
This, and options are documented on the README.
What I'm Trying To Do
I'm trying to create a solution of any kind that will run nightly on a Windows server, authenticate to a website, check a web page on the site for new links indicating a new version of a zip file, use new links (if present) to download a zip file, unzip the downloaded file to an existing folder on the server, use the unzipped contents (sql scripts, etc.) to build an instance of a database, and log everything that happens to a text file.
Forms App: The Part That Sorta Works
I created a Windows Forms app that uses a couple of WebBrowser controls, a couple of threads, and a few timers to do all that except the running nightly. It works great as a Form when I'm logged in and run it, but I need to get it (or something like it) to run on it's own like a Service or scheduled task.
My Service Attempt
So, I created a Windows Service that ticks every hour and, if the System.DateTime.Now.Hour >= 22, attempts to launch the Windows Forms app to do it's thing. When the Service attempts to launch the Form, this error occurs:
ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment.
which I researched and tried to resolve by either placing the [STAThread] attribute on the Main method of the Service's Program class or using some code like this in a few places including the Form constructor:
webBrowseThread = new Thread(new ThreadStart(InitializeComponent));
webBrowseThread.SetApartmentState(ApartmentState.STA);
webBrowseThread.Start();
I couldn't get either approach to work. In the latter approach, the controls on the Form (which would get initialized inside IntializeComponent) don't get initialized and I get null reference exceptions.
My Scheduled Task Attempt
So, I tried creating a nightly scheduled task using my own credentials to run the Form locally on my dev machine (just testing). It gets farther than the Service did, but gets hung up at the File Download Dialog.
Related Note: To send the key sequences to get through the File Download and File Save As dialogs, my Form actually runs a couple of vbscript files that use WScript.Shell.SendKeys. Ok, that's embarassing to admit, but I tried a few different things including SendMessage in Win32 API and referencing IWshRuntimeLibrary to use SendKeys inside my C# code. When I was researching how to get through the dialogs, the Win32 API seemed to be the recommended way to go, but I couldn't figure it out. The vbscript files was the only thing I could get to work, but I'm worried now that this may be the reason why a scheduled task won't work.
Regarding My Choice of WebBrowser Control
I have read about the System.WebClient class as an alternative to the WebBrowser control, but at a glance, it doesn't look like it has what I need to get this done. For example, I needed (or I think I needed) the WebBrowser's DocumentCompleted and FileDownload events to handle the delays in pages loading, files downloading, etc. Is there more to WebClient that I'm not seeing? Is there another class besides WebBrowser that is more Service-friendly and would do the trick?
In Summary
Geez, this is long. Sorry! It would help to even have a high level recommendation for a better way to do what I'm trying to do, because nothing I've tried has worked.
Update 10/22/09
Well, I think I'm closer, but I'm stuck again. I should end up with a decent-sized zip file with several files in it, but the zip file resulting from my code is empty. Here's my code:
// build post request
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetHref);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// encoding to use
Encoding enc = Encoding.GetEncoding(1252);
// build post string containing authentication information and add to post request
string poststring = "returnUrl=" + fixCharacters(targetDownloadFileUrl);
poststring += getUsernameAndPasswordString();
poststring += "&login2.x=0&login2.y=0";
// convert to required byte array
byte[] postBytes = enc.GetBytes(poststring);
request.ContentLength = postBytes.Length;
// write post to request
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
// get response as stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// writes stream to zip file
FileStream writeStream = new FileStream(fullZipFileName, FileMode.Create, FileAccess.Write);
ReadWriteStream(responseStream, writeStream);
response.Close();
responseStream.Close();
The code for ReadWriteStream looks like this.
private void ReadWriteStream(Stream readStream, Stream writeStream)
{
// taken verbatum from http://www.developerfusion.com/code/4669/save-a-stream-to-a-file/
int Length = 256;
Byte[] buffer = new Byte[Length];
int bytesRead = readStream.Read(buffer, 0, Length);
// write the required bytes
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = readStream.Read(buffer, 0, Length);
}
readStream.Close();
writeStream.Close();
}
The building of the post string is taken from my previous forms app that works. I compared the resulting values in poststring for both sets of code (my working forms app and this one) and they're identical.
I'm not even sure how to troubleshoot this further. Anyone see anything obvious as to why this isn't working?
Conclusion 10/23/09
I finally have this working. A couple of important hurdles I had to get over. I had some problems with the ReadWriteStream method code that I got online. I don't know why, but it wasn't working for me. A guy named JB in Claudio Lassala's Virtual Brown Bag meeting helped me to come up with this code which worked much better for my purposes:
private void WriteResponseStreamToFile(Stream responseStreamToRead, string zipFileFullName)
{
// responseStreamToRead will contain a zip file, write it to a file in
// the target location at zipFileFullName
FileStream fileStreamToWrite = new FileStream(zipFileFullName, FileMode.Create);
int readByte = responseStreamToRead.ReadByte();
while (readByte != -1)
{
fileStreamToWrite.WriteByte((byte)readByte);
readByte = responseStreamToRead.ReadByte();
}
fileStreamToWrite.Flush();
fileStreamToWrite.Close();
}
As Will suggested below, I did have trouble with the authentication. The following code is what worked to get around that issue. A few comments inserted addressing key issues I ran into.
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest firstRequest = (HttpWebRequest)WebRequest.Create(targetHref);
firstRequest.AllowAutoRedirect = false; // this is critical, without this, NLM redirects and the whole thing breaks
// firstRequest.Proxy = new WebProxy("127.0.0.1", 8888); // not needed for production, but this helped in order to debug the http traffic using Fiddler
firstRequest.Method = "POST";
firstRequest.ContentType = "application/x-www-form-urlencoded";
// build post string containing authentication information and add to post request
StringBuilder poststring = new StringBuilder("returnUrl=" + fixCharacters(targetDownloadFileUrl));
poststring.Append(getUsernameAndPasswordString());
poststring.Append("&login2.x=0&login2.y=0");
// convert to required byte array
byte[] postBytes = Encoding.UTF8.GetBytes(poststring.ToString());
firstRequest.ContentLength = postBytes.Length;
// write post to request
Stream postStream = firstRequest.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length); // Fiddler shows that post and response happen on this line
postStream.Close();
// get response as stream
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
// create new request for new location and cookies
HttpWebRequest secondRequest = (HttpWebRequest)WebRequest.Create(firstResponse.GetResponseHeader("location"));
secondRequest.AllowAutoRedirect = false;
secondRequest.Headers.Add(HttpRequestHeader.Cookie, firstResponse.GetResponseHeader("Set-Cookie"));
// get response to second request
HttpWebResponse secondResponse = (HttpWebResponse)secondRequest.GetResponse();
// write stream to zip file
Stream responseStreamToRead = secondResponse.GetResponseStream();
WriteResponseStreamToFile(responseStreamToRead, fullZipFileName);
responseStreamToRead.Close();
sl.logScriptActivity("Downloading update.");
firstResponse.Close();
I want to underscore that setting AllowAutoRedirect to false on the first HttpWebRequest instance was critical to the whole thing working. Fiddler showed two additional requests that occurred when this was not set, and it broke the rest of the script.
You're trying to use UI controls to do something in a windows service. This will never work.
What you need to do is just use the WebRequest and WebResponse classes to download the contents of the webpage.
var request = WebRequest.Create("http://www.google.com");
var response = request.GetResponse();
var stream = response.GetResponseStream();
You can dump the contents of the stream, parse the text looking for updates, and then construct a new request for the URL of the file you want to download. That response stream will then have the file, which you can dump on the filesystem and etc etc.
Before you wonder, GetResponse will block until the response returns, and the stream will block as data is being received, so you don't need to worry about events firing when everything has been downloaded.
You definitely need to re-think your approach (as you've already begun to do) to eliminate the Forms-based application approach. The service you're describing needs to operate with no UI at all.
I'm not familiar with the details of System.WebClient, but since it
provides common methods for sending
data to and receiving data from a
resource identified by a URI,
it will probably be your answer.
At first glance, WebClient.DownloadFile(...) or WebClient.DownloadFileAsync(...) will do what you need.
The only thing I can add is that once you have scraped your screen and have the fully qualified name of the file you want to download, you could pass it along to the Windows/DOS command 'get' which will fetch files via HTTP. You can also script a command-line FTP client if desired. It's been a long time since I tried something like this in Windows, but I think you're almost there. Once you have fetched the correct file, building a batch file to do everything else should be pretty easy. If you are more comfortable with Unix, google "unix services for windows" just keep an eye on the services they start running (DHCP, etc). There are some nice utilities which will let your treat dos as a unix-like shell (ls -l, grep, etc) Finally, you could try another language like Perl or Python but I don't think that's the kind of advice you were looking for. :)