We've developed a MassTransit based demo which is working well as long as all processes run on the same server.
However, as sonn as my receiver wants to subscribe himself at another machine it hangs for a while and afterwards we receive the following exception:
"System.InvalidOperationException: Timeout waiting for subscription service to respond."
Checked already: Firewall rules for MSMQ (inbound and outbound), network, etc.
What could have gone wrong?
Subscription queues on the other machine are private? Is this a problem?
Do we have to change the address format in some special way? msmq://computerName/queueName not ok for remote connections?
Looks like we have forgotten some tiny thing, as nobody else had this problem before...
The most likely thing is MT assumes somethings about remote queues - that they are transactional. Local queues can be queried to discover that, but remote queues you cannot. I would add ?tx=false to the end of your remote queue URI if you aren't using transactional queues for the subscription service.
Next, double check the outgoing queues on the local machine. Are the message stuck there or did they disappear? If you are using transactional queues, can the machines enroll in a DTC transaction together?
Answering your question, all queues are private. This is not a problem, they are still remotely addressable.
I hope this helps get you further. After that, I would consider joining the mailing list and posting your questions there: https://groups.google.com/forum/?fromgroups#!forum/masstransit-discuss
For reference: the problem has been a wrong URL in the receiver queue, the receiver queue always resides on the local system of course. Sorry for any inconvenience.
Related
I have an Ubuntu VM installed on a client's VMware system. Recently, the client's IT informed us that his firewall has been detecting consistent potential port scans to our VM's internal IP address (coming from 87.238.57.227). He asked if this was part of a known package update process on our VM.
He sent us a firewall output where we can see several instances of the port scan, but there are also instances of our Ubuntu VM trying to communicate back to the external server on port 37258 (this is dropped by the firewall).
Based on a google lookup, the hostname of the external IP address is "feris.postgresql.org", with the ASN pointing to a European company called Redpill-Linpro. As far as I can tell, they offer IT consulting services, specializing in open source software (like PostgreSQL, which is installed on our VM). I have never heard of them before though and have no idea why our VM would be communicating with them or vice-versa. I'm also not sure if I'm interpreting the IP lookup information correctly: https://ipinfo.io/87.238.57.227
I'm looking for a way to confirm or disprove that this is just our VM pinging for a standard postgres update. If that's the case I'd like to restrict this behaviour. We would prefer to do these types of updates manually and limit the communication outside of the VM to what is strictly necessary for the functionality of our application.
Update
I sent an email to Redpill's abuse account. They responded quickly saying that the server should not be port scanning anyone and if it appears that way, something is wrong.
The server is part of a cluster of machines that serves apt.postgresql.org among other postgres download sites. I don't think we have anything like ansible or puppet installed that would automatically check for updates but I will look into that to make sure. I'm wondering if Ubuntu reaching out to update the MOTD with the number of available packages would explain why our VM is trying to reach out to the external postgres server?
The abuse rep said in any case there should only be outgoing connections from the VM, not incoming. He asked for some additional info so I will keep communicating with him and try to update this post accordingly
My communication with the client's IT dropped off so I did not get a definitive answer on this, but I'll provide some new details:
I reached out to the abuse email for Redpill-Linpro. He got back to me and confirmed the server corresponding to the detected IP address is part of a cluster that hosts postgres download sites, including apt.postgresql.org. He was surprised to learn we had detected a port scan from their server and seems eager to figure out why that is happening.
He asked if the client IT could pass along some necessary info for them to set up tracking on that server. But the client IT never got back to me. I think he was satisfied that it wasn't malicious and stopped pursuing it.
Here's one of the messages the abuse rep sent me that may be relevant:
That does look a lot like the tcp to the apt download server yes. It's
strange that your firewall reports that many incoming connections, but
they could be fallout from some connection tracking that's not
operating as intended. The timing appears to be matching up more or
less perfectly. And there should definitely not be any ping-back
connections from it.
Since you appear to be using the http version of the server (and not https) bringing the data in cleartext, they should be able to just
dump the TCP connection contents and verify exactly what it does. But
I bet they are going to see a number of http requests initiated by the
apt client that is checking for updates.
I'm using Rebus (https://github.com/rebus-org (v.0.83)) and until now it's been all local to a single machine. Now I need to use a remote queue from my website to an app server. It's not abundantly clear to me how to set this up with Rebus. A few questions
I guess I need MSMQ on both machines (web & app) correct? I've configured the web site to UseMsmqInOneWayClientMode;
or is there a way to specify to send it over http?
My configuration looks like this more or less. I'm guessing that inputQUeue needs to point to the local machine not the remote one right?
<rebus inputQueue="mywebqueue" errorQueue="MyErrorQueue#mymachine" workers="1" maxRetries="5">
<endpoints>
<add messages="MyLibrary.CreateMessage, MyLibrary" endpoint="MyQueue#mymachine"/>
</endpoints>
</rebus>
Help would really be appreciated.
Yes. All machines that need to use MSMQ somehow, need to have MSMQ installed. Even as a one-way client, like your webserver, because MSMQ achieves its high availability by providing outgoing queues when you send to remote machines.
I think so. I've never used MSMQ beyond its basic reliable messaging capabilities. Google around, I bet you can find something ;)
Your configuration looks right. And yes - input queues are always local, whereas queues you send to (error queue and all endpoints specified in the endpoint mappings) can be remote too.
I am aware that the Tibco EMS provides Fault Tolerance in a hot backup configuration on the server side as detailed in the User's Guide, this answer and here.
But on the client side does Tibco EMS provide out of the box solution for fault-tolerant clients?
An example: on the topic Sports.F1.PitStop two clients (server1, server2) register as publishers. The idea being that should something go wrong on server1 (i.e. publisher on server1 goes down), server2 would seamlessly continue to publish on the topic. So the question is, does Tibco EMS provide such client-side fault-tolerance capability?
No.
EMS (or JMS) does not support a client-side fault tolerance feature. The reason is simple : typically, publishers processes don't know each other.
To elaborate:
Topics usually accepts many publishers (more than two).
In a pub/sub scenario, publishers don't know subscribers, and to an extent they don't typically know the other publishers.
Your solution:
My first question regarding your solution: Why can't both servers publish messages at the same time ?
I assume you have a good reason (like messages from server1 and server2 being redundant). In that case, then you will have to have some kind of communication between your "active" and "passive" server.
Possibility 1 : Server2 is connected to a simple service/rmi/other heartbeat mechanism, and can tell if/when Server1 as stopped publishing.
Possibility 2 : Server2 is itself subscribed to the topic, and can tell when messages have stopped.
Last note:
In case you meant shared "subscriptions" (as in, one durable subscription being shared between two servers): the new JMS 2.0 API is supporting this feature. EMS 8 is the only version of EMS supporting JMS 2.0.
I'm not sure if I fully understand your question. EMS is a message broker. It brokers messages. Thats all it does. If you have multiple servers publishing to the same topic, then that is fine for EMS. If your publishers themselves are in a fail-over configuration, so that only one is actively sending and the other takes over when the first one fails, then EMS doesn't care.
Managing the fail-over mechanism from one publisher to the other, that is something you have to develop yourself. EMS doesn't offer anything to support that. You need some kind of mechanism for server2 to know when server1 is down. There are several ways you can do that:
Use a heartbeat mechanism where the active server sends periodical heartbeat messages (possibly across EMS as well) and the passive server listens to them and when they stop, assume server1 is down.
Use an active open connection between server1 and server2 (open a simple tcp port for example and stop it from closing automatically by periodically sending a dummy message), and have an open read on that port. You'll get a connection error the moment server1 goes down and you don't have to wait for timeouts on heartbeats.
Use a third party monitoring tool, such as openview, tivoli or nagios to detect server1 is down and inform server2.
There is a feature on EMS you can use. EMS has system admin topics that you can subscribe to and that log basically anything that goes on internally. You can subscribe to connect and disconnect events to monitor any component connecting to or disconnecting from EMS. If server1 fails, this will be visible as a disconnect event.
heartbeat mecanism:
https://support.tibco.com/s/article/Tibco-KnowledgeArticle-Article-33918
For example:
client_heartbeat_server=10
client_timeout_server_connection=35
server_heartbeat_client=10
server_timeout_client_connection=35
If I am using MSMQ over the web, what happens if the network connection is lost between client and server?
So can you still add messages to the Queue, and if so where are they stored? In the client app, or in the clients OS etc?
For instance if I have a windows service which is adding items to a queue in a different country. What if the network connection is lost, and the windows service is restarted. Do the messages get lost forever?
The other part of the question relates to the route that a message takes, is it sent directly to the receiving queue, or is it written into a queue on the client side? Does that require MSMQ to be installed on the sending server, and how about licensing for that?
Is there any good documentation to explain the required setup?
Update: Regarding your follow up question. Yes you have to install msmq on the sending server. There aren't any licensing cost, because MSMQ is part of windows and not a separate software (just like the IIS). Here is documentation on "Setting Up a Message Queue" on windwos 2003.
Before Update: Outgoing message are stored in the outgoing queue of the sending server. They are not lost if the sending service is restarted. They will wait in the outgoing queues ( which can be inspected with the msmq manager ) for I don't know how long.
if the msmq service or the sending server are restarted. Then "express" messages will be lost. express or recoverable are properties of non-transcriptional messages.
I need to develop a server side application that opens sockets and manages communication with multiple clients. Previous answers have told me this is possible using a single script file, which loops forever.
Is this possible using only a PHP/Perl/Python hosting service? or would I need a VPS or shell access?
Any help is appreciated since I've never worked with sockets before. Thanks for your time.
Cheap Perl/PHP hosting services don't want you running your own long-running processes.
This means you will need a VPS (which obviously includes shell account since you can do anything you want on your private server). A few VPS providers might block outgoing IRC port but I think that is rare.
Linode and Slicehost/Rackspace are just two examples very very well run VPS service providers and I guarantee you can run your own socket application on them.
It would make your host very unhappy since their CPU time is valuable! If you use shared hosting, your host might just kick you out for such a solution! (Read your contract for the fine details.)
I think it could be possible but it depends on the setup of your host, plus the permissions your host are granting you. And most will be unhappy about anything that runs forever. (They prefer to see just short, simple applications.)
Usually the service firewall will block any unexpected ports, or if they are not doing it now they will start doing it after they figure out what you are doing and decide they don't like it.
I would say no because it involve too much security problems