I am getting started with OPCUA and eclipse milo and I am trying to understand how to best inform the client that a value or node has changed in the server.
So far my guess is that I need to trigger an event in the node that has changed, and then the client should monitor/subscribe to events in such node. Am I right on this?
If my understanding is correct, which event is most appropriated to trigger for this purpose?
I am using a free UI OPCUA client to test my server changes, and I need to refresh manually to observe my changes. I was expecting that by triggering the correct (OPCUA standard) event I would indicate the client to refresh automatically, is this possible?
Thanks!
You don't need Events to notify a client of an attribute change - that's the entire point of Subscriptions and MonitoredItems.
The client creates a MonitoredItem for the Value attribute (or any other attribute) and the server will report changes when that attribute changes.
As far as what you need to do as a user of the Milo Server SDK - see the ExampleNamespace. Your namespace implements the onDataItemCreated and other related methods to be notified that a client has created a MonitoredItem and you should start sampling values for it.
I am working on a desktop application that requires synchronization between several clients. Basically, a group of people (let's say between 2 and 10) all run the same application. One of them hosts a server and the other clients connect to that server. The client that hosts the server also connects to his own server.
The applications should stay synchronized between all clients, meaning all clients see the same data in the application. Specifically, the data in question I can define in two separate forms:
A simple property with a certain value (this value must stay synchronized)
A list of properties (the items in the list and their values must stay synchronized)
Simple examples of (1) could be: which item in a list does the client currently have selected, and what's the current location of the client's mouse pointer within the application window. These properties keep changing continuously but the number of these properties is constant and does not grow (e.g. defined during design time).
An example of (2) could be a list of chat messages. These lists will grow during runtime with no way to predict how many items there will be.
Here is an example code in C# for the state, client and chat messages:
public class State
{
// A single value shared between all clients
public int SimpleInteger {get;set;}
// List of connected clients and their individual states
public List<Client> Clients {get;set;}
// List of chat messages
public List<ChatMessage> Messages {get;set;}
}
public class Client
{
public string ClientId {get;set;}
public string Username {get;set;}
public ClientState ClientState {get;set;}
}
public class ClientState
{
public string ClientId {get;set;}
public int SelectedIndex {get;set;}
public int MouseX {get;set;}
public int MouseY {get;set;}
}
public class ChatMessage
{
public string ClientId {get;set;}
public string Message {get;set;}
}
I've been working on this on and off for a long time but whatever kind of state synchronization I came up with, it never worked well.
When I search for solutions, I only ever find solutions for games, but those are not very helpful because my requirements are different:
I cannot deal with "dropped updates", I cannot predict (interpolate or extrapolate) what the other clients are doing. Every client needs to receive every update to stay in sync.
On the other hand, I don't care about lag (within reason). It is fine if I see the updates of other client with about a second delay.
When a new client connects (or reconnects), a large portion of the state must be transfered (for example: the list of chat messages from example 2). Each client is required to know about the entire history of the chat so this must be downloaded when a client connects.
My current solution can be summarized as follows:
The server keeps track of the state, e.g. the source of truth.
The state contains the properties that require synchronizing.
The state also contains a list of connected users (and their usernames etc).
Clients also each keep a local copy of the state, which they can act upon immediately. For example, they update their mouse position in their local state continously.
Whenever a client updates his local state, this update is sent to the server.
Potential exceptions here are things that change too fast such as the mouse position, those I will only send in regular intervals.
The server also updates the common "source of truth" state.
Finally, the server updates all other clients with the new updated state.
The last two steps are where I'm struggling. I can think of two methods to synchronize the state, one is easy but probably not efficient and the other is efficient but prone to errors.
The server simply sends the entire state to all clients.
As soon as the server receives an update from the client, the update is applied to the state and the new state is broadcasted. Every other client replaces their local state.
I feel this will probably work, but the state can grow in size quickly due to the "list" items (for example chat messages). In my previous attempts, this quickly became a problem and sending the state back become much too slow.
The server re-sends the same update (that it received) to all other clients.
Each client then only applies the new update to their state locally to sync back with the server.
This is probably much more efficient and sending the entire state is only necessary when a client connects.
However, in the past I frequently ran into desync issues where clients were no longer in sync. I don't really know what caused it, probably conflicts between messages (for example server telling the client to update a value in the state, but the client just updated his local value, which has precedence?). After this happens, everything went completely wrong as the updates are now being applied to two different states and have different outcomes.
I'm looking for some guidance on general concepts on how to achieve this. I'm using several messaging libraries to achieve the actual communication between client and server and that part is not an issue I think. I can make sure in these libraries that every message is received for example (though I'm not sure if the order is guaranteed). Like I said before, lag is not an issue, but I must guarantee every state update is received both by the server and by every other client.
Any help would be great! Thanks.
This is a hard problem and there are enough tricky areas that I wouldn't want to build this myself. Authentication, conflicting updates, API management, network outages, single point of failure, and local persistence come to mind.
If you're up for using a cloud-based solution, Google Cloud Firestore takes care of those tricky areas and does what you need:
Clients save data to the database, by creating, updating, or deleting records. Example code.
Whenever a record is created, updated, or deleted, all clients get realtime notifications. Example code.
(After you follow the links above, make sure you click C# above the code boxes to see the C# code).
This is a complicated issue, with many moving parts, as you seem to understand. As I've been researching this, I've read a couple comments on questions like this one on a variety of Q&A sites, stating this kind of thing is a project all on it's own.
Disclaimer: I haven't done this myself, so I don't know how well this would work, but maybe you can take my suggestions and work with them, if you haven't already done so. I've worked on projects where this was implemented, but I wasn't part of that implementation directly.
Connection
Since you haven't said which library you are using for the connection, I'm going to assume you are using websockets or something similar. If not, I suggest you move to something like websockets. It allows for a (near) constant connection between client and server so that data can be pushed both directions, avoiding the client from having to poll and pull the data. The link below seems to have a decent walk-though on how to do it, so I won't try to. Because links die, here's the first example code they give, which seems pretty simple.
​using System.Net.Sockets;
using System.Net;
using System;
class Server {
public static void Main() {
TcpListener server = new TcpListener(IPAddress.Parse("127.0.0.1"), 80);
server.Start();
Console.WriteLine("Server has started on 127.0.0.1:80.{0}Waiting for a connection...", Environment.NewLine);
TcpClient client = server.AcceptTcpClient();
Console.WriteLine("A client connected.");
}
}
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_server
Client start up
Once you have a stable connection between server and client, you need to make sure the data is in sync. When the user starts the app, you can get the timestamp of the latest change in each table and compare that to the server. If they are exactly the same, you have a somewhat reasonable expectation that the table hasn't changed. I'm assuming each table has a column containing the timestamp for the last edit made to the row.
For the tables that have changed, you can have the server send the new and updated rows to the client based on the client's "last changed timestamp".
Since the internet isn't 100% guaranteed to be connected, you will also need to keep track of the times the client has been connected vs. when they've been on the app (unless the app just won't work without being connected to the server). This information also needs to be sent to the server to compare to data changed during intervals where the client hasn't been connected.
Once timestamp matching has been done, you need to compare the row counts. If they match, you can more reasonably assume the tables are the same. If they aren't, you can see about matching ID/primary keys. There's a variety of different ways to do this, including 1:1 matching (which is slowest but most reliable), or you can do some math with the IDs (assuming numerical IDs) and try to see what's different in batches of 100 rows (for example). Idea: If adding the sorted, auto-increment integer IDs for the first 100 rows is the same on the client and the server, all those rows exist on both servers, but if it doesn't match, you can try the 1:1 match to see what's missing. Because this can be lengthy for large databases, you may want to track this type of sync in another table, so it doesn't need to be done all the time.
Instead, you may want a table to track all the data not sent to a client. This would require a confirmation that the data sent was correctly inserted into the client DB. This could also work on the client side to track what hasn't been sent to the server. Of course, this kind of thing can get cumbersome quickly, even if you're just tracking keys, table names, and timestamps. You can rack up millions of rows quickly, if you don't remove old data periodically. This is why I suggest tracking unsent data, so that anything that becomes "sent" is no longer tracked by this table and removed.
If you don't want to code and manage all that, you can try for a library that does it. There are a variety out there. Even Microsoft has one, but it's on extended support to only 1/1/2021. What happens after that, I doubt even Microsoft knows, but it gets you 1.25 years to come up with a different solution.
Creating Synchronization Providers With The Sync Framework
The Sync Framework can be used to build apps that synchronize data from any data store using any protocol over a network. We'll show you how it works and get you started building a custom sync provider.
https://learn.microsoft.com/en-us/previous-versions/sql/synchronization/mt490616(v=msdn.10)
https://support.microsoft.com/en-us/lifecycle/search?alpha=Microsoft%20Sync%20Framework%202.1
Normal runtime
Once you have your data synced on startup (or in the background after startup), you can simply send the data to the server normally, as in when the user makes changes. Since you'll have a websocket type connection, any changes the server gets from other clients will be able to be pushed to all the other clients.
As far as changing the data in real time in your app, you may have to be constantly polling your local/client DB for timestamp changes so the UI can be appropriately updated. There may be something within C# that does this for you or another library you can find.
Conclusion
At this point, I'm out of ideas. It seems reasonable to me this would work, even though it's a lot of work. Hopefully you can take what I have and use it as a foundation to your own ideas on how to accomplish your task. It seems there's a lot of work ahead of you, so good luck!
Footnote
As I'm currently the only answer after several days of it being unanswered, I'm going to assume no one else has anything better to suggest. If they do, I'd encourage them to make their own answer instead of complaining about mine. People tweaking this answer is expected, but please remember community standards when making comments.
I'm only answering this because I haven't seen anyone else do it on this or other sites. It's only been bits and disconnected pieces here & there, with people still not being able to make sense of it as a whole.
This and similar questions have been asked before on this site and closed as "too broad". If you feel this same way as a reader, please vote so on the Question not this answer.
There are several solutions to your problem.
You could use a BizTalk server out-of-the box. This may not be what you have in mind.
If you want something more home-brewed, you could use WCF (Windows Communication Foundation) with MSMQ (Microsoft Message Queue). This would give you guaranteed message delivery, and durable messages (if you want). You would not have to worry about lost connections, and other errors occurring during messages transmission.
You can go down another level and use direct TCP and UDP protocols to transmit messages. But now, you have to take care of more error cases.
Any SQL DBMS implements one important part of your problem statement: it maintains shared state. Consider what ACID promises:
Consistency. At any one instant, all clients reading from the database are guaranteed to see the same information.
Atomicity. The client updating the database can use as many steps as needed. When the transaction is committed, the data are changed entirely or not at all.
Isolation. The server gives each client the illusion of interacting with it alone. It handles concurrent updates, and updates the database as though the updates arrived serially.
You may not care about durability for this application.
The mediation among the clients is, for my money, the most useful feature of the DBMS for your application. That will save you work, and headaches. Another, non-obvious, benefit is that it can enforce consistency rules for the state information; that can be remarkably useful to prevent an obsolete/corrupt client from munging the shared state.
The second part of your problem statement is notifying 2-10 clients of changed state. There are any number of ways to do that.
Some DBMSs can access OS services from triggers. You could have an update trigger issue a notification. Alternatively, the updating client could do that.
The actual notification mechanism could be quite simple. Clients could connect to a server (that you write) and block on read(2). The server itself listens on a port for update notifications. On receipt of one, it repeats it to all connected clients. When the client's read request returns, it's time to query the database for the updated state, and post a new read.
To prevent a kind of "thundering herd" problem when several updates arrive back-to-back, when a client reads the update message, it could keep reading updates until EWOULDBLOCK, and only then query the DBMS. OTOH, if it's important to see the intermediate states (to see every update, not just the current state), the DBMS is perfectly capable of storing and providing all versions and distinguishing them with a timestamp or serial number.
If you don't want to use TCP sockets directly, you might prefer ZeroMQ.
In this design, each client has three connections: the DBMS, the read-notify socket, and (maybe) the server-notify socket. The server has N+1 connections, for N clients and one listening socket. You have no locks to implement, very little tracking of participation, no problems re-synchronizing, and short windows inconsistency among clients as each one acts on its notification.
When started charles, java app cannot access redis got below error
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
Then I tried to ignore the redis connection to solve it
but the problem still exists
So how to explicitly ignore some connection, e.g redis connection, mongo connection etc. ?
I'm sorry I don't really know what your real problem is. I guess the problem that you have is those HTTP requests still appearing in the Structure view, right?
Being that the case, I would strongly recommend you to use the Focus feature. To use it, you only need to add the domain you are working on to the View -> Focused Hosts... ( you can also do it by right clicking on the request and then selecting "Focus").
By doing this, all the non-focused domains will get grouped in a "Other hosts" entry in the Structure panel so they won't disturb your work anymore.
I'm building an iOS app, but the problem exists on all clients. iChat, Messages, Psi, etc. So because it exists on all clients I'm going to assume it's a server issue.
Has anyone ever experienced something like this? If so, what did you do to fix it? I'm sure it's some silly config setting or something but I simply can't figure this out. This is the only thing that looks like it might be related in ejabberd.log:
=ERROR REPORT==== 2012-09-05 12:07:12 ===
Mnesia(ejabberd#localhost): ** WARNING ** Mnesia is overloaded: {dump_log,
time_threshold}
Thanks in advance for any tips/pointers.
https://github.com/processone/ejabberd/blob/master/src/ejabberd_c2s.erl#L936 seems to have already been patched. The config variable is called resource_conflict and the value you want is setresource.
The above warning is (probably) not related to the issue you are facing. These mnesia events usually happens when the transaction log needs to be dumped, but the previous transaction log dump hasn't finished yet.
Problem that you are facing needs to be debugged for which you can set {log_level, 5} inside ejabberd.cfg. This will enable debug logging for ejabberd. Then look into the logs to find any guesses on why this is happening for you. Also, come back and paste your log file details here, probably we will be able to help you further. I have never faced such non-sensical issues with ejabberd.
Update after log file attachment:
As Joe wrote below, this is indeed happening because of resource conflict. Two of your clients are trying to login with same resource value. But in an ideal world this shouldn't matter. Jabber servers SHOULD take care of this by appending or prepending custom value on top of resource value requested by the client.
For example, here is what gtalk (even facebook chat) servers will do:
SENT <iq xmlns="jabber:client" type="set" id="1"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><resource>jaxl#resource</resource></bind></iq>
RCVD <iq id="1" type="result"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><jid>jabberxmpplibrary#gmail.com/jaxl#resou27F46704</jid></bind></iq>
As you can see my client requested to bind with resource value jaxl#resource but gtalk server actually bound my session with resource value jaxl#resou27F46704. In short, this is not a bug in your client but a bug in ejabberd.
To fix this you can do two things:
Resource value is probably hardcoded somewhere in your client configuration. Simply remove that. A good client will automatically take care of this by generating a random resource value at it's end.
Patch ejabberd to behave how gtalk server does (as shown above). This is the relevant section inside ejabberd_c2s.erl src which needs some tweaking. Also search for Replaced by new connection inside the c2s source file and you will understand what's going on.
This sounds like the "dueling resources" bug in your client. You may have two copies of your client running simultaneously using the same resource, and doing faulty auto-reconnect logic. When the second client logs in, the first client is booted offline with a conflict error. The first client logs back in, causing a conflict error on the second client. Loop.
Evidence for this is in your logfile, on line 3480:
D(<0.373.0>:ejabberd_c2s:1553) : Send XML on stream =
<<"<stream:error><conflict xmlns='urn:ietf:params:xml:ns:xmpp-streams'/>
<text xml:lang='en' xmlns='urn:ietf:params:xml:ns:xmpp-streams'>
Replaced by new connection
</text>
</stream:error>">>
I am trying to write a VPN app using VpnService. I started my app based on the sample ToyVpn. It seems to work fine but I am wondering if there is a way to get rid of the pop up dialog when I click connect. I am hoping that I could just click "connect" and it would start without having to click the "I trust this application..." check box and "Ok".
Thanks.
I don't think it's possible. They seem to be very careful about this class. If you take a look at the documentation you can see it says:
Letting applications intercept packets raises huge security concerns. A VPN application can easily break the network. Besides, two of them may conflict with each other. The system takes several actions to address these issues. Here are some key points:
User action is required to create a VPN connection. [emphasis mine]
There can be only one VPN connection running at the same time. The existing interface is deactivated when a new one is created.
A system-managed notification is shown during the lifetime of a VPN connection.
A system-managed dialog gives the information of the current VPN connection. It also provides a button to disconnect.
The network is restored automatically when the file descriptor is closed. It also covers the cases when a VPN application is crashed or killed by the system.
Since it says that user action is required to create the VPN connection, I assume they mean this is something you cannot control yourself.