I'm seen System.Net.Sockets.SocketException: Connection reset by peer error doing the call to retrieve groups using TransitiveMemberOf method from Microsoft.Graph.dll.
public async Task<List<AdGroup>> GetAllGroupsByUserId(string userId, CancellationToken token)
{
_client = GraphConnection.GetGraphClient(_config);
var memberships = new List<AdGroup>();
try
{
IUserTransitiveMemberOfCollectionWithReferencesRequest? next = null;
do
{
IUserTransitiveMemberOfCollectionWithReferencesPage data;
if (next == null)
{
var id = Guid.NewGuid().ToString();
var options = new List<Option>()
{
new QueryOption("$top", "999"),
new HeaderOption("client-request-id", id)
};
data = await _client.Users[userId].TransitiveMemberOf.Request(options).Select("id,displayName").GetAsync(token);
}
else
{
data = await next.GetAsync(token);
}
next = data.NextPageRequest;
if (data.CurrentPage.Count == 0)
break;
var foundGroups = data.CurrentPage
.Select(i =>
{
if (i is Group group)
return new AdGroup()
{
Id = group.Id,
Name = string.IsNullOrWhiteSpace(group.DisplayName) ? group.AdditionalData?["displayName"].ToString() ?? "" : group.DisplayName
};
if (i is DirectoryRole role)
return new AdGroup()
{
Id = role.Id,
Name = string.IsNullOrWhiteSpace(role.DisplayName) ? role.AdditionalData?["displayName"].ToString() ?? "" : role.DisplayName
};
return null;
})
.Where(g => g != null && Constants.CONTAINS_LIST.Any(c => g.Name.Contains(c, StringComparison.InvariantCultureIgnoreCase)));
if (foundGroups != null)
memberships.AddRange(foundGroups);
} while (next != null);
}
catch (Exception e)
{
_logger.LogError(e, "GetAllGroupsByUserId error");
}
return memberships;
}
I tried to run the code from Azure VM ( I have Linux OS there) as well as from my local machine (Windows OS). Now I'm checking the load of VM because the code is running in threads and thinking about what can be wrong.
I've also created a ticken in Azure Portal and I have the following answer from the Microsoft:
This error is not coming from Microsoft Graph rather from your clients
socket (otherwise meaning a network error)
For these errors, you must start with your networking team to
investigate. There could be a number of networking reasons i.e. your
Firewall, Proxy, Loadbalancer, network bandwidth throttling.
That answer didn't help me a lot, but I'm trying to investigate futher.
I will appreciate any help, thanks
In case if someone face the similar issue, I posted the information which helped me to resolve it:
Increase the number of cores on VM from 4 to 16
Decrease the number of tasks from 200 to 100 which were doing the call to MS Graph and some logic
Initialize a new MS Graph client on each iteration inside of the method GetAllGroupsByUserId
Related
We have two .net services (.Net core console applications) which are accessing a postgres db table.
Service 1 inserts some 500 rows every 1 minute. It runs as a background thread.
Service 2 reads data from the same table continuously. There is an MQTT publisher which keeps reading data from this table when any new data is requested. This also happens very frequently i.e atleast 4/5 times a minute.
We are getting "FATAL: sorry, too many clients already " error.
What I am assuming is since write and read is happening simultaneously too frequently, the connection is not getting dispose properly.
Is there a way to avoid read whenever a write is happening.
EDITED
Thanks for the reply.. I know some connection pooling is happening but not sure where.. so my question was how to avoid concurrent access of postgres db..
Was not sure what part of code I can post to make the question clear
I am having using clause on dbcontext and also disposed like the below..
This is retrieval section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
var data = platinumDBContext.TrendPoints.Where(x => ids.Contains(x.TrendPointID) && x.TimeStamp >= DateTime.Now.AddHours(-timeinHours));
result = data.Select(x => new Last24hours
{
Label = x.TrendPointID.ToString(),
Value = (double)x.TrendPointValue,
time = x.TimeStamp.ToString("MM/dd/yyyy HH:mm:ss")
}).ToList();
}
catch (Exception oE)
{
}
finally {
platinumDBContext.Dispose();
}
}
This is the insertion section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
foreach (var point in trendPoints)
{
if (point != null)
{
TrendPoint item = new TrendPoint();
item.CreatedDate = DateTime.Now;
item.ObjectState = ObjectState.Added;
item.TrendPointID = point.TrendID;
item.TrendPointValue = double.IsNaN(point.Value) ? decimal.MinValue : (decimal)point.Value;
item.TimeStamp = new DateTime(point.TimeStamp);
platinumDBContext.Add(item);
}
}
platinumDBContext.SaveChanges();
}
catch (Exception ex)
{
}
finally
{
platinumDBContext.Dispose();
}
}
Regards,
Geervani
I am building a OPC UA Client using OPC Foundation SDK. I am able to create a subscription containing some Monitoreditems.
On the OPC UA server these monitored items change value constantly (every second or so).
I want to disconnect the client (simulate a connection broken ), keep the subcription alive and wait for a while. Then I reconnect having my subscriptions back, but I also want all the monitored Item values queued up during the disconnect. Right now I only get the last server value on reconnect.
I am setting a queuesize:
monitoredItem.QueueSize = 100;
To kind of simulate a connection error I have set the "delete subscription" to false on ClosesSession:
m_session.CloseSession(new RequestHeader(), false);
My question is how to capture the content of the queue after a disconnect/connection error???
Should the ‘lost values’ be “new MonitoredItem_Notification” automatically when the client reconnect?
Should the SubscriptionId be the same as before the connection was broken?
Should the sessionId be the same or will a new SessionId let med keep the existing subscriptions? What is the best way to simulate a connection error?
Many questions :-)
A sample from the code where I create the subscription containing some MonitoredItems and the MonitoredItem_Notification event method.
Any OPC UA Guru out there??
if (node.Displayname == "node to monitor")
{
MonitoredItem mon = CreateMonitoredItem((NodeId)node.reference.NodeId, node.Displayname);
m_subscription.AddItem(mon);
m_subscription.ApplyChanges();
}
private MonitoredItem CreateMonitoredItem(NodeId nodeId, string displayName)
{
if (m_subscription == null)
{
m_subscription = new Subscription(m_session.DefaultSubscription);
m_subscription.PublishingEnabled = true;
m_subscription.PublishingInterval = 3000;//1000;
m_subscription.KeepAliveCount = 10;
m_subscription.LifetimeCount = 10;
m_subscription.MaxNotificationsPerPublish = 1000;
m_subscription.Priority = 100;
bool cache = m_subscription.DisableMonitoredItemCache;
m_session.AddSubscription(m_subscription);
m_subscription.Create();
}
// add the new monitored item.
MonitoredItem monitoredItem = new MonitoredItem(m_subscription.DefaultItem);
//Each time a monitored item is sampled, the server evaluates the sample using a filter defined for each monitoreditem.
//The server uses the filter to determine if the sample should be reported. The type of filter is dependent on the type of item.
//DataChangeFilter for Variable, Eventfilter when monitoring Events. etc
//MonitoringFilter f = new MonitoringFilter();
//DataChangeFilter f = new DataChangeFilter();
//f.DeadbandValue
monitoredItem.StartNodeId = nodeId;
monitoredItem.AttributeId = Attributes.Value;
monitoredItem.DisplayName = displayName;
//Disabled, Sampling, (Report (includes sampling))
monitoredItem.MonitoringMode = MonitoringMode.Reporting;
//How often the Client wish to check for new values on the server. Must be 0 if item is an event.
//If a negative number the SamplingInterval is set equal to the PublishingInterval (inherited)
//The Subscriptions KeepAliveCount should always be longer than the SamplingInterval/PublishingInterval
monitoredItem.SamplingInterval = 500;
//Number of samples stored on the server between each reporting
monitoredItem.QueueSize = 100;
monitoredItem.DiscardOldest = true;//Discard oldest values when full
monitoredItem.CacheQueueSize = 100;
monitoredItem.Notification += m_MonitoredItem_Notification;
if (ServiceResult.IsBad(monitoredItem.Status.Error))
{
return null;
}
return monitoredItem;
}
private void MonitoredItem_Notification(MonitoredItem monitoredItem, MonitoredItemNotificationEventArgs e)
{
if (this.InvokeRequired)
{
this.BeginInvoke(new MonitoredItemNotificationEventHandler(MonitoredItem_Notification), monitoredItem, e);
return;
}
try
{
if (m_session == null)
{
return;
}
MonitoredItemNotification notification = e.NotificationValue as MonitoredItemNotification;
if (notification == null)
{
return;
}
string sess = m_session.SessionId.Identifier.ToString();
string s = string.Format(" MonitoredItem: {0}\t Value: {1}\t Status: {2}\t SourceTimeStamp: {3}", monitoredItem.DisplayName, (notification.Value.WrappedValue.ToString().Length == 1) ? notification.Value.WrappedValue.ToString() : notification.Value.WrappedValue.ToString(), notification.Value.StatusCode.ToString(), notification.Value.SourceTimestamp.ToLocalTime().ToString("HH:mm:ss.fff"));
richTextBox1.AppendText(s + "SessionId: " + sess);
}
catch (Exception exception)
{
ClientUtils.HandleException(this.Text, exception);
}
}e here
I don't know how much of this, if any, the SDK you're using does for you, but the approach when reconnecting is generally:
try to resume (re-activate) your old session. If this is successful your subscriptions will already exist and all you need to do is send more PublishRequests. Since you're trying to test by closing the session this probably won't work.
create a new session and then call the TransferSubscription service to transfer the previous subscriptions to your new session. You can then start sending PublishRequests and you'll get the queued notifications.
Again, depending on the stack/SDK/toolkit you're using some or none of this may be handled for you.
I have a StreamSocket in UWP and I send my messages like this using a DataWriter object using the StoreAsync() method:
public static async Task<bool> SendNetworkMessage(NetworkMember member, NetworkMessage message)
{
DataWriter writer = member.DataWriter;
//Check that writer is not null
if (writer != null)
{
try
{
//Serialize Message
string stringToSend = SerializeObject<NetworkMessage>(message);
//Send Message Length
writer.WriteUInt32(writer.MeasureString(stringToSend));
//Send Message
writer.WriteString(stringToSend);
await writer.StoreAsync();
return true;
}
catch (Exception e)
{
Debug.WriteLine("DataWriter failed because of " + e.Message);
Debug.WriteLine("");
Disconnect(member);
OnMemberDisconnectedEvent(member);
return false;
}
}
else { return false; }
}
All is well, the only problem is that I don't know if a connection went down.
Now I want to check my connection using a DispatcherTimer like this:
private static async void NetworkTimer_Tick(object sender, object e)
{
foreach (NetworkMember member in networkMemberCollection)
{
if (member.Connected == true && member.Disconnecting == false)
{
await SendNetworkMessage(member, new PingMessage());
}
}}
However, this is causing timing issues which is causing ObjectDisposedExceptions on the DataWriter. It seems that the DispatcherTimer thread cannot use the StreamSocket when I send a message from a different thread. My question is: How can I make sure the Ping is sent each time but that SendNetworkMessage operations are done in order instead of overlapping?
Thanks
It seems that the DispatcherTimer thread cannot use the StreamSocket when I send a message from a different thread.
How can I make sure the Ping is sent each time but that SendNetworkMessage operations are done in order instead of overlapping?
It's possible, and I think your code using foreach and await operation can ensure the work of sending message in order.
the only problem is that I don't know if a connection went down.
If you want to know if the connection went down, you can refer to Handling WinRT StreamSocket disconnects (both server and client side).
I'm developing a cloud service (worker role) for collecting data from a number of instruments. These instruments reports data randomly every minute or so. The service itself is not performance critical and doesn't need to be asynchronous. The instruments are able to resend their data up to an hour on failed connection attempt.
I have tried several implementations for my cloud service including this one:
http://msdn.microsoft.com/en-us/library/system.net.sockets.tcplistener.stop(v=vs.110).aspx
But all of them hang my cloud server sooner or later (sometimes within an hour).
I suspect something is wrong with my code. I have a lot of logging in my code but I get no errors. The service just stops to receive incoming connections.
In Azure portal it seems like the service is running fine. No error logs and no suspicious cpu usage etc.
If I restart the service it will run fine again until it hangs next time.
Would be most grateful if someone could help me with this.
public class WorkerRole : RoleEntryPoint
{
private LoggingService _loggingService;
public override void Run()
{
_loggingService = new LoggingService();
StartListeningForIncommingTCPConnections();
}
private void StartListeningForIncommingTCPConnections()
{
TcpListener listener = null;
try
{
listener = new TcpListener(RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WatchMeEndpoint"].IPEndpoint);
listener.Start();
while (true)
{
_loggingService.Log(SeverityLevel.Info, "Waiting for connection...");
var client = listener.AcceptTcpClient();
var remoteEndPoint = client.Client != null ? client.Client.RemoteEndPoint.ToString() : "Unknown";
_loggingService.Log(SeverityLevel.Info, String.Format("Connected to {0}", remoteEndPoint));
var netStream = client.GetStream();
var data = String.Empty;
using (var reader = new StreamReader(netStream, Encoding.ASCII))
{
data = reader.ReadToEnd();
}
_loggingService.Log(SeverityLevel.Info, "Received data: " + data);
ProcessData(data); //data is processed and stored in database (all resources are released when done)
client.Close();
_loggingService.Log(SeverityLevel.Info, String.Format("Connection closed for {0}", remoteEndPoint));
}
}
catch (Exception exception)
{
_loggingService.Log(SeverityLevel.Error, exception.Message);
}
finally
{
if (listener != null)
listener.Stop();
}
}
private void ProcessData(String data)
{
try
{
var processor = new Processor();
var lines = data.Split('\n');
foreach (var line in lines)
processor.ProcessLine(line);
processor.ProcessMessage();
}
catch (Exception ex)
{
_loggingService.Log(SeverityLevel.Error, ex.Message);
throw new Exception(ex.InnerException.Message);
}
}
}
One strange observation i just did:
I checked the log recently and no instrument has connected for the last 30 minutes (which indicates that the service is down).
I connected to the service myself via a TCP client i've written myself and uploaded some test data.
This worked fine.
When I checked the log again my test data had been stored.
The strange thing is, that 4 other instruments had connected about the same time and send their data successfully.
Why couldn't they connect by themself before I connected with my test client?
Also, what does this setting in .csdef do for an InputEndpoint, idleTimeoutInMinutes?
===============================================
Edit:
Since a cuple of days back my cloud service has been running successfully.
Unfortunately this morning last log entry was from this line:
_loggingService.Log(SeverityLevel.Info, String.Format("Connected to {0}", remoteEndPoint));
No other connections could be made after this. Not even from my own test TCP client (didn't get any error though, but no data was stored and no new logs).
This makes me think that following code causes the service to hang:
var netStream = client.GetStream();
var data = String.Empty;
using (var reader = new StreamReader(netStream, Encoding.ASCII))
{
data = reader.ReadToEnd();
}
I've read somewhere that StremReader's ReadToEnd() could hang. Is this possible?
I have now changed this piece of code to this:
int i;
var bytes = new Byte[256];
var data = new StringBuilder();
const int dataLimit = 10;
var dataCount = 0;
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
{
data.Append(Encoding.ASCII.GetString(bytes, 0, i));
if (dataCount >= dataLimit)
{
_loggingService.Log(SeverityLevel.Error, "Reached data limit");
break;
}
dataCount++;
}
Another explanation could be something hanging in the database. I use the SqlConnection and SqlCommand classes to read and write to my database. I always close my connection afterwards (finally block).
SqlConnection and SqlCommand should have default timeouts, right?
===============================================
Edit:
After some more debugging I found out that when the service wasn't responding it "hanged" on this line of code:
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
After some digging I found out that the NetStream class and its read methods could actually hang. Even though MS declares otherwise.
NetworkStream read hangs
I've now changed my code into this:
Thread thread = null;
var task = Task.Factory.StartNew(() =>
{
thread = Thread.CurrentThread;
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
{
// Translate data bytes to a ASCII string.
data.Append(Encoding.ASCII.GetString(bytes, 0, i));
}
streamReadSucceeded = true;
});
task.Wait(5000);
if (streamReadSucceeded)
{
//Process data
}
else
{
thread.Abort();
}
Hopefully this will stop the hanging.
I'd say that part of your problem is you are processing your data on the thread that listens for connections from clients. This would prevent new clients from connecting if another client has started a long running operation of some type. I'd suggest you defer your processing to worker threads thus freeing the "listener" thread to accept new connections.
Another problem you could be experiencing, if your service throws an error, then the service will stop accepting connections as well.
private static void ListenForClients()
{
tcpListener.Start();
while (true)
{
TcpClient client = tcpListener.AcceptTcpClient();
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object obj)
{
try
{
using(TcpClient tcpClient = (TcpClient)obj)
{
Console.WriteLine("Got Client...");
using (NetworkStream clientStream = tcpClient.GetStream())
using (StreamWriter writer = new StreamWriter(clientStream))
using(StreamReader reader = new StreamReader(clientStream))
{
//do stuff
}
}
}
catch(Exception ex)
{
}
}
I'm using node.js for gaming pairing(for example, several client connect the servers. They are connected to each other if there are more than one players; or they will be kicked for 30sec).
Currently, I'm using socket for connection (and this can detect lost connection unexpected). But I cannot figure out a elegant way for pairing:
var net = require('net');
var _clients = new Array();
net.createServer(function(_socket) {
_socket.on('data', function(_data) {
//Parse client data
_clients.push(_socket);
setTimeout(function(){
_socket.write('kicked\n');
},30*1000);
_socket.on('close', function(data) {
//Delete _socket from _clients
}
}).listen(6969, '127.0.0.1');
setInterval(function(){
var pair = new Array();
while(_clients.length>2)
{
var _s1 = _clients.pop(), _s2 = _clients.pop();
// Pair _s1 & _s2
}
},2*1000);
The current code works, but it is really bad designed :(
(1)SetInterval is used, rather than asynchronous call. (2)Maintaining an array like _clients is inconvenient, since I have to deal with "kicked"/lost connection/pair or other situations.
PS. Currently I'm pairing clients by time order, but maybe random pair or other condition is needed to avoid always pairing the same people when the online players are not so much.
Why not use something like the following?
Without Connection Pool
var net = require('net')
, pendingPair = null;
net.createServer(function(_socket) {
_socket.on('data', function(_data) {
//Parse client data
if(!pendingPair) {
pendingPair = _socket;
} else {
// Now you have a pair!
var p1 = pendingPair
, p2 = _socket;
pendingPair = null;
}
}).listen(6969, '127.0.0.1');
You get automatic pairs as soon as connections are made. You will still need to track those sockets somewhere for kicking clients, and dropped connections, but you should be able to get rid of the setIntervals.
With Connection Pool
var net = require('net')
, _ = require('underscore')._
, clients = {}
, games = {};
function setKickTimer(socket) {
socket.kickTimer = setTimeout(function() {
socket.write('kicked\n');
}, 30 * 1000);
}
net.createServer(function(socket) {
socket.id = Math.floor(Math.random() * 1000);
setKickTimer(socket);
clients[socket.id] = socket;
socket.on('data', function(data) {
socket.data = parseData(data);
}
socket.on('close', function(data) {
var opponent = _.find(clients, function(client) { return client.opponentId === socket.id; });
// The opponent is no longer part of a pair.
if(opponent) {
delete opponent.opponentId;
setKickTimer(opponent);
}
delete clients[socket.id];
}
}).listen(6969, '127.0.0.1');
setInterval(function(){
// Get the client ids of clients who are not matched, and randomize the order
var unmatchedClientIds = _.shuffle(_.keys(_.filter(clients, function(client) { return !client.opponentId; })));
while(unmatchedClientIds > 2) {
var c1 = unmatchedClientIds.pop(),
, c2 = unmatchedClientIds.pop();
clearTimeout(clients[c1].kickTimer);
clearTimeout(clients[c2].kickTimer);
clients[c1].opponentId = c2;
clients[c2].opponentId = c1;
}
},2*1000);
This isn't a complete working solution, but it should give you an idea of how to manage dropped connections and kicking users from the queue. Notice I'm using underscore.js to make the operations on the collections much easier.