I'm running a webapi on an Azure website that makes calls to external web services. The webapi handles approximately 2K-3K requests per minute.
Periodically, lots of socket errors start occurring that indicate: "An attempt was made to access a socket in a way forbidden by its access permissions". This error seems to occur regardless of the ip address of the external web service.
At first, I thought it might be ephemeral port exhaustion, but I've limited "connectionManagement" to a maximum of 100 connections.
What would be causing this?
Thanks very much. Happy to provide whatever information might be helpful.
Update 6/1: - doesn't work per 6/2
I added the following to my web.config system.net section:
<defaultProxy enabled="false" useDefaultCredentials="false">
<proxy/>
<bypasslist/>
<module/>
</defaultProxy>
It appears to have helped as I haven't seen this issue in the last 6 hours. I have no idea why this would actually help though as I'm not using any proxy-related stuff.
Any thoughts?
Update 6/2:
Adding the defaultProxy doesn't actually appear to help. The problem is still occurring. Back to the drawing board.
I've finally figured out the cause of this problem. The issue was occurring due to port exhaustion.
I was using an NLog email target which was grabbing and holding onto too many SMTP connections over time (despite the 100 max connection limit). After removing the email target, the issue no longer occurs. I haven't figured out why NLog was exhibiting this behavior.
Related
I am using an EC2 instance as a backend database server that receives open listings for an AirBnB type site. I've checked on my own browsers and phones and had others check on theres in other regions as well, and these listings load perfectly fine for us. There is one person in another region, however, who is not seeing any listings at all and receives the Failed to load resource: net::ERR_CONNECTION_RESET error instead. I even had them try clearing their cache in Chrome, but that did not help. Below are photos depicting the situation:
Click here to see the problem page (lol)
Photos depicting errors:
What I See/What Should Show Up:
Black Listings Come Up For Them
Here Are The Errors They Receive
Here are the Inbound settings for my security group
I'm thinking it may be a firewall issue, but i'm just not sure. Any help would be greatly appreciated, thank you!
i would suggest you check the security group in which your machine is, since it's possible that you're not allowing traffic to reach your machines.
it's possible that your location IP is open to the API but not other IP's.
I've tried to reach your page from my place and it's timing out as well, that points to the security group.
if possible share a screenshot of the security group set up on your machine, that will help diagnose further.
After some googling I found that this reason if caused by too many scheduled conferences by my Application Endpoint. My only problem is, how do I delete/remove currently stored scheduled conferences for my application endpoint, when the only access I have to my server is through PowerShell??
EDIT:
Just found a command that actually could increase number of scheduled meetings per organizer (Set-CsUserServicesConfiguration -MaxScheduledMeetingsPerOrganizer 2000), but it didn't change the issue. I'm still receiving MaxConferencesExceeded error. Any ideas??
Use Get-CsUserServicesConfiguration to make sure the change has actually occurred, and/or setting it globally to make sure it has the correct context for your users.
I've also found with a large pool, it can take a while to kick in.
Have you tried republishing your Lync topology after making the change? A bit dramatic, but can help with Lync gets itself into a knot with changes.
When I deployed my site that uses nservice to a new production box, it was unusably slow...
After some debugging I discovered that mqsvc.exe was taking up 50% of the CPU usage and the other 50% was being taken up by w3wp.exe
I found this post here:
http://geekswithblogs.net/michaelstephenson/archive/2010/05/07/139717.aspx
which recommended the following:
Make sure you set the windows service for NserviceBus Generic Host to the right credentials
Make sure you have the queue set with the right permissions
Make sure you turn on the right logging configuration in NServiceBus
So I figured the issue was something related to permissions, but even after trying to set the permissions correctly (I thought) I still wasn't able to resolve the issue.
If you allow NServiceBus to create its own queues, then it will create them with the correct permissions it needs.
The problem comes in when you set up a web application, and then the queues are created, and then the identity the application runs under changes. Then you get exactly this problem. NServiceBus tries to check the queue for a message, it does not have access to do so, so it immediately retries over and over, and you spike the processor.
The fix: Delete the queue. Restart the web application. NServiceBus takes over.
Edit: As noted in the comments, NServiceBus 3.x doesn't invoke the installers by default, which means queues are not automatically created in production unless you ask it to. See the documentation page on Installers for more detail.
For a web application (or any other situation where you're not using NServiceBus.Host) you can invoke the installers as part of the fluent config. There is a full example in the NServiceBus download, but here is a link to the relevant file on GitHub.
The issue did end up being that the website needed to be granted explicit permissions to the queues.
I found a number of resources online telling me this, but I still had to spend a good amount of time monkeying around with exactly WHICH account needed access... turned out that since my application pools were set to run as ApplicationPoolIdentity, I need to grant the account permissions by adding the following account to the nservicebus queue:
IIS AppPool\{APP POOL NAME}
I granted full access rights, though I'm sure you could refine that a bit if you needed to.
Hopefully, this will help anyone who runs into the same issues.
(This is my first attempt at the "Answer your own question" mechanism so please let me know if I am doing something wrong..)
I'm building an iOS app, but the problem exists on all clients. iChat, Messages, Psi, etc. So because it exists on all clients I'm going to assume it's a server issue.
Has anyone ever experienced something like this? If so, what did you do to fix it? I'm sure it's some silly config setting or something but I simply can't figure this out. This is the only thing that looks like it might be related in ejabberd.log:
=ERROR REPORT==== 2012-09-05 12:07:12 ===
Mnesia(ejabberd#localhost): ** WARNING ** Mnesia is overloaded: {dump_log,
time_threshold}
Thanks in advance for any tips/pointers.
https://github.com/processone/ejabberd/blob/master/src/ejabberd_c2s.erl#L936 seems to have already been patched. The config variable is called resource_conflict and the value you want is setresource.
The above warning is (probably) not related to the issue you are facing. These mnesia events usually happens when the transaction log needs to be dumped, but the previous transaction log dump hasn't finished yet.
Problem that you are facing needs to be debugged for which you can set {log_level, 5} inside ejabberd.cfg. This will enable debug logging for ejabberd. Then look into the logs to find any guesses on why this is happening for you. Also, come back and paste your log file details here, probably we will be able to help you further. I have never faced such non-sensical issues with ejabberd.
Update after log file attachment:
As Joe wrote below, this is indeed happening because of resource conflict. Two of your clients are trying to login with same resource value. But in an ideal world this shouldn't matter. Jabber servers SHOULD take care of this by appending or prepending custom value on top of resource value requested by the client.
For example, here is what gtalk (even facebook chat) servers will do:
SENT <iq xmlns="jabber:client" type="set" id="1"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><resource>jaxl#resource</resource></bind></iq>
RCVD <iq id="1" type="result"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><jid>jabberxmpplibrary#gmail.com/jaxl#resou27F46704</jid></bind></iq>
As you can see my client requested to bind with resource value jaxl#resource but gtalk server actually bound my session with resource value jaxl#resou27F46704. In short, this is not a bug in your client but a bug in ejabberd.
To fix this you can do two things:
Resource value is probably hardcoded somewhere in your client configuration. Simply remove that. A good client will automatically take care of this by generating a random resource value at it's end.
Patch ejabberd to behave how gtalk server does (as shown above). This is the relevant section inside ejabberd_c2s.erl src which needs some tweaking. Also search for Replaced by new connection inside the c2s source file and you will understand what's going on.
This sounds like the "dueling resources" bug in your client. You may have two copies of your client running simultaneously using the same resource, and doing faulty auto-reconnect logic. When the second client logs in, the first client is booted offline with a conflict error. The first client logs back in, causing a conflict error on the second client. Loop.
Evidence for this is in your logfile, on line 3480:
D(<0.373.0>:ejabberd_c2s:1553) : Send XML on stream =
<<"<stream:error><conflict xmlns='urn:ietf:params:xml:ns:xmpp-streams'/>
<text xml:lang='en' xmlns='urn:ietf:params:xml:ns:xmpp-streams'>
Replaced by new connection
</text>
</stream:error>">>
I have an ASP MVC 2 application, through which I occasionally send emails using SmtpClient.Send(). Typically, emails are sent out in batches of between 1 and 50 emails, with hours or even days passing between batches. I have this all set up so that the emails are actually sending just fine. But, the problem is that when the emails are sent, my site suddenly becomes unavailable for about 15 minutes, and I have no idea why.
My site is hosted on a shared, Windows 2008 server with a third-party web host.
Here is the relevant section in my web.config file, edited for privacy:
<system.net>
<mailSettings>
<smtp deliveryMethod="Network" from="fromemail#doman.com">
<network host="mail.DOMAIN.COM" userName="username" password="password"/>
</smtp>
</mailSettings>
</system.net>
Does anyone have any thoughts or ideas as to why this might be happening? I've been trying to reaserch it and Google it for some time now, but I'm just not coming up with anything.
This really could be many different things but..
The first thing I suggest you do is enable ASP.NET Health Monitoring on your site. This should hopefully help you gain visibility of the exception that is causing this issue (A guide to using Health Monitoring).
For obvious reasons be sure not to choose the Mail Provider to send you your exceptions - perhaps use the SQL provider or write a custom provider that writes to a file.
I would also ask your hosts to look into the Event Log for any information that may be of value.
Hope that helps.
E-mail can be an expensive operation. Have you considered using the asynchronous e-mail send so that the process does not block your main thread?