I recently updated our CDK code to move our OpenSearch cluster from version 1.3 to 2.3. The cluster itself seems to have upgraded to a healthy state and is still accessible / usable by our application, but CloudFormation failed when attempting to update our domain resource with:
Resource handler returned message: "Resource handler returned message: "Invalid request provided: DP Nodes are OOS, Tags operation is not allowed"
This kicked the stack into UPDATE_ROLLBACK_FAILED, which is not allowed. The cluster cannot be downgraded back to 1.3.
I'm struggling to find any information about this error it's kicking out and not quite sure how to resolve it to unblock the CloudFormation stack.
Things I have tried:
Digging through CloudWatch logs only revealed information pertaining to queries.
Forcing the rollback to occur without Domain resource. This got me back to an UPDATE_COMPLETE state, but each subsequent deploy of this stack will cause it to fail again since the core issue is not resolved.
This was an odd presentation of a permissions issue. As I was reading through some docs, I stumbled upon this section, which discusses changes to tag-based access control.
This lead me start looking into CloudTrail a bit and stumbled upon the exact error that was firing when this deploy happened. It was a little odd because the assumed role granted admin access to CloudFormation, but the last line of this event record caught my eye:
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "DP Nodes are OOS, Tags operation is not allowed",
"eventSource": "es.amazonaws.com",
Upon adding es.amazonaws.com to the trust relationship of that role, the deploy fully re-ran successfully.
Hopefully this helps someone else.
I am using sarama(1.27) ClusterAdmin to manage topics in kafka1.1.0. My application that manages kafka topics, is running as a REST service. My application runs fine for a while and I can get/create/delete topic.
But after some time elapses without any activity, a new topic request gets error - write tcp xxxxx:37888->xxxxx:9092: write: broken pipe.
I came across this How to fix broker may not be available after broken pipe.
Since my application is running as a service, how do I prevent broken pipe issue ? I close ClusterAdmin only when application exits. Same ClusterAdmin connection is used to serve all requests. I reinitialize clusterAdmin for each request if for any reason it is nil(Usually it is not nil after first initialization, so same connection is reused).
Should I close clusteradmin after each request is served and open a NewClusterAdmin() for each topic request, or is there a keepalive option that I need to use?
Here is my existing code:
if admin == nil{
admin, err := NewClusterAdmin([]string{"localhost:9092"}, s.config)
..
}
topicMetadata, err := admin.DescribeTopics([]string{topicName})
I also came cross this error. My way to fix this question is try again several times, e.g. 2 to 10 times.
I am trying to establish an SFTP session using JSch. The code is working correctly and I am able to establish a session with multiple servers. However, today I am encountering an issue with one of the server.
Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out:
connect at com.jcraft.jsch.Util.createSocket(Util.java:349) ~[jsch-0.1.54.jar:?]
at com.jcraft.jsch.Session.connect(Session.java:215) ~[jsch-0.1.54.jar:?]
at com.jcraft.jsch.Session.connect(Session.java:183) ~[jsch-0.1.54.jar:?]
After debugging, I see that the issue is happening in Session.class.
tmp.join(timeout);
I tried explicitly setting up the timeout like below but it's still failing:
JSch jsch = new JSch();
Session session = jsch.getSession(userName, ip, port);
session.setPassword(password);
session.setConfig("StrictHostKeyChecking", "no");
session.connect(60000);
Note: Without passing these timeouts also, I never got into an issue so far.
Can someone help me in understanding the possible cause for this behavior and guide me regarding timeouts? Also, why will the below solution help if it will? I am trying to understand the root cause and resolution for the same.
JSch session timeout limit
Thanks
For anyone getting an issue like the one mentioned above, one of the probable cause could be proxy. The JSch Session class was failing at a timeout code without giving detailed stack trace.
I had to enable the proxy in order to get past this issue.
session.setProxy(new ProxyHTTP(PROXY_HOST, PROXY_PORT)). I may need to implement SOCKS4 and SOCKS5 proxy if the proxytype is of those respective types.
I have a Bluemix Node.js (6.1.0) application that uses node-rdkafka 1.0.3. It seems to be working ok but there are tons of error events like Error: Local: Broker Transport Failure or Error: Local: Authentication failure.
The producer options I have set are:
var producer_opts = {
"metadata.broker.list":env.messagehub.brokers,
"security.protocol":"sasl_ssl",
"ssl.ca.location":env.messagehub.calocation,
"sasl.mechanisms":"PLAIN",
"sasl.username":env.messagehub.user,
"sasl.password":env.messagehub.password,
"api.version.request":true,
"socket.timeout.ms": 10000,
"dr_msg_cb":true
};
Consumer has similar settings plus the group.id tag.
I wonder if I should be worrying for theese errors and if there is a way to eliminate them.
Thanks!
You are probably hitting https://github.com/edenhill/librdkafka/issues/1218.
In many cases, as you've noticed, these errors are harmless. The library node-rdkafka is based onto, librdkafka, always connects to all brokers in the cluster. Brokers your applications doesn't interact with will close the idle connections after a while leading to these error messages in your clients.
Unfortunately we don't have a recommended way to eliminate them at the moment. We are currently working on a potential solution to at least reduce their rate and maybe get rid of them.
Update:
With the most recent releases of node-rdkafka (>2.2), you can get rid of all the noisy logs by setting the following properties when creating clients:
'broker.version.fallback': '0.10.2.1',
'log.connection.close' : false
I just started using Postman. I had this error "Error: socket hang up" when I was executing a collection runner. I've read a few post regarding socket hang up and it mention about sending a request and there's no response from the server side and probably timeout. How do I extend the length of time of the request in Postman Collection Runner?
For me it was because my application was switched to https and my postman requests still had http in them. Changing postman to https fixed it.
Socket hang up, error is port related error. I am sharing my experience. When you use same port for connecting database, which port is already in use for other service, then "Socket Hang up" error comes out.
eg:- port 6455 is dedicated port for some other service or connection. You cannot use same port (6455) for making a database connection on same server.
Sometimes, this error rises when a client waits for a response for a very long time. This can be resolved using the 202 (Accepted) Http code. This basically means that you will tell the server to start the job you want it to do, and then, every some-time-period check if it has finished the job.
If you are the one who wrote the server, this is relatively easy to implement. If not, check the documentation of the server you're using.
Postman was giving "Could not get response" "Error: socket hang up".
I solved this problem by adding the Content-Length http header to my request
Are you using nodemon, or some other file-watcher? In my case, I was generating some local files, uploading them, then sending the URL back to my user. Unfortunately nodemon would see the "changes" to the project, and trigger a restart before a response was sent. I ignored the build directories from my file-watcher and solved this issue.
Here is the Nodemon readme on ignoring files: https://github.com/remy/nodemon#ignoring-files
I have just faced the same problem and I fixed it by close my VPN. So I guess that's a network agent problem. You can check if you have some network proxy is on.
this happaned when client wait for response for long time
try to sync your API requests from postman
then make login post and your are done
I defined Authenticate method to generate a token and mentioned its return type as nullable string as:
public string? Authenticate(string username, string password)
{
if(!users.Any(u => u.Key==username && u.Value == password))
{
return null;
}
var tokenHandler = new JwtSecurityTokenHandler();
var tokenKey = Encoding.ASCII.GetBytes(key);
var tokenDescriptor = new SecurityTokenDescriptor()
{
Subject = new ClaimsIdentity(new Claim[]
{
new Claim(ClaimTypes.Name, username)
}),
Expires = DateTime.UtcNow.AddHours(1),
SigningCredentials = new SigningCredentials(new
SymmetricSecurityKey(tokenKey),
SecurityAlgorithms.HmacSha256Signature)
};
var token = tokenHandler.CreateToken(tokenDescriptor);
return tokenHandler.WriteToken(token);
}
Changing nullable string to simply string fixed "Socket Hang Up" issue for me!
If Postman doesn't get response within a specified time it will throw the error "socket hang up".
I was doing something like below to achieve 60 minutes of delay between each scenario in a collection:
get https://postman-echo.com/delay/10
pre request script :-
setTimeout(function(){}, [50000]);
I reduced time duration to 30 seconds:
setTimeout(function(){}, [20000]);
After that I stopped getting this error.
I solved this problem with disconnection my vpn. you should check if there is vpn connected.
What helped for me was replacing 'localhost' in the url to http://127.0.0.1 or whatever other address your local machine has assigned localhost to.
Socket hang up error could be due to the wrong URL of the API you are trying to access in the postman. please check the URL once carefully.
It's possible there are 2 things, happening at the same time.
The url contains a port which is not commonly used AND
you are using a VPN or proxy that does not support that port.
I had this problem. My server port was 45860 and I was using pSiphon anti-filter VPN. In that condition my Postman reported "connection hang-up" only when server's reply was an error with status codes bigger than 0. (It was fine when some text was returning from server with no error code.)
When I changed my web service port to 8080 on my server, WOW, it worked! even though pSiphon VPN was connected.
Following on Abhay's answer: double check the scheme. A server that is secured may disconnect if you call an https endpoint with http.
This happened to me while debugging an ASP.NET Core API running on localhost using the local cert. Took me a while to figure out since it was inside a Postman environment and also it was a Monday.
In my case, adding in the header the "Content-length" parameter did the job.
My environment is
Mac:
[Terminal command: sw_vers]
ProductName: macOS
ProductVersion: 12.0.1. (Monterey)
BuildVersion: 21A559
mysql:
[Terminal command: mysql --version]
Ver 8.0.27 for macos11.6 on x86_64 (Homebrew)
Apache:
[Terminal command: httpd -v]
Server version: Apache/2.4.48 (Unix)
Server built: Oct 1 2021 20:08:18.
*Laravel
[Terminal command: php artisan --version]
Laravel Framework 8.76.2
Postman
Version 9.1.5 (9.1.5)
socket hang up error can also occur due to backend API handling logic.
For example - I was trying to create an Nginx config file and restart the service by using the incoming API request body. This resulted in temporary disconnection of the Nginx service while handling the API request and resulted in socket hang up.
If you have tried all the steps mentioned in other comments, and still face the issue. I suggest you check the API handler code thoroughly.
I handled the above-mentioned example by calling the Nginx reset method with delay and a separate API to check the status of the prev reset request.
For me it was giving Socket Hung Up error only while running Collection Runner not with single request.
Adding a small delay (100-300ms) in the collection Runner solved issue for me.
In my case, I had to provide --ssl-client-key and --ssl-client-cert files to overcome these errors.
Great error, it is so general that for everyone something different helps.
In my case I was not able to fix it and what is really funny is fact that I am expecting to get multipart file on one endpoint. When I prepare request in postman I get "Error: socket hang up". If I change for other endpoint(even not existing) is exactly that same error. But when I call any endpoint without body that request works and after that all subsequent attempts works perfectly.
In my case this is purely postman issue. Any request using curl is never giving that error.
For me the issue was related to the mismatch of the http versions on the client and server.
Client was assuming http v2 while server (spring boot/ tomcat) in the case was http v1
When on the server I configured server to v2, the issue got resolved in a go.
In spring boot you can configure the http v2 as below:-
server.http2.enabled=true
Note - Also the scenario was related to using client auth mechanism (i.e. MTLS)
Without client auth/ MTLS it worked without issues but for client auth the version setting in spring boot was the important rescue point
"socket hang up" is proxy related issue. when we run same collection with the help of newman on jenkins then all test are passed.
change the proxy setting
https://docs.cloudfoundry.org/cf-cli/http-proxy.html
I had the same issue: "Error: socket hang up" when sending a request to store a file and backend logs mentioned a timeout as you described. In my case I was using mongoDB and the real problem was my collection’s array capacity was full. When I cleared the documents in that collection the error was dismissed. Hope this will help someone who faces a similar scenario.
"Socket Hung Up" can be on-premise issue some time's, because, of bottle neck in %temp% folder, try to free up the "temp" folder and give a try
I fixed this issue by disabling Postman token header. Screenshot:
I face the same issue in when calling a SOAP API with POSTMAN
by adding the following data in the header my issue was fixed
Key:Content-Length
Value:<calculated when request is sent>
In my case, I was incorrectly using a port reserved for https version of my api.
For example, I was supposed to use https://localhost:6211, but I was using http://localhost:6211.
It is port related error. I was trying to hit the API with an invalid port.
if it helps to anybody... In my case, i just forgot to use json parser (const jsonParser = express.json();) to have access to json type of objects sending to the server from the client. Be careful, don't waste your time =)
This happened to me while I was learning ASP.NET Web API.
In my case it was because the SSL certificate verification.
I was using VS Code so I oversee about SSL certificate verification and it came with https protocol.
I solved this with testing my endpoints with http protocol.
Another approach can be just disabling the SSL certificate Verification on Postman Settings.
This error was coming for me since the request url is not correct --> here you can see my url does not contains : after http
The url I was using was : http//locahost:9090/someApi
Solution
adding a colon new url is http://localhost:9090/someApi
the socket error was not coming
This is just my case may be your case is totally different as mentioned in the other answers :)