GCP Load Balancer Response 503 When Upload File - google-cloud-http-load-balancer

I have a backend that processes a file. The file is uploaded from a client (Apache HTTP Client) by POST (multipart file). The size of the file is 30kb (average) and the load balancer sometimes responds with a 503 status for some POST. The max simultaneous uploads is 80.
Any idea how to fix that? Thanks!
According to GCP documentation the response status is generated by the load balancer. I checked the field "statusDetails" in the log.

Related

How to view network requests logs in GKE?

How does one view request headers, payloads and response headers from an application thats deployed in GKE? For example if a node api invokes a function cleanData(); and it makes an api request to intuit API, how does one capture the network request/response headers in GCP logs?
I tried going to GCP console logs and its all just console.logs of the application and no network logs

Static webpage redirect http to https using Google loadbalancer

I'm trying to implement URL redirects from http to https as described by [https://cloud.google.com/load-balancing/docs/https/setting-up-traffic-management][1] but I'm getting ERR_TOO_MANY_REDIRECTS
I have a storage bucket with a very simple HTML page.
I have an external HTTP load balancer in front of it. Static IP address. SSL cert. I managed to connect everything so that both http and https requests for the site load the contents of the bucket.
I tried to add the HTTP redirect as per the document:
Changes 'Host and path rules' from 'Simple' to 'Advanced...'.
The default route still points to the bucket
I added a new route. Host is 'www.example.com. The default path rule points to the bucket. The second path rule matches /* and does a prefix/HTTPS redirect as described in the above link.
Once the config is saved, either http or https requests to www.example.com results in ERR_TOO_MANY_REDIRECTS
What am I doing wrong? Really appreciate any help you can provide.
[Backend configuration][2]
[Frontend configuration][3]
[Host and path rules][4]
[Redirect path rule][5]
[1]: https://cloud.google.com/load-balancing/docs/https/setting-up-traffic-management
[2]: https://i.stack.imgur.com/lkhUF.png
[3]: https://i.stack.imgur.com/FYst0.png
[4]: https://i.stack.imgur.com/zsTOX.png
[5]: https://i.stack.imgur.com/2tEDE.png
FYI - someone in Google Groups pointed out that I needed 2 load balancers. 1 to terminate the HTTPS traffic and the second to redirect the HTTP traffic. Works like a charm.

REST API random 404 error when a server is down on load balancer

We have a REST based API project. The project is hosted in IIS on multiple servers on a load balancer.It works great except when a server goes down on the load balancer, we receive a 404 Bad Request error for few seconds and then it resolves itself. How do we resolve this? Are there any changes that need to be done in IIS or on the REST API project itself?
As far as I know, if you use IIS ARR's loadbalance there is no need to add the specail setting to check the server's status. The IIS ARR healthy test will auto send the request to check the server is work well or not. If the IIS ARR find the server is down or return error, it will not redirect the request to the server.
Besides, as far as I know, the 404 error means the page not found not the bad request. Normally, 400 is the bad request status code.
If you find a server is down in your server farms, I suggest you could try to access the server directly through the brwoser to know the details error message.
If the server return 404 error, that means your browser couldn't access the server, it is a network issue normally. You should check the DNS server setting or make sure the IIS web application is running.
If the server return 400 error, you should troublshooting according to the details 400 error message. Normall, it shows when the request contains the wrong header.

AWS Classic Load Balancer + EC2: web API requests returns 404

I have an AWS EC2 Jira instance running behind an AWS Classic load balancer. The site loads in the browser fine, but all API requests are returning 404 for some reason. It is not a Jira 404, but a generic 404 response with no body and minimal headers. Only response useful header seems to be Server: nginx.
Tried white-listing my client IP, opening up all ports, sending request to the LB and directly to the instance with proper security group settings, etc., but same 404 response is returned. I'm using Postman to test the API. I noticed when I load the EC2 instance directly in the browser, it redirects to the load balancer.
Returns 200 with HTML. Basic auth works, too.
GET http://jira (home page)
Returns 404:
GET http://jira/rest/api/2/issue/ticket-num (or any other /rest/ endpoints)
Where should I start looking to debug this 404 issue? I feel like I'm missing something basic. I'm not seeing any Jira configuration for setting up its rest API. I feel like perhaps it's a server configuration issue, although I've never come across manual web server configuration while installing Jira, so maybe on the AWS's side?
EDIT: still waiting to get ssh access to the instance, so I'll update as I get more info and access.
This HTTP 404 responses with very limited set of headers could be from the default (the bottom one) rule in ELB. I experienced similar issue getting HTTP 404 because instead of host header I set path and provided the host domain name in one of ELB rules. So the rule did not work and default rule returned 404 because there is no such path exists on the instance.
I would recommend to try to use Redirect to or Return fixed response options for default rule to check out if it goes to the default rule.

JMeter - load balancing

In our application, We have 6 instance of application server.
When we test it manually, we will land on some node randomly.
When i run our JMeter test (with 20 users), most of the load(70%) goes to 1 specific node out of 6 nodes.
30% goes to another specific node from the rest 5.
I see the information via JSESSIONID set in the request cookies.
Issue:
I was thinking the load will be balanced on all the 6 nodes equally (more or less).
Is this my application issue? Or Does it have to do anything with JMeter?
Note:
Our app server is JBoss. It is a cookie injection based load balancing.
I do not send the request to any specific app server via cookie. The cookie is set automatically.
First of all make sure that HTTP Cookie Manager is added to your test plan, it should be enough to automatically resolve cookie-based load balancing problem.
If you have > 1 load balancer instance or the load balancer resolves in > 1 IP address you may have to add a DNS Cache Manager as well. See The DNS Cache Manager: The Right Way To Test Load Balanced Apps guide for better explanation and configuration details.
Beside what Dmitri suggested you should check the way your LB is configured.
In many cases, the LB distributes the requests based on source IP, so all requests coming form the Jmeter server go to the same target.
This happens especially when the LB does not terminate the https traffic.