Marathon-LB multiple instances through Bridge network doesnt work - marathon

I am using marathon-lb on dcos. When the load increases i get an error 'Maximum connections reached' and marathon-lb fails.
So i am trying to get multiple instances of marathon-lb running on the same node with below config. But this doesnt work, when i try the healthcheck it fails. On the other hand if i give hostport value (9090) one instance runs successfully and second instance keeps waiting. SO no matter what i cant have 2 instances working.
Isn't the Brdige network support to help run multiple instances? Any help is appreciated..
{
"id": "/marathon-lb-test3",
"acceptedResourceRoles": [
"slave_public"
],
"args": [
"sse",
"-m",
"http://marathon.mesos:8080",
"--group",
"external"
],
"backoffFactor": 1.15,
"backoffSeconds": 1,
"container": {
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "tcp",
"servicePort": 10001
},
{
"containerPort": 9090,
"hostPort": 9090,
"protocol": "tcp",
"servicePort": 10006
},
{
"containerPort": 443,
"hostPort": 0,
"protocol": "tcp",
"servicePort": 10007
},
{
"containerPort": 9091,
"hostPort": 0,
"protocol": "tcp",
"servicePort": 10008
},
{
"containerPort": 8080,
"hostPort": 0,
"protocol": "tcp",
"servicePort": 8080
}
],
"type": "DOCKER",
"volumes": [
{
"containerPath": "/marathon-lb/templates",
"hostPath": "/opt/marathon-lb/templates",
"mode": "RW"
}
],
"docker": {
"image": "mesosphere/marathon-lb:v1.11.1",
"forcePullImage": false,
"privileged": true,
"parameters": []
}
},
"cpus": 0.1,
"disk": 0,
"env": {
"HAPROXY_GLOBAL_DEFAULT_OPTIONS": "redispatch,httpclose,forceclose"
},
"healthChecks": [
{
"gracePeriodSeconds": 300,
"ignoreHttp1xx": false,
"intervalSeconds": 60,
"maxConsecutiveFailures": 3,
"portIndex": 1,
"timeoutSeconds": 20,
"delaySeconds": 15,
"protocol": "HTTP",
"path": "/_haproxy_health_check"
}
],
"instances": 2,
"maxLaunchDelaySeconds": 3600,
"mem": 1024,
"gpus": 0,
"networks": [
{
"mode": "container/bridge"
}
],
"requirePorts": false,
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"killSelection": "YOUNGEST_FIRST",
"unreachableStrategy": {
"inactiveAfterSeconds": 300,
"expungeAfterSeconds": 600
},
"fetch": [],
"constraints": []
}

You have at least 2 free Public Agents or change it from "external" to "internal" if you aplications work with a internal network communications, when you add new node to cluster DCOS , you can set as public agent, once time that marathon load in this node, and so you DNS shall resolve domain in this ip

Related

'Create service' for a Cluster with Ec2 type is giving error in AWS console

Trying out sample ECS with EC2 type in AWS free tier.
Created a Cluster for ec2 instance.
Then created task-def for ec2 resources with image uri - public.ecr.aws/ubuntu/nginx:latest, OS : Linux/X86_64, image t2.micro.
While creating/deploying the service, getting error on selecting the task-def created.
There was an error deploying nginx-service
Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: 1ab71394-b41e-190a-df10-6a87d62a7915, HandlerErrorCode: GeneralServiceException)
task-def-json
{
"taskDefinitionArn": "arn:aws:ecs:ap-northeast-1:930446195568:task-definition/ecs-task-def:1",
"containerDefinitions": [
{
"name": "nginx",
"image": "public.ecr.aws/ubuntu/nginx:latest",
"cpu": 0,
"portMappings": [
{
"name": "nginx-80-tcp",
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp",
"appProtocol": "http"
}
],
"essential": true,
"environment": [],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": []
}
],
"family": "ecs-task-def",
"executionRoleArn": "arn:aws:iam::930446195568:role/ecsTaskExecutionRole",
"networkMode": "bridge",
"revision": 1,
"volumes": [],
"status": "ACTIVE",
"placementConstraints": [],
"compatibilities": [
"EC2"
],
"requiresCompatibilities": [
"EC2"
],
"cpu": "512",
"memory": "1024",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
},
"registeredAt": "2023-02-15T17:11:45.596Z",
"registeredBy": "arn:aws:iam::930446195568:user/admin_user",
"tags": []
}

Getting started with KrakenD

I need some beginner help to KrakenD. I am running it on Ubuntu. The config is provided below.
I am able to reach the /healthz API without problem.
My challenge is that the /hello path returns error 500. I want this path to redirect to a Quarkus app that runs at http://getting-started36-getting-going.apps.bamboutos.hostname.us/.
Why is this not working? If I modify the /hello backend and use a fake host, I get the exacts ame result. This suggests that KrakendD is not even trying to connect to the backend.
In logs, KrakendD is saying:
Error #01: invalid character 'H' looking for beginning of value
kraken.json:
{
"version": 2,
"port": 9080,
"extra_config": {
"github_com/devopsfaith/krakend-gologging": {
"level": "DEBUG",
"prefix": "[KRAKEND]",
"syslog": false,
"stdout": true,
"format": "default"
}
},
"timeout": "3000ms",
"cache_ttl": "300s",
"output_encoding": "json",
"name": "KrakenD API Gateway Service",
"endpoints": [
{
"endpoint": "/healthz",
"extra_config": {
"github.com/devopsfaith/krakend/proxy": {
"static": {
"data": { "status": "OK"},
"strategy": "always"
}
}
},
"backend": [
{
"url_pattern": "/",
"host": ["http://fake-backend"]
}
]
},
{
"endpoint": "/hello",
"extra_config": {},
"backend": [
{
"url_pattern": "/hello",
"method": "GET",
"host": [
"http://getting-started36-getting-going.apps.bamboutos.hostname.us/"
]
}
]
}
]
}
What am I missing?
add "encoding": "string" to the backend section.
"backend": [
{
"url_pattern": "/hello",
"method": "GET",
"encoding": "string" ,
"host": [
"http://getting-started36-getting-going.apps.bamboutos.hostname.us/"
]
}
]

ECS CLI efsVolumeConfiguration Docker Compose / ECS Params yml

I have a service on ECS deployed through ecs-cli compose service up
The ECS/EFS documentation says to configure your task definition like this to mount an efs volume in an ecs container:
{
"containerDefinitions": [
{
"memory": 128,
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"mountPoints": [
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "efs-html"
}
],
"name": "nginx",
"image": "nginx"
}
],
"volumes": [
{
"name": "efs-html",
"efsVolumeConfiguration": {
"fileSystemId": "fs-1234",
"rootDirectory": "/path/to/my/data"
}
}
],
"family": "nginx-efs"
}
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_efs.html#efs-create
How does that translate to docker-compose/ecs-params.yml syntax?
Was resolved in 1019.
Documentation can be found here.
It does not seem to be supported just yet. See https://github.com/aws/amazon-ecs-cli/issues/1009

AWS ECS Task Definition: Unknown parameter in volumes[0]: "dockerVolumeConfiguration", must be one of: name, host

I am trying to run Wazuh/Wazuh docker container on ECS. I was able to register task definition and launch container using Terraform. However, I am facing an issue with "Volume"(Data Volume) while registering tak definition using AWS CLI command.
Command: aws ecs --region eu-west-1 register-task-definition --family hids --cli-input-json file://task-definition.json
Error:
ParamValidationError: Parameter validation failed:
Unknown parameter in volumes[0]: "dockerVolumeConfiguration", must be one of: name, host
2019-08-29 07:31:59,195 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
{
"containerDefinitions": [
{
"portMappings": [
{
"hostPort": 514,
"containerPort": 514,
"protocol": "udp"
},
{
"hostPort": 1514,
"containerPort": 1514,
"protocol": "udp"
},
{
"hostPort": 1515,
"containerPort": 1515,
"protocol": "tcp"
},
{
"hostPort": 1516,
"containerPort": 1516,
"protocol": "tcp"
},
{
"hostPort": 55000,
"containerPort": 55000,
"protocol": "tcp"
}
],
"image": "wazuh/wazuh",
"essential": true,
"name": "chids",
"cpu": 1600,
"memory": 1600,
"mountPoints": [
{
"containerPath": "/var/ossec/data",
"sourceVolume": "ossec-data"
},
{
"containerPath": "/etc/filebeat",
"sourceVolume": "filebeat_etc"
},
{
"containerPath": "/var/lib/filebeat",
"sourceVolume": "filebeat_lib"
},
{
"containerPath": "/etc/postfix",
"sourceVolume": "postfix"
}
]
}
],
"volumes": [
{
"name": "ossec-data",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "filebeat_etc",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "filebeat_lib",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "postfix",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
}
]
}
I tried by adding "host" parameter(however it supports Bind Mounts only). But got the same error.
"volumes": [
{
"name": "ossec-data",
"host": {
"sourcePath": "/var/ossec/data"
},
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
}
]
ECS should register the task definition having 4 Data Volumes and associated mount points.
Got the issue.
Removed "dockerVolumeConfiguration" parameter from "Volume" configuration and it worked.
"volumes": [
{
"name": "ossec-data",
"host": {
"sourcePath": "/ecs/ossec-data"
}
},
{
"name": "filebeat_etc",
"host": {
"sourcePath": "/ecs/filebeat_etc"
}
},
{
"name": "filebeat_lib",
"host": {
"sourcePath": "/ecs/filebeat_lib"
}
},
{
"name": "postfix",
"host": {
"sourcePath": "/ecs/postfix"
}
}
]
Can you check on your version of awscli?
aws --version
According to all the documentation, your first task definition should work fine and I tested it locally without any issues.
It might be that you are using an older aws cli version where the syntax was different or parameters were different at the time.
Could you try updating your aws cli to the latest version and try again?
--
Some additional info I found:
Checking on the aws ecs cli command, they added docker volume configuration as part of the cli in v1.80
The main aws-cli releases updates periodically to update the commands but they don't provide much info on what specific versions of each command is changed:
https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst
If you update your aws-cli version things should work

How could a spring-boot application determine if it is running on cloud foundry?

I'm writting a micro service with spring-boot. The db is mongodb. The service works perfectly in my local environment. But after I deployed it to the cloud foundry it doesn't work. The reason is connecting mongodb time out.
I think the root cause is the application doesn't know it is running on cloud. Because it still connecting 127.0.0.1:27017, but not the redirected port.
How could it know it is running on cloud? Thank you!
EDIT:
There is a mongodb instance bound to the service. And when I checked the environment information, I got following info:
{
"VCAP_SERVICES": {
"mongodb": [
{
"credentials": {
"hostname": "10.11.241.1",
"ports": {
"27017/tcp": "43417",
"28017/tcp": "43135"
},
"port": "43417",
"username": "xxxxxxxxxx",
"password": "xxxxxxxxxx",
"dbname": "gwkp7glhw9tq9cwp",
"uri": "xxxxxxxxxx"
},
"syslog_drain_url": null,
"volume_mounts": [],
"label": "mongodb",
"provider": null,
"plan": "v3.0-container",
"name": "mongodb-business-configuration",
"tags": [
"mongodb",
"document"
]
}
]
}
}
{
"VCAP_APPLICATION": {
"cf_api": "xxxxxxxxxx",
"limits": {
"fds": 16384,
"mem": 1024,
"disk": 1024
},
"application_name": "mock-service",
"application_uris": [
"xxxxxxxxxx"
],
"name": "mock-service",
"space_name": "xxxxxxxxxx",
"space_id": "xxxxxxxxxx",
"uris": [
"xxxxxxxxxx"
],
"users": null,
"application_id": "xxxxxxxxxx",
"version": "c7569d23-f3ee-49d0-9875-8e595ee76522",
"application_version": "c7569d23-f3ee-49d0-9875-8e595ee76522"
}
}
From my understanding, I think my spring-boot service should try to connect the port 43417 but not 27017, right? Thank you!
Finally I found the reason is I didn't specify the profile. After adding following code in my manifest.yml it works:
env:
SPRING_PROFILES_ACTIVE: cloud