AWS ECS Container stuck in restart loop - amazon-ecs

Question

Trying to bring up my first ECS Cluster.
Container is brought up successfully, but keeps on restarting every minute or so.
Cluster service is using an application load balancer.
Task definition:
{
"containerDefinitions": [
{
"dnsSearchDomains": null,
"entryPoint": null,
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 80
}
],
"command": [
"bash",
"deploy.sh"
],
"cpu": 512,
"memory": 1961,
"memoryReservation": 256,
"image": "1",
"essential": true,
}
],
"placementConstraints": [],
"memory": null,
"taskRoleArn": null,
"compatibilities": [
"EC2"
],
"taskDefinitionArn": "arn:aws:ecs:...",
"family": "service",
"networkMode": "awsvpc",
"status": "ACTIVE",
"inferenceAccelerators": null,
"proxyConfiguration": null,
"volumes": []
}
ecs agent log (removed and renamed specifics), goes through the following loop every minute or so
00:05Z [INFO] Handling ENI attachment
00:05Z [INFO] Starting ENI ack timer
00:05Z [INFO] Adding task eni attachment
00:05Z [INFO] Emitting task ENI attached event for: ENI Attachment:
00:05Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> NONE, ENI Attachment: sent: false
00:05Z [INFO] TaskHandler: Sending task attachment change: TaskChange: arn:... -> NONE, ENI Attachment: sent: false
00:11Z [INFO] Managed task arn:... unable to create state change event for container create container state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... waiting for any previous stops to complete. Sequence number: 110
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Managed task arn:... no longer waiting
00:11Z [INFO] Managed task arn:... wait over; ready to move towards status: RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [PULLED]
00:11Z [INFO] Managed task arn:... handling container change [{PULLED { <nil> [] <nil> [] map[] {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: PULLED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... creating container: ~internal~ecs~pause
00:11Z [INFO] Task engine arn:... created container name mapping for task: ~internal~ecs~pause ->
00:11Z [INFO] Creating cgroup
00:11Z [INFO] Managed task arn:... transitioned resource [cgroup] to [CREATED]
00:11Z [INFO] Managed task arn:... got resource [cgroup] event: [CREATED]
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... created docker container for task: ~internal~ecs~pause -> ...
00:11Z [INFO] Task engine arn:... created docker container for task: ~internal~ecs~pause -> ..., took 103.387592ms
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [CREATED]
00:11Z [INFO] Managed task arn:... handling container change [{CREATED {... <nil> [] <nil> [] com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } none 0xc00011ab00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: CREATED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... starting container: ~internal~ecs~pause
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RUNNING]
00:11Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> [] com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } none 0xc00011b600} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... setting up container resources for container [~internal~ecs~pause]
00:11Z [INFO] Task engine arn:... started docker container for task: ~internal~ecs~pause -> ..., took 221.148618ms
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RUNNING]
00:11Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> [] com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } none 0xc0007aed00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to RUNNING, but already RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... associated with ip address '1.2.3.4'
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RESOURCES_PROVISIONED]
00:11Z [INFO] Managed task arn:... handling container change [{RESOURCES_PROVISIONED {... <nil> [] <nil> [] map[] {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: internal container: ~internal~ecs~pause
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... pulling image for container concurrently
00:11Z [INFO] Task engine arn:... recording timestamp for starting image pulltime:
00:11Z [INFO] Updating container reference in Image State -
00:11Z [INFO] Task engine arn:... finished pulling image for container in 99.366882ms
00:11Z [INFO] Managed task arn:... got container event: [PULLED]
00:11Z [INFO] Managed task arn:... handling container change [{PULLED { <nil> [] <nil> [] map[] {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container
00:11Z [INFO] Managed task arn:... unable to create state change event for container create container state change event api: status not recognized by ECS: PULLED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... creating container:
00:11Z [INFO] Task engine arn:... created container name mapping for task: ->
00:12Z [INFO] Task engine arn:... created docker container for task: -> ...
00:12Z [INFO] Task engine arn:... created docker container for task: -> ..., took 81.682728ms
00:12Z [INFO] Managed task arn:... got container event: [CREATED]
00:12Z [INFO] Managed task arn:... handling container change [{CREATED {... <nil> [] <nil> [] com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } container: 0xc00062eb00} ContainerStatusChangeEvent}] for container
00:12Z [INFO] Managed task arn:... unable to create state change event for container create container state change event api: status not recognized by ECS: CREATED
00:12Z [INFO] api/task: Updating task's known status to: CREATED, task: arn:aws:ecs:... TaskStatus: (NONE->RUNNING) Containers: (CREATED->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:12Z [INFO] Managed task arn:... container change also resulted in task change [RUNNING]
00:12Z [INFO] Managed task arn:... unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
00:12Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:12Z [INFO] Managed task arn:... waiting for event for task
00:12Z [INFO] Task engine arn:... starting container:
00:12Z [INFO] Managed task arn:... got container event: [RUNNING]
00:12Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> [] com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } container: 0xc0007aeb00} ContainerStatusChangeEvent}] for container
00:12Z [INFO] Managed task arn:... sending container change event arn:aws:ecs:... -> RUNNING, Known Sent: NONE
00:12Z [INFO] Managed task arn:... sent container change event arn:aws:ecs:... -> RUNNING, Known Sent: NONE
00:12Z [INFO] api/task: Updating task's known status to: RUNNING, task: arn:aws:ecs:... TaskStatus: (CREATED->RUNNING) Containers: (RUNNING->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:12Z [INFO] Managed task arn:... container change also resulted in task change [RUNNING]
00:12Z [INFO] Managed task arn:... sending task change event arn:... -> RUNNING, Known Sent: NONE, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: ]
00:12Z [INFO] TaskHandler: batching container event -> RUNNING, Known Sent: NONE
00:12Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> RUNNING, Known Sent: NONE, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: , arn:aws:ecs:... -> RUNNING, Known Sent: NONE] sent: false
00:12Z [INFO] TaskHandler: Sending task change: TaskChange: arn:... -> RUNNING, Known Sent: NONE, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: , arn:aws:ecs:... -> RUNNING, Known Sent: NONE] sent: false
00:12Z [INFO] Managed task arn:... sent task change event arn:... -> RUNNING, Known Sent: NONE, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: ]
00:12Z [INFO] Managed task arn:... task at steady state: RUNNING
00:12Z [INFO] Managed task arn:... waiting for event for task
00:12Z [INFO] Task engine arn:... started docker container for task: -> ..., took 204.620728ms
00:12Z [INFO] Managed task arn:... got container event: [RUNNING]
00:12Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> [] com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } container: 0xc00062f600} ContainerStatusChangeEvent}] for container
00:12Z [INFO] Managed task arn:... redundant container state change. to RUNNING, but already RUNNING
00:12Z [INFO] Managed task arn:... task at steady state: RUNNING
00:12Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got container event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc00 {UNKNOWN <nil> 0 } container: 0xc00062f800} ContainerStatusChangeEvent}] for container
00:14Z [INFO] Task arn:... recording execution stopped time. Essential container stopped at:
00:14Z [INFO] Managed task arn:... sending container change event arn:aws:ecs:... -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] Managed task arn:... sent container change event arn:aws:ecs:... -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: task: arn:aws:ecs:... TaskStatus: (RUNNING->RUNNING) Containers: (STOPPED->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got resource [cgroup] event: [REMOVED]
00:14Z [INFO] Task engine arn:... stopping container [~internal~ecs~pause]
00:14Z [INFO] Task engine arn:... cleaning up the network namespace
00:14Z [INFO] TaskHandler: batching container event -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: task: arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (RESOURCES_PROVISIONED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task: arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (RESOURCES_PROVISIONED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] DockerGoClient: Unable to decode stats for container ...: context canceled
00:14Z [INFO] Container ... is terminal, stopping stats collection
00:14Z [INFO] Task engine arn:... cleaned pause container network namespace
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc00 {UNKNOWN <nil> 0 } none 0xc00062fb00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: internal container: ~internal~ecs~pause
00:14Z [INFO] api/task: Updating task's known status to: STOPPED, task: arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: task: arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task: arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... container change also resulted in task change [~internal~ecs~pause]: [STOPPED]
00:14Z [INFO] Managed task arn:... sending task change event arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt: PullStoppedAt: ExecutionStoppedAt:
00:14Z [INFO] Managed task arn:... sent task change event arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt: PullStoppedAt: ExecutionStoppedAt:
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: task: arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task: arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers: (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task has reached stopped. Waiting for container cleanup
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] DockerGoClient: Unable to decode stats for container ...: context canceled
00:14Z [INFO] Container ... is terminal, stopping stats collection
00:14Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: arn:aws:ecs:... -> STOPPED, Exit 0, , Known Sent: RUNNING] sent: false
00:14Z [INFO] TaskHandler: Sending task change: TaskChange: arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt: PullStoppedAt: ExecutionStoppedAt: arn:aws:ecs:... -> STOPPED, Exit 0, , Known Sent: RUNNING] sent: false
00:14Z [INFO] Managed task arn:... IPAM releasing ip for task eni
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc000489ee0 [] <nil> [] com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } none 0xc000505000} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to STOPPED, but already STOPPED
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc000072500 [] <nil> [] com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:... com.amazonaws.ecs.task-definition-version:15] {UNKNOWN <nil> 0 } none 0xc000505400} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to STOPPED, but already STOPPED
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] Udev watcher event-handler: unable to send state change: udev watcher send ENI state change: eni status already sent: ENI Attachment:
00:52Z [INFO] Handling ENI attachment
00:52Z [INFO] Starting ENI ack timer
00:52Z [INFO] Adding task eni attachment
Can anyone point to a potential cause based on this?
Would anything else be helpful?