GKE service catalog BigQuery ACL/permission problems - The user xx does not have bigquery.jobs.create permission in project yy - kubernetes

I am trying to use the service catalog of Google Kubernetes to connect to BigQuery. I had however a lot of issues regarding IAM/ACL permissions.
I added the Owner role to the myProjectId#cloudservices.gserviceaccount.com account, since Editor was not enough to access IAM during the creation of a binding's service account.
After manually adding projectReaders, projectWriters and projectOwners to the ACL of the dataset, I could finally read and write to BigQuery, but I can not create jobs, since this requires project permissions. The command to update the dataset was
bq update --source /tmp/roles myDatasetId
After that I tried to query bq, but it failed with
root#batch-shell:/app# cat sql/xxx.sql | bq query --format=none --allow_large_results=true --destination_table=myDatasetId.pages_20180730 --maximum_billing_tier 3
BigQuery error in query operation: Access Denied: Project my-staging-project: The user k8s-bigquery-acc#my-staging-project.iam.gserviceaccount.com does not have
bigquery.jobs.create permission in project my-staging-project.
I tried to set the account's role to "Owner" and "BigQuery Job User" with no effect. I even tried all the other accounts as Owner.
This are my current ACL permissions:
[16:52:45] blackfalcon:~/src/myproject/batch :chris $ bq --format=prettyjson show myDatasetId
{
"access": [
{
"role": "WRITER",
"specialGroup": "projectWriters"
},
{
"role": "OWNER",
"specialGroup": "projectOwners"
},
{
"role": "OWNER",
"userByEmail": "myProjectId#cloudservices.gserviceaccount.com"
},
{
"role": "OWNER",
"userByEmail": "k8s-bigquery-acc#my-staging-project.iam.gserviceaccount.com"
},
{
"role": "READER",
"specialGroup": "allAuthenticatedUsers"
},
{
"role": "READER",
"specialGroup": "projectReaders"
}
],
"creationTime": "1532859638248",
"datasetReference": {
"datasetId": "myDatasetId",
"projectId": "my-staging-project"
},
"defaultTableExpirationMs": "8000000000",
"description": "myproject Access myDatasetId",
"id": "my-staging-project:myDatasetId",
"kind": "bigquery#dataset",
"lastModifiedTime": "1533184961736",
"location": "US",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/my-staging-project/datasets/myDatasetId"
}
[16:53:02] blackfalcon:~/src/myproject/batch :chris $ gcloud projects get-iam-policy my-staging-project
bindings:
- members:
- serviceAccount:k8s-bigquery-acc#my-staging-project.iam.gserviceaccount.com
- user:myemail#somedomain.com
role: roles/bigquery.admin
- members:
- serviceAccount:k8s-cloudsql-acc-staging#my-staging-project.iam.gserviceaccount.com
role: roles/cloudsql.client
- members:
- serviceAccount:service-myProjectId#compute-system.iam.gserviceaccount.com
role: roles/compute.serviceAgent
- members:
- serviceAccount:service-myProjectId#container-engine-robot.iam.gserviceaccount.com
role: roles/container.serviceAgent
- members:
- serviceAccount:myProjectId-compute#developer.gserviceaccount.com
- serviceAccount:myProjectId#cloudservices.gserviceaccount.com
- serviceAccount:service-myProjectId#containerregistry.iam.gserviceaccount.com
role: roles/editor
- members:
- serviceAccount:service-myProjectId#cloud-ml.google.com.iam.gserviceaccount.com
role: roles/ml.serviceAgent
- members:
- serviceAccount:myProjectId#cloudservices.gserviceaccount.com
- user:myemail#somedomain.com
role: roles/owner
- members:
- serviceAccount:scg-fv6fz3sjnxo3cfpppcl2qs5edm#my-staging-project.iam.gserviceaccount.com
role: roles/servicebroker.operator
- members:
- serviceAccount:service-myProjectId#gcp-sa-servicebroker.iam.gserviceaccount.com
role: roles/servicebroker.serviceAgent
- members:
- serviceAccount:k8s-bigquery-acc#my-staging-project.iam.gserviceaccount.com
- user:myemail#somedomain.com
role: roles/storage.admin
version: 1
It seems I need to set the projects ACL for BigQuery, but everything I found indicates, that setting the roles with IAM should be enough
Any help would be greatly appreciated.
UPDATE: I solved that for now.
Turns out that the service account itself was not working properly. I tried giving an Owner role to the service account and used the service account locally to access a few gcloud resources, all failed with permission errors.
I created then a new service account with the same permissions and tried again and it worked. So, it seems the service account was somehow broken.
I deleted the bindings, then the IAM and service account and rebuild the bindings.
Now it is working like a charm

Related

Opensearch Failed to set number of replicas due no permissions

I have the problem with running index managment policy for new indices. I get following error on "set number_of_replicas" step:
{
"cause": "no permissions for [indices:admin/settings/update] and associated roles [index_management_full_access, own_index, security_rest_api_access]",
"message": "Failed to set number_of_replicas to 2 [index=sample.name-2022.10.22]"
}
The indices are created by logstash with "sample.name-YYYY.MM.DD" name template, so in the index policy I have "sample.name-*" index pattern.
My policy:
{
"policy_id": "sample.name-*",
"description": "sample.name-* policy ",
"schema_version": 16,
"error_notification": null,
"default_state": "set replicas",
"states": [
{
"name": "set replicas",
"actions": [
{
"replica_count": {
"number_of_replicas": 2
}
}
]
],
"ism_template": [
{
"index_patterns": [
"sample.name-*"
],
"priority": 1
}
]
}
I don't understand the reason of this error.
Am I doing something wrong?
Retry of the policy doesn't work.
The policy works only if I manually reassign it to index by Dashboards or API.
Opensearch version: 2.3.0
First time I created the policy using API under custom internal user with mapped “security_rest_api_access” security role only.
So I added all_access rights to my internal user and re-created policy and it works!
Seems that the policy runs under my internal user, which created it

pod identity on aks cluster crreation

Right now, it's impossible to have assigned user assigned identities on arm templates (and terraform) on cluster creation. I already tried a lot of things, and updates works great, after inserting manually with:
az aks pod-identity add --cluster-name my-aks-cn --resource-group myrg --namespace myns --name example-pod-identity --identity-resource-id /subscriptions/......
But, I want to have this done at once, with the deployment, so I need to insert the pod user identities to the cluster automatically. I also tried to run the command using the DeploymentScripts but the deployment scripts are not ready to use preview aks extersion.
My config looks like this:
{
"type": "Microsoft.ContainerService/managedClusters",
"apiVersion": "2021-02-01",
"name": "[variables('cluster_name')]",
"location": "[variables('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks', variables('vnet_name'))]"
],
"properties": {
....
"podIdentityProfile": {
"allowNetworkPluginKubenet": null,
"enabled": true,
"userAssignedIdentities": [
{
"identity": {
"clientId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity'), '2018-11-30').clientId]",
"objectId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity'), '2018-11-30').principalId]",
"resourceId": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity')]"
},
"name": "managed-indentity",
"namespace": "myns"
}
],
"userAssignedIdentityExceptions": null
},
....
},
"identity": {
"type": "SystemAssigned"
}
},
I'm always getting the same issue:
"statusMessage": "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'deployment_test' is not valid according to the validation procedure. The tracking id is '.....'. See inner errors for details.\",\"details\":[{\"code\":\"PodIdentityAddonUserAssignedIdentitiesNotAllowedInCreation\",\"message\":\"Provisioning of resource(s) for container service cluster-12344 in resource group myrc failed. Message: {\\n \\\"code\\\": \\\"PodIdentityAddonUserAssignedIdentitiesNotAllowedInCreation\\\",\\n \\\"message\\\": \\\"PodIdentity addon does not support assigning pod identities on creation.\\\"\\n }. Details: \"}]}}",
The Product team has shared the answer here: https://github.com/Azure/aad-pod-identity/issues/1123
which says:
This is a known limitation in the existing configuration. We will fix
this in the V2 implementation.
For others who are facing the same issue, please refer to the GitHub issue above.

Mystery "guest" user for rabbitMQ

I know the "guest" user is the default for RabbitMQ, but I thought I'd configured everything to use different names.
My stack is Django / Celery / RabbitMQ, running in Docker.
First up, the error - I jst get loads of these - every few seconds:
rabbitmq_1 | 2020-07-29 08:28:00.775 [warning] <0.1234.0> HTTP access denied: user 'guest' - invalid credentials
rabbitmq_1 | 2020-07-29 08:28:05.775 [warning] <0.1240.0> HTTP access denied: user 'guest' - invalid credentials
rabbitmq_1 | 2020-07-29 08:28:10.776 [warning] <0.1246.0> HTTP access denied: user 'guest' - invalid credentials
rabbitmq_1 | 2020-07-29 08:28:15.776 [warning] <0.1252.0> HTTP access denied: user 'guest' - invalid credentials
rabbitMQ Dockerfile
FROM rabbitmq:management-alpine
ENV RABBITMQ_USER rabbit_user
ENV RABBITMQ_PASSWORD rabbit_user
ADD rabbitmq.conf /etc/rabbitmq/
ADD definitions.json /etc/rabbitmq/
RUN chown rabbitmq:rabbitmq /etc/rabbitmq/rabbitmq.conf /etc/rabbitmq/definitions.json
CMD ["rabbitmq-server"]
rabbitmq.conf
management.load_definitions = /etc/rabbitmq/definitions.json
definitions.json
{
"users": [
{
"name": "rabbit_user",
"password": "rabbit_user",
"tags": ""
},
{
"name": "admin",
"password": "admin",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "\/phoenix"
}
],
"permissions": [
{
"user": "rabbit_user",
"vhost": "\/phoenix",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"parameters": [],
"policies": [],
"exchanges": [],
"bindings": [],
"queues": [
{
"name": "high_prio",
"vhost": "\/phoenix",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "low_prio",
"vhost": "\/phoenix",
"durable": true,
"auto_delete": false,
"arguments": {}
}
]
}
docker-compose.yml
rabbitmq:
build:
context: ./rabbitmq
dockerfile: Dockerfile
# image: rabbitmq:3-management-alpine
ports:
- "15672:15672" # RabbitMQ management plugin
environment:
- RABBITMQ_DEFAULT_USER=rabbit_user
- RABBITMQ_DEFAULT_PASS=rabbit_user
- RABBITMQ_DEFAULT_VHOST=phoenix
expose:
- "5672" # Port exposed between docker containers
depends_on:
- db
- cache
celery_worker:
<<: *django
command: bash -c "celery -A phoenix.celery worker --loglevel=INFO -n worker1#%h"
environment:
- DJANGO_SECRET_KEY=${DJANGO_SECRET_KEY}
- EMAIL_HOST_PASSWORD=${EMAIL_HOST_PASSWORD}
- DJANGO_SETTINGS=${DJANGO_SETTINGS}
# HC the rabbit user. Not secure obvs, but OK for PoC.
- RABBITMQ_DEFAULT_USER=rabbit_user
- RABBITMQ_DEFAULT_PASS=rabbit_user
ports: []
links:
- rabbitmq
- cache
depends_on:
- db
- cache
- rabbitmq
settings.py
CELERY_BROKER_URL = "amqp://rabbit_user:rabbit_user#rabbitmq:5672/phoenix"
CELERY_BROKER_VHOST = "phoenix"
CELERY_RESULT_BACKEND = "django-db"
CELERY_CACHE_BACKEND = "default"
CELERY_TIME_ZONE = TIME_ZONE
I had it all working before when I just pulled the default rabbitMQ container in the docker-compose yaml file. Now I've created a specific Dockerfile for rabbitMQ, and setup rabbit_user and the vhost "phoenix". It all seems to be working - tasks are run, I see the message stats in the rabbit console, but I'm suffering these random "guest" login attempts. The word "guest" appears nowhere in my codebase, so somewhere RabbitMQ is using the default not "rabbit_user", but I can't see where.
Rather typical that I solve this by "fixing" something else ..
I noticed in my RMQ panel that the low_prio and high_prio queues had vhost "/phoenix", while the celery workers had vhost "phoenix" (I'd thought the RMQ config required the leading slash from my reading). I amended this so that all queues were allocated to "phoenix", and the mystery guest login disappeared.
I can only assume that since Celery was configured for the vhost "phoenix", that "/phoenix" was treated as s different vhost, with no users assigned to it, so RabbitMQ tried to use the "guest" default.
Not entirely sure why things were connecting to it - I'd sent nothing to those queues yet - but in case somebody else has this issue, this is what solved it for me.

How to access CloudWatch Event data from triggered Fargate task?

I read the docs on how to Run an Amazon ECS Task When a File is Uploaded to an Amazon S3 Bucket. However, this document stops short of explaining how to get the bucket/key values from the triggering event from within the Fargate task code itself. How can that be done?
I am not sure if you still need the answer for this one. But I did something similar to what Steven1978 mentioned but only using CloudFormation.
The config you're looking for is the InputTransformer. Check this example for a YAML CloudFormation template for an Event Rule:
rEventRuleForFileUpload:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
State: "ENABLED"
EventPattern:
source:
- "aws.s3"
detail-type:
- 'AWS API Call via CloudTrail'
detail:
eventSource:
- s3.amazonaws.com
eventName:
- "PutObject"
- "CompleteMultipartUpload"
requestParameters:
bucketName: "{YOUR_BUCKET_NAME}"
Targets:
- Id: '{YOUR_ECS_CLUSTER_ID}'
Arn: !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${NAME_OF_YOUR_CLUSTER_RESOURCE}"
RoleArn: !GetAtt {YOUR_ROLE}.Arn
EcsParameters:
TaskCount: 1
TaskDefinitionArn: !Ref {YOUR_TASK_DEFINITION}
LaunchType: FARGATE
{... WHATEVER CONFIG YOU MIGHT HAVE...}
InputTransformer:
InputPathsMap:
s3_bucket: "$.detail.requestParameters.bucketName"
s3_key: "$.detail.requestParameters.key"
InputTemplate: '{ "containerOverrides": [ { "name": "{THE_NAME_OF_YOUR_CONTAINER_DEFINITION}", "environment": [ { "name": "EVENT_BUCKET", "value": <s3_bucket> }, { "name": "EVENT_OBJECT_KEY", "value": <s3_key> }] } ] }'
With this approach, you'll be able to get the s3 bucket name (EVENT_BUCKET) and the s3 object key (EVENT_OBJECT_KEY) as environment variables inside your container.
The info isn't very clear, indeed, but here are some sources I used to finally get it working:
Container Override;
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html
InputTransformer:
https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_InputTransformer.html#API_InputTransformer_Contents

Get TargetGroupArn from name?

You use TargetGroupArn in a CF template for ECS services. I have a situation where the target group has already been created and I want to make this a param for the template
But those arn's are awful:
arn:aws:elasticloadbalancing:us-east-1:123456:targetgroup/mytarget/4ed48ba353064a79
That unique number at the end makes this almost impossible. Can I reference the target by name instead of full arn in the template?
Maybe i can use Fn::GetAtt here but not sure what that looks like
This doesn't work:
TargetGroupArn: !GetAtt mytarget.TargetGroupName
I get error: An error occurred (ValidationError) when calling the CreateChangeSet operation: Template error: instance of Fn::GetAtt references undefined resource mytarget
Unfortunately with Target Groups, you won't be able to use convention to determine it's ARN due to the extra string at the end.
If the Target Group was created in Cloudformation, it's easy enough to get the ARN output by using !Ref myTargetGroup.
If the Target Group was created in another CF stack, try Exporting the Target Group ARN and use Fn::ImportValue when creating the ECS Service to input the Target Group ARN.
Type: "AWS::ECS::Service"
Properties:
...
LoadBalancers:
- ContainerName: MyContainer
ContainerPort: 1234
TargetGroupArn: !ImportValue myExportedTargetGroupARN
...
If you want to use the available Target-group, You pass the target group name as the default parameter to the Service CF template.
Internally refer the default parameter as the ref to the TargetGroupArn in the Action section of the LiestnerRule It will get the target group ARN.
Check this link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html
{
"Parameters": {
"VPC": {
...
"TargetGroup": {
"Description": "TargetGroup name for ListenerRule",
"Type": "String",
"Default": "my-target"
}
},
"Resources": {
"Service": {
"TaskDefinition": {
....
"ListenerRule": {
....
"Actions": [
{
"TargetGroupArn": {
"Ref": "TargetGroup"
},
"Type": "forward"
}
]
}
},
"ServiceRole": {
}
}