Problem when deploying a SageMaker Multi-Model Endpoints with AWS CDK/CloudFormation - deployment

I am trying to automate the deployment of a SageMaker multi-model endpoints with AWS CDK using Python language (I guess it would be the same by directly writing a CloudFormation template in json/yaml format), but when trying to deploy it, error occurs at the creation of the SageMaker model.
Here is part of the CloudFormation template made with the cdk synth command:
Resources:
smmodelexecutionrole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: sagemaker.amazonaws.com
Version: "2012-10-17"
Policies:
- PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Resource:
Fn::Join:
- ""
- - "arn:"
- Ref: AWS::Partition
- :s3:::<bucket_name>/deploy_multi_model_artifact/*
Version: "2012-10-17"
PolicyName: policy_s3
- PolicyDocument:
Statement:
- Action: ecr:*
Effect: Allow
Resource:
Fn::Join:
- ""
- - "arn:"
- Ref: AWS::Partition
- ":ecr:"
- Ref: AWS::Region
- ":"
- Ref: AWS::AccountId
- :repository/<my_ecr_repository>
Version: "2012-10-17"
PolicyName: policy_ecr
Metadata:
aws:cdk:path: <omitted>
smmodel:
Type: AWS::SageMaker::Model
Properties:
ExecutionRoleArn:
Fn::GetAtt:
- smmodelexecutionrole
- Arn
Containers:
- Image: xxxxxxxxxxxx.dkr.ecr.<my_aws_region>.amazonaws.com/<my_ecr_repository>/multi-model:latest
Mode: MultiModel
ModelDataUrl: s3://<bucket_name>/deploy_multi_model_artifact/
ModelName: MyModel
Metadata:
aws:cdk:path: <omitted>
When running cdk deploy on the Terminal, the following error occur:
3/6 | 7:56:58 PM | CREATE_FAILED | AWS::SageMaker::Model | sm_model (smmodel)
Could not access model data at s3://<bucket_name>/deploy_multi_model_artifact/.
Please ensure that the role "arn:aws:iam::xxxxxxxxxxxx:role/<my_role>" exists
and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com".
Also ensure that the role has "s3:GetObject" permissions and that the object is located in <my_aws_region>.
(Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: xxxxx)
What I have:
An ECR repository containing the docker image
A S3 bucket containing the model artifacts (.tar.gz files) inside the "folder" "deploy_multi_model_artifact"
To test if it is a IAM role issue, I tried to replace MultiModel by SingleModel and replace s3://<bucket_name>/deploy_multi_model_artifact/ with s3://<bucket_name>/deploy_multi_model_artifact/one_of_my_artifacts.tar.gz, and I could create successfully the model. I am then guessing that it is not a problem related with the IAM contrary to what the error message tells me (but I may make a mistake!) as it seems .
So I am wondering where the problem comes from. This is even more confusing as I have already deployed this multi-model endpoints using boto3 without problem.
Any help would be greatly appreciated !!
(About Multi-Model Endpoints deployment: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.ipynb)

Problem was that I forgot to add SageMaker access permissions to the IAM role.
I can deploy the multi-model endpoints by adding the SageMaker FullAccess managed policy to the IAM role.

Related

The state machine IAM Role is not authorized to access the Log Destination

I am trying to deploy a CF stack which is failing because of an IAM permission issue. The concerning resources in the stack for this issue are:
state machine (step function)
Cloudwatch log group
A subscription filter for the log group
This subscription filter forwards the logs to Kenesis where the logs are streamed into splunk. Coming back to my issue, when I try to deploy the above CF stack, I get the following error: The state machine IAM Role is not authorized to access the Log Destination
I have given the following permissions to the role attached to the state machine
StateMachineRole:
Type: AWS::IAM::Role
DeletionPolicy: Retain
Properties:
RoleName: StateMachineRole-${self:custom.env.stage}
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: states.amazonaws.com
Action: sts:AssumeRole
LambdaPolicy:
Type: AWS::IAM::Policy
DeletionPolicy: Retain
DependsOn: CustomLogGroup
Properties:
PolicyName: LambdaPolicy-${self:custom.env.stage}
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'lambda:InvokeFunction'
Resource: lambdaArn
- Effect: Allow
Action:
- 'logs:CreateLogDelivery'
- 'logs:GetLogDelivery'
- 'logs:UpdateLogDelivery'
- 'logs:DeleteLogDelivery'
- 'logs:ListLogDeliveries'
- 'logs:PutLogEvents'
- 'logs:PutResourcePolicy'
- 'logs:DescribeResourcePolicies'
- 'logs:DescribeLogGroups'
- 'logs:PutDestination'
- 'logs:PutSubscriptionFilter'
- 'logs:PutDestinationPolicy'
Resource: !GetAtt CustomLogGroup.Arn
/*CustomLogGroup*/
CustomLogGroup:
Type: AWS::Logs::LogGroup
Properties:
KmsKeyId: !ImportValue XXXXXXX
LogGroupName: CustomLogGroupName
RetentionInDays: ${file(./.env.${self:custom.env.stage}.yaml):cloudwatchLogs.retentionDays
Referred to the following SO question: Aws step function deployment log access issue

AWS Private API gateway deployment error when using serverless deploy

I am hitting the below error when I try to create a PRIVATE endpoint type using serverless in AWS.
Below is the serverless.yml file for reference
provider:
name: aws
endpointType: PRIVATE
vpcEndpointIds:
- vpce-xxxx
region: ap-southeast-2
apiKeys:
- ${self:custom.actualStage}-xxxxxx2
resourcePolicy:
- Effect: Allow
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*/*/*
Condition:
StringEquals:
aws:SourceVpc:
- "vpc-xxxxx"
plugins:
- serverless-plugin-warmup
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'lambda:InvokeFunction'
Resource: "*"
Error from logs:
CREATE_FAILED: ApiGatewayDeployment1655171823778 (AWS::ApiGateway::Deployment)
Resource handler returned message: "Private REST API doesn't have a resource policy attached to it (Service: ApiGateway, Status Code: 400, Request ID: e321f00e-42b6-4ef6-b984-46500ca40492)" (RequestToken: 475924b8-998d-58fc-89bd-51fc0b80f2d4, HandlerErrorCode: InvalidRequest)
Not sure if AWS released new changes in the serverless configuration for API GW. Your policy version should have worked. However, I have tested this on my end. We now have to mention the resourcePolicy under the apiGateway attribute. Please use the following this will resolve your issue:
apiGateway:
resourcePolicy:
- Effect: Allow
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*
- Effect: Deny
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*
Condition:
StringNotEquals:
aws:SourceVpc:
- 'vpc-*******'

CloudFormation template to update the IAM Role's policy

Is it possible to create a CloudFormation template that takes the IAM role's ARN as the input and updates its policy assigned to it to add more privileges?I tried to work with the CloudFormation designer but it is very confusing and not straightforward,
Yes, you can do this with a template such as this:
Description: Add policy to existing role
Parameters:
MyExistingRoleName:
Type: String
Description: Name of the Role you want to add a policy to
Resources:
MyNewPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: "my-new-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "s3:ListAllMyBuckets"
Resource:
- "*"
Roles:
- !Ref MyExistingRoleName

Run a crawler using CloudFormation template

This CloudFormation template works as expected and creates all the resources required by this article:
Data visualization and anomaly detection using Amazon Athena and Pandas from Amazon SageMaker | AWS Machine Learning Blog
But the WorkflowStartTrigger resource does not actually run the crawler. How do I run a crawler using the CloudFormation template?
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "glue.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
MyDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput:
Name: "dbcrawler123"
Description: "TestDatabaseDescription"
LocationUri: "TestLocationUri"
Parameters:
key1 : "value1"
key2 : "value2"
MyCrawler2:
Type: AWS::Glue::Crawler
Properties:
Description: example classifier
Name: "testcrawler123"
Role: !GetAtt MyRole.Arn
DatabaseName: !Ref MyDatabase
Targets:
S3Targets:
- Path: 's3://nytaxi162/'
SchemaChangePolicy:
UpdateBehavior: "UPDATE_IN_DATABASE"
DeleteBehavior: "LOG"
TablePrefix: test-
Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"
WorkflowStartTrigger:
Type: AWS::Glue::Trigger
Properties:
Description: Trigger for starting the Crawler
Name: StartTrigger
Type: ON_DEMAND
Actions:
- CrawlerName: "testcrawler123"
You should be able to do that by creating a custom resource attached to a lambda whereby the lambda actually does the action of starting the crawler. You should be able to even make it wait for the crawler to complete its execution
CloudFormation directly doesn't run crawlers, it just create them.
But you can create a schedule in order to run a crawler while defining trigger:
ScheduledJobTrigger:
Type: 'AWS::Glue::Trigger'
Properties:
Type: SCHEDULED
StartOnCreation: true
Description: DESCRIPTION_SCHEDULED
Schedule: cron(5 * * * ? *)
Actions:
- CrawlerName: "testcrawler123"
Name: ETLGlueTrigger
If needed to run crawler as part of CloudFormation stack creation, Lambda could be used.

How to call a resource from one yaml template to another yaml template using cloudformation

I need some guidance on cloudformation templates.
I have a stack called test1.yaml, there i created an IAM role called S3Role.
Now I have another stack called test2.yaml, there i created a managed policy to attach to existing iam role.
Now i want to call test1.yml file S3Role in test2.yml file of managed policy.
Can anyone help me with the script?
Can anyone help me with the script.
Obviously due to lack of details in your question, its not possible to provide any script. But I can provide general psudo-code.
test1.yaml
You will have to export the S3Role Arn or Name
Resources:
S3Role:
Type: IAM::ROLE
<rest of role definition>
Outputs:
RoleArn:
Value: !GetAtt S3Role.Arn
Exports:
Name: RoleArn
test2.yml
You will have to import the role exported Arn (or name) from test1.yaml:
Resources:
SomeResouce:
Properties:
Role: !ImportValue RoleArn
Hope this helps.
You need export the role from stack 1 and then import it in stack 2
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-exports.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html
Providing the complete script for cross-referencing an AWS resource in CloudFormation template.
test1.yaml has an IAM role (logical ID: IAMRole) which we export through the Outputs block. Also notice that the indentation of Outputs block is same as that of Resources block.
The Outputs block serves many purposes. From the AWS Documentation
The optional Outputs section declares output values that you can
import into other stacks (to create cross-stack references), return in
response (to describe stack calls), or view on the AWS CloudFormation
console.
test1.yaml
Resources:
IAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: TrustPolicy
Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonS3FullAccess
Path: /
RoleName: IAMRole
Outputs:
ExportIAMRole:
Description: Export the IAMRole to use in test2.yaml
Value: !Ref IAMRole
Export:
Name: IAMRole
In test2.yaml we import the value by referencing the name we have given under Export in Outputs block.
test2.yaml
Resources:
IAMPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
ManagedPolicyName: IAMPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- kms:ListAliases
- kms:Encrypt
- kms:Decrypt
Resource: "*"
Roles:
- !ImportValue IAMRole