409 (request "Conflict") when creating second Endpoint connection in MongoDB Atlas using Terraform - mongodb

I need to create many MongoDB Atlas endpoint connections using terraform.
I successfully create first, using this code:
#Private endpoint connection
resource "mongodbatlas_private_endpoint" "dbpe" {
project_id = var.prj_id
provider_name = "AWS"
region = var.aws_region
}
#AWS endpoint for secure connect to mongo db
resource "aws_vpc_endpoint" "ec2" {
vpc_id = var.sh_vpc
#service_name = "com.amazonaws.${var.aws_region}.ec2"
service_name = mongodbatlas_private_endpoint.dbpe.endpoint_service_name
vpc_endpoint_type = "Interface"
security_group_ids = [
aws_security_group.lb_sg.id,
]
subnet_ids = [
aws_subnet.subnet1.id,
var.sh_subnet
]
tags = {
"Name" = local.tname
}
#private_dns_enabled = true
}
But when I try to use this code second time in another folder (another tfstate) it failed cause error:
Error: error creating MongoDB Private Endpoints Connection: POST https://cloud.mongodb.com/api/atlas/v1.0/groups/***/privateEndpoint: 409 (request "Conflict") A PrivateLink Endpoint Service already exists for AWS region US_EAST_2.
As I understand, a second "mongodbatlas_private_endpoint" "dbpe" trying to create another one Endpoint service. But, when I creating second Endpoint manually through WebUI, it using the same service like first Endpoint.
How I can tell to second Endpoint to use the existing service?
Or maybe it all wrong?
Please, help!
Thank you!

I found the solution.
Creating the "Endpoint Connection" really creates Endpoint only when you do it at first time. All of next times is creating an only association between Atlas endpoint and new AWS Endpoint.
In terraform I tried to create an Atlas endpoint second time and catch an error (because of limit - 1 endpoint per region). All I need to do - is create "Basic Endpoint" one time (by separate folder with own tfstate) and don't delete it. And for each new AWS endpoint need to create a new link from AWS Endpoint to "Basic". I do it by a terraform resource:
mongodbatlas_private_endpoint_interface_link
Resource "mongodbatlas_private_endpoint" is not need now. A "service_name" parameter in "aws_vpc_endpoint" you can hardcoded from "Basic" Endpoint. Use "output" to see mongodbatlas_private_endpoint.test.private_link_id - this is what you need.

Related

How to solve : "Received response status [FAILED] from custom resource. Message returned: Resource is not in the state certificateValidated"? CDK

I have the following error trying to create a static website inspired by https://github.com/aws-samples/aws-cdk-examples/blob/master/typescript/static-site/static-site.ts
const certificateArn = new acm.DnsValidatedCertificate(
this,
"SiteCertificateR53",
{
domainName: props.siteDomain,
hostedZone: props.zone,
region: "us-east-1", // Cloudfront only checks this region for certificates.
}
).certificateArn;
new cdk.CfnOutput(this, "CertificateR53", {value: certificateArn});
Error:
Received response status [FAILED] from custom resource. Message returned: Resource is not in the state certificateValidated
If you don't need to do cross region stuff (e.g. us-east-1 needs a resource from us-west-2) using the following method provides the same benefit as DnsValidatedCertificate
const certificate = new acm.Certificate(this, `SiteCertificateR53`, {
domainName: props.siteDomain,
validation: acm.CertificateValidation.fromDns(props.zone)});
If you still gotta do cross-region stuff, then you should create and deploy your zone via AWS console first. That won't guarantee a fix though, this page can help if you're still stuck: https://docs.aws.amazon.com/acm/latest/userguide/troubleshooting-DNS-validation.html

How to create a bucket using the python SDK?

I'm trying to create a bucket in cloud object storage using python. I have followed the instructions in the API docs.
This is the code I'm using
COS_ENDPOINT = "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
# Create client
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
s3 = ibm_boto3.resource('s3')
def create_bucket(bucket_name):
print("Creating new bucket: {0}".format(bucket_name))
s3.Bucket(bucket_name).create()
return
bucket_name = 'test_bucket_442332'
create_bucket(bucket_name)
I'm getting this error - I tried setting CreateBucketConfiguration={"LocationConstraint":"us-south"}, but it doesnt seem to work
"ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to."
Resolved by going to https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints#endpoints
And choosing the endpoint specific to the region I need. The "Endpoint" provided with the credentials, is not the actual endpoint.

gRPC Node microservice talking to another microservice in istio mesh

I've got several gRPC microservices deployed via Istio in my k8s pod behind a gateway that handles the routing for web clients. Things work great when I need to send an RPC from client (browser) to any of these services.
I'm now at the point where I want to call service A from service B directly. How do I go about doing that?
Code for how both the servers are instantiated:
const server = new grpc.Server();
server.addService(MyService, new MyServiceImpl());
server.bindAsync(`0.0.0.0:${PORT_A}`, grpc.ServerCredentials.createInsecure(), () => {
server.start();
});
A Service Account is being used with GOOGLE_APPLICATION_CREDENTIALS and a secret in my deployment YAML.
To call service A from service B, I was thinking the code in service B would look something like:
const serviceAClient: MyServiceClient = new MyServiceClient(`0.0.0.0:${PORT_A}`, creds);
const req = new SomeRpcRequest()...;
serviceAClient.someRpc(req, (err: grpc.ServiceError, response: SomeRpcResponse) => {
// yay!
});
Is that naive? One thing I'm not sure about is the creds that I need to pass when instantiating the client. I get complaints that I need to pass ChannelCredentials, but every mechanism I've tried to create those creds has not worked.
Another thing I'm realizing is that 0.0.0.0 can't be correct because each service is in its own container paired with a sidecar proxy... so how do I route RPCs properly and attach proper creds?
I'm trying to construct the creds this way:
let callCreds = grpc.CallCredentials.createFromGoogleCredential(myOauthClient);
let channelCreds = grpc.ChannelCredentials.createSsl().compose(callCreds);
const serviceAClient = new MyServiceClient(`0.0.0.0:${PORT_A}`, channcelCreds);
and I'm mysteriously getting the following error stack:
UnhandledPromiseRejectionWarning: TypeError: Channel credentials must be a ChannelCredentials object
at new ChannelImplementation (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/channel.js:69:19)
at new Client (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/client.js:58:36)
at new ServiceClientImpl (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/make-client.js:58:5)
at PresenceService.<anonymous> (/bish/src/servers/presence/dist/presence.js:348:44)
at step (/bish/src/servers/presence/dist/presence.js:33:23)
at Object.next (/bish/src/servers/presence/dist/presence.js:14:53)
at fulfilled (/bish/src/servers/presence/dist/presence.js:5:58)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Which is odd because channelCreds is a ComposedChannelCredentialsImpl which, in fact, extends ChannelCredentials
Ok, at least the root cause of the "Channel credentials must be a ChannelCredentials object" error is now known. I'm developing node packages side-by-side as symlinks and each of the dependencies has their own copy of grpc-js in it.
https://github.com/npm/npm/issues/7742#issuecomment-257186653

How to create/start cluster from data bricks web activity by invoking databricks rest api

I have 2 requirements:
1:I have a clusterID. I need to start the cluster from a "Wb Activity" in ADF. The activity parameters look like this:
url:https://XXXX..azuredatabricks.net/api/2.0/clusters/start
body: {"cluster_id":"0311-004310-cars577"}
Authentication: Azure Key Vault Client Certificate
Upon running this activity I am encountering with below error:
"errorCode": "2108",
"message": "Error calling the endpoint
'https://xxxxx.azuredatabricks.net/api/2.0/clusters/start'. Response status code: ''. More
details:Exception message: 'Cannot find the requested object.\r\n'.\r\nNo response from the
endpoint. Possible causes: network connectivity, DNS failure, server certificate validation or
timeout.",
"failureType": "UserError",
"target": "GetADBToken",
"GetADBToken" is my activity name.
The above security mechanism is working for other Databricks related activity such a running jar which is already installed on my databricks cluster.
2: I want to create a new cluster with the below settings:
url:https://XXXX..azuredatabricks.net/api/2.0/clusters/create
body:{
"cluster_name": "my-cluster",
"spark_version": "5.3.x-scala2.11",
"node_type_id": "i3.xlarge",
"spark_conf": {
"spark.speculation": true
},
"num_workers": 2
}
Upon calling this api, if a cluster creation is successful I would like to capture the cluster id in the next activity.
So what would be the output of the above activity and how can I access them in an immediate ADF activity?
For #2 ) Can you please check if you change the version
"spark_version": "5.3.x-scala2.11"
to
"spark_version": "6.4.x-scala2.11"
if that helps

Terraform - AWS - API Gateway dependency conundrum

I am trying to provision some AWS resources, specifically an API Gateway which is connected to a Lambda. I am using Terraform v0.8.8.
I have a module which provisions the Lambda and returns the lambda function ARN as an output, which I then provide as a parameter to the following API Gateway provisioning code (which is based on the example in the TF docs):
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
}
# Variables
variable "myregion" { default = "eu-west-2" }
variable "accountId" { default = "" }
variable "lambdaArn" { default = "" }
variable "stageName" { default = "lab" }
# API Gateway
resource "aws_api_gateway_rest_api" "api" {
name = "myapi"
}
resource "aws_api_gateway_method" "method" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "GET"
authorization = "NONE"
}
resource "aws_api_gateway_integration" "integration" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "${aws_api_gateway_method.method.http_method}"
integration_http_method = "POST"
type = "AWS"
uri = "arn:aws:apigateway:${var.myregion}:lambda:path/2015-03-31/functions/${var.lambdaArn}/invocations"
}
# Lambda
resource "aws_lambda_permission" "apigw_lambda" {
statement_id = "AllowExecutionFromAPIGateway"
action = "lambda:InvokeFunction"
function_name = "${var.lambdaArn}"
principal = "apigateway.amazonaws.com"
source_arn = "arn:aws:execute-api:${var.myregion}:${var.accountId}:${aws_api_gateway_rest_api.api.id}/*/${aws_api_gateway_method.method.http_method}/resourcepath/subresourcepath"
}
resource "aws_api_gateway_deployment" "deployment" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
stage_name = "${var.stageName}"
}
When I run the above from scratch (i.e. when none of the resources exist) I get the following error:
Error applying plan:
1 error(s) occurred:
* aws_api_gateway_deployment.deployment: Error creating API Gateway Deployment: BadRequestException: No integration defined for method
status code: 400, request id: 15604135-03f5-11e7-8321-f5a75dc2b0a3
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
If I perform a 2nd TF application it consistently applies successfully, but every time I destroy I then receive the above error upon the first application.
This caused me to wonder if there's a dependency that I need to explicitly declare somewhere, I discovered #7486, which describes a similar pattern (although relating to an aws_api_gateway_integration_response rather than an aws_api_gateway_deployment). I tried manually adding an explicit dependency from the aws_api_gateway_deployment to the aws_api_gateway_integration but this had no effect.
Grateful for any thoughts, including whether this may indeed be a TF bug in which case I will raise it in the issue tracker. I thought I'd check with the community before doing so in case I'm missing something obvious.
Many thanks,
Edd
P.S. I've asked this question on the Terraform user group but this seems to get very little in the way of responses, I'm yet to figure out the cause of the issue hence now asking here.
You are right about the explicit dependency declaration.
Normally Terraform would be able to figure out the relationships and schedule create/update/delete operations accordingly to that - this is mostly possible because of the interpolation mechanisms under the hood (${resource_type.ref_name.attribute}). You can display the relationships affecting this in a graph via terraform graph.
Unfortunately in this specific case there's no direct relationship between API Gateway Deployments and Integrations - meaning the API interface for managing API Gateway resources doesn't require you to reference integration ID or anything like that to create deployment and the api_gateway_deployment resource in turn doesn't require that either.
The documentation for aws_api_gateway_deployment does mention this caveat at the top of the page. Admittedly the Deployment not only requires the method to exist, but integration too.
Here's how you can modify your code to get around it:
resource "aws_api_gateway_deployment" "deployment" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
stage_name = "${var.stageName}"
depends_on = ["aws_api_gateway_method.method", "aws_api_gateway_integration.integration"]
}
Theoretically the "aws_api_gateway_method.method" is redundant since the integration already references the method in the config:
http_method = "${aws_api_gateway_method.method.http_method}"
so it will be scheduled for creation/update prior to the integration either way, but if you were to change that to something like
http_method = "GET"
then it would become necessary.
I have submitted PR to update the docs accordingly.