Hello every one i have just started to get my hands dirty with ceph object storage i.e. radosgateway and for this purpose have spun out a very basic single node ceph/daemon docker container which works perfectly fine for both s3cmd and java s3 API (the mgr dashboard don't work though container shuts down when issuing command ceph mgr module enable dashboard) but one thing i cant seem to figure out is how can we specify more than one endpoint for our java s3 client to connect to our cluster? does it have something to do with HTTP front-ends? please need some pointers or a sample example would be great.Following is my code to connect to a single node ceph cluster built using ceph/daemon image's docker container.
String accessKey = "demoKey";
String secretKey = "demoKey";
try {
ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setProtocol(Protocol.HTTP);
System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY,"true");
if (SDKGlobalConfiguration.isCertCheckingDisabled())
{
System.out.println("Cert checking is disabled");
}
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 conn = new AmazonS3Client(credentials);
conn.setEndpoint("http://ubuntu:8080"); //more than one endpoint ??
List<Bucket> buckets = conn.listBuckets();
for (Bucket bucket : buckets) {
System.out.println(bucket.getName() + "\t" +
StringUtils.fromDate(bucket.getCreationDate()));
}
}catch(Exception ex)
{
ex.printStackTrace();
}
Finally my ceph version
ceph version 14.2.4 nautilus (stable)
The Ceph Object Gateway can have multiple instances. These are combined by some load balancer. You have one end point that is distributing the load onto the ceph object gateway instances. The load balancer itself can be scaled as well (i.e. round-robin DNS or whatnot).
I found a nice use case here. Maybe it helps. Have a look at the media storage architecture.
Related
I created a simple node.js/express app and created docker image and successfully pushed it to aws ecr.
Next, I created a cdk project to deploy this container to fargate with public application load balancer. ecs_patterns.ApplicationLoadBalancedFargateService
Although the deployment cmd (cdk deploy) was successful, cluster page in aws console shows "No tasks running" and Services tab within the cluster shows red bar with "0/1 Tasks running" and Tasks tab within cluster shows tasks are getting created and stopped (every 1 or 2 min, a task is created and eventually stopped and a new one is created and this keeps on going forever)
Going inside a stopped task and its Log tab shows
ERROR: Connecting to MongoDB failed. Please check if MongoDB server is running at the correct host/port.. This is the error message I have in my app when connection to mongodb fails when the server is initialized.
The DB credentials and connection url are valid (see below) and it runs in a separate EC2 instance with EIP and domain name. In fact, I can connect to the DB from my dev machine which is outside aws.
Also, just for trial, I created a stack manually through console by creating security groups (for load balancer and service), target group, application load balancer, listener (port 80 HTTP), cluster, task definition (with correct db credentials set in env var), service, etc., it's working without any issue.
All I want is to create similar stack using cdk (I don't want to manually create/maintain it)
Any clue on why connection to external server/db is failing from a fargate container would be very useful. I'm unable to compare the "cdk created cloudformation template" (that's not working) with the "manually created stack" (that's working) as there are too many items in the autogenerated template.
Here is the cdk code based on aws sample code:
const vpc = new ec2.Vpc(this, "MyVpc", { maxAzs: 2 });
const cluster = new ecs.Cluster(this, "MyCluster", { vpc });
const logDriver = ecs.LogDriver.awsLogs({ streamPrefix: "api-log" });
const ecrRepo = ecr.Repository.fromRepositoryName(this, "app-ecr", "abcdef");
new ecs_patterns.ApplicationLoadBalancedFargateService(
this, "FargateService", {
assignPublicIp: true,
cluster,
desiredCount: 1,
memoryLimitMiB: 1024,
cpu: 512,
taskImageOptions: {
containerName: "api-container",
image: ecs.ContainerImage.fromEcrRepository(ecrRepo),
enableLogging: true,
logDriver,
environment: { MONGO_DB_URL: process.env.DB_URL as string }
},
publicLoadBalancer: true,
loadBalancerName: "api-app-lb",
serviceName: "api-service"
}
);
It turned out to be a silly mistake! Instead of MONGO_DB_URL it should be DB_URL because that's what my node.js/express server in the container is using.
Description
Is it possible for Spring Data Redis to use Elasticache's configuration endpoint to perform all cluster operations (i.e., reading, writing, etc.)?
Long Description
I have a Spring Boot application that uses a Redis cluster as data store. The Redis cluster is hosted on AWS Elasticache running in cluster-mode enabled. The Elasticache cluster has 3 shards spread out over 12 nodes. The Redis version that the cluster is running is 6.0.
The service isn't correctly writing or retrieving data from the cluster. Whenever performing any of these operations, I get a message similar to the following:
io.lettuce.core.RedisCommandExecutionException: MOVED 16211 10.0.7.254:6379
In searching the internet, it appears that the service isn't correctly configured for a cluster. The fix seems to be set the spring.redis.cluster.nodes property with a list of all the nodes in the Elasticache cluster (see here and here). I find this rather needless, considering that the Elasticache configuration endpoint is supposed to be used for all read and write operations (see the "Finding Endpoints for a Redis (Cluster Mode Enabled) Cluster" section here).
My question is this: can Spring Data Redis use Elasticache's configuration endpoint to perform all reads and writes, the way the AWS documentation describes? I'd rather not hand over a list of all the nodes if Spring Data Redis can use the configuration endpoint the way its meant to be used. This seems like a serious limitation to me.
Thanks in advance!
Here is what I found works:
#Bean
public RedisConnectionFactory lettuceConnectionFactory()
{
LettuceClientConfiguration config =
LettucePoolingClientConfiguration
.builder()
.*your configuration settings*
.build();
RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration();
clusterConfig.addClusterNode(new RedisNode("xxx.v1tc03.clustercfg.use1.cache.amazonaws.com", 6379));
return new LettuceConnectionFactory(clusterConfig, config);
}
where xxx is the name from your elasticache cluster.
We have a cluster of 3 nodes, 2 of them are offline (missing) and I cannot get them to rejoin the cluster automatically only the master is Online.
Usually, you can use innodb admin:
var cluster = dba.getCluster();
but I cannot use the cluster instance because the metadata is not up to date. But I cannot upgrade the meta data because the missing members are required to be online to use dba.upgradeMetadata(). (Catch 22)
I tried to dissolve the cluster by using:
var cluster = dba.rebootClusterFromCompleteOutage();
cluster.dissolve({force:true});
but this requires the metadata to be updated as well.
Question is, how do I dissolve the cluster completely or upgrade the metadata so that I can use the cluster. methods.
This "chicken-egg" issue was fixed in MySQL Shell 8.0.20. dba.rebootClusterFromCompleteOutage() is now allowed in such situation:
BUG#30661129 – DBA.UPGRADEMETADATA() AND DBA.REBOOTCLUSTERFROMCOMPLETEOUTAGE() BLOCK EACH OTHER
More info at: https://mysqlserverteam.com/mysql-shell-adminapi-whats-new-in-8-0-20/
If you have a cluster where each node upgrades to the latest version of mysql and the cluster isn't fully operational and you need to update your metadata for mysqlsh, you'll need to use an older version of mysqlsh for example, https://downloads.mysql.com/archives/shell/ to get the cluster back up and running. Once it is up and running you can use the dba.upgrademetadata on the R/W node - make sure you update all of your routers or they will lose connection.
I'd like to use Fargate to run two containers - one for the main project's backend, and another for the database (MongoDB). The basic example included in the GitHub repo shows how to run a single container on Fargate using CDK, however I still have 2 issues:
The example doesn't show how to run two containers.
I'd like to scale the database containers but have them share the data storage (so that the data gets stored in a central place and stays synchronized between the different containers).
I've figured out how to (sort of) fixed the first issue, similarly to how ecs.LoadBalancedFargateService is implemented, however the second issue still remains.
For reference, this is what I have so far in stack.ts (the rest is the basic boilerplate cdk init app --language typescript generates for you):
import cdk = require("#aws-cdk/cdk");
import ec2 = require("#aws-cdk/aws-ec2");
import ecs = require("#aws-cdk/aws-ecs");
import elbv2 = require("#aws-cdk/aws-elasticloadbalancingv2");
const {ApplicationProtocol} = elbv2;
export class AppStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create VPC and Fargate Cluster
const vpc = new ec2.VpcNetwork(this, "FargateVPC", {
maxAZs: 2
});
const cluster = new ecs.Cluster(this, "Cluster", {vpc});
// Create task definition
const fargateTaskDefinition = new ecs.FargateTaskDefinition(this, "FargateTaskDef", {
memoryMiB: "512",
cpu: "256"
});
// Create container from local `Dockerfile`
const appContainer = fargateTaskDefinition.addContainer("Container", {
image: ecs.ContainerImage.fromAsset(this, "Image", {
directory: ".."
})
});
// Set port mapping
appContainer.addPortMappings({
containerPort: 5000
});
// Create container from DockerHub image
const mongoContainer = fargateTaskDefinition.addContainer("MongoContainer", {
image: ecs.ContainerImage.fromDockerHub("mongo")
});
// Set port mapping
mongoContainer.addPortMappings({
containerPort: 27017
});
// Create service
const service = new ecs.FargateService(this, "Service", {
cluster,
taskDefinition: fargateTaskDefinition,
desiredCount: 2
});
// Configure task auto-scaling
const scaling = service.autoScaleTaskCount({
maxCapacity: 5
});
scaling.scaleOnCpuUtilization("CpuScaling", {
targetUtilizationPercent: 70
});
// Create service with built-in load balancer
const loadBalancer = new elbv2.ApplicationLoadBalancer(this, "AppLB", {
vpc,
internetFacing: true
});
// Allow incoming connections
loadBalancer.connections.allowFromAnyIPv4(new ec2.TcpPort(5000), "Allow inbound HTTP");
// Create a listener and listen to incoming requests
const listener = loadBalancer.addListener("Listener", {
port: 5000,
protocol: ApplicationProtocol.Http
});
listener.addTargets("ServiceTarget", {
port: 5000,
protocol: ApplicationProtocol.Http,
targets: [service]
});
// Output the DNS where you can access your service
new cdk.Output(this, "LoadBalancerDNS", {
value: loadBalancer.dnsName
});
}
}
Thanks in advance.
Generally, running a database in a Fargate container is not recommended since there is not currently a good solution for persisting data. You could integrate a hook that copies data into something like S3 prior to a task stopping, but generally those kinds of solutions are very fragile and not recommended.
You may want to check out DocumentDB as an alternative to running your own MongoDB instances, though support for DocumentDB constructs in the CDK are not yet fully fleshed out.
Another alternative is to run regular ECS tasks and attach an EBS volume on your EC2 Instance. Then you can use docker volumes to mount the EBS volume to your container. With this approach, you'll need to tag the instance metadata and use an ECS placement constraint to ensure that your task gets placed on the instance that has the EBS volume attached.
If either of these approaches works for you, feel free to open a feature request on the CDK repository. Hope this helps!
Is AWS Fargate a hard requirement?
If not, you could opt for simple ECS + Ec2, it supports the use of persistent data volumes:
Fargate tasks only support nonpersistent storage volumes.
For EC2 tasks, use data volumes in the following common examples:
To provide persistent data volumes for use with a container
To define an empty, nonpersistent data volume and mount it on multiple containers
To share defined data volumes at different locations on different containers on the same container instance
To provide a data volume to your task that is managed by a third-party volume driver
I haven't tried it myself but it seems that CDK has stable support for ECS + Ec2.
PS the link to the basic example is broken, I tried to find the new location but in the new example repository without success.
I am trying to setup my microservices architecture using AWS Elastic Beanstalk and Docker. That is very easy to do, but when I launch the environment, it launches into the default VPC, thus giving public IP's to the instances. Right now, that's not too much of a concern.
What I am having a problem with is how to set up the MongoDB architecture. I have read: recommended way to install mongodb on elastic beanstalk but still remain unsure on how to set this up.
So far I have tried:
Using the CloudFormation template from AWS here: http://docs.aws.amazon.com/quickstart/latest/mongodb/step2b.html to launch a primary with 2 replica node setup into the default VPC, but this gives and assigns public access to the Mongo nodes. I also am not sure how to connect my application since this does not add a NAT instance - do I simply connect directly to the primary node? In case of failure for this node, will the secondary node's IP become the same as that of the primary node so that all connections remain consistent? Or do I need to add my own NAT instance?
I have also tried launching MongoDB into its own VPC (https://docs.aws.amazon.com/quickstart/latest/mongodb/step2a.html) and giving access via the NAT, but this means having two different VPCs (one for my EB instances and one for the MongoDB). In this case would I connect to the NAT from my EB VPC in order to route requests to the databases?
I have also tried launching a new VPC for the MongoDB architecture first and then trying to launch EB into this VPC. For some reason, the load balancing setup won't let me add into the subnets, giving me the error: "Custom Availability Zones option not supported for VPC environments".
I am trying to launch all this in us-west-1. It's been two days now and I have no idea where to go or what the right way is to tackle this issue. I want the databases to be private (no public access) with a NAT gateway, so ideally my third method seems what I want, but I cannot seem to add the new EB instances/load balancer into the newly-created MongoDB VPC. This is the setup I'm going for: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/images/default-vpc-diagram.png but I am trying to use the templates to do this.
What am I doing wrong here? Any help would be much, much appreciated. I have read up a lot about this but still am not sure where to go from here.
Thanks a lot in advance!
Im having this same issue. There seems to be a complete lack of documentation on how to connect an Elastic Beanstalk node.js / express app with the aws Quickstart mongodb cluster set up documentation.
When I run the aws mongo quickstart though it launches a NAT which is public and also a private primary node... maybe this is part of your issue?