I am working on spring cloud dataflow stream app. I am able to run Spring cloud data flow server locally with the skipper running in Cloud Foundry with below configuration . Now i am trying to run the same with the skipper running in kubernetes cluster. How can i specify the same ?
manifest.yml
---
applications:
- name: poc-scdf-server
memory: 1G
instances: 1
path: ../target/scdf-server-1.0.0-SNAPSHOT.jar
buildpacks:
- java_buildpack
env:
JAVA_VERSION: 1.8.0_+
JBP_CONFIG_SPRING_AUTO_RECONFIGURATION: '{enabled: false}'
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL:
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: <org>
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: <space>
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_DOMAIN: <url>
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: <user>
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: <pwd>
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SKIPSSLVALIDATION: true
SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI: <skipper_url> SPRING_CLOUD_GAIA_SERVICES_ENV_KEY_PREFIX:spring.cloud.dataflow.task.platform.cloudfoundry.accounts[default].connection.
In v2.3, we have recently added the platform-specific docker-compose.yml experience for the Local mode. You can find the new files here.
With this infrastructure, you could start SCDF locally, but also bring the platform accounts for CF, K8s, or even both! See docs.
You can also use the DockerComposeIT.java to bring thigs up and running with automation, as well.
Related
Platform used : Kubernetes.
I have an issue with Spring cloud stream url. I am launching my spring cloud tasks using spring cloud stream. Streams are deployed in kubernetes platform. Stream contains http-kafka as source and taskLauncerKafka as sink. I used http-kafka kubernetes service url to launch tasks. Kubernetes service url changes after each deployment which causes problem.The changes in the service name after each stream deployment is difficult to manage. I have tried enabling loadbalacer also. In that case also external ip-address changed after each stream roll-out.
I am using skipper for managing the deployments. Every time stream is deployed stream version changes which also changes stream url.
In my case , I have multiple instances from where I can launch spring-cloud task. If the stream url changes I need to make changes in the configmap of the deployment project for all instance and need to redeploy all instance.
Any solution ? I am thinking of centralised configuration management using spring-cloud-config server or zookeeper. In this case also I need to update the url. I can avoid deploying multiple instances using centralised configuration management.
Skipper server version : 2.4.1.RELEASE
Dataflow server version : 2.5.1.RELEASE
Which version of SCDF/Skipper you are running?
This looks similar to the issue https://github.com/spring-cloud/spring-cloud-skipper/issues/953 which was subsequently addressed in Skipper 2.6.0.
I did research but couldn't find any "howTo" for deploying Metabase with Cloud Run on GCP, I only found Q&A the problems by deploying it.
My goal is to deploy Metabase with Cloud run and use Postgres as database. I have already deployed app by Cloud Run and have cloudbuild.yml pipeline on git that I use it for building my app I just wanna add Metabase.
Any directions or solutions?
It is not recommended to deploy Metabase on Cloud Run because:
Computation is scoped to a request. You should only expect to be able
to do computation within the scope of a request: a container instance
does not have any CPU available if it is not processing a request.
Container runtime contract
Therefore you might be facing issues loading the application and database connections timeout.
I think your best option is using AppEngine Flexible.
Install Metabase on Google Cloud with Docker – App Engine
It's very hard to use Metabase on Cloud Run. And Running it on Flexible App Engine is more expensive (~100$/month). Metabase entrypoint doing lot of processes and if Cloud Run decide to create another container instance it will display the loading metabase page so we have to do multiple manually refresh to get the new container ready.
I managed this problem by using more JVM JAVA_OPTS -Xmx4g and limit the MAX_INSTANCES to 1. I set a Cloud Scheduler task who does a get query to the metabase service base URL every 10 minutes.
With this configuration metabase will stay alive and Cloud Run will never create more than 1 container. So the refresh problem will not appear any more.
For the moment I don't have problems with this configuration but I think the best way to host metabase is to run it on Compute Engine VM.
I am trying to deploy an application to AWS ECS using codeship. I have my docker-compose file and everything is ready to be deployed. Codeship documentation says to do something like this in the codeship-steps.yml file:
aws ecs register-task-definition --cli-input-json file:///deploy/tasks/backend.json
aws ecs update-service --service my-backend-service --task-definition backend
My question is, is this file file:///deploy/tasks/backend.json something I have to provide manually or is it created automatically as well as the ECS task. because I keep getting this error from codeship
An error occurred (ClientException) when calling the RunTask operation: TaskDefinition not found.
file:///deploy/tasks/backend.json is something you provide to aws ecs register-task-definition.
Looked it up here, this will generate the structure of that file you want:
aws ecs register-task-definition --generate-cli-skeleton
You can then modify it, after throwing it in backend.json, for instance (provided you redirect the output of the command into a file called backend.json).
If you look at the suggested service definition:
awsdeployment:
image: codeship/aws-deployment
encrypted_env_file: aws-deployment.env.encrypted
environment:
- AWS_DEFAULT_REGION=us-east-1
volumes:
- ./:/deploy
You can see that ./ is mapped to the /deploy mount point. This means that if in your repo, you create a directory called tasks, then you place your json file there, and you should be all set.
I wanna deploy the Spring-cloud-data-flow on several hosts.
I will deploy the server of Spring-cloud-data-flow on one host-A, and deploy the agents on the other hosts(These hosts are in charge of executing the tasks).
Except the host-A, all the other hosts run the same tasks.
Shall I modify on the basis of the Spring Cloud Data Flow Local Server or on the Spring Cloud Data Flow Apache Yarn Server or other better choice?
Do you mean how the apps are deployed on several hosts? If so, the apps are deployed using the underlying deployer implementation. For instance, if it is local deployer then, each app is deployed by spawning a new process. You can scale out the number apps deployment using the count property during the stream deploy. I am not sure what do you mean by the agents here.
We have an existing microservice environment with logstash, config and eureka servers. We are now setting up a Spring Cloud Dataflow (Kubernetes) environment (primarily intially to run tasks/batch jobs).
Ideally we would like the tasks to use the existing logstash, config and eureka servers via the standard spring boot configuration (annotations etc) to support the following scenarios:
Logstash: When a task runs its logs are output to logstash and viewable from Kibana
Config Server: To support changing configuration properties for tasks. eg a periodic task's configuration can be tweaked by altering the values on the configuration server and next time the task runs it will use the new values.
My understanding is that config server properties will override properties in the task definition which override properties in the internal application.properties.
Eureka: Each task would register itself in Eureka. The main reason for this is that our tasks have web actuator endpoints exposed and we can then can use Spring Boot Admin (which can discover services via eureka) to access the actuator endpoints and information while a task is running.
(Some of our tasks can take hours to run and this would enable us to monitor them, adjust logging etc)
Is this a sensible approach - or are there any potential issues to look out for here (eg short lived tasks with eureka). I can’t find any discussion of this in the existing spring cloud data flow or spring cloud task documentation.
You may try logstash-logback-encoder for SCDF integration with ELK stack. It works fine for our SCDF on Yarn stream application.
Config Server should work for any Spring Boot application.