I'm working on python web site which uses Celery for running long running background procesess. I plan to deploy my web site with celery worker to Beanstalk container.
Please share you expirence or ideas how to deploy and run celeary workers as demons at Beanstalk container.
I built a custom AMI using AWS linux ami as a base, with celery set to run at start up in daemon mode. I customised the AMI so eveything was running under Python 2.7 rather than the default 2.6. This meant rebuilding mod_wsgi and using virtual environments.
Related
I am an engineering newbie who is trying to learn something while experimenting with Raspberry Pi. I wanted to try out Docker, so I followed along with a tutorial, which installed Docker, docker-compose and via that, Portainer.
Now I am a bit lost.
What is the purpose of docker-compose?
What is the purpose of Portainer?
And if I want to add some container like NextCloud/openVPN how do I do that? Thru Portainer? Or docker-compose?
Thanks!
Portainer is just a visualized tool for Docker suite. You can run docker commands everywhere a portainer-agent runs as long as you have access to it and it is not an official Docker product.
Docker-compose from the other is a set of docker commands as part of docker engine (community and enterprise) that will help you to orchestrate containers in a single node (PC or VM). If you want to orchestrate more than a single node, you should read about Docker-Swarm or Kubernetes.
A very nice article to understand swarm vs compose differences is here.
Portainer is just a tool on top of Docker that gives you a UI for free because Native Docker Universal Control Panel is on enterprise edition only.
With the use of Jenkins Docker Plugin we can provision the slaves dynamically.
My need is to run UI tests on the automatically created slaves. Is that feasible? If yes, how can we achieve that?
UI tests are WindowTester test cases for eclipse based tool.
I am doing same kind of stuff, On successful build we are running all automated test cases on windows machine.
In your Jenkins, you need to add Windows machine as a Slave machine.
Try below tutorial -
https://wiki.jenkins.io/display/JENKINS/Step+by+step+guide+to+set+up+master+and+slave+machines+on+Windows
Once node is up and running then in your Job make sure you selected windows slave node.
I`m looking for a hosting package for scala and play in Amazon elastic servers. If there is anybody here is done it before? what are the ideal solution for a small ERP application?
you can easily deploy playframework web application on elastic beanstalk using docker container.
Create environment on EBS using docker container.
package your app using sbt docker:stage
make a zip of target/docker/stage/*
deploy this zip on EBS environment.
you can explore docker # https://www.docker.com/
I'm playing with Kafka-Connect. I've got the HDFS connector working both in stand-alone mode and distributed mode.
They advertise that the workers (which are responsible for running the connectors) can be managed via YARN However, I haven't seen any documentation that describes how to achieve this goal.
How do I go about getting YARN to execute workers? If there is no specific approach, are there generic how-to's as to how to get an application to run within YARN?
I've used YARN with SPARK using spark-submit however, I cannot figure out how to get the connector to run in YARN.
You can theoretically run anything on YARN, even a simple hello world program. Which is why saying Kafka-Connect runs on YARN is technically correct. The caveat is that getting Kafka-Connect to run on YARN will take a fair amount of elbow grease at the moment. There are two ways to do it:
Directly talk to the YARN API to acquire a container, deploy the Kafka-Connect binaries and launch Kafka-Connect.
Use the separate Slider project https://slider.incubator.apache.org/docs/getting_started.html that Stephane has already mentioned in the comments.
Slider
You'll have to read quite a bit of documentation to get it working but the idea behind Slider is that you can get any program to run on YARN without dealing with the YARN API and writing a YARN app master by doing the following:
Create a slider package out of your program
Define a configuration for you package
Use the slider cli to deploy your application onto YARN
Slider handles container deployment and recovery of failed containers for you, which is nice. Also Slider is becoming a native part of YARN when YARN 3.0 is released.
Alternatives
Also as a side note, getting Kafka-Connect to deploy on Kubernetes or Mesos / Marathon is probably going to be easier. The basic workflow to do that would be:
Create a Kafka-Connect docker container or just use confluent's docker container
Create a deployment config for Kubernetes or Marathon
Click a button / run a command
Tutorials
A good Mesos / Marathon tutorial can be found here
Kubernetes tutorial here
Confluent Kubernetes Helm Charts here
I packaged my Scala/LiftWeb app with the sbt one-jar plugin into a single executable jar file and packed it up with Docker, exposing the embedded Jetty's port in the Dockerfile.
It runs fine locally on Docker and appearently deploys clean on AWS EB using the CLI deployment tools. On the received EB URL however, all I see is the congrats page saying "Your Docker Container is now running in Elastic Beanstalk on your own dedicated environment in the AWS Cloud.".
So, where is my app? Do I miss any steps making my app publicly available on my EB instance?
For future reference, the problem was caused by using an obsolete 2.x version of the aws-eb-cli tools package. Upgrading it to 3.x made the error obvious - building the docker image has failed on AWS.
What I was looking for was running an existing docker image, I found instruction for this scenario at https://aws.amazon.com/blogs/aws/aws-elastic-beanstalk-for-docker/.
Thanks a lot for Nick for asking the right questions which made me realize the obsolete tools package!