i have a production cluster using ambari from hortonworks. Now cloudera has blocked every access to hdp repository, because a paid support license is needed.
This hit us really hard because we have big infrastructure using ambari, kafka, ans storm.
I'm trying to build ambari from source but i think that a local hdp repository is needed.
Anyone know how to build a repo strarting from kafka and storm source?
Related
How Kafka will release the patch updates?
How users will get to know Kafka patch updates?
Kafka is typically available as a zip/tar that contains the binary files which we will use to start/stop/manage Kafka. You may want to:
Subscribe to https://kafka.apache.org/downloads by generating a feed for it.
Subscribe to any feeds that give you updates
Write a script that checks for new kafka releases https://downloads.apache.org/kafka/ periodically to notify or download.
The Kafka versioning format typically is major.minor.patch release.
Every time, there is a new Kafka release, we need to download the latest zip, use the old configuration files (make changes if required) and start Kafka using new binaries. The upgrade process is fully documented in the Upgrading section at https://kafka.apache.org/documentation
For production environments, we have several options:
1. Using Managed Kafka Service (like in AWS, Azure, Confluent etc)
In this case, we need not worry about patching and security updates to Kafka because it is taken care by the service provider itself. For AWS, you will typically get notifications in the Console regarding when your Kafka update is scheduled.
It is easy to get started to use Managed Kafka service for production environments.
2. Using self-hosted kafka in Kubernetes (eg, using Strimzi)
If you are running Kafka in Kubernetes environment, you can use Strimzi operator and helm upgrade to update to the version you require. You need to update helm chart info from repository using helm repo update.
Managed services and Kubernetes operators make managing easy, however, manually managing Kafka clusters is relatively difficult.
We are working with Kafka, and have three environments set up: A test environment for the test system, a QA environment and a production environment. Pretty standard. Now we are getting started with using Avro, and have set up a schema registry. The same installation setup: Test, QA and production. But now we get a bit uncertain about how to use the environments together. We have looked around a bit but haven't really found any examples of how to setup a Kafka test/qa/prod environment with a schema registry. These are the three approaches we have discussed internally:
Should we use Schema Registry Prod for all environments? Just as we do with our other artifact repositories.
With Nexus, Artifactory, Harbor etc. we use one instance for handling both developer version and release versions of artifacts. So our initial approach was to do the same with the Schema Registry. But here there is a difference though: with our other artifact repositories we have SNAPSHOT support and different spaces (builds/releases etc.) which we have not seen people using with a Schema Registry. So even though this approach was our initial, and it should work since we plan on using compatibility FULL_TRANSITIVE, we are now getting doubtful about sending each development/test version of a contract all the way to production. For example, FULL_TRANSITIVE would make it impossible to make non-compatible changes even during development.
Use the Test schema registry for test environment, and prod schema registry for Prod environment.
Straight forward, use "Kafka Schema Registry Test" in our test environment, with our Kafka test installation. And "Kafka Schema Registry Prod" only in our production environment. Then our build pipeline would deploy the Avro schemas to the production schema registry at an appropriate stage.
Use snapshots schemas
This would be to try and mimic the setup with our other repositories. That is, we use one schema registry in all our environments (the "Schema Registry Prod"). But during development and test we use a "snapshot" version of the schema (perhaps prepend "-snapshot" to the subject name). In the build pipeline we change to a 'non-snapshot' subject when ready to release.
So we would like to hear how other people work with Avro and Schema Registry. What does your setup look like?
Adding snapshot to the subject name would cause the default TopicNameStrategy serializer function to fail
In a primarily Java dev environment, we do have 3 Registries, but they are only at backwards transitivity, not full
Where possible, we use the Maven Plugins provided by Confluent to test the schema compatibility before registering. This will cause the CI pipeline to fail if the schema is not compatible, and we can use Maven profiles to override the registry url per environment, but the assumption is that if it fails in a lower environment, it'll never reach higher. Test environment can have SNAPSHOT artifacts manually be deployed to, but this is only referenced by code artifacts and not Registry subjects, so schemas need to be manually deleted if there are any mis-registrations. That being said, "staging/qa" and prod are generally the exact same, so barring network connectivity around "production", you'd only need two Registries
For non Java projects, we force them to use a standalone custom Maven archetype repo with the same plugins, so that allows them to version the schema artifacts in Maven, but also allows for JVM consumers to still use their data. It also simplifies the build pipeline support to a standard lifecycle
For a project I am working on, we have CI setup using Jenkins.
We now want to setup Continuous Delivery (CD) using Puppet.
Here is our dev environment specs
Windows 2008 Server
Jboss-SOA-P (jboss AS 5.1 app server) - 2 instances
Jenkins for CI
Installed Puppet Learning VM (as we are evaluating, so we don't have a license to install Puppet Enterprise).
My question is: How can I automate deployment of my application(s) on already installed Jboss servers (on Windows m/c) using Puppet?
For my organization, I am using the following tools to achieve Continuous Delivery and Continuous Integration
Foreman: A provisioning and ECN ( External Node Classifier )
Puppet Master :
This will be running in our main server
Puppet Agents:
On rest of the servers
Jenkins:
On main server
Nexus repository for maintaining staging and release repositories on another server
A nexus repository module installed in puppet master. It has the logic to connect to nexus repository, fetch the latest release from "release" repository.
Flow:
I have a jenkins job defined whose purpose is making sure the build doesn't break and at the time of release, I perform maven release
that in turn upload the latest version to nexus release repository.
Load nexus repository module in foreman and map "Nexus" class to all my servers. This is the crucial part that allows me to perform
cloud deployment with a single button click in foreman. To do this, you need to have your own puppet files that perform dbmigration, undeploy and deployment. I always write a deploy maven module for all my projects which will have only these puppet files that allows me to perform deployment in all servers in one shot.
Since you are already familiar with CI and CD, I hope my statements are self explanatory
Without knowing how you want your application deployed, it's hard to answer exactly how to do it. But I'll see if I can point you in the right direction:
There are existing modules on the forge to deploy JBoss with Puppet. I'd recommend looking through and seeing if you can find one that suits your requirements.
You could then integrate your source control to automatically deploy changes to your JBoss instance as they get checked in.
There's an example of using Puppet with Tomcat and Maven for continuous delivery here. It's a few years old, but the concepts still apply: http://www.slideshare.net/carlossg/2013-02-continuous-delivery-apache-con
Here's also an example from CloudBees for contionous delivery with Puppet and Jenkins (which has a lot of the engineers behind Jenkins) https://www.cloudbees.com/event/continuous-delivery-jenkins-and-puppet-debug-bad-bits-production
Plus a general PuppetLabs one here: https://puppetlabs.com/blog/whats-continuous-delivery-get-speed-these-great-puppetconf-decks
Also: I don't want to sound too salesly, and full disclosure, I work at PuppetLabs. But you can use Puppet Enterprise with up to 10 nodes for free, so I'd recommend that over using the Learning VM, which isn't really designed for hosting applications.
I've multiple consuming app instances connecting to spring cloud config server. Config server gets config from SVN Repo.
Just wanted to understand how config server (instance) is managing possibly concurrent requests.
Thanx,
That's a bug in the SVN support (the git version has a synchronized method). https://github.com/spring-cloud/spring-cloud-config/issues/128
We have two distributed dev teams. Each with it's own Archiva instance. Each team deploys artifacts into their own repository. I'd like to configure each of these Archiva instances so that they each can discover artifacts from the other.
I set up Proxy Connectors on each Archiva.
Archiva A -> Archiva B
Archiva B -> Archiva A
But when someone requests an artifact one of the two Archiva's hangs, requiring the service to be rebooted.
Is this Proxy Connector configuration creating an infinite loop? Perhaps I'm confused by how they work. If they shouldn't be configured in this manner, how would I solve my original problem of two Archiva's sharing artifacts?
are you using some black/white lists to limit the proxying to only your company groupId ?
Hangs but nothing in the logs ?