I'm using kafka-connect-elasticsearch with a custom converter, which extents standard org.apache.kafka.connect.json.JsonConverter.
In my custom converter I need to access an environment variable.
Let's assume, I need to append to every message the name of the cluster, which is written to environment variable CLUSTER.
How can I access my environment variable in the converter?
Maybe read it at converter configuration phase (configure(Map<String, ?> configs) methond)?
How can I forward CLUSTER env variable value to this configs map?
You can't get it in that map.
You would need to use System.getenv("CLUSTER")
Related
I don't have any experience with Kafka yet and need to automate a task. Is there a way that I can use env variables in the configuration file?
To be more specific:
advertised.listeners=INSIDE://:9092,OUTSIDE://<hostname>:29092
I'd like to extract and use the hostname from my env variables.
Property files offer no variable interpolation
If you started Kafka via Docker processes, or write your own shell scripts which generate a property file prior to starting the broker, then you could inject values
Some examples include confd, consul-template, dockerize
I would like for a job J from a release R in a bosh deployment to start with a certain environmental variable E set, which is not available in the job's properties for configuration
Can this be specified in the deployment file or when calling the bosh cli?
Unfortunately, I am pretty sure this is not possible. BOSH does not understand environment variables. Instead, it executes an ERB template with the properties configured in the manifest. For example in this job template from log-cache is executed with the properties from a manifest along with defaults from the job spec.
If you need to have a particular environment variable set for testing/development, you can bosh ssh on to an instance where you are going to run the job and then mutate the generated file. Given the CF deployment example, bosh ssh doppler/0 and then modify the generated bpm.yml in /var/vcap/jobs/log-cache/config/bpm.yml. This is a workaround for debugging and development, if you need to set a field in a manifest reach out to the release author and open an issue or PR the ability to set environment variable as a property by adding it to the job spec.
(note the versions used in the example are just from HEAD and may not actually work)
I'm setting up a Kafka cluster, in which I'm setting the broker.id=-1 so that broker.ids are automatically generated. but in some cases want to set them using environment variables (i.e. KAFKA_BROKER_ID).
If done so, will the nodes with the KAFKA_BROKER_ID env variables use the env variable or auto-generate them?
Depends on how you are deploying your Kafka installation.
Out of the box, Kafka does not use system properties to configure broker id, so you need to put the value into .properties file.
(among others: grepping for KAFKA_BROKER_ID in Kafka source returns nothing)
KAFKA_BROKER_ID appears to be added by multiple Docker images, you'd need to contact the author of the one you are using.
I have multiple kind of environment on which my application runs like Test environment or prod environment.I have different configuration files for each environment named application-prod.properties and application-test.properties and all other common but necessary properties are saved in application.properties file.Here application.properties file is overridden by respective environment properties file thus getting the desired environment variables at runtime.
I want to store these environment specific variables in Postgresql database so that at runtime I could retrieve those variables based on the environment the application is running.
Can anyone help me to achieve this goal ? How should I design this database for storing environment variables ?
So I have configured an OpenShift 3.9 build configuration such that environment variables are populated from an OpenShift secret at build-time. I am using these environment variables for setting passwords up for PostgreSQL roles in the image's ENTRYPOINT script.
Apparently these environment variables are baked into the image, not just the build image, but also the resulting database image. (I can see their values when issuing set inside the running container.) On one hand this seems necessary because the ENTRYPOINT script needs access to them, and it executes only at image run-time (not build-time). On the other this is somewhat disconcerting, because FWIK one who obtained the image could now extract those passwords. Unsetting the environment variables after use would not change that.
So is there a better way (or even best practice) for handling such situations in a more secure way?
UPDATE At this stage I see two possible ways forward (better choice first):
Configure DeploymentConfig such that it mounts the secret as a volume (not: have BuildConfig populate environment variables from it).
Store PostgreSQL password hashes (not: verbatim passwords) in secret.
As was suggested in a comment, what made sense was to shift the provision of environment variables from the secret from BuildConfig to DeploymentConfig. For reference:
oc explain bc.spec.strategy.dockerStrategy.env.valueFrom.secretKeyRef
oc explain dc.spec.template.spec.containers.env.valueFrom.secretKeyRef