I'm trying to figure out how to pass custom port and cache size and vcl file in docker-compose.yml to varnish (I'm using the official varnish image https://hub.docker.com/_/varnish )
I Tried using environment variables by adding the following lines into my docker-compose.yml but it seems that they are not taken into consideration
environment:
- VARNISH_CONFIG="/etc/varnish/custom.vcl"
- CACHE_SIZE=512m
- VARNISH_PORT=8080
Any clue on how to pass these params to a varnish container ?
In case some one needs to achieve the same config, I ended up by creating a new image based on the official one and adding these params to the CMD instruction :
EXPOSE **8080**
CMD ["varnishd", "-F", "-f", "**/etc/varnish/custom.vcl**", "-s", "**malloc,2G**"]
The official Varnish Docker image is quite basic at this point. As you concluded in your second post, you can use our the image as the basis, and then customize the behavior in your own Dockerfile.
However, for Varnish Software, it's an iterative process, so we'll gradually add features to the image. In the end, you'll be able to configure most parameters through environment variables in your docker-compose.yml file.
We're working on it, thanks for your patience.
Related
I am attempting to learn the various features of something called Open Policy Agent because I think it may be a useful tool in a microservices based application.
Here is a link to the 'Running with Docker' section of the documentation for this application: https://www.openpolicyagent.org/docs/latest/deployments/#running-with-docker
Currently, I am running Docker using the Docker Desktop in a Windows 10 environment and I already have a docker-compose file set up for my main application which includes various docker images. My thoughts were that I could simply add the latest openpolicyagent image as well as the openpolicyagent demo-restful api so that I could begin learning about the service. To do this, I added the following lines to my docker-compose.yml:
opa:
image: openpolicyagent/opa:0.34.2
ports:
- 8181:8181
command:
- "run"
- "--server"
- "--log-level=debug"
- "api_authz.rego"
volumes:
- C:\Sites\prosaurus\policy\api_authz.rego:/api_authz.rego
api_server:
image: openpolicyagent/demo-restful-api:latest
ports:
- 5000:5000
environment:
- OPA_ADDR=http://opa:8181
- POLICY_PATH=/v1/data/httpapi/authz
This appears to have worked in that I can go to localhost:8181 and i see the Query and Input Data (JSON) boxes as I presume is supposed to happen, however I would like to test some of the command line functions as are mentioned here:
https://www.openpolicyagent.org/docs/latest/#2-try-opa-eval
However I can not seem to access the command line of the docker container which is running the OPA agent. The way I have attempted this is via the Docker Desktop application GUI in Windows. In this application I can see all of the docker instances which are running and each one has an option to run the CLI (you click the button and the cli opens). They all work except for the OPA one. When I click on that one a cmd window opens for a split second, displays something too fast for me to read it and then closes.
What have I done wrong?
OPA can be run in a few different ways, and opa eval is distinctly different from running OPA as a server, i.e. opa run --server.
When you run OPA as a server - which is how you'd normally run OPA in production - you query OPA for policy decisions through OPA's REST API.
opa eval on the other hand is more like a Swiss army knife of OPA, allowing you to quickly evaluate a rule or expression given some provided policy and data.
You can think of them as two entirely different tools.
We let the golem package automatically create a Dockerfile for us and can run the docker image and see the app at the root directory: http://localhost:3838/?...
But we would like the app to appear in a subdirectory like http://localhost:3838/myApp/v1/?... so that we can set up the necessary proxying for Apache and have this and other apps all available from a single server.
We can manually edit the Dockerfile to copy a shiny-server.conf file with the following information:
# Define a server that listens on port 3838
server {
listen 3838;
# Define a location at the base URL
location /myApp/v1/ {
# Host the directory of Shiny Apps stored in this directory
site_dir /srv/shiny-server;
# Log all Shiny output to files in this directory
log_dir /var/log/shiny-server;
}
}
The above solution feels like a hack and we are hoping there is functionality inside of golem that will allow us to set the subdirectory at which the app will appear.
Unfortunately there is no way to include an nginx configuration inside the Dockerfile programmatically: {golem} tries to help with the creation of the file, but some things still need to be done manually.
Also, note that {golem} doesn't create a Dockerfile with a shiny server in it, it creates a standalone docker image that launches the app, so there is no shiny server running, just an R process. {shiny} being what it is, there is no way to natively run it on a given path, it's always at the root, on a port.
That being said, what you can do is either edit the dockerfile so that it also bundle nginx (or any other load balancer), so that you can serve the app on a path, or serve your application on another port, using the port argument of add_dockerfile(): that might be easier to configure it with you Apache proxy.
Colin
We're developing several Lagom-based Scala micro-services. They are configured using variable replacement in application.conf, eg.
mysql = {
url = "jdbc:mysql://"${?ENV_MYSQL_DATABASE_URL}
During development, we set these variables as Java System Properties via a env.sbt file that calls System.setProperty("ENV_MYSQL_DATABASE_URL", url). This is working fine.
Now I want to deploy this in a container to my local Docker installation. We are using the SbtReactiveAppPlugin to build the Docker image from build.sbt and simply run sbt Docker/publishLocal. This works as expected, a Docker image is created and I can fire it up.
However, passing in environment variables using the standard docker or docker-compose mechanisms does not seem to work. While I can see that the environment variables are set correctly inside the Docker container (verified using env on a bash and also by doing log.debug("ENV_MYSQL_DATABASE_URL via env: " + sys.env("ENV_MYSQL_DATABASE_URL")) inside the service), they are not used by the application.conf and not available in the configuration system. The values are empty/unset (verified through configuration.getString("ENV_MYSQL_DATABASE_URL").toString() and the exceptions thrown by the mysql system and other systems).
The only way I've gotten it to work was by fudging this into the JAVA_OPTS via JAVA_OPTS=-D ENV_MYSQL_DATABASE_URL=..... However, this seems like a hack, and doesn't appear to scale very well with dozens of environment parameters.
Am I missing something, is there a way to easily use the environment variables inside the Lagom application and application.conf?
Thanks!
I've used Lightbend config to configure Lagom services via environment variables in docker containers for many years, so know that it can be done and has been pretty straightforward in my experience.
With that in mind, when you say that they're not used by application.conf, do you mean that they're unset? Note that unless you're passing a very specific option as a Java property, configuration.getString("ENV_MYSQL_DATABASE_URL") will not read from an environment variable, so checking that will not tell you anything about whether mysql.url is affected by the environment variable. configuration.getString("mysql.url") will give you a better idea of what's going on.
I suspect that in fact your Docker image is being built with the dev-mode properties hardcoded in, and since Java system properties take precedence over everything else, they're shadowing the environment variable.
You may find it useful to structure your application.conf along these lines:
mysql_database_url = "..." # Some reasonable default default for dev-mode
mysql_database_url = ${?ENV_MYSQL_DATABASE_URL}
mysql {
url = "jdbc://"${mysql_database_url}
}
In this case, you have a reasonable default for a developer (probably including in the docs some instructions for running MySQL in a way compatible with that configuration). The default can then be overridden via setting a Java property (e.g. JAVA_OPTS=-Dmysql_database_url) or by setting the ENV_MYSQL_DATABASE_URL environment variable.
While I agree with the answer provided by Levi Ramsey, I would suggest you to use typesafe's config to load the your config
Hi I setup my zookeeper cluster and it seems to be running fine. But I'm trying to setup the heap size and it doesn't seem to be respected. I created the java.env with export JVMFLAGS="-Xms3000m -Xmx3000m" file inside conf/...
When I ps -aux | grep java I can see -Xmx1000m -Xms3000m -Xmx3000m. But when I check with free -m I only see 200M used and 3.3G free.
I noticed that the default value is set regardless. Does this affects it?
Shouldn't Xms fill up the used RAM?
The file zkEnv.sh contains the following line:
export SERVER_JVMFLAGS="-Xmx${ZK_SERVER_HEAP}m $SERVER_JVMFLAGS"
and the "-Xmx${ZK_SERVER_HEAP}m" caused the trouble.
This is what finally worked for me.
In the conf/java.env
export SERVER_JVMFLAGS="-Xms6144m -Xmx6144m -XX:+AlwaysPreTouch"
If you are using /usr/bin/zookeeper-server-start from the confluent-kafka-* package, you might need to set KAFKA_HEAP_OPTS inside your systemd unit file.
Environment="KAFKA_HEAP_OPTS=-Xmx1024M -Xms1024M"
I am trying to set up Stackdriver debugging using Go. Using the article and this great medium post I came up with this solution.
Key parts, in cloudbuild.yaml
- name: gcr.io/cloud-builders/wget
args: [
"-O",
"go-cloud-debug",
"https://storage.googleapis.com/cloud-debugger/compute-go/go-cloud-debug"
]
...
Dockerfile I have
...
COPY gopath/bin/stackdriver-demo /stackdriver-demo
ADD go-cloud-debug /
ADD source-context.json /
CMD ["/go-cloud-debug","-sourcecontext=./source-context.json", "-appmodule=go-errrep","-appversion=1.0","--","/stackdriver-demo"]
...
However the pods keeps crashing, the container logs show this error:
Error loading program: decoding dwarf section info at offset 0x0: too short
EDIT: Using https://storage.googleapis.com/cloud-debugger/compute-go/go-cloud-debug may be outdated as I haven't seen it used outside Daz's medium post. The official docs uses the package cloud.google.com/go/cmd/go-cloud-debug-agent
I have update cloudbuild.yaml file to install this package:
- name: 'gcr.io/cloud-builders/go'
args: ["get", "-u", "cloud.google.com/go/cmd/go-cloud-debug-agent"]
env: ['PROJECT_ROOT=github.com/roberson34/stackdriver-demo', 'CGO_ENABLED=0', 'GOOS=linux']
- name: 'gcr.io/cloud-builders/go'
args: ["install", "cloud.google.com/go/cmd/go-cloud-debug-agent"]
env: ['PROJECT_ROOT=github.com/roberson34/stackdriver-demo', 'CGO_ENABLED=0', 'GOOS=linux']
And in the Dockerfile I can get access to the binary in gopath/bin/go-cloud-debug-agent
When I execute the gopath/bin/go-cloud-debug-agent with my own program as an argument:
/go-cloud-debug-agent -sourcecontext=./source-context.json -appmodule=go-errrep -appversion=1.0 -- /stackdriver-demo
I get another opaque error:
Error loading program: AttrStmtList not present or not int64 for unit 88
So basically using the cloud-debug binary from https://storage.googleapis.com/cloud-debugger/compute-go/go-cloud-debug and cloud-debug-agent binary from the package cloud.google.com/go/cmd/go-cloud-debug-agent both don't work and give different errors.
Would appreciate any tips on what I'm doing wrong and how to fix it.
OK :-)
Yes, you should follow the current Stackdriver documentation, e.g. go-cloud-debug-agent
Unfortunately, there are now various issues with my post including a (currently broken) gcr.io/cloud-builders/kubectl for regions.
I think your issue pertains to your use of golang:alpine. Alpine uses musl rather than the glibc that you find on most other Linux distro's and so, you really must compile for Alpine to ensure your binaries reference the correct libc.
I'm able to get your solution working primarily by switching your Dockerfile to pull the Cloud Debug Agent while on Alpine and to compile your source on Alpine:
FROM golang:alpine
RUN apk add git
RUN go get -u cloud.google.com/go/cmd/go-cloud-debug-agent
ADD main.go src
RUN CGO_ENABLED=0 go build -gcflags=all='-N -l' src/main.go
ADD source-context.json /
CMD ["bin/go-cloud-debug-agent","-sourcecontext=/source-context.json", "-appmodule=stackdriver-demo","-appversion=1.0","--","main"]
I think that should get you beyond the errors that you documented and you should be able to deploy your container to Kubernetes.
I've made my version of your image publicly available (and will retain it for a few days for you):
gcr.io/dazwilkin-190402-55473323/roberson34#sha256:17cb45f1320e2fe04e0681310506f4c229896429192b0d1c2c8dc20ed54adb0d
You may wish to reference it (by that digest) in your deployment.yaml
NB For Error Reporting to be "interesting", your code needs to generate errors and, with your example, this is going to be challenging (usually a good thing). You may consider adding another errorful handler that always results in errors so that you may test the service.