i am currently using rabbitmq version 3.7 on my project.
I'm generating my rabbitmq image on a docker-compose.
My question is how do i change from the default message size of 512mb to ~2gb.
rabbit:
container_name: rabbitMQ
hostname: rabbit
image: "rabbitmq:3.7-management"
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "15672:15672"
# - "5672:5672"
volumes:
- ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
From what i gathered i needed to change a property on rabbitmq.conf, but everything i tried didn't have an effect on the allowed size in production.
Is it even possible to change this value?
Related
I'm a little bit confused. I've been following this to get started and the installation method.
https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-kafka-connector-msk/
Setup AWS cli and configure it
Install maven
Compile connector
Set classpath with the jar generated
Set up the properties file of the connector
FYI: I have a docker-compose that creates all my containers (kafka, mqtt, etc.)
(all of the above is setup on-premise)
And then, I executed all this on my machine itself and not on the Kafka container, so for the last step how would that work when I try to run it standalone?
version: '3'
services:
nodered:
container_name: nodered
image: nodered/node-red
ports:
- "1880:1880"
volumes:
- ./nodered:/data
depends_on:
- mosquitto
environment:
- TZ=America/Toronto
- NODE_RED_ENABLE_PROJECTS=true
restart: always
mosquitto:
image: eclipse-mosquitto
container_name: mqtt
restart: always
ports:
- "1883:1883"
volumes:
- "./mosquitto/config:/mosquitto/config"
- "./mosquitto/data:/mosquitto/data"
- "./mosquitto/log:/mosquitto/log"
environment:
- TZ=America/Toronto
user: "${PUID}:${PGID}"
portainer:
ports:
- "9000:9000"
container_name: portainer
restart: always
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "./portainer/portainer_data:/data"
image: portainer/portainer-ce
zookeeper:
image: zookeeper:3.4
container_name: zookeeper
ports:
- "2181:2181"
volumes:
- "zookeeper_data:/data"
kafka:
image: wurstmeister/kafka:1.0.0
container_name: kafka
ports:
- "9092:9092"
- "9093:9093"
volumes:
- "kafka_data:/data"
environment:
- KAFKA_ZOOKEEPER_CONNECT=10.0.0.129:2181
- KAFKA_ADVERTISED_HOST_NAME=10.0.0.129
- JMX_PORT=9093
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_LOG_RETENTION_HOURS=1
- KAFKA_MESSAGE_MAX_BYTES=10000000
- KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
- KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
- KAFKA_NUM_PARTITIONS=2
- KAFKA_DELETE_RETENTION_MS=1000
depends_on:
- zookeeper
restart: on-failure
cmak:
image: hlebalbau/kafka-manager:1.3.3.16
container_name: kafka-manager
restart: always
depends_on:
- kafka
- zookeeper
ports:
- "9080:9080"
environment:
- ZK_HOSTS=10.0.0.129
- APPLICATION_SECRET=letmein
command: -Dconfig.file=/kafka-manager/conf/application.conf -Dapplication.home=/kafkamanager -Dhttp.port=9080
volumes:
zookeeper_data:
driver: local
kafka_data:
driver: local
I guess I have to go into my Kafka container and run the below code, but how can I reference by machine path... I'm stuck here or perhaps I'm missing something:
./bin/connect-standalone.sh {{path_from_machine_where_jar_is}}/kinesis-kafka-connector/config/worker.properties {{path_from_machine_where_jar_is}}/kinesis-kafka-connector/config/kinesis-streams-kafka-
connector.properties
Or I have to run all the previous steps in my Kafka container directly...
I was thinking of just doing this, copy my jar file and just moving it in my kafka container.
docker cp /hostfile (container_id):/(to_the_place_you_want_the_file_to_be)
Thank you!
guess I have to go into my Kafka container and run the below code
No. Containers should only run one process, which is kafka server.
So, either you download Kafka locally and run the Connect scripts from your host.
Or you simply add a new container for Kafka Connect, which will run Connect distributed mode, rather than standalone.
In either case, yes, you need to copy (or mount) the jar into Connect's plugin path
Alternatively, run MSK rather than Kinesis and produce your Kafka data there rather than locally.
Cheers
I am currently trying to reproduce the tutorial to develop an application / solution with Fiware seen in: Desarrolla tu primer aplicación en Fiware and I am having difficulty connecting Grafana with the Crate database (recommended by Fiware).
My docker-compose file configuration is:
version: "3.5"
services:
orion:
image: fiware/orion
hostname: orion
container_name: fiware-orion
networks:
- fiware_network
depends_on:
- mongo-db
ports:
- "1026:1026"
command: -dbhost mongo-db -logLevel DEBUG -noCache
mongo-db:
image: mongo:3.6
hostname: mongo-db
container_name: db-mongo
networks:
- fiware_network
ports:
- "27017:27017"
command: --bind_ip_all --smallfiles
volumes:
- mongo-db:/data
grafana:
image: grafana/grafana:8.2.6
container_name: fiware-grafana
networks:
- fiware_network
ports:
- "3000:3000"
depends_on:
- crate
quantumleap:
image: fiware/quantum-leap
container_name: fiware-quantumleap
networks:
- fiware_network
ports:
- "8668:8668"
depends_on:
- mongo-db
- orion
- crate
environment:
- CRATE_HOST=crate
crate:
image: crate:1.0.5
networks:
- fiware_network
ports:
# Admin UI
- "4200:4200"
# Transport protocol
- "4300:4300"
command: -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
volumes:
- cratedata:/data
volumes:
mongo-db:
cratedata:
networks:
fiware_network:
driver: bridge
After starting the containers, I have a positive response from OCB (Orion), from Quantumleap and even after creating the subscription between Orion and quantumleap, in the Crate database, the data is stored and updated correctly.
Unfortunately I am not able to get the visualization of the data in grafana.
I thought that the reason was the fact that crateDB was removed as a grafana plugin versions ago, but after researching how to connect crateDB with grafana, through a postgres data source (I read on: Visualizing time series data with Grafana and CrateDB), I'm still having difficulty getting the connection, getting in grafana "Query error
dbquery error: EOF"
grafana settings img
The difference with respect to the guide is the listening port, since with input port 5432 I get a response indicating that it is not possible to connect to the server. I'm using port 4300.
After configuring, and trying to query from grafana, I get the mentioned EOF error
EOF error in grafana img
I tried to connect from a database IDE (Dbeaver) and I get exactly the same problem.
EOF error in Dbeaver img
Is there something I am missing?
What should I change in my docker configuration, or ports, or anything else to fix this?
I think it is worth mentioning that I am studying this because I am being asked in a project to visualize context switches in real time with Fiware and grafana.
Thanks in advance
The PostgreSQL port 5432 must be exposed as well by the CrateDB docker image.
Additionally, I highly recommend to use a recent (or just the latest) CrateDB version, current stable is 5.0.0, your used version 1.0.5 is very old and not maintained anymore.
Full example entry:
crate:
image: crate:latest
networks:
- fiware_network
ports:
# Admin UI
- "4200:4200"
# Transport protocol
- "4300:4300"
# PostgreSQL protocol
- "5432:5432"
command: -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
volumes:
- cratedata:/data
General Information:
Os : Ubuntu 20.04 LTS
docker : v. 20.10.6
docker-compose: v. 1.29.2
Soundcard : Steinberg UR22mkII
Drivers : Alsa
We are developing a web service to record audio signals and display various properties and analysis for anomaly detection. For some analyses larger window sizes are necessary but also some realtime plots are included. The realtime plots are done via java-script(p5-Module) while everything else is processed via flask and python and visualized via grafana.
We have now encountered the problem that these two different clients cannot access the same audio device at the same time. In the main system this behavior can be solved with the dsnoop plugin from asoundrc (https://www.alsa-project.org/wiki/Asoundrc). But until now we were not able to implement this functionality in the docker-environment.
We have already tried to tunnel the virtual audio devices via the docker-compose file, but without success. (The compose file is enclosed). The Alsa drivers inside the container are also installed correctly. We suspect that it has something to do with the setup of the internal docker-environment, but are stuck at this point.
We are grateful for any tips or hints!
Docker-Compose File:
version: "3.8"
services:
webapp:
build: .
restart: always
depends_on:
- influxdb
- grafana
ports:
- 5000:5000
volumes:
- ./:/aad
devices:
- /dev/snd:/dev/snd
environment:
# output gets written to the docker-compse console without buffering it
- PYTHONUNBUFFERED=1
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
influxdb:
image: influxdb
restart: always
ports:
- 8086:8086
volumes:
# mount volumes to store data and configuration files. Dirs are createt if necessary
- ./influxdb/data:/var/lib/influxdb2
- ./influxdb/config:/etc/influxdb2
# mount script to be executed (only) after initial setup is done
- ./influxdb/scripts:/docker-entrypoint-initdb.d
environment:
# setup of database is only executet if no boltdb file is found in the specified path so the container with influx can be rebooted same as once setup
DOCKER_INFLUXDB_INIT_USERNAME: aad
DOCKER_INFLUXDB_INIT_PASSWORD: .......
DOCKER_INFLUXDB_INIT_ORG: aaddev
DOCKER_INFLUXDB_INIT_BUCKET: training
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: .......
DOCKER_INFLUXDB_INIT_MODE: setup
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
grafana:
image: grafana/grafana
# port mapping only need for easieser debugging
ports:
- 3000:3000
restart: always
depends_on:
- influxdb
volumes:
- grafana-storage:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
GF_SECURITY_ADMIN_USER: aad
GF_SECURITY_ADMIN_PASSWORD: .......
GF_PATHS_CONFIG: /etc/grafana/grafana.ini
GF_USERS_DEFAULT_THEME: light
GF_AUTH_ANONYMOUS_ENABLED: "True"
GF_SECURITY_ALLOW_EMBEDDING: "True"
GF_AUTH_ANONYMOUS_ORG_NAME: Main Org.
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
GF_DASHBOARDS_MIN_REFRESH_INTERVAL: 1s
GF_AUTH_BASIC_ENABLED: "True"
GF_DISABLE_LOGIN_FORM: "True"
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
volumes:
grafana-storage:
Python Environment:
name: aad
channels:
- anaconda
- conda-forge
- defaults
dependencies:
- portaudio=19.6.0=h7b6447c_4
- flask=1.1.2=pyhd3eb1b0_0
- librosa=0.8.0=pyh9f0ad1d_0
- matplotlib=3.3.4=py38h06a4308_0
- numpy=1.20.1=py38h93e21f0_0
- pandas=1.2.4=py38h2531618_0
- pip=21.0.1=py38h06a4308_0
- pyaudio=0.2.11=py38h7b6447c_2
- python=3.8.8=hdb3f193_5
- scikit-learn=0.24.1=py38ha9443f7_0
- scipy=1.6.2=py38had2a1c9_1
- tqdm=4.59.0=pyhd3eb1b0_1
- werkzeug=1.0.1=pyhd3eb1b0_0
- pip:
- influxdb-client==1.16.0
- rx==3.2.0
I'm trying to run an elastic stack on my RasPi4 using docker compose. Problem is that Elastic does not provide images for ARM archtiecture ... only X86. So raspi is not supported out of the box.
Everytime I start my docker compose config I get this message
7.9.3: Pulling from elasticsearch/elasticsearch
ERROR: no matching manifest for linux/arm/v7 in the manifest list entries
Google search mostly gives results pointing to an unofficial image ... which I would try ... but this one is 4 years old: https://hub.docker.com/r/ind3x/rpi-elasticsearch/. So I guess I don#t get an up to date elasticsearch.
Anyone got an idea on how I get my elastic to run? This is my docker-compose.yml ... pretty straigt forward.
version: '3.3'
services:
elastic-node-1:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
container_name: elastic-node-1
restart: always
environment:
- node.name=elastic-node-1
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elastic-node-2
- cluster.initial_master_nodes=elastic-node-1,elastic-node-2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elastic-data-1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic-net
elastic-node-2:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
container_name: elastic-node-2
restart: always
environment:
- node.name=elastic-node-2
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elastic-node-1
- cluster.initial_master_nodes=elastic-node-1,elastic-node-2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elastic-data-2:/usr/share/elasticsearch/data
ports:
- 9201:9201
networks:
- elastic-net
kibana:
image: docker.elastic.co/kibana/kibana:7.9.3
container_name: kibana
restart: always
depends_on:
- elastic-node-1
- elastic-node-2
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elastic-node-1:9200
ELASTICSEARCH_HOSTS: http://elastic-node-1:9200
networks:
- elastic-net
volumes:
elastic-data-1:
driver: local
elastic-data-2:
driver: local
networks:
elastic-net:
driver: bridge
If there is no way to get this elastic setup to run, can you recommend any other hardware similar to raspi (using linux) which is x86 and can take the place of my raspi? Then I would switch hardware for my elastic stack.
I have made some experience with Elastic in larger business applications so just some additional food for thought - I do not have a direct answer here yet:
indeed an image that ks 4 years old is not worth the effort. Elsstic is stable in version 7.x and 8.x is in progress and there have been massive changes.
you need to consider that Heapsize available to Elastic actually should be configured to 50% as it is shared with Lucene.
meaning Elastic can be quite RAM hungry. Depending on your use case and given the limits of Raspi to max 8GB at this point in time you may want to consider that.
For a small application it may work, but I would not consider it more than experimental.
If you do not have any other way you may have two options:
build a docker image (or find someone who is interested enough to join the effort maybe the orinibal author of that old docker image)
go step by step and first deploy elastic on a headless raspi standalone (even avoid docker for the moment and reduce any overhead) and then add some elastic node configs (elastic usually only works well with at least three nodes)
indeed build a cluster which offers at least 8 -16 GB per node - I believe an Ubuntu based setup will do with an X86.
So I have the following docker-compose.yml
version: "3.7"
services:
roundclinic-mysql:
image: mysql:5.7
networks:
- spring-boot-mysql-network
environment:
- MYSQL_DATABASE=
- MYSQL_USER=
- MYSQL_PASSWORD=
- MYSQL_ROOT_PASSWORD=
volumes:
- ./mysqldata:/var/lib/mysql:rw,delegated
ports:
- "3306:3306"
web-service:
image: roundclinic/roundclinic:latest
networks:
- spring-boot-mysql-network
- traefik-network
depends_on:
- roundclinic-mysql
ports:
- 8080:8080
environment:
- "SPRING_PROFILES_ACTIVE=dev"
links:
- roundclinic-mysql
labels:
- "--providers.docker.network=traefik_default"
- "traefik.enable=true"
- "traefik.http.routers.roundclinic.rule=Host(`api-dev.roundclinic.app`)"
- "traefik.http.routers.roundclinic.entrypoints=web"
- "traefik.http.services.cal.loadbalancer.server.port=8080"
traefik:
image: "traefik:v2.2"
container_name: "traefik"
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "traefik.docker.network=traefik-network"
ports:
- "80:80"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
traefik-network:
driver: bridge
external: true
spring-boot-mysql-network:
driver: bridge
volumes:
my-db:
Spring boot starts up fine and can connect to mysql.
When I connect to http://api-dev.roundclinic.app:8080/../ I can hit my application just fine
When I connect to http://api-dev.roundclinic.app/../ I get a gateway timeout. I can see in the traefik logs that it's forwarding the request to what seems to be the correct IP and port, but nothing hits the actual application. I'm not sure what's going on here. Any help?
When accessing port 8080 you are bypassing Traefik and directly access your application, correct?
Generally speaking the Traefik labels look good. Entrypoint, Port and Host are defined, router and service port are present. These are usually all the requirements for Docker-based setups.
One thing that I noticed is that the traefik container uses "traefik.docker.network=traefik-network", but your web app uses:
"--providers.docker.network=traefik_default".
I am not sure if traefik_default is something that traefik provides but that mismatch in network names might be the issue.
I can't test if that is the problem but that would be the first thing to check.
One way would be to simplify your config but just always using the networks key from docker compose instead of mixing it with labels and arguments.