PDNS_* options at docker-compose.yml has no effect. Why? - docker-compose

I have docker-compose.yml file to build PowerDNS auth server:
version: '3.9'
services:
auth:
build:
context: pdns/
dockerfile: Dockerfile-auth
environment:
- PDNS_AUTH_API_KEY
- PDNS_LOGLEVEL=7
- PDNS_PRIMARY=yes
- PDNS_LAUNCH=gpgsql
- PDNS_GPGSQL_HOST=pgdb
- PDNS_GPGSQL_USER=pdns
- PDNS_GPGSQL_PASSWORD=***
- PDNS_GPGSQL_DBNAME=pdns
ports:
- "1053:53"
- "1053:53/udp"
- "8081:8081"
user: "1000:1000"
But when starting built container, I see that LAUNCH=gpgsql has no effect.
Oct 17 05:59:57 Loading '/usr/local/lib/pdns/libgsqlite3backend.so'
Oct 17 05:59:57 This is a standalone pdns
Oct 17 05:59:57 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
Oct 17 05:59:57 UDP server bound to 0.0.0.0:53
Oct 17 05:59:57 UDP server bound to [::]:53
Oct 17 05:59:57 TCP server bound to 0.0.0.0:53
Oct 17 05:59:57 TCP server bound to [::]:53
Oct 17 05:59:57 PowerDNS Authoritative Server 4.8.0-alpha0.494.master.g4189b2d41 (C) 2001-2022 PowerDNS.COM BV
Oct 17 05:59:57 Using 64-bits mode. Built using gcc 10.2.1 20210110 on Oct 16 2022 13:14:45 by root#localhost.
Oct 17 05:59:57 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Oct 17 05:59:57 Not validating response for security status update, this is a non-release version
Oct 17 05:59:57 gsqlite3: connection failed: SQLite database '/var/lib/powerdns/pdns.sqlite3' does not exist yet
Oct 17 05:59:57 Caught an exception instantiating a backend: Unable to launch gsqlite3 connection: SQLite database '/var/lib/powerdns/pdns.sqlite3' does not exist yet
Oct 17 05:59:57 Cleaning up
Oct 17 05:59:57 PDNSException while filling the zone cache: Unable to launch gsqlite3 connection: SQLite database '/var/lib/powerdns/pdns.sqlite3' does not exist yet
Notice. PowerDNS tries to load sqlite3.
When I mounted configuration file, then option is recognized:
volumes:
- ./volumes/pdns.conf:/etc/powerdns/pdns.conf
cat ./volumes/pdns.conf:
loglevel=7
primary=yes
launch=gpgsql
gpgsql-host=pgdb
gpgsql-user=pdns
gpgsql-password=***
gpgsql-dbname=pdns
Why environment variable - PDNS_LAUNCH=gpgsql, setted up from docker compose, was not recognized by PowerDNS?

Related

Cannot access apache server from foreign ip address

I have a site running on my computer using Apache 2.4 which I can easily access by using my local ipv4 address and respective port 80. The port 80 is bound to port 22*** using portmap.io and is configured with OpenVpn/tcp on my computer.I have allowed access to Apache HTTP server and Apache Server Monitor through the firewall.I have also increased keepAlive timeout in apche server to 600s, max connections.i have Listen 80 and LISTEN 22*** and ServerName as http://awm-22***.portmap.host:22*** in my httpd.conf file.You can look into for more options here.Apache handler configuration
I am using PHP as backend language.
Since the port 80 is bound to port 22470, whenever I try access my site from another device(which also uses the same wifi network as the computer running the server) using the local ipv4 address of my computer(which is running the server) and port 80 i.e
192.168..:80, it automatically redirects the browser to 192.168..:22*** and I can access my site with no difficuty. Access log in apcache server:
192.168.**.** - - [15/Dec/2022:10:08:02 +0530] "GET /abc%20xyz%20klm/ HTTP/1.1" 200 12049
192.168.**.** - - [15/Dec/2022:10:08:02 +0530] "GET /SPR/b/get_captcha.php?rand=29842778 HTTP/1.1" 200 4057
But when I try access the same site from another device(which also uses the same wifi network as the computer running the server) using the ipv4 address provided by OpenVpn to my computer(which is running the server):10.9..* and port 80 i.e 10.9..*:80, it shows TOOK TOO LONG TO RESPPOND error on the browser.But the browser's header has the following:
http://10.9.**.**4:22470/abc%20xyz%20klm/
Then why is it not loading the page. No log in apache access log file.
When I try access the same site from another device(which also uses the same wifi network as the computer running the server) using the url provided to me by portmap.io i.e: http://awm-22***.portmap.host:22***/,
The server takes too long to respond error is shown in the browser.
Access log in apcache server:
10.9.0.1 - - [15/Dec/2022:10:21:33 +0530] "GET / HTTP/1.0" 302 -
10.9.0.1 - - [15/Dec/2022:10:21:34 +0530] "GET /abc%20xyz%20klm HTTP/1.0" 301 256
OpenVpn Log:
Thu Dec 15 10:32:30 2022 SIGHUP[hard,] received, process restarting
Thu Dec 15 10:32:30 2022 --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Thu Dec 15 10:32:30 2022 OpenVPN 2.5.7 [git:release/2.5/3d792ae9557b959e] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Oct 28 2022
Thu Dec 15 10:32:30 2022 Windows version 10.0 (Windows 10 or greater) 64bit
Thu Dec 15 10:32:30 2022 library versions: OpenSSL 1.1.1o 3 May 2022, LZO 2.10
Thu Dec 15 10:32:35 2022 TCP/UDP: Preserving recently used remote address: [AF_INET]193.161.193.99:1194
Thu Dec 15 10:32:35 2022 Attempting to establish TCP connection with [AF_INET]193.161.193.99:1194 [nonblock]
Thu Dec 15 10:32:35 2022 TCP connection established with [AF_INET]193.161.193.99:1194
Thu Dec 15 10:32:35 2022 TCP_CLIENT link local: (not bound)
Thu Dec 15 10:32:35 2022 TCP_CLIENT link remote: [AF_INET]193.161.193.99:1194
Thu Dec 15 10:32:41 2022 [193.161.193.99] Peer Connection Initiated with [AF_INET]193.161.193.99:1194
Thu Dec 15 10:32:42 2022 WARNING: INSECURE cipher (BF-CBC) with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Support for these insecure ciphers will be removed in OpenVPN 2.7.
Thu Dec 15 10:32:42 2022 WARNING: INSECURE cipher (BF-CBC) with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Support for these insecure ciphers will be removed in OpenVPN 2.7.
Thu Dec 15 10:32:42 2022 WARNING: cipher with small block size in use, reducing reneg-bytes to 64MB to mitigate SWEET32 attacks.
Thu Dec 15 10:32:42 2022 open_tun
Thu Dec 15 10:32:42 2022 tap-windows6 device [OpenVPN TAP-Windows6] opened
Thu Dec 15 10:32:42 2022 Notified TAP-Windows driver to set a DHCP IP/netmask of 10.9.**.234/255.255.255.252 on interface {798F492A-574C-4BC6-87C5-A62C6D058EC1} [DHCP-serv: 10.9.**.233, lease-time: 31536000]
Thu Dec 15 10:32:42 2022 Successful ARP Flush on interface [12] {798F492A-574C-4BC6-87C5-A62C6D058EC1}
Thu Dec 15 10:32:42 2022 IPv4 MTU set to 1500 on interface 12 using service
These are my firewall rules:
Inbound rules
Inbound rule for Port 80 Outbound rules
Firewall monitoring Domain and Private Profiles
Firewall monitoring Public Profile
What is causing the problem? Any solution will be of great help. Thanks in advance.

redis rdb file permission denied in AWS ECS

i have this redis service in my docker compose file
redis:
container_name: redis
image: redis:${VERSION_REDIS}
ports:
- "6379:6379"
networks:
- backend
if i run docker-compose locally, than the redis db is working as intended.
1:M 12 Jun 2022 14:47:51.212 * Running mode=standalone, port=6379.
1:M 12 Jun 2022 14:47:51.213 # Server initialized
1:M 12 Jun 2022 14:47:51.215 * Loading RDB produced by version 6.2.5
1:M 12 Jun 2022 14:47:51.216 * RDB age 728 seconds
1:M 12 Jun 2022 14:47:51.219 * RDB memory usage when created 0.79 Mb
1:M 12 Jun 2022 14:47:51.219 * DB loaded from disk: 0.004 seconds
1:M 12 Jun 2022 14:47:51.220 * Ready to accept connections
The version used is 6.2.5-buster. If i try to use docker-compose to my AWS ECS cluster, every service starts normal but the redis task has the following protocol output:
12.6.2022, 16:32:34 1:M 12 Jun 2022 14:32:34.734 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:34 1:M 12 Jun 2022 14:32:34.164 # Failed opening the RDB file zzh (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.836 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.646 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.267 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:30 1:M 12 Jun 2022 14:32:30.939 # Failed opening the RDB file backup.db (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:40:47 1:M 12 Jun 2022 14:40:47.168 # Background saving error
i tried to search for solutions, but i am just starting out with container services on AWS. How can i configure my docker-compose file or how to change the permissions inside aws to make redis work? Are there other ways? Thank you. The cluster is running on fargate.

nodered on Raspberry PI is not starting any longer

I am not quite sure what node is causing this behaviour and there are tooo many flows so I can not install from scratch and yes I do not have a backup of them.
I realized today in the morning that I can not access the http gui of my nodered instance any longer on my raspberrypi zero. Just edited some flows but nothing real serious.
I am trying to start my node red on my Rapsberry PI zere and no GUI and UI is starting up to access the node red instance. I don't know how to solve and troubleshoot this. What I am doing or trying to do is:
pi#nodered-pi:~/.node-red $ node-red-start
Start Node-RED
Once Node-RED has started, point a browser at http://192.168.1.42:1880
On Pi Node-RED works better with the Firefox or Chrome browser
Use node-red-stop to stop Node-RED
Use node-red-start to start Node-RED again
Use node-red-log to view the recent log output
Use sudo systemctl enable nodered.service to autostart Node-RED at every boot
Use sudo systemctl disable nodered.service to disable autostart on boot
To find more nodes and example flows - go to http://flows.nodered.org
Starting as a systemd service.
Started Node-RED graphical event wiring tool.
19 Aug 15:13:55 - [info]
Welcome to Node-RED
===================
19 Aug 15:13:55 - [info] Node-RED version: v0.18.7
19 Aug 15:13:55 - [info] Node.js version: v8.11.1
19 Aug 15:13:55 - [info] Linux 4.14.52+ arm LE
19 Aug 15:14:06 - [info] Loading palette nodes
19 Aug 15:14:37 - [info] Dashboard version 2.9.6 started at /ui
19 Aug 15:14:49 - [warn] ------------------------------------------------------
19 Aug 15:14:49 - [warn] [node-red-contrib-delta-timed/delta-time] 'delta' already registered by module node-red-contrib-change-detect
19 Aug 15:14:49 - [warn] ------------------------------------------------------
19 Aug 15:14:49 - [info] Settings file : /home/pi/.node-red/settings.js
19 Aug 15:14:49 - [info] User directory : /home/pi/.node-red
19 Aug 15:14:49 - [warn] Projects disabled : set editorTheme.projects.enabled=true to enable
19 Aug 15:14:49 - [info] Flows file : /home/pi/.node-red/flows_nodered-pi.json
19 Aug 15:14:50 - [info] Server now running at http://127.0.0.1:1880/
19 Aug 15:14:50 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
19 Aug 15:14:50 - [warn] Error loading credentials: SyntaxError: Unexpected token T in JSON at position 0
19 Aug 15:14:50 - [warn] Error loading flows: Error: Failed to decrypt credentials
19 Aug 15:14:51 - [info] Starting flows
19 Aug 15:15:01 - [warn] [telegram receiver:Telegram Receiver] bot not initialized
19 Aug 15:15:01 - [warn] [telegram sender:Temperatur Wetterstation] bot not initialized.
19 Aug 15:15:01 - [error] [function:Versorge mit Information] SyntaxError: Invalid or unexpected token
19 Aug 15:15:01 - [info] Started flows
19 Aug 15:15:02 - [info] [sonoff-server:166ef3ba.0029bc] SONOFF Server Started On Port 1080
19 Aug 15:15:02 - [red] Uncaught Exception:
19 Aug 15:15:02 - Error: listen EACCES 0.0.0.0:443
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
nodered.service: Main process exited, code=exited, status=1/FAILURE
nodered.service: Unit entered failed state.
nodered.service: Failed with result 'exit-code'.
nodered.service: Service hold-off time over, scheduling restart.
Stopped Node-RED graphical event wiring tool.
Started Node-RED graphical event wiring tool.
19 Aug 15:15:20 - [info]
Welcome to Node-RED
===================
19 Aug 15:15:20 - [info] Node-RED version: v0.18.7
19 Aug 15:15:02 - Error: listen EACCES 0.0.0.0:443
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
This error implies that something else is already running on port 443. This could be an existing copy of Node-RED or something else. You can search what applications are listening on what ports with the following command
lsof -i :443
This will list what is listening on port 443

freeradius startup error (code=exited, status=1/FAILURE)

I am new to freeradius. I have installed freeradius on centos-7 and planned to manage it using daloradius.
After installing freeradius it was working fine and started normally by "systemctl start radiusd".
but after installing and configuring MariaDB, freeradius is not starting while working with "radiusd -X" is working.
following is the error message:
[root#radiusx1 ~]# systemctl start radiusd
Warning: radiusd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for radiusd.service failed because the control process exited with error code. See "systemctl status radiusd.service" and "journalctl -xe" for details.
[root#radiusx1 ~]# systemctl status radiusd
● radiusd.service - FreeRADIUS high performance RADIUS server.
Loaded: loaded (/usr/lib/systemd/system/radiusd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2017-03-02 17:21:00 PKT; 10s ago
Process: 26712 ExecStartPre=/usr/sbin/radiusd -C (code=exited, status=1/FAILURE)
Process: 26708 ExecStartPre=/bin/chown -R radiusd.radiusd /var/run/radiusd (code=exited, status=0/SUCCESS)
Mar 02 17:20:59 radiusx1.xoultech.com systemd[1]: Starting FreeRADIUS high performance RADIUS server....
Mar 02 17:21:00 radiusx1.xoultech.com systemd[1]: radiusd.service: control process exited, code=exited status=1
Mar 02 17:21:00 radiusx1.xoultech.com systemd[1]: Failed to start FreeRADIUS high performance RADIUS server..
Mar 02 17:21:00 radiusx1.xoultech.com systemd[1]: Unit radiusd.service entered failed state.
Mar 02 17:21:00 radiusx1.xoultech.com systemd[1]: radiusd.service failed.
Please help me to sort out this issue.
Any help will be appreciated.
That's likely a permissions issue. -X runs the process as the current user, whereas -C will change to the configured user/group.
You can use this invocation to perform a config check with debugging info written to stdout.
/usr/sbin/radiusd -C -lstdout -xxx
-C config check
-lstdout override configured logging destination and set to stdout.
-xxx set debugging to level 3. Should show any useful output.
Report back with the output if it doesn't make sense to you, and i'll interpret.
I match same issue, after check by Arran's command that will let you know reason, but suggest you to enable radius by "radius -X" under root environment, that will give you root cause, as follow my issue:
Under user environment:
[cisco#frank ~]$ /usr/sbin/radiusd -C -lstdout -xxx
Wed May 17 04:47:36 2017 : Info: radiusd: FreeRADIUS Version 3.0.1, for host x86_64-redhat-linux-gnu, built on Mar 5 2014 at 05:31:12
Wed May 17 04:47:36 2017 : Debug: Server was built with:
Wed May 17 04:47:36 2017 : Debug: accounting
Wed May 17 04:47:36 2017 : Debug: authentication
Wed May 17 04:47:36 2017 : Debug: ascend binary attributes
Wed May 17 04:47:36 2017 : Debug: coa
Wed May 17 04:47:36 2017 : Debug: control-socket
Wed May 17 04:47:36 2017 : Debug: detail
Wed May 17 04:47:36 2017 : Debug: dhcp
Wed May 17 04:47:36 2017 : Debug: dynamic clients
Wed May 17 04:47:36 2017 : Debug: proxy
Wed May 17 04:47:36 2017 : Debug: regex-pcre
Wed May 17 04:47:36 2017 : Debug: session-management
Wed May 17 04:47:36 2017 : Debug: stats
Wed May 17 04:47:36 2017 : Debug: tcp
Wed May 17 04:47:36 2017 : Debug: threads
Wed May 17 04:47:36 2017 : Debug: tls
Wed May 17 04:47:36 2017 : Debug: unlang
Wed May 17 04:47:36 2017 : Debug: vmps
Wed May 17 04:47:36 2017 : Debug: Server core libs:
Wed May 17 04:47:36 2017 : Debug: talloc : 2.0.*
Wed May 17 04:47:36 2017 : Debug: ssl : OpenSSL 1.0.1e-fips 11 Feb 2013
Wed May 17 04:47:36 2017 : Info: Copyright (C) 1999-2014 The FreeRADIUS server project and contributors
Wed May 17 04:47:36 2017 : Info: There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
Wed May 17 04:47:36 2017 : Info: PARTICULAR PURPOSE
Wed May 17 04:47:36 2017 : Info: You may redistribute copies of FreeRADIUS under the terms of the
Wed May 17 04:47:36 2017 : Info: GNU General Public License
Wed May 17 04:47:36 2017 : Info: For more information about these matters, see the file named COPYRIGHT
Wed May 17 04:47:36 2017 : Info: Starting - reading configuration files ...
Wed May 17 04:47:36 2017 : Debug: including dictionary file /etc/raddb/dictionary
Wed May 17 04:47:36 2017 : Debug: including configuration file /etc/raddb/radiusd.conf
Wed May 17 04:47:36 2017 : Error: Unable to open file "/etc/raddb/radiusd.conf": Permission denied
Wed May 17 04:47:36 2017 : Error: Errors reading or parsing /etc/raddb/radiusd.conf
[cisco#frank ~]$ radiusd -X
radiusd: FreeRADIUS Version 3.0.1, for host x86_64-redhat-linux-gnu, built on Mar 5 2014 at 05:31:12
Copyright (C) 1999-2014 The FreeRADIUS server project and contributors
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE
You may redistribute copies of FreeRADIUS under the terms of the
GNU General Public License
For more information about these matters, see the file named COPYRIGHT
Starting - reading configuration files ...
including dictionary file /etc/raddb/dictionary
including configuration file /etc/raddb/radiusd.conf
Unable to open file "/etc/raddb/radiusd.conf": Permission denied
Errors reading or parsing /etc/raddb/radiusd.conf
Under root environment:
[root#frank ~]# radiusd -X
radiusd: FreeRADIUS Version 3.0.1, for host x86_64-redhat-linux-gnu, built on Mar 5 2014 at 05:31:12
Copyright (C) 1999-2014 The FreeRADIUS server project and contributors
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE
You may redistribute copies of FreeRADIUS under the terms of the
GNU General Public License
For more information about these matters, see the file named COPYRIGHT
Starting - reading configuration files ...
including dictionary file /etc/raddb/dictionary
including configuration file /etc/raddb/radiusd.conf
including configuration file /etc/raddb/proxy.conf
including configuration file /etc/raddb/clients.conf
......
client 10.75.13.13 {
require_message_authenticator = no
secret = "cisco"
shortname = "asr9006-3"
limit {
max_connections = 16
lifetime = 0
idle_timeout = 30
}
}
Failed to add duplicate client asr9006-3
/etc/raddb/clients.conf[297]: Failed to add client 10.75.13.13
After correct config issue by root, then restart or reboot, then check again under user:
[cisco#frank ~]$ systemctl status radiusd
radiusd.service - FreeRADIUS high performance RADIUS server.
Loaded: loaded (/usr/lib/systemd/system/radiusd.service; enabled)
Active: active (running) since Wed 2017-05-17 05:10:15 CST; 10min ago
Process: 2438 ExecStart=/usr/sbin/radiusd -d /etc/raddb (code=exited, status=0/SUCCESS)
Process: 1956 ExecStartPre=/usr/sbin/radiusd -C (code=exited, status=0/SUCCESS)
Process: 1951 ExecStartPre=/bin/chown -R radiusd.radiusd /var/run/radiusd (code=exited, status=0/SUCCESS)
Main PID: 2439 (radiusd)
CGroup: /system.slice/radiusd.service
└─2439 /usr/sbin/radiusd -d /etc/raddb

Zookeeper / Exhibitor recurring JMX Error

I don't know why this is occurring, but occasionally I will get this series of repeating errors and the zookeeper instances will go into a bad state.
Tue Feb 16 07:05:04 EST 2016 ERROR ZooKeeper Server: Using config: /opt/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Tue Feb 16 07:05:04 EST 2016 ERROR ZooKeeper Server: JMX enabled by default
Tue Feb 16 07:05:04 EST 2016 INFO Process started via: /opt/zookeeper/zookeeper-3.4.6/bin/zkServer.sh
Tue Feb 16 07:05:03 EST 2016 INFO Kill attempted result: 0
Tue Feb 16 07:05:03 EST 2016 INFO Attempting to start/restart ZooKeeper
Tue Feb 16 07:05:03 EST 2016 INFO Attempting to stop instance
Tue Feb 16 07:05:03 EST 2016 INFO Restarting down/not-serving ZooKeeper after 60037 ms pause
Tue Feb 16 07:04:33 EST 2016 INFO ZooKeeper down/not-serving waiting 30026 of 40000 ms before restarting
Tue Feb 16 07:04:05 EST 2016 INFO ZooKeeper Server: Starting zookeeper ... STARTED
Tue Feb 16 07:04:04 EST 2016 ERROR ZooKeeper Server: Using config: /opt/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Tue Feb 16 07:04:04 EST 2016 ERROR ZooKeeper Server: JMX enabled by default
The exhibitor stuff uses shared storage on a NAS. The servers are centOs 6.6. It is a three node ensemble, and the one noticible problem I have seen is that the "ensemble" connection string inside of Exhibitor GUI all of a sudden becomes different between the three nodes (one node may "forget" about some of the other nodes in the ensemble).
I don't even know where to look to dig into these causes. Any help or direction will be greatly appreciated. Its trully odd...
update versions
zk: 3.4.6
Exhibitor: 1.5.5