I'm running a dockerized distribution of Confluent platform:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6963904b485 confluentinc/cp-enterprise-control-center:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:9021->9021/tcp, :::9021->9021/tcp control-center
49ade0e752b4 confluentinc/cp-ksqldb-cli:7.0.1 "/bin/sh" 11 hours ago Up 11 hours ksqldb-cli
95b0982c0159 confluentinc/ksqldb-examples:7.0.1 "bash -c 'echo Waiti…" 11 hours ago Up 11 hours ksql-datagen
e28e3b937f6e confluentinc/cp-ksqldb-server:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp ksqldb-server
af92bfb84cb1 confluentinc/cp-kafka-rest:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:8082->8082/tcp, :::8082->8082/tcp rest-proxy
318a999e76dc cnfldemos/cp-server-connect-datagen:0.5.0-6.2.0 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp connect
0c299fbda7c5 confluentinc/cp-schema-registry:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp schema-registry
a33075002386 confluentinc/cp-server:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:9101->9101/tcp, :::9101->9101/tcp broker
135f832fbccb confluentinc/cp-zookeeper:7.0.1 "/etc/confluent/dock…" 11 hours ago Up 11 hours 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
Kafka REST Proxy is running on port 8082
When I issue an HTTP GET call against the REST proxy:
curl --silent -X GET http://10.0.0.253:8082/kafka/clusters/ | jq
All I get is:
{
"error_code": 404,
"message": "HTTP 404 Not Found"
}
Given my configuration, what can I change to actually get some useful information out of Kafka REST Proxy?
Is it possible to install collections in a axw container ?
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c4d255148cc8 netboxcommunity/netbox:latest "/opt/netbox/docker-…" 5 days ago Up 5 days 0.0.0.0:8000->8080/tcp, :::8000->8080/tcp netbox-docker_netbox_1
ac0784c16861 netboxcommunity/netbox:latest "/opt/netbox/venv/bi…" 5 days ago Up 5 days netbox-docker_netbox-worker_1
31b850bf8d12 redis:6-alpine "docker-entrypoint.s…" 5 days ago Up 5 days 6379/tcp netbox-docker_redis-cache_1
df0977f446f4 postgres:12-alpine "docker-entrypoint.s…" 5 days ago Up 5 days 5432/tcp netbox-docker_postgres_1
983b698274af redis:6-alpine "docker-entrypoint.s…" 5 days ago Up 5 days 6379/tcp netbox-docker_redis_1
4150e6ae71cc ansible/awx:17.1.0 "/usr/bin/tini -- /u…" 6 days ago Up 6 days 8052/tcp awx_task
5583bbf60d45 ansible/awx:17.1.0 "/usr/bin/tini -- /b…" 6 days ago Up 6 days 0.0.0.0:80->8052/tcp, :::80->8052/tcp awx_web
c9d92412d1cd redis "docker-entrypoint.s…" 6 days ago Up 6 days 6379/tcp awx_redis
71b99bde5d41 postgres:12 "docker-entrypoint.s…" 6 days ago Up 6 days 5432/tcp awx_postgres
Got the solution.
Create a folder: collections
Create file: requirements.yml
File content:
collections:
- name: community.general
I am trying to setup an keycloak instance with an ssl connection over an nginx proxy my 'docker ps' output:
d7fd473cc77b jboss/keycloak "/opt/jboss/tools/do…" 34 minutes ago Up 8 minutes 0.0.0.0:8080->8080/tcp, 8443/tcp auth
76e757bbe129 mariadb "sh -c ' echo 'CREAT…" 34 minutes ago Up 8 minutes 0.0.0.0:3306->3306/tcp backend-database
d99e23470955 stilliard/pure-ftpd:hardened-latest "/bin/sh -c '/run.sh…" 34 minutes ago Up 8 minutes 0.0.0.0:21->21/tcp, 0.0.0.0:30000->30000/tcp, 30001-30009/tcp ftp-server
95f4fbdea0de wordpress:latest "docker-entrypoint.s…" 35 minutes ago Up 35 minutes 80/tcp wordpress
b3e40ca6de48 mariadb:latest "docker-entrypoint.s…" 35 minutes ago Up 35 minutes 3306/tcp database
e5c12bb5ba52 nginx "/docker-entrypoint.…" 37 minutes ago Up 37 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp nginx-web
c0ac90a6c408 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 37 minutes ago Up 37 minutes nginx-letsencrypt
33ae7de5f598 jwilder/docker-gen "/usr/local/bin/dock…" 37 minutes ago Up 37 minutes nginx-gen
As you can see at the above console output, I am also running an instance of wordpress in a docker container and this works like a charm, no problems with unsigned or invalid ssl certificates, just erverything fine. But when I am trying to call the web interface of keycloak over the domain with the corresponding port (in my case: 8080), I got the following error:
Fehlercode: SSL_ERROR_RX_RECORD_TOO_LONG
And when I am trying to call the web interface over the ip address also with the corresponding port, I got the message that the connection isn't safe.
Hopefully this are enough information for you guys, to figure out what I've done wrong.
So far,
Daniel
I'm setting up a plex requests server (https://github.com/lokenx/plexrequests-meteor/blob/master/README.md) on a completely fresh ubuntu installation.
I started by downloading meteor curl https://install.meteor.com/ | sh which seemed to work without any issues. I then cloned the plex-requests project, and ran meteor from inside that directory.
This is the error that is generated:
plex#johnpc:/home/plex/plexrequests-meteor$ meteor
[[[[[ /home/plex/plexrequests-meteor ]]]]]
=> Started proxy.
=> Started MongoDB.
Unexpected mongo exit code null. Restarting.
/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/fibers/future.js:245
throw(ex);
^
Error: spawn ENOMEM
at errnoException (child_process.js:1011:11)
at ChildProcess.spawn (child_process.js:958:11)
at Object.exports.spawn (child_process.js:746:9)
at spawnMongod (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/runners/run-mongo.js:39:24)
at launchOneMongoAndWaitForReadyForInitiate (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/runners/run-mongo.js:426:12)
at launchMongo (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/runners/run-mongo.js:610:7)
at [object Object]._.extend._startOrRestart (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/runners/run-mongo.js:704:19)
at [object Object].<anonymous> (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/runners/run-mongo.js:762:14)
at runWithEnvironment (/var/lib/plexmediaserver/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/utils/fiber-helpers.js:143:21)
I am wondering whether there are some other dependencies that I am missing, or what else could be causing this issue. My guess is that it could be a permissions issue of some kind, because I've run into a handful of those already on this server, but I'm not sure where. It appears as though the permissions are correct on the directory where the error was thrown.
plex#johnpc:/home/plex/plexrequests-meteor$ ls -al /var/lib/plexmediaserver
total 40
drwxr-xr-x 8 plex plex 4096 Nov 29 17:31 .
drwxr-xr-x 45 root root 4096 Nov 29 19:28 ..
drwx------ 2 plex plex 4096 Nov 28 22:33 .cache
drwxrwxr-x 3 plex plex 4096 Nov 29 17:10 .cordova
drwxrwxr-x 3 plex plex 4096 Nov 28 21:26 Library
drwxr-xr-x 4 plex plex 4096 Oct 26 20:16 .meteor
-rw------- 1 plex plex 90 Nov 29 17:31 .meteorsession
drwxrwxr-x 11 plex plex 4096 Nov 29 17:31 .npm
drwxrwxr-x 3 plex plex 4096 Nov 29 19:34 .s3ql
-rw------- 1 plex plex 638 Nov 29 17:17 .viminfo
The other issue is it may be that I'm missing dependencies - I installed mongo to see if that would fix the problem but it seems to be persisting. I tried the solutions suggested here, by setting the lang env variables and reseting meteor, nothing there solved the issue. Thoughts?
The system shown below works flawlessly on my macbook pro, but once I try to recreate it on my fresh ubuntu (Ubuntu 14.04 x64) box this problem arises.
Another possibility is that this could be memory related. I've seen people online struggle with meteor when the memory runs out. It doesn't look like that's happening to me, but correct me if I'm wrong about that.
free:
/home/plex/plexrequests-meteor master 83s
# ❯ free -m
total used free shared buffers cached
Mem: 490 225 264 0 7 50
-/+ buffers/cache: 168 322
Swap: 0 0 0
/proc/meminfo:
# ❯ cat /proc/meminfo
MemTotal: 501800 kB
MemFree: 269708 kB
Buffers: 8004 kB
Cached: 51856 kB
SwapCached: 0 kB
Active: 147216 kB
Inactive: 48484 kB
Active(anon): 135940 kB
Inactive(anon): 260 kB
Active(file): 11276 kB
Inactive(file): 48224 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 135880 kB
Mapped: 13188 kB
Shmem: 348 kB
Slab: 18584 kB
SReclaimable: 9436 kB
SUnreclaim: 9148 kB
KernelStack: 1208 kB
PageTables: 5264 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 250900 kB
Committed_AS: 876812 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 7892 kB
VmallocChunk: 34359722172 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 32760 kB
DirectMap2M: 491520 kB
DirectMap1G: 0 kB
From my experience with cloud9, which has 768MB of memory, I'd get the very same exception after meteor is running for few minutes and the memory is full.
If you want to develop with meteor, you'll need much more RAM than you have available. If you just want to run it, you can build the application and run it as a simple node app, but this doesn't come with MongoDB, that one you'd need to add extra.
Customers are reporting problems almost every day on about the same hours. This app is running on 2 nodes. It is Metastorm BPM platform and it's calling our code.
In some dumps I noticed very long running threads (~50 minutes) but not in all of them. Administrators are also telling me that just before users report problems memory usage goes up. Then everything slows down to the point they can't work and admins have to restart platforms on both nodes. My first thought was deadlocks (long running threads) but didn't manage to confirm that. !syncblk isn't returning anything. Then I looked at memory usage. I noticed a lot of dynamic assemblies so thought maybe assemblies leak. But it looks it's not that. I have received dump from day where everything was working fine and number of dynamic assemblies is similar. So maybe memory leak I thought. But also cannot confirm that. !dumpheap -stat shows memory usage grows but I haven't found anything interesting with !gcroot. But there is one thing I don't know what it is. Threadpool Completion Port. There's a lot of them. So maybe sth is waiting on sth? Here is data I can give You so far that will fit in this post. Could You suggest anything that could help diagnose this situation?
Users not reporting problems:
Node1 Node2
Size of dump: 638MB 646MB
DynamicAssemblies 259 265
GC Heaps: 37MB 35MB
Loader Heaps: 11MB 11MB
Node1:
Number of Timers: 12
CPU utilization 2%
Worker Thread: Total: 5 Running: 0 Idle: 5 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 2 Free: 2 MaxFree: 16 CurrentLimit: 4 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat (biggest)
0x793041d0 32,664 2,563,292 System.Object[]
0x79332b9c 23,072 3,485,624 System.Int32[]
0x79330a00 46,823 3,530,664 System.String
0x79333470 22,549 4,049,536 System.Byte[]
Node2:
Number of Timers: 12
CPU utilization 0%
Worker Thread: Total: 7 Running: 0 Idle: 7 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 3 Free: 1 MaxFree: 16 CurrentLimit: 5 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x793041d0 30,678 2,537,272 System.Object[]
0x79332b9c 21,589 3,298,488 System.Int32[]
0x79333470 21,825 3,680,000 System.Byte[]
0x79330a00 46,938 5,446,576 System.String
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Users start to report problems:
Node1 Node2
Size of dump: 662MB 655MB
DynamicAssemblies 236 235
GC Heaps: 159MB 113MB
Loader Heaps: 10MB 10MB
Node1:
Work Request in Queue: 0
Number of Timers: 14
CPU utilization 20%
Worker Thread: Total: 7 Running: 0 Idle: 7 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 48 Free: 1 MaxFree: 16 CurrentLimit: 49 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x7932a208 88,974 3,914,856 System.Threading.ReaderWriterLock
0x79333054 71,397 3,998,232 System.Collections.Hashtable
0x24f70350 319,053 5,104,848 Our.Class
0x79332b9c 53,190 6,821,588 System.Int32[]
0x79333470 52,693 6,883,120 System.Byte[]
0x79333150 72,900 11,081,328 System.Collections.Hashtable+bucket[]
0x793041d0 247,011 26,229,980 System.Object[]
0x79330a00 644,807 34,144,396 System.String
Node2:
Work Request in Queue: 1
Number of Timers: 17
CPU utilization 17%
Worker Thread: Total: 6 Running: 0 Idle: 6 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 48 Free: 2 MaxFree: 16 CurrentLimit: 49 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x7932a208 76,425 3,362,700 System.Threading.ReaderWriterLock
0x79332b9c 42,417 5,695,492 System.Int32[]
0x79333150 41,172 6,451,368 System.Collections.Hashtable+bucket[]
0x79333470 44,052 6,792,004 System.Byte[]
0x793041d0 175,973 18,573,780 System.Object[]
0x79330a00 397,361 21,489,204 System.String
Edit:
I downloaded debugdiag and let it analyze my dumps. Here is part of output:
The following threads in process_name name_of_dump.dmp are making a COM call to thread 193 within the same process which in turn is waiting on data to be returned from another server via WinSock.
The call to WinSock originated from 0x0107b03b and is destined for port xxxx at IP address xxx.xxx.xxx.xxx
( 18 76 172 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 210 211 212 213 214 215 216 217 218 224 225 226 227 228 229 231 232 233 236 239 )
14,79% of threads blocked
And the recommendation is:
Several threads making calls to the same STA thread can cause a performance bottleneck due to serialization. Server side COM servers are recommended to be thread aware and follow MTA guidelines when multiple threads are sharing the same object instance.
I checked using windbg what thread 193 does. It is calling our code. Our code is calling some Metastorm engine code and it hangs on some remoting call. But !runaway shows it is hanging for 8 seconds. So not that long. So I checked what are those waiting threads. All except thread 18 are:
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) I could understand one, but why so many of them. Is it specific to business process modeling engine we're using or is it something typical? I guess it's taking threads that could be used by other clients and that's why the slowdown reported by users. Are those threads Completion Port Threads I asked about before? Can I do anything more to diagnose or did I found our code to be the cause?
From the looks of the output most of the memory is not on the .net heaps (only 35 MB out of ~650) so if you are looking at the .net heaps I think you are looking in the wrong place. The memory is probably either in assemblies or in native memory if you are using some native component for file transfers or similar. You would want to use Debug Diag to monitor that.
It is hard to say if you are leaking dynamic assemblies without looking at the pattern of growth so I would suggest for that that you look at perfmon and #current assemblies to see if it keeps growing over time, if it does then you would have to investigate that further by looking at what the dynamic assemblies are with !dda