I'm trying to figure out, what is import/export best practices in K8S keycloak(version 3.3.0.CR1). Here is keycloak official page import/export explanation, and they example of export to single file json. Going to /keycloak/bin folder and the run this:
./standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.file=keycloak-export.json
I logged in to pod, and I get errors after run this command:
12:23:32,045 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("core-service" => "management"),
("management-interface" => "http-interface")
]) - failure description: {
"WFLYCTL0080: Failed services" => {"org.wildfly.management.http.extensible" => "java.net.BindException: Address already in use /127.0.0.1:9990"},
"WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => ["org.wildfly.management.http.extensible.shutdown"],
"Services that may be the cause:" => ["jboss.remoting.remotingConnectorInfoService.http-remoting-connector"]
}
}
As I see, Keycloak server run on the same port, where I ran backup script. Here helm/keycloak values.yml:
Service:
Name: keycloak
Port: 8080
Type: ClusterIP
Deployment:
Image: jboss/keycloak
ImageTag: 2.5.1.Final
ImagePullPolicy: IfNotPresent
ContainerPort: 8080
KeycloakUser: Admin
KeycloakPassword: Admin
So, server should be stopped, before we ran this scripts? I can't stop keycloak process inside of pod, because ingress will close pod and will create new one.
Any suggestions for any other way to export/import(backup/restore) data? Or I missing something?
P.S.
I even tried UI import/export. Export work good, and I see all data. But import worked in half way. He Brought me all "Clients", but not my "Realm" and "User Federation".
Basically, you just have to start the exporting Keycloak instance on ports that are different from your main instance. I used something like this just now:
bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.file=keycloak-export.json -Djboss.http.port=8888 -Djboss.https.port=9999 -Djboss.management.http.port=7777
The important part are all the ports. If you get more error messages, you might need to add more properties (grep port standalone/configuration/standalone.xml is your friend for finding out property names), but in the end, all error messages stop and you see this message instead:
09:15:26,550 INFO [org.keycloak.exportimport.singlefile.SingleFileExportProvider] (ServerService Thread Pool -- 52) Exporting model into file /opt/jboss/keycloak/keycloak-export.json
[...]
09:15:29,565 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 3.2.0.Final (WildFly Core 2.0.10.Final) started in 12156ms - Started 444 of 818 services (558 services are lazy, passive or on-demand)
Now you can stop the server with Ctrl-C, exit the container and copy the export file away with kubectl cp.
The Export functionality changed with the Quarkus distribution.
See Documentation.
Be aware, that there is a bug, when the InfinispanCache is enabled. In short, the port for the cache is blocked, when trying to export.
Keycloak Issue
Bitnami Issue, with workaround
Related
I deployed an aws eks cluster via terraform. I also deployed Consul following hasicorp’s tutorial and I see the nodes in consul’s UI.
Now I’m wondering how al the consul agents will know about the pods I deploy? I deploy something and it’s not shown anywhere on consul.
I can’t find any documentation as to how to register pods (services) on consul via the node’s consul agent, do I need to configure that somewhere? Should I not use the node’s agent and register the service straight from the pod? Hashicorp discourages this since it may increase resource utilization depending on how many pods one deploy on a given node. But then how does the node’s agent know about my services deployed on that node?
Moreover, when I deploy a pod in a node and ssh into the node, and install consul, consul’s agent can’t find the consul server (as opposed from the node, which can find it)
EDIT:
Bottom line is I can't find WHERE to add the configuration. If I execute ON THE POD:
consul members
It works properly and I get:
Node Address Status Type Build Protocol DC Segment
consul-consul-server-0 10.0.103.23:8301 alive server 1.10.0 2 full <all>
consul-consul-server-1 10.0.101.151:8301 alive server 1.10.0 2 full <all>
consul-consul-server-2 10.0.102.112:8301 alive server 1.10.0 2 full <all>
ip-10-0-101-129.ec2.internal 10.0.101.70:8301 alive client 1.10.0 2 full <default>
ip-10-0-102-175.ec2.internal 10.0.102.244:8301 alive client 1.10.0 2 full <default>
ip-10-0-103-240.ec2.internal 10.0.103.245:8301 alive client 1.10.0 2 full <default>
ip-10-0-3-223.ec2.internal 10.0.3.249:8301 alive client 1.10.0 2 full <default>
But if i execute:
# consul agent -datacenter=voip-full -config-dir=/etc/consul.d/ -log-file=log-file -advertise=$(wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4)
I get the following error:
==> Starting Consul agent...
Version: '1.10.1'
Node ID: 'f10070e7-9910-06c7-0e12-6edb6cc4c9b9'
Node name: 'ip-10-0-3-223.ec2.internal'
Datacenter: 'voip-full' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 10.0.3.223 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2021-08-16T18:23:06.936Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2021-08-16T18:23:06.936Z [WARN] agent: Node name "ip-10-0-3-223.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-08-16T18:23:06.946Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2021-08-16T18:23:06.947Z [WARN] agent.auto_config: Node name "ip-10-0-3-223.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-08-16T18:23:06.948Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: ip-10-0-3-223.ec2.internal 10.0.3.223
2021-08-16T18:23:06.948Z [INFO] agent.router: Initializing LAN area manager
2021-08-16T18:23:06.950Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp
2021-08-16T18:23:06.950Z [WARN] agent.client.serf.lan: serf: Failed to re-join any previously known node
2021-08-16T18:23:06.950Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp
2021-08-16T18:23:06.951Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
2021-08-16T18:23:06.951Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2021-08-16T18:23:06.953Z [INFO] agent: started state syncer
2021-08-16T18:23:06.953Z [INFO] agent: Consul agent running!
2021-08-16T18:23:06.953Z [WARN] agent.router.manager: No servers available
2021-08-16T18:23:06.954Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2021-08-16T18:23:34.169Z [WARN] agent.router.manager: No servers available
2021-08-16T18:23:34.169Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
So where to add the config?
I also tried adding a service in k8s pointing to the pod, but the service doesn't come up on consul's UI...
What do you guys recommend?
Thanks
Consul knows where these services are located because each service
registers with its local Consul client. Operators can register
services manually, configuration management tools can register
services when they are deployed, or container orchestration platforms
can register services automatically via integrations.
if you planning to use manual option you have to register the service into the consul.
Something like
echo '{
"service": {
"name": "web",
"tags": [
"rails"
],
"port": 80
}
}' > ./consul.d/web.json
You can find the good example at : https://thenewstack.io/implementing-service-discovery-of-microservices-with-consul/
Also this is a very nice document for having detailed configuration of the health check and service discovery : https://cloud.spring.io/spring-cloud-consul/multi/multi_spring-cloud-consul-discovery.html
Official document : https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery
BTW, I was finally able to figure out the issue.
consul-dns is not deployed by default, i had to manually deploy it, then forward all .consul requests from coredns to consul-dns.
All is working now. Thanks!
With my team we are trying to move our micro-services to openj9, they are running on kubernetes. However, we encounter a problem on the configuration of JMX. (openjdk8-openj9)
We have a connection refused when we try a connection with jvisualvm (and a port-forwarding with Kubernetes).
We haven't changed our configuration, except for switching from Hotspot to OpenJ9.
The error :
E0312 17:09:46.286374 17160 portforward.go:400] an error occurred forwarding 1099 -> 1099: error forwarding port 1099 to pod XXXXXXX, uid : exit status 1: 2020/03/12 16:09:45 socat[31284] E connect(5, AF=2 127.0.0.1:1099, 16): Connection refused
The java options that we use :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
We are using the last adoptopenjdk/openjdk8-openj9 docker image.
Do you have any ideas?
Thank you !
Regards.
I managed to figure out why it wasn't working.
It turns out that to pass the JMX options to the service we were using the Kubernetes service descriptor in YAML. It looks like this:
- name: _JAVA_OPTIONS
value: -Dzipkinserver.listOfServers=http://zipkin:9411 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099
I realized that the JMX properties were not taken into account from _JAVA_OPTIONS when the application is not launch with ENTRYPOINT in the docker container.
So I pass the properties directly into the Dockerfile like this and it works.
CMD ["java", "-Dcom.sun.management.jmxremote", "-Dcom.sun.management.jmxremote.authenticate=false", "-Dcom.sun.management.jmxremote.ssl=false", "-Dcom.sun.management.jmxremote.local.only=false", "-Dcom.sun.management.jmxremote.port=1099", "-Dcom.sun.management.jmxremote.rmi.port=1099", "-Djava.rmi.server.hostname=127.0.0.1", "-cp","app:app/lib/*","OurMainClass"]
It's also possible to keep _JAVA_OPTIONS and setup an ENTRYPOINT in the dockerfile.
Thanks!
I'm trying to configure a PostgresSQL datasource by following the
tutorial presented in the documentation:
http://www.keycloak.org/docs/1.9/server_installation_guide/topics/database.html
.
I'm doing the configuration for a standalone server by running
standalone.bat and by configuring everything in
keycloak-3.0.0.Final/standalone/configuration/standalone.xml .
I'm following the documentation step-by-step, but I get the following
error running standalone.bat every-time, and thus I'm unable to connect to the database.
15:33:11,684 ERROR [org.jboss.as.controller.management-operation]
(Controller Boot Thread) WFLYCTL0013: Operation ("add"
) failed - address: ([
("subsystem" => "datasources"),
("data-source" => "KeycloakDS")
]) - failure description: {"WFLYCTL0180: Services with
missing/unavailable dependencies" => [
"org.wildfly.data-source.KeycloakDS is missing
[jboss.jdbc-driver.postgres-driver]",
"jboss.driver-demander.java:jboss/datasources/KeycloakDS is missing
[jboss.jdbc-driver.postgres-driver]"
]}
15:33:11,687 ERROR [org.jboss.as.controller.management-operation]
(Controller Boot Thread) WFLYCTL0013: Operation ("add"
) failed - address: ([
("subsystem" => "datasources"),
("data-source" => "KeycloakDS")
]) - failure description: {"WFLYCTL0180: Services with
missing/unavailable dependencies" => [
"org.wildfly.data-source.KeycloakDS is missing
[jboss.jdbc-driver.postgres-driver]",
"jboss.driver-demander.java:jboss/datasources/KeycloakDS is missing
[jboss.jdbc-driver.postgres-driver]",
"org.wildfly.data-source.KeycloakDS is missing
[jboss.jdbc-driver.postgres-driver]"
I have tried different versions of JDBC drivers provided by:
https://jdbc.postgresql.org/, but to no avail, even by
using the version mentioned in the documentation.
At this point I am stuck since my configuration files look exactly like
in the documentation. What exactly could I be missing? Is the
documentation up to date?
you cant find the same question- http://lists.jboss.org/pipermail/keycloak-user/2017-March/010120.html,
but couldnt find any solution there,so re posting the same here.
I use jboss-cli.sh (or .bat if you're on Windows) to manage this for me. With Keycloak stopped and your PATH including $KEYCLOAK_HOME/bin I put the script below into a file (example: config_db.cli):
embed-server --server-config=standalone.xml --std-out=echo
batch
#
# remove the default provided datasource
#
/subsystem=datasources/data-source=KeycloakDS/:remove
#
# add it back using PostgreSQL
#
module add --name=org.postgres --resources=/path/to/your/postgresql-42.1.4.jar --dependencies=javax.api,javax.transaction.api
/subsystem=datasources/jdbc-driver=postgres:add(driver-name="postgres",driver-module-name="org.postgres",driver-class-name=org.postgresql.Driver)
/subsystem=datasources/data-source=KeycloakDS/:add(connection-url=jdbc:postgresql://localhost:5432/db_name,driver-name=postgres,jndi-name=java:jboss/datasources/KeycloakDS,password=db_password,user-name=db_user)
run-batch
And run with bin/jboss-cli.sh --file=config_db.cli
Note that this assumes you've already created a PostgreSQL database user (db_user above) with a password (db_password) and a database owned by the db_user (db_name).
In this way I can create my datasources the same way on my local machine as I do on my servers.
CORRECTIONS MADE: The format for running the jboss-cl.sh is:
bin/jboss-cli.sh --file=filename
I am attempting to port the Hyperledger Fabric Getting Started to Kubernetes. But am struggling to get peer1's to deploy. If I enable CORE_PEER_GOSSIP_BOOTSTRAP, I receive errors "Received AliveMessage from a peer with the same PKI-ID as myself".
How can I debug a peer reportedly having the same PKI-ID as another?
Using this as a starting point:
https://hyperledger-fabric.readthedocs.io/en/latest/getting_started.html
I am able to create:
orderer and cli pods in default namespace
peer0's one in each org1|org2 namespace.
peer1's but only if I disable (comment out) CORE_PEER_GOSSIP_BOOTSTRAP
If I enable CORE_PEER_GOSSIP_BOOTSTRAP for the peer1's, I receive the following warning and error:
[gossip/gossip#10.0.0.10:7051] NewGossipService -> WARN 01c External endpoint is empty, peer will not be accessible outside of its organization
...
[gossip/discovery#10.0.0.10:7051] handleAliveMessage -> ERRO 02a Bad configuration detected: Received AliveMessage from a peer with the same PKI-ID as myself: tag:EMPTY alive_msg:<membership:<pki_id:"[[REDACTED]]" > timestamp:<inc_number:1495468533769417608 seq_num:416 > >
In order to better map the Orderer, Peers to DNS names, I'm using Kubernetes Namespaces and this configuration:
OrdererOrgs:
- Name: Orderer
Domain: default.svc.cluster.local
Specs:
- Hostname: orderer
PeerOrgs:
- Name: Org1
Domain: org1.svc.cluster.local
Template:
Count: 2
Users:
Count: 2
- Name: Org2
Domain: org2.svc.cluster.local
Template:
Count: 2
Users:
Count: 2
In order to expose the peer0's to the other peers in the org and to expose the orderer, I have ClusterIP services for the peer0's (selecting only the peer0's) and orderer. It's inelegant but I'm trying to get it to work before I get it working more beautifully.
I am able to resolve orderer.default.svc.cluster.local, peer0.org1.svc.cluster.local, `peer0.org2.svc.cluster.local' using nslookup from within a pod deployed to default on the cluster.
Absent a curl-like tool for gPRC, I am able to open sockets against these endpoints on 7051 and 7053.
First, make sure you are using the right certificates.
Second, verify that your environment/configuration for gossip is set correctly
environment:
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer1.org1.example.com:8051
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051
- CORE_PEER_GOSSIP_ENDPOINT=peer0.org1.example.com:7051
OR in core.yaml
peer:
gossip:
bootstrap: peer0.org1.example.com:7051
externalEndpoint: peer1.org1.example.com:8051
endpoint: peer0.org1.example.com:7051
Edited: Also make sure that you have properly setup your CA
Hope this helps, it worked for me. And I was successfully able to connect peers.
If the peers are started from the same node, its possible that you are mounting the same crypto-material (path to mspconfig directory) for both the peers. If that is the case, separate the directory structures for both the peers and keep their respective certificates in them, update the respective paths for msp in docker-compose file and try to run.
I am new to jboss server. When I am trying to deploy .war file on server the following exception gets print on console:
6:38:04,388 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
16:38:05,642 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = Aerobay.war_com.mysql.jdbc.Driver_5_1
16:38:09,548 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#5f88823f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1292)
at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180)
at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590)
at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313)
at org.jboss.as.server.ServerService.boot(ServerService.java:384)
at org.jboss.as.server.ServerService.boot(ServerService.java:359)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271)
at java.lang.Thread.run(Thread.java:745)
Thanks in advance for the help !
I had the same problem when I tried to deploy the WAR file on my Red Hat Jboss EAP 7.0.
But the server was integrated into my IDE (Eclipse Neon) and the problem only occured in Debug-Modus.
I was able to solve the problem by removing all breakpoints and after that i started the server again.
Try increasing timeout by adding java option "blocking.timeout". You can do it in bin/standalone.conf.bat (depends on how you configure wildfly) by adding line:
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
Change the number if it's not enough.
increasing the timeout doesn't solve the root cause of the problem. You need to check the cause of the time of the block and solve the issue. Maybe in some cases the solution is to increase the timeout.
In most cases, increasing resources is a bad way to solve issues. I had this case, the Wildfly took a lot of time to boot. I increased the timeout to 600 and solved the issue but was still having issue with the wildfly booting time which was so annoying.
2018-03-26 07:50:36,523 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("path" => "xxxxxxxxxxxxxxxx")]'
Finally I checked the block cause in and found the block was due to network host resolving (NAS storage defined as a path in wildfly).
I jumped to the network setting and found that my local DNS was not set properly. I added the local DNS instead of the public DNS and the block issue was gone. Hope this helps
Regards
Sleem
When i tried to debug and started the server with debug mode got the following error:
16:19:50,096 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "ViprWeb.war")]'
16:19:50,096 ERROR [org.jboss.as.server] (management-handler-thread - 1) JBAS015870
16:20:00,117 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013413: Timeout after [5000] seconds waiting for service container stability while finalizing an operation.
I removed all my breakpoints and restarted my server jboss and it resolved the issue.
just increase time out in standalone.conf.bat
set as set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
It worked for me.
I had the same problem running a "dockerized" application locally - turns out increasing the resources fixed the issue. What I finally settled on:
CPUs: 4
Memory: 8GB
Swap: 2GB
Same problem, with netbeans
but I had not break points.
Running jboss by command line, helped me
Stop jboss
Close Netbeans
open command line
Go to jboss folder > bin >
type: standalone.bat (this starts jboss)
open Netbeans
worked fine!
Hope it'll help someone else.
I've been facing the same problem recently with WildFly 18 and 21, trying to run a WAR file containing JSR-352 batch jobs that worked fine on WildFly 14.
Increasing the timeout did not solve the situation, only prolonged the time before the TimeoutException was casted, no matter the value (e.g. 5, 10 or 20 minutes).
I've just found that to turn off microprofile-metrics-smallryesubsystem seems to be a possible solution.
After commenting out this line from the standalone.xml file, the war deploy was successful and much faster (about 2 minutes):
<subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="${wildfly.metrics.prefix:wildfly}"/>
I am having problem with keycloak server 15.0.2.
WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#410c55ac for operation add-deployer-chains at address [] failed
I am using mysql5.7 with jconnect8.0 jar.
I had the same problem. Then I killed the Kaspersky process and it helped!
I tackled a similar problem and had only succeed with undeploy the the apps. This gave a clean environment for Wildfly to restart and start the management and http-service. Then deploy the apps/WARs and identify what got you to this state.
In my case it was transactions that wanted to recover and deleting those from DB solve the problem bot to re-occur.