I am now using Alibaba Canal to sync MySQL from datacenter A to datacenter B(the canal deploy in kubernetes), after I start the canal-server, shows error like this:
[root#canal-server-stable-0 bin]# tail -f /home/canal/logs/canal/canal.log
2021-05-26 11:47:32.329 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-05-26 11:47:32.366 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-05-26 11:47:32.849 [main] ERROR com.alibaba.otter.canal.deployer.CanalLauncher - ## Something goes wrong when starting up the canal Server:
com.alibaba.otter.canal.common.CanalException: load manager config failed.
Caused by: com.alibaba.otter.canal.common.CanalException: requestGet for canal config error: auth :admin is failed
2021-05-26 11:52:50.402 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-05-26 11:52:50.432 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-05-26 11:52:50.836 [main] ERROR com.alibaba.otter.canal.deployer.CanalLauncher - ## Something goes wrong when starting up the canal Server:
com.alibaba.otter.canal.common.CanalException: load manager config failed.
Caused by: com.alibaba.otter.canal.common.CanalException: requestGet for canal config error: auth :admin is failed
this is my canal server config:
[root#canal-server-stable-0 bin]# cat ../conf/canal.properties
# register ip
# canal.register.ip = canal-server-stable-0.canal-server-discovery-svc-stable.hades-pro.svc.cluster.local
canal.register.ip = 10.244.5.5
# canal admin config
canal.admin.manager = 10.105.49.36:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 6bb4837eb74329105ee4568dda7dc67ed2ca2ad9
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster = online
the hash password was encrypt from 123456. I am sure the password is right. I tried to find the password in database, it matched with my config:
I also using Arthas to trace the online app of canal-admin:
watch com.alibaba.otter.canal.admin.controller.PollingConfigController auth "{params,returnObj}" -x 3 -b
shows the password I pass is: 6bb4837eb74329105ee4568dda7dc67ed2ca2ad9. I did not know where is going wrong now, what should I do to fix it?
you can check canal admin conf/application.yaml file
canal:
adminUser: admin
adminPasswd: 123456
if you modified "canal.adminPasswd" attribute, you can modified it correct.
hope, help you.
Related
Recently i've been experimenting with deploying Statefull applications onto Kubernetes. For my dev environment everything is on-premise, either on my local machine or on remote VMs. I deployed OpenSearch through its helm chart, got it and dashboards up and running, and everything was going well. I am now trying to setup data-prepper running as a docker container on my local machine (the kubernetes cluster is on remote VMs, not sure if this matters). I have the kube service that defines access to OpenSearch port-forwarded to my machine and am able to access it using "curl -u : https://localhost:9200 -k". Since my only interest is seeing it up and running I don't care (yet) that it is insecure. When I setup my data-prepper pipeline to hit OpenSearch in the exact same way, it is refusing the connection and I'm at a loss as to why.
pipelines.yaml:
simple-sample-pipeline:
workers: 2
delay: "5000"
source:
random:
sink:
- opensearch:
hosts: [ "https://localhost:9200" ]
insecure: true
username: <user>
password: <admin>
index: test
data-prepper-config.yaml
ssl: false
Docker command to run container:
docker run --name data-prepper \
-v C:/users/<profile>/documents/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml \
-v C:/users/<profile>/documents/data-prepper.yaml:/usr/share/data-prepper/data-prepper-config.yaml \
opensearchproject/data-prepper:latest
logs exerpt:
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-06-07T19:39:50,959 [main] INFO com.amazon.dataprepper.parser.config.DataPrepperAppConfiguration - Command line args: /usr/share/data-prepper/pipelines.yaml,/usr/share/data-prepper/data-prepper-config.yaml
2022-06-07T19:39:50,960 [main] INFO com.amazon.dataprepper.parser.config.DataPrepperArgs - Using /usr/share/data-prepper/pipelines.yaml configuration file
2022-06-07T19:39:54,599 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building pipeline [simple-sample-pipeline] from provided configuration
2022-06-07T19:39:54,600 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building [random] as source component for the pipeline [simple-sample-pipeline]
2022-06-07T19:39:54,624 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building buffer for the pipeline [simple-sample-pipeline]
2022-06-07T19:39:54,634 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building processors for the pipeline [simple-sample-pipeline]
2022-06-07T19:39:54,635 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building sinks for the pipeline [simple-sample-pipeline]
2022-06-07T19:39:54,635 [main] INFO com.amazon.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
2022-06-07T19:39:54,643 [main] INFO com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2022-06-07T19:39:54,649 [main] INFO com.amazon.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2022-06-07T19:39:54,789 [main] INFO com.amazon.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2022-06-07T19:39:54,881 [main] ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink
java.lang.reflect.InvocationTargetException: null
-----
Caused by: java.net.ConnectException: Connection refused
I have installed Kafka single-node using Confluent. There is an error in Kafka pod :
[WARN] 2022-04-26 14:29:47,008 [main-SendThread(zookeeper.confluent.svc.cluster.local:2181)] org.apache.zookeeper.ClientCnxn run - Session 0x0 for sever zookeeper.confluent.svc.cluster.local:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper.confluent.svc.cluster.local:2181 because it's not resolvable
at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
[INFO] 2022-04-26 14:29:47,273 [main] kafka.zookeeper.ZooKeeperClient info - [ZooKeeperClient Kafka server] Closing.
[ERROR] 2022-04-26 14:29:48,112 [main-SendThread(zookeeper.confluent.svc.cluster.local:2181)] org.apache.zookeeper.client.StaticHostProvider resolve - Unable to resolve address: zookeeper.confluent.svc.cluster.local:2181
java.net.UnknownHostException: zookeeper.confluent.svc.cluster.local
Error messages :
Unable to canonicalize address zookeeper.confluent.svc.cluster.local:2181 because it's not resolvable
Unable to resolve address: zookeeper.confluent.svc.cluster.local:2181
I checked my zookeper ... it's good and works without a problem. Also, check DNS using dnsutils :
$ kubectl -n default exec -it dnsutils -- nslookup zookeeper.confluent.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: zookeeper.confluent.svc.cluster.local
Address: 192.168.0.111
What can I do? Is this a k8s related problem?
This happens with me, but on docker-compose project
finally, I found that no space left on server causes this issue.
clean some spaces and it worked.
I have configured my multi-cluster Hazelcast server on Kubernetes via the Kubernetes API discovery strategy. (Please see Two separate hazelcast clusters in kubernetes) And the members of each cluster are successfully discovering each other.
My client project is running on the k8s cluster as my Hazelcast server.
I have added the following dependency to my client project pom:
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-kubernetes</artifactId>
<version>1.3.1</version>
</dependency>
I have configured my Hazelcast client as given in the official documentation:
clientConfig.getNetworkConfig().getKubernetesConfig()
.setEnabled(true)
.setProperty("namespace", "default")
.setProperty("service-name", "xyz");
(I have a namespace called "default" and k8s service object named "xyz")
These are the logs on client startup. Although it recognized the Hazelcast server pod, it gave an AuthenticationException (as expanded below). Also, want to point out that it did not try to connect to the correct port.
2019-09-18 12:59:36,699 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.client.HazelcastClient (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] A non-empty group password is configured for the Hazelcast client. Starting with Hazelcast version 3.11, clients with the same group name, but with different group passwords (that do not use authentication) will be accepted to a cluster. The group password configuration will be removed completely in a future release.
2019-09-18 12:59:36,709 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.core.LifecycleService (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] HazelcastClient 3.11.1 (20181218 - d294f31) is STARTING
2019-09-18 12:59:36,977 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.spi.discovery.integration.DiscoveryService (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: xyz, service-port: 0, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
2019-09-18 12:59:36,980 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.spi.discovery.integration.DiscoveryService (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] Kubernetes Discovery activated resolver: ServiceEndpointResolver
2019-09-18 12:59:36,999 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.client.spi.ClientInvocationService (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] Running with 2 response threads
2019-09-18 12:59:37,060 [instance=local-service_01.devciny-dock] [localhost-startStop-1] INFO com.hazelcast.core.LifecycleService (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] HazelcastClient 3.11.1 (20181218 - d294f31) is STARTED
2019-09-18 12:59:37,390 [instance=local-service_01.devciny-dock] [local-service_01.devciny-dock.cluster-] INFO com.hazelcast.client.connection.ClientConnectionManager (Slf4jFactory.java:65) - local-service_01.devciny-dock [instance_identifier] [3.11.1] Trying to connect to [10.42.1.111]:5701 as owner member
2019-09-18 12:59:37,432 [instance=local-service_01.devciny-dock] [local-service_01.devciny-dock.internal-3] WARN com.hazelcast.client.connection.nio.ClientConnection (Slf4jFactory.java:67) - local-service_01.devciny-dock [instance_identifier] [3.11.1] ClientConnection{alive=false, connectionId=1, channel=NioChannel{/10.42.1.121:39003->/10.42.1.111:5701}, remoteEndpoint=null, lastReadTime=2019-09-18 12:59:37.426, lastWriteTime=2019-09-18 12:59:37.425, closedTime=2019-09-18 12:59:37.431, connected server version=null} closed. Reason: com.hazelcast.client.AuthenticationException[Invalid credentials! Principal: null]
com.hazelcast.client.AuthenticationException: Invalid credentials! Principal: null
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$AuthCallback.onResponse(ClientConnectionManagerImpl.java:747)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$AuthCallback.onResponse(ClientConnectionManagerImpl.java:702)
at com.hazelcast.client.spi.impl.ClientInvocationFuture$InternalDelegatingExecutionCallback.onResponse(ClientInvocationFuture.java:130)
at com.hazelcast.client.spi.impl.ClientInvocationFuture$InternalDelegatingExecutionCallback.onResponse(ClientInvocationFuture.java:118)
at com.hazelcast.client.spi.impl.ClientInvocationFuture$InternalDelegatingExecutionCallback.onResponse(ClientInvocationFuture.java:130)
at com.hazelcast.client.spi.impl.ClientInvocationFuture$InternalDelegatingExecutionCallback.onResponse(ClientInvocationFuture.java:118)
at com.hazelcast.spi.impl.AbstractInvocationFuture$1.run(AbstractInvocationFuture.java:255)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Your Hazelcast client tries to connect to 10.42.1.111:5701 and it find Hazelcast server there. The port looks correct and it correctly finds the Hazelcast server there.
What happens next is that it cannot authenticate with the server, which probably means that you didn't specify the cluster password in your Hazelcast configuration. You can read more on how to do it in this StackOverflow question.
You didn't share the most important part of configuration related to client authentication. That is the group config of clusters and group config from your client. I anticipate, the problem is rooted there.
The default authentication compares group name on the member with username coming from the client. The username is filled by the client's group name (by default).
Check the AuthenticationBaseMessageTask code
private AuthenticationStatus authenticate(UsernamePasswordCredentials credentials) {
GroupConfig groupConfig = nodeEngine.getConfig().getGroupConfig();
String nodeGroupName = groupConfig.getName();
boolean usernameMatch = nodeGroupName.equals(credentials.getUsername());
return usernameMatch ? AuthenticationStatus.AUTHENTICATED : AuthenticationStatus.CREDENTIALS_FAILED;
}
I have a jar which runs fine on my host; specifically, when I run
java -jar myjar.jar
I get the expected output:
[2018-12-05 16:46:53.917] boot - 21252 INFO [main] --- Application: No active profile set, falling back to default profiles: default
[2018-12-05 16:47:00.855] boot - 21252 INFO [main] --- Application: Started Application in 8.176 seconds (JVM running for 9.106)
This is the Core Data Micro Service.
[2018-12-05 16:47:00.856] boot - 21252 INFO [main] --- Application: Registering to queue for events
[2018-12-05 16:47:00.857] boot - 21252 INFO [main] --- ZeroMQEventSubscriber: Getting subscriber, listening to tcp://localhost:5565
[2018-12-05 16:47:00.915] boot - 21252 INFO [main] --- ZeroMQEventSubscriber: Watching for new Event messages...
But then, I try to run the same jar inside a docker container. So I create the image like this:
FROM openjdk:8-jdk-alpine
COPY myjar.jar /opt/spring-cloud/lib/
ENTRYPOINT ["/usr/bin/java"]
CMD ["-jar", "/opt/spring-cloud/lib/myjar.jar"]
EXPOSE 48080
and run it:
sudo docker run [ID]
but this time, I get this exception from the container logs (this is only a part of the exception because it is too big, but I can show it all if needed):
[2018-12-07 08:30:31.447] boot - 1 INFO [main] --- Application: No active profile set, falling back to default profiles: default
[2018-12-07 08:32:35.423] boot - 1 ERROR [main] --- SpringApplication: Application startup failed
...
...
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'readingControllerImpl': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: org.edgexfoundry.dao.ValueDescriptorRepository org.edgexfoundry.controller.impl.ReadingControllerImpl.valDescRepos; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'valueDescriptorRepository': Invocation of init method failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 120000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=localhost:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 120000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=localhost:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
...
...
Caused by: com.mongodb.MongoTimeoutException: Timed out after 120000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=localhost:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.BaseCluster.getServer(BaseCluster.java:82)
at com.mongodb.DBTCPConnector.getServer(DBTCPConnector.java:664)
at com.mongodb.DBTCPConnector.access$500(DBTCPConnector.java:40)
at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:513)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:456)
at com.mongodb.DBTCPConnector.getPrimaryPort(DBTCPConnector.java:415)
at com.mongodb.DBCollectionImpl.createIndex(DBCollectionImpl.java:378)
at com.mongodb.DBCollection.createIndex(DBCollection.java:597)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.createIndex(MongoPersistentEntityIndexCreator.java:142)
... 57 more
Mongo has been started in another container through docker-compose (together with other services in other containers):
ps aux | grep mongo
root 16226 0.0 0.0 4340 768 ? Ss 10:27 0:00 /bin/sh -c /edgex/mongo/config/launch-edgex-mongo.sh
root 16292 0.0 0.0 4340 764 ? S 10:27 0:00 /bin/sh /edgex/mongo/config/launch-edgex-mongo.sh
root 16293 0.5 0.3 961168 61400 ? SLl 10:27 0:05 mongod --smallfiles
This is the docker-compose file:
version: '3'
services:
volume:
image: edgexfoundry/docker-edgex-volume:0.6.0
container_name: edgex-files
networks:
- edgex-network
volumes:
- db-data:/data/db
- log-data:/edgex/logs
- consul-config:/consul/config
- consul-data:/consul/data
mongo:
image: edgexfoundry/docker-edgex-mongo:0.6.0
ports:
- "27017:27017"
container_name: edgex-mongo
hostname: edgex-mongo
networks:
- edgex-network
volumes:
- db-data:/data/db
- log-data:/edgex/logs
- consul-config:/consul/config
- consul-data:/consul/data
depends_on:
- volume
.... more services...
networks:
edgex-network:
driver: "bridge
And the mongo db configuration properties:
spring.data.mongodb.username=core
spring.data.mongodb.password=password
spring.data.mongodb.database=coredata
#change to localhost when running locally during development
# (or set hosts to point edgex-mongo to the mongo host
spring.data.mongodb.host=localhost
#spring.data.mongodb.host=edgex-mongo
spring.data.mongodb.port=27017
spring.data.mongodb.connectTimeout=120000
spring.data.mongodb.socketTimeout=60000
spring.data.mongodb.maxWaitTime=120000
spring.data.mongodb.socketKeepAlive=true
Any ideas what may be going wrong?
There are two things going wrong here, first of all spring tries to connect to your mongodb on localhost, within docker this does not work since localhost references to the current container where of course no mongodb is available. To fix this you have to comment out this line and uncomment the next line which lists the host as edgex-mongo which corresponds with the hostname of your mongodb container, so spring knows to connect to that container.
However when you would do this you would run into the issue that it would not recognize edgex-mongo since it has no connection to this container. edgex-mongo is inside a bridged network which requires you to add the spring container to this network by using the following command:
docker run --network edgex--network [image]
I hope this helps you
I am trying to run Apache Atlas in a standalone fashion on Ubuntu - meaning without having to setup Solr and/or HBase.
What I did (according to the documentation: http://atlas.apache.org/0.8.1/InstallationSteps.html) was cloning the Git repository, build the maven project with embadded HBase an dSolr:
mvn clean package -Pdist,embedded-hbase-solr
Unpacked the resuting tar.gz file and executed bin/atlas_start.py - without having changed any configuration. To my understanding of the documentatino that should actually start up HBase along with Atlas - right?
The is what I find in logs/applocation.log:
2017-11-30 17:14:24,093 INFO - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216)
2017-11-30 17:14:24,093 INFO - [main:] ~ Server starting with TLS ? false on port 21000 (Atlas:217)
2017-11-30 17:14:24,093 INFO - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218)
2017-11-30 17:14:27,684 INFO - [main:] ~ No authentication method configured. Defaulting to simple authentication (LoginProcessor:102)
2017-11-30 17:14:28,527 INFO - [main:] ~ Logged in user daniel (auth:SIMPLE) (LoginProcessor:77)
2017-11-30 17:14:31,777 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)
2017-11-30 17:14:39,456 WARN - [main-SendThread(localhost:2181):] ~ Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (ClientCnxn$SendThread:110$
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-11-30 17:14:39,594 WARN - [main:] ~ Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti$
2017-11-30 17:14:40,593 WARN - [main-SendThread(localhost:2181):] ~ Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (ClientCnxn$SendThread:110$
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
...
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-11-30 17:14:56,185 WARN - [main:] ~ Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti$
2017-11-30 17:14:56,186 ERROR - [main:] ~ ZooKeeper exists failed after 4 attempts (RecoverableZooKeeper:277)
2017-11-30 17:14:56,186 WARN - [main:] ~ hconnection-0x1dba4e060x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid) (ZKUtil:544)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
To me it reads as if no HBase (and Zookeeper) are started by the script.
Am I missing something?
Thanks for your hints!
OK, meanwhile I figured out the issue. The start script obviously does not execute the script conf/atlas-env.sh which sets some environment variable. Among this are MANAGE_LOCAL_HBASE and MANAGE_LOCAL_SOLR. So if you set those two env vars to true (and set JAVA_HOME properly which is needed for the embedded HBase), then Atlas automatically starts HBase and Solr - and we get a local running instance of Atlas!
Maybe this helps someone who comes across the same issue in future!
Update March 2021
There are two ways of running apache atlas:
A) Building it from scratch:
git clone https://github.com/apache/atlas
mvn clean install -DskipTests
mvn clean package -Pdist -DskipTests
Running atlas_start.py:
python <atlas-directory>/conf/atlas_start.py
B) Using docker image:
docker-compose.yml
version: "3.3"
services:
atlas:
image: sburn/apache-atlas
container_name: atlas
ports:
- "21000:21000"
volumes:
- "./bash_script:/app"
command: bash -exc "/opt/apache-atlas-2.1.0/bin/atlas_start.py"
docker-compose up