Deploy Graylog on GKE - kubernetes

I'm having a hard time deploying Graylog on Google Kubernetes Engine, I'm using this configuration https://github.com/aliasmee/kubernetes-graylog-cluster with some minor modifications. My Graylog server is up but show this error in the interface:
Error message
Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.
Original Request
GET http://ES_IP:12900/system/sessions
Status code
undefined
Full error message
Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.
Graylog logs show nothing in particular other than this:
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
but at the end :
2019-01-16 13:35:00,255 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
Elastic search health check is green, no issues in ES nor Mongo logs.
I suspect a problem with the connection to Elastic Search though.
curl http://ip_address:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

After reading the tutorial you shared, I was able to identify that kubelet needs to run with the argument --allow-privileged.
"Elasticsearch pods need for an init-container to run in privileged mode, so it can set some VM options. For that to happen, the kubelet should be running with args --allow-privileged, otherwise the init-container will fail to run."
It's not possible to customize or modify kubelet parameter/arguments, there is a feature request found here: https://issuetracker.google.com/118428580, so this can be implemented in a future.
Also in case you are modifying kubelet directly on the node(s), it's possible that the master resets the configuration and it isn't guaranteed that the configurations will be persistent.

Related

How to register app from private repo in Spring Cloud dataflow 2.6.1

I'm using SCDF 2.6.1 in Openshift 3, and I'm facing error while registering the app, error log like below :
java.lang.NullPointerException: null
at org.springframework.cloud.dataflow.configuration.metadata.container.DefaultContainerImageMetadataResolver.getRegistryRequest(DefaultContainerImageMetadataResolver.java:162)
at org.springframework.cloud.dataflow.configuration.metadata.container.DefaultContainerImageMetadataResolver.getImageLabels(DefaultContainerImageMetadataResolver.java:110)
at org.springframework.cloud.dataflow.configuration.metadata.BootApplicationConfigurationMetadataResolver.resolvePortNamesFromContainerImage(BootApplicationConfigurationMetadataResolver.java:215)
at org.springframework.cloud.dataflow.configuration.metadata.BootApplicationConfigurationMetadataResolver.listPortNames(BootApplicationConfigurationMetadataResolver.java:163)
at org.springframework.cloud.dataflow.server.controller.AppRegistryController.getInfo(AppRegistryController.java:193)
at org.springframework.cloud.dataflow.server.controller.AppRegistryController.info(AppRegistryController.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
I checked the line of code in DefaultContainerImageMetadataResolver.java:162
// Convert the image name into a well-formed ContainerImage
ContainerImage containerImage = this.containerImageParser.parse(imageName);
// Find a registry configuration that matches the image's registry host
RegistryConfiguration registryConf = this.registryConfigurationMap.get(containerImage.getRegistryHost());
// Retrieve a registry authorizer that supports the configured authorization type.
RegistryAuthorizer registryAuthorizer = this.registryAuthorizerMap.get(registryConf.getAuthorizationType());
I'm pretty sure the error is because registryConf is null as result from
RegistryConfiguration registryConf = this.registryConfigurationMap.get(containerImage.getRegistryHost());
How to put my private repo URI in registryConfigurationMap ?
I have tried to put imagePullSecret in the deployment.yml which is registered with the private repo, but I think it doesn't work because in the startup log, I still see :
2020-09-03 04:55:24.111 INFO 1 --- [ main] urationMetadataResolverAutoConfiguration :
Final Registry Configurations: {registry-1.docker.io=RegistryConfiguration{registryHost='registry-1.docker.io', user='null', secret='****'', authorizationType=dockeroauth2, manifestMediaType='application/vnd.docker.distribution.manifest.v2+json', disableSslVerification='false',
extra={registryAuthUri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:{repository}:pull&offline_token=1&client_id=shell }}}
The only place where SCDF server downloads the container image layer is when it looks for app metadata.
Currently, this is configured to use the docker registry host (as this is where all the out-of-the-box applications are hosted).
If you want to override, you can modify these property values at the time of server startup and proceed.
Remember the fact that this configuration is only needed to download the app metadata layer of the image - not to download the entire container image at the SCDF server side.

Service Fabric Replica Stuck

I am upgrading an application on Service Fabric and one of the replicas is showing the following warning:
Unhealthy event: SourceId='System.RAP', Property='IStatefulServiceReplica.ChangeRole(S)Duration', HealthState='Warning', ConsiderWarningAsError=false.
The api IStatefulServiceReplica.ChangeRole(S) on node _gtmsf1_0 is stuck. Start Time (UTC): 2018-03-21 15:49:54.326.
After some debugging, I suspect I'm not properly honoring a cancellation token. In the meantime, how do I safely force a restart of this stuck replica to get the service working again?
Partial results of Get-ServiceFabricDeployedReplica:
...
ReplicaRole : ActiveSecondary
ReplicaStatus : Ready
ServiceTypeName : MarketServiceType
...
ServicePackageActivationId :
CodePackageName : Code
...
HostProcessId : 6180
ReconfigurationInformation : {
PreviousConfigurationRole : Primary
ReconfigurationPhase : Phase0
ReconfigurationType : SwapPrimary
ReconfigurationStartTimeUtc : 3/21/2018 3:49:54 PM
}
You might be able to pipe that directly to Restart-ServiceFabricReplica. If that remains stuck, then you should be able to use Get-ServiceFabricDeployedCodePackage and Restart-ServiceFabricDeployedCodePackage to restart the surrounding process. Since Restart-ServiceFabricDeployedCodePackage has options for selecting random packages to simulate failure, just be sure to target the specific code package you're interested in restarting.

Zoomdata error - Connect to localhost:3333 [localhost/127.0.0.1] failed: Connection refused

Good morning.
I am an intern currently working on a proof of concept requiring me to use zoomdata to visualize data from elasticsearch on ubuntu 16.04 LTS.
I managed to connect both on docker - in short, each process ran in a separate, isolated container and communicated through ports - but the performance wasn't good enough (the visualisation encountered bugs for files > 25 mb).
So I decided to install zoomdata on my computer (trial, .deb version)
I also installed mongodb first.
However, when I run zoomdata, I have two issues, and I believe that solving the second might solve the first:
-the first is that when I create a new elasticsearch connexion, I enter exactly the same parameters as with docker (I've triple checked, they are accurate):
node name: elasticsearch (the name of my node)
client type : HTTP node
adress: 192.168.10.4 and port: 9200
-the second is when I run zoomdata:
During the initialisation (more accurately, the spark one) I have an error message (more details below):
c.z.s.c.r.impl.CategoriesServiceClient : Attempt: 1 to send
categories registration failed. Connect to localhost:3333
[localhost/127.0.0.1] failed: Connection refused
followed by the same message over and over again (the attempt number changes) as the rest of the program executes normally.
I took a look at the logs - computer(bug) and docker(works).
Note: the SAML error doesn't stop the docker version from working, so unless fixing it fixes other problems It's not my priority.
Computer:
2016-06-15 16:04:06.680 ERROR 1 --- [ main]
c.z.core.service.impl.SamlService : Error initializing SAML.
Disabling SAML.
2016-06-15 15:58:12.125 INFO 8149 --- [ main]
com.zoomdata.dao.mongo.KeyValueDao : Inserting into key value
collection com.zoomdata.model.dto.SamlConfig#4f3431a1
2016-06-15 15:58:12.789 INFO 8149 --- [ main]
c.z.core.init.SystemVersionBeanFactory : Server version 2.2.6,
database version 2.2.6, git commit :
0403c951946e5daf03000d83ec45ad85d0ce4a56, built on 06060725
2016-06-15 15:58:17.571 ERROR 8149 --- [actory-thread-1]
c.z.s.c.r.impl.CategoriesServiceClient : Attempt: 1 to send
categories registration failed. Connect to localhost:3333
[localhost/127.0.0.1] failed: Connection refused
2016-06-15 15:58:17.776 ERROR 8149 --- [actory-thread-1]
c.z.s.c.r.impl.CategoriesServiceClient : Attempt: 2 to send
categories registration failed. Connect to localhost:3333
[localhost/127.0.0.1] failed: Connection refused
2016-06-15 15:58:18.537 INFO 8149 --- [ main]
c.z.c.s.impl.SparkContextManager$2 : Running Spark version 1.5.1
Docker:
2016-06-15 16:04:06.680 ERROR 1 --- [ main]
c.z.core.service.impl.SamlService : Error initializing SAML.
Disabling SAML.
2016-06-15 16:04:06.681 INFO 1 --- [ main]
com.zoomdata.dao.mongo.KeyValueDao : Inserting into key value
collection com.zoomdata.model.dto.SamlConfig#43b09564
2016-06-15
16:04:07.209 INFO 1 --- [ main]
c.z.core.init.SystemVersionBeanFactory : Server version 2.2.7,
database version 2.2.7, git commit :
97c9ba3c3662d1646b4e380e7640766674673039, built on 06131356
2016-06-15
16:04:12.117 INFO 1 --- [actory-thread-1]
c.z.s.c.r.impl.CategoriesServiceClient : Registered to handle
categories: [field-refresh, source-refresh]
2016-06-15 16:04:12.427 INFO 1 --- [ main]
c.z.c.s.impl.SparkContextManager$2 : Running Spark version 1.5.1
The server versions are different (2.2.6 for computer, 2.2.7 for docker) - I'll try to update the computer one but I don't have much hope that it will work.
What I tried already:
-use zoomdata as root: zoomdata refused, and internet told me why it was a bad idea to begin with.
-deactivate all firewalls : didn't do anything.
-search on the internet: didn't manage to solve.
I'm all out of ideas, and would be grateful for any assistance.
EDIT : I updated to 2.2.7, but the problem persists.
I'm wondering if there may be a problem with authorisations (since the connexion is refused).
I also tried disabling SSL on my zoomdata server.
EDIT: after a discussion on the logs and .properties files seen here
the issue is the scheduler service not connecting to mongodb, although zoomdata does connect.
I filled the files following the installation guide but to no avail.
From the logs I see that scheduler cannot connect to mongo because socket exception. Found possible solution here:
stackoverflow.com/questions/15419325/mongodb-insert-fails-due-to-socket-exception
Comment out the line in your mongod.conf that binds the IP to 127.0.0.1. Usually, it is set to 127.0.0.1 by default.
For Linux, this config file location should be be /etc/mongod.conf.
Once you comment that out , it will receive connections from all interfaces.
This fixed it for me as i was getting these socket exceptions as well.

Is CMS Replication required for ApplicationPool also?

Is CMS Replication required for ApplicationPool also?
When I run the command Get-CsManagementStoreReplicationStatus I get UpToDate : True for my domain but it comes False for my ApplicationPool.
UpToDate : True
ReplicaFqdn : ****.*****
LastStatusReport : 07-08-2014 11:42:26
LastUpdateCreation : 07-08-2014 11:42:26
ProductVersion : 5.0.8308.0
UpToDate : False
ReplicaFqdn : MyApplicationPool.****.*****
LastStatusReport :
LastUpdateCreation : 08-08-2014 15:16:03
ProductVersion :
UpToDate : False
ReplicaFqdn : ****.*****
LastStatusReport :
LastUpdateCreation : 08-08-2014 15:10:59
Am I on the right track? Have I created my ApplicationPool wrongly?
Yes, UCMA applications running on an app server generally require access to the CMS, so replication should be enabled.
On the app server, you'd need to:
Ensure the "Lync Server Replica Replicator Agent" service is running
Run Enable-CsReplica in the management shell
Run Enable-CsTopoloy
Then run Invoke-CSManagementStoreReplication to force a replication
I've noticed that it often takes a while for the CMS to be replicated to the app server, so you might need to run Get-CsManagementStoreReplicationStatus a few times before you see UpToDate change to True.

mongod crashed without logging

I'm using mongodb v2.2.2 on single server(Ubuntu 12.04).
It crashed with no log on /var/log/mongodb/mongodb.log.
It seemed crashed during logging.(Character is interrupted. And, this log is normal query log.)
And, I checked on syslog about memory-issue(for example, killed proccess),
but couldn't find it.
Then, I found the following error on mongo-shell(db.printCollectionStats() command).
DLLConnectionResultData
{
"ns" : "UserData.DLLConnectionResultData",
"count" : 8215398,
"size" : 4831306500,
"avgObjSize" : 588.0794211065611,
"errmsg" : "exception: assertion src/mongo/db/database.cpp:300",
"code" : 0,
"ok" : 0
}
How do I figure out problems?
Thank you,
I checked that line in the source code for 2.2.2 (see here for reference). That error is specifically related to enforcing quotas on MongoDB. You haven't mentioned enforcing quotas here or what you have set the files limit to (default is 8) but you could be running into the limit here.
First, I would recommend getting onto a more recent version of 2.2 (and upgrading to 2.4 eventually, but definitely 2.2.7+ initially). If you are using quotas, this fix which went into 2.2.5 would log quota exceeded messages (previously logged only at log level 1, default is log level 0). Hence if a quota violation is the culprit here, you may get an early warning.
If that is the root cause, then you have a couple of options:
After upgrading to the latest version of 2.2, of the issue happens repeatedly, file a bug report for the crash on 2.2
Upgrade to 2.4, verify that the issue still occurs, and file a bug (or add to the above report for 2.2)
In either case, turning off quotas in the interim would be the obvious way to prevent the crash.