According to the documentation there are two ways to send log information to the SwisscomDev ELK service.
Standard way via STDOUT: Every output to stdout is sent to Logstash
Directly send to Logstash
Asking about way 2. How is is this achieved, especially how is the input expected?
We're using Monolog in our PHP buildpack based application and using its stdout_handler is working fine.
I was trying the GelfHandler (connection refused), SyslogUdPHandler (no error, but no result), both configured to use VCAPServices logstashHost and logstashPort as API endpoint / host to send logs to.
Binding works, env variables are set, but I have no idea how to send SwisscomDev ELK service Logstash API endpoint compatible log information from our application.
Logstash is configured with a tcp input, which is reachable via logstashHost:logstashPort. The tcp input is configured with its default codec, which is the line codec (source code; not the plain codec as stated in the documentation).
The payload of the log event should be encoded in JSON so that the fields are automatically recognized by Elasticsearch. If this is the case, the whole log event is forwarded without further processing to Elasticsearch.
If the payload is not JSON, the whole log line will end up in the field message.
For your use case with Monolog, I suggest you to use the SocketHandler (pointing it to logstashHost:logstashPort) in combination with the LogstashFormatter which will take care of the JSON encoding with the log events being line delimited.
Related
I'm running a test Dgraph instance in a dgraph/standalone Docker container, using the github.com/dgraph-io/dgo/v200/protos/api API on port 9080 to write data, but can't see the changes in the Console on port 8000. Using the API to query the previously written data works fine, so I wonder if the API and the Console are somehow using different name spaces?
Are you committing the transaction? I have seen users complaining about this, but they forgot to commit the txn.
For a sample project for Weather service I needed to store time series data. This is the first time I am using any time-series database. I did some reading on those and their comparison and found that InfluxDB is open-source and is one of the best, so decided to use that.
For my PoC I installed it locally on my machine and connecting it from my application. However, when I see the logs for various queries run against InfluxDB, found that it makes http calls to InfluxDB and it passes the username and password in the Query params in URL. This certainly seems like bad practice to pass credentials as as plain text in the URL while making the http call. Can someone comment why is it designed like this and is it supposed to be like this in real world scenario as well?
Logs:
2019-07-19 12:01:00.304 INFO 69709 --- [pool-1-thread-1] okhttp3.OkHttpClient : --> POST http://127.0.0.1:8086/write?u=admin&p=admin&db=weatherdata&rp=defaultPolicy&precision=n&consistency=one (78-byte body)
2019-07-19 13:48:28.461 INFO 69709 --- [nio-8080-exec-9] okhttp3.OkHttpClient : --> GET http://127.0.0.1:8086/query?u=admin&p=admin&db=weatherdata&q=Select+*+from+weather
2019-07-19 13:48:28.530 INFO 69709 --- [nio-8080-exec-9] okhttp3.OkHttpClient : <-- 200 OK http://127.0.0.1:8086/query?u=admin&p=admin&db=weatherdata&q=Select+*+from+weather (68ms, unknown-length body)
InfluxDB supports HTTP Basic Auth where username and password are passed via HTTP auth headers instead of the URL. I suspect you just need to configure your client to do that instead of using the URL parameters. Credentials are still in plaintext, but I think if you set up HTTPS, Basic Auth is secure-ish.
In general I don't think the Influx Devs expect InfluxDB to be a standalone, public-facing service. Instead, they expect that you're going to use InfluxDB as a backend, and then use something like Chronograf (which is their own visualization tool) or Grafana as a front end. So if they're going to spend time on more sophisticated authentication protocols, they're going to do it on the front end side.
The expectation would be that the front end and back end run on the same network, and communications between them can be secured via network segmentation.
I want to call a Nifi custom processor as a REST Api and pass the parameters at run-time through pyspark. And retrieve the results in the response object.
Can anyone please help me in suggesting different approaches for the same.
use the following sequence of processors:
HandleHttpRequest
extract patameters
your other processors...
prepare response
HandleHttpResponse
The steps are:
Configure HandleHttpRequest processor.
Enable the required HTTP methods (GET, POST, DELETE, etc.).
Set the listening port.
Attached the Context Map to a service (the listener).
5. Enable the service and the processor.
Bonus:
If you run Nifi from a Docker container, as I do, you should get the container's IP:
docker inspect <container-name> --format='{{.NetworkSettings.IPAddress}}'
Now, you can run Postman, and the HandleHttpRequest processor will fetch it. For example:
I created a simple template to exemplify this scenario. The HTTP request's body is saved into a directory:
I have a Message Hub instance on Bluemix, and am able to produce / consume messages off it. I was looking for a quick, reasonable way to browse topics / messages to see what's going on. Something along the lines of kafka-topics-ui.
I installed kafka-topics-ui locally, but could not get it to connect to Message Hub. I used the kafka-rest-url value from the MessageHub credentials in the kafka-topics-ui configuration file (env.js), but could not figure out where to provide the API key.
Alternatively, in the Bluemix UI, under Kibana, I can see log entries for creating the topic. Unfortunately, I could not see log entries for messages in the topic (perhaps I'm looking the wrong place or have wrong filters?).
My guess is I'm missing something basic. Is there a way to either:
configure a tool such as kafka-topics-ui to connect to MessageHub,
or,
browse topic messages easily?
Cheers.
According to Using the Kafka REST API on Bluemix you need an additional header in all API requests:
-H "X-Auth-Token: APIKEY"
A quick solution is to edit the topic-ui code and include your token in every request. Another solution would be to use a Chrome plugin that can inject the above header. For a more formal solution, i have opened a ticket on github
Hello I would like to enquire if there is an API that can be used to retrieve Ganglia stats for all clients from a single ganglia server?
The Ganglia gmetad component listens on ports 8651 and 8652 by default and replies with XML metric data. The XML data type definition can be seen on GitHub here.
Gmetad needs to be configured to allow XML replies to be sent to specific hosts or all hosts. By default only localhost is allowed. This can be changed in /etc/ganglia/gmetad.conf.
Connecting to port 8651 will get you a default XML report of all metrics as a response.
Port 8652 is the interactive port which allows for customized queries. Gmetad will recognize raw text queries sent to this port, i.e. not HTTP requests.
Here are examples of some queries:
/?filter=summary (returns a summary of the whole grid, i.e. all clusters)
/clusterName (returns raw data of a cluster called "clusterName")
/clusterName/hostName (returns raw data for host "hostName" in cluster "clusterName")
/clusterName?filter=summary (returns a summary of only cluster "clusterName")
The ?filter=summary parameter changes the output to contain the sum of each metric value over all hosts. The number of hosts is also provided for each metric so that the mean value may be calculated.
Yes, there's an API for Ganglia: https://github.com/guardian/ganglia-api
You should check this presentation from 2012 Velocity Europe - it was really a great talk: http://www.guardian.co.uk/info/developer-blog/2012/oct/04/winning-the-metrics-battle
There is also an API you can install from pypi with 'pip install gangliarest' and sets up a configurable API backed with a Redis cache and indexer to improve performance.
https://pypi.python.org/pypi/gangliarest