How can I centrally log my spring boot REST services which are running in different applications on the cloud foundry platform? For example I want to log how much a particular services is requested. The log should also be persistent even if I have to restart / reset my application. I don't only want to see the last log entries with cf logs --recent. Is there a best practise?
Your applications should configure their logging to write to stdout and stderr. The Cloud Foundry logging subsystem will automatically pick up everything written to stdout and stderr and send it to the log aggregator. See the Application Logging docs for more info.
To persist the logs and make them available for viewing and analysis, they should be streamed to an external log capture system. Some Cloud Foundry docs contain some general information about configuring log streaming and some specific instructions for some popular log capture systems.
Related
I am trying to implement a Rest client for Flink to send jobs via Restful Flink services. And also I want to integrate Flink and Kubernetes natively. I have decided to use “Application Mode” as deployment mode according to Flink documentation .
I have already implemented a job and packaged it as jar. And I have tested it on Standalone Flink. But my aim is to move on Kubernetes and deploy my application in Application mode via Rest API of Flink.
I have already investigated the samples at Flink documentation - Native Kubernetes. But I cannot find a sample for executing same samples via Restful services (esp. how to set --target kubernetes-application/kubernetes-session or other parameters).
In addition to samples, I checked out the Flink sources from GitHub and tried to find some sample implementation or get some clue.
I think the below ones are related with my case.
org.apache.flink.client.program.rest. RestClusterClient
org.apache.flink.kubernetes. KubernetesClusterDescriptorTest. testDeployApplicationCluster
But they are all so complicated for me to understand below points.
For application mode, are there any need to initialize a container to serve Flink Rest services before submitting job? If so, is it JobManager?
For application mode, how can I set the same command line parameters via Rest services?
For session mode, in command line samples, kubernetes-session.sh is executed before job submission to initialize a JobManager container. How sould I do this step via Rest client?
For session mode, how can I set the same command line parameters via Rest services? Although the command line samples send .jar job as parameter, should I upload jar before submitting job?
Could you please provide me some clue/sample to continue my implementation?
Best regards,
Burcu
I suspect that if you study the implementation of the Apache Flink Kubernetes Operator you'll find some clues.
I'm designing a distributed application, comprised of several Spring microservices that will be deployed with Kubernetes. It is a batch processing app, and a typical request could take several minutes of processing, with the processing getting distributed across the services, using Kafka as a message broker.
A requirement of the project is that each request will generate a log file, which will need to be stored on the application file store for retrieval. The current design is, all the processing services write log messages (with the associated unique request ID) to Kafka, and there is a dedicated logging microservice that reads these messages down, does some formatting and should persist them to the log file associated with the given request ID.
I'm very unfamiliar with how files should be stored in web applications. Should I be storing these log files to the local file system? If so, wouldn't that mean this "logging service" couldn't be scaled? For example, if I scaled the log service to 2 instances, then each instance would only have access to half of the log files in theory. And if a user makes a request to retrieve a log file, there is no guarantee that the requested log file will be at whatever log service instance the Kubernetes load balancer routed them too.
What is the currently accepted "best practice" for having a file system in a distributed application? Or should I just accept that the logging service can never be scaled up?
A possible solution I can think of would just store the text log files in our MySQL database as TEXT rows, making the logging service effectively stateless. If someone could point out any potential issues with this that would be much appreciated?
deployed with Kubernetes
each request will generate a log file, which will need to be stored on the application file store
Don't do this. Use a Fluentd / Filebeat / promtail / Splunk forwarder side car that gathers stdout from the container processes.
Or have your services write to a kafka logs topic rather than create files.
With either option, use a collector like Elasticsearch, Grafana Loki, or Splunk
https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent
wouldn't that mean this "logging service" couldn't be scaled?
No, each of these services are designed to be scaled
possible solution I can think of would just store the text log files in our MySQL database as TEXT rows,
Sure, but Elasticsearch or Solr are purpose-built for gathering and searching plaintext, not MySQL.
Don't treat logs as something application specific. In other words, your solution shouldn't be unique to Spring
I am investigating options for monitoring our installation in Swisscom's cloud-foundry. My objectives are the following:
monitor performance indicators for deployed application (such as cpu, disk, memory)
monitor performance indicators for services (slow queries, number of queries, ideally also some metrics on hitting quotas)
So far, I understand the options are the following (including some BUTs):
I used a very nice TOP cf-plugin (github)
This works very well. It seems that it registers itself to get the required firehose nozzles and consume data.
That is very useful for tracing / ad-hoc monitoring, but not very good for a serious infrastructure monitoring.
Another way I found is to use firehose-syslog solution.
This can be deployed as an app to (as far as I understand) do the job in similar way, as the TOP cf plugin.
The problem is, that it requires registered client, so it can authenticate with the doppler endpoint. For some reason, the top-cf-plugin does that automatically / in another way.
Last option i am considering is to build the monitoring itself to the App (using a special buildpack)
That can be for example done with Datadog. But it seems to also require a dedicated uaa client to register the Nozzle.
I would like to check, if somebody is (was) on the similar road, has some findings.
Eventually I would like to raise the following questions towards the swisscom community support:
is it possible to register uaac client to be able to ingest events through the firehose nozzle from external service? (this requires admin credentials if I was reading correctly)
is there an alternative way to authenticate with the nozzle (for example using a special user and his authentication token?)
is there any alternative to monitor the CF deployments in Swisscom? Eventually, is there a paper, blogpost or other form of documentation, that would be helpful in this respect (also for other users of AppCloud)?
Since it requires admin permissions, we can not give out UAA clients for the firehose.
However, there are different ways to get metrics in context of a user.
CF API
You can obtain basic metrics of a specific app by polling the CF API:
https://apidocs.cloudfoundry.org/5.0.0/apps/get_detailed_stats_for_a_started_app.html
However, since you have to poll (and for each app), it's not the recommended way.
Metrics in syslog drain
CF allows devs to forward their logs to syslog drains; in more recent versions, CF also sends metrics to this syslog drain (see https://docs.cloudfoundry.org/devguide/deploy-apps/streaming-logs.html#container-metrics).
For example, you could use Swisscom's Elasticsearch service to store these metrics and then analyze it using Kibana.
Metrics using loggregator (firehose)
The firehose allows streaming logs to clients for two types of roles:
Streaming all logs to admins (which requires a UAA client with admin permissions) and streaming app logs and metrics to devs with permissions in the app's space. This is also what the cf logs command uses. cf top also works this way (it enumerates all apps and streams the logs of each app).
However, you will find out that most open source tools that leverage the firehose only work in admin mode, since they're written for the platform operator.
Of course you also have the possibility to monitor your app by instrumenting it (white box approach), for example by configuring Spring actuator in a Spring boot app or by including an agent of your favourite APM vendor (Dynatrace, AppDynamics, ...)
I guess this is the most common approach; we've seen a lot of teams having success by instrumenting their applications. Especially since advanced monitoring anyway requires you to create your own metrics as the firehose provided cpu/memory metrics are not that powerful in a microservice world.
However, option 2. would be worth a try as well, especially since the ELK's stack metric support is getting better and better.
Is there a way to get some notification when a Cloud Foundry application fails or is unreachable? I mean to register to some deployed app and if the status of the application is changed to failed or something, I want to receive a notification.
On Pivotal Cloud Foundry, when a app crashes, an event is emitted thru the firehose.
PCF Metrics tile, available from Pivotal, can be deployed to your PCF foudnation. PCF Metrics will track all events for apps running on the foundation and are accessible to developers (thru Apps Manager). I believe Metrics tile tracks history for up to two weeks. I am not aware of any alerting capabilities in the PCF Metrics tile (I could be wrong, in which case, please correct me), that will prompt you when an app crashes.
Other approaches are to implement event logging tools like Splunk, New Relic etc. They support alerts. You will have to build those.
API monitoring tools like AppD, Apigee, and New Relic provide alerting and can notify you went the response time to an app has degraded (as in your app has crashed). This approach is a little more involved. You may require to add an agent to your buildpack, depending on the tool you choose.
IMHO there is no such built-in feature for Cloud Foundry, but IBM Cloud offers the Availability Monitoring service to monitor apps and send out alerts in case of unavailability or other similar events. The service is part of the DevOps category in the IBM Cloud catalog.
There is also Alert Notification to manage alerts, the notification of the right groups via all kinds of channels and to track the alert status. For your question you should start with the Availability Monitoring and then work towards how those events are handled.
You can use the cf events appname command to get a list of all events about the application, this will print out all the recent events such as application crashes.
if run the cf events appname -v you will see the json rest calls the cf cli makes to Cloud Foundry.
You can use Cloud Foundry Java Client to write you own code to interact with Cloud Foundry.
Another thing you can do is stream your application logs to any syslog compatible log aggregation service for example splunk. Then have splunk monitor for app crash events in the log. You can read how to configure app log streaming at the docs
This functionality is scheduled to be available with PCF Metrics 1.5 and can be seen with PWS (Pivotal Web Services) in Alpha Mode.
The functionality is available under the Monitors Tab inside of PCF Metrics (1.5).
Webhook notifications (i.e. Slack) can be configured for a number of Events (including as you discussed crashes).
You can create a User Provided service and Add a syslog drain URL. And then bind the service to your application. Now in case of any events happening it will put the logs into the URL you have provided.
I have app that uses several Bluemix services including database service.
Is it possible to have access to all available operational info via single dashboard or I need inspect all components' logs separately?
-Thanks in advance
Service log files are not available. You might consider looking at the Monitoring and Analytics service to see if will meet your needs. There is both a free and a diagnostics plan available.