I want to configure Service Fabric to keep rolling fixed number of logs of specific size Y on disk.
For example, keep a maximum of 5 log files on disk with a file size not to exceed 5MB, file #1 is replaced when the last file meets the rollover criteria.
I tried following links
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-actors-reliabledictionarystateprovider-configuration
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-configuration
but I still can not find info I am looking for. I also tried following setting but with this, I can not specify number of files.
<Parameter Name="MaxDiskQuotaInMB" Value="10" />
<Parameter Name="DataDeletionAgeInDays" Value="3" />
Thank you for help.
Starting in version 5.4 (2.4 SDK), the local development clusters will use rolling file traces similar to what you described. The size of the log is fixed and a single file though.
This isn't supported in production clusters due to its performance impact.
Related
There's two parts to this question. First, what falls under the purview of the Diagnostics---MaxDiskQuotaInMB configuration? Is it everything under SvcFab/Log? Just SvcFab/Log/AppInstanceData/? Having more info on this would be nice.
Second, what is the proper course of action if the FabricDCA.exe is running but the SvcFab/Log and SvcFab/Log/AppInstanceData/ folders exceed the limits we've set on their size? My team set them to 10,000 MB, but SvcFab/Log regularly takes up 12-16 GB.
The cluster configuration on Azure recognizes the change to the MaxDiskQuotaInMB configuration but there seems to be no impact on the node itself. I've tried resetting FabricDCA.exe as well and so far it has not helped either (after several hours).
One node in our cluster had so much space taken up by logs (over our limit) that remaining storage space was reduced to 1 MB.
Posting a more complete answer since it may be helpful to other people.
Most of the things under SvcFab/Log folder should fall under the quota set by MaxDiskQuotaInMB. There are a few things that may not, but the majority of things that usually take disk space are included. Keep in mind also that the task cleaning the disk usually runs every 5 minutes so you may see usage go over the quota within this timeframe.
If FabricDCA.exe is not properly cleaning files from this folder it is possible that you are hitting a bug in .Net runtime where all system.threading.timers stop firing and the disk to not be cleaned because FabricDCA relies on these timers to do so.
This is the bug on the .NET core side tracking the issue: (https://github.com/dotnet/coreclr/issues/26771). It seems to happen when the machine is running out of memory intermittently.
There is an auto-mitigation added in FabricDCA in Service Fabric 7.0.
The manual mitigation is usually to kill FabricDCA.exe process.
The process should start again and after a few minutes it will start cleaning again.
You mentioned that you already tried killing FabricDCA.exe so maybe the solution above does not work for you. In this case, try taking a look at the Service Fabric cluster manifest directly, it might be the case where your new configurations seem to be accepted by the ARM template deployment but the new configuration doesn't reach the cluster manifest which is the source of truth in this case.
Update:
There was a regression introduced as part of the auto-mitigation above which caused The AppInstanceFolder to fill up the disk. This is fixed in SF version 7.0.466
In the Open shift i have 4 projects and 25Gb of space allocated to the projects.
And db i use is Mongo Db(3.2 Version).
So in openshift iam getting the message has Quota limit reached and if i check all the 25 GB has been used as per openshift
But in Mongo db if i check using db.stats() for all the projects i have used 5.7GB
I want to know where the remaining space is used Or how to find exact space that i am using.
I think you’d like to do double checks about your resource issues.
check what resource limit was reached, is it a storage?
you should check the event logs which provide more details.
check what quota limits were configured your cluster or project.
have you been experienced some troubles after the showing the messages? Such as db hanging up, no response from pod and so on.
They are just troubleshooting guides, but i hope it help you.
I have deployed my dockerized micro services in AWS server using Elastic Beanstalk which is written using Akka-HTTP(https://github.com/theiterators/akka-http-microservice) and Scala.
I have allocated 512mb memory size for each docker and performance problems. I have noticed that the CPU usage increased when server getting more number of requests(like 20%, 23%, 45%...) & depends on load, then it automatically came down to the normal state (0.88%). But Memory usage keeps on increasing for every request and it failed to release unused memory even after CPU usage came to the normal stage and it reached 100% and docker killed by itself and restarted again.
I have also enabled auto scaling feature in EB to handle a huge number of requests. So it created another duplicate instance only after CPU usage of the running instance is reached its maximum.
How can I setup auto-scaling to create another instance once memory usage is reached its maximum limit(i.e 500mb out of 512mb)?
Please provide us a solution/way to resolve these problems as soon as possible as it is a very critical problem for us?
CloudWatch doesn't natively report memory statistics. But there are some scripts that Amazon provides (usually just referred to as the "CloudWatch Monitoring Scripts for Linux) that will get the statistics into CloudWatch so you can use those metrics to build a scaling policy.
The Elastic Beanstalk documentation provides some information on installing the scripts on the Linux platform at http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html.
However, this will come with another caveat in that you cannot use the native Docker deployment JSON as it won't pick up the .ebextensions folder (see Where to put ebextensions config in AWS Elastic Beanstalk Docker deploy with dockerrun source bundle?). The solution here would be to create a zip of your application that includes the JSON file and .ebextensions folder and use that as the deployment artifact.
There is also one thing I am unclear on and that is if these metrics will be available to choose from under the Configuration -> Scaling section of the application. You may need to create another .ebextensions config file to set the custom metric such as:
option_settings:
aws:elasticbeanstalk:customoption:
BreachDuration: 3
LowerBreachScaleIncrement: -1
MeasureName: MemoryUtilization
Period: 60
Statistic: Average
Threshold: 90
UpperBreachScaleIncrement: 2
Now, even if this works, if the application will not lower its memory usage after scaling and load goes down then the scaling policy would just continue to trigger and reach max instances eventually.
I'd first see if you can get some garbage collection statistics for the JVM and maybe tune the JVM to do garbage collection more often to help bring memory down faster after application load goes down.
I have cluster running in Azure.
I have multiple gigabytes of log data under D:\SvcFab\Log\Traces. Is there way to control amount of trace data that is collected/stored? Will the logs grow indefinitely?
Also the D:\SvcFab\ReplicatorLog has 8GB of preallocated data as specified by SharedLogSizeInMB parameter (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-configuration). How can I change this setting in Azure cluster or should it always be kept default?
For Azure clusters the SvcFab\Log folder will grow up to 5GB. It will also shrink if detects your disk is running out of space (<1GB). There are no controls for this in Azure.
This may be old but if you still have this issue, the solution for this is to add parameter in arm template for service fabric cluster .. there are some other ways to do this but this one is the most guaranteed one
https://techcommunity.microsoft.com/t5/azure-paas-developer-blog/reduce-log-size-on-service-fabric-node/ba-p/1017493
We are using wildfly and in our wildflyhome/standalone/log directory it's filling up with logs and eventually running out of disc space. I would like to set up rolling logs and know that it's possible but just don't know how to do it. Any help would be appreciated.
If you're on linux just setup a cron job to purge the files.
If you want to use a size rotation that will only keep as many rotations as you tell it you can use a size-rotating-file-handler.
The following CLI command will replace the default periodic-rotating-file-handler with a size-rotating-file-handler which rotates when the log file reaches 50MB and will only keep 10 rotations.
batch
/subsystem=logging/root-logger=ROOT:remove-handler(name=FILE)
/subsystem=logging/periodic-rotating-file-handler=FILE:remove
/subsystem=logging/size-rotating-file-handler=FILE:add(append=true, autoflush=true, named-formatter=PATTERN, max-backup-index=10, rotate-size=50m, file={relative-to=jboss.server.log.dir, path=server.log})
/subsystem=logging/root-logger=ROOT:add-handler(name=FILE)
run-batch