I'm looking for a way to increase the Jobs - Standard Out pane cache for a fast loading experience.
It is rough to debug a 1500 lines output because when you scroll it at certain velocity it takes around 5-8 seconds to load the logs.
I have been checking the AWX documentation but couldn't find any related config.
Any suggestions?
Related
I've been tinkering with WebGPU and, when profiling the page on a couple different pages, I see that the script is marked as Idle. This makes it hard to estimate the actual rendering budget, as the time that the JS is executing is actually quite small but it looks much larger due to the Idle marking. Is there anyway to clear up this Idle when using WebGPU? Thanks!
I'm using server side rendering with Angular universal, and PM2 as the process manager, in a Digital Ocean droplet of 8 GB Memory / 80 GB Disk / Ubuntu 16.04.3 x64 / 4 vCPUs.
I use a 6GB swap file, and the available memory when "free -m" is the following:
total used free shared buff/cache available
Mem: 7983 1356 5290 16 1335 6278
Swap: 6143 88 6055
The ram used looks fine. There are 4 processes with Cluster Mode of PM2.
Every 6-8 hours, when the memory reaches ~88% in my Digital Ocean panel, the CPU goes very high, the web application does not respond correctly and PM2 has to restart the process, not sure for how long the web application does not work well.
Here is an image of what happens:
Performance is fine when working normally:
I think I'm missing some sort of configuration or something, since this happens always at the same periods of time.
EDIT1 So far I fixed some incompatibilities in my code (the app was working, but sometimes failed due to this), and added a memory limit in pm2 of 1GB. I'm not sure if this is the way to go since I'm a bit new to process management, but the CPU levels are fine now. Any comment is appreciated. I leave a picture of the current behaviour, every time one of the four processes reach 1GB, its restarted:
EDIT2 I add 3 more images, 2 showing top processes from Digital Ocean, and one showing Keymetrics status:
EDIT3 I figured out some memory leaks from my Angular app (I forgot to unsubscribe from a couple of subscriptions) and the system behaviour improved, but the memory line is still going up. I'll keep investigating about memory leaking in Angular and see if I've made some other mistakes:
It looks like your Angular Universal app is leaking memory, it should not increase over time as you observe but stay mostly flat.
You can try to find the memory leak (looks like you already found an issue and have a suspicion what else it could be).
Another thing you can try is periodically restart your app.
See here for example how to restart your pm2 process every couple of hours to reset and prevent the OOM situation that you've been running into.
In our (edge) case, the kubernetes healthcheck was the cause of the issue. The healthcheck accessed the main page by an internal IP. The page used the caller URL (in this case its IP) to load some resources which it couldn't find that way. This lead to an error and was somehow cached and slowly used up all memory. We had the same very linear rise in memory even during nights because of the regularity of the healthcheck.
We solved the problem by letting the healthcheck call "/health" where we only return a 200 code.. as one should do anyway.
I've been doing some refactoring on a slow running web application, and managed to reduce the number of requests and the size of the downloads to help improve the situation. Now the loading time is consistently shorter. However, consistently before there was hardly any time elapsed before the last 2 requests. Now consistently there is a gap.
Q1: What do these 'gaps' indicate in Chrome Network view?
Q2: Looking at the screenshots, the DOMContentLoaded time vs. the overall Finish time, are there any conclusions I can draw that could help me optimise further?
Record the page load in the Performance panel. See Get Started With Analyzing Runtime Performance to get the gist of how to use the panel. Understanding the network bottleneck can also help get you oriented.
However, you'll want to press the Reload page button (like Sam does in the "understanding the network bottleneck" video) instead of the Record button to record the page load performance, as the "get started with analyzing runtime performance" instructs you to do.
Once you've got a recording, the Main section shows you all of the main thread activity that occurs while the page is loading. The Network section shows you all of the Network requests. You'll probably be able to visually verify that there's a bunch of JavaScript work going on during the gap that you're seeing in your screenshots.
If it's still not clear to you, post a screenshot of your Performance panel recording and I'll help you decode the results.
Can anybody help me understand what the following is telling me in Chrome Dev Tools on the Network tab Timeline? Specifically, what does the gap mean?
During this terribly slow page load the resources display some info on how they are loading up until ~7 seconds or so. This is followed by a large gap. And then near ~13 seconds there is info on the resources loading again.
I've truncated the full list of resources, but the only resource bar that spans the gap, is the first resource for the primary page. It it identifies what is happening in the gap as “Content Download” (the blue bar):
The total resources on the page appear to be around 2.5MB in size. In this example, the DOMContentLoaded and Load events are taking place at the same time near the ~13 second mark.
Chrome starts loading external resources like images or scripts before the page HTML download has finished.
For example, here I've got an image at the top of the page, and then further down in the middle of the HTML a script tag.
This also means that the Load event follows quickly after DOMContentLoaded, because Chrome has already fetched the additional resources.
In your case, there's some JavaScript code that waits for either one of those events and then triggers additional requests.
The reason for the initial 13s load time is that after the waiting time of ~4s it takes another 9s to actually download the whole document.
This could be because the browser's internet connection is poor, or because the server takes a long time to generate the response, or because the server doesn't have enough bandwidth.
One of the new performance enhanchements for .NET 4.5 is the introduction of the 'MultiCode JIT'.
See here for more details.
I have tried this, but it seems to have no effect on my application.
The reason why I am interested is that my app (IronScheme) takes a good long time to startup if not NGEN'd, which implies a fair amount of JIT'ng is involved at startup. (1.4 sec vs 0.1 sec when NGEN'd).
I have followed the instructions on how to enable this, and I can see a 'small' (4-12KB) is created. But on subsequent startup, it seems to have absolutely no effect on improving the startup time. It is still 1.4 sec.
Has anyone actually seen (or made) this work in practice?
Also, are there any limitations on which code will be 'tracked'? Eg: assembly loading contexts, transient assemblies, etc. I ask this as the created file never seems to grow, but I am in fact generating a fair amount of code (in a transient assembly).
One bug that I did encounter was that SetProfileRoot does not seem to understand a / as a path separator, make sure to use \ .
The rule of thumb we use at Microsoft is that Multicore JIT gets you about half way towards NGEN startup performance. Thus if your app starts in 0.1 seconds with NGEN and 1.4 seconds without NGEN, we would expect Multicore JIT startup to take about 0.75 seconds.
That being said, we had to put some limitations in place to guarantee that program execution order is the same with and without MCJ. MCJ will sometimes pause the background thread waiting for modules to be loaded by the foreground thread, and will abort background compilation if there is an assembly resolve or module resolve event.
If you want to find out what's happening in your case, we have ETW (Event Tracing For Windows) instrumentation of the MCJ feature and we will be releasing a version of PerfView soon which will be able to collect these events by if you take a trace of your app startup.
Update: PerfView has been updated to be able to show background JIT information. Here are the steps to diagnosing with the latest version (1.2.2.0):
Collect a trace using PerfView of your application startup, either using Collect->Run or Collect->Collect from the main PerfView menu.
Assuming you used Collect-> Run, put the name of your .exe in the Command text box, pick a filename (i.e. IronScheme.etl), select Background JIT from Advanced Options, and click Run Command.
Close your application and double click on the IronScheme.etl file that gets generated.
Double click on the JIT Stats view in the list underneath IronScheme.etl, you should see something like this in the view that pops up:
This process uses Background JIT compilation (System.Runtime.ProfileOptimize)
Methods Background JITTed : 2,951
Percent # Methods Background JITTed : 52.9%
MSec Background JITTing : 3,901
Percent Time JITTing is Background : 50.9%
Background JIT Thread : 11308
You can click on "View Raw Background Jit Diagnostics" to see all of the MCJ events in excel. One question I forgot to ask: are you running this on a multicore machine or multicore VM? It is a common mistake to test out MCJ in a VM that only has a single logical processor.
Calling Activator.CreateInstance during startup seems to kill MCJ?
Or rather that triggered an Assembly Resolve, which completely seems to stop MCJ. And never work after that. Maybe the MSDN docs should mention this.