Large-scale models and FMU for model exchange - modelica

I want to export large-scale models from Openmodelica to FMU for model exchange.
As first step, I'm using this model for testing purposes:
model BigModel "Tank-Pipe-Tank cloned n times"
//
constant Real g=9.8;
//
parameter Integer n = 1;
parameter Real A_a = 1;
parameter Real A_b = 1;
parameter Real c = 0.2 "Kinematic viscosity";
parameter Real rho = 1000.0;
//
Real P_a[n];
Real P_b[n];
Real H_a[n] (start=fill(2,n),each fixed=true);
Real H_b[n] (start=fill(1,n),each fixed=true);
Real F[n] "Flow";
equation
for i in 1:n loop
P_a[i] = rho * g * H_a[i];
P_b[i] = rho * g * H_b[i];
F[i] = c * (H_a[i] - H_b[i]);
A_a * der(H_a[i]) = -F[i];
A_b * der(H_b[i]) = F[i];
end for;
end BigModel;
Changing "n" we can generate big models.
Depending on the OpenModelica version and the OS, the maximum number of equations changes a lot:
Windows 10 64 bits - OM 32 bits - 16 GB RAM - 4 CPU - i7) => n = 2600; Equations = 13000.
Windows 10 64 bits - OM 64 bits - 8 GB RAM - i5 => n = 100;
Equations = 500.
Windows 10 64 bits - OM 64 bits - 16 GB RAM - 4 CPU - i7) => n =
Similar to the previous one.
Ubuntu 17.10.1 64 bits - OM 64 bits - Virtual Machine - 8GB RAM - 1
CPU=> n = 22500; Equations = 112500
Does this have any sense for you?
Is it possible to reach n = 22500 in Windows?
Thanks!!!


The problem is not OMC, is GCC here.
In Windows 64bit OMC consumes 4.1G max with n=22500 and generates the files.
Then GCC runs on the files and goes out of memory. Also use -n=1 to not run GCC in parallel as that will consume even more memory.
I ran this script: fmu.mos
loadFile("BigModel.mo"); getErrorString();
translateModelFMU(BigModel); getErrorString();
using command line:
adrpo33#ida-0030 MINGW64 /c/home/adrpo33/dev/OMTesting/test
$ time ~/dev/OpenModelica/build/bin/omc -n=1 -d=execstat fmu.mos > trace.txt 2>&1
With this result:
true
"Notification: Performance of loadFile(BigModel.mo): time 0.001069/0.00107, allocations: 19.95 kB / 3.378 MB, free: 1.066 MB / 4.402 MB
"
""
"Notification: Performance of FrontEnd - loaded program: time 2.53e-005/2.53e-005, allocations: 8.109 kB / 3.803 MB, free: 0.6406 MB / 4.402 MB
Notification: Performance of FrontEnd - Absyn->SCode: time 8.684e-006/8.08e-005, allocations: 4.109 kB / 3.807 MB, free: 0.6367 MB / 4.402 MB
Notification: Performance of FrontEnd - scodeFlatten: time 0.008003/0.008111, allocations: 2.238 MB / 6.045 MB, free: 1.133 MB / 5.871 MB
Notification: Performance of FrontEnd - mkProgramGraph: time 5.286e-005/0.008211, allocations: 36.11 kB / 6.08 MB, free: 1.098 MB / 5.871 MB
Notification: Performance of FrontEnd: time 130.7/130.7, allocations: -1064517280 / -1058141792, free: 24 MB / 383.5 MB
Notification: Performance of Transformations before backend: time 0.03745/130.7, allocations: 6.872 MB / -1050935456, free: 24 MB / 383.5 MB
Notification: Performance of Generate backend data structure: time 3.862/134.6, allocations: 339.6 MB / -694807840, free: 0 / 0.5191 GB
Notification: Performance of prepare preOptimizeDAE: time 9.779e-005/134.6, allocations: 14.38 kB / -694793120, free: 0 / 0.5191 GB
Notification: Performance of preOpt normalInlineFunction (simulation): time 0.4386/135, allocations: 37.08 MB / -655909072, free: 101.4 MB / 0.5191 GB
Notification: Performance of preOpt evaluateParameters (simulation): time 1.076/136.1, allocations: 95.27 MB / -556013488, free: 68.34 MB / 0.5191 GB
Notification: Performance of preOpt simplifyIfEquations (simulation): time 0.0443/136.2, allocations: 13.73 MB / -541611840, free: 54.61 MB / 0.5191 GB
Notification: Performance of preOpt expandDerOperator (simulation): time 0.324/136.5, allocations: 46.69 MB / -492650352, free: 9.816 MB / 0.5191 GB
Notification: Performance of preOpt removeEqualFunctionCalls (simulation): time 3.973/140.4, allocations: 434.9 MB / -36674560, free: 6.402 MB / 0.5347 GB
Notification: Performance of preOpt clockPartitioning (simulation): time 4.345/144.8, allocations: 0.5446 GB / 0.5105 GB, free: 7.621 MB / 0.7378 GB
Notification: Performance of preOpt findStateOrder (simulation): time 0.1166/144.9, allocations: 3.431 MB / 0.5138 GB, free: 5.531 MB / 0.7378 GB
Notification: Performance of preOpt replaceEdgeChange (simulation): time 0.2595/145.2, allocations: 2.064 MB / 0.5158 GB, free: 5.527 MB / 0.7378 GB
Notification: Performance of preOpt inlineArrayEqn (simulation): time 0.0404/145.2, allocations: 12.36 MB / 0.5279 GB, free: 5.527 MB / 0.7378 GB
Notification: Performance of preOpt removeSimpleEquations (simulation): time 3.416/148.6, allocations: 449.9 MB / 0.9673 GB, free: 29.4 MB / 0.9097 GB
Notification: Performance of preOpt comSubExp (simulation): time 7.473/156.1, allocations: 1.242 GB / -1922531216, free: 230.5 MB / 0.9722 GB
Notification: Performance of preOpt resolveLoops (simulation): time 2.721/158.8, allocations: 403.8 MB / -1499167600, free: 219.4 MB / 0.9722 GB
Notification: Performance of preOpt evalFunc (simulation): time 0.1624/159, allocations: 20.26 MB / -1477922608, free: 219.4 MB / 0.9722 GB
Notification: Performance of preOpt encapsulateWhenConditions (simulation): time 0.07544/159.1, allocations: 39.34 MB / -1436668032, free: 211.6 MB / 0.9722 GB
Notification: Performance of pre-optimization done (n=112500): time 0.01513/159.1, allocations: 0 / -1436668032, free: 211.6 MB / 0.9722 GB
Notification: Performance of matching and sorting (n=112500): time 6.358/165.4, allocations: 0.7035 GB / -681302800, free: 57.4 MB / 0.9722 GB
Notification: Performance of inlineWhenForInitialization (initialization): time 0.8272/166.3, allocations: 0.5764 GB / -62439520, free: 133.4 MB / 1.3 GB
Notification: Performance of selectInitializationVariablesDAE (initialization): time 0.03557/166.3, allocations: 102.6 kB / -62334496, free: 133.3 MB / 1.3 GB
Notification: Performance of collectPreVariables (initialization): time 0.198/166.5, allocations: 1.402 MB / -60864704, free: 133.3 MB / 1.3 GB
Notification: Performance of collectInitialEqns (initialization): time 0.01097/166.5, allocations: 1.795 MB / -58982608, free: 131.5 MB / 1.3 GB
Notification: Performance of collectInitialBindings (initialization): time 1.716/168.2, allocations: 425.5 MB / 369.2 MB, free: 332.3 MB / 1.3 GB
Notification: Performance of simplifyInitialFunctions (initialization): time 0.2769/168.5, allocations: 0 / 369.2 MB, free: 332.3 MB / 1.3 GB
Notification: Performance of setup shared object (initialization): time 0.2535/168.8, allocations: 51.25 MB / 420.5 MB, free: 329.1 MB / 1.3 GB
Notification: Performance of preBalanceInitialSystem (initialization): time 3.38/172.1, allocations: 352.8 MB / 0.7552 GB, free: 336.1 MB / 1.3 GB
Notification: Performance of partitionIndependentBlocks (initialization): time 4.389/176.5, allocations: 0.5341 GB / 1.289 GB, free: 124.8 MB / 1.3 GB
Notification: Performance of analyzeInitialSystem (initialization): time 4.051/180.6, allocations: 468.3 MB / 1.747 GB, free: 140.9 MB / 1.3 GB
Notification: Performance of solveInitialSystemEqSystem (initialization): time 0.03169/180.6, allocations: 1.374 MB / 1.748 GB, free: 140.9 MB / 1.3 GB
Notification: Performance of matching and sorting (n=157500) (initialization): time 8.06/188.7, allocations: 0.9828 GB / -1362827296, free: 35.2 MB / 1.378 GB
Notification: Performance of prepare postOptimizeDAE: time 3.512e-005/188.7, allocations: 5.625 kB / -1362821536, free: 35.2 MB / 1.378 GB
Notification: Performance of postOpt simplifyComplexFunction (initialization): time 0.05533/188.7, allocations: 4.807 MB / -1357780960, free: 35.2 MB / 1.378 GB
Notification: Performance of postOpt tearingSystem (initialization): time 0.07246/188.8, allocations: 10.99 MB / -1346260768, free: 35.2 MB / 1.378 GB
Notification: Performance of postOpt solveSimpleEquations (initialization): time 4.998/193.8, allocations: 315.5 MB / -1015414128, free: 94.34 MB / 1.472 GB
Notification: Performance of postOpt calculateStrongComponentJacobians (initialization): time 0.06548/193.9, allocations: 13.39 MB / -1001371280, free: 94.34 MB / 1.472 GB
Notification: Performance of postOpt simplifyAllExpressions (initialization): time 1.557/195.4, allocations: 59.39 MB / -939091840, free: 74.02 MB / 1.472 GB
Notification: Performance of postOpt collapseArrayExpressions (initialization): time 0.3376/195.7, allocations: 40.51 MB / -896611008, free: 74.02 MB / 1.472 GB
Notification: Performance of prepare postOptimizeDAE: time 0.2474/196, allocations: 1.451 MB / -895089264, free: 73.95 MB / 1.472 GB
Notification: Performance of postOpt createAliasVarsForOutputStates (simulation): time 7.001/203, allocations: 0.718 GB / -124186368, free: 158.8 MB / 1.472 GB
Notification: Performance of postOpt lateInlineFunction (simulation): time 0.465/203.5, allocations: 44.3 MB / -77732736, free: 143.9 MB / 1.472 GB
Notification: Performance of postOpt wrapFunctionCalls (simulation): time 0.4331/203.9, allocations: 71.41 MB / -2853904, free: 143.9 MB / 1.472 GB
Notification: Performance of postOpt inlineArrayEqn (simulation): time 0.04548/203.9, allocations: 15.79 MB / 13.07 MB, free: 143.9 MB / 1.472 GB
Notification: Performance of postOpt constantLinearSystem (simulation): time 0.04134/204, allocations: 6.181 MB / 19.25 MB, free: 143.9 MB / 1.472 GB
Notification: Performance of postOpt simplifysemiLinear (simulation): time 0.06582/204, allocations: 10.65 MB / 29.91 MB, free: 143.9 MB / 1.472 GB
Notification: Performance of postOpt removeSimpleEquations (simulation): time 4.006/208.1, allocations: 339 MB / 369 MB, free: 156 MB / 1.472 GB
Notification: Performance of postOpt simplifyComplexFunction (simulation): time 0.03146/208.1, allocations: 4.805 MB / 373.8 MB, free: 156 MB / 1.472 GB
Notification: Performance of postOpt solveSimpleEquations (simulation): time 5.964/214.1, allocations: 316.5 MB / 0.6741 GB, free: 151.8 MB / 1.472 GB
Notification: Performance of postOpt tearingSystem (simulation): time 0.06915/214.1, allocations: 9.619 MB / 0.6835 GB, free: 148.4 MB / 1.472 GB
Notification: Performance of postOpt inputDerivativesUsed (simulation): time 0.4287/214.5, allocations: 30.9 MB / 0.7137 GB, free: 128.5 MB / 1.472 GB
Notification: Performance of postOpt calculateStrongComponentJacobians (simulation): time 0.04856/214.6, allocations: 12.02 MB / 0.7254 GB, free: 117.5 MB / 1.472 GB
Notification: Performance of postOpt calculateStateSetsJacobians (simulation): time 0.02431/214.6, allocations: 7.555 MB / 0.7328 GB, free: 109.9 MB / 1.472 GB
Notification: Performance of postOpt detectJacobianSparsePattern (simulation): time 37.72/252.3, allocations: -1209710736 / -422866672, free: 0.7053 GB / -1724342272
Notification: Performance of postOpt removeConstants (simulation): time 1.288/253.6, allocations: 113.4 MB / -303983904, free: 0.5939 GB / -1724342272
Notification: Performance of postOpt simplifyTimeIndepFuncCalls (simulation): time 0.5424/254.2, allocations: 24.38 MB / -278421456, free: 0.57 GB / -1724342272
Notification: Performance of postOpt simplifyAllExpressions (simulation): time 1.631/255.8, allocations: 4.121 MB / -274100176, free: 0.566 GB / -1724342272
Notification: Performance of postOpt findZeroCrossings (simulation): time 0.9685/256.8, allocations: 69.35 MB / -201383472, free: 0.9268 GB / -1724342272
Notification: Performance of postOpt collapseArrayExpressions (simulation): time 0.2593/257, allocations: 34.33 MB / -165385120, free: 0.9268 GB / -1724342272
Notification: Performance of sort global known variables: time 8.42e-005/257, allocations: 84.09 kB / -165299008, free: 0.9268 GB / -1724342272
Notification: Performance of remove unused functions: time 1.37/258.4, allocations: 116.7 MB / -42898336, free: 0.9268 GB / -1724342272
Notification: Performance of Backend phase and start with SimCode phase: time 39.46/297.9, allocations: 393.4 MB / 352.5 MB, free: 0.89 GB / -1715953664
Notification: Performance of simCode: created initialization part: time 5.951/303.8, allocations: 0.8172 GB / 1.161 GB, free: 0.8811 GB / -1715953664
Notification: Performance of simCode: created event and clocks part: time 0.00486/303.8, allocations: 0.6865 MB / 1.162 GB, free: 0.8811 GB / -1715953664
Notification: Performance of simCode: created simulation system equations: time 2.295/306.1, allocations: 397.2 MB / 1.55 GB, free: 0.8525 GB / -1715953664
Notification: Performance of simCode: created of all other equations (e.g. parameter, nominal, assert, etc): time 1.112/307.2, allocations: 12.37 MB / 1.562 GB, free: 0.8525 GB / -1715953664
Notification: Performance of simCode: created linear, non-linear and system jacobian parts: time 11.74/319, allocations: -2053937424 / -376601600, free: 0.7194 GB / -1699176448
Notification: Performance of simCode: all other stuff during SimCode phase: time 3.699/322.7, allocations: 480.8 MB / 121.6 MB, free: 0.7004 GB / -1699176448
Notification: Performance of SimCode: time 2.643e-006/322.7, allocations: 3.594 kB / 121.6 MB, free: 0.7004 GB / -1699176448
Notification: Performance of buildModelFMU: Generate the FMI files: time 63.56/386.2, allocations: -2089164704 / -1961640176, free: 7.172 MB / -843538432
Notification: Performance of buildModelFMU: configured platform dynamic using cached values: time 0.008376/386.2, allocations: 145.7 kB / -1961490976, free: 6.98 MB / -843538432
Error: Error building simulator. Build log: gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_FMU.o BigModel_FMU.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel.o BigModel.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_functions.o BigModel_functions.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_records.o BigModel_records.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_01exo.o BigModel_01exo.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_02nls.o BigModel_02nls.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_03lsy.o BigModel_03lsy.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_04set.o BigModel_04set.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_05evt.o BigModel_05evt.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_06inz.o BigModel_06inz.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_07dly.o BigModel_07dly.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_08bnd.o BigModel_08bnd.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_09alg.o BigModel_09alg.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_10asr.o BigModel_10asr.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_11mix.o BigModel_11mix.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_12jac.o BigModel_12jac.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_13opt.o BigModel_13opt.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_14lnz.o BigModel_14lnz.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_15syn.o BigModel_15syn.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_16dae.o BigModel_16dae.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_17inl.o BigModel_17inl.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_init_fmu.o BigModel_init_fmu.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_06inz_part0.o BigModel_06inz_part0.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_06inz_part1.o BigModel_06inz_part1.c
gcc -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel_06inz_part2.o BigModel_06inz_part2.c
cc1.exe: out of memory allocating 16777144 bytes
<builtin>: recipe for target 'BigModel_12jac.o' failed
mingw32-make: *** [BigModel_12jac.o] Error 1
mingw32-make: *** Waiting for unfinished jobs....
cc1.exe: out of memory allocating 142198032 bytes
<builtin>: recipe for target 'BigModel.o' failed
mingw32-make: *** [BigModel.o] Error 1
cc1.exe: out of memory allocating 65536 bytes
<builtin>: recipe for target 'BigModel_init_fmu.o' failed
mingw32-make: *** [BigModel_init_fmu.o] Error 1
RESULT: 2
Error: Error opening file: BigModel.log: No such file or directory.
"
In Linux we use CLANG and in Windows we use GCC and it seems that GCC is way more memory hungry.
Is should be possible to reach n=22500 but maybe with a newer GCC because the one we have in OMC seems to be crashing:
adrpo33#ida-0030 MINGW64 /c/home/adrpo33/dev/OMTesting/test/BigModel.fmutmp/sources
$ make SIM_OR_DYNLOAD_OPT_LEVEL=-O0
gcc -O0 -falign-functions -msse2 -mfpmath=sse -Iinclude/ -Iinclude/fmi2 -I. -DOMC_MODEL_PREFIX=BigModel -DOMC_NUM_MIXED_SYSTEMS=0 -DOMC_NUM_LINEAR_SYSTEMS=0 -DOMC_NUM_NONLINEAR_SYSTEMS=0 -DOMC_NDELAY_EXPRESSIONS=0 -DOMC_NVAR_STRING=0 -c -o BigModel.o BigModel.c
BigModel.c: In function 'BigModel_eqFunction_228127':
BigModel.c:847581:1: internal compiler error: Segmentation fault

Related

MongoDB slow - memory usage very high

I have a 5 note replicaset mongoDB - 1 primary, 3 secondaries and 1 arbiter.
I am using mong version 4.2.3
Sizes:
“dataSize” : 688.4161271536723,
“indexes” : 177,
“indexSize” : 108.41889953613281
My Primary is very slow - each command from the shell takes a long time to return.
Memory usage seems very high, and looks like mongodb is consuming more than 50% of the RAM:
# free -lh
total used free shared buff/cache available
Mem: 188Gi 187Gi 473Mi 56Mi 740Mi 868Mi
Low: 188Gi 188Gi 473Mi
High: 0B 0B 0B
Swap: 191Gi 117Gi 74Gi
------------------------------------------------------------------
Top Memory Consuming Process Using ps command
------------------------------------------------------------------
PID PPID %MEM %CPU CMD
311 49145 97.8 498 mongod --config /etc/mongod.conf
23818 23801 0.0 3.8 /bin/prometheus --config.file=/etc/prometheus/prometheus.yml
23162 23145 0.0 8.4 /usr/bin/cadvisor -logtostderr
25796 25793 0.0 0.4 postgres: checkpointer
23501 23484 0.0 1.0 /postgres_exporter
24490 24473 0.0 0.1 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
311 systemd+ 20 0 313.9g 184.6g 2432 S 151.7 97.9 26229:09 mongod
23818 nobody 20 0 11.3g 150084 17988 S 20.7 0.1 8523:47 prometheus
23162 root 20 0 12.7g 93948 5964 S 65.5 0.0 18702:22 cadvisor
serverStatus memeory shows this:
octopusrs0:PRIMARY> db.serverStatus().mem
{
"bits" : 64,
"resident" : 189097,
"virtual" : 321404,
"supported" : true
}
octopusrs0:PRIMARY> db.serverStatus().tcmalloc.tcmalloc.formattedString
------------------------------------------------
MALLOC: 218206510816 (208097.9 MiB) Bytes in use by application
MALLOC: + 96926863360 (92436.7 MiB) Bytes in page heap freelist
MALLOC: + 3944588576 ( 3761.9 MiB) Bytes in central cache freelist
MALLOC: + 134144 ( 0.1 MiB) Bytes in transfer cache freelist
MALLOC: + 713330688 ( 680.3 MiB) Bytes in thread cache freelists
MALLOC: + 1200750592 ( 1145.1 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 320992178176 (306122.0 MiB) Actual memory used (physical + swap)
MALLOC: + 13979086848 (13331.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 334971265024 (319453.5 MiB) Virtual address space used
MALLOC:
MALLOC: 9420092 Spans in use
MALLOC: 234 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
How can I detrmine what is causing this high memory consumption, and what can I do to return to nornal memory consumption?
Thanks,
Tamar

FFMPEG, faulty input with only a few keyframes?

I have this video I can watch on MPC-HC alright, but I've been trying to re-encode it with FFMPEG to no avail.
The video unerringly stops 5 minutes into the clip and freezes. Checking the key frames it would seem like the last keyframe is indeed at around the 5 minute mark, even though the video is completely watchable for its entire duration of upwards an hour. I've been looking into finding a way to repopulate the index of keyframes or something, but the answer eludes me.
Both Handbrake and Premiere Pro fails to handle this video properly as well. Premiere Pro just imports it as though it's 5 minutes long, and Handbrake freezes when the encoding reaches the 5 minute mark.
Even doing a -c copy would give me an output that terminates at the 5 minute mark.
What can I do to fix this?
Edit: Added log as requested.
ffmpeg version git-2020-02-27-9b22254 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.2.1 (GCC) 20200122
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil 56. 42.100 / 56. 42.100
libavcodec 58. 73.102 / 58. 73.102
libavformat 58. 39.101 / 58. 39.101
libavdevice 58. 9.103 / 58. 9.103
libavfilter 7. 77.100 / 7. 77.100
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100
[mov,mp4,m4a,3gp,3g2,mj2 # 0000019d5f659b00] st: 0 edit list: 1 Missing key frame while searching for timestamp: 20
[mov,mp4,m4a,3gp,3g2,mj2 # 0000019d5f659b00] st: 0 edit list 1 Cannot find an index entry before timestamp: 20.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '[REDACTED].mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
creation_time : 2009-07-27T12:06:40.000000Z
Duration: 01:13:58.42, start: 0.000000, bitrate: 2671 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 716x480 [SAR 8:9 DAR 179:135], 2504 kb/s, SAR 29127:32768 DAR 1007794:760071, 29.97 fps, 29.97 tbr, 48k tbn, 59.94 tbc (default)
Metadata:
rotate : 0
creation_time : 2007-09-08T18:58:09.000000Z
encoder : AVC Coding
Side data:
displaymatrix: rotation of -0.00 degrees
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 159 kb/s (default)
Metadata:
creation_time : 2009-07-27T12:06:47.000000Z
handler_name : GPAC ISO Audio Handler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 # 0000019d5f65de40] using SAR=8/9
[libx264 # 0000019d5f65de40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 # 0000019d5f65de40] profile High 4:4:4 Predictive, level 3.0, 4:2:0, 8-bit
[libx264 # 0000019d5f65de40] 64 - core 159 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0
Output #0, mp4, to '[REDACTED].mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
encoder : Lavf58.39.101
Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p(progressive), 716x480 [SAR 29127:32768 DAR 1007794:760071], q=-1--1, 29.97 fps, 11988 tbn, 29.97 tbc (default)
Metadata:
encoder : Lavc58.73.102 libx264
creation_time : 2007-09-08T18:58:09.000000Z
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
displaymatrix: rotation of -0.00 degrees
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 192 kb/s (default)
Metadata:
creation_time : 2009-07-27T12:06:47.000000Z
handler_name : GPAC ISO Audio Handler
encoder : Lavc58.73.102 aac
frame= 9958 fps= 70 q=-1.0 Lsize= 960472kB time=00:11:13.32 bitrate=11685.6kbits/s speed=4.71x
video:944292kB audio:15836kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.035744%
[libx264 # 0000015f858dde40] frame I:46 Avg QP: 0.00 size:113701
[libx264 # 0000015f858dde40] frame P:9912 Avg QP: 0.00 size: 97026
[libx264 # 0000015f858dde40] mb I I16..4: 52.6% 10.7% 36.7%
[libx264 # 0000015f858dde40] mb P I16..4: 22.5% 5.5% 10.5% P16..4: 31.8% 15.2% 11.1% 0.0% 0.0% skip: 3.4%
[libx264 # 0000015f858dde40] 8x8 transform intra:14.3% inter:40.9%
[libx264 # 0000015f858dde40] coded y,uvDC,uvAC intra: 95.6% 84.9% 84.0% inter: 81.2% 83.0% 82.5%
[libx264 # 0000015f858dde40] i16 v,h,dc,p: 53% 44% 2% 1%
[libx264 # 0000015f858dde40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 40% 47% 8% 1% 1% 1% 1% 1% 1%
[libx264 # 0000015f858dde40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 44% 5% 3% 5% 4% 2% 3% 1%
[libx264 # 0000015f858dde40] i8c dc,h,v,p: 12% 44% 43% 1%
[libx264 # 0000015f858dde40] Weighted P-Frames: Y:1.0% UV:0.2%
[libx264 # 0000015f858dde40] ref P L0: 59.1% 10.2% 27.8% 2.9% 0.0%
[libx264 # 0000015f858dde40] kb/s:23281.49
[aac # 0000015f85b5b880] Qavg: 182.070

Postgres Out of Memory

I am setting up a new install of postgres 10.4 and during load testing I keep receiving an out of memory error.
Configuration:
$ cat /etc/security/limits.conf
postgres hard memlock 508559360
postgres soft memlock 508559360
$ cat /etc/sysctl.conf
vm.nr_hugepages = 248320
vm.hugetlb_shm_group = 118
vm.overcommit_memory=2
vm.swappiness=1
vm.vfs_cache_pressure=50
kernel.sem = 250 32000 32 128
System Specs:
$ cat /proc/cpuinfo | grep "core id" | wc -l
32
Each CPU is an Intel(R) Xeon(R) CPU E5-2695 v4 # 2.10GHz. 16 physical cores, 32 logical as shown above.
$ free -m
free -m
total used free shared buff/cache available
Mem: 510969 498048 2434 3098 10485 8343
Swap: 1071 1071 0
Note: we have the memlock set at approximately 485 GB dedicated directly to postgres which shows in the high "used" column.
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 250G 64K 250G 1% /dev/shm
$ cat /proc/meminfo
MemTotal: 523232496 kB
MemFree: 1216648 kB
MemAvailable: 8348808 kB
Buffers: 65220 kB
Cached: 10211780 kB
SwapCached: 7620 kB
Active: 5556092 kB
Inactive: 5432428 kB
Active(anon): 2208064 kB
Inactive(anon): 1593084 kB
Active(file): 3348028 kB
Inactive(file): 3839344 kB
Unevictable: 24128 kB
Mlocked: 24128 kB
SwapTotal: 1097724 kB
SwapFree: 0 kB
Dirty: 52 kB
Writeback: 0 kB
AnonPages: 728044 kB
Mapped: 62904 kB
Shmem: 3085592 kB
Slab: 1454276 kB
SReclaimable: 1345008 kB
SUnreclaim: 109268 kB
KernelStack: 11984 kB
PageTables: 22856 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 15770860 kB
Committed_AS: 7920136 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 71680 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 248320
HugePages_Free: 245883
HugePages_Rsvd: 1800
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 329600 kB
DirectMap2M: 534444032 kB
Postgres Info:
SELECT version();
PostgreSQL 10.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, 64-bit
max_connections=600
shared_buffers= '8192 MB'
work_mem = '20 MB'
maintenance_work_mem = 2GB
max_parallel_workers = 8
wal_buffers = 16MB
max_wal_size = 20GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = '364 GB'
default_statistics_target = 1000
log_timezone = 'US/Eastern'
track_activities = on
track_counts = on
track_io_timing = on
stats_temp_directory = 'pg_stat_tmp'
datestyle = 'iso, mdy'
timezone = 'US/Eastern'
default_text_search_config = 'pg_catalog.english'
transform_null_equals = on
shared_preload_libraries = 'pg_stat_statements'
track_activity_query_size = 16384
track_functions = all
track_io_timing = true
pg_stat_statements.track = all
session_preload_libraries = 'auto_explain'
auto_explain.log_min_duration = '3s'
auto_explain.log_nested_statements='on'
auto_explain.log_analyze=true
Note: I've started w/ work memory set as high as 250 MB and have slowly brought it down to 20 MB and still receive the errors. I've also verified that connections don't go higher than 120 connections. We use PGBouncer in front of the instance in session mode.
The errors are too large for stack overflow but here they are linked:
https://codepad.co/snippet/derPU4E8
The standout errors are:
2018-06-18 15:02:22 EDT,28197,mydb,ERROR: could not resize shared memory segment "/PostgreSQL.1552129380" to 192088 bytes: No space left on device
I don't understand what device its talking about. I don't have any that appear to be even close to OOM let alone the tiny 192088 bytes its talking about.
2018-06-18 15:02:22 EDT,19708,mydb,ERROR: out of memory
2018-06-18 15:02:22 EDT,19708,mydb,DETAIL: Failed on request of size 7232.
2018-06-18 15:02:22 EDT,16688,,LOG: could not fork worker process: Cannot allocate memory
2018-06-18 15:02:22 EDT,4555,,ERROR: out of memory
2018-06-18 15:02:22 EDT,4555,,DETAIL: Failed on request of size 78336.
2018-06-18 15:02:22 EDT,4552,,LOG: could not open directory "/usr/lib/postgresql/10/share/timezone": Cannot allocate memory
2018-06-18 15:02:22 EDT,19935,mydb,ERROR: could not load library "/usr/lib/postgresql/10/lib/auto_explain.so": /usr/lib/postgresql/10/lib/auto_explain.so: failed to map segment from shared object
2018-06-18 15:02:22 EDT,28193,mydb,ERROR: out of memory
2018-06-18 15:02:22 EDT,28193,mydb,DETAIL: Failed on request of size 8192.
2018-06-18 15:02:22 EDT,26927,mydb,ERROR: could not resize shared memory segment "/PostgreSQL.1101931262" to 192088 bytes: No space left on device
Question: How do I debug this issue and more importantly how can I resolve it?

I followed Laurenz advice and no longer run OOM.
vm.overcommit_ratio = 100.
Additionally, I removed the memlock and huge pages definition in sysctl/limits.

Check to see if your queries are so complicated that they need to consume a large amount of temporary tablespace.

How to find out what caused PTE exhaustion?

I have my Windows 10 bluescreen several times and I have memory dump and running !vm produces output below showing 0 available PTEs. How do I find out postmortem who leaked those or monitor on live system which process/driver responsible for leak?
0: kd> !vm
Page File: \??\C:\pagefile.sys
Current: 9961472 Kb Free Space: 9961464 Kb
Minimum: 9961472 Kb Maximum: 62351824 Kb
Page File: \??\C:\swapfile.sys
Current: 16384 Kb Free Space: 16376 Kb
Minimum: 16384 Kb Maximum: 49881460 Kb
No Name for Paging File
Current: 129378408 Kb Free Space: 129362880 Kb
Minimum: 129378408 Kb Maximum: 129378408 Kb
Physical Memory: 16756646 ( 67026584 Kb)
Available Pages: 11876350 ( 47505400 Kb)
ResAvail Pages: 15917045 ( 63668180 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 0 ( 0 Kb)
********** Running out of system PTEs **************
Modified Pages: 249289 ( 997156 Kb)
Modified PF Pages: 249261 ( 997044 Kb)
Modified No Write Pages: 25 ( 100 Kb)
NonPagedPool Usage: 3042 ( 12168 Kb)
NonPagedPoolNx Usage: 103383 ( 413532 Kb)
NonPagedPool Max: 4294967296 (17179869184 Kb)
PagedPool 0: 144048 ( 576192 Kb)
PagedPool 1: 31595 ( 126380 Kb)
PagedPool 2: 31923 ( 127692 Kb)
PagedPool 3: 31631 ( 126524 Kb)
PagedPool 4: 31714 ( 126856 Kb)
PagedPool Usage: 270911 ( 1083644 Kb)
PagedPool Maximum: 4294967296 (17179869184 Kb)
Processor Commit: 1348 ( 5392 Kb)
Session Commit: 17782 ( 71128 Kb)
Shared Commit: 658461 ( 2633844 Kb)
Special Pool: 0 ( 0 Kb)
Kernel Stacks: 26919 ( 107676 Kb)
Pages For MDLs: 395401 ( 1581604 Kb)
Pages For AWE: 0 ( 0 Kb)
NonPagedPool Commit: 97838 ( 391352 Kb)
PagedPool Commit: 270911 ( 1083644 Kb)
Driver Commit: 19721 ( 78884 Kb)
Boot Commit: 2732 ( 10928 Kb)
PFN Array Commit: 196913 ( 787652 Kb)
System PageTables: 3267 ( 13068 Kb)
ProcessLockedFilePages: 306 ( 1224 Kb)
Pagefile Hash Pages: 0 ( 0 Kb)
Sum System Commit: 1691599 ( 6766396 Kb)
Total Private: 4330147 ( 17320588 Kb)
Misc/Transient Commit: 9281 ( 37124 Kb)
Committed pages: 6031027 ( 24124108 Kb)
Commit limit: 19247014 ( 76988056 Kb)
Pid ImageName Commit SharedCommit Debt
598c vmmem 3677256 Kb 0 Kb 0 Kb
4d4 RemoteDesktopManager.exe 1367276 Kb 473008 Kb 0 Kb
6300 vmmem 1050684 Kb 0 Kb 0 Kb
207c vmmem 1050684 Kb 0 Kb 0 Kb
1d18 vmmem 1050684 Kb 0 Kb 0 Kb
276c powershell.exe 713052 Kb 4660 Kb 0 Kb
483c chrome.exe 635112 Kb 130192 Kb 0 Kb
3ad4 chrome.exe 525988 Kb 20672 Kb 0 Kb

From Russinovich, Mark; Solomon, David; Ionescu, Alex. Windows Internals, Part 2 (6th Edition) (Developer Reference)
you can enable system PTE tracking by creating a new DWORD value in
the HKLM\ SYSTEM\ CurrentControlSet\ Control\ Session Manager\ Memory
Management key called TrackPtes and setting its value to 1. You can
then use !sysptes 4 to show a list of allocators
So you could try to
set the registry key and reboot
use livekdon the running system and execute !sysptes 4 to get al list of allocators
or execute !sysptes 4 on a dump you collect

Move last column title name and put to be on first column on all rows

Want to get the last column in the title in that example is "db5b", and to place in to the first column on every next row.
Output:
Filesystem utilization for db5b
/dev/hd4 3.00 2.87 5% 5728 1% /
/dev/hd2 7.00 2.79 61% 61845 9% /usr
/dev/hd9var 7.00 5.88 17% 861 1% /var
/dev/hd3 3.00 2.98 1% 205 1% /tmp
Expected result:
db5b 5% /
db5b 61% /usr
db5b 17% /var
db5b 1% /tmp
I use that but not able to figure out how to place db5b to be on the first column:
awk '{print $3, $7}'
I guess if want to finish the result with awk to use NF but not sure how to finish the command.

Like this, for example:
$ awk 'NR==1 {a=$NF} NR>1 && $0 {print a,$4,$7}' file
db5b 5% /
db5b 61% /usr
db5b 17% /var
db5b 1% /tmp
Explanation
NR==1 {a=$NF} on first line, save the last field into var a.
NR>1 && $0 {print a,$4,$7} on lines after the first with some data, print the value plus 4th and 7th fields.
And for an input like Vijay's:
$ awk '/Filesystem/ {a=$NF} /^\// {print a,$4,$7}' file
db5b 5% /
db5b 61% /usr
db5b 17% /var
db5b 1% /tmp
db6b 5% /
db6b 61% /usr
db6b 17% /var
db6b 1% /tmp

this one-liner gives what you wanted:
awk 'NR==1{t=$NF;next}$0{print t,$4,$NF}' file
EDIT
for different titles:
awk 'NF==4{t=$NF;next}$0{print t,$4,$NF}' file

awk '$4!~/%/ && NF>0{a=$NF;next}{print a,$4,$7}' your_file
Test:
> cat temp
Filesystem utilization for db5b
/dev/hd4 3.00 2.87 5% 5728 1% /
/dev/hd2 7.00 2.79 61% 61845 9% /usr
/dev/hd9var 7.00 5.88 17% 861 1% /var
/dev/hd3 3.00 2.98 1% 205 1% /tmp
Filesystem utilization for db6b
/dev/hd4 3.00 2.87 5% 5728 1% /
/dev/hd2 7.00 2.79 61% 61845 9% /usr
/dev/hd9var 7.00 5.88 17% 861 1% /var
/dev/hd3 3.00 2.98 1% 205 1% /tmp
> awk '$4!~/%/ && NF>0{a=$NF;next}{print a,$4,$7}' temp
db5b 5% /
db5b 61% /usr
db5b 17% /var
db5b 1% /tmp
db6b 5% /
db6b 61% /usr
db6b 17% /var
db6b 1% /tmp
>

If you wanted something non-awk:-
#!/usr/bin/env bash
set -x
while read -r
do
output=$(echo "${REPLY}" | tr -s ' ')
drive=$(echo "${output}" | cut -f7)
percent=$(echo "${output}" | cut -f4)
echo -e "db5b\t${percent}\t${drive}"
done < fs_utilisation
The -e switch to echo may be bash-only, and thus non-portable. I apologise if so; you will have to check. The reason why I used the -e switch is because it will interpret \t to give you tabs between your fields.