I would like to run !refs command against each address from following command
!dumpgen 2 -type System.DateTime[]
How this could be done.I know a loop can be created as follows
.foreach (myvar {!dumpgen 2 -type System.DateTime[]})
But how can I access object address that could be used in loop with !refs?
!dumpgen does not have a -short argument like !dumpheap has and I'd really like to see a simpler answer than this one.
Approach 1: dump a generation manually
Get the addresses of the heap
0:003> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x026f1018
generation 1 starts at 0x026f100c
generation 2 starts at 0x026f1000
Use the addresses to limit the output to the generation you want:
!dumpheap -type X <start> <end>
Use the -short parameter on !dumpheap, which outputs the address only. This address of an object can then be processed by other commands.
Also note: using -type may result in other types as well. Better use the method table with -mt since only that guarantees the uniqueness of types. Use !name2ee if you don't get that from elsewhere.
A complete session could look like this:
0:003> !dumpheap -stat
total 345 objects
Statistics:
MT Count TotalSize Class Name
53ab421c 1 12 System.Text.DecoderExceptionFallback
[...]
53ab0d48 135 6640 System.String
53a84518 26 9452 System.Object[]
Total 345 objects
0:003> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x026f1018
generation 1 starts at 0x026f100c
generation 2 starts at 0x026f1000
[...]
0:003> !name2ee *!System.String
Module: 53841000 (mscorlib.dll)
Token: 0x02000024
MethodTable: 53ab0d48
[...]
0:003> !dumpheap -short -mt 53ab0d48 0x026f1000 0x026f100c
(Ok, all my strings seem to be in generation 0, damn :-)
0:003> .foreach (addr {!dumpheap -short -mt 53ab0d48 0x026f1018}) {!refs ${addr}}
Disadvantage: you need to do that for all GC heaps separately. There could be several of them.
Approach 2: decide for the generation on each single object
Another ugly solution is to
Dump all object addresses (!dumpheap -short and -type or -mt)
Query the generation of each object via !gcgen
Depending on the generation, execute a command
Here's how to do it (formatted for readability, put it all into one line):
.foreach (addr {!dumpheap -short -mt 53ab0d48}) {
.foreach /pS 1 (gen {!gcgen ${addr}}) {
.if ($scmp("${gen}","2")==0) {
!refs ${addr}
}
}
}
where 53ab0d48 is the method table of the type you're looking for and "2" is the generation you want. /pS 1 skips the word "Gen" in the output of !gcgen.
Disadvantage: might be slow since it works on all objects.
Related
I am running the following script:
# Usage: sh hmmscan-parser.sh hmmscan-file
# 1. take hmmer3 output and generate the tabular output
# 2. sort on the 6th and 7th cols
# 3. remove overlapped/redundant hmm matches and keep the one with the lower e-values
# 4. calculate the covered fraction of hmm (make sure you have downloaded the "all.hmm.ps.len" file to the same directory of this perl script)
# 5. apply the E-value cutoff and the covered faction cutoff
cat $1 | perl -e 'while(<>){if(/^\/\//){$x=join("",#a);($q)=($x=~/^Query:\s+(\S+)/m);while($x=~/^>> (\S+.*?\n\n)/msg){$a=$&;#c=split(/\n/,$a);$c[0]=~s/>> //;for($i=3;$i<=$#c;$i++){#d=split(/\s+/,$c[$i]);print $q."\t".$c[0]."\t$d[6]\t$d[7]\t$d[8]\t$d[10]\t$d[11]\n" if $d[6]<1;}}#a=();}else{push(#a,$_);}}' \
| sort -k 1,1 -k 6,6n -k 7,7n | uniq \
| perl -e 'while(<>){chomp;#a=split;next if $a[-1]==$a[-2];push(#{$b{$a[0]}},$_);}foreach(sort keys %b){#a=#{$b{$_}};for($i=0;$i<$#a;$i++){#b=split(/\t/,$a[$i]);#c=split(/\t/,$a[$i+1]);$len1=$b[-1]-$b[-2];$len2=$c[-1]-$c[-2];$len3=$b[-1]-$c[-2];if($len3>0 and ($len3/$len1>0.5 or $len3/$len2>0.5)){if($b[2]<$c[2]){splice(#a,$i+1,1);}else{splice(#a,$i,1);}$i=$i-1;}}foreach(#a){print $_."\n";}}' \
| uniq | perl -e 'open(IN,"all.hmm.ps.len");
while(<IN>)
{
chomp;
#a=split;
$b{$a[0]}=$a[1]; # creates hash of hmmName : hmmLength
}
while(<>)
{
chomp;
#a=split;
$r=($a[3]-$a[2])/$b{$a[1]}; # $a[4] = hmm end $a[3] = hmm start ; $b{$a[1]} = result of the hash of the name of the hmm (hmm length).
print $_."\t".$r."\n";
}' \
| perl -e 'while(<>){#a=split(/\t/,$_);if(($a[-2]-$a[-3])>80){print $_ if $a[2]<1e-5;}else{print $_ if $a[2]<1e-3;}}' | awk '$NF>0.3'
When I run the file, I get "Illegal division by zero at -e line 14, <> line 1"
Line 14 is :
$r=($a[3]-$a[2])/$b{$a[1]};
The first input (hmmscan-file) is under this form :
Query: NODE_1_length_300803_cov_11.207433_1 [L=264]
Description: # 139 # 930 # 1 # ID=1_1;partial=00;start_type=ATG;rbs_motif=AGGAG;rbs_spacer=5-10bp;gc_cont=0.543
Scores for complete sequence (score includes all domains):
--- full sequence --- --- best 1 domain --- -#dom-
E-value score bias E-value score bias exp N Model Description
------- ------ ----- ------- ------ ----- ---- -- -------- -----------
[No hits detected that satisfy reporting thresholds]
Domain annotation for each model (and alignments):
[No targets detected that satisfy reporting thresholds]
Internal pipeline statistics summary:
-------------------------------------
Query sequence(s): 1 (264 residues searched)
Target model(s): 641 (202466 nodes)
Passed MSV filter: 18 (0.0280811); expected 12.8 (0.02)
Passed bias filter: 18 (0.0280811); expected 12.8 (0.02)
Passed Vit filter: 1 (0.00156006); expected 0.6 (0.001)
Passed Fwd filter: 0 (0); expected 0.0 (1e-05)
Initial search space (Z): 641 [actual number of targets]
Domain search space (domZ): 0 [number of targets reported over threshold]
# CPU time: 0.02u 0.02s 00:00:00.04 Elapsed: 00:00:00.03
# Mc/sec: 1357.56
//
Query: NODE_1_length_300803_cov_11.207433_2 [L=184]
Description: # 945 # 1496 # 1 # ID=1_2;partial=00;start_type=ATG;rbs_motif=GGA/GAG/AGG;rbs_spacer=5-10bp;gc_cont=0.431
the second input named (all.hmm.ps.len) is under this form whoever the third pearl command calls him.
BM10.hmm 28
CBM11.hmm 163
I did not understand where the problem is knowing that the script in general aims to create an array to clearly read the input (hmmscan-file).
thank you very much
So you have this error:
Illegal division by zero at -e line 14, <> line 1
And you say that line 14 is this:
$r=($a[3]-$a[2])/$b{$a[1]};
Well, there's only one division in that line of code, so it seems clear that when you're executing that line $b{$a[1]} contains zero (but note that because you don't have use warnings in your code, it's possible that it might also contain an empty string or undef and that's being silently converted to a zero).
So as the programmer, your job is to trace through your code and find out where these values are being set and what might be causing it not to contain a value that can be used in your division.
I would be happy to help work out what the problem is, but for a few problems:
Your program reads from two input files and you've only given us one.
Your variables all have single-letter names, making it impossible to understand what they are for.
Your use of a string of command-line programs is bizarre and renders your code pretty much unreadable.
If you want help (not just from here, but from any forum), your first move should be to make your code as readable as possible. It's more than likely that by doing that, you'll find the problem yourself.
But your current habit of reposting the question with incremental changes is winning you no friends here.
Update: One idea might be to rewrite your line as something like this:
if (exists $b{$a[1]}) {
$r=($a[3]-$a[2])/$b{$a[1]};
} else {
warn "Hey, I can't find the key '$a[1]' in the hash \%b\n";
}
I have some array (export from csv file) with ~ 100k lines . File size is ~ 22mb on disk
All i need is to find line with some data, process it and load to mssql (i need to sync csv data with mssql)
Problem is that search tooks almost 1 second (~ TotalMilliseconds : 655,0788)!
$csv.Where({$_.'Device UUID' -eq 'lalala'})
Any way just to speed it up?
Load all 100K rows into a hashtable, using the Device UUID property as the key - this will make it much faster to locate a row than by iterating the whole array with .Where({...}):
$deviceTable = #{}
Import-Csv .\path\to\device_list.csv |ForEach-Object {
$deviceTable[$_.'Device UUID'] = $_
}
This will now take significantly less than 1 second:
$matchingDevice = $deviceTable['lalala']
If you only need one or a few lookups, you can consider the following alternative to Mathias R. Jessen's helpful answer. Note that, like Mathias' solution, it requires reading all rows into memory at once:
# Load all rows into memory.
$allRows = Import-Csv file.csv
# Get the *index* of the row with the column value of interest.
# Note: This lookup is case-SENSITIVE.
$rowIndex = $allRows.'Device UUID'.IndexOf('lalala')
# Retrieve the row of interest by index, if found.
($rowOfInterest = if ($rowIndex -ne -1) { $allRows[$rowIndex] })
Once the rows are loaded into memory (as [pscustomobject] instances, which itself won't be fast), the array lookup - via member-access enumeration - is reasonably fast, thanks to .NET performing the (linear) array search, using the System.Array.IndexOf() method.
The problem with your .Where({ ... }) approach is that iteratively calling a PowerShell script block ({ ... }) many times is computationally expensive.
It comes down to the following trade-off:
Either: Spend more time up front to build up a data structure ([hashtable]) that allows efficient lookup (Mathias' answer)
Or: Read the file more quickly, but spend more time on each lookup (this answer).
Playing around with sqlite shell.
'Device UUID' | set-content file.csv
1..2200kb | % { get-random } | add-content file.csv # 1.44 sec, 25mb
'.mode csv
.import file.csv file' | sqlite3 file # 2.92 sec, 81mb
# last row
'select * from file where "device uuid" = 2143292650;' | sqlite3 file
# 'select * from file where "device uuid" > 2143292649 and "device uuid" < 2143292651;' | sqlite3 file
2143292650
(history)[-1] | % { $_.endexecutiontime - $_.startexecutiontime }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 570
Ticks : 5706795
TotalDays : 6.60508680555556E-06
TotalHours : 0.000158522083333333
TotalMinutes : 0.009511325
TotalSeconds : 0.5706795
TotalMilliseconds : 570.6795
# 34 ms after this:
# 'create index deviceindex on file("device uuid");' | sqlite3 file
# with ".timer on", it's 1ms, after the table is loaded
PowerShell scripts sometimes pipe to %{0}:
[byte[]]$bytes = 0..65535 | %{0};
I know % is an alias for ForEach-Object and $_ represents the current pipeline object.
Is this solely to avoid output? Isn’t there a smarter way of doing it?
The pipeline on the right-hand side simply outputs the integer 0 65536 times - which when cast to byte[] produces a byte array of length 65536 with all values initialized to 0.
You could also have done:
[byte[]]$bytes = ,0 * 65536
As PetSerAl hints at, this is unnecessary since arrays of numerical value types initialize all items to 0 anyways, meaning that simply creating a new array, like so:
# using the new constructor keyword, PowerShell version > 5,
[byte[]]$bytes = [byte[]]::new(65536)
# using New-Object, PowerShell version > 2
[byte[]]$bytes = New-Object 'byte[]' 65536
would also have given you the exact same result
I have a dump file from a w3c process that I need to analyze.
According to a "!DumpHeap -type HttpRequest", there are currently some three thousand active connections to the server.
The question is if it is possible to get the requested URLs of these connections? I would really like to avoid doing a !do for each object to find the reference of the "url" property..
.foreach (object {!DumpHeap -type System.Web.HttpRequest -short}) { !do ${object} }
This will dump every HttpRequest. The URL is a bit deeper down. First you have to find the offset of the _url property:
MT Field Offset Type VT Attr Value Name
000007feedc1cc70 4000d7d 90 System.Uri 0 instance 00000000025f2020 _url
In this case (64 bit) it's at offset 0x90. To dump all the Uri objects, replace the !do ${object} by !do poi(${object}+90). But still this is not the URL, so let's see:
MT Field Offset Type VT Attr Value Name
000007feeeaa68f0 400161c 8 System.String 0 instance 00000000025f1e18 m_String
000007feeeaa68f0 400161d 10 System.String 0 instance 0000000000000000 m_originalUnicodeString
At offset 0x8, the URI has a string and at 0x10 another string. Again we add the offset, so exchange !do poi(${object}+90) by !do poi(poi(${object}+90)+8) (or +10). This will print the .NET string object with all fields. If you want the pure string, do it once again:
MT Field Offset Type VT Attr Value Name
000007feeeaab318 4000104 c System.Char 1 instance 68 m_firstChar
This time we're not using !do any more, because we're on raw bits and bytes and dump a unicode string with du poi(poi(${object}+90)+8)+c. The total command for all HttpRequests is then:
.foreach (object {!DumpHeap -type System.Web.HttpRequest -short}) { du poi(poi(${object}+90)+8)+c }
Some variant of the .foreach command, e.g.
.foreach (object {!sos.DumpHeap -type System.String -short}) { !sos.DumpObj object }
Why am I getting a Invalid parameter poi(adr+4) when I run the following command in WinDbg while debugging a dump file?
.foreach ( adr { !dumpheap -mt 66df13d4 -short } ) { !do poi(adr+4); }
The following shows that the value of adr is getting populated just fine.
.foreach ( adr { !dumpheap -mt 66df13d4 -short } ) { .echo adr; }
I want to get the contents of a .NET string variable that is stored at the 4th offset of a System.Web.Caching.CacheEntry object.
You need to have spaces around adr or use ${adr}. This is documented in MSDN
Note When the string Variable appears within OutCommands, it must be
surrounded by spaces. If it is adjacent to any other text -- even a
parenthesis -- it will not be replaced by the current token value,
unless you use the ${ } (Alias Interpreter) token.