I would like to build powershell pipeline like
cmd | transform_a | stdout_and | transform_b | store_variable
^
|
copy input to next consumer and to console
I tried to utilize Tee-Object but without success. I dont want to do this
dir | select -last 5 | tee lastFiveLines | select -first 1
echo $lastFiveLines
altough it works. Instead I want the content to be printed directly.
You can try a foreach-loop and Out-Default or Out-Host to skip the rest of the pipeline (Host is Default output anyways), while also sending the object down the pipeline. Sample:
Get-ChildItem |
Select-Object Name, FullName |
ForEach-Object {
#Send Name-value directly to console (default output)
$_.Name | Out-Default
#Send original object down the pipeline
$_
} |
Select-Object -ExpandProperty FullName | % { Start-sleep -Seconds 1; "Hello $_" }
You can create a filter to make it easy to reuse it.
#Bad filter-name, but fits the question.
filter stdout_and {
#Send Name-value directly to console (default output)
$_.Name | Out-Default
#Send original object down the pipeline
$_
}
Get-ChildItem |
Select-Object Name, FullName |
stdout_and |
Select-Object -ExpandProperty FullName | % { Start-sleep -Seconds 1; "Hello $_" }
Related
There is such a shell command in the chapter "transformational programming" of "The Pragmatic Programmer".
Its function is to list the five files with the most lines in the current directory.
$ find . -type f | xargs wc -l | sort -n | tail -6 | head -5
470 ./debug.pml
470 ./test_to_build.pml
487 ./dbc.pml
719 ./domain_languages.pml
727 ./dry.pml
I'm trying to do the same thing with PowerShell，But it seems too wordy
(Get-ChildItem .\ | ForEach-Object {$_ | Select-Object -Property 'Name', #{label = 'Lines'; expression = {($_ | Get-Content).Length}}} |Sort-Object -Property 'Lines')|Select-Object -Last 5
I believe there will be a simpler way, but I can't think of it.
How to get files with most lines in the current directory by simplest way using PowerShell?
Of course, you don't need to use custom aliases and abbreviations to shorten the length. Although it looks more concise, it loses readability.
Get-Content * | Group-Object PSChildName | Select-Object Count, Name |
Sort-Object Count | Select-Object -Last 5
I finally found my own satisfactory answer!
Used 3 pipeline operators, shell used 5!
What's more, what we get is the object, which can be used for more extensible operations.
I feel better than shell of linux.
dir -file | sort {($_ | gc).Length} | select -l 5
Try either File.ReadLines with Linq or File.ReadAllLines with Count property.
File.ReadLines
Get-ChildItem .\ -File |
Select-Object -Property Name, #{n='Lines'; e= {
[System.Linq.Enumerable]::Count([System.IO.File]::ReadLines($_.FullName))
}
} | Sort-Object -Property 'Lines' -Descending | Select-Object -First 5
File.ReadAllLines
Get-ChildItem .\ -File |
Select-Object -Property Name, #{n='Lines'; e= {
[System.IO.File]::ReadAllLines($_.FullName).Count
}
} | Sort-Object -Property 'Lines' -Descending | Select-Object -First 5
A fast approach would be to use switch -File:
$files = (Get-ChildItem -File ).FullName
$result = foreach ($file in $files) {
$lineCount = 0
switch -File $file {
default { $lineCount++ }
}
[PsCustomObject]#{
File = $file
Lines = $lineCount
}
}
$result | Sort-Object Lines | Select-Object -Last 5
I have the following code.
$summary = . {
while ($true) {
# Generating huge list of psobject
}
} |
Tee-Object -FilePath 'fname.csv' | # Need to process the input objects before writing to CSV
Group-Object -Property xxx | Select Name,Count
However, I need to process the input objects before writing to fname.csv. Is it possible to Tee the object to two pipelines?
I tried
$summary = . {
while ($true) {
# Generating huge list of psobject
}
} |
For-Each {
$_ | ConvertTo-Csv -NoTypeInformation | Out-File -Append 'file.csv'
$_
} |
Group-Object -Property xxx | Select Name,Count
But the headers are repeated every line in file.csv.
I'm not sure what you want to do. Does this help? The objects from get-process get passed to two different pipelines.
get-process cmd | foreach-object { $_ | measure-object
$_ | export-csv -append whatever.csv }
This should do the work in the question. It doesn't Tee to two pipelines, which may be needed for some use cases, though.
$summary = . {
while ($true) {
# Generating huge list of psobject
}
} |
ForEach {
$_ | Export-Csv -NoTypeInformation -Append 'file.csv'
$_
} |
Group-Object -Property xxx | Select Name,Count
Going after #mklement0 's guidance, does the below make it any simpler,
Get-Process excel | Tee-Object -Variable process | group processname | select name, count
$process | Export-Csv "D:\Op_GetProcessExcel.csv" -NoTypeInformation
Previous suggestion
As #AnsgarWiechers pointed out something like the following should work for you,
Get-Process |
group processname |
select name, count |
ConvertTo-Csv -NoTypeInformation |
Tee-Object "C:\Op_GetProcess.csv"
I am using Powershell v4.0 to remove some duplicate files off of our drive and generate some reports. Our hope is no one will care that we're removing duplicate files - but if they do then I can just restore the files to their location.
I create a report showing all duplicates. I create a report of all the files I'll keep. I create a report of all the files I'll erase. And finally I actually make the deletion.
The first command is to get a report of all the duplicate files that I have:
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % { $_.group | select } | Out-File "c:\users\me\desktop\duplicatelist.txt"
The second command is to get a report of all the files I'll keep:
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % { $_.group[0] | select } | Out-File "c:\users\me\desktop\keep.txt"
The third command is to get a report of all the files I'll erase:
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % { $_.group | select -skip 1} | Out-File "c:\users\me\desktop\delete.txt"
The fourth command is to delete all of the duplicate files:
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % { $_.group | select -skip 1} | del
Can I combine multiple commands? It takes eight hours and thirty minutes per iteration of the command, so running it four times doesn't seem like a great solution.
In the least to be able to print out what I'm deleting, and to delete the files at the same step would be a good step forward.
Rather than hashing 4 times, save the hashes to a variable to reuse:
$groupedhashes = ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 }
$groupedhashes | % { $_.group | select } | Out-File "c:\users\me\desktop\duplicatelist.txt"
$groupedhashes | % { $_.group[0] | select } | Out-File "c:\users\me\desktop\keep.txt"
$groupedhashes | % { $_.group | select -skip 1} | Out-File "c:\users\me\desktop\delete.txt"
$groupedhashes | % { $_.group | select -skip 1} | del
Similar to #BenH I'd suggest only getting the files and hashing once, but there's no reason to loop through the whole thing 4 times. Just put all four commands in one ForEach loop.
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % {
$_.group | Out-File "c:\users\me\desktop\duplicatelist.txt"
$_.group[0] | Out-File "c:\users\me\desktop\keep.txt"
$_.group | select -skip 1 | Out-File "c:\users\me\desktop\delete.txt"
$_.group | select -skip 1 | del
}
Edit: Better yet, combine the last two commands with Tee-Object so it comes out to this:
ls *.*| Get-FileHash | group -Property hash | where { $_.count -gt 1 } | % {
$_.group | Out-File "c:\users\me\desktop\duplicatelist.txt"
$_.group[0] | Out-File "c:\users\me\desktop\keep.txt"
$_.group | select -skip 1 | Tee-Object -FilePath "c:\users\me\desktop\delete.txt" -Append | del
}
I've built the following PowerShell script and it doesn't seem to work:
"\\example\examplepath\" | % { $_ | select name, #{n="lines"; e={ get-content
$_.FullName | measure-object -line | Select -expand lines } } } | ft -
Autosize | Out-file c:\counts\result.csv
The script is supposed to get a line count for each file and output them to a CSV. Admittedly there around 140,000 files in the folder. Any ideas?
You are missing the Get-ChildItem cmdlet to retrieve all files. The Foreach-Object (%) cmdlet is obsolete here so I removed it. I also removed the Format-Table cmdlet because you are piping the result to Out-File:
Get-ChildItem "\\example\examplepath\" |
Select-Object name, #{n="lines"; e={ get-content $_.FullName | measure-object -line | Select-Object -expand lines } } |
Out-file c:\counts\result.csv
This powershell code searches the directory and outputs a list of all the files and how old they are to a log file that is parsed buy a different script. all that is working correctly but i also need to keep track of the number of files it found for that dir and the number of files found globally. Thats what the two foreach-Object statements do. but they are staying at 0.
gci -filter *.avi | Select-Object Name, #{Name="Age"; Expression= { (((Get-Date) - $_.CreationTime).Days) }} | Where {$_.Age -ge $daysToKeep} | Out-File -filepath $logFile -append | Foreach-Object {$fileCountCam1++} | Foreach-Object {$fileCount++}
mjolinor's solution is valid, but there's another way (if you can use v3). You can use Tee-Object to write to the file without a loop.
You can also combine your two variable increments into the same script block in the final foreach-object which will speed things up significantly.
gci -filter *.avi |
Select-Object Name, #{Name="Age"; Expression= { (((Get-Date) - $_.CreationTime).Days) }} |
Where {$_.Age -ge $daysToKeep} | Tee-Object -filepath $logFile -append |
Foreach-Object {$fileCountCam1++;$fileCount++}
Out-File is a termnating cmdlet (it doesn't ouput the object to the pipeline), so everything after it isn't getting any input from the pipeline.
See if this works better:
gci -filter *.avi |
Select-Object Name, #{Name="Age"; Expression= { (((Get-Date) - $_.CreationTime).Days) }} |
Where {$_.Age -ge $daysToKeep} |
Foreach-Object {
$_ | Out-File -filepath $logFile -append
$fileCountCam1++
$fileCount++
}