I want to be able to use my function like:
Get-Process | export-xsl;
Right now I'm manually calling Get-Process inside my function:
function export-xsl() {
$path = "{0}\Downloads\test.csv" -f $home;
Get-Process | export-csv -Path $path -NoTypeInformation
Invoke-item $path;
}
The examples I found seem to iterate on each item, which I believe will create multiple .csv files.
I tried, but this creates the CSV dozens of times, once per iteration. I'm trying to get the entire object as one CSV file.
function export-xsl() {
process {
$path = "{0}\Downloads\test.csv" -f $home;
$_ | export-csv -Path $path -NoTypeInformation
Invoke-item $path;
}
}
function export-xsl {
$path = "{0}\Downloads\test.csv" -f $home;
$input | export-csv -Path $path -NoTypeInformation;
Invoke-item $path;
}
$input will allow you to pipe all the data at once, instead of many iterations.
Related
I am trying to construct a script that moves through specific folders and the log files in it, and filters the error codes. After that it passes them into a new file.
I'm not really sure how to do that with for loops so I'll leave my code bellow.
If someone could tell me what I'm doing wrong, that would be greatly appreciated.
$file_name = Read-Host -Prompt 'Name of the new file: '
$path = 'C:\Users\user\Power\log_script\logs'
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
param([string]$zipfile, [string]$outpath)
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
if ([System.IO.File]::Exists($path)) {
Remove-Item $path
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
} else {
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
}
$folder = Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles'
$files = foreach($logfolder in $folder) {
$content = foreach($line in $files) {
if ($line -match '([ ][4-5][0-5][0-9][ ])') {
echo $line
}
}
}
$content | Out-File $file_name -Force -Encoding ascii
Inside the LogFiles folder are three more folders each containing log files.
Thanks
Expanding on a comment above about recursing the folder structure, and then actually retrieving the content of the files, you could try something line this:
$allFiles = Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' -Recurse
# iterate the files
$allFiles | ForEach-Object {
# iterate the content of each file, line by line
Get-Content $_ | ForEach-Object {
if ($_ -match '([ ][4-5][0-5][0-9][ ])') {
echo $_
}
}
}
It looks like your inner loop is of a collection ($files) that doesn't yet exist. You assign $files to the output of a ForEach(...) loop then try to nest another loop of $files inside it. Of course at this point $files isn't available to be looped.
Regardless, the issue is you are never reading the content of your log files. Even if you managed to loop through the output of Get-ChildItem, you need to look at each line to perform the match.
Obviously I cannot completely test this, but I see a few issues and have rewritten as below:
$file_name = Read-Host -Prompt 'Name of the new file'
$path = 'C:\Users\user\Power\log_script\logs'
$Pattern = '([ ][4-5][0-5][0-9][ ])'
if ( [System.IO.File]::Exists( $path ) ) { Remove-Item $path }
Expand-Archive 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
Select-String -Path 'C:\Users\user\Power\log_script\logs\LogFiles\*' -Pattern $Pattern |
Select-Object -ExpandProperty line |
Out-File $file_name -Force -Encoding ascii
Note: Select-String cannot recurse on its own.
I'm not sure you need to write your own UnZip function. PowerShell has the Expand-Archive cmdlet which can at least match the functionality thus far:
Expand-Archive -Path <SourceZipPath> -DestinationPath <DestinationFolder>
Note: The -Force parameter allows it to over write the destination files if they are already present. which may be a substitute for testing if the file exists and deleting if it does.
If you are going to test for the file that section of code can be simplified as:
if ( [System.IO.File]::Exists( $path ) ) { Remove-Item $path }
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
This is because you were going to run the UnZip command regardless...
Note: You could also use Test-Path for this.
Also there are enumerable ways to get the matching lines, here are a couple of extra samples:
Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' |
ForEach-Object{
( Get-Content $_.FullName ) -match $Pattern
# Using match in this way will echo the lines that matched from each run of
# Get-Content. If nothing matched nothing will output on that iteration.
} |
Out-File $file_name -Force -Encoding ascii
This approach will read the entire file into an array before running the match on it. For large files it may pose a memory issue, however it enabled the clever use of -match.
OR:
Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' |
Get-Content |
ForEach-Object{ If( $_ -match $Pattern ) { $_ } } |
Out-File $file_name -Force -Encoding ascii
Note: You don't need the alias echo or its real cmdlet Write-Output
UPDATE: After fuzzing around a bit and trying different things I finally got it to work.
I'll include the code below just for demonstration purposes.
Thanks everyone
$start = Get-Date
"`n$start`n"
$file_name = Read-Host -Prompt 'Name of the new file: '
Out-File $file_name -Force -Encoding ascii
Expand-Archive -Path 'C:\Users\User\Power\log_script\logs.zip' -Force
$i = 1
$folders = Get-ChildItem -Path 'C:\Users\User\Power\log_script\logs\logs\LogFiles' -Name -Recurse -Include *.log
foreach($item in $folders) {
$files = 'C:\Users\User\Power\log_script\logs\logs\LogFiles\' + $item
foreach($file in $files){
$content = Get-Content $file
Write-Progress -Activity "Filtering..." -Status "File $i of $($folders.Count)" -PercentComplete (($i / $folders.Count) * 100)
$i++
$output = foreach($line in $content) {
if ($line -match '([ ][4-5][0-5][0-9][ ])') {
Add-Content -Path $file_name -Value $line
}
}
}
}
$end = Get-Date
$time = [int]($end - $start).TotalSeconds
Write-Output ("Runtime: " + $time + " Seconds" -join ' ')
I am trying to get complete a request of amount of files in a directory and then the total size of that directory. I've come up with this:
Get-Content -Path C:\Users\$USERNAME%\Documents\list.txt |
Foreach-Object {
cd $_
Write-Host $_
(Get-ChildItem -Recurse -File | Measure-Object).Count
(ls -r|measure -sum Length).Sum
}
The txt file has contents such as:
\\directory\on\network\1
\\directory\on\network\also
Ultimately I need this in a spreadsheet, but I am failing with formatting. As is, it outputs straight to powershell, so with thousands of directories this isn't ideal. I've tried exporting to CSV but it overwrites the CSV with each result, and when I tried setting the function equal to a variable array and then exporting that, it simply output a blank file.
Any assistance with this is appreciated.
In order to export to CSV you will need an object with properties. Your code generates a few values without any structure. Surely the % in your sample code is a typo, it definitely doesn't belong there. It is generally considered bad practice to use aliases in scripts, however you should, at a minimum, keep it consistent. One line you use Get-ChildItem/Measure-Object and the next use ls/measure. Regardless you don't show your export, so it's hard to help with what we can't see. You also don't need to CD into the directory, it seems it would only slow the script down if anything.
The easiest way I know to create an object is to use the [PSCustomObject] type accelerator.
$infile = "C:\Users\$USERNAME\Documents\list.txt"
$outfile = "C:\some\path\to.csv"
Get-Content -Path $infile |
Foreach-Object {
Write-Host Processing $_
[PSCustomObject]#{
Path = $_
Total = (Get-ChildItem $_ -Recurse -File | Measure-Object).Count
Size = (Get-ChildItem $_ -Recurse -File | Measure-Object -sum Length).Sum
}
} | Export-Csv $outfile -NoTypeInformation
Edit
We should've ran the Get-Childitem call once and then pulled the info out. The first option is in "pipeline" mode can save on memory usage but might be slower. The second puts it all in memory first so it can be much quicker if it's not too large.
Get-Content -Path $infile |
Foreach-Object {
Write-Host Processing $_
$files = Get-ChildItem $_ -Recurse -File | Measure-Object -sum Length
[PSCustomObject]#{
Path = $_
Total = $files.Count
Size = $files.Sum
}
} | Export-Csv $outfile -NoTypeInformation
or
$results = foreach($folder in Get-Content -Path $infile)
{
Write-Host Processing $folder
$files = Get-ChildItem $folder -Recurse -File | Measure-Object -sum Length
[PSCustomObject]#{
Path = $folder
Total = $files.Count
Size = $files.Sum
}
}
$results | Export-Csv $outfile -NoTypeInformation
The -append flag in Export-Csv allows you to add to an existing file rather than overwriting.
getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
ForEach-Object { Import-Csv $_.FullName } |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
May be can you export and filter your files one by one and append result into your output file like this :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }
Note: The reason for not using Get-ChildItem ... | Import-Csv ... - i.e., for not directly piping Get-ChildItem to Import-Csv and instead having to call Import-Csv from the script block ({ ... } of an auxiliary ForEach-Object call, is a bug in Windows PowerShell that has since been fixed in PowerShell Core - see the bottom section for a more concise workaround.
However, even output from ForEach-Object script blocks should stream to the remaining pipeline commands, so you shouldn't run out of memory - after all, a salient feature of the PowerShell pipeline is object-by-object processing, which keeps memory use constant, irrespective of the size of the (streaming) input collection.
You've since confirmed that avoiding the aux. ForEach-Object call does not solve the problem, so we still don't know what causes your out-of-memory exception.
Update:
This GitHub issue contains clues as to the reason for excessive memory use, especially with many properties that contain small amounts of data.
This GitHub feature request proposes using strongly typed output objects to help the issue.
The following workaround, which uses the switch statement to process the files as text files, may help:
$header = ''
Get-ChildItem $inputFolder -Filter *.csv | ForEach-Object {
$i = 0
switch -Wildcard -File $_.FullName {
'*workstations*' {
# NOTE: If no other columns contain the word `workstations`, you can
# simplify and speed up the command by omitting the `ConvertFrom-Csv` call
# (you can make the wildcard matching more robust with something
# like '*,workstations,*')
if ((ConvertFrom-Csv "$header`n$_").machine_type -ne 'workstations') { continue }
$_ # row whose 'machine_type' column value equals 'workstations'
}
default {
if ($i++ -eq 0) {
if ($header) { continue } # header already written
else { $header = $_; $_ } # header row of 1st file
}
}
}
} | Set-Content $outputFile
Here's a workaround for the bug of not being able to pipe Get-ChildItem output directly to Import-Csv, by passing it as an argument instead:
Import-Csv -LiteralPath (Get-ChildItem $inputFolder -File -Filter *.csv) |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Note that in PowerShell Core you could more naturally write:
Get-ChildItem $inputFolder -File -Filter *.csv | Import-Csv |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Solution 2 :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8 # modify encoding if necessary
$Delimiter=','
#find header for your files => i take first row of first file with data
$Header = Get-ChildItem -Path $inputFolder -Filter *.csv | Where length -gt 0 | select -First 1 | Get-Content -TotalCount 1
#if not header founded then not file with sise >0 => we quit
if(! $Header) {return}
#create array for header
$HeaderArray=$Header -split $Delimiter -replace '"', ''
#open output file
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
#write header founded
$w.WriteLine($Header)
#loop on file csv
Get-ChildItem $inputFolder -File -Filter "*.csv" | %{
#open file for read
$r = New-Object System.IO.StreamReader($_.fullname, $encoding)
$skiprow = $true
while ($line = $r.ReadLine())
{
#exclude header
if ($skiprow)
{
$skiprow = $false
continue
}
#Get objet for current row with header founded
$Object=$line | ConvertFrom-Csv -Header $HeaderArray -Delimiter $Delimiter
#write in output file for your condition asked
if ($Object.machine_type -eq 'workstations') { $w.WriteLine($line) }
}
$r.Close()
$r.Dispose()
}
$w.close()
$w.Dispose()
You have to read and write to the .csv files one row at a time, using StreamReader and StreamWriter:
$filepath = "C:\Change\2019\October"
$outputfile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8
$files = Get-ChildItem -Path $filePath -Filter *.csv |
Where-Object { $_.machine_type -eq 'workstations' }
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
$skiprow = $false
foreach ($file in $files)
{
$r = New-Object System.IO.StreamReader($file.fullname, $encoding)
while (($line = $r.ReadLine()) -ne $null)
{
if (!$skiprow)
{
$w.WriteLine($line)
}
$skiprow = $false
}
$r.Close()
$r.Dispose()
$skiprow = $true
}
$w.close()
$w.Dispose()
get-content *.csv | add-content combined.csv
Make sure combined.csv doesn't exist when you run this, or it's going to go full Ouroboros.
I am trying to do a simple script that pulls in the name of the file and the contents of said text file into a CSV file. I am able to pull in all of the information well enough but it's not splitting up into different columns in the CSV file. When I open up the CSV file in excel everything is in the first column, and I need the two bits of information separated into separate columns. So far my working code is as follows:
$Data = Get-ChildItem -Path c:path -Recurse -Filter *.txt |
where {$_.lastwritetime -gt(Get-Date).addDays`enter code here`(-25)}
$outfile = "c:path\test.csv"
rm $outfile
foreach ($info in $Data) {
$content = Get-Content $info.FullName
echo "$($info.BaseName) , $content" >> $outfile
}
I figured out how to seperate the information by rows but I need it by columns. I'm new to powershell and can't seem to get past this little speed bump. Any input would be greatly appreciated. Thanks in advance!
Output:
Itm# , TextContent
Itm2 , NextTextContent
What I need:
Itm# | Text Content |
Itm2 | NextTextContent |
Except for a few syntactical errors your code appears to be working as expected. I worry if you are having issues in Excel with you text import. I touched up your code a bit but it is functionally the same as what you had.
$Data = Get-ChildItem -Path "C:\temp" -Recurse -Filter *.txt |
Where-Object {$_.LastWriteTime -gt (Get-Date).addDays(-25)}
$outfile = "C:\temp\test.csv"
If(Test-Path $outfile){Remove-Item $outfile -Force}
foreach ($info in $Data) {
$content = Get-Content $info.FullName
"$($info.BaseName) , $content" | Add-Content $outfile
}
I don't know what version of Excel you have but look for the text import wizard.
Do you mean something like this?
Get-ChildItem -Path C:\path -Recurse -Filter *.txt |
Where-Object { $_.LastWriteTime -gt (Get-Date).AddDays(-25) } | ForEach-Object {
New-Object PSObject -Property #{
"Itm#" = $_.FullName
"TextContent" = Get-Content $_.FullName
} | Select-Object Itm#,TextContent
} | Export-Csv List.csv -NoTypeInformation
Excel will treat the data in csv files which are delimited bij the ; as a single columns.
I always use the -delimiter switch on export-csv or convertto-csv to set this as a delimiter.
I have trouble to use several commands in one Foreach or Foreach-Object loop
My situation is -
I have many text files, about 100.
So they are read Get-ChildItem $FilePath -Include *.txt
Every file's structure is same only key information is different.
Example
User: Somerandomname
Computer: Somerandomcomputer
With -Replace command I remove "User:" and "Computer:" so $User = Somerandomname and $computer = "Somerandomcomputer.
In each circle $user and $Computer with -Append should be written to one file. And then next file should be read.
foreach-object { $file = $_.fullname;
should be used, but I can not figure out the right syntax for it. Could someone help me with it?
Assuming you've defined $FilePath, $user, and/or $computer elsewhere, try something like this.
$files = Get-ChildItem $FilePath\*.txt
foreach ($file in $files)
{
(Get-Content $file) |
Foreach-Object { $content = $_ -replace "User:", "User: $user" ; $content -replace "Computer:", "Computer: $computer" } |
Set-Content $file
}
You can use ; to delimit additional commands in within the Foreach-Object, for example if you wanted to have separate commands for your user and computer name. If you don't enclose the Get-Content cmdlet with parenthesis you will get an error because that process will still have $file open when Set-Content tries to use it.
Also note that with Powershell, strings in double quotes will evaluate variables, so you can put $user in the string to do something like "User: $user" if you so desired.
Try this:
gci $FilePath -Include *.txt | % {
gc $_.FullName | ? { $_ -match '^(?:User|Computer): (.*)' } | % { $matches[1] }
} | Out-File 'C:\path\to\output.txt'
If User and Computer are on separate lines, you need to read the lines two at a time. The ReadCount parameter of Get-Content allows you to do that.
Get-ChildItem $FilePath -Include *.txt `
| Get-Content -ReadCount 2 `
| %{ $user = $_[0] -replace '^User: ', ''; $computer = $_[1] -replace '^Computer: ', ''; "$user $computer" } `
| Out-File outputfile.txt
This makes the assumption that every file contains only lines of the exact form
User: someuser
Computer: somecomputer
User: someotheruser
Computer: someothercomputer
...
If this is not the case, you will need to provide whatever is the exact file format.