I have below PowerShell script which generate some set of CSV files from large CSV files
$BatchNr = 1
Import-Csv -Path .\Master.csv |Create-Batch -Size 50 |ForEach-Object {
$_ |ForEach-Object {
$_ # do something with each item in the batch of 50
} |Export-Csv ".\Batch$BatchNr.csv"
$BatchNr++
}
And results will be
Batch1.CSV,
Batch2.CSV and so on
Now I have another powershell script which do some operations with this Batch1.CSV,
Batch2.CSV. example below
$Users = Import-Csv -Path "Batch1.CSV"
But now i have to do it manually take each Batch. How can call this files from my PS1 file as soon as file create it, PS2 take files and execute it and after 15 minutes next files like this
Change the second script to accept the CSV path as a parameter:
# script2.ps1
param(
[string]$LiteralPath
)
$Users = Import-Csv -LiteralPath $LiteralPath
# ...
Then change the first script to pass the next CSV name to it:
$_ |ForEach-Object {
$_ # do something with each item in the batch of 50
} |Export-Csv ".\Batch$BatchNr.csv"
.\script2.ps1 -LiteralPath ".\Batch$BatchNr.csv"
$BatchNr++
Related
I got a powershell query, it works fine for smaller amount of data but i am trying to run my CSV against a folder which has multiple folders and files within. Folder size is nearly 800GB and 180 folders within.
I want to see if the file exists in the folder, I can manually search the files within Windows and does not take to long to return a result but my CSV has 3000 rows and i do not wish to do this for 3000 rows. My script works fine for a smaller amount of data.
The script has been running for 6 days and it has not generated a file with data as of yet. it is 0KB and I am running it via task scheduler.
Script is below.
$myFolder = Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction
SilentlyContinue -Force
$myCSV = Import-Csv -Path 'C:\Test\differences.csv' | % {$_.'name' -replace "\\", ""}
$compare = Compare-Object -ReferenceObject $myCSV -DifferenceObject $myFolder
Write-Output "`n_____MISSING FILES_____`n"
$compare
Write-Output "`n_____MISSING FILES DETAILS____`n"
foreach($y in $compare){
if($y.SideIndicator -eq "<="){
write-output "$($y.InputObject) Is present in the CSV but not in Missing folder."
}
}
I then created another script which runs the above script and contains an out file command and runs with Task scheduler.
C:\test\test.ps1 | Out-File 'C:\test\Results.csv'
is there a better way of doing this?
Thanks
is there a better way of doing this?
Yes!
Add each file name on disk to a HashSet[string]
the HashSet type is SUPER FAST at determining whether it contains a
specific value or not, much faster than Compare-Object
Loop over your CSV records, check if each file name exists in the set from step 1
# 1. Build our file name index using a HashSet
$fileNames = [System.Collections.Generic.HashSet[string]]::new()
Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction
SilentlyContinue -Force |ForEach-Object {
[void]$fileNames.Add($_.Name)
}
# 2. Check each CSV record against the file name index
Import-Csv -Path 'C:\Test\differences.csv' |ForEach-Object {
$referenceName = $_.name -replace '\\'
if(-not $fileNames.Contains($referenceName)){
"${referenceName} is present in CSV but not on disk"
}
}
Another option is to use the hash set from step 1 in a Where-Object filter:
$csvRecordsMissingFromDisk = Import-Csv -Path 'C:\Test\differences.csv' |Where-Object { -not $fileNames.Contains($_) }
I have narrowed down a little more what exactly my end game is.
I have pre-created the file that I want the results to write to.
Here is a rough script of what I want to do:
$computers = Get-content "C:\users\nicholas.j.nedrow\desktop\scripts\lists\ComputerList.txt"
# Ping all computers in ComputerList.txt.
# Need WinEst_Computers.csv file created with 3 Columns; Computer Name | Online (Y/N) | Does File Exist (Y/N)
$output = foreach ($comp in $computers) {
$TestConn = Test-connection -cn $comp -BufferSize 16 -Count 1 -ea 0 -quiet
if ($TestConn -match "False")
{
#Write "N" to "Online (Y/N)" Column in primary output .csv file
}
if ($TestConn -match "True")
{
#Write "Y" to "Online (Y/N)" Column in primary output .csv file
}
#For computers that return a "True" ping value:
#Search for WinEst.exe application on C:\
Get-ChildItem -Path "\\$comp\c$\program files (x86)\WinEst.exe" -ErrorAction SilentlyContinue |
if ("\\$comp\c$\program files (x86)\WinEst.exe" -match "False")
{
#Write "N" to "Does File Exist (Y/N)" Column in primary output .csv file
}
if ("\\$comp\c$\program files (x86)\WinEst.exe" -match "True")
{
#Write "Y" to "Does File Exist (Y/N)" Column in primary output .csv file
}
Select #{n='ComputerName';e={$comp}},Name
}
$output | Out-file "C:\users\nicholas.j.nedrow\desktop\scripts\results\CSV Files\WinEst_Computers.csv"
What I need help with is the following:
How to get each result to either write to the appropriate line (I.e. computername, online, file exist?) or would it be easier to do one column at a time;
--Write all PC's to Column A
--Ping each machine and record results in Column B
--Search each machine for the .exe and record results.
Any suggestions? Sorry I keep changing things. Just trying to figure out the best way to do this.
You are using the foreach command, which has a syntax foreach ($itemVariable in $collectionVariable) { }. If $computer is your collection, then your current item cannot also be $computer inside your foreach.
Get-Item does not return a property computerName. Therefore you cannot explicitly select it with Select-Object. However, you can use a calculated property to add a new property to the custom object that Select-Object outputs.
If your CSV file has a row of header(s), it is simpler to use Import-Csv to read the file. If it is just a list of computer names, then Get-Content works well.
If you are searching for a single file and you know the exact path, then just stick with -Path or -LiteralPath and forget -Include. -Include is not intuitive and isn't explained well in the online documentation.
If you are piping output to Export-Csv using a single pipeline, there's no need for -Append unless you already have an existing CSV with data you want to retain. However, if you choose to pipe to Export-Csv during each loop iteration, -Append would be necessary to retain the output.
Here is some updated code using the recommendations:
$computers = Get-content "C:\users\nicholas.j.nedrow\desktop\scripts\lists\ComputerList.txt"
$output = foreach ($comp in $computers) {
Get-Item -Path "\\$comp\c$\program files (x86)\WinEst.exe" -ErrorAction SilentlyContinue |
Select #{n='ComputerName';e={$comp}},Name
}
$output | Export-Csv -Path "C:\users\nicholas.j.nedrow\desktop\scripts\results\CSV Files\WinEst_Computers.csv" -NoType
I try to create a powershell script, to perform a few steps:
In a specific folder, I put a .xlsx file, it converts it to csv. Until now I got this:
$ErrorActionPreference = 'Stop'
Function Convert-CsvInBatch
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$Folder
)
$ExcelFiles = Get-ChildItem -Path $Folder -Filter *.xlsx -Recurse
$excelApp = New-Object -ComObject Excel.Application
$excelApp.DisplayAlerts = $false
$ExcelFiles | ForEach-Object {
$workbook = $excelApp.Workbooks.Open($_.FullName)
$csvFilePath = $_.FullName -replace "\.xlsx$", ".csv"
$workbook.SaveAs($csvFilePath, [Microsoft.Office.Interop.Excel.XlFileFormat]::xlCSV)
$workbook.Close()
}
# Release Excel Com Object resource
$excelApp.Workbooks.Close()
$excelApp.Visible = $true
Start-Sleep 5
$excelApp.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excelApp) | Out-Null
}
#
# 0. Prepare the folder path which contains all excel files
$FolderPath = "C:\exacthpath"
Convert-CsvInBatch -Folder $FolderPath
The columns in the file, are still there, so I want to remove them, and insert a ';' instead, like:
H;1;43;185;
At this point I'm stuck. I can import it into Powershell like:
Import-Csv -Path 'C:\folder\filename.csv' | ForEach-Object {
$_
}
I get this look, and the most important task is here, in the first row only:
H;1;43;185;
This should be modified into:
H;01;43;185
the rest should be left untouched.
After I need to export back it into a CSV file, like:
Export-Csv -Path 'C:\folder\modified_filename.csv'
But this whole process should be inserted in one single powershell script, which performs the above steps on it's own. So in short:
identifies any .xlsx file - regardless of it's name
convers it into .csv
modifies the outlook of the document, to separate the columns with a ";"
modify the first line to have 'H;01;43;185' - this is a static line, it will always look like this
save the created file as a final .csv file
Can you help me somehow to include/optimize the above scripts and let powershell perform the modification too? Example content of a file like this (final look) Usually it includes more 1000+ lines:
H;01;43;185
D;111;3;1042;2
D;222;3;1055;3
D;333;3;1085;1
T;3;;;
Any help is highly appreciated.
Regards,
Armin
If as you say in your comment, your Excel already creates a csv with the semi-colon as delimiter, you can do this inside the loop, just below $workbook.Close()
# read the file created by Excel as string array
$data = Get-Content $csvFilePath
# overwrite the file with just the new header
Set-Content -Path $csvFilePath -Value 'H;01;43;185'
# add the rest of the data to the file
$data[1..($data.Count -1)] | Add-Content -Path $csvFilePath
P.S. I would delete the lines
$excelApp.Visible = $true
Start-Sleep 5
because I don't see the need to have Excel show itself and pause the function for 5 seconds.. Instead, have Excel not show at all so it will work a lot faster by adding
$excelApp.Visible = $false
right after you have created the $excelApp
I'm parsing a CSV file to get the names of folders which I need to copy to another location. Because there are hundreds of them, I need to select the first 10 or so and run the copy routine but to avoid copying them again I'm removing them from the list and saving the file.
I'll run this on a daily scheduled task to avoid having to wait for the folders to finish copying. I'm having a problem using the 'Select' and 'Skip' options in the code (see below), if I remove those lines the folders are copied (I'm using empty folders to test) but if I have them in, then nothing happens when I run this in PowerShell.
I looked around in other questions about similar issues but did not find anything that answers this particular issue selecting and skipping rows in the CSV.
$source_location = 'C:\Folders to Copy'
$folders_Needed = gci $source_location
Set-Location -Path $source_location
$Dest = 'C:\Transferred Folders'
$csv_name = 'C:\List of Folders.csv'
$csv_Import = Get-Content $csv_name
foreach($csv_n in $csv_Import | Select-Object -First 3){
foreach ($folder_Tocopy in $folders_Needed){
if("$folder_Tocopy" -contains "$csv_n"){
Copy-Item -Path $folder_Tocopy -Destination $Dest -Recurse -Verbose
}
}
$csv_Import | Select-Object -Skip 3 | Out-File -FilePath $csv_name
}
It should work with skip/first as in your example, but I cannot really test it without your sample data. Also, it seems wrong that you write the same output to the csv file at every iteration of the loop. And I assume it's not a csv file but actually just a plain text file, a list of folders? Just folder names or full paths? (I assume the first.)
Anyways, here is my suggested update to the script (see comments):
$source_location = 'C:\Folders to Copy'
$folders_Needed = Get-ChildItem $source_location
$Dest = 'C:\Transferred Folders'
$csv_name = 'C:\List of Folders.csv'
$csv_Import = #(Get-Content $csv_name)
# optional limit
# set this to $csv_Import.Count if you want to copy all folders
$limit = 10
# loop over the csv entries
for ($i = 0; $i -lt $csv_Import.Count -and $i -lt $limit; $i++) {
# current line in the csv file
$csv_n = $csv_Import[$i]
# copy the folder(s) which name matches the csv entry
$folders_Needed | where {$_.Name -eq $csv_n} | Copy-Item -Destination $Dest -Recurse -Verbose
# update the csv file (skip all processed entries)
$csv_Import | Select-Object -Skip ($i + 1) | Out-File -FilePath $csv_name
}
I have put together a script inspired from a number of sources. The purpose of the powershell script is to scan a directory for files (.SQL), copy all of it to a new directory (retain the original), and scan each file against a list file (CSV format - containing 2 columns: OldValue,NewValue), and replace any strings that matches. What works: moving, modifying, log creation.
What doesn't work:
Recording in the .log for the changes made by the script.
Sample usage: .\ConvertSQL.ps1 -List .\EVar.csv -Files \SQLFiles\Rel_1
Param (
[String]$List = "*.csv",
[String]$Files = "*.sql"
)
function Get-TimeStamp {
return "[{0:dd/MM/yyyy} {0:HH:mm:ss}]" -f (Get-Date)
}
$CustomFiles = "$Files\CUSTOMISED"
IF (-Not (Test-Path $CustomFiles))
{
MD -Path $CustomFiles
}
Copy-Item "$Files\*.sql" -Recurse -Destination "$CustomFiles"
$ReplacementList = Import-Csv $List;
Get-ChildItem $CustomFiles |
ForEach-Object {
$LogFile = "$CustomFiles\$_.$(Get-Date -Format dd_MM_yyyy).log"
Write-Output "$_ has been modified on $(Get-TimeStamp)." | Out-File "$LogFile"
$Content = Get-Content -Path $_.FullName;
foreach ($ReplacementItem in $ReplacementList)
{
$Content = $Content.Replace($ReplacementItem.OldValue, $ReplacementItem.NewValue)
}
Set-Content -Path $_.FullName -Value $Content
}
Thank you very much.
Edit: I've cleaned up a bit and removed my test logging files.
Here's the snippet of code that I've been testing with little success. I put the following right under $Content= Content.Replace($ReplacementItem.OldValue, $ReplacementItem.NewValue)
if ( $_.FullName -like '*TEST*' ) {
"This is a test." | Add-Content $LogFile
}
I've also tried to pipe out the Set-Content using Out-File. The outputs I end up with are either a full copy of the contents of my CSV file or the SQL file itself. I'll continue reading up on different methods. I simply want to, out of hundreds to a thousand or so lines, to be able to identify what variables in the SQL has been changed.
Instead of piping output to Add-Content, pipe the log output to: Out-File -Append
Edit: compare the content using the Compare-Object cmdlet and evaluate it's ouput to identify where the content in each string object differs.