I manage database servers and often I have to apply scripts into different servers or databases.
Sometimes these scripts are all saved in a directory and need to be open and run in the target server\database.
As I have been looking at automating this task I came across how Run All PowerShell Scripts In A Directory and also How can I execute a set of .SQL files from within SSMS? and that is exactly what I needed, however I stumbled over a few issues:
I don't know the file names
:setvar path "c:\Path_to_scripts\"
:r $(path)\file1.sql
:r $(path)\file2.sql
I tried to add all .sql files into one big thing, but when I copied from powershell into sql, in many of the procedures that had long lines, the lines got messed up
cls
$Radhe = Get-Content 'D:\apply all scripts to SQLPRODUCTION\*.sql' -Raw
$Radhe.Count
$Radhe.LongLength
$Radhe
If I could read all the files in that specific folder and save them all into a single the_scripts_to_run.sql file, without changing the line endings, that would be perfect.
I don't need to use get-content or any command in particular, I just would like to get all my scripts into a big single script with everything in it, without changes.
How can I achieve that?
I even found Merge multiple SQL files into a single SQL file but I want to get it done via powershell.
This should work fine, I'm not sure what you mean by not needing to use Get-Content you could use [System.IO.File]::ReadAllLines( ) or [System.IO.File]::ReadAllText( ) but this should work fine too. Try it and let me know if it works.
$path = "c:\Path_to_scripts"
$scripts = (Get-ChildItem "$path\*.sql" -Recurse -File).FullName
$merged = [system.collections.generic.list[string[]]]::new()
foreach($script in $scripts)
{
$merged.Add((Get-Content $script))
}
$merged | Out-File "$path\mergedscripts.sql"
This is actually much simpler than the proposed solutions. Get-Content takes a list of paths and supports wildcards, so no loop is required.
$path = 'c:\temp\sql'
Set-Content -Path "$path\the_scripts_to_run.sql" -Value (Get-Content -Path "$path\*.sql" -Raw)
Looks like me and #Santiago had the same idea:
Get-ChildItem -Path "$path" -Filter "*.sql" | ForEach-Object -Process {
Get-Content $_.FullName | Out-File $Path\stuff.txt -Append utf8
}
Goal: Update text entry on one line within many files distributed on a server
Summary: As part of an application migration between datacenters the .rdp files on end-user desktops need to be updated to point to the new IP address of their Remote Desktop Server. All the .rdp files reside on Windows servers in a redirected folders SMB share where I have Administrative access.
Powershell experience: minimal. Still trying to wrap my head around the way variables, output and piping work.
Was originally trying to make a single line of powershell code to complete this task but got stuck and had to make script file with the two lines of code below.
-Line 1: Search for all .rdp files in the folder structure and store the full path with file name in a variable. Every file will be checked since the users tend to accidentally change file names, eliminating absolute predictability.
-Line 2: I want to make one pass through all the files to replace only instances of two particular IP addresses with the new address. Then write the changes into the original file.
$Path = ls 'C:\Scripts\Replace-RDP\TESTFILES\' -Include *.rdp -Recurse -Force -ErrorAction SilentlyContinue | foreach fullname
$Path | (Get-Content -Path $Path) -Replace 'IPserver1','newIPserver1' -Replace 'IPserver2','newIPserver2' | Set-Content $Path -Force
Have found most of the solution with Powershell but have a problem with the results. The second line of code when output to the screen changes contents correctly in memory. The content written to file however resulted in the new server IP address being written into ALL rdp files even if the source rdp file's target IP address doesn't match the -Replace criterion.
Text inside a .rdp on the relevant line is:
full address:s:192.168.1.123
changes to:
full address:s:172.16.1.23
Thank you for all assistance in reaching the endpoint. Have spent hours learning from various sites and code snippets.
You need to keep track of each file that you are reading so that you can save changes to that file. Foreach-Object makes this process easy. Inside of the Foreach-Object script block, the current object $_ is the FullName value for each of your files.
$CurrentIP1 = '192\.168\.1\.123'
$CurrentIP2 = '192\.168\.1\.124'
$NewIP1 = '172.16.1.23'
$NewIP2 = '172.16.1.24'
$files = (Get-ChildItem 'C:\Scripts\Replace-RDP\TESTFILES\' -Filter *.rdp -Recurse -Force -File -ErrorAction SilentlyContinue).FullName
$files | Foreach-Object {
if (($contents = Get-Content $_) -match "$CurrentIP1|$CurrentIP2") {
$contents -replace $CurrentIP1,$NewIP1 -replace $CurrentIP2,$NewIP2 |
Set-Content $_
}
}
Note that using the -File switch on Get-ChildItem (alias ls) outputs only files. Since -replace uses regex to do matching, you must backslash escape literal . characters.
I created simple nagios plugin check_log.ps1 to check log file on windows machine. It works in way that make copy content of log and in next time look for specified string in difference between copy of log and original log file.
The problem is that sometimes in random moments check_log.ps1 locks log file so it cause stop of the application which create log file.
Generally plugin use original log file in two places
# compare content of $Logfile and $Oldlog, save diff to $tempdiff
Compare-Object -ReferenceObject (Get-Content -Path $Logfile) -DifferenceObject (Get-Content -Path $Oldlog) | Select-Object -Property InputObject > $tempdiff
# override file $Oldlog using conetent of $Logfile
Copy-Item $Logfile $Oldlog
I make test. In one PS session I run while($true) { [string]"test" >> C:\test\test.log }, in second session I run plugin C:\test\check_log.ps1 C:\test\test.log C:\test\Old_log.log test
I'm not fully sure if my test is correct but I think that Copy-Item command cause problem. When I comment this line in script I don't see any errors in terminals. I tested some custom functions to copy file which I found in internet but I din't find solution for my problem.
Do you have an idea how to make it work fully?
if you think the copy-item is locking the file, try reading the content and then saving it to another location. Something like this:
Get-Content $Logfile | Set-Content $Oldlog
I have a simple PowerShell script that replaces "false" or "true" with "0" or "1":
$InputFolder = $args[0];
if($InputFolder.Length -lt 3)
{
Write-Host "Enter a path name as your first argument" -foregroundcolor Red
return
}
if(-not (Test-Path $InputFolder)) {
Write-Host "File path does not appear to be valid" -foregroundcolor Red
return
}
Get-ChildItem $InputFolder
$content = [System.IO.File]::ReadAllText($InputFolder).Replace("`"false`"", "`"0`"").Replace("`"true`"", "`"1`"").Replace("`"FALSE`"", "`"0`"").Replace("`"TRUE`"", "`"1`"")
[System.IO.File]::WriteAllText($InputFolder, $content)
[GC]::Collect()
This works fine for almost all files I have to amend, with the exception of one 808MB CSV.
I have no idea how many lines are in this CSV, as nothing I have will open it properly.
Interestingly, the PowerShell script will complete successfully when invoked manually via either PowerShell directly or via command prompt.
When this is launched as part of the SSIS package it's required for, that's when the error happens.
Sample data for the file:
"RowIdentifier","DateProfileCreated","IdProfileCreatedBy","IDStaffMemberProfileRole","StaffRole","DateEmploymentStart","DateEmploymentEnd","PPAID","GPLocalCode","IDStaffMember","IDOrganisation","GmpID","RemovedData"
"134","09/07/1999 00:00","-1","98","GP Partner","09/07/1999 00:00","14/08/2009 15:29","341159","BRA 871","141","B83067","G3411591","0"
Error message thrown:
I'm not tied to PowerShell - I'm open to other options. I had a cribbed together C# script previously, but that died on small files than this - I'm no C# developer, so was unable to debug it at all.
Any suggestions or help gratefully received.
Generally, avoiding read large files all at once, as you can run out of memory, as you've experienced.
Instead, process text-based files line by line - both reading and writing.
While PowerShell generally excels at line-by-line (object-by-object) processing, it it is slow with files with many lines.
Using the .NET Framework directly - while more complex - offers much better performance.
If you process the input file line by line, you cannot directly write back to it and must instead write to a temporary output file, which you can replace the input file with on success.
Here's a solution that uses .NET types directly for performance reasons:
# Be sure to use a *full* path, because .NET typically doesn't have the same working dir. as PS.
$inFile = Convert-Path $Args[0]
$tmpOutFile = [io.path]::GetTempFileName()
$tmpOutFileWriter = [IO.File]::CreateText($tmpOutFile)
foreach ($line in [IO.File]::ReadLines($inFile)) {
$tmpOutFileWriter.WriteLine(
$line.Replace('"false"', '"0"').Replace('"true"', '"1"').Replace('"FALSE"', '"0"').Replace('"TRUE"', '"1"')
)
}
$tmpOutFileWriter.Dispose()
# Replace the input file with the temporary file.
# !! BE SURE TO MAKE A BACKUP COPY FIRST.
# -WhatIf *previews* the move operation; remove it to perform the actual move.
Move-Item -Force -LiteralPath $tmpOutFile $inFile -WhatIf
Note:
UTF-8 encoding is assumed, and the rewritten file will not have a BOM. You can change this by specifying the desired encoding to the .NET methods.
As an aside: Your chain of .Replace() calls on each input line can be simplified as follows, using PowerShell's -replace operator, which is case-insensitive, so only 2 replacements are needed:
$line -replace '"false"', '"0"' -replace '"true"', '"1"'
However, while that is shorter to write, it is actually slower than the .Replace() call chain, presumably because -replace is regex-based, which incurs extra processing.
You could read the file Per line with get-content -readcount, Out-file a temp file, then delete old file and rename-item the temp file the old files name.
Small things that would need fixing. This will add a new empty line at end of file. This will change the encoding. You could try and get the current file encoding and set the encoding on the Out-file -encoding
function Replace-LargeFilesInFolder(){
Param(
[string]$DirectoryPath,
[string]$OldString,
[string]$NewString,
[string]$TempExtention = "temp",
[int]$LinesPerRead = 500
)
Get-ChildItem $DirectoryPath -File | %{
$File = $_
Get-Content $_.FullName -ReadCount $LinesPerRead |
%{
$_ -replace $OldString, $NewString |
out-file "$($File.FullName).$($TempExtention)" -Append
}
Remove-Item $File.FullName
Rename-Item "$($File.FullName).$($TempExtention)" -NewName $($File.FullName)
}
}
Replace-LargeFilesInFolder -DirectoryPath C:\TEST -LinesPerRead 1 -OldString "a" -NewString "5"
I am trying to do find and replace in a file using following approach.
Function Find-Replace ($FileFullpath, $FindString, $ReplacementString) {
Get-Content $FileFullpath |
Foreach-Object {$_ -replace $FindString, $ReplacementString } |
Set-Content $FileFullpath
}
Find-Replace "c:\program files (x86)\MyProj\web.config" $OldServiceName $NewServiceName
But i am always getting error.
Set-Content : The process cannot access the file 'c:\program files
(x86)\MyProj\web.config' because it is being used by another process.
The file is not opened any where. I think Get-content is yet to release the file.
Why it happens ? How to do find and replace in the same file without issue?
You can't read and write to the same file while it's open, Get-Content opens the file for reading and in the same time Set-Content tries to write to it. Put the Get-Conetnt call in parentheses, it will open the file, read it's content and close it.
(Get-Content $FileFullpath) | ...