Controlling network speed for any process by powershell? - powershell

I am using powershell to do folder sync. My question is that how does "Copy-Item" use the network bandwidth / speed ? Does it use all available bandwidth? or it is controlled by Windows automatically.
Because my sync process will be performed through vpn, so I would like to control the speed of sync to 500K, is that possible to do it in powershell Like this software does?
function Get-FileMD5 {
Param([string]$file)
$md5 = [System.Security.Cryptography.HashAlgorithm]::Create("MD5")
$IO = New-Object System.IO.FileStream($file, [System.IO.FileMode]::Open)
$StringBuilder = New-Object System.Text.StringBuilder
$md5.ComputeHash($IO) | % { [void] $StringBuilder.Append($_.ToString("x2")) }
$hash = $StringBuilder.ToString()
$IO.Dispose()
return $hash
}
#VARIABLES
$DebugPreference = "continue"
#parameters
$SRC_DIR = 'VPN POINT'
$DST_DIR = 'VPN POINT'
#SCRIPT MAIN
clear
$SourceFiles = GCI -Recurse $SRC_DIR | ? { $_.PSIsContainer -eq $false} #get the files in the source dir.
$SourceFiles | % { # loop through the source dir files
$src = $_.FullName #current source dir file
Write-Debug $src
$dest = $src -replace $SRC_DIR.Replace('\','\\'),$DST_DIR #current destination dir file
if (test-path $dest) { #if file exists in destination folder check MD5 hash
$srcMD5 = Get-FileMD5 -file $src
Write-Debug "Source file hash: $srcMD5"
$destMD5 = Get-FileMD5 -file $dest
Write-Debug "Destination file hash: $destMD5"
if ($srcMD5 -eq $destMD5) { #if the MD5 hashes match then the files are the same
Write-Debug "File hashes match. File already exists in destination folder and will be skipped."
$cpy = $false
}
else { #if the MD5 hashes are different then copy the file and overwrite the older version in the destination dir
$cpy = $true
Write-Debug "File hashes don't match. File will be copied to destination folder."
}
}
else { #if the file doesn't in the destination dir it will be copied.
Write-Debug "File doesn't exist in destination folder and will be copied."
$cpy = $true
}
Write-Debug "Copy is $cpy"
if ($cpy -eq $true) { #copy the file if file version is newer or if it doesn't exist in the destination dir.
Write-Debug "Copying $src to $dest"
if (!(test-path $dest)) {
New-Item -ItemType "File" -Path $dest -Force
}
Copy-Item -Path $src -Destination $dest -Force
}
}

There's a Network Emulator feature in Visual Studio 2010 (and probably newer). Maybe you can get that to work from PowerShell. Otherwise I think you'll have to resort to NetLimiter or one of the alternatives:
TMNetSim
Network Simulator
netem

Robocopy does have the ability though and you can wrap it in PowerShell via Invoke-Expression.

Related

How to unzip all files in folder? (Not .zip extension)

Currently, I am writing a script that moves PDF files that are wrongfully zipped to a certain folder. I have achieved that. The next thing I need to get to work, is that the zipped .pdf files get unzipped into a different folder.
This is my whole script. Everything except for the last 2 lines is dedicated to finding the PDF files that are zipped and moving them.
In the first parts, the script checks the first few bytes of every pdf file in the folder. If they start with "PK*", they are zip files and get moved to the zipped folder.
For every PDF/zip file, there is one associated HL7 file in the folder next to it.
These also need to get moved to the same folder. From there the zip files need to be unzipped and relocated to "unzipped"
The last 2 lines are for unzipping.
$pdfDirectory = 'Z:\Documents\16_Med._App\Auftraege\PDFPrzemek\struktur_id_1225\ext_dok'
$newLocation = 'Z:\Documents\16_Med._App\Auftraege\PDFPrzemek\Zip'
Get-ChildItem "$pdfDirectory" -Filter "*.pdf" | foreach {
if ((Get-Content $_.FullName | select -First 1 ) -like "PK*") {
$HL7 = $_.FullName.Replace("ext_dok","MDM")
$HL7 = $HL7.Replace(".pdf",".hl7")
move $_.FullName $newLocation;
move $HL7 $newLocation
}
}
Get-ChildItem 'Z:\Documents\16_Med._App\Auftraege\PDFPrzemek\Zip' |
Expand-Archive -DestinationPath 'Z:\Documents\16_Med._App\Auftraege\PDFPrzemek\Zip\unzipped' -Force
This, sadly, doesn't work.
I suspect that it's because these files dont have the .zip extension. The only Filter that works for Expand-Archive is .zip.
So I need to find a way to get this function to unzip the files, even though they dont have the fitting extension...
Like #Ansgar said this would be the way to go:
Param (
$SourcePath = 'C:\Users\xxx\Downloads\PDF',
$ZipFilesPath = 'C:\Users\xxx\Downloads\ZIP',
$UnzippedFilesPath = 'C:\Users\xxx\Downloads\Unzipped'
)
$VerbosePreference = 'Continue'
#region Test folders
#($SourcePath, $ZipFilesPath, $UnzippedFilesPath) | Where-Object {
-not (Test-Path -LiteralPath $_)
} | ForEach-Object {
throw "Path '$_' not found. Make sure that the folders exist before running the script."
}
#endregion
#region Get all files with extension .pdf
$Params = #{
Path = Join-Path -Path $SourcePath -ChildPath 'ext_dok'
Filter = '*.pdf'
}
$PDFfiles = Get-ChildItem #Params
Write-Verbose "Got $($PDFfiles.count) files with extension '.pdf' from '$($Params.Path)'"
#endregion
#region Move PDF and HL7 files
$MDMpath = Join-Path -Path $SourcePath -ChildPath 'MDM'
foreach ($PDFfile in ($PDFfiles | Where-Object {
(Get-Content $_.FullName | Select-Object -First 1) -like 'PK*'})
) {
$MoveParams = #{
Path = $PDFfile.FullName
Destination = Join-Path -Path $ZipFilesPath -ChildPath ($PDFfile.BaseName + '.zip')
}
Move-Item #MoveParams
Write-Verbose "Moved file '$($MoveParams.Path)' to '$($MoveParams.Destination)'"
$GetParams = #{
Path = Join-Path -Path $MDMpath -ChildPath ($PDFfile.BaseName + '.hl7')
ErrorAction = 'Ignore'
}
if ($HL7file = Get-Item #GetParams) {
$MoveParams = #{
Path = $HL7file
Destination = $ZipFilesPath
}
Move-Item #MoveParams
Write-Verbose "Moved file '$($MoveParams.Path)' to '$($MoveParams.Destination)$($HL7file.Name)'"
}
}
#endregion
#region Unzip files
$ZipFiles = Get-ChildItem -Path $ZipFilesPath -Filter '*.zip' -File
foreach ($ZipFile in $ZipFiles) {
$ZipFile | Expand-Archive -DestinationPath $UnzippedFilesPath -Force
Write-Verbose "Unzipped file '$($ZipFile.Name)' in folder '$UnzippedFilesPath'"
}
#endregion
Some tips:
Add a Param () clause at the beginning of the script to contain all your variables that can change.
Try to use the full parameter name to clearly indicate what is what. Use Get-ChildItem -Path xxx instead of Get-ChildItem xxx.
Use hash tables for long parameters. This makes the code more compact in width and more easily to read.
Use #region and #endregion to group your code.

How to backup these files into specific folders using powershell

I've finally have given up googling and come here out of desperation. Go easy on me I'm fairly new to Powershell.
So, the objective of the code below was to first look through the source folder, then read through each .zip file and move to the directory specified by the value in the hashtable. Unfortunately, this is not how they want it to work anymore.
Now I need to retain the parent folder from source: for example "DAL" and then create the proceeding folders based on the file names and finally move each .zip to its file specified folder. Also, it needs to go through each folder under source which will be at least 20 other folders with a unique 3 character names.
$srcRoot = "C:\Cloud\source\dal"
$dstRoot = "C:\Cloud\Destination"
##$map = #{}; dir -recurse | ? { !$_.psiscontainer} | % { ##$map.add($_.name,$_.PSChildName) }
# DAT and DEV will have to be excluded from folder creation
$map = {
#AEODDAT_201901 = "AEOD\2019\01"
#AEOMDEV_201902 = "AEOM\2019\01"
#AEOYDAT_201902 = "AEOY\2019\01"
}
$fileList = Get-ChildItem -Path $srcRoot -Filter "*.zip*" -File -Force -Recurse
foreach ($file in $fileList)
{
#Go through each file up to mapped string
$key = $file.BaseName.Substring(0,14)
if ($key -in $map.Keys)
{
$fileName = $file.Name
$dstDir = Join-Path -Path $dstRoot -ChildPath $map[$key]
#create direcotory if not in path
if (-not (Test-Path -Path $dstDir))
{
mkdir -Path $dstDir
}
Write-Verbose "Moving $($file.FullName)"
if (Test-Path -Path (Join-Path -Path $dstDir -ChildPath $fileName))
{
#Write error if name exists
Write-Error -Message "File $fileName already exists at $dstDir"
#move path
} else {
Move-Item -Path $($file.FullName) -Destination $dstDir
}
}
}
So C:\Cloud\source\DAL\AEODDAT20190101.zip should create folders in C:\Cloud\Destination\DAL\AEOD\2019\01\AEODDAT20190101.zip would be my desired output.
Welcome, Matt! (no pun intended) One of the habits I have in similar situations with destination folders is to Set-Location $dstRoot and create folders from the relative path. You can execute New-Item with the relative path and the syntax is simpler. For example, your If statement could look like this and it would work the same way (with a slightly different error message):
if ($key -in $map.Keys){
Set-Location $dstRoot
New-Item -ItemType Directory $map[$key] -ErrorAction Ignore #won't raise an error if it exists
Write-Verbose "Moving $($file.FullName)"
#this will raise an error if the file already exists, unless you specify -Force
Move-Item "$($file.FullName)" $map[$key]
}
EDIT: Found 2 issues.
$map is a Hashtable literal that should be preceded with #:
$map = #{
AEODDAT20190101 = "AEOD\2019\01"
You were missing the last character of the base file name by taking only the first 14 characters. AEODDAT2019010 didn't match AEODDAT20190101. This should fix it:
$key = $file.BaseName.Substring(0,15)

Powershell script to deploy files with necessary backup

Team,
I'm writing a powershell script to automate one of the manual task we are doing daily. I need expert suggestion and help here as I'm new to powershell scripting.
Requirement:
There is a source, destination and a backup folder.
Source may have diff files and files within multiple folders of source.
eg: source\Login.aspx or source\App_Code\BLogic.vb or source\bin\servr.dll etc.
While copying to destination, my source file has to be checked for existence at destination, if exists then we need to copy the current existing file in destination to backup folder, then copy source to destination.
Below are the scripts that i have tried till now.
Script1:
I'm able to list files which are same but not able to copy those array elements to backup location:
#Declare Source and Destination
$source = Get-ChildItem -Recurse -path \\server1\\e$\ps\src\
$dest = Get-ChildItem -Recurse -path \\server2\\e$\ps\dest\
#lists only objects which are equal and assign to variable
$files=compare-Object -DifferenceObject $source -ReferenceObject $dest -IncludeEqual -ExcludeDifferent
$array = #($files)
$len=$array.length
for ($i=0; $i -lt $array.length; $i++)
{
$array[$i]
}
Script2:
As script1 i was facing problem i tried a silly logic with this script, but i feel this is also having too much manual work :( please help me.
Function cpytest ($s, $d)
{
Copy-Item -Path $s -Destination $d;
}
$n = Read-Host "Enter no of files to be uploaded"
$b = Read-Host "Enter Backup Location for file backups"
for ($i=0;$i -lt $n;$i++)
{
$s = Read-Host "Enter Source with complete file name"
$d = Read-Host "Enter Destination with file name too"
$r = Test-Path $d
If ($r -eq "True")
{
cpytest $d $b
cpytest $s $d
}
Else
{
cpytest $s $d
}
}
Try something like this:
Function BackupAndMoveFile($filePath, $backupPath, $deployPath) {
$fileName = Split-Path $filePath -Leaf
$deployFile = Join-Path $deployPath $fileName
if ((Test-Path $deployFile)) {
$backupFile = Join-Path $backupPath $fileName
Move-Item $deployFile $backupFile -Force
if (!(Test-Path $backupFile)) {
Write-Warning "Can't backup $fileName to $backupFile"
return $false
}
}
Copy-Item $filePath $deployPath -Force
if (!(Test-Path $deployPath)) {
Write-Warning "Can't deploy $fileName to $deployPath"
return $false
}
return $true
}
BackupAndMoveFile 'C:\temp\backup-test.txt' `
'C:\temp\backup' `
'C:\temp\deploy'
It will overwrite anything in the backup directory so you might want to modify to add a timestamp.

Copy items from Source to Destination if they don't already exist

I have a pretty basic powershell copy script that copies items from a source folder to a destination folder. However this is moving way too much data, and I'd like to check if the filename already exists so that file can be ignored. I don't need this as complex as verifying created date/checksum/etc.
Currently it's along the lines of:
Copy-Item source destination -recurse
Copy-Item source2 destination2 -recurse
I'd imagine I need to add the Test-Path cmdlet, but I'm uncertain how to implement it.
You could always call ROBOCOPY from PowerShell for this.
Use the /xc (exclude changed) /xn (exclude newer) and /xo (exclude older) flags:
robocopy /xc /xn /xo source destination
This will ONLY copy those files that are not in the destination folder.
For more option type robocopy /?
$exclude = Get-ChildItem -recurse $dest
Copy-Item -Recurse $file $dest -Verbose -Exclude $exclude
While I agree that Robocopy is the best tool for something like this, I'm all for giving the customer what they asked for and it was an interesting PowerShell exercise.
This script should do just what you asked for: copy a file from Source to Destination only if it does not already exist in the Destination with a minimum of frills. Since you had the -recurse option in your example, that made for a bit more coding than just simply testing for the filename in the Destination folder.
$Source = "C:\SourceFolder"
$Destination = "C:\DestinationFolder"
Get-ChildItem $Source -Recurse | ForEach {
$ModifiedDestination = $($_.FullName).Replace("$Source","$Destination")
If ((Test-Path $ModifiedDestination) -eq $False) {
Copy-Item $_.FullName $ModifiedDestination
}
}
Building off of Wai Ha Lee's post, here's an example that worked for me:
$Source = "<your path here>"
$Dest = "<your path here>"
$Exclude = Get-ChildItem -recurse $Dest
Get-ChildItem $Source -Recurse -Filter "*.pdf" | Copy-Item -Destination $Dest -Verbose -Exclude $Exclude
This builds a list to exclude, then copies any pdf in the source directory and sub-directories to the destination in a single folder...excluding the existing files. Again, this is an example from my needs, but similar to yours. Should be easy enough to tweak to your hearts content.
Function Copy-IfNotPresent will accept one file at a time but it's easy to loop for all files you want to copy. Here's an example:
gci c:\temp\1\*.* -Recurse -File | % { Copy-IfNotPresent -FilePath $_ -Destination "C:\temp\2\$(Resolve-Path $_ -relative)" -Verbose }
Here's the function. It will generate the folder tree if necessary. Here's the gists link: https://gist.github.com/pollusb/cd47b4afeda8edbf8943a8808c880eb8
Function Copy-IfNotPresent {
<#
Copy file only if not present at destination.
This is a one file at a time call. It's not meant to replace complex call like ROBOCOPY.
Destination can be a file or folder. If it's a folder, you can use -Container to force Folder creation when not exists
#>
[CmdletBinding()]
Param (
[Parameter(Mandatory)]
$FilePath,
[Parameter(Mandatory)]
[string]$Destination,
[switch]$Container,
[switch]$WhatIf
)
#region validations
if ($FilePath -isnot [System.IO.FileInfo]){
$File = Get-ChildItem $FilePath -File
} else {
$File = $FilePath
}
if (!$File.Count){
Write-Warning "$FilePath no file found."
return
} elseif ($File.Count -gt 1) {
Write-Warning "$FilePath must resolve to one file only."
return
}
#endregion
# Destination is a folder
if ($Container -or (Test-Path -Path $Destination -PathType Container)) {
if (!(Test-Path $Destination)) {
New-Item -Path $Destination -ItemType Container | Out-Null
}
$Destination += "\$($File.Name)"
}
# Destination is a file
if (!(Test-Path $Destination)) {
if ($WhatIf) {
Write-Host "WhatIf:Copy-IfNotPresent $FilePath -> $Destination"
} else {
# Force creation of parent folder
$Parent = Split-Path $Destination -Parent
if (!(Test-Path $Parent)) {
New-Item $Parent -ItemType Container | Out-Null
}
Copy-Item -Path $FilePath -Destination $Destination
Write-Verbose "Copy-IfNotPresent $FilePath -> $Destination (is absent) copying"
}
} else {
Write-Verbose "Copy-IfNotPresent $Destination (is present) not copying"
}
}
$source = "c:\source"
$destination = "c:\destination"
Create a list of files to exclude, i.e. files already existing in the destination.
$exclude = Get-Childitem -Recurse $destination | ForEach-Object { $_.FullName -replace [Regex]::Escape($destination ), "" }
Recursively copy all contents from the source to the destination excluding the previously collected files.
Copy-Item -Recurse -Path (Join-Path $source "*") -Destination $destination -Exclude $exclude -Force -Verbose
(Join-Path $source "*") add a wildcard at end ensuring that you get the children of the source folder instead of the source folder itself.
Force is used because I don't mind that there are already existing folders (results in error messages). Use with caution.
ForEach-Object { $_.FullName -replace [Regex]::Escape($destination ), "" } transforms the existing file full names into values which can be used as Exclude parameter
Here is a recursive script that syncronizes 2 folders ignoring existing files:
function Copy-FilesAndFolders([string]$folderFrom, [string]$folderTo) {
$itensFrom = Get-ChildItem $folderFrom
foreach ($i in $itensFrom)
{
if ($i.PSIsContainer)
{
$subFolderFrom = $folderFrom + "\" + $i.BaseName
$subFolderTo = $folderTo + "\" + $i.BaseName
Copy-FilesAndFolders $subFolderFrom $subFolderTo | Out-Null
}
else
{
$from = $folderFrom + "\" + $i.Name
$to = $folderTo + "\" + $i.Name
if (!(Test-Path $from)) # only copies non-existing files
{
if (!(Test-Path $folderTo)) # if folder doesn't exist, creates it
{
New-Item -ItemType "directory" -Path $folderTo
}
Copy-Item $from $folderTo
}
}
}
}
To call it:
Copy-FilesAndFolders "C:\FromFolder" "C:\ToFolder"
Lots of great answers in here, here's my contribution as it relates to keeping an mp3 player in sync with a music library.
#Tom Hubbard, 10-19-2021
#Copy only new music to mp3 player, saves time by only copying items that don't exist on the destination.
#Leaving the hardcoded directories and paths in here, sometimes too much variable substitution is confusing for newer PS users.
#Gets all of the albums in the source directory such as your music library
$albumsInLibrary = gci -Directory -path "C:\users\tom\OneDrive\Music" | select -ExpandProperty Name
#Gets all of the albums of your destination folder, such as your mp3 player
$albumsOnPlayer = gci -Directory -Path "e:\" | select -ExpandProperty name
#For illustration, it will list the differences between the music library and the music player.
Compare-Object -DifferenceObject $albumsInLibrary -ReferenceObject $albumsOnPlayer
#Loop through each album in the library
foreach ($album in $albumsInLibrary)
{
#Check to see if the music player contains this directory from the music library
if ($albumsOnPlayer -notcontains $album)
{
#If the album doesn't exist on the music player, copy it and it's child items from the library to the player
write-host "$album is not on music player, copying to music player" -ForegroundColor Cyan
Copy-Item -path "C:\users\Tom\OneDrive\music\$album" -Recurse -Destination e:\$album
}
}

Most elegant way to extract a directory from a zipfile using PowerShell?

I need to unzip a specific directory from a zipfile.
Like for example extract the directory 'test\etc\script' from zipfile 'c:\tmp\test.zip' and place it in c:\tmp\output\test\etc\script.
The code below works but has two quirks:
I need to recursively find the directory ('script') in the zip file (function finditem) although I already know the path ('c:\tmp\test.zip\test\etc\script')
With CopyHere I need to determine the targetdirectory, specifically the 'test\etc' part manually
Any better solutions? Thanks.
The code:
function finditem($items, $itemname)
{
foreach($item In $items)
{
if ($item.GetFolder -ne $Null)
{
finditem $item.GetFolder.items() $itemname
}
if ($item.name -Like $itemname)
{
return $item
}
}
}
$source = 'c:\tmp\test.zip'
$target = 'c:\tmp\output'
$shell = new-object -com shell.application
# find script folder e.g. c:\tmp\test.zip\test\etc\script
$item = finditem $shell.NameSpace($source).Items() "script"
# output folder is c:\tmp\output\test\etc
$targetfolder = Join-Path $target ((split-path $item.path -Parent) -replace '^.*zip')
New-Item $targetfolder -ItemType directory -ErrorAction Ignore
# unzip c:\tmp\test.zip\test\etc\script to c:\tmp\output\test\etc
$shell.NameSpace($targetfolder).CopyHere($item)
I don't know about most elegant, but with .Net 4.5 installed you could use the ZipFile class from the System.IO.Compression namespace:
[Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem') | Out-Null
$zipfile = 'C:\path\to\your.zip'
$folder = 'folder\inside\zipfile'
$dst = 'C:\output\folder'
[IO.Compression.ZipFile]::OpenRead($zipfile).Entries | ? {
$_.FullName -like "$($folder -replace '\\','/')/*"
} | % {
$file = Join-Path $dst $_.FullName
$parent = Split-Path -Parent $file
if (-not (Test-Path -LiteralPath $parent)) {
New-Item -Path $parent -Type Directory | Out-Null
}
[IO.Compression.ZipFileExtensions]::ExtractToFile($_, $file, $true)
}
The 3rd parameter of ExtractToFile() can be omitted. If present it defines whether existing files will be overwritten or not.
As far as the folder location in a zip is known, the original code can be simplified:
$source = 'c:\tmp\test.zip' # zip file
$target = 'c:\tmp\output' # target root
$folder = 'test\etc\script' # path in the zip
$shell = New-Object -ComObject Shell.Application
# find script folder e.g. c:\tmp\test.zip\test\etc\script
$item = $shell.NameSpace("$source\$folder")
# actual destination directory
$path = Split-Path (Join-Path $target $folder)
if (!(Test-Path $path)) {$null = mkdir $path}
# unzip c:\tmp\test.zip\test\etc\script to c:\tmp\output\test\etc\script
$shell.NameSpace($path).CopyHere($item)
Windows PowerShell 5.0 (included in Windows 10) natively supports extracting ZIP files using Expand-Archive cmdlet:
Expand-Archive -Path Draft.Zip -DestinationPath C:\Reference