Split XML by node - powershell

I've got a big XML file (more than 4Gb) and i should integrate it into SQL table.
SQL has a 2 Gb limitation. The idea is to split the file into serveral XML. For example 500Mb by file.
I try to adapt a powershell script but i still have the following error
Here is the PowerShell script i'm using and i can't find what's wrong
param( [string]$file = $(throw "ENTITY.XML"), $matchesPerSplit = 50, $maxFiles = [Int32]::MaxValue, $splitOnNode = $(throw "entity"), $offset = 0 )
$ErrorActionPreference = "Stop";
trap {
$ErrorActionPreference = "Continue"
write-error "Script failed: $_ \r\n $($_.ScriptStackTrace)"
exit (1);
}
$file = (resolve-path $file).path
$fileNameExt = [IO.Path]::GetExtension($file)
$fileNameWithoutExt = [IO.Path]::GetFileNameWithoutExtension($file)
$fileNameDirectory = [IO.Path]::GetDirectoryName($file)
$reader = [System.Xml.XmlReader]::Create($file)
$matchesCount = $idx = 0
try {
"Splitting $from on node name='$splitOnNode', with a max of $matchesPerSplit matches per file. Max of $maxFiles files will be generated."
$result = $reader.ReadToFollowing($splitOnNode)
$hasNextSibling = $true
while (-not($reader.EOF) -and $result -and $hasNextSibling -and ($idx -lt $maxFiles + $offset)) {
if ($matchesCount -lt $matchesPerSplit) {
if($offset -gt $idx) {
$idx++
continue
}
$to = [IO.Path]::Combine($fileNameDirectory, "$fileNameWithoutExt.$($idx -$offset)$fileNameExt")
"Writing to $to"
$toXml = New-Object System.Xml.XmlTextWriter($to, $null)
$toXml.Formatting = 'Indented'
$toXml.Indentation = 2
try {
$toXml.WriteStartElement("split")
$toXml.WriteAttributeString("cnt", $null, "$idx")
do {
$toXml.WriteRaw($reader.ReadOuterXml())
$matchesCount++;
$hasNextSibling = $reader.ReadToNextSibling($splitOnNode)
} while($hasNextSibling -and ($matchesCount -lt $matchesPerSplit))
$toXml.WriteEndElement();
}
finally {
$toXml.Flush()
$toXml.Close()
}
$idx++
$matchesCount = 0;
}
}
}
finally {
$reader.Close()
}
My XML file is on the same folder as the PS file and "entity" is the node.

Related

Coding with powershell

I'm new with powershell and i would like to use a loop to ping several Printers on my network.
My problem is : once i'm in the loop of pinging , i can't go out of the loop ...
I tried several things from google but without success ( start-stop , Timer ) . Does anybody have any idea?
Here is the code :
$BtnStartPingClicked = {
if ($LblFileSelectPing.Text -eq "*.txt") {
Add-Type -AssemblyName PresentationCore,PresentationFramework
$ButtonType = [System.Windows.MessageBoxButton]::OK
$MessageIcon = [System.Windows.MessageBoxImage]::Error
$MessageBody = "Please select a list of printer first"
$MessageTitle = "Error"
$Result = [System.Windows.MessageBox]::Show($MessageBody,$MessageTitle,$ButtonType,$MessageIcon)
Write-Host "Your choice is $Result"
}
else {
do {
$IPList = Get-Content ($LblFileSelectPing.Text)
$snmp = New-Object -ComObject olePrn.OleSNMP
$ping = New-Object System.Net.NetworkInformation.Ping
$i = 11
$j = 1
foreach ($Printer in $IPList) {
try {
$result = $ping.Send($Printer)
} catch {
$result = $null
}
if ($result.Status -eq 'Success') {
$((Get-Variable -name ("GBMachine"+$j+"Ping")).value).Visible = $True
$j++
test-Connection -ComputerName $Printer -Count 1 -Quiet
$printerip = $result.Address.ToString()
# OPEN SNMP CONNECTION TO PRINTER
$snmp.open($Printer, 'public', 2, 3000)
# MODEL
try {
$model = $snmp.Get('.1.3.6.1.2.1.25.3.2.1.3.1')
} catch {
$model = $null
}
# Serial
try {
$serial = $snmp.Get('.1.3.6.1.4.1.1602.1.2.1.8.1.3.1.1').toupper()
} catch {
$Dns = $null
}
# progress
$TBMonitoringPing.SelectionColor = "green"
$TBMonitoringPing.AppendText("$Printer is Pinging")
$TBMonitoringPing.AppendText("`n")
$mac = (arp -a $Printer | Select-String '([0-9a-f]{2}-){5}[0-9a-f]{2}').Matches.Value
# OPEN SNMP CONNECTION TO PRINTER
$((Get-Variable -name ('LblMach' + $i)).value).Text = "IP : $Printerip"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "Model : $Model"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "MAC : $mac"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "Serial : $serial"
$TBAnswerMachine.AppendText("$Model")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("$Printer - $Serial")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("$Mac")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("`n")
Get-Content ($LblFileSelectPing.Text) | Where-Object {$_ -notmatch $Printer} | Set-Content ("C:\_canonsoftware\out.txt")
$i = $i+7
$snmp.Close()
Start-Sleep -milliseconds 1000 # Take a breather!
}
else {
$TBMonitoringPing.selectioncolor = "red"
$TBMonitoringPing.AppendText("$Printer not pinging")
$TBMonitoringPing.AppendText("`n")
Start-Sleep -milliseconds 1000 # Take a breather!
}
}
$LblFileSelectPing.Text = "C:\_canonsoftware\out.txt"
} until($infinity)
}
}
thanks for your answers...
1 - part of my object are indeed not declared in the code because they are in my other PS1 file....
2 - I do a do until infinity because i don't want to stop the code before i decide it...
3 - I didn't explain my problem correctly ( excuse my poor english ) ... i would like to be able to go out of the loop do until at the moment i click on a stop button ... but apprently the windows doens't respond while in the loop ... i have to stop the script with powershell ... which is annoying because i'd like to make an executable with it ... and not have to go out of my program ...
thank you for your ideas

How to handle create a file and write it using Powershell?

I want to create a file and write content to the file.
The new file will create based on the total SWPO file exist. I tried this scirpt, I can create new file. But when I put only 1 $PO_Path file, it will create 2 new file. Actually those 2 file are the same but one of the file without $c. Like this
ID_ABC18XXR3CT123_.job
ID_ABC18XXR3CT123_AE.job
BUt if I put $PO_Path file more than 1, it works well.
Anyone can help me please. Thanks.
Function Create_OriJob
{
$BID = "18XXR3CT123"
$Job_Path = $Config_File.Automation_Config.Path.OriJob
$PO_Path = $Config_File.Automation_Config.Path.POfiles
if(Test-Path -Path "$PO_Path\*$BID*")
{
Write-Output "SWPO File Found"
# Start-Sleep -s 3
$PO_Content = Get-Content -path "$PO_Path\*$BID*"
$POfile = Get-ChildItem -Name "$PO_Path\*$BID*"
$Get_CRM = $PO_Content | Where-Object {$_ -like "*;CRM*"}
$CRM = $Get_CRM.Substring(5,2)
$CRM = $CRM.split()
$POCountry = Get-ChildItem -Name "$PO_Path"
$GetCountry = $POCountry.Substring(15,3)
$GetCountry = $GetCountry.split()
For($i = 0; $i -lt $POfile.Length; $i++){
try{
$po = $POfile[$i]
$c = $CRM[$i]
$cc = $GetCountry[$i]
New-Item -ItemType File -Path "$Job_Path\$JobType`_$Prefix$BID`_$c.job" -Force
$Title = $Config_File.Automation_Config.Out_Job.Title
$Auto = $Config_File.Automation_Config.Out_Job.Auto
$Proc = $Config_File.Automation_Config.Out_Job.Process
$Auto = $Config_File.Automation_Config.Out_Job.Auto
$PO_Conf = $Config_File.Automation_Config.Out_Job.PO
$BIDINFO = $Config_File.Automation_Config.Out_Job.BIDINFO
$BuildID = $Config_File.Automation_Config.Out_Job.BID
$PFX = $Config_File.Automation_Config.Out_Job.PFX
$CRM_Conf = $Config_File.Automation_Config.Out_Job.CRM
$CountryConf = $Config_File.Automation_Config.Out_Job.Country
$Platform = $Config_File.Automation_Config.Out_Job.Platform
$TSJobcreate = Get-Date
$Output_JOB = #"
<?xml version="1.0" encoding="UTF-8"?>
<$Title>
<$Auto>
<$Proc>$Auto</$Proc>
<$PO_Conf>$po</$PO_Conf>
</$Auto>
<$BIDINFO>
<$BuildID>$BID</$BuildID>
<$PFX>$Prefix</$PFX>
<$CRM_Conf>$c</$CRM_Conf>
<$CountryConf>$cc</$CountryConf>
</$BIDINFO>
<$Platform>
$All_SSID
</$Platform>
<Timestamp>
<JobCreate>$TSJobcreate</JobCreate>
</Timestamp>
</$Title>
"#
$Output_JOB | Out-File "$Job_Path\$JobType`_$Prefix$BID`_$c.job" -NoNewline
Write-Host "Output"
}
catch{
Write-Output "Something wrong!"
}
}
Write-Output "Continue to create operational job"
Create_OpJob
}
else{
Write-Host "SWPO Not Found, Do Error checking file"
Error_Monitoring
}
#Error_Monitoring
}

Dynamic Parameters - with Dynamic ValidateSet

I have a script that I've been working on to provide parsing of SCCM log files. This script takes a computername and a location on disk to build a dynamic parameter list and then present it to the user to choose the log file they want to parse. Trouble is I cannot seem to get the ValidateSet portion of the dynamic parameter to provide values to the user. In addition the script won't display the -log dynamic parameter when attempting to call the function.
When you run it for the first time you are not presented with the dynamic parameter Log as I mentioned above. If you then use -log and then hit tab you’ll get the command completer for the files in the directory you are in. Not what you’d expect; you'd expect that it would present you the Logfile names that were gathered during the dynamic parameter execution.
PSVersion 5.1.14409.1012
So the question is how do I get PowerShell to present the proper Validate set items to the user?
If you issue one of the items in the error log you get the proper behavior:
Here are the two functions that i use to make this possible:
function Get-CCMLog
{
[CmdletBinding()]
param([Parameter(Mandatory=$true,Position=0)]$ComputerName = '$env:computername', [Parameter(Mandatory=$true,Position=1)]$path = 'c:\windows\ccm\logs')
DynamicParam
{
$ParameterName = 'Log'
if($path.ToCharArray() -contains ':')
{
$FilePath = "\\$ComputerName\$($path -replace ':','$')"
if(test-path $FilePath)
{
$logs = gci "$FilePath\*.log"
$LogNames = $logs.basename
$logAttribute = New-Object System.Management.Automation.ParameterAttribute
$logAttribute.Position = 2
$logAttribute.Mandatory = $true
$logAttribute.HelpMessage = 'Pick A log to parse'
$logCollection = New-Object System.Collections.ObjectModel.Collection[System.Attribute]
$logCollection.add($logAttribute)
$logValidateSet = New-Object System.Management.Automation.ValidateSetAttribute($LogNames)
$logCollection.add($logValidateSet)
$logParam = New-Object System.Management.Automation.RuntimeDefinedParameter($ParameterName,[string],$logCollection)
$logDictionary = New-Object System.Management.Automation.RuntimeDefinedParameterDictionary
$logDictionary.Add($ParameterName,$logParam)
return $logDictionary
}
}
}
begin {
# Bind the parameter to a friendly variable
$Log = $PsBoundParameters[$ParameterName]
}
process {
# Your code goes here
#dir -Path $Path
$sb2 = "$((Get-ChildItem function:get-cmlog).scriptblock)`r`n"
$sb1 = [scriptblock]::Create($sb2)
$results = Invoke-Command -ComputerName $ComputerName -ScriptBlock $sb1 -ArgumentList "$path\$log.log"
[PSCustomObject]#{"$($log)Log"=$results}
}
}
function Get-CMLog
{
param(
[Parameter(Mandatory=$true,
Position=0,
ValueFromPipelineByPropertyName=$true)]
[Alias("FullName")]
$Path,
$tail =10
)
PROCESS
{
if(($Path -isnot [array]) -and (test-path $Path -PathType Container) )
{
$Path = Get-ChildItem "$path\*.log"
}
foreach ($File in $Path)
{
if(!( test-path $file))
{
$Path +=(Get-ChildItem "$file*.log").fullname
}
$FileName = Split-Path -Path $File -Leaf
if($tail)
{
$lines = Get-Content -Path $File -tail $tail
}
else {
$lines = get-cotnet -path $file
}
ForEach($l in $lines ){
$l -match '\<\!\[LOG\[(?<Message>.*)?\]LOG\]\!\>\<time=\"(?<Time>.+)(?<TZAdjust>[+|-])(?<TZOffset>\d{2,3})\"\s+date=\"(?<Date>.+)?\"\s+component=\"(?<Component>.+)?\"\s+context="(?<Context>.*)?\"\s+type=\"(?<Type>\d)?\"\s+thread=\"(?<TID>\d+)?\"\s+file=\"(?<Reference>.+)?\"\>' | Out-Null
if($matches)
{
$UTCTime = [datetime]::ParseExact($("$($matches.date) $($matches.time)$($matches.TZAdjust)$($matches.TZOffset/60)"),"MM-dd-yyyy HH:mm:ss.fffz", $null, "AdjustToUniversal")
$LocalTime = [datetime]::ParseExact($("$($matches.date) $($matches.time)"),"MM-dd-yyyy HH:mm:ss.fff", $null)
}
[pscustomobject]#{
UTCTime = $UTCTime
LocalTime = $LocalTime
FileName = $FileName
Component = $matches.component
Context = $matches.context
Type = $matches.type
TID = $matches.TI
Reference = $matches.reference
Message = $matches.message
}
}
}
}
}
The problem is that you have all the dynamic logic inside scriptblock in the if statement, and it handles the parameter addition only if the path provided contains a semicolon (':').
You could change it to something like:
if($path.ToCharArray() -contains ':') {
$FilePath = "\\$ComputerName\$($path -replace ':','$')"
} else {
$FilePath = $path
}
and continue your code from there
PS 6 can do a dynamic [ValidateSet] with a class:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_functions_advanced_parameters?view=powershell-6#dynamic-validateset-values

How to track path/key during recursion through hash of hashes in PowerShell?

I've been trying to compare two hashes of hashes against each other. Unfortunately even with the great help from here I struggled to pull together what I was trying to do.
So, I've resorted to searching the internet again and I was lucky enough to find Edxi's code (full credit to him and Dbroeglin who looks to have written it originally)https://gist.github.com/edxi/87cb8a550b43ec90e4a89d2e69324806 His compare-hash function does exactly what I needed in terms of its comparisons. However, I'd like to report of the full path of the hash in the final output. So I've tried to update the code to allow for this. My thinking being that I should be able to take the Key (aka path) while the code loops over itself but I'm clearly going about it in the wrong manner.
Am I going about this with the right approach, and if not, would someone suggest another method please? The biggest problem I've found so far is that if I change the hash, my code changes don't work e.g. adding 'More' = 'stuff' So it becomes $sailings.Arrivals.PDH083.More breaks the path in the way I've set it.
My version of the code:
$Sailings = #{
'Arrivals' = #{
'PDH083' = #{
'GoingTo' = 'Port1'
'Scheduled' = '09:05'
'Expected' = '10:11'
'Status' = 'Delayed'
}
}
'Departures' = #{
'PDH083' = #{
'ArrivingFrom' = 'Port1'
'Scheduled' = '09:05'
'Expected' = '09:05'
'Status' = 'OnTime'
'Other' = #{'Data' = 'Test'}
}
}
}
$Flights = #{
'Arrivals' = #{
'PDH083' = #{
'GoingTo' = 'Port'
'Scheduled' = '09:05'
'Expected' = '10:20'
'Status' = 'Delayed'
}
}
'Departures' = #{
'PDH083' = #{
'ArrivingFrom' = 'Port11'
'Scheduled' = '09:05'
'Expected' = '09:05'
'Status' = 'NotOnTime'
'Other' = #{'Data' = 'Test_Diff'}
}
}
}
function Compare-Hashtable {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[Hashtable]$Left,
[Parameter(Mandatory = $true)]
[Hashtable]$Right,
[string] $path,
[boolean] $trackpath = $True
)
write-host "PAth received as: $path"
function New-Result($Key, $LValue, $Side, $RValue) {
New-Object -Type PSObject -Property #{
key = $Key
lvalue = $LValue
rvalue = $RValue
side = $Side
}
}
[Object[]]$Results = $Left.Keys | ForEach-Object {
if ($trackpath ) {
write-host "Working on Path: " $path + $_
}
if ($Left.ContainsKey($_) -and !$Right.ContainsKey($_)) {
write-host "Right key not matched. Report path as: $path"
New-Result $path $Left[$_] "<=" $Null
}
else {
if ($Left[$_] -is [hashtable] -and $Right[$_] -is [hashtable] ) {
if ($path -ne $null -or $path -ne "/") {
$path = $path + "/" + $_
write-host "Sending Path to function as: $path"
}
Compare-Hashtable $Left[$_] $Right[$_] $path
}
else {
$LValue, $RValue = $Left[$_], $Right[$_]
if ($LValue -ne $RValue) {
$path = $path + "/" + $_
write-host "Not a hash so must be a value at path:$path"
New-Result $path $LValue "!=" $RValue
}
else {
Write-Host "Before changing path: $path "
if (($path.Substring(0, $path.lastIndexOf('/'))).length >0) {
$path = $path.Substring(0, $path.lastIndexOf('/'))
Write-Host "After changing path: $path "
}
}
}
}
# if (($path.Substring(0, $path.lastIndexOf('/'))).length >0) {
# Tried to use this to stop error on substring being less than zero
# but clearly doesnt work when you add more levels to the hash
$path = $path.Substring(0, $path.lastIndexOf('/'))
# } else { $path = $path.Substring(0, $path.lastIndexOf('/')) }
}
$Results += $Right.Keys | ForEach-Object {
if (!$Left.ContainsKey($_) -and $Right.ContainsKey($_)) {
New-Result $_ $Null "=>" $Right[$_]
}
}
if ($Results -ne $null) { $Results }
}
cls
Compare-Hashtable $Sailings $Flights
outputs
key lvalue side rvalue
--- ------ ---- ------
/Arrivals/PDH083/Expected 10:11 != 10:20
/Arrivals/GoingTo Port1 != Port
/Departures/PDH083/ArrivingFrom Port1 != Port11
/Departures/PDH083/Status OnTime != NotOnTime
/Departures/PDH083/Other/Data Test != Test_Diff
I won't do that much string manipulation on the $Path but threat $Path as an array and join it to a string at the moment you assign it as a property (key = $Path -Join "/") to the object:
function Compare-Hashtable {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[Hashtable]$Left,
[Parameter(Mandatory = $true)]
[Hashtable]$Right,
[string[]] $path = #(),
[boolean] $trackpath = $True
)
write-host "Path received as: $path"
function New-Result($Path, $LValue, $Side, $RValue) {
New-Object -Type PSObject -Property #{
key = $Path -Join "/"
lvalue = $LValue
rvalue = $RValue
side = $Side
}
}
$Left.Keys | ForEach-Object {
$NewPath = $Path + $_
if ($trackpath ) {
write-host "Working on Path: " $NewPath
}
if ($Left.ContainsKey($_) -and !$Right.ContainsKey($_)) {
write-host "Right key not matched. Report path as: $NewPath"
New-Result $NewPath $Left[$_] "<=" $Null
}
else {
if ($Left[$_] -is [hashtable] -and $Right[$_] -is [hashtable] ) {
Compare-Hashtable $Left[$_] $Right[$_] $NewPath
}
else {
$LValue, $RValue = $Left[$_], $Right[$_]
if ($LValue -ne $RValue) {
New-Result $NewPath $LValue "!=" $RValue
}
}
}
}
$Right.Keys | ForEach-Object {
$NewPath = $Path + $_
if (!$Left.ContainsKey($_) -and $Right.ContainsKey($_)) {
New-Result $NewPath $Null "=>" $Right[$_]
}
}
}
cls
Compare-Hashtable $Sailings $Flights
side-note: Write Single Records to the Pipeline (SC03), see: Strongly Encouraged Development Guidelines. In other words, don't do the $Results += ... but intermediately put any completed result in the pipeline for the next cmdlet to be picked up (besides, it unnecessarily code)...

Unable to read an open file with binary reader

I have this function to read the SQL Server errorlog but the problem is that I'm not able to read the errorlog that the server is using at the time. I have been google-ing and it seems that the Fileshare flag isn't working for powershell. Is there some way to set the the Fileshare flag when I try to open the file?
function check_logs{
param($logs)
$pos
foreach($log in $logpos){
if($log.host -eq $logs.host){
$currentLog = $log
break
}
}
if($currentLog -eq $null){
$currentLog = #{}
$logpos.Add($currentLog)
$currentLog.host = $logs.host
$currentLog.event = $logs.type
$currentLog.lastpos = 0
}
$path = $logs.file
if($currentLog.lastpos -ne $null){$pos = $currentLog.lastpos}
else{$pos = 0}
if($logs.enc -eq $null){$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open))}
else{
$encoding = $logs.enc.toUpper().Replace('-','')
if($encoding -eq 'UTF16'){$encoding = 'Unicode'}
$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open), [System.Text.Encoding]::$encoding)
}
$required = $br.BaseStream.Length - $pos
if($required -lt 0){
$pos = 0
$required = $br.BaseStream.Length
}
if($required -eq 0){$br.close(); return $null}
$br.BaseStream.Seek($pos, [System.IO.SeekOrigin]::Begin)|Out-Null
$bytes = $br.ReadBytes($required)
$result = [System.Text.Encoding]::Unicode.GetString($bytes)
$split = $result.Split("`n")
foreach($s in $split)
{
if($s.contains(" Error:"))
{
$errorLine = [regex]::Split($s, "\s\s+")
$err = [regex]::Split($errorLine[1], "\s+")
if(log_filter $currentLog.event $err[1..$err.length]){$Script:events = $events+ [string]$s + "`n" }
}
}
$currentLog.lastpos = $br.BaseStream.Position
$br.close()
}
To be clear the error comes when I try to open the file. The error message is:
Exception calling "Open" with "2" argument(s): "The process cannot access the file
'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Log\ERRORLOG'
because it is being used by another process."
Gísli
So I found the answer and it was pretty simple.
The binary reader constructor takes as input a stream. I didn't define the stream seperately and that's why I didn't notice that you set the FileShare flag in the stream's constructor.
What I had to do was to change this:
{$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open))}
To this:
{$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite))}
And then it worked like a charm.
Gísli