I am trying to use Powershell to create a dhcpd.leases file for an ISC DHCP Server. I have imported an array in the variable $lease_import and have run the following code.
Beware; This code is not pretty (at all) and does not intend to be. (It is a one time use type of deal)
$input_file= Get-InputFile
$lease_import = ipcsv -Path $file_path -Delimiter "," | where {
($_.binding_state -ne "ABANDONED") -and ($_.mac_address -ne "") -and ($_.ends -match "2014")
}
$output_file = $($input_file | Split-Path) + "\dhcpd.leases"
if (Test-Path $output_file){
ri $output_file
}
ac $output_file "# The format of this file is documented in the dhcpd.leases(5) manual page.`n# This lease file was written by isc-dhcp-4.2.4-P2`n"
$end_time = $($(Get-Date).AddDays(7).ToString('yyyy-MM-dd') +" "+ $(Get-Date -Format T) -replace "-","/")
$lease_import | foreach {
ac $output_file "lease $($_.ip_address) {`n starts $($($($_.starts -replace ".......$","00") -replace "T"," ") -replace "-","/");`n ends $end_time;`n hardware ethernet $($_.mac_address);`n binding state active;`n}`n"
}
ac $output_file 'server-duid "\000\001\000\001\025\034G\301\000\000^\000\001\002";'
Running this code will generate a statement as follows:
lease 10.116.3.9 {
starts 2014/10/31 09:59:00;
ends 2014/11/09 17:01:21;
hardware ethernet 00:1c:c4:96:bc:50;
binding state active;
}
For each row in the array. Problems occur after running this script. Every statement as shown above occurs twice in dhcpd.leases. I have looked over this piece of code for hours on end but I can't figure out why this is happening. The imported CSV file does not have cloned rows. Each row is unique.
Am I overlooking something ?
EDIT: The above code would work when importing it into a DHCP server as double entries simply get applied twice, but I'd really like to figure out why they occur twice in dhcpd.leases in the first place.
Related
I'm writing a script that iterates over files that are copied from a MAC computer to a Exfat disk and checks the name of the files for Windows forbidden characters.(Writing it in PowerShell)
And hopefully replace the forbidden characters with another character, for example a "-".
Why i am doing this is because i see it as a good way for me to practice coding and it might be used in my work when we get users with a lot of local files that we want to move to Onedrive.(Onedrive has a function to rename but it doesn't touch the forbidden characters, and i don't know Bash)
The issue is when I'm trying to find the characters within the script itself, it cant find the characters if i write them in the script.(For example if i write that it should look for ">")
Even if i escape the characters it just skips it(Or rather doesn't find it).
It just skips over the file i know has one in it, at first i though it might be due to encoding, but no matter what i use as a default encoding it wont display the filename correctly.(I assume this is due to how Windows reads filenames?)
edit: These are the forbidden characters im goint to look for " * : < > ? / \ |
The script itself is able to remove letters and stuff if i ask it to.
I also tried getting the char of the byte([byte][char]"") but i get this error:
Cannot convert value "" to type "System.Byte". Error: "Value was either too large or too small for an unsigned byte." Edit: the changed by itself during the day not sure what to say.
If i just add it to the function it just returns the error:Rename-Item : The input to the script block for parameter 'NewName' failed. Exception calling "Replace" with "2" argument(s): "String cannot be of zero length.
The characters are displayed like this in visual studio code.
Any ideas are welcome or if you know of any better ways of doing it?(Maybe its better if i just learn bash. )
Here is the script itself:
edit: cleaned up the script abit and some small changes.
$provided_path = Read-Host "What Directory and its subfolders do you want to check?"
# Iterates over the folders and files within.
Write-Host "Creating list of files..."
$dictionary_filenames = ""
$dictionary_filenames = Get-ChildItem -Path $provided_path -Recurse -Force -File | Select-Object FullName,BaseName,Extension
# Resetting counter
$counter_skipped = 0
# Function for character replacment
function rename_file_name($names_function,$forbidden_char){
$old_name = $names_function.BaseName
$new_name = $names_function | Rename-Item -LiteralPath $names_function.FullName -NewName{$_.BaseName.Replace("$forbidden_char","-") + $_.extension} -PassThru
if($old_name -ne $new_name.BaseName){
Write-host "$old_name changed to $new_name"
}
}
foreach($names in $dictionary_filenames){
if($names.BaseName[0] -eq "." -and $names.BaseName[1] -match "_"){
$counter_skipped ++
$counter ++
continue
}
else{
if($names.BaseName[0] -eq " " -or $names.BaseName[-1] -eq " "){
$old_name = $names.BaseName
$new_name = $names | Rename-Item -LiteralPath $names.FullName -NewName{$_.BaseName.trim() + $_.extension} -PassThru
Write-Host "Trimming whitespace: $old_name"
}
else{
Write-Host "Trimming not needed "$names.BaseName
}
Write-host "Checking forbidden characters "$names.BaseName
rename_file_name $names ">"
rename_file_name $names "<"
rename_file_name $names "/"
}
}
Write-Host "Files checked: "$dictionary_filenames.Count
Write-Host "Files Skipped: "$counter_skipped
Read-Host 'Close window by pressing "Enter"'```
is U+F021, two bytes long or [int16]. You can put it in a utf8 with bom encoded script.
I have a requirement, in which I need to do read line by line, and then do string/character replacement in a datafile having data in windows latin 1.
I've written this powershell (my first one) initially using out-file -encoding option. However the output file thus created was doing some character translation. Then I searched and came across WriteAllLines, but I'm unable to use it in my code.
$encoding =[Text.Encoding]::GetEncoding('iso-8859-1')
$pdsname="ABCD.XYZ.PQRST"
$datafile="ABCD.SCHEMA.TABLE.DAT"
Get-Content ABCD.SCHEMA.TABLE.DAT | ForEach-Object {
$matches = [regex]::Match($_,'ABCD')
$string_to_be_replaced=$_.substring($matches.Index,$pdsname.Length+10)
$string_to_be_replaced="`"$string_to_be_replaced`""
$member = [regex]::match($_,"`"$pdsname\(([^\)]+)\)`"").Groups[1].Value
$_ -replace $([regex]::Escape($string_to_be_replaced)),$member
} | [System.IO.File]::WriteAllLines("C:\Users\USer01", "ABCD.SCHEMA.TABLE.NEW.DAT", $encoding)
With the help of an answer from #Gzeh Niert, I updated my above script. However, when I execute the script the output file being generated by the script has just the last record, as it was unable to append, and it did an overwrite, I tried using System.IO.File]::AppendAllText, but this strangely creates a larger file, and has only the last record. In short its likely that empty lines are being written.
param(
[String]$datafile
)
$pdsname="ABCD.XYZ.PQRST"
$encoding =[Text.Encoding]::GetEncoding('iso-8859-1')
$datafile = "ABCD.SCHEMA.TABLE.DAT"
$datafile2="ABCD.SCHEMA.TABLE.NEW.DAT"
Get-Content $datafile | ForEach-Object {
$matches = [regex]::Match($_,'ABCD')
if($matches.Success) {
$string_to_be_replaced=$_.substring($matches.Index,$pdsname.Length+10)
$string_to_be_replaced="`"$string_to_be_replaced`""
$member = [regex]::match($_,"`"$pdsname\(([^\)]+)\)`"").Groups[1].Value
$replacedContent = $_ -replace $([regex]::Escape($string_to_be_replaced)),$member
[System.IO.File]::AppendAllText($datafile2, $replacedContent, $encoding)
}
else {
[System.IO.File]::AppendAllText($datafile2, $_, $encoding)
}
#[System.IO.File]::WriteAllLines($datafile2, $replacedContent, $encoding)
}
Please help me figure out where I am going wrong.
System.IO.File.WriteAllLines is getting either an array of strings or an IEnumerable of strings as second parameter and cannot be piped to a command because it is not a CmdLet handling pipeline input but a .NET Framework method.
You should try storing your replaced content into a string[]to use it as parameter when saving the file.
param(
[String]$file
)
$encoding =[Text.Encoding]::GetEncoding('iso-8859-1')
$replacedContent = [string[]]#(Get-Content $file | ForEach-Object {
# Do stuff
})
[System.IO.File]::WriteAllLines($file, $replacedContent, $encoding)
I have a PowerShell script that reads a large CSV file (4GB+), finds certain lines, then writes the lines to other files.
I'm noticing that when it gets to "echo "Processed $datacounter total lines in the $datafile file"" the last line of the script, it doesn't actually finish until 5-10 minutes later.
What is it doing for that period? When it does finish, memory usage drops off significantly. Is there a way to force it to clear memory at the end of the script?
Screenshot of Memory Usage
Screenshot of script timestamps
Here is the final version of my script for reference.
# Get the filename
$datafile = Read-Host "Filename"
$dayofweek = Read-Host "Day of week (IE 1 = Monday, 2 = Tuesday..)"
$campaignWriters = #{}
# Create campaign ID hash table
$campaignByID = #{}
foreach($c in (Import-Csv 'campaigns.txt' -Delimiter '|')) {
foreach($id in ($c.CampaignID -split ' ')) {
$campaignByID[$id] = $c.CampaignName
}
foreach($cname in ($c.CampaignName)) {
$writer = $campaignWriters[$cname] = New-Object IO.StreamWriter($dayofweek + $cname + '_filtered.txt')
if($dayofweek -eq 1) {
$writer.WriteLine("ID1|ID2|ID3|ID4|ID5|ID6|Time|Time-UTC-Sec")
}
}
}
# Display the campaigns
$campaignByID.GetEnumerator() | Sort-Object Value
# Read in data file
$encoding = [Text.Encoding]::GetEncoding('iso-8859-1')
$datareader = New-Object IO.StreamReader($datafile, $encoding)
$datacounter = 0
echo "Starting.."
get-date -Format g
while (!$datareader.EndOfStream) {
$data = $datareader.ReadLine().Split('þ')
# Find the Campaign in the hashtable
$campaignName = $campaignByID[$data[3]]
if($campaignName) {
$writer = $campaignWriters[$campaignName]
# If a campaign name was returned from the hash, add the line using that campaign's writer
$writer.WriteLine(($data[20,3,5,8,12,14,0,19] -join '|'))
}
$datacounter++;
}
$datareader.Close()
foreach ($writer in $campaignWriters.Values) {
$writer.Close()
}
echo "Done!"
get-date -Format g
echo "Processed $datacounter total lines in the $datafile file"
I'm assuming that campaigns.txt is the mult-gigabyte file you are referring to. If it's the other file(s), this might not make as much sense.
If so, invoking import-csv the inside parenthesis then using the foreach statement to iterate through them is what's driving your memory usage so high. A better alternative would be use a PowerShell pipeline to stream records from the file without needing to keep all of them in memory at the same time. You achieve this by changing the foreach statment into a ForEach-Object cmdlet:
Import-Csv 'campaigns.txt' -Delimiter '|' | ForEach-Object {
foreach($id in ($_.CampaignID -split ' ')) {
$campaignByID[$id] = $_.CampaignName
}
}
The .NET garbage collector is optimized cases where the majority of objects are short-lived. Therefor this change should result in a noticeable performance increase, as well as reduced wind-down time at the end.
I advise against forcing garbage collection with [System.GC]::Collect(), the garbage collector knows best when it should run. The reasons for this are complex, if you really want to know details why this is true, Maoni's blog has a wealth of details about garbage collection in the .NET environment.
It may or may not work, but you can try to tell garbage collection to run:
[System.GC]::Collect()
You don't have fine grained control over it though, and it may help to Remove-Variable or set variables to $null for some things before running it so that there aren't references to the data anymore.
I am currently using the below PS script to check if the currents months MS patches are installed on the system. The script is set to check the $env:COMPUTERNAME.mbsa and the Patch_NA.txt file and send the result to the $env:COMPUTERNAME.csv file.
I now need to modify this script to also pull information from other POS devices in the same location (C:\Users\Cambridge\SecurityScans) and send the results to the $env:COMPUTERNAME.csv file.
The POS devices are listed like this:
172.26.210.1.mbsa
172.26.210.2.mbsa
172.26.210.3.mbsa
and so forth.
The IP range at all our locations (last octet) is 1 - 60. Any ideas on how I can set this up?
Script:
$logname = "C:\temp\PatchVerify\$env:COMPUTERNAME.csv"
[xml]$x=type "C:\Users\Cambridge\SecurityScans\$env:COMPUTERNAME.mbsa"
#This list is created based on a text file that is provided.
$montlyPatches = type "C:\Temp\PatchVerify\Patches_NA.txt"|
foreach{if ($_ -mat"-KB(? <KB>\d+)"){$matches.KB}}
$patchesNotInstalled=$x.SecScan.check | where {$_.id -eq 500} |foreach{`
$_.detail.updatedata|where {$_.isinstalled -eq "false"}}|Select -expandProperty KBID
$patchesInstalled =$x.SecScan.check | where {$_.id -eq 500} |foreach{`
$_.detail.updatedata|where {$_.isinstalled -eq "true"}}|Select -expandProperty KBID
"Store,Patch,Present"> $logname
$store = "$env:COMPUTERNAME"
foreach ($patch in $montlyPatches)
{
$result = "Unknown"
if ( $patchesInstalled -contains $patch)
{
$result = "YES"
}
if ( $patchesNotInstalled -contains $patch)
{
$result = "NO"
}
"$store,KB$($patch),$result" >>$logname
}
You can find lots of information on creating functions on the web, but a simple example would be:
Function Check-Patches{
Param($FileName)
$logname = "C:\temp\PatchVerify\$FileName.csv"
[xml]$x=type "C:\Users\Cambridge\SecurityScans\$FileName.mbsa"
The rest of your existing code goes here...
}
Check-Patches "$env:ComputerName"
For($i=1;$i -le 60;$i++){
Check-Patches "172.26.210.$i"
}
If you need me to break down anything in that let me know and I'll go into further explanation, but from what you already have it looks like you have a decent grasp on PowerShell theory and just needed to know what resources are available.
Edit: I updated my example to better fit your script, having it accept a file name, and then applying that file name to the $logname and $x variables within the function.
The break down...
First we declare that we are creating a Function using the Function keyword. Following that is the name of the function that you will use later to call it, and an opening curly brace to start the scriptblock that makes up the actual function.
Next is the Param line, which in this case is very simple only declaring one variable as input. This could alternatively be done as Function Check-Patches ($FileName){ but when you start getting into more advanced functions that only gets confusing, so my recommendation is to stick with putting the parameters inside the function's scriptblock. This is the first thing you want inside of your function in most cases, excluding any Help that you would write up for the function.
Then we have updated lines for $logname and [xml]$x that use the $FileName that the function gets as input.
After that comes all of your code that parses the patch logs, and outputs to your CSV, and the closing curly brace that ends the scriptblock, and the function.
Then we call it for the ComputerName, and run a For loop. The For loop runs everything between 1 and 60, and for each loop it uses that number as the last octet of the file name to feed into the function and check those files.
A few comments on the rest of your code. $monthlypatches = could be changed to = type | ?{$_ -match "-KB(? <KB>\d+)"}|%{$matches.KB} so that the results are filtered before the ForEach loop, which could cut down on some time.
On the $patchesInstalled and $patchesNotInstalled lines you don't need the backtick at the end of that line. You can naturally have a linebreak after the beginning of the scriptblock for a ForEach loop. Having it there can be hard to see later if the script breaks, and if there is anything after it (including a space) the script can break and throw errors that are hard to track down.
Lastly, you loop through $x twice, and then $monthlyPatches once, and do a lot of individual writes to the log file. I would suggest creating an array, filling it with custom objects that have 3 properties (Store, Patch, and Present), and then outputting that at the end of the function. That changes things a little bit, but then your function outputs an object, which you could pipe to Export-CSV, or maybe later you could want it to do something else, but at least then you'd have it. To do that I'd run $x through a switch to see if things are installed, then I'd flush out the array by setting all of the monthlypatches that aren't already in that array to Unknown. That would go something like:
Function Check-Patches{
Param($FileName)
$logname = "C:\temp\PatchVerify\$FileName.csv"
[xml]$x=type "C:\Users\Cambridge\SecurityScans\$FileName.mbsa"
$PatchStatus = #()
#This list is created based on a text file that is provided.
$monthlyPatches = GC "C:\Temp\PatchVerify\Patches_NA.txt"|?{$_ -match "-KB(? <KB>\d+)"} | %{$matches.KB}
#Create objects for all the patches in the updatelog that were in the monthly list.
Switch($x.SecScan.Check|?{$_.KBID -in $monthlyPatches -and $_.id -eq 500}){
{$_.detail.updatedata.isinstalled -eq "true"}{$PatchStatus+=[PSCustomObject][Ordered]#{Store=$FileName;Patch=$_.KBID;Present="YES"};Continue}
{$_.detail.updatedata.isinstalled -eq "false"}{$PatchStatus+=[PSCustomObject][Ordered]#{Store=$FileName;Patch=$_.KBID;Present="NO"};Continue}
}
#Populate all of the monthly patches that weren't found on the machine as installed or failed
$monthlyPatches | ?{$_ -notin $PatchStatus.Patch} | %{$PatchStatus += [PSCustomObject][Ordered]#{Store=$FileName;Patch=$_;Present="Unknown"}}
#Output results
$PatchStatus
}
#Check patches on current computer
Check-Patches "$env:ComputerName"|Export-Csv "C:\temp\PatchVerify\$env:ComputerName.csv" -NoTypeInformation
#Check patches on POS Devices
For($i=1;$i -le 60;$i++){
Check-Patches "172.26.210.$i"|Export-Csv "C:\temp\PatchVerify\172.26.210.$i.csv" -NoTypeInformation
}
I'm reading in an csv file (list of students, school, birthdays, etc) creating their login names and exporting the data to another csv that will be imported into another system.
Everything is working great except i can only get one line of data in my csv file (the last user name). i assume it is overwriting the same line each time. Doing this in powershell. help. Here's my code:
Add-PSSnapin Quest.ActiveRoles.ADManagement
#import list of students from hourly IC extract
$users = Import-Csv C:\Users\edge.brandon\Desktop\enrollment\mbcextract.csv |
where {$_.'grade' -ne 'PK' -and $_.'name' -ne 'Ombudsman' -and $_.'name' -ne 'Ombudsman MS' -and $_.'name' -ne 'z Transition Services' -and $_.'name' -ne 'z Home Services'}
if ($users) {
foreach ($u in $users) {
# sets middle name and initial to null so that variable does not cary over to next student if they have no middle name
$middle= $null
$mi= $null
$first= ($u.'firstname')
$last= ($u.'lastname')
$middle= ($u.'middlename')
$birth= ($u.'birthdate')
$grade= ($u.'grade')
$id= ($u.'studentNumber')
$schoolid= ($u.'sch.number')
#Removes spaces, apostrophes, hyphens, periods, commas from name
$first=$first.Replace(" ", "")
$middle=$middle.Replace(" ", "")
$last=$last.Replace(" ", "")
$first=$first.Replace("'", "")
$middle=$middle.Replace("'", "")
$last=$last.Replace("'", "")
$first=$first.Replace("-", "")
$middle=$middle.Replace("-", "")
$last=$last.Replace("-", "")
$first=$first.Replace(".", "")
$middle=$middle.Replace(".", "")
$last=$last.Replace(".", "")
$first=$first.Replace(",", "")
$middle=$middle.Replace(",", "")
$last=$last.Replace(",", "")
# sets 1st and middle initial. also sets mmdd of birth
$fi= $first.substring(0,1)
$mi= $middle.substring(0,1)
$mmdd =$birth.substring(0,4)
#sets username and then makes sure it truncates anything after 20 characters
$un= ($last + $fi + $mi +$mmdd)
$un= $un.substring(0,20)
}
} **$users |
select $id,$un,$first,$last,$schoolid,$grade,$upn,"1"," ","0" | export-csv MBC.csv**
Remove-PSSnapin Quest.ActiveRoles.ADManagement
i found a mistake i changed the $users to $u ($u | select $id,$un,$first,$last,$schoolid,$grade,$upn,"1"," ","0" | export-csv mbc.csv -NoTypeInformation)
and added a break point step thru the code and if i open the csv file as the code is running the first line of the csv file populates correctly with each pass. it continues to write on the first line...I tried the append statement but it didn't help...how do u make it go to the next line in the csv file (ie write a line of data on row 1, go to row 2 write another line of data)
I can't provide a full answer (I don't have csv data to test with offhand and don't know the relevant bits of powershell well enough from memory to comment specifically) but at the very least part of the problem is likely that you are using the values of variables in your select statement and I can't imagine you actually meant to do that.
I imagine you meant to grab the properties with those names instead (select id,un,first,last,... instead of select $id,$un,$first,$last,...).
I know this is nearly two years old, but others might find this topic.
The problem here is that the export-csv will recreate the file each time it is called. Combine that with the fact that you are sending export-csv the variable values calculated for the last record in $users, and that would explain your results.
I found an article that outlines how to do what you're intending to do (I have a similar project that I'm starting on). Working with Custom Objects: https://technet.microsoft.com/en-us/library/ff730946.aspx
As in the article, you would create a array of custom objects, then load each object with property values from your calculated variable values, as you read through the objects in $users
At the end, you would export your array of objects (not $users or $un) using the export-csv command.