Output Values from gc into Hash Table - powershell

Trying to make a hash table with 2 categories: Users and Passwords.
This is my code thus far but the issue is the output only displays the command and does not execute it.
for ($i=1; $i -le 10; $i++){
$caps = [char[]] "ABCDEFGHJKMNPQRSTUVWXY"
$lows = [char[]] "abcdefghjkmnpqrstuvwxy"
$nums = [char[]] "2346789"
$spl = [char[]] "!##$%^&*?+"
$first = $lows | Get-Random -count 1;
$second = $caps | Get-Random -count 1;
$third = $nums | Get-Random -count 1;
$forth = $lows | Get-Random -count 1;
$fifth = $spl | Get-Random -count 1;
$sixth = $caps | Get-Random -count 1;
$pwd = [string](#($first) + #($second) + #($third) + #($forth) + #($fifth) + #($sixth))
Out-File C:\Users\Administrator\Documents\L8_userpasswords.txt -InputObject $pwd -Append
}
$here = #'
$users=Get-Content C:\\Users\\Administrator\\Desktop\\L8_users.txt
$passwords=Get-Content C:\\Users\\Administrator\\Documents\\L8_userpasswords.txt
'#
convertfrom-stringdata -stringdata $here
This is the output I am getting:
PS C:\Users\Administrator> C:\Users\Administrator\Documents\l8.ps1
Name Value
---- -----
$users Get-Content C:\Users\Administrator\Desktop\Lab8_users.txt
$passwords Get-Content C:\Users\Administrator\Documents\L8_userpasswords.txt

I think you want this, which will turn the list of users and passwords into a HashTable, and then cast it to a PSCustomObject, which will have two properties: Users and Passwords.
$Data = [PSCustomObject]#{
Users = Get-Content -Path C:\Users\Administrator\Desktop\L8_users.txt;
Passwords = Get-Content -Path C:\Users\Administrator\Desktop\L8_userpasswords.txt;
}
$Data;

Or hey, you could probably just replace the entire script with a one liner:
GC C:\Users\Administrator\Desktop\L8_users.txt|%{[PSCustomObject]#{User=$_;Password=[System.Web.Security.Membership]::GeneratePassword(10,3)}}
Unless you are super attached to your password generation loop that is. [System.Web.Security.Membership]::GeneratePassword(X,Y) will generate complex passwords where X is the length and Y is the number of special characters (the rest will be a random mix of upper case letters, lower case letters, and numbers). So in my code (10,3) is a 10 character password with 3 non-alphanumeric characters.
You want it saved to a file? Pipe that to Export-CSV. Or assign it to a variable by prefixing it with something like $UserList = <code>.
Or if you really, really want a Hash Table you could make an empty one and then alter it just a little to add each pair to the table like this:
$UserList = #{}
GC C:\Users\Administrator\Desktop\L8_users.txt|%{$UserList.add($_,[System.Web.Security.Membership]::GeneratePassword(10,3))}

Assuming that L8_users.txt and L8_userpasswords.txt contain the same number of items, you could do something like this:
$users = Get-Content 'C:\Users\Administrator\Desktop\L8_users.txt'
$passwords = Get-Content 'C:\Users\Administrator\Documents\L8_userpasswords.txt'
$userpasswords = #{}
for ($i = 0; i -lt $users.Length; $i++) {
$userpasswords[$users[$i]] = $passwords[$i]
}

Related

Split values table for extract with powershell

I would like to make a new line in my hashtable to extract it in a csv.
I initialize my variable in hastable
$vlr=#{}
$vlr["OS"]=,#("test","test2")
I extract my variable in a .csv
$Output += New-Object PSObject -Property $vlr
$output | Convert-OutputForCSV | export-csv -NoTypeInformation -Delimiter ";" -Path $filepath
and the problem is in the extraction the result of the values ​​is on the same line
My goal is that each value is in a different line
You might want to use the Out-String cmdlet for this:
$vlr=#{}
$vlr["OS"]=,#("test","test2") | Out-String
$Object = New-Object PSObject -Property $vlr
$Object | ConvertTo-Csv
"OS"
"test
test2
"
this solution does not work because in the case where $vlr with several names the extraction will be complicated
$vlr=#{}
$vlr["OS"]=,#("test","test2")
$vlr["PS"]=,#("lous","tique")
it's a problem
https://gallery.technet.microsoft.com/scriptcenter/Convert-OutoutForCSV-6e552fc6
For the function Convert-OutputForCSV
I don't know what the posted function does, but you can make your own function to handle a single-key or multi-key hash table provided all of the key value counts are the same.
function Convert-OutputForCsv {
param(
[parameter(ValueFromPipeline)]
[hashtable]$hash
)
# Array of custom object property names
$keys = [array]$hash.Keys
# Loop through each key's values
for ($i = 0; $i -lt $hash.($keys[0]).count; $i++) {
# Custom object with keys as properties. Property values are empty.
$obj = "" | Select $keys
# Loop through key names
for ($j = 0; $j -lt $keys.Count; $j++) {
$obj.($keys[$j]) = $hash.($Keys[$j])[$i]
}
$obj
}
}
$vlr=[ordered]#{}
$vlr["OS"]='test','test2'
$vlr["PS"]='lous','tique'
$vlr | Convert-OutputForCsv | Export-Csv -NoTypeInformation -Delimiter ";" -Path $filepath
Honestly, if you are in control of the input data, I would just type out a CSV instead of typing out hash tables.
this solution is good in my simplified case but not adapted to my case unfortunately
I'm merging my old base2 array with my new base array and my goal is to concatenate the values ​​in an excel to make them usable
$base2 = Get-content $filepath2 | select -first 1
$base2 = $base2 -split ";"
$base2 = $base2.Replace("`"", "")
$cunt2 = $base2.count - 1
$h2 = ipcsv $filepath2 -Delimiter ";"
$HashTable2 = #{}
for ($i = 0 ; $i -le $cunt2 ; $i++) {
foreach ($r in $h2) {
$HashTable2[$base2[$i]] = $r.($base2[$i])
}
base2 = old tables
$base = Get-content $filepath2 | select -first 1
$base = $base -split ";"
$base = $base.Replace("`"", "")
$cunt = $base.count - 1
$h1 = ipcsv $filepath -Delimiter ";"
$HashTable = #{}
for ($i = 0 ; $i -le $cunt ; $i++) {
foreach ($r in $h1) {
$HashTable[$base[$i]] = $r.($base[$i])
}
New tables $base
once the two arrays are initialized, I merge them and this is where I have to separate the values ​​row by row
$csvfinal = $hashtable, $hashtable2 | Merge-Hashtables

PowerShell Slowness Two Large CSV Files

I have this script working, but with 100k+ rows in File1 and 200k+ in file 2, it will take days to complete. I got the where.({ part down to less than a second, with both csv files as data tables, but with that route I can't get the data out the way I want. This script outputs the data the way I want, but it takes 4 seconds per lookup. What can I do to speed this up?
I thought containskey somewhere might help, but on PRACT_ID there is a one to many relationship, so not sure how to handle those? Thx.
Invoke-Expression "C:\SHC\MSO\DataTable\functionlibrary.ps1"
[System.Data.DataTable]$Script:MappingTable = New-Object System.Data.DataTable
$File1 = Import-csv "C:\File1.csv" -Delimiter '|' | Sort-Object PRACT_ID
$File2 = Get-Content "C:\File2.csv" | Select-Object -Skip 1 | Sort-Object
$Script:MappingTable = $File1 | Out-DataTable
$Logs = "C:\Testing1.7.csv"
[System.Object]$UserOutput = #()
foreach ($name in $File1) {
[string]$userMatch = $File2.Where( { $_.Split("|")[0] -eq $name.PRACT_ID })
if ($userMatch) {
# Process the data
$UserOutput += New-Object PsObject -property #{
ID_NUMBER = $name.ID_NUMBER
PRACT_ID = $name.PRACT_ID
LAST_NAME = $name.LAST_NAME
FIRST_NAME = $name.FIRST_NAME
MIDDLE_INITIAL = $name.MIDDLE_INITIAL
DEGREE = $name.DEGREE
EMAILADDRESS = $name.EMAILADDRESS
PRIMARY_CLINIC_PHONE = $name.PRIMARY_CLINIC_PHONE
SPECIALTY_NAME = $name.SPECIALTY_NAME
State_License = $name.State_License
NPI_Number = $name.NPI_Number
'University Affiliation' = $name.'University Affiliation'
Teaching_Title = $name.Teaching_Title
FACILITY = $userMatch
}
}
}
$UserOutput | Select-Object ID_NUMBER, PRACT_ID, LAST_NAME, FIRST_NAME, MIDDLE_INITIAL, DEGREE, EMAILADDRESS, PRIMARY_CLINIC_PHONE, SPECIALTY_NAME, State_License, NPI_Number, 'University Affiliation', Teaching_Title, FACILITY |
Export-Csv $logs -NoTypeInformation
Load $File2 into a hashtable with the $_.Split('|')[0] value as the key - you can then also skip the object creation completely and offload everything to Select-Object:
$File2 = Get-Content "C:\File2.csv" | Select-Object -Skip 1 | Sort-Object
# load $file2 into hashtable
$userTable = #{}
foreach($userEntry in $File2){
$userTable[$userEntry.Split('|')[0]] = $userEntry
}
# prepare the existing property names we want to preserve
$propertiesToSelect = 'ID_NUMBER', 'PRACT_ID', 'LAST_NAME', 'FIRST_NAME', 'MIDDLE_INITIAL', 'DEGREE', 'EMAILADDRESS', 'PRIMARY_CLINIC_PHONE', 'SPECIALTY_NAME', 'State_License', 'NPI_Number', 'University Affiliation', 'Teaching_Title'
# read file, filter on existence in $userTable, add the FACILITY calculated property before export
Import-csv "C:\File1.csv" -Delimiter '|' |Where-Object {$userTable.ContainsKey($_.PRACT_ID)} |Select-Object $propertiesToSelect,#{Name='FACILITY';Expression={$userTable[$_.PRACT_ID]}} |Export-Csv $logs -NoTypeInformation
There are a multitude of ways to increase the speed of the operations you're doing and can be broken down to in-script and out-of-script-possibilities:
Out of Script Possibilities:
Since the files are large, how much memory does the machine you're running this on have? And are you maxing it out during this operation?
If you are paging to disk, this will be the single biggest impact to the overall process!
If you are, the two ways to address this are:
Throw more hardware at the problem (easiest to deal with)
Write your code to iterate over each file small chunks at a time so you don't load it all into RAM at the same time. (very difficult if you're not familiar with it)
In-Script:
Dont use #() with += (it's really slow (especially over large datasets))
Use instead an ArrayList. Here is a quick sample of the perf difference (ArrayList ~40x faster on 10,000 and 500x faster on 100,000 entries, consistently -- this difference gets larger as the dataset gets larger or in other words, #() += gets slower as the dataset gets bigger)):
(Measure-Command {
$arr = [System.Collections.ArrayList]::new()
1..100000 | % {
[void]$arr.Add($_)
}
}).TotalSeconds
(Measure-Command {
$arr = #()
1..100000 | % {
$arr += $_
}
}).TotalSeconds
0.8258113
451.5413987
If you need to do multiple key-based lookups on the data, iterating over the data millions of times will be slow. Import the data as a CSV and then structure a couple of hashtables with the associated information with key -> data and/or key -> data[] and then you can do index lookups instead of iterating through the arrays millions of times... will be MUCH faster; assuming you have available RAM for the extra objects..
EDIT for #RoadRunner:
My experience with GC may be old... it used to be horrendously slow on large files but appears in newer PowerShell versions, may have been fixed:
[System.IO.File]::WriteAllLines("$($Env:UserProfile)\Desktop\10MB.txt", ('8' * 10MB))
[System.IO.File]::WriteAllLines("$($Env:UserProfile)\Desktop\50MB.txt", ('8' * 50MB))
[System.IO.File]::WriteAllLines("$($Env:UserProfile)\Desktop\100MB.txt", ('8' * 100MB))
[System.IO.File]::WriteAllLines("$($Env:UserProfile)\Desktop\500MB.txt", ('8' * 500MB))
$10MB = gi .\10MB.txt
$50MB = gi .\50MB.txt
$100MB = gi .\100MB.txt
$500MB = gi .\500MB.txt
0..10 | % {
$n = [pscustomobject] #{
'GC_10MB' = (Measure-Command { Get-Content $10MB }).TotalSeconds
'RAL_10MB' = (Measure-Command { [System.IO.File]::ReadAllLines($10MB) }).TotalSeconds
'GC_50MB' = (Measure-Command { Get-Content $50MB }).TotalSeconds
'RAL_50MB' = (Measure-Command { [System.IO.File]::ReadAllLines($50MB) }).TotalSeconds
'GC_100MB' = (Measure-Command { Get-Content $100MB }).TotalSeconds
'RAL_100MB' = (Measure-Command { [System.IO.File]::ReadAllLines($100MB) }).TotalSeconds
'GC_500MB' = (Measure-Command { Get-Content $500MB }).TotalSeconds
'RAL_500MB' = (Measure-Command { [System.IO.File]::ReadAllLines($500MB) }).TotalSeconds
'Delta_10MB' = $null
'Delta_50MB' = $null
'Delta_100MB' = $null
'Delta_500MB' = $null
}
$n.Delta_10MB = "{0:P}" -f ($n.GC_10MB / $n.RAL_10MB)
$n.Delta_50MB = "{0:P}" -f ($n.GC_50MB / $n.RAL_50MB)
$n.Delta_100MB = "{0:P}" -f ($n.GC_100MB / $n.RAL_100MB)
$n.Delta_500MB = "{0:P}" -f ($n.GC_500MB / $n.RAL_500MB)
$n
}

How to set a variable to a column with no header in a tab delimited text file

Barcode1 Plate # 12/29/2017 07:35:56 EST
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
Above is an example of a tab delimited text file. I need to get the data from the column with no header; namely, the columns at the end and I don't know how to identify it. I am trying to swap columns and output a text file. The source data file format is the same every time.
This is part of what I have:
$swapColumns = #{
column1 = #{
name = "date-header"
instance = 1
}
column2 = #{
name = "Blank"
instance = 1
}
}
$formats = #(
'XR-{0:yyyyMMdd}-01.txt'
)
$date = [datetime]::now
$ErrorActionPreference = 'Stop'
function Get-HeaderIndex {
param(
[System.Collections.Generic.List[string]]$Source,
[string]$Header,
[uint16]$Instance
)
$index = 0;
for ($i = 0; $i -lt $Instance; $i++) {
$index = $Source.IndexOf($Header, $index, ($Source.Count - $index))
if (($index -eq -1) -or (($i + 1) -eq $Instance)) {
break
}
$index = $index + 1
}
if ($index -eq -1) { throw "index not found" }
return $index
}
#grabs the first item in folder matching UCX-*.txt
$fileDetails = Get-ChildItem $PSScriptRoot\UCX-*.txt | select -First 1
#gets the file contents
$file = Get-Content $fileDetails
#break up script in sections that look like '======section======'
#and store the section name and line number it starts on
$sections = #()
for ($i = 0; $i -lt $file.Count; $i++) {
if ($file[$i] -match '^=+(\w+)=+$') {
$section = $Matches[1]
$sections += [pscustomobject]#{line = $i; header = $section}
}
}
#get the data section
$dataSection = $sections | ? {$_.header -eq 'data'}
#get the section following data
$nextSection = $sections | ? {$_.line -gt $dataSection.line} | sort
-Property line | select -First 1
#get data column headers
$dataHeaders = New-Object System.Collections.Generic.List[string]
$file[$dataSection.line + 1].split("`t") | % {
[datetime]$headerDateValue = [datetime]::MinValue
$headerIsDate = [datetime]::TryParse($_.Replace('EST','').Trim(),
[ref] $headerDateValue)
if ($headerIsDate) {
$dataHeaders.Add('date-header')
}
else {
$dataHeaders.Add($_)
}
}
#get index of columns defined in $swapColumns
$column1 = Get-HeaderIndex -Source $dataHeaders -Header
$swapColumns.column1.name -Instance $swapColumns.column1.instance
$column2 = Get-HeaderIndex -Source $dataHeaders -Header
swapColumns.column2.name -Instance $swapColumns.column2.instance
#iterate over each row in data section, swap data from column1/column2
for ($i = $dataSection.line + 2; $i -lt $nextSection.line - 1; $i++) {
$line = $file[$i]
$parts = $line.split("`t")
$tmp1 = $parts[$column1]
$parts[$column1] = $parts[$column2]
$parts[$column2] = $tmp1
$file[$i] = $parts -join "`t"
}
#write new file contents to files with names defined in $formats
$formats | % {
$file | Out-File ($_ -f $date) -Force
}
If you know what your file format is going to be then forget whatever the current header is and assume when we convert the file to a CSV object.
It looks like you need to parse the date of out the header which should be trivial. Grab it from $fileheader however you would like.
$wholeFile = Get-Content C:\temp\test.txt
$fileHeader = $wholeFile[0] -split "`t"
$newHeader = "Barcode1", "Plate #", "Date", "Plumbus", "Dinglebop"
$wholeFile |Select-Object -Skip 1 | ConvertFrom-Csv -Delimiter "`t" -Header $newHeader
If the columns length is always the same, there's another option, specify manually the width of the columns, See example:
$content = Get-Content C:\temp.tsv
$columns = 13, 24, 35 | Sort -Descending
$Delimiter = ','
$Results = $content | % {
$line = $_
$columns | % {
$line = $line.Insert($_, $Delimiter)
}
$line
} |
ConvertFrom-Csv -Delimiter $Delimiter
Results:
Barcode1 Plate # H1 12/29/2017 07:35:56 EST
--------- ----------- -- -----------------------
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
Then you can easily get the data you need:
$Results[0].H1
4
[This answer doesn't solve the OP's problem after clarifying the exact requirements, but may be of general interest to some, given the question's generic title.]
If the file is really tab-delimited, you can use Import-Csv -Delimiter "`t" to read it, in which case PowerShell will autogenerate header names as H<n> if they're missing, where <n> is a sequence number starting with 1.
Caveat: This doesn't work if the unnamed column is the last one, because - inexplicably - Import-Csv then ignores the entire column (more generally, any run of trailing delimiters).
Import-Csv -Delimiter "`t" file.tsv | Select-Object -ExpandProperty H1

Powershell - Empty entries in CSV

I don't have much experience with CSV, so apologies if I'm really blind here.
I have a basic CSV and script setup to test this with. The CSV has two columns, Letter and Number. Letter goes from A-F and Number goes from 1-10. This means that Number has more rows than Letter, so when running the following script, the output can sometimes provide an empty Letter.
$L = ipcsv ln.csv | Get-Random | Select-Object -ExpandProperty Letter
$N = ipcsv ln.csv | Get-Random | Select-Object -ExpandProperty Number
Write-Output $L
Write-Output $N
Some outputs come out as
B
9
while others can come out as
5
I don't know whether the issue is my script not ignoring empty lines or my CSV being written incorrectly, which is posted below.
Letter,Number
A,1
B,2
C,3
D,4
E,5
F,6
,7
,8
,9
,10
What's my issue here and how do I go about fixing it?
Your asking for a random object from your CSV, not a random letter. Since some of the lines are missing a letter, you might end up picking one that has an empty Letter-value.
If you want to pick any line with a letter, you need to filter the rows first to only pick from the ones with a value. Also, you sould avoid reading the same file twice, use a varible
#$csv = Import-CSV -Path ln.csv
$csv = #"
Letter,Number
A,1
B,2
C,3
D,4
E,5
F,6
,7
,8
,9
,10
"# | ConvertFrom-Csv
$L = $csv | Where-Object { $_.Letter } | Get-Random | Select-Object -ExpandProperty Letter
$N = $csv | Where-Object { $_.Number } | Get-Random | Select-Object -ExpandProperty Number
Write-Output $L
Write-Output $N
CSV migtht not be the best solution for this scenario. Ex. you could store these as arrays in the script, like:
$chars = [char[]](65..70) #A-F uppercase letters
$numbers = 1..10
$L = $chars | Get-Random
$N = $numbers | Get-Random
Write-Output $L
Write-Output $N
Import-Csv turns each line into an object, with a property for each column.
Even though one or more property values may be empty, the object still exists, and Get-Random has no reason determine that an object with a certain property (such as Letter) having the value "" (ie. an empty string), should not be picked.
You can fix this by expanding the property values first, then filter for empty values and then finally pick the random value from those that weren't empty:
$L = ipcsv ln.csv |Select-Object -ExpandProperty Letter |Where-Object {$_} |Get-Random
$N = ipcsv ln.csv |Select-Object -ExpandProperty Number |Where-Object {$_} |Get-Random

powershell concat all columns in a row

i have 20+ columns in a csv file like
empid ename deptid mgrid hiredon col6 .... col20
10 a 10 5 10-may-2010
11 b 10 5 08-aug-2005
12 c 11 3 11-dec-2008
i would like to get the output as csv like
empid, all_other_details
10 , {ename:a;deptid:10;mgrid:5; like this for all 19 columns }
except employee id all other columns should be wrapped into a string containing key:value pairs. Is there a way to join all the columns without mentioning each column as $_. ?
I have come up with this, I hope comments are self explanatory.
It should work with 2 or more columns.
Delimiters can be changed (on my computer, CSV delimiter is ; not , for example, and I know it can be different with other Cultures).
#declare delimiters
$CSVdelimiter = ";"
$detailsDelimiter = ","
#load file in array
$data = Get-Content "Book1.csv"
#isolate headers
$headers = $data[0].Split($CSVdelimiter)
#declare row counter
$rowCount = 0
#declare results array with headers
$results = #($headers[0] + "$CSVdelimiter`details")
#for each row except first
$data | Select-Object -Skip 1 | % {
#split on $csvDelimiter
$rowArray = $_.Split($CSVdelimiter)
#declare details array
$details = #()
#for each column except first
for($i = 1; $i -lt $rowArray.Count; $i++) {
#add to details array (header:value)
$details += $headers[$i] + ":" + $rowArray[$i]
}
#join details array with $detailsDelimiter to build new row
#append to first column value
#add to results array
$results += "$($rowArray[0])$CSVdelimiter{$($details -join $detailsDelimiter)}"
#increment row counter
$rowCount++
}
#output results to new csv file
$results | Out-File "Book2.csv"
Output looks like this :
empid;details
10;{ename:a,deptid:10,mgrid:5,hiredon:10-may-2010}
11;{ename:b,deptid:10,mgrid:5,hiredon:08-aug-2005}
12;{ename:c,deptid:11,mgrid:3,hiredon:11-dec-2008}
Try this:
$csv = Get-Content .\input_file.csv
$keys = $csv[0] -split '\s+'
$c = $keys.count - 1
$keys = ($keys[1..$c] | % {$i = -1}{$i += 1; "$($_):{$i}"}) -join '; '
$csv[1..($csv.count -1)] | % {
$a = $_ -split '\s+'
New-Object psobject -Property #{
empid = $a[0]
all_other_details = "{$($keys -f $a[1..$c])}"
}
} | Export-Csv output_file.csv -NoTypeInformation