Split values table for extract with powershell - powershell

I would like to make a new line in my hashtable to extract it in a csv.
I initialize my variable in hastable
$vlr=#{}
$vlr["OS"]=,#("test","test2")
I extract my variable in a .csv
$Output += New-Object PSObject -Property $vlr
$output | Convert-OutputForCSV | export-csv -NoTypeInformation -Delimiter ";" -Path $filepath
and the problem is in the extraction the result of the values ​​is on the same line
My goal is that each value is in a different line

You might want to use the Out-String cmdlet for this:
$vlr=#{}
$vlr["OS"]=,#("test","test2") | Out-String
$Object = New-Object PSObject -Property $vlr
$Object | ConvertTo-Csv
"OS"
"test
test2
"

this solution does not work because in the case where $vlr with several names the extraction will be complicated
$vlr=#{}
$vlr["OS"]=,#("test","test2")
$vlr["PS"]=,#("lous","tique")
it's a problem
https://gallery.technet.microsoft.com/scriptcenter/Convert-OutoutForCSV-6e552fc6
For the function Convert-OutputForCSV

I don't know what the posted function does, but you can make your own function to handle a single-key or multi-key hash table provided all of the key value counts are the same.
function Convert-OutputForCsv {
param(
[parameter(ValueFromPipeline)]
[hashtable]$hash
)
# Array of custom object property names
$keys = [array]$hash.Keys
# Loop through each key's values
for ($i = 0; $i -lt $hash.($keys[0]).count; $i++) {
# Custom object with keys as properties. Property values are empty.
$obj = "" | Select $keys
# Loop through key names
for ($j = 0; $j -lt $keys.Count; $j++) {
$obj.($keys[$j]) = $hash.($Keys[$j])[$i]
}
$obj
}
}
$vlr=[ordered]#{}
$vlr["OS"]='test','test2'
$vlr["PS"]='lous','tique'
$vlr | Convert-OutputForCsv | Export-Csv -NoTypeInformation -Delimiter ";" -Path $filepath
Honestly, if you are in control of the input data, I would just type out a CSV instead of typing out hash tables.

this solution is good in my simplified case but not adapted to my case unfortunately
I'm merging my old base2 array with my new base array and my goal is to concatenate the values ​​in an excel to make them usable
$base2 = Get-content $filepath2 | select -first 1
$base2 = $base2 -split ";"
$base2 = $base2.Replace("`"", "")
$cunt2 = $base2.count - 1
$h2 = ipcsv $filepath2 -Delimiter ";"
$HashTable2 = #{}
for ($i = 0 ; $i -le $cunt2 ; $i++) {
foreach ($r in $h2) {
$HashTable2[$base2[$i]] = $r.($base2[$i])
}
base2 = old tables
$base = Get-content $filepath2 | select -first 1
$base = $base -split ";"
$base = $base.Replace("`"", "")
$cunt = $base.count - 1
$h1 = ipcsv $filepath -Delimiter ";"
$HashTable = #{}
for ($i = 0 ; $i -le $cunt ; $i++) {
foreach ($r in $h1) {
$HashTable[$base[$i]] = $r.($base[$i])
}
New tables $base
once the two arrays are initialized, I merge them and this is where I have to separate the values ​​row by row
$csvfinal = $hashtable, $hashtable2 | Merge-Hashtables

Related

Find out Text data in CSV File Numeric Columns in Powershell

I am very new in powershell.
I am trying to validate my CSV file by finding out if there is any text value in my numeric fields. I can define with columns are numeric.
This is my source data like this
ColA ColB ColC ColD
23 23 ff 100
2.30E+01 34 2.40E+01 23
df 33 ss df
34 35 36 37
I need output something like this (only text values if found in any column)
ColA ColC ColD
2.30E+01 ff df
df 2.40E+01
ss
I have tried some code but not getting any results, get only some output like as under
System.Object[]
---------------
xxx fff' ddd 3.54E+03
...
This is what I was trying
#
cls
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$arrResult = #()
$arraycol = #()
$FileCol = #("ColA","ColB","ColC","ColD")
$dif_file_path = "C:\Users\$env:username\desktop\f2.csv"
#Importing CSVs
$dif_file = Import-Csv -Path $dif_file_path -Delimiter ","
############## Test Datatype (Is-Numeric)##########
foreach($col in $FileCol)
{
foreach ($line in $dif_file) {
$val = $line.$col
$isnum = Is-Numeric($val)
if ($isnum -eq $false) {
$arrResult += $line.$col
$arraycol += $col
}
}
}
[pscustomobject]#{$arraycol = "$arrResult"}| out-file "C:\Users\$env:username\Desktop\Errors1.csv"
####################
can someone guide me right direction?
Thanks
You can try something like this,
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$dif_file_path = "C:\Users\$env:username\desktop\f2.csv"
#Importing CSVs
$dif_file = Import-Csv -Path $dif_file_path -Delimiter ","
#$columns = $dif_file | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
# Use this to specify certain columns
$columns = "ColB", "ColC", "ColD"
foreach($row in $dif_file) {
foreach ($col in $columns) {
if ($col -in $columns) {
if (!(Is-Numeric $row.$col)) {
$row.$col = ""
}
}
}
}
$dif_file | Export-Csv C:\temp\formatted.txt
Look up name of columns as you go
Look up values of each col in each row and if it is not numeric, change to ""
Exported updated file.
I think not displaying columns that have no data creates the challenge here. You can do the following:
$csv = Import-Csv "C:\Users\$env:username\desktop\f2.csv"
$finalprops = [collections.generic.list[string]]#()
$out = foreach ($line in $csv) {
$props = $line.psobject.properties | Where {$_.Value -notmatch '^[\d\.]+$'} |
Select-Object -Expand Name
$props | Where {$_ -notin $finalprops} | Foreach-Object { $finalprops.add($_) }
if ($props) {
$line | Select $props
}
$out | Select-Object ($finalprops | Sort)
Given the nature of Format-Table or tabular output, you only see the properties of the first object in the collection. So if object1 has ColA only, but object2 has ColA and ColB, you only see ColA.
The output order you want is quite different than the input CSV; you're tracking bad text data not by first occurrence, but by column order, which requires some extra steps.
test.csv file contents:
ColA,ColB,ColC,ColD
23,23,ff,100
2.30E+01,34,2.40E+01,23
df,33,ss,df
34,35,36,37
Sample code tested to meet your description:
$csvIn = Import-Csv "$PSScriptRoot\test.csv";
# create working data set with headers in same order as input file
$data = [ordered]#{};
$csvIn[0].PSObject.Properties | foreach {
$data.Add($_.Name, (New-Object System.Collections.ArrayList));
};
# add fields with text data
$csvIn | foreach {
$_.PSObject.Properties | foreach {
if ($_.Value -notmatch '^-?[\d\.]+$') {
$null = $data[$_.Name].Add($_.Value);
}
}
}
$removes = #(); # remove `good` columns with numeric data
$rowCount = 0; # column with most bad values
$data.GetEnumerator() | foreach {
$badCount = $_.Value.Count;
if ($badCount -eq 0) { $removes += $_.Key; }
if ($badCount -gt $rowCount) { $rowCount = $badCount; }
}
$removes | foreach { $data.Remove($_); }
0..($rowCount - 1) | foreach {
$h = [ordered]#{};
foreach ($key in $data.Keys) {
$h.Add($key, $data[$key][$_]);
}
[PSCustomObject]$h;
} |
Export-Csv -NoTypeInformation -Path "$PSScriptRoot\text-data.csv";
output file contents:
"ColA","ColC","ColD"
"2.30E+01","ff","df"
"df","2.40E+01",
,"ss",
#Jawad, Finally I have tried
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$arrResult = #()
$columns = "ColA","ColB","ColC","ColD"
$dif_file_path = "C:\Users\$env:username\desktop\f1.csv"
$dif_file = Import-Csv -Path $dif_file_path -Delimiter "," |select $columns
$columns = $dif_file | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
foreach($row in $dif_file) {
foreach ($col in $columns) {
$val = $row.$col
$isnum = Is-Numeric($val)
if ($isnum -eq $false) {
$arrResult += $col+ " " +$row.$col
}}}
$arrResult | out-file "C:\Users\$env:username\desktop\Errordata.csv"
I get correct result in my out file, order is very ambiguous like
ColA ss
ColB 5.74E+03
ColA ss
ColC rrr
ColB 3.54E+03
ColD ss
ColB 8.31E+03
ColD cc
any idea to get proper format? thanks
Note: with your suggested code, I get complete source file with all data , not the specific error data.

How to set a variable to a column with no header in a tab delimited text file

Barcode1 Plate # 12/29/2017 07:35:56 EST
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
A 1 4 5 6
Above is an example of a tab delimited text file. I need to get the data from the column with no header; namely, the columns at the end and I don't know how to identify it. I am trying to swap columns and output a text file. The source data file format is the same every time.
This is part of what I have:
$swapColumns = #{
column1 = #{
name = "date-header"
instance = 1
}
column2 = #{
name = "Blank"
instance = 1
}
}
$formats = #(
'XR-{0:yyyyMMdd}-01.txt'
)
$date = [datetime]::now
$ErrorActionPreference = 'Stop'
function Get-HeaderIndex {
param(
[System.Collections.Generic.List[string]]$Source,
[string]$Header,
[uint16]$Instance
)
$index = 0;
for ($i = 0; $i -lt $Instance; $i++) {
$index = $Source.IndexOf($Header, $index, ($Source.Count - $index))
if (($index -eq -1) -or (($i + 1) -eq $Instance)) {
break
}
$index = $index + 1
}
if ($index -eq -1) { throw "index not found" }
return $index
}
#grabs the first item in folder matching UCX-*.txt
$fileDetails = Get-ChildItem $PSScriptRoot\UCX-*.txt | select -First 1
#gets the file contents
$file = Get-Content $fileDetails
#break up script in sections that look like '======section======'
#and store the section name and line number it starts on
$sections = #()
for ($i = 0; $i -lt $file.Count; $i++) {
if ($file[$i] -match '^=+(\w+)=+$') {
$section = $Matches[1]
$sections += [pscustomobject]#{line = $i; header = $section}
}
}
#get the data section
$dataSection = $sections | ? {$_.header -eq 'data'}
#get the section following data
$nextSection = $sections | ? {$_.line -gt $dataSection.line} | sort
-Property line | select -First 1
#get data column headers
$dataHeaders = New-Object System.Collections.Generic.List[string]
$file[$dataSection.line + 1].split("`t") | % {
[datetime]$headerDateValue = [datetime]::MinValue
$headerIsDate = [datetime]::TryParse($_.Replace('EST','').Trim(),
[ref] $headerDateValue)
if ($headerIsDate) {
$dataHeaders.Add('date-header')
}
else {
$dataHeaders.Add($_)
}
}
#get index of columns defined in $swapColumns
$column1 = Get-HeaderIndex -Source $dataHeaders -Header
$swapColumns.column1.name -Instance $swapColumns.column1.instance
$column2 = Get-HeaderIndex -Source $dataHeaders -Header
swapColumns.column2.name -Instance $swapColumns.column2.instance
#iterate over each row in data section, swap data from column1/column2
for ($i = $dataSection.line + 2; $i -lt $nextSection.line - 1; $i++) {
$line = $file[$i]
$parts = $line.split("`t")
$tmp1 = $parts[$column1]
$parts[$column1] = $parts[$column2]
$parts[$column2] = $tmp1
$file[$i] = $parts -join "`t"
}
#write new file contents to files with names defined in $formats
$formats | % {
$file | Out-File ($_ -f $date) -Force
}
If you know what your file format is going to be then forget whatever the current header is and assume when we convert the file to a CSV object.
It looks like you need to parse the date of out the header which should be trivial. Grab it from $fileheader however you would like.
$wholeFile = Get-Content C:\temp\test.txt
$fileHeader = $wholeFile[0] -split "`t"
$newHeader = "Barcode1", "Plate #", "Date", "Plumbus", "Dinglebop"
$wholeFile |Select-Object -Skip 1 | ConvertFrom-Csv -Delimiter "`t" -Header $newHeader
If the columns length is always the same, there's another option, specify manually the width of the columns, See example:
$content = Get-Content C:\temp.tsv
$columns = 13, 24, 35 | Sort -Descending
$Delimiter = ','
$Results = $content | % {
$line = $_
$columns | % {
$line = $line.Insert($_, $Delimiter)
}
$line
} |
ConvertFrom-Csv -Delimiter $Delimiter
Results:
Barcode1 Plate # H1 12/29/2017 07:35:56 EST
--------- ----------- -- -----------------------
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
A 1 4 5
Then you can easily get the data you need:
$Results[0].H1
4
[This answer doesn't solve the OP's problem after clarifying the exact requirements, but may be of general interest to some, given the question's generic title.]
If the file is really tab-delimited, you can use Import-Csv -Delimiter "`t" to read it, in which case PowerShell will autogenerate header names as H<n> if they're missing, where <n> is a sequence number starting with 1.
Caveat: This doesn't work if the unnamed column is the last one, because - inexplicably - Import-Csv then ignores the entire column (more generally, any run of trailing delimiters).
Import-Csv -Delimiter "`t" file.tsv | Select-Object -ExpandProperty H1

powershell concat all columns in a row

i have 20+ columns in a csv file like
empid ename deptid mgrid hiredon col6 .... col20
10 a 10 5 10-may-2010
11 b 10 5 08-aug-2005
12 c 11 3 11-dec-2008
i would like to get the output as csv like
empid, all_other_details
10 , {ename:a;deptid:10;mgrid:5; like this for all 19 columns }
except employee id all other columns should be wrapped into a string containing key:value pairs. Is there a way to join all the columns without mentioning each column as $_. ?
I have come up with this, I hope comments are self explanatory.
It should work with 2 or more columns.
Delimiters can be changed (on my computer, CSV delimiter is ; not , for example, and I know it can be different with other Cultures).
#declare delimiters
$CSVdelimiter = ";"
$detailsDelimiter = ","
#load file in array
$data = Get-Content "Book1.csv"
#isolate headers
$headers = $data[0].Split($CSVdelimiter)
#declare row counter
$rowCount = 0
#declare results array with headers
$results = #($headers[0] + "$CSVdelimiter`details")
#for each row except first
$data | Select-Object -Skip 1 | % {
#split on $csvDelimiter
$rowArray = $_.Split($CSVdelimiter)
#declare details array
$details = #()
#for each column except first
for($i = 1; $i -lt $rowArray.Count; $i++) {
#add to details array (header:value)
$details += $headers[$i] + ":" + $rowArray[$i]
}
#join details array with $detailsDelimiter to build new row
#append to first column value
#add to results array
$results += "$($rowArray[0])$CSVdelimiter{$($details -join $detailsDelimiter)}"
#increment row counter
$rowCount++
}
#output results to new csv file
$results | Out-File "Book2.csv"
Output looks like this :
empid;details
10;{ename:a,deptid:10,mgrid:5,hiredon:10-may-2010}
11;{ename:b,deptid:10,mgrid:5,hiredon:08-aug-2005}
12;{ename:c,deptid:11,mgrid:3,hiredon:11-dec-2008}
Try this:
$csv = Get-Content .\input_file.csv
$keys = $csv[0] -split '\s+'
$c = $keys.count - 1
$keys = ($keys[1..$c] | % {$i = -1}{$i += 1; "$($_):{$i}"}) -join '; '
$csv[1..($csv.count -1)] | % {
$a = $_ -split '\s+'
New-Object psobject -Property #{
empid = $a[0]
all_other_details = "{$($keys -f $a[1..$c])}"
}
} | Export-Csv output_file.csv -NoTypeInformation

Output Values from gc into Hash Table

Trying to make a hash table with 2 categories: Users and Passwords.
This is my code thus far but the issue is the output only displays the command and does not execute it.
for ($i=1; $i -le 10; $i++){
$caps = [char[]] "ABCDEFGHJKMNPQRSTUVWXY"
$lows = [char[]] "abcdefghjkmnpqrstuvwxy"
$nums = [char[]] "2346789"
$spl = [char[]] "!##$%^&*?+"
$first = $lows | Get-Random -count 1;
$second = $caps | Get-Random -count 1;
$third = $nums | Get-Random -count 1;
$forth = $lows | Get-Random -count 1;
$fifth = $spl | Get-Random -count 1;
$sixth = $caps | Get-Random -count 1;
$pwd = [string](#($first) + #($second) + #($third) + #($forth) + #($fifth) + #($sixth))
Out-File C:\Users\Administrator\Documents\L8_userpasswords.txt -InputObject $pwd -Append
}
$here = #'
$users=Get-Content C:\\Users\\Administrator\\Desktop\\L8_users.txt
$passwords=Get-Content C:\\Users\\Administrator\\Documents\\L8_userpasswords.txt
'#
convertfrom-stringdata -stringdata $here
This is the output I am getting:
PS C:\Users\Administrator> C:\Users\Administrator\Documents\l8.ps1
Name Value
---- -----
$users Get-Content C:\Users\Administrator\Desktop\Lab8_users.txt
$passwords Get-Content C:\Users\Administrator\Documents\L8_userpasswords.txt
I think you want this, which will turn the list of users and passwords into a HashTable, and then cast it to a PSCustomObject, which will have two properties: Users and Passwords.
$Data = [PSCustomObject]#{
Users = Get-Content -Path C:\Users\Administrator\Desktop\L8_users.txt;
Passwords = Get-Content -Path C:\Users\Administrator\Desktop\L8_userpasswords.txt;
}
$Data;
Or hey, you could probably just replace the entire script with a one liner:
GC C:\Users\Administrator\Desktop\L8_users.txt|%{[PSCustomObject]#{User=$_;Password=[System.Web.Security.Membership]::GeneratePassword(10,3)}}
Unless you are super attached to your password generation loop that is. [System.Web.Security.Membership]::GeneratePassword(X,Y) will generate complex passwords where X is the length and Y is the number of special characters (the rest will be a random mix of upper case letters, lower case letters, and numbers). So in my code (10,3) is a 10 character password with 3 non-alphanumeric characters.
You want it saved to a file? Pipe that to Export-CSV. Or assign it to a variable by prefixing it with something like $UserList = <code>.
Or if you really, really want a Hash Table you could make an empty one and then alter it just a little to add each pair to the table like this:
$UserList = #{}
GC C:\Users\Administrator\Desktop\L8_users.txt|%{$UserList.add($_,[System.Web.Security.Membership]::GeneratePassword(10,3))}
Assuming that L8_users.txt and L8_userpasswords.txt contain the same number of items, you could do something like this:
$users = Get-Content 'C:\Users\Administrator\Desktop\L8_users.txt'
$passwords = Get-Content 'C:\Users\Administrator\Documents\L8_userpasswords.txt'
$userpasswords = #{}
for ($i = 0; i -lt $users.Length; $i++) {
$userpasswords[$users[$i]] = $passwords[$i]
}

Get a variable by dynamic variable name

How does one access data imported from a CSV file by using dynamic note property names? That is, one doesn't know the colunm names beforehand. They do match a pattern and are extracted from the CSV file when the script runs.
As for an example, consider a CSV file:
"Header 1","Header A","Header 3","Header B"
0,0,0,0
1,2,3,4
5,6,7,8
I'd like to extract only columns that end with a letter. To do this, I read the header row and extract names with a regex like so,
$reader = new-object IO.StreamReader("C:\tmp\data.csv")
$line = $reader.ReadLine()
$headers = #()
$line.Split(",") | % {
$m = [regex]::match($_, '("Header [A-Z]")')
if($m.Success) { $headers += $m.value } }
This will get all the column names I care about:
"Header A"
"Header B"
Now, to access a CSV file I import it like so,
$csvData = import-csv "C:\tmp\data.csv"
Import-CSV will create a custom object that has properties as per the header row. One can access the fields by NoteProperty names like so,
$csvData | % { $_."Header A" } # Works fine
This obviously requires one to know the column name in advance. I'd like to use colunn names I extracted and stored into the $headers. How would I do that?
Some things I've tried so far
$csvData | % { $_.$headers[0] } # Error: Cannot index into a null array.
$csvData | % { $np = $headers[0]; $_.$np } # Doesn't print anything.
$csvData | % { $_.$($headers[0]) } # Doesn't print anything.
I could change the script like so it will write another a script that does know the column names. Is that my only solution?
I think you want this:
[string[]]$headers = $csvdata | gm -MemberType "noteproperty" |
?{ $_.Name -match "Header [a-zA-Z]$"} |
select -expand Name
$csvdata | select $headers
Choose the headers that match the condition (in this case, ones ending with characters) and then get the csv data for those headers.
the first thing ( and the only one... sorry) that came in my mind is:
$csvData | % { $_.$(( $csvData | gm | ? { $_.membertype -eq "noteproperty"} )[0].name) }
for get the first's column values and
$csvData | % { $_.$(( $csvData | gm | ? { $_.membertype -eq "noteproperty"} )[1].name) }
for second column and so on....
is this what you need?
you can use custom script to parse csv manually:
$content = Get-Content "C:\tmp\data.csv"
$header = $content | Select -first 1
$columns = $header.Split(",")
$indexes = #()
for($i; $i -lt $columns.Count;$i++)
{
# to verify whether string end with letter matches this regex: "[A-Za-z]$"
if ($column[$i] -match "[A-Za-z]$")
{
$indexes += $i
}
}
$outputFile = "C:\tmp\outdata.csv"
Remove-Item $outputFile -ea 0
foreach ($line in $content)
{
$output = ""
$rowcol = $line.Split(",")
[string]::Join(",", ($indexes | foreach { $rowcol[$_] })) | Add-Content $outputFile
}