Commas in Powershell hash table key - powershell

Take this .txt file:
11111-2222, My, file, is, here.xml
22222-1111, My, filename.xml
22222-2323, My filename 2.xml
22222-2323, Myfilename3.xml
This text file represents a map linking ID's with names of files on a filesystem directory (each row is an entry, seperated by the first comma following the ID number). I have a powershell script that at a high-level will import this .txt file as CSV and put it in a map to where I can match a filename with it's corresponding ID number, which I need to append to a PUT request to an endpoint.
My script is working great for lines 3 and 4 above in the .txt... except for the filenames that have commas. Since we are delimiting with commas, Powershell is cutting off those field names short, causing for incorrect field values.
Code snippet:
$mapFile = "location\of\mytextfilehere.txt"
$contentObjects = "location\of\filesIwantToPost"
$map = #{}
Import-Csv $mapFile -Header ID,Filename | ForEach-Object { $map[$_.Filename] = $_.ID }
foreach ($file in $contentObjects) {
$content = Get-Content $object.PSPath
$putURI = "http://myendpoint:3000/" + $map[$file.Name]
$request = Invoke-WebRequest -Uri $putURI -Method PUT -Body $content
}
The above breaks when trying to PUT the file "My, file, is, here.xml" and "My, filename.xml". The map building saves only "My, " as the key value, since we are comma delimited.
Is there a way to help me deal with these commas and save these fields completely and correctly? Perhaps by delimiting my .txt with pipes instead of commas? Or is there a different alternate approach?

Your file is not CSV. Wrap names with quotes or parse it manually like below:
#test data
new-item sample.txt -ItemType File -Value "11111-2222, My, file, is, here.xml
22222-1111, My, filename.xml
22222-2323, My filename 2.xml
22222-2323, Myfilename3.xml"
#parse
cat sample.txt | %{
$ID, $File = $_ -split ',',2;
[PSCustomObject]#{ ID=$ID; File=$File.Trim() }
}

Related

Powershell - adding a string to each row

I have a csv with over 1,000 rows.
In powershell how do I add a fixed string (e.g. "File1")to each row of the csv file.
For example:
Cat,dog,mouse should become file1,cat,dog,mouse.
Hopefully this is clear. Thanks for assistance.
Try like this:
Import-Csv "CSV FILE Path" |
Select-Object #{n='YOUR NEW COLUMN HEADER NAME';e={"file1"}},* |
Export-Csv "CSV FILE Path"
Looking at your latest comment, I think the fastest way here is to read the file line-by-line and output that same line with the wanted prefix attached.
Something like:
$prefix = 'File1'
$result = switch -File 'D:\Test\TheInputFile.csv' {
default { "$prefix,$_" }
}
$result | Set-Content -Path 'D:\Test\TheOutputFile.csv'
If the input file is a csv with headers, you'll need to supply a new column header for the inserted field:
$prefix = 'File1'
$header = 'Path' # or some other not already used column name
$firstLine = $true
$result = switch -File 'D:\Test\TheInputFile.csv' {
default {
if ($firstLine) { "$header,$_"; $firstLine = $false }
else { "$prefix,$_" }
}
}
$result | Set-Content -Path 'D:\Test\TheOutputFile.csv'
A CSV file is a text file. The difference is that in a csv, data is presented in a structured way. Ideally it starts with a header line, defining the field names for the rest of the lines which contain the actual data.
Each field is separated from the next by a specific character, usually a comma or semi-colon or tab.
To make sure a field can also include such a character, these fields must be enclosed in double-quotes.
See here

Powershell command required to strip ID/Passport No's from columns delimited text file

Have text File as per below;
New to Powershell so need guidance here.
INPUT TXT
membercode|id_number|passportnumber
BED_ODG0001|5007160081|
PF-000552516|7605430081|
PF-000704976|0385084|
PF-000678375||EP3800795
Need ONLY the ID_Number Column, but if there is no value move the Passportnumber value into the column.
REQUIRED OUTPUT TXT
5007160081
7605430081
0385084
EP3800795
I got a bit bored waiting on this, so here you go:
Have a look at the Import-Csv cmdlet.
Basically, your input txt file looks like a valid CSV file, including field headers, where the fields are separated using a pipe-symbol |.
Using Import-Csv in combination with parameter -Delimiter, it is really easy to load the data as an array of objects.
Every field in the file will be represented in these objects as properties that have names as defined in the file headers.
$inputFile = 'YOUR PATH AND FILENAME TO THE INPUT TXT FILE'
$outputFile = 'YOUR PATH AND FILENAME FOR THE OUTPUT TXT FILE'
# load the data from the CSV as an array of objects into a variable
$data = Import-Csv -Path $inputFile -Delimiter '|'
# loop through the elements of this array (the data rows in the file).
# inside the loop, the automatic variable '$_' is an object representing
# one element in the array (~~> row in the file)
$data | ForEach-Object {
# output the field 'id_number' or if that is not there, use the field 'passportnumber'
# and write it to the $outputFile file
if ($_.id_number) { $_.id_number } else { $_.passportnumber }
} | Set-Content -Path $outputFile
Result:
5007160081
7605430081
0385084
EP3800795

Append text to certain values in text file with PowerShell

I have a CSV text file separated with ; and it's in the format as:
USER_EMPLOYEE_ID;SYSTEM1;USERNAME1
The first column is an identity and the following pairs of columns are user's account on different active directories. I have placed garbage data but the idea is there.
ay7suve0001;ADDPWN;ay7suve0001
AAXMR3E0001;ADDPWN;AAXMR3E0001
ABABIL;ADDPWN;ABABIL
ABDF17;ADDPWN;ABDF17;
ABKMPPE0001;ADDPWN;ABKMPPE0001
ABL1FL;ADDPWN;ABL1FL
AB6JG8E0004;ADDPWN;AB6JG8E0004;
ACB4YB;ADDPWN;ACB4YB
ACK7J9;ADDPWN;ACK7J9
ACLZFS;ADDPWN;ACLZFS;
ACQXZ3;ADDPWN;ACQXZ3
Now there is a requirement that I have to append a fixed string like #ADDPWN.com to all the USERNAME1 values. Some records are having a ; and some don't.
Is there a quick way to append the #ADDPWN.com to each line taking care of:
any ;
any already #ADDPWN.com
From PowerShell?
Import-Csv is your friend. The following should get you on the right track.
Import-Csv "import.csv" -Delimiter ';' |
foreach {
if ($_.username1 -notlike '*#ADDPWN.com') { $_.username1 += '#ADDPWN.com' }
$_
} |
Export-Csv "export.csv" -Delimiter ';'
This assumes the first line of your csv file is your header line. If it's not, you can pass -Header 'USER_EMPLOYEE_ID','SYSTEM1','USERNAME1' as another parameter to Import-Csv.
Export-Csv adds some extra stuff like quotes around parameters, so you may need to play with the output format if you don't want that.
For another explanation how this works, check out Changes last name, first name to first name, last name in last column CSV powershell
This was a solution that worked for me.........
#opens list of file names
$file2 ="F:\OneDrive_Biz\PowerApps\SecurityCameraVideoApp\file_list_names.csv"
$x = Get-Content $file2
#appends URl to beginning of file name list
for($i=0; $i -lt $x.Count; $i++){
$x[$i] = "https://analytics-my.sharepoint.com/personal/gpowell_analytics_onmicrosoft_com/Documents/PowerApps/SecurityCameraVideoApp/Video_Files/" + $x[$i]
}
$x
#remove all files in target directory prior to saving new list
get-childitem -path C:\_TEMP\file_list_names.csv | remove-item
Add-Content -Path C:\_TEMP\file_list_names_url.csv -Value $x

How to read multiple data sets from one .csv file in powershell

I have a temp recorder that (daily) reads multiple sensors and saves the data to a single .csv with a whole bunch of header information before each set of date/time and temperature. the file looks something like this:
"readerinfo","onlylistedonce"
"downloadinfo",YYYY/MM/DD 00:00:00
"timezone",-8
"headerstuff","headersuff"
"sensor1","sensorstuff"
"serial#","0000001"
"about15lines","ofthisstuff"
"header1","header2"
datetime,temp
datetime,temp
datetime,temp
"sensor2","sensorstuff"
"serial#","0000002"
"about15lines","ofthisstuff"
"header1","header2"
datetime,temp
datetime,temp
datetime,temp
"downloadcomplete"
My aim is to pull out the date/time and temp data for each sensor and save it as a new file so that I can run some basic stats(hi/lo/avg temp)on it. (It would be beautiful if I could somehow identify which sensor the data came from based on a serial number listed in the header info, but that's less important than separating out the data into sets) The lengths of the date/time lists change from sensor to sensor based on how long they've been recording and the number of sensors changes daily also. Even if I could just split the sensor data, header info and all, into however many files there are sensors, that would be a good start.
This isn't exactly a CSV file in the traditional sense. I imagine you already know this, given your description of the file contents.
If the lines with datetime,temp truly do not have any double quotes in them, per your example data, then the following script should work. This script is self-containing, since it declares the example data in-line.
IMPORTANT: You will need to modify the line containing the declaration of the $SensorList variable. You will have to populate this variable with the names of the sensors, or you can parameterize the script to accept an array of sensor names.
UPDATE: I changed the script to be parameterized.
Results
The results of the script are as follows:
sensor1.csv (with corresponding data)
sensor2.csv (with corresponding data)
Some green text will be written to the PowerShell host, indicating which sensor is currently detected
Script
The contents of the script should appear as follows. Save the script file to a folder, such as c:\test\test.ps1, and then execute it.
# Declare text as a PowerShell here-string
$Text = #"
"readerinfo","onlylistedonce"
"downloadinfo",YYYY/MM/DD 00:00:00
"timezone",-8
"headerstuff","headersuff"
"sensor1","sensorstuff"
"serial#","0000001"
"about15lines","ofthisstuff"
"header1","header2"
datetime,tempfromsensor1
datetime,tempfromsensor1
datetime,tempfromsensor1
"sensor2","sensorstuff"
"serial#","0000002"
"about15lines","ofthisstuff"
"header1","header2"
datetime,tempfromsensor2
datetime,tempfromsensor2
datetime,tempfromsensor2
"downloadcomplete"
"#.Split("`n");
# Declare the list of sensor names
$SensorList = #('sensor1', 'sensor2');
$CurrentSensor = $null;
# WARNING: Clean up all CSV files in the same directory as the script
Remove-Item -Path $PSScriptRoot\*.csv;
# Iterate over each line in the text file
foreach ($Line in $Text) {
#region Line matches double quote
if ($Line -match '"') {
# Parse the property/value pairs (where double quotes are present)
if ($Line -match '"(.*?)",("(?<value>.*)"|(?<value>.*))') {
$Entry = [PSCustomObject]#{
Property = $matches[1];
Value = $matches['value'];
};
if ($matches[1] -in $SensorList) {
$CurrentSensor = $matches[1];
Write-Host -ForegroundColor Green -Object ('Current sensor is: {0}' -f $CurrentSensor);
}
}
}
#endregion Line matches double quote
#region Line does not match double quote
else {
# Parse the datetime/temp pairs
if ($Line -match '(.*?),(.*)') {
$Entry = [PSCustomObject]#{
DateTime = $matches[1];
Temp = $matches[2];
};
# Write the sensor's datetime/temp to its file
Add-Content -Path ('{0}\{1}.csv' -f $PSScriptRoot, $CurrentSensor) -Value $Line;
}
}
#endregion Line does not match double quote
}
Using the data sample you provided, the output of this script would as follows:
C:\sensoroutput_20140204.csv
sensor1,datetime,temp
sensor1,datetime,temp
sensor1,datetime,temp
sensor2,datetime,temp
sensor2,datetime,temp
sensor2,datetime,temp
I believe this is what you are looking for. The assumption here is the new line characters. The get-content line is reading the data and breaking it into "sets" by using 2 new line characters as the delimiter to split on. I chose to use the environment's (Windows) new line character. Your source file may have different new line characters. You can use Notepad++ to see which characters they are e.g. \r\n, \n, etc.
$newline = [Environment]::NewLine
$srcfile = "C:\sensordata.log"
$dstpath = 'C:\sensoroutput_{0}.csv' -f (get-date -f 'yyyyMMdd')
# Reads file as a single string with out-string
# then splits with a delimiter of two new line chars
$datasets = get-content $srcfile -delimiter ($newline * 2)
foreach ($ds in $datasets) {
$lines = ($ds -split $newline) # Split dataset into lines
$setname = $lines[0] -replace '\"(\w+).*', '$1' # Get the set or sensor name
$lines | % {
if ($_ -and $_ -notmatch '"') { # No empty lines and no lines with quotes
$data = ($setname, ',', $_ -join '') # Concats set name, datetime, and temp
Out-File -filepath $dstpath -inputObject $data -encoding 'ascii' -append
}
}
}

Splitting one file into many with Powershell

I have a file called one_to_many.txt. In the file is the data:
a,aaa
b,bbb
c,ccc
I want to use powershell to create 3 files a.txt, b.txt and c.txt
File a.txt to contain data "aaa", b to contain "bbb ...
I'm new to Powershell and can't work out how to do it. Thanks.
One possible solution:
Get-Content one_to_many.txt | Foreach-Object {
$fileName, $content = $_ -split ','
$content | Set-Content "$fileName.txt"
}
Get-Content reads the file and returns lines, one by one. Each line is piped to Foreach-Object where it can be accessed as $_.
You need to split the line by comma, so invoke operator -split which returns in this case array of 2 items. The assignment $filename, $content = .. causes that content of first item in array (from -split) is assigned to $filename and the rest to `$content.
Then simply store that content in the file via Set-Content