I would like to use PowerShell to parse a log, and output to a CSV file.
What is the basic way to accomplish this?
Each log format involves it own logic when doing pattern matching, but I would propose to you something like:
(test.log is something like:
date - context - type - msg)
$file = get-content test.log
$pattern = '^(.*) - (.*) - (.*) - (.*)$'
$findings = $file | select-string -pattern $pattern
$lines = foreach($f in $findings) {
if ($f -match $pattern) {
$line = #{
'date' = $matches[1]
'context' = $matches[2]
'level' = $matches[3]
'message' = $matches[4]
}
new-object -typename psobject -property $line
}
}
$lines | export-csv -notypeinformation -delimiter ';' -path 'test.csv'
Get-Content | do something | Export-CSV
Related
I try to replace some texts inside a csv like following:
$csv = Import-Csv $csvFileName -Delimiter ';'
foreach ($line in $csv)
{
$properties = $line | Get-Member -MemberType Properties
for ($i = 0; $i -lt $properties.Count;$i++)
{
$column = $properties[$i]
$value = $line | Select -ExpandProperty $column.Name
# Convert numbers with , as separator to . separator
if ($value -match "^[+-]?(\d*\,)?\d+$")
{
$value = $value -replace ",", "."
# HOW TO update the CSV cell with the new value here???
# ???
}
}
}
$csv | Export-Csv $csvFileName -Delimiter ',' -NoTypeInformation -Encoding UTF8
As you can see, I miss the line where I want to update the csv line's cell value with the new value => can someone tell me how I can do that?
Assuming the regex you have in place will match the pattern you expect, you can simplify your code using 2 foreach loops, easier than a for. This method invokes the intrinsic PSObject member, available to all PowerShell objects.
$csv = Import-Csv $csvFileName -Delimiter ';'
foreach ($line in $csv) {
foreach($property in $line.PSObject.Properties) {
if ($property.Value -match "^[+-]?(\d*\,)?\d+$") {
# regex operator not needed in this case
$property.Value = $property.Value.Replace(",", ".")
}
}
}
$csv | Export-Csv ....
You can also do all process in pipeline (method above should be clearly faster, however this method is likely to be memory friendlier):
Import-Csv $csvFileName -Delimiter ';' | ForEach-Object {
foreach($property in $_.PSObject.Properties) {
if ($property.Value -match "^[+-]?(\d*\,)?\d+$") {
# regex operator not needed in this case
$property.Value = $property.Value.Replace(",", ".")
}
}
$_ # => output this object
} | Export-Csv myexport.csv -NoTypeInformation
I have to process some text and got some difficulties:
The text .\text.txt is formatted like that:
name,
surname,
address,
name.
surname,
address,
etc.
What I want to achieve is join the objects that ends with the "," like this:
name,surname,address
name,surname,address
etc
I was working on something like this:
$content= path to the text.txt
$result= path to the result file
Get-Content -Encoding UTF8 $content | ForEach-object {
if ( $_ -match "," ) {
....join the selected lines....
}
} |Set-Content -Encoding UTF8 $result
What I need to consider is also that lines which terminate with "," may have a next line empty which should be a CR in the $result
You can do this by splitting the blocks of data on the empty newlines first:
# read the content of the file as one single multiline string
$content = Get-Content -Path 'Path\To\The\file.txt' -Raw -Encoding UTF8
# split on two or more newlines and dispose of empty blocks
$content -split '(\r?\n){2,}' | Where-Object { $_ -match '\S' } | ForEach-Object {
# trim the text block, split on newline and remove the trailing commas (or dots)
# output these joined with a comma
($_.Trim() -split '\r?\n' ).TrimEnd(",.") -join ','
} | Set-Content -Path 'Path\To\The\NEW_file.txt' -Encoding UTF8
Output:
name,surname,address
name,surname,address
all your terms ends with a , so you could use regex:
$content= "C:\test.txt"
$result= "path to the result file"
$CR = "`r`n"
$lines = Get-Content -Encoding UTF8 $content -raw
$option = [System.Text.RegularExpressions.RegexOptions]::Singleline
$lines = [regex]::new(',(?:\r?\n){2,}', $option).Replace($lines, $CR + $CR)
$lines = [regex]::new(',\r?\n', $option).Replace($lines, ",")
$lines | Out-File -FilePath $result -Encoding utf8
result:
name,surname,address
name1,surname,address
name,surname,address
name,surname,address
Below piece of code will give the required result.
$content= "Your file path"
$resultPath = "result file path"
Get-Content $content | foreach {
$data = $_
if($data -eq "address,")
{
$NewData = $data -replace ',',''
$data = $NewData + "`r`n"
}
$out = $out + $data
}
$out | Out-File $resultPath
Hi all I have written the following code to export the text file content to HTML formatted output, but I am not getting the result as expected can some one help me
function TextToHtml {
$SourceFile = "C:\sample.txt"
$TargetFile = "C:\TestOutput.htm"
$TextData = Get-Content $SourceFile
foreach ($Line in $TextData) {
$LineData = $FileLine + $Line
}
$LineData | ConvertTo-HTML | Out-File $TargetFile
}
ConvertTo-Html turns object properties into values in a list or table. A string only has one property, Length - that's what you see the value of in your output.
Create a new object with $LineData as a property value and you'll get meaningful output
function TextToHtml
{
$SourceFile = "C:\sample.txt"
$TargetFile = "C:\TestOutput.htm"
$TextData = Get-Content $SourceFile
Foreach ($Line in $TextData) {
$LineData = $LineData + $Line
}
New-Object psobject -Property #{Text = $LineData} | ConvertTo-HTML -Property Text | Out-File $TargetFile
}
To make your function more reusable, turn the source and target file paths into parameters.
You also don't need to iterate over each string in $TextData to append them to each other, just use the -join operator:
function TextToHtml
{
param(
[string]$SourceFile = "C:\sample.txt",
[string]$TargetFile = "C:\TestOutput.htm"
)
$TextData = Get-Content $SourceFile
$LineData = $TextData -join ''
New-Object psobject -Property #{Text = $LineData} | ConvertTo-HTML | Out-File $TargetFile
}
If you want to avoid the * header for objects that only have one property, use the -Property parameter with ConvertTo-Html to explicitly select the Text property:
ConvertTo-Html -Property Text
If you want to show each line in it's own table row, skip the concatenation and pipe the strings directly to ConvertTo-Html instead:
function TextToHtml
{
param(
[string]$SourceFile = "C:\sample.txt",
[string]$TargetFile = "C:\TestOutput.htm"
)
Get-Content $SourceFile | ConvertTo-HTML -Property #{Label='Text';Expression={$_}} | Out-File $TargetFile
}
If we go just with the code pasted in the question above. It will give you the value of $FileLine (here it is null or blank) and text from last line in that file.
But if you want to have all lines as well use below code:
function TextToHtml
{
$SourceFile = "C:\sample.txt"
$TargetFile = "C:\TestOutput.htm"
$TextData = Get-Content $SourceFile
Foreach ($Line in $TextData) {
$LineData = $FileLine + $LineData + $Line
}
$LineData | ConvertTo-HTML | Out-File $TargetFile
}
Working on a code which will replace a set of characters from a text files in a folder. IS there a way where it can do it for all the files in the folder. I am using a Windows 7 OS and Powershell Version 3.Attaching the code which I have. The issue is it creates a new file when I run the code (New_NOV_1995.txt) but it doesn't change any character in the new file as mentioned in the code. Help very much Appreciated.
$lookupTable = #{
'¿' = '|'
'Ù' = '|'
'À' = '|'
'Ú' = '|'
'³' = '|'
'Ä' = '-'
}
$original_file = 'C:\FilePath\NOV_1995.txt'
$destination_file = 'C:\FilePath\NOV_1995_NEW.txt'
Get-Content -Path $original_file | ForEach-Object {
$line = $_
$lookupTable.GetEnumerator() | ForEach-Object {
if ($line -match $_.Key)
{
$line = $line -replace $_.Key, $_.Value
}
}
$line
} | Set-Content -Path $destination_file
In the following example, I'm assuming that H:\Replace_String is a directory. In your code above, you don't have a backslash so it would only select files in the root of H:.
$configFiles = Get-ChildItem -path H:\Replace_String\*.txt
foreach ($file in $configFiles)
{
(Get-Content $file) |
Foreach-Object { $_ -replace "Cat", "New_Cat" } |
Foreach-Object { $_ -replace "Dog", "New_Dog" } |
Set-Content $file
}
The (original) answer proposed by Tony Hinkle needs another loop. The reason for this is that Get-Content produces an array. Each line represents an element of the array.
$configFiles = Get-ChildItem -path 'H:\Replace_String\*.txt'
foreach ($file in $configFiles){
$output = #()
$content = Get-Content $file
foreach ($line in $content) {
$line = $content.Replace("Cat", "New_Cat")
$line = $content.Replace("Dog", "New_Dog")
$output += $line
}
$output | Set-Content -Path $file
}
Edit: I noticed that Tony Hinkle's answer was modified as I posted this. He's sending everything through a pipeline where I'm storing the array in a variable then looping through. The pipeline method is probably more memory efficient. The variable with second loop for each element of the array is more easily modified to do more than just the two replacments.
I have files which need to be modified according to mapping provided in CSV. I want to read each line of my txt file and depending if specified value exist I want to replace other strings in that line according to my CSV file (mapping). For that purpose I have used HashTable. Here is my ps script:
$file ="path\map.csv"
$mapping = Import-CSV $file -Encoding UTF8 -Delimiter ";"
$table = $mapping | Group-Object -AsHashTable -AsString -Property Name
$original_file = "path\input.txt"
$destination_file = "path\output.txt"
$content = Get-Content $original_file
foreach ($line in $content){
foreach ($e in $table.GetEnumerator()) {
if ($line -like "$($e.Name)") {
$line = $line -replace $e.Values.old_category, $e.Values.new_category
$line = $line -replace $e.Values.old_type, $e.Values.new_type
}
}
}
Set-Content -Path $destination_file -Value $content
My map.csv looks as follows:
Name;new_category;new_type;old_category;old_type
alfa;new_category1;new_type1;old_category1;old_type1
beta;new_category2;new_type2;old_category2;old_type2
gamma;new_category3;new_type3;old_category3;old_type3
And my input.txt content is:
bla bla "bla"
buuu buuu 123456 "test"
"gamma" "old_category3" "old_type3"
alfa
When I run this script it creates exactly the same output as initial file. Can someone tell me why it didn't change the line where "gamma" appears according to my mapping ?
Thanks in advance
Couple of things to change.
Firstly there is no need to change $mapping to a hash, Import-Csv already gives you an object array to work with.
Secondly, if you want to update the elements of $content, you need to use a for loop such that you can directly access modify them. Using a foreach creates a new variable in the pipeline and you were previously modifying it but then never writing it back to $content
Below should work:
$file ="map.csv"
$mapping = Import-CSV $file -Encoding UTF8 -Delimiter ";"
$original_file = "input.txt"
$destination_file = "output.txt"
$content = Get-Content $original_file
for($i=0; $i -lt $content.length; $i++) {
foreach($map in $mapping) {
if ($content[$i] -like "*$($map.Name)*") {
$content[$i] = $content[$i] -replace $map.old_category, $map.new_category
$content[$i] = $content[$i] -replace $map.old_type, $map.new_type
}
}
}
Set-Content -Path $destination_file -Value $content