create file index manually using powershell, tab delimited - powershell

Sorry in advance for the probably trivial question, I'm a powershell noob, please bear with me and give me advice on how to get better.
I want to achieve a file index index.txt that contains the list of all files in current dir and subdirs in this format:
./dir1/file1.txt 07.05.2020 16:16 1959281
where
dirs listed are relative (i.e. this will be run remotely and to save space, the relative path is good enough)
the delimiter is a tab \t
the date format is day.month.fullyear hours:minutes:seconds, last written (this is the case for me, but I'm guessing this would be different on system setting and should be enforced)
(the last number is the size in bytes)
I almost get there using this command in powershell (maybe that's useful to someone else as well):
get-childitem . -recurse | select fullname,LastWriteTime,Length | Out-File index.txt
with this result
FullName LastWriteTime Length
-------- ------------- ------
C:\Users\user1\Downloads\test\asdf.txt 07.05.2020 16:19:29 1490
C:\Users\user1\Downloads\test\dirtree.txt 07.05.2020 16:08:44 0
C:\Users\user1\Downloads\test\index.txt 07.05.2020 16:29:01 0
C:\Users\user1\Downloads\test\test.txt 07.05.2020 16:01:23 814
C:\Users\user1\Downloads\test\text2.txt 07.05.2020 15:55:45 1346
So the questions that remain are: How to...
get rid of the headers?
enforce this date format?
tab delimit everything?
get control of what newline character is used (\n or \r or both)?

Another approach could be this:
$StartDirectory = Get-Location
Get-ChildItem -Path $StartDirectory -recurse |
Select-Object -Property #{Name='RelPath';Expression={$_.FullName.toString() -replace [REGEX]::Escape($StartDirectory.ToString()),'.'}},
#{Name='LastWriteTime';Expression={$_.LastWriteTime.toString('dd.MM.yyyy HH:mm:ss')}},
Length |
Export-Csv -Path Result.csv -NoTypeInformation -Delimiter "`t"
I recommend to use proper CSV files if you have structured data like this. The resulting CSV file will be saved in the current working directory.

If the path you are running this from is NOT the current scrip path, do:
$path = 'D:\Downloads' # 'X:\SomeFolder\SomeWhere'
Set-Location $path
first.
Next, this ought to do it:
Get-ChildItem . -Recurse -File | ForEach-Object {
"{0}`t{1:dd.MM.yyyy HH:mm}`t{2}" -f ($_ | Resolve-Path -Relative), $_.LastWriteTime, $_.Length
} | Out-File 'index.txt'
On Windows the newline will be \r\n (CRLF)
If you want control over that, this should do:
$newline = "`n" # for example
# capture the lines as string array in variable $lines
$lines = Get-ChildItem . -Recurse -File | ForEach-Object {
"{0}`t{1:dd.MM.yyyy HH:mm}`t{2}" -f ($_ | Resolve-Path -Relative), $_.LastWriteTime, $_.Length
}
# join the array with the chosen newline and save to file
$lines -join $newline | Out-File 'index.txt' -NoNewline
Because your requirement is to NOT have column headers in the output file, I'm using Out-File here instead of Export-Csv

Related

Multiple Select Strings in a for loop to separate files

I wrote this script to search a lot of text files (~100,000) for 4 different search criteria and export to 4 separate files, I thought it would be more efficient to perform all 4 searches on each file as it is loaded vs doing 4 full searches like the first iteration below does. I may be missing some other major inefficiencies as I am pretty new to powershell.
I have this script re written from the first version to the second, but can't figure out how to get the path and data to display together like the first version did. I am struggling to reference the object within the loop, and have pieced this second version together, which is working, but not giving me the path to the file which is necessary.
It seems like I am just missing one or two little things to get me going in the right direction. Thanks in advance for your help
1st version:
Get-ChildItem -Filter *.txt -Path "\\file\to\search" -Recurse | Select-String -Pattern "abc123" -Context 0,3 | Out-File -FilePath "\\c:\out.txt"
Get-ChildItem -Filter *.txt -Path "\\file\to\search2" -Recurse | Select-String -Pattern "abc124" -Context 0,3 | Out-File -FilePath "\\c:\out2.txt"
Get-ChildItem -Filter *.txt -Path "\\file\to\search3" -Recurse | Select-String -Pattern "abc125" -Context 0,3 | Out-File -FilePath "\\c:\out3.txt"
Get-ChildItem -Filter *.txt -Path "\\file\to\search4" -Recurse | Select-String -Pattern "abc126" -Context 0,3 | Out-File -FilePath "\\c:\out4.txt"
Output:
\\file\that\was\found\example.txt:84: abc123
\\file\that\was\found\example.txt:90: abc123
\\file\that\was\found\example.txt:91: abc123
2nd version:
##$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ Configuration $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
############################################ Global Parameters #############################################
$SearchPath="\\file\to\search"
$ProgressFile=""\\progress\file\ResultsCount.txt"
$records = 105325
##----------------------------------------- End Global Parameters -----------------------------------------
########################################### Search Parameters ##############################################
##Search Pattern 1
$Pattern1="abc123"
$SaveFile1="\\c:\out.txt"
##Search Pattern 2
$Pattern2="abc124"
$SaveFile2="\\c:\out2.txt"
##Search Pattern 3
$Pattern3= "abc125"
$SaveFile3= "\\c:\out3.txt"
##Search Pattern 4
$Pattern4= "abc126"
$SaveFile4="\\c:\out4.txt"
##Search Pattern 5
$Pattern5= ""
$SaveFile5=""
##----------------------------------------- End Search Parameters ------------------------------------------
##$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ End of Config $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
############################### SCRIPT #####################################################################
## NOTES
## ------
##$files=Get-ChildItem -Filter *.txt -Path $SearchPath -Recurse ## Set all files to variable #### Long running, needs to be a better way #######
##$records=$files.count ## Set record #
Get-ChildItem -Filter *.txt -Path $SearchPath -Recurse | Foreach-Object { ## loop through search folder
$i=$i+1 ## increment record
##
Get-Content $_.FullName | Select-String -Pattern $Pattern1 -Context 0,3 | Out-File -FilePath $SaveFile1 ## pattern1 search
Get-Content $_.FullName | Select-String -Pattern $Pattern2 | Out-File -FilePath $SaveFile2 ## pattern2 search
Get-Content $_.FullName | Select-String -Pattern $Pattern3 -Context 0,1 | Out-File -FilePath $SaveFile3 ## pattern3 search
Get-Content $_.FullName | Select-String -Pattern $Pattern4 -Context 0,1 | Out-File -FilePath $SaveFile4 ## pattern4 search
##Get-Content $_.FullName | Select-String -Pattern $Pattern5 -Context 0,1 | Out-File -FilePath $SaveFile5 ## pattern5 search (Comment out unneeded search lines like this one)
$progress ="Record $($i) of $($records)" ## set progress
Write-Host "Record $($i) of $($records)" ## Writes progress to window
$progress | Out-File -FilePath $ProgressFile ## progress file
} ##
############################################################################################################
Output:
abc123
abc123
abc123
Edit: Also I am trying to figure out a good way to not have to hard code in the number of records for a decent progress readout, I commented out the way I thought would work (1st & 2nd line of the script), but there needs to be a more efficient way than rerunning the same search twice, one for a count and one for the for loop.
I would be very interested in any runtime efficiency information you could provide.
[edit - thanks to mklement0 for pointing out the errors about speed and the -SimpleMatch switch. [grin]]
the Select-String cmdlet will accept a -Path parameter ... and it is FAR [i was thinking of Get-Content, not Get-ChidItem] faster than using Get-ChildItem to feed the files to S-S. [grin]
also, the -Pattern parameter accepts a regex OR pattern like Thing|OtherThing|YetAnotherThing - and it accepts simple string patterns if you use the -SimpleMatch switch parameter.
what the code does ...
defines the source dir
defines the file spec
joins those two into a wildcard file path
builds an array of string patterns to use
calls Select-String with a path and an array of strings to search for
uses Group-Object and a calculated property to group the matches by the last part of .Line property from the S-S call
saves that to a $Var
shows that on screen
at that point, you can use the .Name property of each GroupInfo to select the items to send out to each file AND to build your file names.
the code ...
$SourceDir = 'D:\Temp\zzz - Copy'
$FileSpec = '*.log'
$SD_FileSpec = Join-Path -Path $SourceDir -ChildPath $FileSpec
$TargetPatternList = #(
'Accordion Cajun Zydeco'
'better-not-be-there'
'Piano Rockabilly Rowdy'
)
$GO_Results = Select-String -Path $SD_FileSpec -SimpleMatch $TargetPatternList |
Group-Object -Property {$_.Line.Split(':')[-1]}
$GO_Results
output ...
Count Name Group
----- ---- -----
6 Accordion Cajun Zydeco {D:\Temp\zzz - Copy\Grouping-List_08-02.log:11:Accordion Cajun Zydeco, D:\Temp\zzz - Copy\Grouping-List_08-09.log:11:Accordion Cajun Zy...
6 Bawdy Dupe Piano Rocka... {D:\Temp\zzz - Copy\Grouping-List_08-02.log:108:Bawdy Dupe Piano Rockabilly Rowdy, D:\Temp\zzz - Copy\Grouping-List_08-09.log:108:Bawdy...
6 Bawdy Piano Rockabilly... {D:\Temp\zzz - Copy\Grouping-List_08-02.log:138:Bawdy Piano Rockabilly Rowdy, D:\Temp\zzz - Copy\Grouping-List_08-09.log:138:Bawdy Pian...
6 Dupe Piano Rockabilly ... {D:\Temp\zzz - Copy\Grouping-List_08-02.log:948:Dupe Piano Rockabilly Rowdy, D:\Temp\zzz - Copy\Grouping-List_08-09.log:948:Dupe Piano ...
6 Instrumental Piano Roc... {D:\Temp\zzz - Copy\Grouping-List_08-02.log:1563:Instrumental Piano Rockabilly Rowdy, D:\Temp\zzz - Copy\Grouping-List_08-09.log:1563:I...
6 Piano Rockabilly Rowdy {D:\Temp\zzz - Copy\Grouping-List_08-02.log:1781:Piano Rockabilly Rowdy, D:\Temp\zzz - Copy\Grouping-List_08-09.log:1781:Piano Rockabil...
note that the .Group contains an array of lines from the matches sent out by the S-S call. you can send that to your output file.
Here is my take at solving this problem, very similar to Lee_Dailey's nice answer but with a foreach loop. I would recommend investing some time into researching the multi-threading options available on PowerShell in case you need to increase the performance of the script, you can look specifically at the ThreadJob module by Microsoft which is really easy to use or if you can't install modules due to some work policy, you can use Runspace.
It is worth adding that you can use the -List switch on Select-String, this way the performance of the script would be increased even more:
-List
Only the first instance of matching text is returned from each input file. This is the most efficient way to retrieve a list of files that have contents matching the regular expression.
$map = #{
abc123 = 'C:\out_abc123.txt'
abc124 = 'C:\out_abc124.txt'
abc125 = 'C:\out_abc125.txt'
}
$pattern = $map.Keys -join '|'
$match = foreach($file in Get-ChildItem *.txt)
{
Select-String -LiteralPath $file.FullName -Pattern $pattern
}
$match | Group-Object { $_.Matches.Value } | ForEach-Object {
$_.Group | Select-Object Path, LineNumber, Line | Out-File $map[$_.Name]
}
To compliment the answers #Santiago Squarzon and Lee_Dailey, I think you were actually on the good way yourself knowing that the Group-Object cmdlet is pretty expensive especially in memory usage as it chokes the PowerShell pipeline causing all the search results to be piled up in memory.
Besides, the Select-String cmdlet supports multiple (-SimpleMatch) patterns, where concatenating the search patters with an | (-join '|') will force you to use an (escaped) regular expression.
To continue on your approach:
(note that in the example, I am using my own settings to search through my script files)
$ProgressFile = '.\ResultsCount.txt'
$SearchRoot = '..\'
$Filter = '*.ps1'
$Searches = #{
'Null' = '.\Null.txt'
'Test' = '.\Test.txt'
'Object' = '.\Object.txt'
}
$Files = Get-ChildItem -Filter $Filter -Path $SearchRoot -Recurse
$Total = $Files.count
$Searches.Values |ForEach-Object { Set-Content -LiteralPath $_ -Value '' }
$i = 0
ForEach ($File in $Files) {
Get-Content -LiteralPath $File.FullName |
Select-String #($Searches.Keys) -AllMatches |ForEach-Object {
$Value = '{0}:{1}:{2}' -f $File.FullName, $_.LineNumber, $_
Add-Content -LiteralPath $Searches[$_.Pattern] -Value $Value
}
'Record {0} of {1}' -f ++$i, $Total |Tee-Object -Append .\ProgressFile.txt
}
Explanations
$Searches = #{ ...
Maps the search patters with the files, you might also use a PSObject list to specify each search (where you could add columns with e.g. context start/end values, etc.)
$Searches.Values |ForEach-Object { Set-Content -LiteralPath $_ -Value '' }
Empties the result files (knowing that they are not part of the main stream you can't use Add-Content)
$i = 0
Unfortunately there is no automatic index that initializes with a foreach loop (yet, see: #13772 Automatic variable for the pipeline index)
Get-Content -LiteralPath $File.FullName
Load the content once into memory
Note1: this is a string array.
Note2: the $Content will be reused each iteration and therefore overwrites the previous one and unloads it from memory
Select-String #($Searches.Keys) -AllMatches |ForEach-Object {
Searches the string array using your (multiple) defined patterns. (you might consider to use the -SimpleMatch parameter if your search strings contain special characters.)
Note: Unfortunately you need to embedded the $Searches.Keys in a array subexpression operator #( ), for details see .Net issue: #56835 Make OrderedDictionaryKeyValueCollection implement IList
$Value = '{0}:{1}:{2}' -f $File.FullName, $_.LineNumber, $_
Build an result output string.
Note: the result of the Select-String does have a (hidden) LineNumber and (matched) Pattern property.
Add-Content -LiteralPath $Searches[$_.Pattern] -Value $Value
Add the result string to the specific mapped output file.
'Record {0} of {1}' -f $i++, $Total |Tee-Object -Append .\ProgressFile.txt
Tee-Object will write the progress to the standard output (display) and also to the specific file.

Remove entries of one text file present in another

I have 2 text file like below
exclude.txt
10.1.1.3
10.1.1.4
10.1.1.5
10.1.1.6
free.txt
10.1.1.3
10.1.1.4
10.1.1.5
10.1.1.6
10.1.1.7
10.1.1.8
10.1.1.9
10.1.1.10
I want to write exclude the entries of exclude.txt from free.txt and write to another file
10.1.1.7
10.1.1.8
10.1.1.9
10.1.1.10
I tried :
compare-object (get-content $freeips) (get-content $excludeip) -PassThru | format-list | Out-File $finalips
Here in the final output I am always getting the first IP of the exclude.txt
10.1.1.7
10.1.1.8
10.1.1.9
10.1.1.10
10.1.1.3
and another way I tried
$exclude = Get-Content "C:\exclude.txt"
foreach($ip in $exclude)
{
get-content "C:\free.txt" | select-string -pattern $ip -notmatch | Out-File "C:\diff.txt"
}
But in this case also I am getting the entries of exclude.txt in the final output.
Please let me know where I am doing wrong here
The Select-String solution is probably faster. Besides it doesn't require the iteration through the IP addresses as the -Pattern parameters accepts a string array (String[]). The point is thou that by default the pattern(s) repressent a regular expression where a dot (.) is a place holder for any character. To search for a literal pattern you should use the -SimpleMatch switch:
$exclude = Get-Content .\exclude.txt
get-content .\free.txt |Select-String -pattern $exclude -NotMatch -SimpleMatch
Note: The space in top of the displayed exclude.txt file suggests that there might be an empty line in top of the file (which regex matches any string). To get rid of any empty lines, use:
$exclude = Get-Content .\exclude.txt |Where-Object { $_ }
When comparing, $excludeip should be the referenceObject and $freeips comes after, like this:
compare-object (get-content $excludeip) (get-content $freeips) -PassThru | Out-File $finalips

Copy altered CSV Data to new CSV

The whole point of this issue is going to be: How to copy data from one CSV to another without knowing/listing the headers of the original CSV.
The cmdlet I'm building is meant to convert a report from CSV to a spreadsheet eventually. And if I write the column headers to the code, each time somebody changes the report, the code will break and it would have to be updated.
The steps I would take right now:
# Import the Source CSV. Gonna pull data from this later.
$SourceCSV = Import-Csv -Path $reportSourceCSV -Delimiter ";"
# Remove NULL characters, white spaces and change comma separator to semicolon.
(Get-Content -Path $reportSourceCSV | Where-Object {-not [string]::IsNullOrWhiteSpace($PSItem)}).Replace('","',";") | Out-File -FilePath $TMP1
# Import the modified new temp CSV.
$Input = Import-Csv -Path $TMP1 -Delimiter ";"
# Take existing CSV file headers and append some new ones. Rename a long column name.
((($GetHeaders = foreach ($Header in $SourceCSV[0].PSObject.Properties.Name) {
"`"$Header`""
}) + '"column4"','"column5"','"column6"') -join ";").Replace("VerylongOldColumnName","ShortName") | Out-File -FilePath $TMP2
foreach ($Item in $Input) {
"`"$($Item.column1)`";`"$($Item.'column2')`";`"$($Item.column3)`"" | Out-File -FilePath $TMP2 -Append
}
$exportToXLSX = Import-Csv -Path $TMP2 -Delimiter ";" | Export-Excel -Path $Target -WorkSheetname "reportname" -TableName "tablename" -TableStyle Medium2 -FreezeTopRow -AutoSize -PassThru
$exportToXLSX.Save()
$exportToXLSX.Dispose()
Remove-Item -Path $TMP1, $TMP2
This works! But I don't want to create infinite amount of different reports and just as many different logic blocks to process all these reports.
So this is as far as I was able to get trying a more dynamic way of processing the report CSVs:
(Get-Content -Path $reportSourceCSV | Where-Object {-not [string]::IsNullOrWhiteSpace($PSItem)}).Replace('","',";") | Out-File -FilePath $TMP1
$import = Import-Csv -Path $TMP1 -Delimiter ";"
$headers = ($import[0].PSObject.Properties.Name).Replace("VerylongOldColumnName","ShortName")
$headers | Out-File -FilePath "C:\TEMP\test.csv"
foreach ($item in $import) {
for ($h = 0; $h -le ($headers).Count; $h++) {
$($item.$($headers[$h]))
}
}
Now, this works... kind of. If I run the script like this, it shows me the output I want, but I was NOT able to export this to CSV.
I added Export-Csv to this line: $($item.$($headers[$h])) so this particular line would look like this:
$($item.$($headers[$h])) | Export-Csv -Path $Output -Delimiter ";" -Append -NoTypeInformation
And this is the error I get:
Export-Csv : Cannot append CSV content to the following file: C:\TEMP\test.csv.
The appended object does not have a property that corresponds to the following
column: column1. To continue with mismatched properties, add the -Force parameter,
and then retry the command.
At line:11 char:36
+ ... ers[$h])) | Export-Csv -Path $Output -Delimiter ";" -Append -NoTypeIn ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (column1:String) [Export-Csv], InvalidOperationException
+ FullyQualifiedErrorId : CannotAppendCsvWithMismatchedPropertyNames,Microsoft.PowerShell.Commands.ExportCsvCommand
If I add -Force parameter, the output will be the headers and a bunch of empty lines.
As little as I understand, is that the output is for some reason a string? To my knowledge everything should be an object in PS, unless converted to string (Write-Host cmdlet being an exception). And I don't really know how to force the output back to being objects.
Edit: Added sample source CSV
"Plugin","Plugin Name","Family","Severity","IP Address","Protocol","Port","Exploit?","Repository","DNS Name","NetBIOS Name","Plugin Text","Synopsis","Description","Solution","See Also","Vulnerability Priority Rating","CVSS V3 Base Score","CVSS V3 Temporal Score","CVSS V3 Vector","CPE","CVE","Cross References","First Discovered","Last Observed","Vuln Publication Date","Patch Publication Date","Exploit Ease","Exploit Frameworks"
"65057","Insecure Windows Service Permissions","Windows","High","127.0.0.1","TCP","445","No","Individual Scan","computer.domain.tld","NetBIOS Name","Plugin Output:
Path : c:\program files (x86)\application\folder\service.exe
Used by services : application
File write allowed for groups : Users, Authenticated Users
Full control of directory allowed for groups : Users, Authenticated Users","At least one improperly configured Windows service may have a privilege escalation vulnerability.","At least one Windows service executable with insecure permissions was detected on the remote host. Services configured to use an executable with weak permissions are vulnerable to privilege escalation attacks.
An unprivileged user could modify or overwrite the executable with arbitrary code, which would be executed the next time the service is started. Depending on the user that the service runs as, this could result in privilege escalation.
This plugin checks if any of the following groups have permissions to modify executable files that are started by Windows services :
- Everyone
- Users
- Domain Users
- Authenticated Users","Ensure the groups listed above do not have permissions to modify or write service executables. Additionally, ensure these groups do not have Full Control permission to any directories that contain service executables.","http://www.nessus.org/u?e4e766b2","","8.4","","AV:L/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H","cpe:/o:microsoft:windows","","","Jul 11, 2029 06:48:20 CEST","Jul 11, 2029 06:48:20 CEST","N/A","N/A","",""
Edit: I think I found another way how to accomplish this and looking at it, it looks I tried to overdo it quite a bit.
# Doing cleanup, changing delimiters, renaming that one known column. All in one line.
$importCSV = 'C:\TEMP\sourceReport.csv'
(Get-Content -Path $importCSV | Where-Object {-not [string]::IsNullOrWhiteSpace($PSItem)}).Replace('","','";"').Replace"VerylongOldColumnName","ShortName") | Out-File -FilePath C:\TEMP\tmp1.csv
# Adding additional columns and exporting it all to result CSV.
Import-Csv -Path C:\TEMP\tmp1.csv -Delimiter ";" | Select-Object *, "Column1", "Column2" | Export-Csv -Path C:\TEMP\result.csv -NoTypeInformation -Delimiter ";"
You should not simply replace , with ; because the fields actually contain commas as in ..Additionally, ensure these groups .. By replacing just like that, the field will get separated from the rest of its content and you'll end up with a mis-aligned csv.
The below approach will do what you want, leaving the structure of the csv file intact:
$importCSV = 'C:\TEMP\sourceReport.csv'
$exportCSV = 'C:\TEMP\result.csv'
$columnsToAdd = "Column1", "Column2"
# read the file as string array, not including empty lines
$content = Get-Content -Path $importCSV | Where-Object { $_ -match '\S' }
# replace the column header in the top line only
$content[0] = $content[0].Replace("VerylongOldColumnName", "ShortName")
# join the string array with newlines and convert that to an object with ConvertFrom-Csv
# add the columns to the object and export it using the semi-colon as delimiter
($content -join [Environment]::NewLine) | ConvertFrom-Csv |
Select-Object *, $columnsToAdd |
Export-Csv -Path $exportCSV -NoTypeInformation -Delimiter ";"

I use -NoTypeInformation so why do I get header back when using Out-File?

I filtered by date this file data1.csv
2017.11.1,09:55,1.1,1.2,1.3,1.4,1
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
I don't get a header with -NoTypeInformation:
$CutOff = (Get-Date).AddDays(-2)
$filePath = "data1.csv"
$Data = Import-Csv $filePath -Header Date,Time,A,B,C,D,E
$Data2 = $Data | Where-Object {$_.Date -as [datetime] -gt $Cutoff} | convertto-csv -NoTypeInformation -Delimiter "," | % {$_ -replace '"',''}
But when rewriting with Out-File
$Data2 | Out-File "data2.csv" -Encoding utf8 -Force
I get header back as data2.csv contains:
Date,Time,A,B,C,D,E
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
Why do I have Date,Time,A,B,C,D,E ?
-NoTypeInformation is not about the header but the data type of the rows in the file. Remove it to see what shows up. From Microsoft
Omits the type information header from the output. By default, the string in the output contains #TYPE followed by the fully-qualified name of the object type.
Emphasis mine.
CSVs need headers. That is why it is making one. If you don't want to see the header in the output use Select-Object -Skip 1 to remove it.
$Data |
Where-Object {$_.Date -as [datetime] -gt $Cutoff} |
ConvertTo-CSV -NoTypeInformation -Delimiter "," |
Select-Object -Skip 1 |
% {$_ -replace '"'}
I would not pipe Out-File to itself. You could pipe to Set-Content here just as well.
I am guessing this whole process is to keep the source file in the same state just with some lines filtered out based on date. You could skip most of this just by parsing the date out in each line.
$threshold = (Get-Date).AddDays(-2)
$filePath = "c:\temp\bagel.txt"
(Get-Content $filePath) | Where-Object{
$date,$null=$_.Split(",",2)
[datetime]$date -gt $threshold
} | Set-Content $filePath
Now you don't have to worry about PowerShell CSV object structure or output since we act on the raw data of the file itself.
That will take each line of the input file and filter it out if the parsed date does not match the threshold. Change encoding on the input output cmdlets as you see necessary. What $date,$null=$_.Split(",",2) is doing is splitting the line
on the comma into 2 parts. First of which becomes $date and since this is just a filtering condition we dump the rest of the line into $null.
Properly-formed CSV files must have column headers. Your use of -NoTypeInformation in generating the CSV does not affect column headers; instead, it affects whether the PowerShell object type information is included. If you Export-CSV without -NoTypeInformation, the first line of your CSV file will have a line that looks like #TYPE System.PSCustomObject, which you don't want if you're going to open the CSV in a spreadsheet program.
If you subsequently Import-CSV, the headers (Date, Time, A, B, C) are used to create the fields of a PSObject, so that you can refer to them using the standard dot notation (e.g., $CSV[$line].Date).
The ability to specify -Header on Import-CSV is essentially a "hack" to allow the cmdlet to handle files that are comma-separated, but which did not include column headers.

powershell filter to remove .pdf extension in the name of a file

I am trying to use powershell to get all child elements in a folder the code I am using is
Get-ChildItem -Recurse -path C:\clntfiles
this code gives output like
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 4/29/2015 9:11 AM 6919044 HD 100616 Dec2014.pdf
-a--- 5/1/2015 11:42 AM 7091019 HD 101642 Jan2015.pdf
I don't want Mode lastWriteTime Length and name of file without .pdf extension
the output should be like
Dec2014
Jan2015
I am not sure how to filter that. please advise
I'll start by posting something similar to Leptonator's answer, but simplified by using the Select-Object command (alias Select used in code because it's habit, and I'm lazy).
$files = Get-ChildItem -Recurse -path C:\clntfiles | Select -ExpandProperty BaseName
Now that gets you the file names without extension. But, you actually asked for only part of the file names, as the first file name is "HD 100616 Dec2014.pdf" and you specified that you actually only want "Dec2014" to be returned. We can do that a couple different ways, but my favorite of them would be a RegEx match (because RegEx is awesome, and I think the LastIndexOf/SubString combo is overly complicated imho).
So, a RegEx match of "\w+$" will get what you want. That is broken down like this:
\w means any letter or number
+ means 1 or more of them
$ means the end of the string/line
So that's 1 or more alpha-numeric characters at the end of the string. We pipe our array of file names into a ForEach-Object loop (alias ForEach used out of habit), and then we have:
$Files | ForEach{ [RegEx]::Matches($_,"\w+$")}
Now, this outputs a [System.Text.RegularExpressions.Match] object, which is more than you want, but it does have a property Value which is exactly what you asked for! So we use Select -Expand again for that property and the output is precisely what you asked for:
$files = Get-ChildItem -Recurse -path C:\clntfiles | Select -ExpandProperty BaseName
$files | ForEach{[regex]::Matches($_,"\w+$")} | Select -Expand Value
RegEx matches are really handy, and if you learn about them you can simplify that quite a bit more like this:
gci C:\clntfiles -Rec | ?{$_.BaseName -match "(\w+)$"} | %{$Matches[1]}
That one line, as well as the two line code above it both should output:
Dec2014
Jan2015
Something like this should do it for you..
$files = Get-ChildItem -Recurse -path C:\clntfiles
if ($files -ne $null)
{
foreach ($file in $files)
{
$file.BaseName
}
}
In my folder, it shows:
> 2014-03-28_exeresult_file
> 2014-03-30_exeresult_file
> 2014-03-31_exeresult_file
> 2014-04-02_exeresult_file
> 2014-04-03_exeresult_file
> 2014-04-04_exeresult_file
> 2014-04-06_exeresult_file
> 2014-04-08_exeresult_file
and are indeed .txt files
Hope this helps!
Use the following Get-ChildItem -Recurse -name -path C:\clntfiles. This will get you only the file names.
Working solution:
$names = Get-ChildItem -name
foreach($n in $names) {$n.Substring(0,$n.IndexOf("."))}
You can also use LastIndexOf if part of the file name is .