I am faced with a task convert 1 or more CSV files to JSON.
I have been able to do that with the following command:
Import-Csv "File1.csv" |
ConvertTo-Json -Compress |
Add-Content -Path "File1_output.JSON"
However, this only does one file at a time and you have to enter the name of the file manually.
To give you a background, say I have 2x CSV file, one called File1.csv and another File2.csv, what I would like to loop through all .csv files in the Documents folders and created a .json output file with the name appended to it.
Ideally I would like to call it as a .ps1 file or even a command line bat file as SSIS will be calling it ultimately.
Use Get-ChildItem for enumerating files, and a loop for processing the enumerated files.
Something like this should work for converting all CSVs in the current working directory:
Get-ChildItem -Filter '*.csv' | ForEach-Object {
Import-Csv $_.FullName |
ConvertTo-Json -Compress |
Set-Content ($_.Basename + '_output.json')
}
Note that this will convert each file individually. If you want to combine all CSVs into a single JSON document you could probably eliminate the loop
Get-ChildItem -Filter '*.csv' |
Import-Csv |
ConvertTo-Json -Compress |
Set-Content 'output.json'
at least as long as the CSVs have the same columns.
Otherwise you need to define the desired structure of the resulting JSON file first, and then merge the data from the CSVs in a way that conforms to the desired output format.
Related
I am just starting to learn PowerShell and have run into a hurdle where I'm trying to use gci and import-csv. My goal is to run a script in a folder directory that has numerous subfolders that contain a specific csv file that I would like to import and consolidate the data. These subfolders have additional subfolders that have other file types including csv that I don't have any use for. I am interested in the specific path below. The csv files have a specific header type called location that I care about and have to parse out into a string.
Folder directory example
This is my code so far:
$files = gci .\ -Recurse
foreach($file in $files) {
$foldername = $file.Name
$csv = Import-Csv -Path ".\$foldername\$foldername.csv"
foreach ($line in $csv) {
$outputlines = $line.location -split '/'
Export-csv -Path .\Test.csv -NoTypeInformation -Append
}
}
This is the message I get when I run it:
cmdlet Export-Csv at command pipeline position 1
Supply values for the following parameters:
InputObject:
Can someone please guide me in the right direction on what I'm doing wrong?
As others have commented and strictly speaking, Export-Csv needs something to export. However, that's not the only issue. Import-Csv will return objects based on the data in the csv file. So -split is likely to provide strange results as it's designed to run against strings and/or arrays of strings.
Withstanding the -split that is unlikely to work, you can address consolidation more simply by simply feeding the results from many Import-Csv commands to a single Export-csv command:
$OutputCsv = 'c:\temp\Output.csv'
Get-ChildItem .\ -Directory |
ForEach-Object{
$FolderName = $_.Name
".\$FolderName \$FolderName .csv"
} |
Import-Csv |
Export-Csv -Path $OutputCsv -NoTypeInformation -Append
The loop will output a series of strings that are piped to Import-Csv. So, all the files will get imported and the resulting objects will be streamed to Export-Csv consolidating everything into $Outputcsv which is c:\temp\Output.csv.
Note: The use of the -Directory parameter. Since you are only leveraging the folder names that should prevent a few errors, particularly for file names that may not exist.
If you want to clarify the question with an example of the CSV contents and the desired output we can take this further.
I need to merge CSV files and want to use following command from post
Merging multiple CSV files into one using PowerShell.
Get-ChildItem -Filter *.csv |
Select-Object -ExpandProperty FullName |
Import-Csv |
Export-Csv .\merged\merged.csv -NoTypeInformation -Append
However, only the first column of the source CSV files ends up in merged.csv.
Found the issue ...
The tool generating the csv files added as top line
sep=,
With this Excel openes the csv already correctly formatted.
After removing this line from the csv files all is working as expected with the Import-csv and Export-csv command.
Best regards,
Axel
I have got a set of txt files in a directory that I want to merge together.
The contents of all the txt files are in the same format as follows:
IPAddress Description DNSDomain
--------- ----------- ---------
{192.168.1.2} Microsoft Hyper-V Network Adapter
{192.168.1.30} Microsoft Hyper-V Network Adapter #2
I have the below code that combines all the txt files in to one txt file called all.txt.
copy *.txt all.txt
From this all.txt I can't see what lines came from what txt file. Any ideas on any bits of code that would add an extra column to the end txt file with the file name the rows come from?
As per the comments above, you've put the output of Format-Table into a text file. Note that Format-Table might be visually structured on screen, but is just lines of text. By doing that you have made it harder to work with the data.
If you just want a few properties from the results of the Get-WMIObject cmdlet, use Select-Object which (in the use given here) will effectively filter the data for just the properties you want.
Instead of writing text to a simple file, you can preserve the tabular nature of the data by writing to a structured file (i.e. CSV):
Get-WmiObject -Class Win32_NetworkAdapterConfiguration -Filter IPEnabled=TRUE -ComputerName SERVERNAMEHERE |
Select-Object PSComputerName, IPAddress, Description, DNSDomain |
Export-Csv 'C:\temp\server.csv'
Note that we were able to include the PScomputerName property in each line of data, effectively giving you the extra column of data you wanted.
So much for getting the raw data. One way you could read in all the CSV data and write it out again might look like this:
Get-ChildItem *.csv -Exclude all.csv |
Foreach-Object {Import-Csv $_} |
Export-Csv all.csv
Note that we exclude the output file in the initial cmdlet to avoid reading and writing form/to the same file endlessly.
If you don't have the luxury to collect the data again you'll need to spool the files together. Spooling files together is done with Get-Content, something like this:
Get-ChildItem *.txt -Exclude all.txt |
Foreach-Object {Get-Content $_ -Raw} |
Out-File all.txt
In your case, you wanted to suffix each line, which tricker as you need to process the files line-by-line:
$files = Get-ChildItem *.txt
foreach($file in $files) {
$lines = Get-Content $file
foreach($line in $lines) {
"$line $($file.Name)" | Out-File all.txt -Append
}
}
My task is to merge two xml files together. Before merging these two files, I need to remove the first line of the second file. I was able to receive the needed output file by writing these two lines:
#case if both files exists - remove first line from the file
(Get-Content $JfilePath | Select-Object -Skip 1) | Set-Content $JfilePath
#mergeFiles together
Get-Content $MfilePath, $JfilePath | Set-Content $mergedFile
The issue is that I am modifying the second file by executing the first cmdlet. I would like to keep both files in original form. I dont want to also create any temporary files.
I was trying to perform the following:
Get-Content $MfilePath, (Get-Content $JfilePath | Select-Object -Skip 1) | Set-Content $mergedFile
but I received the error:
Get-Content : Cannot convert 'System.Object[]' to the type 'System.String' required by parameter 'LiteralPath'. Specified method is not supported.
Could you please help how the output file could be received without modifying these input files?
Try this:
(Get-Content $MfilePath), (Get-Content $JfilePath | Select-Object -Skip 1) | Set-Content $mergedFile
My Old Bat file
Copy F:\File.hdr+F:*.csv F:\FinalOutput.csv
the HDR file is a single entry file that has only header data for the CSV files
Is there a way to perform this in PowerShell (to combine all the CSV files into a single file)?
Here is my powershell script that doesn't work
$CSVFolder = 'F:\Input\';
$OutputFile = 'F:\Output\NewOutput.csv';
$CSV= #();
Get-ChildItem -Path $CSVFolder -Filter *.inv | ForEach-Object {
$CSV += #(Import-Csv -Path $CSVFolder\$_)
}
$CSVHeader = Import-Csv 'F:\Input\Headings.hdr'
$CSV = $CSVHeader + $CSV
$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;
I get the list of FileNames that are exported and not the content of the Files.
The script is also modifying the date/time stamp on my INV files. It shouldn't be doing that.
You can skip the whole CSV bit if you just append the files as you would before.
Something like this should work:
# First we create the new file and add the header.
get-content $headerfile | set-content $outputfile
# Then we get the input files, read them out with get-content
# and append them to the output file (add-content).
get-childitem -path $csvfolder *.inv | get-content | add-content $outputfile
The CSV commandlets are handy if you want to be processing the CSV data in your script, but in your case simply appending the files will do the trick. Not bothering with the CSV conversion will be a lot faster as Powershell doesn't have to parse the CSV lines and create PS-objects. It's really fast with pure text though.
Another trick here is how the get-content and add-content are used in the pipeline. Since they are aware of the pipeline you can pass in file objects without having to use a foreach loop. This makes your statements a lot shorter.
How about:
get-childitem *.inv | foreach-object {
import-csv $_ -header (get-content Headings.hdr)
} | export-csv NewOutput.csv -notypeinformation