As you probably understand from the title, I'm new to PowerShell, having a hard time even trying to describe my question. So please forgive my terminology.
Scenario
I am using PowerShell to query the audit log of Office 365. The cmdlet Search-UnifiedAuditLog returns "multiple sets of objects"(?), one of which has an array of other objects(?). The output is JSON if I got this right.
Here is an example of what is returned (I will call it one "Set of Objects"):
RunspaceId : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
RecordType : AzureActiveDirectoryStsLogon
CreationDate : 21/02/2017 12:05:23
UserIds : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Operations : UserLoggedIn
AuditData : {"CreationTime":"2017-02-21T12:05:23","Id":"{"ID":"00000000000000000","Type":3}],"ActorContextId":"xxxxxxxxxxxxxxxxxxxxxxxxx","ActorIpAddress":"xxxxxxxxxxxxx","InterSystemsId":"xxxxxxxxxxxxxxxxx","IntraSystemId":"000000000000-000000-000","Target":[{"ID":"00-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"xxxxxxxxxxxxxxxxxx","ApplicationId":"xxxxxxxxxxxxxxxxxxxxxxxxxx"}
ResultIndex : 1
ResultCount : 16
Identity : xxxxxxxxxxxxxxxxxxxxxxxxxxx
IsValid : True
ObjectState : Unchanged
Now, I want some of the content of the AuditData line exported to a csv (normally containing much more data than copied here). This works fine for one "set of objects" (like the one above). To do this I use:
$LogOutput = Search-UnifiedAuditLog -StartDate 2/20/2017 -EndDate 2/23/2017 -ResultSize 1
$ConvertedOutput = ConvertFrom-Json -InputObject $LogOutput.AuditData
$ConvertedOutput | Select-Object CreationTime,UserId,ClientIP | Export-Csv -Path "C:\users\some.user\desktop\users.csv
ResultSize returns 1 instead of multiple "sets of objects". The ConvertFrom-Json does not work if I remove ResultSize.
So the question is:
Can I loop through all the "set of objects" and convert from json and have this exported line-by-line on a csv? Resulting in something like this:
UserId,Activity,UserIP
this#user.com, loggedIn, 10.10.10.10
that#user.org, accessedFile, 11.11.11.11
A pedagogic answer would be very, very much appreciated. Many thanks!
Instead of -ResultSize, try using Search-UnifiedAuditLog <args> | Select-Object -ExpandProperty AuditData | ConvertFrom-Json
This will make only the AuditData property get forwarded into ConvertFrom-Json and ignore the rest of the object from Search-UnifiedAuditLog
Related
I am pretty sure there is a simple answer, but I cannot figure out how to ask it accurately enough as I am new to PowerShell.
Simple put, I am doing some API calls and then running through the results. The results include various properties that I converted from JSON into a Powershell Object.
A record looks simply like this:
id : 10123
path : /a/path/here/file.pptx
creationDate : 2019-06-28T09:37:32.000-04:00
modifiedDate : 2020-03-09T13:56:13.000-04:00
description : Record Description
creator : asmith
lastModifiedBy : jsmith
I then want to interact with the records, so I use a FOR loop on the IDs:
Foreach($SimpleID in $Records.id){
Write-Host $SimpleID + "`t|`t"+ $Records.path >> file.log
}
My question is, I want that output in the file to be:
10123 | /a/path/here/file.pptx
10124 | /next/path/file.txt
...etc
I know that the $Records.path is not correct there, but how do I filter it to only print the single path for the ID? I am sure there is a simple way, but I cannot figure out what to look up to start.
Any ideas on where to start?
You cannot use Write-Host to produce data output - see this post.
It sounds like you're looking for the following:
$Records | ForEach-Object { $_.id + "`t|`t"+ $_.path } > file.log
I would like to provide this alternative to mklement0's solution using Set-Content and Add-Content:
Set-Content -Path '.\log' -Value ''
$Records | ForEach-Object { Add-Content -Path '.\log' -Value "$_.id | $_.path" }
Loop over the Records objects and grab only what you need.
When i look into my object arra and type a $obj[2] (as exam) I'm getting
Name : audit_event
UUID : c6479a6f-f1bd-4759-9881-fcb493821aff
Timestamp : 17-06-20 13:30:48 +00:00
Fields : {[audit_schema_version, 1], [event_time, 17-06-20 13:30:48 +00:00], [sequence_number, 1], [action_id, 541934402]...}
Actions : {}
I would like to get all fields into a single csv file.
So first I started to find at least the fields, but dispite some solutions I saw i'm not getting it OK.
foreach ($UUID in $logsOBJECT[2].UUID) {
echo $UUID
foreach ($field in $logsOBJECT.$UUID.Keys) {
echo $field
}
}
This doesn't work.
I'm not a Powershell developer so quite novice.
I have to use Powershell because Synaps Analytics doesn't give a better option to read and process its logfiles.
Kind regards, Harry
Given an object that looks like this in JSON:
[{
"Name": "audit_event",
"UUID": "c6479a6f-f1bd-4759-9881-fcb493821aff",
"Timestamp": "17-06-20 13:30:48 +00:00",
"Fields": [["audit_schema_version", 1], ["event_time", "17-06-20 13:30:48 +00:00"], ["sequence_number", 1], ["action_id", 541934402]],
"Actions": {}
}]
(You can generate that by using $obj | ConvertTo-Json so it's easier for others to reproduce)
First we loop through the $obj list by passing it to ForEach-Object, or % for short. For each element we create a $result object that contains all the data, except the Fields property.
Then we loop through the fields property on the object. Each field is itself a list, where the first (0) element is the name, and the second (1) is the value. For each field, we add a property to the $result object with the name and value of the field.
When this is done we return the $result object to the pipeline, and that gets passed to Export-Csv which writes it to a file.
$obj | % {
$result = $_ | Select-Object * -ExcludeProperty Fields
$_.Fields | % {
$result | Add-Member -Name $_[0] -Value $_[1] -MemberType NoteProperty
}
return $result
} | Export-Csv -Path C:\test.csv -Encoding UTF8 -NoTypeInformation
I don't have your exact PS module of SynapsAnalystics...
But it seems to be a problem accessing nested arrays in Powershell.
Here I have an example with Windows services:
PS C:\WINDOWS\system32> $servic[20] | fl
Name : BrokerInfrastructure
DisplayName : Background Tasks Infrastructure Service
Status : Running
DependentServices : {workfolderssvc, WMPNetworkSvc, WSearch, embeddedmode}
ServicesDependedOn : {DcomLaunch, RpcSs, RpcEptMapper}
CanPauseAndContinue : False
CanShutdown : False
CanStop : False
ServiceType : Win32ShareProcess
Here, if I want the output of $servic.DependentServices I need to know the Keys\Propertys of $servic.DependentServices. ie,
You can get that by :
PS C:\WINDOWS\system32> $servic[20].DependentServices
Status Name DisplayName
------ ---- -----------
Stopped workfolderssvc Work Folders
Running WMPNetworkSvc Windows Media Player Network Sharin...
Running WSearch Windows Search
Stopped embeddedmode Embedded Mode
So the Propertys here are
Status Name DisplayName etc...
$servic[20].DependentServices would be similar to $obj[2].Fields in your case.
Try and see the Keys or Property's within $obj[2].Fields then decide which Property you want to loop through.
you can get that with
$obj[2].Fields | get-Module (this will give all parameters)
Then you can loop through the required Properties, like in my case:
foreach ($echserv in $servic[0-2])
{
write-host "*****************Service******************"
echo $echserv.Name
Write-Host "####DependentServices####"
foreach ($echDependServic in $servic.DependentServices.DisplayName)
{
echo $echDependServic
}
}
which would give output:
*****************Service******************
XboxGipSvc
####DependentServices####
Smartlocker Filter Driver
Agent Activation Runtime_ea2d3
Agent Activation Runtime
Windows Audio
Agent Activation Runtime_ea2d3
Agent Activation Runtime
Xbox Live Networking Service
.
.
.
Hope this helps with your problem.
PS: There are better ways to display your output using hashtables in Powershell. This can later be used to export to CSV etc..
Question
Should the script block of Invoke-Command, when run with a PSSession, always run on the remote computer?
Context
I ran the below powershell against a list of servers:
Clear-Host
$cred = get-credential 'myDomain\myUsername'
$psSessions = New-PSSession -ComputerName #(1..10 | %{'myServer{0:00}' -f $_}) -Credential $cred
Invoke-Command -Session $psSessions -ScriptBlock {
Get-Item -Path 'HKLM:\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters'
} | Sort-Object PSComputerName
# $psSessions | Remove-PSSession
This returned:
Hive: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos
Name Property PSComputerName
---- -------- --------------
Parameters MaxPacketSize : 1 myServer01
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer02
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer03
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer04
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer05
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer06
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer07
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer08
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer09
MaxTokenSize : 65535
Parameters MaxPacketSize : 1 myServer10
MaxTokenSize : 65535
All looks good; onlyl I'd not expected to see these values / I was running this as a quick sense check before setting the values on these servers to ensure I didn't overwrite anything.
I had a quick look at one of the servers using regedit; and found that MaxTokenSize and MaxPacketSize did not exist.
I then amended the command to use Get-ItemProperty instead of Get-Item:
Invoke-Command -Session $psSessions -ScriptBlock {
Get-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters' -Name 'MaxTokenSize'
} | Sort-Object PSComputerName
This time I got 10 errors:
Property MaxTokenSize does not exist at path HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters.
+ CategoryInfo : InvalidArgument: (MaxTokenSize:String) [Get-ItemProperty], PSArgumentException
+ FullyQualifiedErrorId : System.Management.Automation.PSArgumentException,Microsoft.PowerShell.Commands.GetItemPropertyCommand
+ PSComputerName : myServer01
# ... (and the same for the other 9 servers, with only PSComputerName changing)
Regarding where the values that were returned came from... they're from my local machine. Amending my local registry entries and rerunning the original command showed all "servers" as having the new value.
I'm guessing this is a bug; but because I've not played with PSSessions much so far wanted to check here in case it's an issue with my understanding / usage of these commands, or if there are known gotchas to watch out for when using PSSessions.
tl;dr:
The root cause is a bug in the formatting instructions for registry keys (as of Windows PowerShell 5.1.18362.125 and PowerShell Core 7.0.0-preview.2) leading to the unexpected mix of remote and local information - see GitHub issue #10341.
As an aside: Similarly, PowerShell's ETS (extended type system) can introduce problems too, as shown in Mathias' answer here.
The best workaround is to simply use Get-ItemProperty (without a -Name argument) instead of Get-Item.
Mathias R. Jessen has provided the crucial pointer in a comment on the question, and js2010's answer provides a limited workaround and a pointer to the root cause, but it's worth providing more background information:
PowerShell comes with formatting instructions for type Microsoft.Win32.RegistryKey, as output by Get-Item with a registry path.
These formatting instructions define a calculated column named Property for the default (tabular) view, which helpfully shows a summary of the output registry key's values, which involves accessing the registry again, using Get-ItemProperty as shown in js2010's answer.
However, that behind-the-scenes Get-ItemProperty call always accesses the local registry - even when the keys were retrieved from a different machine, via PowerShell remoting, so you'll end up with a spurious mix of remote and local information.
Note that, technically, when Get-Item is run remotely, what you receive locally is an approximation of the original Microsoft.Win32.RegistryKey object, due to the serialization and deserialization involved in remoting. This approximation is a custom object with static copies of the original object's property values, and its (simulated) type name is Deserialized.Microsoft.Win32.RegistryKey - note the prefix.
PowerShell applies formatting instructions based on the full type name of output objects, but in the absence of a specific instructions or a given Deserialized.<originalTypeName> type, PowerShell applies the instructions for <originalTypeName>, which is what causes the problems here.
A - cumbersome, but edition-agnostic[1] - way to see the problematic formatting instruction is to run the following command:
(Get-FormatData Microsoft.Win32.RegistryKey -PowerShellVersion $PSVersionTable.PSVersion).FormatViewDefinition.Control | % {
$colNames = $_.Headers.Label
$colValues = $_.Rows.Columns.DisplayEntry.Value
foreach ($i in 0..($colNames.Count-1)) {
[pscustomobject] #{
ColumnName = $colNames[$i]
ColumnValue = $colValues[$i]
}
}
} | Format-Table -Wrap
This yields the column names and definitions for the table view:
ColumnName ColumnValue
---------- -----------
Name PSChildName
Property
$result = (Get-ItemProperty -LiteralPath $_.PSPath |
Select * -Exclude PSPath,PSParentPath,PSChildName,PSDrive,PsProvider |
Format-List | Out-String | Sort).Trim()
$result = $result.Substring(0, [Math]::Min($result.Length, 5000) )
if($result.Length -eq 5000) { $result += "..." }
$result
The workaround suggested in js2010's answer - piping to Format-Table * or Format-List * is effective in the sense that it prevents the inapplicable local information from being displayed: by specifying properties explicitly (even by wildcard pattern *), only those properties are displayed on output - not also the flawed calculated column.
However, while the true Property property of the output objects provides access to the value names in the registry key at hand, it doesn't provide the actual data, the way that the calculated Property column does.
By contrast, using Get-ItemProperty without a -Name argument in lieu of Get-Item as a workaround returns both value names and data (correctly even when remoting) and even does so without restrictions (whereas Get-Item limits output to 5000 chars.)
The output format will be slightly different, but all the information is there.
[1] That is, the command works also in PowerShell Core, where the built-in formatting instructions are no longer maintained as external *.format.ps1xl files and are instead compiled into the executable.
Pipe it to fl * or ft * so it doesn't use the format file to display the registry keys. The format file runs get-itemproperty locally to try to display the properties.
From the bottom of $PSHOME\Registry.format.ps1xml for type Microsoft.Win32.RegistryKey:
<ScriptBlock>
$result = (Get-ItemProperty -LiteralPath $_.PSPath |
Select * -Exclude PSPath,PSParentPath,PSChildName,PSDrive,PsProvider |
Format-List | Out-String | Sort).Trim()
$result = $result.Substring(0, [Math]::Min($result.Length, 5000) )
if($result.Length -eq 5000) { $result += "..." }
$result
</ScriptBlock>
I would like to create a CSV file (with no headers if possible) from a SQL query I make. My powershell is below. It reads the DB no problem and I see row results come back from printint to stdout. But I can't seem to figure out how to get the results into a CSV file.
The txt file created from my powershell only has one line it:
TYPE System.Int32
Which seems to me like it is just outputing the data type or something. It seems odd it is In32 but maybe that is because it is a pointer to the object?
I saw one article that used Tables[0] when using a data set but when I replaced my last line with
$QueryResults.Tables[0] | export-csv c:\tests.txt
It gave me an error saying it couldn't bind an argument to parameter because it was null.
function Get-DatabaseData {
param (
[string]$connectionString,
[string]$query,
[switch]$isSQLServer
)
if ($isSQLServer) {
Write-Verbose 'in SQL Server mode'
$connection = New-Object -TypeName System.Data.SqlClient.SqlConnection
} else {
Write-Verbose 'in OleDB mode'
$connection = New-Object -TypeName System.Data.OleDb.OleDbConnection
}
$connection.ConnectionString = $connectionString
$command = $connection.CreateCommand()
$command.CommandText = $query
if ($isSQLServer) {
$adapter = New-Object -TypeName System.Data.SqlClient.SqlDataAdapter $command
} else {
$adapter = New-Object -TypeName System.Data.OleDb.OleDbDataAdapter $command
}
$dataset = New-Object -TypeName System.Data.DataSet
$adapter.Fill($dataset)
$dataset.Tables[0]
}
$QueryResults = Get-DatabaseData -verbose -connectionString 'Server=localhost\SQLEXPRESS;;uid=sa; pwd=Mypwd;Database=DanTest;Integrated Security=False;' -isSQLServer -query "SELECT * FROM Computers"
foreach($MyRow in $QueryResults) {
write-output $MyRow.computer
write-output $MyRow.osversion
}
$QueryResults | export-csv c:\tests.txt
I added a line to print $QueryResults to stdout and below is the output in the console of that and the write form my for loop to show what my $QueryResults has.
5
computer : localhost
osversion :
biosserial :
osarchitecture :
procarchitecture :
computer : localhost
osversion :
biosserial :
osarchitecture :
procarchitecture :
computer : not-online
osversion :
biosserial :
osarchitecture :
procarchitecture :
computer :
osversion : win8
biosserial :
osarchitecture :
procarchitecture :
computer : dano
osversion : win8
biosserial :
osarchitecture :
procarchitecture :
localhost
localhost
not-online
win8
dano
win8
You might have looping issues as from the sounds of it your $QueryResults might not contain the data you are looking for. What does $QueryResults look like while in the console? Once you address that this should take care of the output for you.
From TechNet
By default, the first line of the CSV file contains "#TYPE " followed by the fully-qualified name of the type of the object.
To get around that you would just use the -NoTypeInformation switch like Alroc suggested. However you also mentioned removing the header which would require a different approach.
$QueryResults | ConvertTo-CSV -NoTypeInformation | Select-Object -Skip 1 | Set-Content "c:\tests.txt"
or something slightly different with the same result.
$QueryResults | ConvertTo-CSV | Select -Skip 2 | Set-Content "c:\tests.txt"
Convert the $QueryResults to a csv object. However you choose to we omit the lines containing the type information and the header / title row. Should just have nothing but data at that point.
Update From Discussion
The ConvertTo-CSV and Export-CSV were not showing expected data but nothing except type information. I didnt really notice at first but the output of $QueryResults contained a first line that was just the number 5. Presumably it was the # of records from the result dataset. If we skip the first record of $QueryResults then we are left with just the data needed. Might be better to look into ther reason that first line is there ( maybe there is a way to supress it. ). As the question stands currently we address this as follows. The first skip is to ignore the "5" line and the second is to remove the header from the csv output.
$QueryResults | Select -Skip 1 | Convert-ToCSV -NoTypeInformation | Select-Object -Skip 1 | Set-Content "c:\tests.txt"
If you want just the data in the CSV file, pass the -NoTypeInformation switch into Export-CSV.
Since, i am a beginner, i 've no much hands-on to the powershell programming.Although
i had a script developed to insert data from an array to the csv file as follows:
#Following is the array
$InventoryReport = New-Object -TypeName PSobject -Property #{
ComputerName = "1myComputerName"
DomainName = "2myComputerDomain"
Manufacturer = "3myComputerManufacturer"
}
#Now to export the data to csv, i am using following:
$InventoryReport |Select-Object -Property ComputerName, DomainName, Manufacturer | Export-Csv -Path "c:\abc.csv" -NoTypeInformation -ErrorAction Stop
#This works fine
and the output of above is :
"ComputerName","DomainName","Manufacturer"
"1myComputerName","2myComputerDomain","3myComputerManufacturer"
....
Now, i don't want this , i want the ouput to appear in columnar fashion i.e.
"ComputerName","1myComputerName"
"DomainName","2myComputerDomain"
"Manufacturer","3myComputerManufacturer"
What code changes should be done to achieve this. ?
Either you want CSV, which you already have, or you want a custom txt-file. If you want the latter, try this:
$comp = gwmi win32_computersystem
#"
"ComputerName","$($comp.Name)"
"DomainName","$($comp.Domain)"
"Manufacturer","$($comp.Manufacturer)"
"# | Out-File test.txt
sample of test.txt output below. I've got a non-domain, custom built pc, so don't worry about the values.
"ComputerName","GRAIMER-PC"
"DomainName","WORKGROUP"
"Manufacturer","System manufacturer"
EDIT I suggest you learn what CSV is. Remember that CSV is not a fileformat, it's a formatting-style used in a normal textfile. The .csv extension is just cosmetic to let people know that the textfile uses the csv-style. Check out Wikipedia and Technet
In the CSV file, each object is represented by a comma-separated list
of the property values of the object. The property values are
converted to strings (by using the ToString() method of the object),
so they are generally represented by the name of the property value.
Export-CSV does not export the methods of the object.
The format of an exported file is as follows:
-- The first line of the CSV file contains the string '#TYPE ' followed by the fully qualified name of the object, such as #TYPE
System.Diagnostics.Process. To suppress this line, use the
NoTypeInformation parameter.
-- The next line of the CSV file represents the column headers. It contains a comma-separated list of the names of all the properties of
the first object.
-- Additional lines of the file consist of comma-separated lists of the property values of each object.
You could try something like this:
$InventoryReport | Format-List ComputerName, DomainName, Manufacturer `
| Out-String -Stream `
| ? { $_ -ne '' } `
| % { $_ -replace '\s+:\s+', '","' -replace '(^|$)', '"' }