Pivot Table directly from SQL using powershell - powershell

I am using a combination of powershell scripts I've found online (shown below) to create an Excel document with pivot tables. But, I've only found examples where the datasource for the pivot tables was data already-imported into the excel document via CSV, or other similar methods. I can get those to work. But, my problem is that I would like to bypass the step of loading .csv data into excel and then creating the pivot tables...my whole issue is my dataset is way too large for excel...so I want to create the pivot table directly from a SQL connection. I cannot figure out the right syntax to to set the datasource. Below is my closest attempt...should use "xlExternal" for the datasource? But how do I set the datasource then? Micorsoft's documentation/APIs are somwhat helpful, but their examples are all written in VB and they also use other methods I'm not even familiar with in VB, so I'm having a hard time translating the QueryTables.Add method and some others.
I believe the only 1 or 2 lines that I'm stuck on are this one:
$qt = $ws.QueryTables.Add("ODBC;DSN=$connectionString", $ws.Range("A1"),
$SQL)
And/or this one:
$PivotTable =
$wb.PivotCaches().Create($xlExternal,$selection,$xlPivotTableVersion10)
But my code gives the exception on this line:
$PivotTable.CreatePivotTable("R1C1","Tables1") | Out-Null
and the error is saying:
Exception from HRESULT: 0x800A03EC
At C:\Users\me\Desktop\NEWpsCode.ps1:107 char:1
+ $PivotTable.CreatePivotTable("R1C1","Tables1") | Out-Null
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
So, here is my code:
$DirectoryToSaveTo='C:\Users\me\Desktop\'
$Filename='myfile'
$ServerName = "BLAH1"
$DatabaseName = "BLAH2"
$userName = "BLAH3"
$password = "BLAH4"
$connectionString = "Server = $ServerName; Database = $DatabaseName; User ID
= $userName; Password = $password;"
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection
$connectionString
$sqlConnection.Open()
$xlCenter=-4108
$xlTop=-4160
$xlOpenXMLWorkbook=[int]51
$SQL=#"
SELECT Account1,Account2,Account3 from myTable
"#
$excel = New-Object -Com Excel.Application #open a new instance of Excel
$excel.Visible = $True
$wb = $Excel.Workbooks.Add()
$currentWorksheet=1
if ($currentWorksheet-lt 4)
{
$ws = $wb.Worksheets.Item($currentWorksheet)
}
else
{
$ws = $wb.Worksheets.Add()
}
$currentWorksheet += 1
$qt = $ws.QueryTables.Add("ODBC;DSN=$connectionString", $ws.Range("A1"),
$SQL)
$xlPivotTableVersion12 = 3
$xlPivotTableVersion10 = 1
$xlCount = -4112
$xlDescending = 2
$xlDatabase = 1
$xlExternal = 2
$xlHidden = 0
$xlRowField = 1
$xlColumnField = 2
$xlPageField = 3
$xlDataField = 4
$xlDirection = [Microsoft.Office.Interop.Excel.XLDirection]
$range1=$ws.range("A1")
$range1=$ws.range($range1,$range1.End($xlDirection::xlDown))
$range2=$ws.range("B1")
$range2=$ws.range($range2,$range2.End($xlDirection::xlDown))
$selection = $ws.range($range1, $range2)
$PivotTable =
$wb.PivotCaches().Create($xlExternal,$selection,$xlPivotTableVersion10)
$PivotTable.CreatePivotTable("R1C1","Tables1") | Out-Null
$filename = "$DirectoryToSaveTo$filename.xlsx"
if (test-path $filename ) { rm $filename }
$wb.SaveAs($filename, $xlOpenXMLWorkbook)
$wb.Saved = $True
$wb.Close()
$Excel.Quit()
$wb = $Null
$ws = $Null
$Excel=$Null
So, if anyone can please direct me to setting up a pivot table using SQL connection directly, that would be extremely helpful! Even a very simple example will help immensely.

Related

Powershell opc xml write UInt32 array

I've managed to read an OPC XML item with a Powershell script
$wsdl=(Join-path $PSScriptRoot "OpcXmlDa1.00.wsdl")
$global:Proxy = New-WebserviceProxy $wsdl –Namespace X
$global:Options = New-Object X.RequestOptions
$Options.ReturnItemName = $true;
$Options.ReturnItemTime = $true;
$global:ItemList = New-Object X.ReadRequestItemList
$global:RItemList = New-Object X.ReplyItemList
$global:Errors = New-Object X.OPCError
$global:Item = New-Object X.ReadRequestItem
# Read from OPC server
$Proxy.Url = "http://xxx.xx.xx.xxx:xxxx"
$Item.ItemName = "xxx/xxx/xxx"
$ItemList.Items = $Item
$Proxy.Read($Options, $ItemList, [ref]$RItemList, [ref]$Errors)
#Show result
$rTimeStamp = $RItemList.Items[0].Timestamp.ToString('yyyy-MM-dd HH:mm:ss')
$rItemName = $RItemList.Items[0].ItemName
$rValue = $RItemList.Items[0].Value
Write-Host ("{0}: {1}={2}" -f $rTimeStamp, $rItemName, $rValue)
In my result above I see the following in $RItemList.Items[0]
DiagnosticInfo:$null
Value: [UInt32[32]]
Quality: [OPCQuality]
ValueTypeQualifier: $null
ItemPath: ""
ItemName: "xxx/xxx/xxx"
ClientItemHandle: $null
Timestamp: 2020-12 19 01:00:00
TimestampSpecified: $true
ResultID: $null
and when I dive deeper into Value above I see that it is read as expected. So far so good...
Now to my problem; I cannot understand how to write to this same Item via another script.
This is my attempt
$wsdl=(Join-path $PSScriptRoot "OpcXmlDa1.00.wsdl")
$Global:Proxy = New-WebserviceProxy $wsdl –Namespace X
$Global:Options = New-Object X.RequestOptions
$Options.ReturnItemName = $true;
$Options.ReturnItemTime = $true;
$Options.ReturnDiagnosticInfo = $true;
$Global:writeItemList = New-Object X.WriteRequestItemList
$Global:RItemList = New-Object X.ReplyItemList
$Global:Errors = New-Object X.OPCError
$Global:ItemSessionRequest = New-Object X.ItemValue
$ItemSessionRequest.ValueTypeQualifier = 'xsd:unsignedInt' #<- I believe that this might be the issue?
$ItemSessionRequest.ItemPath = ""
$ItemSessionRequest.ItemName = "xxx/xxx/xxx"
$ItemSessionRequest.Value = #($x;$y;$z) #<- I have what I want to write in $x, $y, $z
# Write to OPC server
$Proxy.Url = "http://xxx.xx.xx.xxx:xxxx"
$ReturnValuesOnReply = $true
$writeItemList.Items = $ItemSessionRequest
$Proxy.Write($Options, $writeItemList, $ReturnValuesOnReply, [ref]$RItemList, [ref]$Errors)
I get two errors (I'm running via VSC-code).
Error #1:
DiagnosticInfo : This value to write cannot be NULL
Value :
Quality :
ValueTypeQualifier :
ItemPath :
ItemName : xxx/xxx/SessionRequest
ClientItemHandle :
Timestamp : 1970-01-01 00:59:59
TimestampSpecified : True
ResultID : http://opcfoundation.org/webservices/XMLDA/1.0/:E_BADTYPE
I've assumed that this has to do with my choice of $ItemSessionRequest.ValueTypeQualifier (where I've tried a lot of alternatives).
Error #2: This error comes if I run the script a second time in the same instance
Cannot convert argument "Options", with value: "X.RequestOptions", for "Write" to type "X.RequestOptions": "Cannot convert the "X.RequestOptions" value of type "X.RequestOptions" to type "X.Request
Options"."
At C:\bla.bla.ps1:79 char:1
+ $Proxy.Write($Options, $writeItemList, $ReturnValuesOnReply, [ref]$RI ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument
I guess this has something to do with the $Proxy variable
I'm grateful for help and I apologize if I missed something relevant, but I have done my best...
In this case problem no 1 was solved by the following
$ItemSessionRequest.Value = #($x;$y;$z)
A work around for problem no 2 was by using a new session inside the script, see 1

I am trying to update the data source of an oledb data adapter and nothing happens

I am attempting to update data in an oledb datasource. I pull data from a table that who's schema has been written out as an access create table script (i could probably find a better way to get that schema information) and stored in the AccessCreateScript field of the lookup table.
I am able get the following things to work:
pull the data out of the source table on sql server
put it into a dataset
verify its good
create a second dataset based off the created table on the access file
copy the data from the source dataset to the destination dataset
verify that the data has been copied
create the update command using a command builder
verify the commands look correct
But, then when it comes to using the update method on the ole data adapter, it just doesnt do anything.
The data contained in the lookup table has been validated to be correct thoroughly.
There currently is only one record in the lookup table.
$LookupServer = "Redacted"
$LookupDatabase = "redacted"
$LookupSQL = "Select [DBServer]
,[SourceDB]
,[SourceTableOrScript]
,[DestinationFile]
,[DestinationTable]
,[AccessDropScript]
,[AccessCreateScript]
,[Active]
FROM [AccessDataExport_Lookup_Data]"
$LookupConnection = New-Object System.Data.SQLClient.SQLConnection
$LookupCSBuilder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder
$LookupCSBuilder['server'] = $LookupServer
$LookupCSBuilder['trusted_connection'] = $True
$LookupCSBuilder['database'] = $LookupDatabase
$LookupConnection.ConnectionString = $LookupCSBuilder.ConnectionString
$LookupConnection.Open()
$LookupCommand = New-Object System.Data.SQLClient.SQLCommand
$LookupCommand.Connection = $LookupConnection
$LookupCommand.CommandText = $LookupSQL
$LookupDataAdapter = new-object System.Data.SqlClient.SqlDataAdapter $LookupCommand
$LookupDataset = new-object System.Data.Dataset
$LookupDataAdapter.Fill($LookupDataset)
$LookupConnection.Close()
foreach( $r in $LookupDataset.Tables[0].Rows){
$SourceConnection = New-Object System.Data.SQLClient.SQLConnection
$SourceCSBuilder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder
$SourceCSBuilder['server'] = $r.DBServer
$SourceCSBuilder['trusted_connection'] = $true
$SourceCSBuilder['database'] = $r.SourceDB
$SourceConnection.ConnectionString = $SourceCSBuilder.ConnectionString
$SourceCmd = $SourceConnection.CreateCommand()
$DestConnection = New-Object System.Data.OleDb.OleDbConnection
$DestCSBuilder = New-Object System.Data.OleDb.OleDbConnectionStringBuilder
$DestCSBuilder['Provider'] = 'Microsoft.ACE.OLEDB.12.0'
$DestCSBuilder['Persist Security Info'] = $false
$DestCSBuilder['Data Source'] = $r.DestinationFile
$DestConnection.ConnectionString = $DestCSBuilder.ConnectionString
$DestCmd = $DestConnection.CreateCommand()
$DestConnection.Open()
$SourceConnection.Open()
$DestCmd.CommandText = $r.AccessDropScript
try
{
$DestCmd.ExecuteNonQuery()
}
catch
{
write-Output $_.Exception.Message
write-Output $_.InvocationInfo.ScriptLineNumber
}
$DestCmd.CommandText = $r.AccessCreateScript
try{
$DestCmd.ExecuteNonQuery()
$SourceCmd.CommandText = $r.SourceTableOrScript
$SourceDA = New-object System.Data.SqlClient.SqlDataAdapter $SourceCmd
$SourceData = new-object System.Data.Dataset
$DestCmd.CommandText = $r.SourceTableOrScript
$DestDA = New-Object System.Data.OleDb.OleDbDataAdapter $DestCmd
$DestData = new-object System.Data.Dataset
$result = $SourceDA.fill($SourceData)
Write-Output "Source filled $result rows"
$result = $DestDA.fill($DestData)
Write-Output "Dest filled $result rows"
$DestData.load($SourceData.CreateDataReader(),1,$SourceData.Tables[0].TableName)
$destRowCount = $DestData.Tables[0].Rows.Count
Write-Output "DestData Row Count $destrowcount"
$DestCb = New-Object System.Data.OleDb.OleDbCommandBuilder $DestDA
$DestDA.UpdateCommand = $DestCb.GetUpdateCommand()
$DestDA.InsertCommand = $DestCb.GetInsertCommand()
$DestDA.DeleteCommand = $DestCb.GetDeleteCommand()
$result = $DestDA.Update($DestData)
Write-Output "Copy data copied $result rows"
}
Catch
{
write-Output $_.Exception.Message
write-Output $_.InvocationInfo.ScriptLineNumber
}
finally{
$DestConnection.Close()
$SourceConnection.Close()
}
}
and the output is
PS C:\Users\****> & 'c:\Users\***\source\repos\Powershell Scripts\Modular-Export.ps1'
1
0
0
Source filled 1274 rows
Dest filled 0 rows
DestData Row Count 1274
Copy data copied 0 rows
Table data since this is pertinate, but with redacted sensitive information.
https://gist.github.com/KySoto/5a764f3fa3c0d139131056ed6d44529b
In the end a reddit user helped me figure it out, Their post was the thing that got everything rolling. What i needed to do was set the state of every datarow in my datatable to "Added" using the SetAdded method. Then i needed to set $DestCb.QuotePrefix = "[" and $DestCb.QuoteSuffix = "]" Then it properly filled.

Data type mismatch when querying a CSV with ACE OLEDB provider

I am attempting to query a CSV file using the Microsoft ACE OLEDB provider. When I add "PrctBusy > 60" to the where clause I receive the Error "Data type mismatch in criteria expression." I have searched StackOverFlow and used google to search for solutions, I see this is not an uncommon issue. From my readings it looks to be datatype issue. The data in the column PrctBusy is all numeric. I think I need to force it to be number but I have not found a solution.
Below is the code I am currently working with:
$ArrayNameUtil = "000198701258"
$CatNameUtil = "FE_DIR"
$sdLocalPath = "D:\Logs\SANData\Perf"
$InputCSV = "VMaxSANReportUtilFile.csv"
$csv = Join-Path $sdLocalPath $InputCSV
$provider = (New-Object System.Data.OleDb.OleDbEnumerator).GetElements() | Where-Object { $_.SOURCES_NAME -like "Microsoft.ACE.OLEDB.*" }
if ($provider -is [system.array]) { $provider = $provider[0].SOURCES_NAME } else { $provider = $provider.SOURCES_NAME }
$connstring = "Provider=$provider;Data Source=$(Split-Path $csv);Extended Properties='text;HDR=$firstRowColumnNames;';"
$firstRowColumnNames = "Yes"
$delimiter = ","
$tablename = (Split-Path $csv -leaf).Replace(".","#")
$conn = New-Object System.Data.OleDb.OleDbconnection
$conn.ConnectionString = $connstring
$provider = (New-Object System.Data.OleDb.OleDbEnumerator).GetElements() | Where-Object { $_.SOURCES_NAME -like "Microsoft.ACE.OLEDB.*" }
if ($provider -is [system.array]) { $provider = $provider[0].SOURCES_NAME } else { $provider = $provider.SOURCES_NAME }
$connstring = "Provider=$provider;Data Source=$(Split-Path $csv);Extended Properties='text;HDR=$firstRowColumnNames;';"
$firstRowColumnNames = "Yes"
$delimiter = ","
$tablename = (Split-Path $csv -leaf).Replace(".","#")
$conn = New-Object System.Data.OleDb.OleDbconnection
$conn.ConnectionString = $connstring
$conn.Open()
#
$sql = "SELECT TimeStamp, count(PrctBusy) AS Above60 FROM [$tablename] WHERE array = '$ArrayNameUtil' and Category like '$CatNameUtil' and PrctBusy > 60 Group by TimeStamp "
$cmd = New-Object System.Data.OleDB.OleDBCommand
$cmd.Connection = $conn
$cmd.CommandText = $sql
$dtp = New-Object System.Data.DataTable
$dtp.Load($cmd.ExecuteReader())
Because of the pointer from TessellatingHeckler to Codeproject and some follow on queries, I was lead to http://aspdotnetcodes.com/Importing_CSV_Database_Schema.ini.aspx. I found that a schema.ini file in the same directory as the CSV file could specify the data type.
The schema.ini file ended up in the following format:
[VMaxSANReportUtilFile.csv]
ColNameHeader=True
Format=CSVDelimited
Col1=Array Text Width 20
Col2=TimeStamp Text Width 20
Col3=Category Text Width 20
Col4=Instance Text Width 20
Col5=PrctBusy Short
Col6=QueUtil Short
I went through several revisions to get the data type correct for an ACE OLE DB provider. If the columns are named the names need to be in the schema.ini file.

Powershell iSeries DB2 ODBC returns "Arithmetic operation resulted in an overflow."

Summary: Powershell 5 using 64 bit ODBC driver for DB2. SQL query Select "Count(*)" works. Query for actual data "Select field" returns an error. 32 bit ODBC driver will not work at all. One answer here pointed to an ODBC setting (Convert binary data ...) for conversion but that did not resolve the problem.
Code:
$SqlQuery = #"
SELECT MMITNO,
MMITDS
FROM MITMAS
WHERE MMCONO = 3
AND MMITNO = '2658591351'
"#
$Db2Connection = new-object system.data.odbc.odbcconnection
$Db2Connection.connectionstring = "DSN=DB2_MR0P_64;Userid=USSRF_ADM;Password=########"
$Db2Connection.open()
# Set up to run the query
$SqlCmd = New-Object system.Data.Odbc.OdbcCommand($SqlQuery,$Db2Connection)
$DataAdapter = New-Object system.Data.Odbc.OdbcDataAdapter($SqlCmd)
$DataTable = New-Object system.Data.datatable
$RowCount = $DataAdapter.fill($DataTable)
$Db2Connection.close()
# Process the returned datatable
$DataTable.Table[0] | ForEach-object {
write-host "Total rows in MITMAS is $("{0:N0}" -f $_)."
}
Returns:
Exception calling "Fill" with "1" argument(s): "Arithmetic operation resulted in an overflow."
At C:\PowerShellScripts\TestDB2.ps1:19 char:1
+ $RowCount = $DataAdapter.fill($DataTable)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : OverflowException
To be quite frank, my knowledge of what i'm using is very limited and i've been successful by copying and pasting various posts from you experts. So, i apologize if the answer is simple and i'm just not seeing it. The only machine i have any control over is the local PC with no admin rights on the iSeries machine.
Fill method work with dataset, not datatable.
modify you code like this:
$DataAdapter = new-object System.Data.odbc.odbcDataAdapter($SqlCmd)
$DataSet = new-object System.Data.DataSet
$DataAdapter.Fill($DataSet)
foreach ($Row in $DataSet.Tables[0].Rows)
{
#your traitment here
}

Convert xlsx to CSV without using Excel

I get following error:
Cannot index into a null array.
At C:\tmp\Folder\excel\output\net45\test.ps1:14 char:1
+ $Data = $Reader.AsDataSet().Tables[0].Rows
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
# Zero based index. The second row has index 1.
$StartRow = 2
# Input File
$InputFileName = "C:\tmp\Folder\excel\output\net20\test.xlsx"
# Output File
$OutputFileName = "C:\tmp\Folder\excel\output\net20\SomeFile.csv"
# Path to Excel.dll is saved (downloaded from http://exceldatareader.codeplex.com/)
$DllPath = "C:\tmp\Folder\excel\output\net45\Excel.4.5.dll"
[void]([Reflection.Assembly]::LoadFrom($DllPath))
$Stream = New-Object IO.FileStream($InputFileName, "Open", "Read")
$Reader = [Excel.ExcelReaderFactory]::CreateBinaryReader($Stream)
$Data = $Reader.AsDataSet().Tables[0].Rows
# Read the column names. Order should be preserved
$Columns = $Data[$StartRow].ItemArray
# Sort the remaining data into an object using the specified columns
$Data[$($StartRow + 1)..$($Data.Count - 1)] | % {
# Create an object
$Output = New-Object Object
# Read each column
for ($i = 0; $i -lt $Columns.Count; $i++) {
$Output | Add-Member NoteProperty $Columns[$i] $_.ItemArray[$i]
}
# Leave it in the output pipeline
$Output
} | Export-CSV $OutputFileName -NoType
You're calling the binary method (.xls) and using an Open XML format file (.xlsx). Try using [Excel.ExcelReaderFactory]::CreateOpenXmlReader($Stream) instead.
This works for me:
$DllPath = 'C:\Excel.DataReader.45\Excel.4.5.dll';
$FilePath = 'C:\Students.xlsx';
$FileMode = [System.IO.FileMode]::Open;
$FileAccess = [System.IO.FileAccess]::Read;
Add-Type -Path $DllPath;
$FileStream = New-Object -TypeName System.IO.FileStream $FilePath, $FileMode, $FileAccess;
$ExcelDataReader = [Excel.ExcelReaderFactory]::CreateOpenXmlReader($FileStream);
$ExcelDataReader.IsFirstRowAsColumnNames = $true;
$ExcelDataSet = $ExcelDataReader.AsDataSet();
$ExcelDataReader.Dispose();
$FileStream.Close();
$FileStream.Dispose();
$ExcelDataSet.Tables | Format-Table -AutoSize
If you're still having trouble, you might consider using the Microsoft.ACE.OLEDB.12.0 provider, which you install separately from Office. There's some doc here.
I've read this "Convert XLS to CSV on command line" and this "convert-xlsx-file-to-csv-using-batch" before in a similar doubt I have. Try too see if it helps.