I am trying to upload some string values into an Oracle table by means of powershell. However when I upload strings directly some characters are shown up like ? in the table.
Actually, I first parse a text and retrieve some results through regex as below:
if($wiki_link -match "http:\/\/en\.wikipedia\.org\/wiki\/(.*)") {$city = $matches[1]}
Then I wanna upload this $city variable into a table as below:
[System.Reflection.Assembly]::LoadWithPartialName("System.Data.OracleClient")
$connectionString = "Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(Host=xxxxxxxxx)(Port=1521)))(CONNECT_DATA=(SERVER = DEDICATED) (SERVICE_NAME =xxxxx)));user id=xxxxxx;password=xxxxx"
$connection = New-Object System.Data.OracleClient.OracleConnection($connectionString)
$connection.Open()
$cmd2=$connection.CreateCommand()
$cmd2.CommandText="insert into mehmet.goo_region (city) values ('$city')"
$rdr2=$cmd2.ExecuteNonQuery()
When I apply this method, the city named Elâzığ appears as Elaz?? in the table cell.
I guess I have to convert string into UTF-8 but I could not find a solution through web.
Thanks in advance...
Try this, it should work:
$u = New-Object System.Text.UTF8Encoding
$s = $u.GetBytes("YourStringGoesHere")
$u.GetString($s) ## this is your UTF-8 string
So your code becomes
$u = New-Object System.Text.UTF8Encoding
$s = $u.GetBytes($city)
$utf8city = $u.GetString($s)
Related
Currently the code uses a comma for the column and a new line for row.
This is an issue because some of the data in the tables are paragraphs which already include commas and new lines.
I want to be able to use a delimiter with multiple characters but that is returning an error
Cannot bind parameter Delimiter. Cannot convert value '/~/' to type System.Char
$server = "(server)\instance"
$database = "DBName"
$tablequery = "SELECT name from sys.tables"
#Delcare Connection Variables
$connectionTemplate = "Data Source={0};Integrated Security=SSPI;Initial Catalog={1};"
$connectionString = [string]::Format($connectionTemplate, $server, $database)
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$command = New-Object System.Data.SqlClient.SqlCommand
$command.CommandText = $tablequery
$command.Connection = $connection
#Load up the Tables in a dataset
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
# Loop through all tables and export a CSV of the Table Data
foreach ($Row in $DataSet.Tables[0].Rows)
{
$queryData = "SELECT * FROM [$($Row[0])]"
#Specify the output location of your dump file
$extractFile = "C:\temp\backups\$($Row[0]).csv"
$command.CommandText = $queryData
$command.Connection = $connection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
$DataSet.Tables[0] | Export-Csv $extractFile -Delimiter '/~/'
}
Export-Csv and ConvertTo-Csv correctly handles newline and comma characters
It is not a problem that the data could potentially contain commas and/or new lines.
If you look at the CSV "specification" (written in quotation signs here because a lot of usages of the term CSV is not referring to anything following this specification) you'll see that a field in a CSV file can be enclosed in quation characters. If the data of that field contains a quotation character, the delimiter character or a newline character it must be enclosed in quotation characters. If the data contains a quotation character, that quotation character should be doubled.
This will all be handled correctly by the ConvertTo-Csv and the Export-Csv cmdlets.
$obj = New-Object PSObject -Property ([ordered]#{
FirstColumn = "First Value";
SecondColumn = "Second value, including a comma";
ThirdColumn ="Third Value including two`nnewline`ncharacters";
FourthColumn = 'Fourth value including a " character'}
)
$obj | ConvertTo-Csv -NoTypeInformation
This will give us the following output:
"FirstColumn","SecondColumn","ThirdColumn","FourthColumn"
"First Value","Second value, including a comma","Third Value including two
newline
characters","Fourth value including a "" character"
Which is correctly handled according to the CSV specification.
So you do not need to worry about the data containing comma characters or newline characters, since it is handled by the CSV format.
Open a CSV file in Excel
I don't know what your current problem with the data is, but I'm guessing you're trying to open the resulting file in Excel and seeing incorrect data. This is because Excel unfortunately doesn't open .csv files as... well... CSV files.
One way (there might be more ways) to open it in Excel is to go on the Data tab and in the "Get & Transform Data" section press the "From Text/CSV" button. This way, Excel should open the file correctly according to the CSV standard.
I'm trying to convert a text file to PDF using powershell in landscape.
Currently using a iTextSharp that has got me to great place with text conversion to PDF however I'm not able to find anything to place it in landscape when the lines are added to create the PDF.
Here is what I'm currently using.
[System.Reflection.Assembly]::LoadFrom("I:\powershell\itextsharp.dll")
$doc = New-Object itextsharp.text.document
$stream = [IO.File]::OpenWrite("I:\powershell\test.pdf")
$writer = [itextsharp.text.pdf.PdfWriter]::GetInstance($doc, $stream)
$doc.Open()
[IO.File]::ReadAllLines("I:\powershell\test.txt") | foreach {
$line = New-Object itextsharp.text.Paragraph($_)
$doc.Add($line)
}
$doc.Close()
$stream.Close()
I have found some for C# and Java just nothing with powershell.
Tried this which did not work:
$doc = New-Object iTextSharp.text.Document([iTextSharp.text.PageSize]::LEGAL_LANDSCAPE)
Thank you so much.
D
Use the overloaded constructor that takes a Rectangle, which explicitly sets the page size:
$doc = New-Object itextsharp.text.document(New-Object itextsharp.text.Rectangle(792, 612));
I have been wrestling with database connection to PostgreSQL from Powershell. I finally am able to connect to and insert into the database. Now I can't figure out how to extract data from a DB select into a variable.
I'm not including my insert for the sake of clarity but will tack it onto this thread later as I know it was super hard to find and may be helpful to someone.
so here's my code:
# use existing 64 bit ODBC System DSN that we set up manually
$DBconn = New-Object -comobject ADODB.Connection
$DBconn.Open("PostgreSQL35W")
$theQuery = "select * from test1"
$theObject = $DBconn.Execute($theQuery) # $theObject is a System.__ComObject
$numRecords = $theObject.RecordCount
write-host "found $numRecords records" # getting -1
$theObject.MoveFirst() # throws no error
# $theValue = $theObject.DataMember # throws no error, but gives no result
$theValue = $theObject.Index[1] # throws "Cannot index into a null array"
write-host($theValue)
try this
replace "#database#" with your database name in $cnString
replace "#server_ip#" with your server ip address in $cnString
replace "#user#" with a valid user in $cnString and $user
replace "#pass#" with a valid pass in $pass
replace "#table#" with a valid table name of your db
replace 5432 with your db port
$cnString = "DRIVER={PostgreSQL Unicode(x64)};DATABASE=#database#;SERVER=#server_ip#;PORT=5432;UID=#user#;"
$user="#user#"
$pass="#pass#"
$conn = New-Object -comobject ADODB.Connection
$conn.Open($cnString,$user,$pass)
$recordset = $conn.Execute("SELECT * FROM #table# limit 1;")
while ($recordset.EOF -ne $True)
{
foreach ($field in $recordset.Fields)
{
'{0,30} = {1,-30}' -f # this line sets up a nice pretty field format, but you don't really need it
$field.name, $field.value
}
'' # this line adds a line between records
$recordset.MoveNext()
}
$conn.Close();
Via psql, which comes with postgresql
$dburl="postgresql://exusername:expw#exhostname:5432/postgres"
$data="select * from extable" | psql --csv $dburl | ConvertFrom-Csv
You must have psql in your path or reference it, its within e.g. C:\Program Files\PostgreSQL\12\bin. Should be able to type "psql" and see output within powershell.
As a warning, expect strings. E.g $data[0].age.GetType() would be string, despite being stored in the database as an integer. You can immediately cast it, cast it later, or hope powershell infers type correctly.
If you want to add back in type information can do e.g.:
$data = $data | %{[pscustomobject]#{name=$_.name;age=[int]$_.age}}
I ended up figuring it out - here's what I did
$conn = New-Object -comobject ADODB.Connection
# use existing 64 bit ODBC System DSN that we set up manually
$conn.Open("PostgreSQL35W")
$recordset = $conn.Execute("SELECT * FROM JobHistory")
while ($recordset.EOF -ne $True)
{
foreach ($field in $recordset.Fields)
{
'{0,30} = {1,-30}' -f # this line sets up a nice pretty field format, but you don't really need it
$field.name, $field.value
}
'' # this line adds a line between records
$recordset.MoveNext()
}
$conn.Close();
Exit
use the dot notation. You don't need to split the data.
$list = New-Object Collections.Generic.List[OnlineCourse]
foreach($element in $results)
{
$tempObj= New-Object OnlineCourse($element.id,$element.courseName,$element.completedRatio,$element.completedRatio,$element.lastActivity, $element.provider)
$list.add($tempObj)
}
I have a slightly different approach to #dog, I couldn't get the --csv to work, so I resorted to tuple only rows returned, then parse them into a List of Classes (which happen to be called OnlineCourses):
class OnlineCourse
{
[int]$id
[string]$email
[string]$courseName
[int]$completedRatio
[datetime]$lastActivity
[String]$provider
OnlineCourse([int]$id,
[string]$email,
[string]$courseName,
[int]$completedPerc,
[datetime]$lastActivity,
[String]$provider) {
$this.id = $id
$this.email = $email.Trim()
$this.courseName = $courseName.Trim()
$this.completedRatio = $completedPerc
$this.lastActivity = $lastActivity
$this.provider = $provider.Trim()
}
}
$connstr="postgresql://exusername:expw#exhostname:5432/postgres"
$data = "select * from onlinecourses" | .\psql -t $connstr
$list = New-Object Collections.Generic.List[OnlineCourse]
foreach ($field in $data) {
$id, $email, $courseName, $completedratio, $lastactivity, $provider = $field.split('|')
$course = [OnlineCourse]::new($id, $email, $courseName, $completedratio, $lastactivity, $provider)
$list.Add($course)
}
This is slightly adapted from another answer and it worked for me.
$dburl="postgresql://postgres:secret_pwd#database-host:5432/dbname"
$psqlPath = 'C:\Program Files\PostgreSQL\11\bin\psql.exe'
function Query {
param($Sql)
Write-Host $Sql
$rows = $Sql `
| &$psqlPath "-A" $dburl | ConvertFrom-Csv -Delimiter '|'
$result = #($rows | Select-Object -SkipLast 1)
Write-Host "-> " (ConvertTo-Json $result)
$result
}
$rows = Query "select ... from ..."
I am trying to load extremely large CSV files into SQL Server using Powershell. The code also has to apply on the fly regex replacements, allow for various delimiters, EOR, and EOF markers. For maintenance, I would really like all of this logic to exist in Powershell without importing assemblies.
To be efficient, I know I need to use the SQLBulkCopy method. But, all of the Powershell examples I see fill a DataTable and pass it which is not possible for me because of the file size.
I am pretty sure I need to wrap StreamReader in an Idatareader and then pass that to SQLBulkcopy. I found a couple great examples of this implemented in C#:
http://archive.msdn.microsoft.com/FlatFileDataReader
http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader
Is it possible to accomplish this functionality using native PowerShell without importing the C# assembly? I am specifically having a hard time converting the abstract class wrapper.
This is the code I have so far that does not pass an IdataReader and breaks on memory limits.
function Get-CSVDataReader()
{
param (
[string]$path
)
$parsedData = New-Object 'System.Collections.Generic.List[string]'
#List<string[]> parsedData = new List<string[]>()
$sr = new-object IO.StreamReader($path)
while ($line = $sr.ReadLine())
{
#regex replace and other logic here
$parsedData.Add($line.Split(','))
}
,$parsedData #if this was an idatareader, the comma keeps it from exploding
}
$MyReader = Get-CSVDataReader('This should not fill immediately. It needs a Read Method.')
Thanks a bunch for the help.
If all you want to do is use a DataReader with SqlBulkCopy you could use the ACE drivers which comes with Office 2007/2010 and is also available as a separate download to open an OLEDB connection to to CSV file, open a reader and call WriteToServer
$ServerInstance = "$env:computername\sql1"
$Database = "tempdb"
$tableName = "psdrive"
$ConnectionString = "Server={0};Database={1};Integrated Security=True;" -f $ServerInstance,$Database
$filepath = "C:\Users\Public\bin\"
get-psdrive | export-csv ./psdrive.csv -NoTypeInformation -Force
$connString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=`"$filepath`";Extended Properties=`"text;HDR=yes;FMT=Delimited`";"
$qry = 'select * from [psdrive.csv]'
$conn = new-object System.Data.OleDb.OleDbConnection($connString)
$conn.open()
$cmd = new-object System.Data.OleDb.OleDbCommand($qry,$conn)
$dr = $cmd.ExecuteReader()
$bulkCopy = new-object ("Data.SqlClient.SqlBulkCopy") $connectionString
$bulkCopy.DestinationTableName = $tableName
$bulkCopy.WriteToServer($dr)
$dr.Close()
$conn.Close()
#CREATE TABLE [dbo].[psdrive](
# [Used] [varchar](1000) NULL,
# [Free] [varchar](1000) NULL,
# [CurrentLocation] [varchar](1000) NULL,
# [Name] [varchar](1000) NULL,
# [Provider] [varchar](1000) NULL,
# [Root] [varchar](1000) NULL,
# [Description] [varchar](1000) NULL,
# [Credential] [varchar](1000) NULL,
# [DisplayRoot] [varchar](1000) NULL
#)
I'm importing large CSV's by a datatable and performing batch updates after 1 million rows.
if ($dt.rows.count -eq 1000000) {
$bulkCopy.WriteToServer($dt)
$dt.Clear()
}
Here is the link where I detail my own script on my blog, but the above code outlines the basic concept. My PowerShell script took 4.x minutes to import 9 million rows from a 1.1 GB CSV. The script relied on SqlBulkCopy, [System.IO.File]::OpenText and a datatable.
I am trying to convert some excel (.xlsx) and Access (.accdb) to CSV files.
I quickly found a way to do this with Excel but now I cannot find any helpful documentation on converting .accdb files.
So far I have:
$adOpenStatic = 3
$adLockOptimistic = 3
$objConnection = New-Object -com "ADODB.Connection"
$objRecordSet = New-Object -com "ADODB.Recordset"
$objConnection.Open("Provider = Microsoft.ACE.OLEDB.12.0; Data Source = " + $Filepath)
$objRecordset.Open("Select * From TableName",$objConnection,$adOpenStatic, $adLockOptimistic)
#Here I need some way to either saveas .csv or loop through
#each row and pass to csv.
$objRecordSet.Close()
$objConnection.Close()
Any Ideas?
I would be willing to do this with another language (VB, Java, PHP) if anyone knows a way.
If you use .NET rather than COM it's a lot easier. Here's some code to handle the Excel XLSX files
#Even /w Excel 2010 installed, needed to install ACE:
#http://www.microsoft.com/downloads/en/details.aspx?FamilyID=c06b8369-60dd-4b64-a44b-84b371ede16d&displaylang=en
#Becareful about executing in "right" version x86 vs. x64
#Change these settings as needed
$filepath = 'C:\Users\u00\Documents\backupset.xlsx'
#Comment/Uncomment connection string based on version
#Connection String for Excel 2007:
$connString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=`"$filepath`";Extended Properties=`"Excel 12.0 Xml;HDR=YES`";"
#Connection String for Excel 2003:
#$connString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=`"$filepath`";Extended Properties=`"Excel 8.0;HDR=Yes;IMEX=1`";"
$qry = 'select * from [backupset$]'
$conn = new-object System.Data.OleDb.OleDbConnection($connString)
$conn.open()
$cmd = new-object System.Data.OleDb.OleDbCommand($qry,$conn)
$da = new-object System.Data.OleDb.OleDbDataAdapter($cmd)
$dt = new-object System.Data.dataTable
[void]$da.fill($dt)
$conn.close()
$dt | export-csv ./test.csv -NoTypeInformation
If you want to stick with ADODB COM object:
# loop through all records - do work on each record to convert it to CSV
$objRecordset.Open("Select * FROM Tablename", $objConnection,$adOpenStatic,$adLockOptimistic)
$objRecordset.MoveFirst()
do {
# do your work to get each field and convert this item to CSV
# fields available thru: $objRecordset.Fields['fieldname'].Value
$objRecordset.MoveNext()
} while ($objRecordset.EOF -eq $false)
$objRecordset.Close()