I got a table where the data may contain null between the characters. As I have already defined the table as VARCHAR, it throws me an error
org.postgresql.util.PSQLException: ERROR: invalid byte sequence for
encoding "UTF8": 0x00
There should be a way where I can insert a null based string in postgres.
This is the sample insert that has failed while inserting onto postgres
private void postGrestest() throws ClassNotFoundException, SQLException
{
Class.forName("org.postgresql.Driver");
String dropStmt = "DROP TABLE PUBLIC.TEST";
String createStmt = "CREATE TABLE PUBLIC.TEST(COL1 VARCHAR(50), COL2 BOOLEAN)";
String insertStmt = "INSERT INTO PUBLIC.TEST VALUES (?, ?)";
try (Connection connection = DriverManager.getConnection(
"jdbc:postgresql://url:5432/objectserver?stringtype=unspecified",
"username", "password");
Statement stmt = connection.createStatement();
PreparedStatement ps = connection.prepareStatement(insertStmt);)
{
stmt.execute(dropStmt);
stmt.execute(createStmt);
Random r = new Random();
for (int i = 0; i < 100; i++)
{
Object str = "Test" + i;
str = ((String) str).replace('s', '\0');
logger.info("Inserting " + str);
// str = ((String) str).replace("\0", "");
ps.setObject(1, str);
Object obj = String.valueOf(r.nextBoolean());
ps.setObject(2, obj);
ps.executeUpdate();
}
}
}
Are there any considerations before dealing with this type of data? This data is a string based one where the source may contain data containing null between them. This is handled well on a different database instance SQL Server using NVARCHAR.
You can't include a null in a string in PostgreSQL. From the documentation:
The character with the code zero cannot be in a string constant.
Java uses a slightly modified Unicode scheme where U+0000 can be encoded as 0xC0 0x80, a two-byte encoding. You might replace these values in the string rather than a binary null. PostgreSQL will gladly ingest it.
Related
I have been breaking my head over the past few hours at trying to figure out what's wrong with my code. This piece of code was all working fine until i received a file which had japanese characters in it. Notepad++ and even some online utility tools say that the encoding of the file is UTF-8. Notepad says its UTF-8-BOM.
I have read my data from the file and I have processed it and finally want to write it out to the database.
I get the error org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0xee
My database encoding is UTF8 only..
package citynet.dta.pump;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import org.postgresql.copy.CopyManager;
import org.postgresql.core.BaseConnection;
import citynet.common.ServerException;
public class TestEncoding {
public static void main(String[] args) {
byte[] bytes = null;
try {
//use the below sql to create table 'testtable'
// create table testtable (text1 character varying, text2 character varying,text3 character varying)
try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
DataOutputStream out = new DataOutputStream(baos);
out.writeBytes("INR,字仮名交じり文,3255104BTK1");
bytes = baos.toByteArray();
}
Class.forName("org.postgresql.Driver");
Connection c = DriverManager.getConnection("jdbc:postgresql://server:5432/dbname", "username", "password");
if (bytes != null) {
try (ByteArrayInputStream input = new ByteArrayInputStream(bytes)) {
String sql = "COPY testtable FROM stdin delimiter ',' NULL AS 'null' ENCODING 'UTF8' ";
BaseConnection pgcon = (BaseConnection) c;
CopyManager mgr = new CopyManager(pgcon);
try {
mgr.copyIn(sql, input);
} catch (SQLException ex) {
throw new ServerException("Error while copying data in Postgres DB:" + ex);
}
}
}
} catch (Exception e) {
System.out.println("Error:" + e);
}
}
}
The issue is DataOutputStream#writeBytes("INR,字仮名交じり文,3255104BTK1") is not doing what you expect.
You should refrain from using BaseConnection as it is an internal class. Application code should use PGConnection
Here's how you get CopyManager:
Connection con = ...;
PGConnection pgcon = con.unwrap(org.postgresql.PGConnection.class);
CopyManager mgr = pgcon.getCopyAPI();
The source of your data might be different so there are multiple ways to execute copyAPI.
If you want to convert String to UTF-8 bytes via your own code, then you need getBytes.
String sql = "COPY testtable FROM stdin delimiter ',' NULL AS 'null' ENCODING 'UTF8' ";
byte[] bytes = "INR,字仮名交じり文,3255104BTK1".getBytes(StandardCharsets.UTF_8);
mgr.copyIn(sql, new ByteArrayInputStream(bytes));
Note: there's no need to close ByteArrayInputStream (see its Javadoc).
If you need to stream a CSV file to the database, you might use FileInputStream:
try (InputStream fis = new FileInputStream("file.csv")) {
mgr.copyIn(sql, fis);
}
If you want to build the contents incrementally, then you might use ByteArrayOutputStream + OutputStreamWriter
Note: all the rows would need to fit in the memory otherwise you get OutOfMemoryError.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (OutputStreamWriter wr = new OutputStreamWriter(baos, StandardCharsets.UTF_8)) {
// Write 10 rows
for (int i = 0; i < 10; i++) {
wr.write("INR,字仮名交じり文,3255104BTK1\n");
}
}
String sql = "COPY testtable FROM stdin delimiter ',' NULL AS 'null' ENCODING 'UTF8'";
mgr.copyIn(sql, new ByteArrayInputStream(baos.toByteArray()));
An alternative option is to use Reader
Note: encoding is not specified, and it is using connection-default encoding (which is utf-8 in 99.42% of the cases since the driver defaults to utf-8 connection encoding).
String sql = "COPY testtable FROM stdin delimiter ',' NULL AS 'null'";
mgr.copyIn(sql, new StringReader("INR,字仮名交じり文,3255104BTK1"));
Yet another alternative is to use copyIn(String sql, ByteStreamWriter from) API which might be more efficient for certain use-cases (e.g. all the data is in-memory, and you know the number of bytes you are going to write)
I tried to call my user defined function in pgresql from C# code,
my function creation script is as follows,
CREATE OR REPLACE FUNCTION public."GetUserDailyData"(
cid integer,
hid integer,
currday integer)
RETURNS character varying AS
$BODY$
DECLARE
returndata varchar = '';
BEGIN
SELECT data->20+currday into returndata FROM pops
WHERE hybid = hid and cropid = cid;
return returndata;
END
$BODY$
LANGUAGE plpgsql
COST 100;
My method to call this function is as follows,
public static object ExecuteScalar(string conString, string spName, NpgsqlParameter[] param)
{
using (var conn = new NpgsqlConnection(conString))
{
conn.Open();
using (var tran = conn.BeginTransaction())
using (var command = conn.CreateCommand())
{
command.CommandText = spName;
command.CommandType = CommandType.StoredProcedure;
for (var i = 0; i < param.Length; i++)
{
command.Parameters.Add(new NpgsqlParameter());
command.Parameters[i] = param[i];
}
var result = command.ExecuteScalar();
return result;
}
}
}
I tried everything even checked the existence of this function in pg_proc using
select * from pg_proc where proname = 'GetUserDailyData'
and it reflected the function details row.
But every time it is giving the same error.
Any kind of suggestion would be highly appreciated. Thanks.
Adding objects with case sensitive names in PostgreSQL can lead to these complications; in this case you need to specify the name of the stored procedure between quotes, however it would be advisable to simply not create any objects that rely on case sensitivity, use underscores instead, or when create/refer to objects using CamelCase without the quotes (which creates/refers to the objects in low-caps). In any case, you may also need to specify the whole interface (not just the name) as the CommandText, and specify the data types of the parameters (see this).
...
command.CommandText = "\"" + spName + "\"";
...
I have now facing the problem in bytea to Base64, actually I have save the image in below query,
user_profile_pic is defind in bytea in table
Update user_profile_pic
Set user_profile_pic = (profilepic::bytea)
Where userid = userid;
after that I have select the below query,
case 1:
SELECT user_profile_pic
FROM user_profile_pic;
its return exact same as I have updated, but after passing service its display a byte format
case 2:
Select encode(user_profile_pic::bytea, 'base64')
FROM user_profile_pic;
it returns totally different result.
I want to result case 1 along with service?
its working for me, not working query if write procedure/function, i write direct code behind
conn.Open();
NpgsqlCommand command = new NpgsqlCommand("SELECT profile_pic FROM userlog WHERE cust_id = '" + CustID + "'", conn);
Byte[] result = (Byte[])command.ExecuteScalar();
if(result.Length > 0)
{
ProfilePicture = Convert.ToBase64String(result);
ErrorNumber = 0;
ErrorMessage = "Successful operation";
}
else
{
ErrorNumber = 1;
}
conn.Close();
I am using NpgSQL with PostgreSQL and ADO.NET. Forgive the simplicity of the question as I just started using PostgreSQL and NpgSQL this week.
Something like this works fine:
[Test]
public void InsertNoParameters()
{
NpgsqlConnection conn = new NpgsqlConnection("Host=localhost; Database=postgres; User ID=postgres; Password=password");
conn.Open();
IDbCommand command = conn.CreateCommand();
string sql = "INSERT INTO Customers (FirstName,LastName) VALUES ('Test','Tube')";
command.CommandText = sql;
command.ExecuteNonQuery();
conn.Close();
}
When I put in parameters I get the error message:
Npgsql.NpgsqlException : ERROR: 42703: column "_firstname" does not exist
[Test]
public void InsertWithParameters()
{
NpgsqlConnection conn = new NpgsqlConnection("Host=localhost; Database=postgres; User ID=postgres; Password=password");
conn.Open();
IDbCommand command = conn.CreateCommand();
string sql = "INSERT INTO Customers (FirstName,LastName) VALUES (_FirstName,_LastName)";
command.CommandText = sql;
var parameter = command.CreateParameter();
parameter.ParameterName = "_FirstName";
parameter.Value = "Test";
command.Parameters.Add(parameter);
parameter = command.CreateParameter();
parameter.ParameterName = "_LastName";
parameter.Value = "Tube";
command.Parameters.Add(parameter);
command.ExecuteNonQuery();
conn.Close();
}
The responses in the comments are correct:
Npgsql doesn't support _ as a parameter placeholder notation. You should be using # or : (so #FirstName or :FirstName, not _FirstName).
PostgreSQL will automatically lower-case your table and column names unless they are double-quoted. Either use lower-case names for everything (simpler) or quote identifiers in your SQL queries.
So your code should look more or less like this:
IDbCommand command = conn.CreateCommand();
string sql = "INSERT INTO Customers (first_name, last_name) VALUES (#FirstName,#LastName)";
command.CommandText = sql;
var parameter = command.CreateParameter();
parameter.ParameterName = "FirstName";
parameter.Value = "Test";
command.Parameters.Add(parameter);
I'm trying to copy data from a MySQL database to an equivalent PostgreSQL database, doing a load/insert through Java using JDBC. I get this error whenever I try to copy an enum column:
org.postgresql.util.PSQLException: ERROR: column "mycol" is of type mytable_mycol_enum but expression is of type character varying
Hint: You will need to rewrite or cast the expression.
Position: 194
The Java code (simplified and anonymized):
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection mysqlConn = DriverManager.getConnection(
"jdbc:mysql://localhost/mysqldb", "user", "pass");
Class.forName("org.postgresql.Driver").newInstance();
Connection pgConn = DriverManager.getConnection(
"jdbc:postgresql://othercomp/pgdb", "user", "pass");
Statement selStatement = mysqlConn.createStatement();
ResultSet selSet = selStatement.executeQuery("SELECT * FROM mytable");
Statement insStatement = pgConn.createStatement(
ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_UPDATABLE);
Resultset insSet = insStatement.executeQuery("SELECT * FROM mytable WHERE 0=1");
insSet.moveToInsertRow();
while(selSet.next())
{
ResultSetMetaData metaData = selSet.getMetaData();
for (int i = 1; i <= metaData.getColumnCount(); i++)
{
String colName = metaData.getColumnName(i);
Object obj = selSet.getObject(colName);
insertSet.updateObject(colName, obj);
}
insertSet.insertRow();
}
This works fine for tables without an enum column, but
In MySQL I have a column mycol of type enum('A','B','C')
In PostgreSQL I have mycol of type mytable_mycol_enum with CREATE TYPE mytable_mycol_enum AS ENUM ('A','B','C')
How do I set the enum value?
You have to cast the value, so that postgresql understands it:
insert into mytable (mycol) values ('A'::mytable_mycol_enum)
You might have to use a named insert statement, not the generic one in your example. Also, try it in psql first to see if it works.
HTH