is varchar(n) saving memory space in postgresql? [closed] - postgresql

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
If I define a varchar(25) and my string is less than 25 characters (e.g. 12 chars), SQL shows the vector of char as a length of 12 and no trailing spaces are added (unlike character does).
My question is the following : behind the fact that SQL displays just the string as it was inserted in the field (or truncated if longer than the maximum length), how Postgresql is storing such a data type?
Is it padding with extra bytes as :
twelvecharxx............. (length : 25)
or just storing 12 bytes?
I assume this could be more complex internally. I just need to know if the maximum-length optional argument is a safety to disallow large string storage or just a performance question (regarding if all the subsequence stored strings are expected to be less or equal to 25 characters).

SQL defines two primary character types: character varying(n) and
character(n), where n is a positive integer. Both of these types can
store strings up to n characters in length. An attempt to store a
longer string into a column of these types will result in an error,
unless the excess characters are all spaces, in which case the string
will be truncated to the maximum length. (This somewhat bizarre
exception is required by the SQL standard.) If the string to be stored
is shorter than the declared length, values of type character will be
space-padded; values of type character varying will simply store the
shorter string.
and
The storage requirement for a short string (up to 126 bytes) is 1 byte
plus the actual string, which includes the space padding in the case
of character. Longer strings have 4 bytes overhead instead of 1.
finally
Tip: There are no performance differences between these three types,
apart from increased storage size when using the blank-padded type,
and a few extra cycles to check the length when storing into a
length-constrained column. While character(n) has performance
advantages in some other database systems, it has no such advantages
in PostgreSQL.
from here

No padding is stored for varchar, according to the documentation. For character it is, however. There's also some overhead in storing the length of the string.

Related

Does it waste space defining a column in IBM Db2 on Cloud a longer VARCHAR() than required?

We often have columns that can contain values of varying sizes. For these, I like to set the data type to VARCHAR with a size way beyond the current maximum length. For example, if I have a column where the current minimum length for a value is 10 and the maximum length is 35, I might set the data type to VARCHAR(64). My rationale is that Db2 stores the 2 byte length followed by the exact value, therefore, there is no difference, from a storage perspective, defining the data type as VARCHAR(64) instead of VARCHAR(35). And I don't get an error if I a value with a length of 36 comes along.
Is there a nuance that I'm missing and should I not be so glib about my VARCHAR assignments?
The exact formula to calculate row length is described in the docs for CREATE TABLE. VARCHAR(64) or VARCHAR(35) should not make a difference.
Be aware that rows a stored in data pages in tablespaces. Database systems usually pre-allocate pages for performance reasons. Moreover, pages might not be fully filled or there is compression. And you might have defined indexes which require their own pages with structures. Plus there is metadata in the system catalog.

What will be the change in Application behavior after altering table with CODEUNITS32 for supporting unicode behavior?

we are in a phase of migration of some tables from AS400 DB to DB2 LUW(V11.1).
While migrating we found some special character(€) in the source database(AS400)- (Column with CHAR) and that lead to error if we are unable to alter table column with CODEUNITS32, DB2 LUW Database configuration Byte Encoding Set at UTF-8.
We want to understand, what would be the behavior of the application after changing the char column to CODEUNITS32, Do I need to update any Configuration at the application level (C & Java Application) to handle both Character Encoding Set?
After changing to CODEUNITS32
- My C application able to compile and able to handle change in Character byte from 8 bit per character(UTF-8) to 4 Byte Per Character(CODEUNITS32)?
- My Java application is able to handle change in Character byte from 8 bit per character(UTF-8) to 4 Byte Per Character(CODEUNITS32)?
We did some pilot testing by inserting Special character manually to the table after setting column definition to CODEUNITS32 from CHAR and testing was successful.
Using a string units specification of CODEUNITS32 for a column does not change the encoding of a column, the data is still stored in UTF-8 for CHAR/VARCHAR columns.
It alters the physical length (CHAR) or max length (VARCHAR) of the column by a factor of 4.
It also enables "character semantics" in some functions such as SUBSTR(), such that they work on characters, not bytes when processing CODEUNITS32 columns. (SUBSTRING() will always use character semantics (unless processing a FOR BIT DATA column))
So a CHAR(4) is CHAR(4 OCTETS) is 4 bytes long, and can hold at most 4 characters if they are all single byte in UTF-8. For € which is 3 bytes long, it could only hold say €4 but not €42
ACHAR(4 CODEUNTIS32) is 16 bytes long, and is allowed to hold up to 4 characters. It could hold €€€€ but not €2345
It is worth considering avoiding CHAR(x CODEUNITS32) and prefering VARCHAR(x CODEUNITS32). UTF-8 does not really play well with fixed width data types. The more common UTF-8 characters are 1 or 2 bytes long, so typically a CHAR(x CODEUNITS32) column will hold be more than 50% space padding.
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.sql.ref.doc/doc/r0008470.html
CODEUNITS32
Indicates that the units for the length attribute are Unicode UTF-32 code units which approximate counting in characters.
This unit of length does not affect the underlying code page of the data type.
The actual length of a data value is determined by counting the UTF-32
code units as if the data was converted to UTF-32.
A string unit of CODEUNITS32 can be used only in a Unicode database.
CODEUNITS32 can be
explicitly specified or determined based on an environment setting.
Also, out of interest, GRAPHIC/VARGRAPHIC and columns are stored in UTF-16, and default to CODEUNITS16, but can also use CODEUNITS32.
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.sql.ref.doc/doc/r0008471.html

What is the difference between decimal and numeric in Postgres?

I'm changing a column in the database from "money" to "numeric" per some previous advice here.
Looking through the data types in postgres -- https://www.postgresql.org/docs/current/static/datatype-numeric.html -- I can't see any differences between numeric and decimal in the descriptions.
What is the difference between decimal and numeric, and is there any reason I should use numeric instead of decimal for prices in my database?
According to the manual they are the same.
The types decimal and numeric are equivalent. Both types are part of
the SQL standard.
https://www.postgresql.org/docs/current/static/datatype-numeric.html
The difference lies in the SQL standard which allows for different behaviour:
NUMERIC must be exactly as precise as it is defined — so if you define 4 decimal places, the DB must always store 4 decimal places.
DECIMAL must be at least as precise as it is defined. This means that the database can actually store more digits than specified (due to the behind-the-scenes storage having space for extra digits). This means the database might store 1.00005 instead of 1.0000, affecting future calculations.
Difference between DECIMAL and NUMERIC

Comparison of PostgreSQL text types

I'm migrating from MySQL to PostgreSQL because Oracle. There is a great MySQL text type reference, here is the relevant information for MySQL...
CHAR( ) A fixed section from 0 to 255 characters long.
VARCHAR( ) A variable section from 0 to 255 characters long.
TINYTEXT A string with a maximum length of 255 characters.
TEXT A string with a maximum length of 65535 characters.
BLOB A string with a maximum length of 65535 characters.
MEDIUMTEXT A string with a maximum length of 16777215 characters.
MEDIUMBLOB A string with a maximum length of 16777215 characters.
LONGTEXT A string with a maximum length of 4294967295 characters.
LONGBLOB A string with a maximum length of 4294967295 characters.
PostgreSQL seems a bit different, there is a text type looking through phppgAdmin, not sure what else there is and I'm not finding any good comparison tables.
What are all the available text types in PostgreSQL?
PostgreSQL has more advanced types but doesn't need the distinction between text sizes.
There are 3 string types in PostgreSQL and a binary type:
text
Just a text object with a non-specified size. You can put anything in here and it will be stored. Size doesn't matter.
varchar(n) / character varying(n)
Basically a text which has a size check, there is virtually no (except for checking the size while inserting) performance difference here.
char(n) / character(n)
Just a text where all the extra characters will be padded with space characters so you always get n characters back.
bytea
The blob type you've mentioned is a totally different type alltogether. You could replace it with the bytea type: http://www.postgresql.org/docs/9.3/static/datatype-binary.html
Source: http://www.postgresql.org/docs/9.3/static/datatype-character.html

SQL Server 2008 R2 : detect encoding in nvarchar field

I have a 1,000,000 rows plus string table, that has some garbage inside due to encoding errors.
The garbage is minimal, but needs to be found.
The column in question is a NVARCHAR column that normally contains text in one of 11 languages.
All of the text should be unicode (utf-8 when we process it application side).
The corrupt columns contain ? characters and or a very limited unusual glyph set, by eye they can be very easily seen not to be valid language. It is likely that these columns have been encoded backwards and forwards into total garbage.
So in the name of speed, is there anything I can do on SQL Server to detect bad encoding / string garbage?
Thanks.
EDIT to add garbage example:
This was Russian или Ð˜Ð¼Ñ Ð£Ñ‡Ð°Ñтника