In order to hide the output of DBCC SHRINKFILE I tried:
CREATE TABLE #Swallow(DbId int, Fileld int, CurrentSize int, MininumSize int, UsedPages int, EstimatedPages int)
INSERT INTO #Swallow EXEC('DBCC SHRINKFILE(''filename'', 0, TRUNCATEONLY)')
but it returns the following error:
Cannot perform a shrinkfile operation inside a user transaction. Terminate the transaction and reissue the statement.
How can this be done?
It seems there's a WITH NO_INFOMSGS option:
DBCC SHRINKFILE('filename', 0, TRUNCATEONLY) WITH NO_INFOMSGS
Ref: http://ss64.com/sql/dbcc_shrinkfile.html
Related
Is there a way in KsqlDB to add headers while creating a stream from AS SELECT?
For example, I have a stream DomainOrgs(DomainId INT,OrgId INT,someId INT), now I need to create a stream with all the values in DomainOrgs also DomainId should go to Header
I tried to create like
CREATE STREAM DomainOrgs_with_header AS
SELECT DomainId,
OrgId,
someId,
DomainId HEADER('DomainId')
FROM DomainOrgs
EMIT CHANGES;
Also tried
CREATE STREAM DomainOrgs_with_header
(
DomainId INT,
OrgId INT,
someId INT,
DomainId_Header Header('DomainId')
)
INSERT INTO DomainOrgs_with_header
SELECT DomainId,OrgId,someId,DomainId FROM DomainOrgs
Here, stream will create but INSERT INTO will fail.
Is there any way to select data into the stream with headers?
I am trying to create a conditional insert into a table with responses for an event. The event may have a limit on how many responses/attendees it can support so to prevent overbooking I want to check the status before I insert a new response.
The tables match the database models I have (IDs are generated by DB/auto inc):
case class Event(id: Option[Int], name: String, limitedCapacity: Option[Int])
case class Response(id: Option[Int], eventId: Int, email: String)
I have constructed a SQL statement that describes my conditional insert (eg, only insert if event has no limited capacity or if number of responses are less than the limit) that looks like this:
INSERT INTO responses(eventId, email)
SELECT :eventId, :email
FROM events
WHERE id = :eventId
AND (limitedCapacity IS NULL
OR (SELECT COUNT(1) FROM responses WHERE eventId = :eventId) < limitedCapacity);
but I don't know how to translate that into Slick DSL that also returns the inserted row. I am using PostgreSQL so I know return-row-on-insert is possible for normal insertions at least.
Here is some code that shows what I am after but I want this as a single transaction:
def create(response: Response): Future[Option[Response]] = {
val event = db.run(events.filter(_.id === response.eventId).result.head)
if (event.limitedCapacity.isEmpty) {
db.run((responses returning responses) += response).map(Some(_))
}
else {
val responseCount = db.run(responses.filter(_.eventId === response.id).length.result)
if (responseCount < event.limitedCapacity.get) {
db.run((responses returning responses) += response).map(Some(_))
}
else {
Future.sucessful(None)
}
}
}
If it is not possible to return the inserted row that is fine but I need some kind of confirmation of the insertion at least.
Introduction:
I'm trying to insert data with Python/psycopg2 into Postgres in the following format:
(integer, date, integer, customtype[], customtype[], customtype[], customtype[])
However, as I try to insert them, I always get this error:
'"customtype[]" does not exist'
How is my setup:
I have a dict with the data I need, like so:
data_dict = {'integer1':1, 'date': datetime(),
'integer2': 2, 'custom1':[(str, double, double),(str, double, double)],
'custom2':[(str, double, double),(str, double, double),(str, double, double)],
'custom3':[(str, double, double),(str, double, double),(str, double, double)],
'custom4':[(str, double, double)]}
Each custom array can have as many custom tuples as needed.
I've already created a type for these custom tuples, as such:
"CREATE TYPE customtype AS (text, double precision, double precision)"
And I've created a table with columns of customtype[].
What I've tried so far:
query = """INSERT INTO table (column names...) VALUES
(%(integer1)s, %(date)s, %(integer2)s,
%(custom1)s::customtype[], [...]);"""
And:
query = """INSERT INTO table (column names...) VALUES
(%(integer1)s, %(date)s, %(integer2)s,
CAST(%(custom1)s AS customtype[]), [...]);"""
But both options render the same results.
The final question:
How to insert these record-type arrays in Postgresql with Psycopg2?
Maybe I'm probably misunderstanding completely how Postgresql works. I'm comming from a BigQuery Record/Repeated type background.
Ps.: This is how I'm querying:
cursor.execute(query,data_dict)
The problem is that I created the type inside the Database.
When referencing custom types in Postgresql, there's a need to reference the database where the type was created as well as the type.
Like so:
(%(dict_name)s)::"database_name".type
#or
CAST(%(dict_name)s as "database_name".type)
Be carefull with the quoting!
This is a bit of an old question and it set me on the right track to finding the answer but didn't quite get me all the way.
I had a function which needed to return a custom type. I needed to attach public before the custom type.
RETURNS TABLE (
"status" "public"."my_custom_status_type",
"ID" VARCHAR(255)
)
So directly answering OP's question, try adding public before the type.
query = """INSERT INTO table (column names...) VALUES
(%(integer1)s, %(date)s, %(integer2)s,
CAST(%(custom1)s AS public.customtype[]), [...]);"""
Input dataset was generated by pyspark below command,
dfs_ids1.write.parquet('./outputs-parquet/i94-apr16.parquet', mode='overwrite')
COPY statement & error message are,
COPY staging_ids FROM 's3://sushanth-dend-capstone-files/i94-apr16.parquet/' IAM_ROLE 'arn:aws:iam::164084742828:role/dwhRole' FORMAT AS PARQUET ;
S3 Query Exception (Fetch)
DETAIL:
-----------------------------------------------
error: S3 Query Exception (Fetch)
code: 15001
context: Task failed due to an internal error. Unmatched number of columns between table and file. Table columns: 23, Data columns: 22, File name: https://s3.us-west-2.amazonaws.com/sushanth-dend-capstone-files/i94-apr16.parquet/part-00000-6034cb60-860e-4a6c-a86d-6
query: 1867
location: dory_util.cpp:1119
process: fetchtask_thread [pid=1331]
-----------------------------------------------
When checking the parquet using S3 "Select from", found that for some JSON sets, column occup is missing, this specific column mostly contains "null or STU".
{
"CoC": 101,
"CoR": 101,
"PoE": "BOS",
"landing_state": "MA",
"age": 58,
"visa_issued_in": "TIA",
"biryear": 1958,
"gender": "M",
"airline": "LH",
"admnum": 2147483647,
"fltno": "00422",
"visatype": "B1",
"arrival_mode": "Air",
"visit_purpose": "Business",
"arrival_dt": "2016-04-01",
"departure_dt": "2016-04-05",
"DaysinUS": 4,
"added_to_i94": "2016-04-01",
"allowed_until": "2016-04-06",
"entry_exit": "exit",
"month": 4
}
To verify this column is missing, i read this parquet file in Athena using below parameters,
S3 Path : s3://sushanth-dend-capstone-files/i94-apr16.parquet/
Bulk Load of columns : CoC int, CoR int, PoE string, landing_state string, age int, visa_issued_in string, occup string, biryear int, gender string, airline string, admnum int, fltno string, visatype string, arrival_mode string, visit_purpose string, arrival_dt date, departure_dt date, DaysinUS int, added_to_i94 date, allowed_until date, entry_exit string, month int
Athena DDL
CREATE EXTERNAL TABLE IF NOT EXISTS capstone.staging_ids (
`coc` int,
`cor` int,
`poe` string,
`landing_state` string,
`age` int,
`visa_issued_in` string,
`occup` string,
`biryear` int,
`gender` string,
`airline` string,
`admnum` int,
`fltno` string,
`visatype` string,
`arrival_mode` string,
`visit_purpose` string,
`arrival_dt` date,
`departure_dt` date,
`daysinus` int,
`added_to_i94` date,
`allowed_until` date,
`entry_exit` string,
`month` int
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1'
) LOCATION 's3://sushanth-dend-capstone-files/i94-apr16.parquet/'
TBLPROPERTIES ('has_encrypted_data'='false');
When i run the query i could see column 'occup' with STU for some rows.
Question is,
How to write all the columns from spark dataframe in parquet format
file ?
How to load this sort of file into redshift via COPY statement, is parquet not the right format to load into redshift ?
I have the objective of breaking out the results of a query on a table with a json column that contains an array into individual rows. However I'm not sure about the syntax to write this query. I'm using this:
For the following query
SELECT
jobs.id,
templates.Id,
templates.Version,
templates.StepGroupId,
templates.PublicVersion,
templates.PlannedDataSheetIds,
templates.SnapshottedDataSheetValues
FROM jobs,
jsonb_to_recordset(jobs.source_templates) AS templates(Id, Version, StepGroupId, PublicVersion,
PlannedDataSheetIds, SnapshottedDataSheetValues)
On the following table:
create table jobs
(
id uuid default uuid_generate_v4() not null
constraint jobs_pkey
primary key,
source_templates jsonb,
);
with the jsonb column containing data in this format:
[
{
"Id":"94729e08-7d5c-459d-9244-f66e17059fc4",
"Version":1,
"StepGroupId":"0274590b-c08d-4963-b37e-8fc8f25151d2",
"PublicVersion":1,
"PlannedDataSheetIds":null,
"SnapshottedDataSheetValues":null
},
{
"Id":"66791bfd-8cdb-43f7-92e6-bfb45b0f780f",
"Version":4,
"StepGroupId":"126404c5-ed1e-4796-80b1-ca68ad486682",
"PublicVersion":1,
"PlannedDataSheetIds":null,
"SnapshottedDataSheetValues":null
},
{
"Id":"e3b31b98-8052-40dd-9405-c316b9c62942",
"Version":4,
"StepGroupId":"bc6a9dd3-d527-449e-bb36-39f03eaf87b9",
"PublicVersion":1,
"PlannedDataSheetIds":null,
"SnapshottedDataSheetValues":null
}
]
I get an error:
[42601] ERROR: a column definition list is required for functions returning "record"
What is the right way to do this without generating the error?
You need to define datatypes:
SELECT
jobs.id,
templates.Id,
templates.Version,
templates.StepGroupId,
templates.PublicVersion,
templates.PlannedDataSheetIds,
templates.SnapshottedDataSheetValues
FROM jobs,
jsonb_to_recordset(jobs.source_templates)
AS templates(Id UUID, Version INT, StepGroupId UUID, PublicVersion INT,
PlannedDataSheetIds INT, SnapshottedDataSheetValues INT)