I am pushing data from a SQL query on cloud to db2 on cloud, when I am querying data and storing it in my s3 bucket it is saving jobid of that particular query result.
But when I am pushing data into db2 it is not saving any jobid, in fact data has been inserted. I am checking by going again on db2.
But how do I came to know on SQL query that my query has succeeded or not? I want to confirm it when I am running the SQL query on cloud.
My query:
select
a.col, a.col2, explode (a.col3) company_list
from
cos://us-east/akshay-test-bucket1/Receive-2020-03-11/CP-2020-03-11/ STORED AS PARQUET c
into
crn:v1:bluemix:public:dashdb-for-transactions:eu-gb:a/2c36cd86085f442b915f0fba63138e0c:61f353e4-6640-4599-b1dd-48ee52ee008d::/schema_name.table_name
Here I am storing data into db2 and SQL query is saying "A preview is only possible for results stored in Cloud Object Storage" as you can see in above screenshot and see my query.
Related
In Azure Data Factory, I'm doing a pretty vanilla 'Copy Data' operation. One dataset to another.
Can I view the query being used to perform the copy operation? Apparently, it's a syntax error, but I've only used drag-and-drop menus. Here's the error:
ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A
database operation failed. Please search error to get more
details.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=Incorrect
syntax near the keyword 'from'.,Source=.Net SqlClient Data
Provider,SqlErrorNumber=156,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=156,State=1,Message=Incorrect
syntax near the keyword 'from'.,},],'
Extra context
1.Clear Schema and import Schema again.
2. This mostly happens when there are some changes to table schema after creating pipeline and datasets. verify once.
3.The schema and datasets should be refreshed when there are some changes in the SQL table schema.
4. For table name or view name used in the query, use []. ex: [dbo].[persons]
5. In datasets select table name
6. try to publish before testing.
I followed the same scenario and reproduced the same thing. I didn’t get any error as you can see, the above error mainly happens because of schema.
Source dataset:
In source dataset, I manually added and connected to SQL database with Sample SQL table.
Sink dataset:
In sink dataset, I added another SQL database with auto create table, write behavior: Upset, key column: PersonID.
Before execution there is no table on SQL database, then after the execution was successful, I got this sample output in azure SQL database.
Wanted to check if Large Object replication is supported by AWS DMS when Source and destination DB's are PostgreSQL?
I just used pglogical to replicate a DB which has Large Object (Like IOD's etc) and the target DB does not have LO's.
When I query a table on the destination which uses a OID column:
select id, lo_get(json) from table_1 where id=998877;
ERROR: large object 6698726 does not exist
The json column is oid datatype
If AWS DMS takes care of it, I will start using it.
Thanks
i have two database residing on same server , say I have db1 and db2 , now i want to access some tables from db1 from db2 , how can I do this in Sql Server-12?
If you're using Azure SQL database(single database), USE statement is not supported. We can use Elastic query to achieve across database query.
Ref: Azure SQL Database elastic query overview:
The elastic query feature (in preview) enables you to run a
Transact-SQL query that spans multiple databases in Azure SQL
Database. It allows you to perform cross-database queries to access
remote tables, and to connect Microsoft and third-party tools (Excel,
Power BI, Tableau, etc.) to query across data tiers with multiple
databases. Using this feature, you can scale out queries to large
data tiers and visualize the results in business intelligence (BI)
reports.
If you're using Azure SQL managed instance, it's same with on-premise SQL Server.
for example you want to access to table[A] > schema[dbo] > database[C]
Assume current user has access to both database
SELECT * FROM [C].[dbo].[A]
Join [A] table from [schema_1] schema from [db_1] Database and [B] table from [schema_2] schema from [db_2] Database from different database on same server
When use db_1
USE [db_1]
SELECT * FROM [schema_1].[A] AS [db_1_A] INNER JOIN [db_2].[schema_2].[B] AS [db_2_B] ON [db_1_A].[field_1]=[db_2_B].[field_1]
When use db_2
USE [db_2]
SELECT * FROM [db_1].[schema_1].[A] AS [db_1_A] INNER JOIN [schema_2].[B] AS [db_2_B] ON [db_1_A].[field_1]=[db_2_B].[field_1]
After changes to some Terraform code, I can no longer access the data I've added into an Aurora (PostgreSQL) database. The data gets added into the database as expected without errors in the logs but I can't find the data after connecting to the database with AWS RDS Query Editor.
I have added thousands of rows with Python code that uses the SQLAlchemy/PostgreSQL engine object to insert a batch of rows from a mappings dictionary, like so:
if (count % batch_size) == 0:
self.engine.execute(Building.__table__.insert(), mappings)
self.session.commit()
The logs from this data ingest show no errors, the commits all appear to have completed successfully. So the data was inserted someplace, I just can't work out where that is, as it's not showing up in the AWS Console RDS Query Editor. I run the SQL below to find the table, with zero rows returned:
SELECT * FROM information_schema.tables WHERE table_name = 'buildings'
This has worked as expected before (i.e. I could see the data in the Aurora database via the Query Editor) so I'm trying to work out which of the recently modified Terraform settings have caused the issue.
Where else can I look to find where the data was inserted, assuming that it was actually inserted somewhere? If I can work that out it may help reveal the culprit.
I suspect misleading capitalization. Like "Buildings". Search again with:
SELECT * FROM information_schema.tables WHERE table_name ~* 'building';
Or:
SELECT * FROM pg_catalog.pg_tables WHERE tablename ~* 'building';
Or maybe your target wasn't a table? You can "write" to simple views. Check with:
SELECT * FROM pg_catalog.pg_class WHERE relname ~* 'building';
None of this is specific to RDS. It's the same in plain Postgres.
If the last query returns nothing, you are in the wrong database. (You are aware that there can be multiple databases in one DB cluster?) Or you have a serious problem.
See:
How to check if a table exists in a given schema
Are PostgreSQL column names case-sensitive?
Once I logged more information regarding the connection I discovered that the database name being used was incorrect, so I have been querying the Aurora instance using the wrong database name. Once I worked this out and used the correct database name the select statements in AWS RDS Query Editor worked as expected.
I have a UDF function in Cosmos DB , it takes a parameter and returns the documents that meets the condition based on the parameter.
Each document returned by this UDF has 3 fields,
Customer ID
Modified Date
Customer Status
I need this information in a SQL Server SP present in another database.
I am thinking of having a powershell script to bring this data from the Cosmos DB , store it in a table local to the SQL server database , and then use this table eventually in the SP.
I wondering if my above approach to fetch data from Cosmos DB to SQL Server database is right, and if so could I know if we can execute a cosmos DB UDF from a powershell script and use the result set returned by the UDF.
Based on your description,maybe you could use Azure Data Factory.
Step1: Follow the article to create Copy activity.
Step2: Configure Cosmos db source data:
sql:
SELECT udf.adf(c.fields).CustomerID,
udf.adf(c.fields).ModifiedDate,
udf.adf(c.fields).CustomerStatus FROM c
Then,please follow the steps from this doc:
Step 3: Configure your Sink dataset:
Step 4: Configure Sink section in copy activity as follows:
Step 5: In your database, define the table type with the same name as sqlWriterTableType. Notice that the schema of the table type should be same as the schema returned by your input data.
CREATE TYPE [dbo].[CsvType] AS TABLE(
[ID] [varchar](256) NOT NULL,
[Date] [varchar](256) NOT NULL,
[Status ] [varchar](256) NOT NULL
)
Step 6: In your database, define the stored procedure with the same name as SqlWriterStoredProcedureName. It handles input data from your specified source, and merge into the output table. Notice that the parameter name of the stored procedure should be the same as the "tableName" defined in dataset.
Create PROCEDURE convertCsv #ctest [dbo].[CsvType] READONLY
AS
BEGIN
MERGE [dbo].[adf] AS target
USING #ctest AS source
ON (1=1)
WHEN NOT MATCHED THEN
INSERT (id,data,status)
VALUES (source.ID,source.Date,source.Status );
END