I have installed apache drill and it is working fine. I am trying to fetch data from MYSQL table in postman REST APi
localhost:8047/query.json - launching properly
mysqlplugin configured successfully
From drill command line, MYSQL command working properly
apache drill> SELECT * FROM mysqlplugin.test.`table` LIMIT 10;
+----+-------+
| id | name |
+----+-------+
| 1 | anil |
| 2 | amit |
| 3 | ankur |
But when I connect REST API to localhost:8047/query.json
using body:
{
"queryType" : "SQL",
"query" : "select * from mysqlplugin.test.`table`;"
}
It gives response:
{
"errorMessage": "HTTP 405 Method Not Allowed"
}
What is the problem?
I'm trying to save a file from Azure File Storage into Azure SQL Database table varbinary(max) column (store whole content as advised in this answer). I've tried a few times to adjust my SQL query but without success. Here's the code which results in error 'Bad or inaccessible location specified in external data source "my_Azure_Files".' when it invokes OPENROWSET:
OPEN MASTER KEY DECRYPTION BY PASSWORD = 'mypassword123'
GO
CREATE DATABASE SCOPED CREDENTIAL [https://mystorageaccount.file.core.windows.net/]
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'sas_token_generated_on_azure_portal';
CREATE EXTERNAL DATA SOURCE my_Azure_Files
WITH (
LOCATION = 'https://mystorageaccount.file.core.windows.net/test',
CREDENTIAL = [https://mystorageaccount.file.core.windows.net/],
TYPE = BLOB_STORAGE
);
Insert into dbo.myTable(targetColumn)
Select BulkColumn FROM OPENROWSET(
BULK 'test.csv',
DATA_SOURCE = 'my_Azure_Files',
SINGLE_BLOB) AS testFile;
CLOSE MASTER KEY;
GO
I'm able to download the test.csv file by a web-browser using the same SAS token and url path. I'm also able to verify that the credential and the external source are successfully created in the database:
+-------------------------------------------------+------------------+-----------------------------------------------------+-------------------------+------------------+---------------------------+---------------+---------------+----------------+--------------------+----------+
| data_source_id | name | location | type_desc | type | resource_manager_location | credential_id | database_name | shard_map_name | connection_options | pushdown |
+-------------------------------------------------+------------------+-----------------------------------------------------+-------------------------+------------------+---------------------------+---------------+---------------+----------------+--------------------+----------+
| 65540 | my_Azure_Files | https://mystorageaccount.file.core.windows.net/test | BLOB_STORAGE | 05/01/1900 00:00 | NULL | 65539 | NULL | NULL | NULL | ON |
| name | principal_id | credential_id | credential_identity | create_date | modify_date | target_type | target_id | | | |
+-------------------------------------------------+------------------+-----------------------------------------------------+-------------------------+------------------+---------------------------+---------------+---------------+----------------+--------------------+----------+
| https://mystorageaccount.file.core.windows.net/ | 1 | 65539 | SHARED ACCESS SIGNATURE | 15/07/2020 13:14 | 15/07/2020 13:14 | NULL | NULL | | | |
When creating SAS on Azure portal I checked all allowed resource types and all allowed permissions, except 'Delete'. I also removed the leading '?' from SAS to use in the SECRET field.
I've tried variations of TYPE = BLOB_STORAGE and TYPE = HADOOP as well as SINGLE_BLOB, SINGLE_CLOB and SINGLE_NCLOB parameters.
Please help me solve my problem.
By following below steps, able to successfully insert into the target table:
While generating the SAS, please select Allowed Resource Type as ‘Container’ and ‘Object’:
Copy the SAS and use below command:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'password#123'
Use SAS token generated without ‘?’ at the start and create Scoped Credentials:
CREATE DATABASE SCOPED CREDENTIAL MyAzureBlobStorageCredential WITH IDENTITY =
'SHARED ACCESS SIGNATURE', SECRET = 'sv=2019-10-
10XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
Create External Data Source referencing your blob path:
CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://mystorageaccount.file.core.windows.net'
, CREDENTIAL= MyAzureBlobStorageCredential
);
Run the insert using OPENROWSET:
Insert into dbo.test(name1)
Select BulkColumn FROM OPENROWSET(
BULK 'test/test.csv',
DATA_SOURCE = 'MyAzureBlobStorage',
SINGLE_BLOB) AS testFile;
Can also use Bulk insert:
BULK INSERT dbo.test
FROM 'test/test.csv'
WITH (DATA_SOURCE = 'MyAzureBlobStorage',
FORMAT = 'CSV');
Assuming table dbo.test is already created
I am trying to create shortcuts for DBAs/admins in psql.
I have created some functions and queries, which give me information such as schema size in bytes, or a list of tables in order of storage used, etc. These will be used whenever an admin or DBA wants to get some info about the database.
I can run these queries naturally with select, or as a function like get_size(), but I want them to be accessible as shortcuts, similar to the native backslash commands (\dx, \dt, etc).
So I have used psql's \set feature to store queries/functions as variables, which I will put in the psqlrc file:
\set size 'select pg_size_pretty(my_size_function(''public''));'
Then when I type :size in psql I would get the size of the "public" schema.
What I want though, is to be able to dynamically pass a schema name, so I could run things like
:size public, :size schema2 etc.
I tried changing the \set to: \set size 'select pg_size_pretty(my_size_function(:schema));', but I can only call that by executing \set schema '''public''' first.
Since the whole point is to use these universally as shortcuts, having to manually run \set commands each time defeats the purpose.
In Oracle the would be colon a bind variable, which would be read at runtime.
How can I do this with psql?
I use these ways.
Postgres Functions (Inside Postgres CLI)
postgres=# CREATE OR REPLACE FUNCTION getSize(tableName varchar) RETURNS varchar LANGUAGE SQL as
postgres-# $$
postgres$# SELECT pg_size_pretty(pg_relation_size(tableName));
postgres$# $$;
CREATE FUNCTION
postgres=# select getSize('test');
getsize
------------
8192 bytes
(1 row)
From Shell (Outside Postgres CLI)
psqlgettablesize(schema, tablename) - Get table size. Pass all for schema argument to get for all schemas.
$ psqlgettablesize customer events
table_name | total_size | table_size | index_size
--------------------------+---------------------+-------------+---------------
test(complete database) | 19039892127 (18 GB) | |
--------- | --------- | --------- | ---------
customer.events | 24576 (24 kB) | 0 (0 bytes) | 24576 (24 kB)
(3 rows)
psqlgettablecount(schema, tablename) - Get count of rows in a table, in a schema
$ psqlgettablecount customer events
count
----------
51850000
(1 row)
psqlgetvacuumdetails(schema, tablename) - Get vacuum details of a table, in a schema
$ psqlgetvacuumdetails customer events
schemaname | relname | n_live_tup | n_dead_tup | last_analyze | analyze_count | last_autoanalyze | autoanalyze_count | last_vacuum | vacuum_count | last_autovacuum | autovacuum_count
------------+-----------+------------+------------+----------------------------+---------------+----------------------------+-------------------+---------------------------+--------------+-----------------+------------------
customer | events | 0 | 0 | 2019-12-02 18:25:04.887653 | 2 | 2019-11-27 18:49:19.002405 | 1 | 2019-11-29 13:11:15.92002 | 1 | | 0
(1 row)
psqltruncatetable(schema, tablename) - Truncate a table, in a schema, after authorization.
$ psqltruncatetable customer events
Are you sure to truncate table 'customer.events' (y/n)? y
Time: 4.944 ms
psqlsettings(category) - Get settings of Postgres
$ psqlsettings Autovacuum
name | setting | unit | category | short_desc | extra_desc | context | vartype | source | min_val | max_val | enumvals | boot_val | reset_val | sourcefile | sourceline | pending_restart
-------------------------------------+-----------+------+------------+-------------------------------------------------------------------------------------------+------------+------------+---------+---------+---------+------------+----------+-----------+-----------+------------+------------+-----------------
autovacuum | on | | Autovacuum | Starts the autovacuum subprocess. | | sighup | bool | default | | | | on | on | | | f
autovacuum_analyze_scale_factor | 0.1 | | Autovacuum | Number of tuple inserts, updates, or deletes prior to analyze as a fraction of reltuples. | | sighup | real | default | 0 | 100 | | 0.1 | 0.1 | | | f
psqlselectrows(schema, tablename) - Get rows from a table, in a schema
$ psqlselectrows customer events
id | name
----+------
1 | Clicked
2 | Page Loaded
(10 rows)
#Colors
B_BLACK='\033[1;30m'
B_RED='\033[1;31m'
B_GREEN='\033[1;32m'
B_YELLOW='\033[1;33m'
B_BLUE='\033[1;34m'
B_PURPLE='\033[1;35m'
B_CYAN='\033[1;36m'
B_WHITE='\033[1;37m'
RESET='\033[0m'
#Postgres Command With Params
psqlcommand="$POSTGRES_BIN/psql -U postgres test -q -c"
function psqlgettablesize()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing. Schema name needed. ${B_YELLOW}(Pass 'all' to get details from all schema(s)${RESET}" ||
{
criteria="and table_schema = '$1'"
if [ "$1" == "all" ]; then
criteria=""
fi
if [ "$2" != "" ]; then
criteria+=" and table_name = '$2'"
fi
[ -z "$2" ] && echo -e "${B_YELLOW}Table name not given. ${B_GREEN}Showing size of all tables in $1 schema(s)${RESET}"
query="SELECT
concat(current_database(), '(complete database)') AS table_name, concat(pg_database_size(current_database()), ' (', pg_size_pretty(pg_database_size(current_database())), ')') AS total_size, '' AS table_size, '' AS index_size
UNION ALL SELECT '---------','---------','---------','---------'
UNION ALL SELECT
table_name,
concat(total_table_size, ' (', pg_size_pretty(total_table_size), ')'),
concat(table_size, ' (', pg_size_pretty(table_size), ')'),
concat(index_size, ' (', pg_size_pretty(index_size), ')')
FROM (
SELECT
concat(table_schema, '.', table_name) AS table_name,
pg_total_relation_size(concat(table_schema, '.', table_name)) AS total_table_size,
pg_relation_size(concat(table_schema, '.', table_name)) AS table_size,
pg_indexes_size(concat(table_schema, '.', table_name)) AS index_size
FROM information_schema.tables where table_schema !~ '^pg_' AND table_schema <> 'information_schema' $criteria ORDER BY total_table_size) AS sizes";
$psqlcommand "$query"
}
}
function psqlgettablecount()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}"
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "select count(*) from $1.$2;"
}
function psqlgetvacuumdetails()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "SELECT schemaname, relname, n_live_tup, n_dead_tup, last_analyze::timestamp, analyze_count, last_autoanalyze::timestamp, autoanalyze_count, last_vacuum::timestamp, vacuum_count, last_autovacuum::timestamp, autovacuum_count FROM pg_stat_user_tables where schemaname = '$1' and relname='$2';"
}
function psqltruncatetable()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
{
read -p "$(echo -e ${B_YELLOW}"Are you sure to truncate table '$1.$2' (y/n)? "${RESET})" choice
case "$choice" in
y|Y ) $psqlcommand "TRUNCATE $1.$2;";;
n|N ) echo -e "${B_GREEN}Table '$1.$2' not truncated${RESET}";;
* ) echo -e "${B_RED}Invalid option${RESET}";;
esac
}
}
function psqlsettings()
{
query="select * from pg_settings"
if [ "$1" != "" ]; then
query="$query where category like '%$1%'"
fi
query="$query ;"
$psqlcommand "$query"
if [ -z "$1" ]; then
echo -e "${B_YELLOW}Passing Category as first argument will filter the related settings.${RESET}"
fi
}
function psqlselectrows()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "SELECT * from $1.$2"
}
I couldn't figure out how to dynamically pass variables so I settled for a system of aliases and variables that work together to provide what I was looking for. So I can run :load_schema <schemaname> and then run commands interacting with the "loaded" schema. Loading a schema is simply aliasing a \set command. This way people who are unfamiliar with psql can just press : and tab autocomplete will show them all of my shortcuts.
All my code is here.
This isn't exactly what I was looking for so I won't accept it but it's close enough to post in case anyone else wants to do this.
I'm trying to automate a TSQL select statement on a website. Each year the column names change with the number values at the end of the name increasing by 1 so instead of manually updating the site I'm trying to figure out how to include the dynamic column name in the where clause.
The data looks something like this.
+------+------+------+------+------+
| FY18 | FY19 | FY20 | FY21 | FY22 |
+------+------+------+------+------+
| 1 | 0 | 1 | 0 | 1 |
| 1 | 1 | 0 | 1 | 0 |
| 1 | 0 | 0 | 1 | 0 |
+------+------+------+------+------+
Here is what I've come up with so far... The select statement looks something like this
Select distinct
POS
from TBL_Staff
where [' 'FY'+right(year(dateadd(month,3,getdate()))-1,2) '] = 1
What I'm trying to figure it out is if there is a way to dynamically generate the date and get SQl to recognize name+date as a column
Note: This is fake data so please let me know if something isn't clear
Any help on this is most appreciated.
You could use dynamic SQL
DECLARE #sql NVARCHAR(MAX) = 'Select distinct
POS
from TBL_Staff
where FY' + CAST(right(year(dateadd(month,3,getdate()))-1,2) AS VARCHAR) + ' = 1'
exec sp_executesql #sql
#sql is a variable that is used to construct a SQL statement dynamically. Then the sp_executesql procedure executes it. Beware of using dynamic sql when other alternatives exist. It's harder to read, can be difficult to maintain, and can be a security issue if taking input from users and you're not careful to sanitize input parameters.
My example code output:
time | name | status | s_used | s_max |
+------------+-------------+-----------+------------+-----------+
| 1482222363 | asf | Closed | 0/16 | 0 |
| 1482222363 | as0 | Available | 4/16 | 4 |
I have attached the part of my output which is generated using perl cgi script and mysql database.
My query is how to take denominator value from the column s_used and store only the denominator values in the s_max column using perl.
3.I had attached the following part of code which i tried.
if($i == 4){
if(/s_used/){
print;
}
else{
chomp();
my($num,$s_max)=split /\//,$table_data{2}{'ENTRY'};
print $s_max;
}
}
Code Explanation:
$i == 4 is the column where should I store the variable.
I got time column from the sql database $time, name I got from $table_data{0}{'ENTRY'}, status from $table_data{1}{'ENTRY'}, s_used from $table_Data{2}{'ENTRY'}.
Expected output:
time | name | status | s_used | s_max |
+------------+-------------+-----------+------------+-----------+
| 1482222363 | asf | Closed | 0/16 | 16 |
| 1482222363 | as0 | Available | 4/16 | 16 |
Seems your code "my($num,$s_max)=split /\//,$table_data{2}{'ENTRY'};" is right.
Somehow the value $s_max at the time it's writing to the DB is incorrect. Since you did not post the portion of code to show the part $s_max writing back to the DB, you need to check what value is in $s_max (e.g. printing the $s_max value) at the time right before writing it back to DB. From there, please try to trace back why an incorrect value is assigned to $s_max. Then, the problem would be solved.