I am new to Druid and PlyQL and i could not find any guides on how to use it. I have installed the package but where should i input the query from example (in documentation https://plywood.imply.io/plyql):
plyql -h 192.168.60.100:8082 -q 'SHOW TABLES'
The is a CLI command. You need to run this on terminal with appropriate parameters. Alternatively you can use any visualation tools for druid eg. Grafana, Imply or Turnilo.
Related
I am new to Postgresql. I am using SQL shell(psql) for working. When I use any psql commands, it is not displaying any output. And I am using psql(13.3) version.
When I use psql --help, as per documentation, it should display the available options. But I dint get any output dispalyed. Similarly when I use anyother commands starting with psql, it is not displaying any results.
Can anyone help me on this?
I am running a set of SQL statements sequentially in the AWS Redshift Query editor.
sql-1
sql-2
sql-3
....
sql-N
However, in the Redshift Query editor, I cannot run multiple SQL statements. So currently I am running SQL statements one by one manually.
What is the alternative approach for me? For me looks like, I can use the DBeaver.
Is there more programmatic approach i.e. just by using a simple bash script?
If you have a Linux instance that can access the cluster you can use the psql command line tool. For example:
yum install postgresql
psql -h my-cluster.cjmul6ivnpa4.us-east-2.redshift.amazonaws.com \
-p 5439 \
-d my_db \
-f my_sql_script.sql
We recently announced a way to schedule queries: https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-redshift-supports-scheduling-sql-queries-by-integrating-with-amazon-eventbridge/ And even more recently published this blog post that walks you through all the steps using the Console or the CLI: https://aws.amazon.com/blogs/big-data/scheduling-sql-queries-on-your-amazon-redshift-data-warehouse/ Hope these links help.
I am currently trying to use a dashDB database with the db2cli utility and ODBC (values are from Connect/Connection Information on the dashDB web console). At this moment I can perfectly do SELECT or INSERT statements and fetch data from custom tables which I have created, thanks to the command:
db2cli execsql -connstring "DRIVER={IBM DB2 ODBC DRIVER - IBMDBCL1}; DATABASE=BLUDB; HOSTNAME=yp-dashdb-small-01-lon02.services.eu-gb.bluemix.net; PORT=50000; PROTOCOL=TCPIP; UID=xxxxxx; PWD=xxxxxx" -inputsql /tmp/input.sql
Now I am trying to do a DB2 LOAD operation through the db2cli utility, but I don't know how to proceed or even if it is possible to do so.
The aim is to import data from a file without cataloging the DB2 dashDB database on my side, but only through ODBC. Does someone know if this kind of operation is possible (with db2cli or another utility)?
The latest API version referenced from the DB2 on Cloud (ex DashDB) dashboard is available here. It requires first to call the /auth/tokens endpoint to generate an auth token based on your Bluemix credentials to be used to authorize the API calls.
I've published recently a npm module - db2-rest-client - to simplify the usage of these operations. For example, to load data from a .csv file you can use the following commands:
# install the module globally
npm i db2-rest-client -g
# call the load job
export DB_USERID='<USERID>'
export DB_PASSWORD='<PASSWORD>'
export DB_URI='https://<HOSTNAME>/dbapi/v3'
export DEBUG=db2-rest-client:cli
db2-rest-client load --file=mydata.csv --table='MY_TABLE' --schema='MY_SCHEMA'
For the load job, a test on Bluemix dedicated with a 70MB source file and about 4 million rows took about 4 minutes to load. There are also other CLI options as executing export statement, comma separated statements and uploading files.
This is not possible. LOAD is not an SQL statement, therefore it cannot be executed via an SQL interface such as ODBC, only using the the DB2 CLP, which in turn requires a cataloged database.
ADMIN_CMD() can be invoked via an SQL interface, however, it requires that the input file be on the server -- it won't work with a file stored on your workstation.
If JDBC is an option, you could use the CLPPlus IMPORT command.
You can try loading data using REST API.
Example:
curl --user dashXXX:XXXXXX -H "Content-Type: multipart/form-data" -X POST -F loadFile1=#"/home/yogesh/Downloads/datasets/order_details_0.csv" "https://yp-dashdb-small-01-lon02.services.eu-gb.bluemix.net:8443/dashdb-api/load/local/del/dashXXX.ORDER_DETAILS?hasHeaderRow=true×tampFormat=YYYY-MM-DD%20HH:MM:SS.U"
I have used the REST API and have not seen any size limitations. In ver 1.11 of dashDB local (warehouse db) external tables have been included. As long as file is on the container it can be loaded. Also the DB2 Load locks the table until load is finished where a external table load won't
There are a number of ways to get data into Db2 Warehouse on Cloud. From a command line you can use Lift CLI https://lift.ng.bluemix.net/ which provides the best performance for large data sets
You can also use EXTERNAL TABLEs https://www.ibm.com/support/knowledgecenter/ean/SS6NHC/com.ibm.swg.im.dashdb.sql.ref.doc/doc/r_create_ext_table.html which are also high performance and have lots of options
This is a quick example using a local file (not on the server) hence the REMOTESOURCE YES option
db2 "create table foo(i int)"
echo "1" > /tmp/foo.csv
db2 "insert into foo select * from external '/tmp/foo.csv' using (REMOTESOURCE YES)"
db2 "select * from foo"
I
-----------
1
1 record(s) selected.
for large files, you can use gzip, either on the fly
db2 "insert into foo select * from external '/tmp/foo.csv' using (REMOTESOURCE GZIP)"
or from gzip'ed files
gzip /tmp/foo.csv
db2 "insert into foo select * from external '/tmp/foo2.csv.gz' using (REMOTESOURCE YES)"
I'm able to see queries in MongoDB, but I've tried to see what writes are being performed on a MongoDB database without success.
My application code doesn't have any write commands in it. Yet, when I load test my app, I'm seeing a whole bunch of writes in mongostat. I'm not sure where they're coming from.
Aside from logging writes (which I'm unable to do), are there any other methods that I can use to determine where those writes are coming from?
You have a few options that I'm aware of:
a) If you suspect that the writes are going to a particular database, you can set the profiling level to 2 to log all queries
use [database name]
db.setProfilingLevel(2)
...
// disable when done
db.setProfilingLevel(0)
b) You can start the database with various levels of versbosity using -v
-v [ --verbose ] be more verbose (include multiple times for more
verbosity e.g. -vvvvv)
c) You can use mongosniff to sniff the port
d) If you're using replication, you could also check the local.oplog.rs collection
I've tried all of jeffl's suggestions, and one of them was able to show me the writes: mongosniff. Thanks jeffl!
Here are the commands that I used to install mongosniff on my Ubuntu 10 box, in case someone else finds this useful:
git clone git://github.com/mongodb/mongo.git
cd mongo
git checkout r2.4.6
apt-get install scons libpcap-dev g++
scons mongosniff
build/linux2/normal/mongo/mongosniff --source NET lo 27017
I made a command line tool to see the logs, and also activate the profiler activity first without the need of others client tools: "mongotail".
To activate the log profiling to level 2:
mongotail databasename -l 2
Then to show the latest 10 queries:
mongotail databasename
Also you can use the tool with the -f option, to see the changes in "real time".
mongotail databasename -f
And finally, filter the result with egrep to find a particular operation, like only show writes operations:
mongotail databasename -f | egrep "(INSERT|UPDATE|REMOVE)"
See documentation and installation instructions in: https://github.com/mrsarm/mongotail
I'd like to monitor the queries getting sent to my database from an application. To that end, I've found pg_stat_activity, but more often then not, the rows which are returned read " in transaction". I'm either doing something wrong, am not fast enough to see the queries come through, am confused, or all of the above!
Can someone recommend the most idiot-proof way to monitor queries running against PostgreSQL? I'd prefer some sort of easy-to-use UI based solution (example: SQL Server's "Profiler"), but I'm not too choosy.
PgAdmin offers a pretty easy-to-use tool called server monitor
(Tools ->ServerStatus)
With PostgreSQL 8.4 or higher you can use the contrib module pg_stat_statements to gather query execution statistics of the database server.
Run the SQL script of this contrib module pg_stat_statements.sql (on ubuntu it can be found in /usr/share/postgresql/<version>/contrib) in your database and add this sample configuration to your postgresql.conf (requires re-start):
custom_variable_classes = 'pg_stat_statements'
pg_stat_statements.max = 1000
pg_stat_statements.track = top # top,all,none
pg_stat_statements.save = off
To see what queries are executed in real time you might want to just configure the server log to show all queries or queries with a minimum execution time. To do so set the logging configuration parameters log_statement and log_min_duration_statement in your postgresql.conf accordingly.
pg_activity is what we use.
https://github.com/dalibo/pg_activity
It's a great tool with a top-like interface.
You can install and run it on Ubuntu 21.10 with:
sudo apt install pg-activity
pg_activity
If you are using Docker Compose, you can add this line to your docker-compose.yaml file:
command: ["postgres", "-c", "log_statement=all"]
now you can see postgres query logs in docker-compose logs with
docker-compose logs -f
or if you want to see only postgres logs
docker-compose logs -f [postgres-service-name]
https://stackoverflow.com/a/58806511/10053470
I haven't tried it myself unfortunately, but I think that pgFouine can show you some statistics.
Although, it seems it does not show you queries in real time, but rather generates a report of queries afterwards, perhaps it still satisfies your demand?
You can take a look at
http://pgfouine.projects.postgresql.org/