This is a hybrid question & public service announcement. The basic question is whether there are convenient and/or efficient workarounds to the limitations listed below. I spent half the morning discovering that one cannot just transplant SQL code from Access to Matlab. I've boiled it down to 3 points so far.
Can't use double quotes in SQL statements to avoid collision with
Matlab's string delimiter. The Matlab code for the SQL code strings
can become quite complicated, especially if the SQL strings already
use repeated single-quotes to represent a single quote within a
string constant.
Must always specify a source table from which to query. What won't
work is "SELECT #2015-07-28#". One basically needs to create a 1-row dummy table.
Must always select at least one field in the table being queried.
An asterisk does not seem to suffice.
The above limitations do not exist when submitting SQL code using the Access Query Designer (either in SQL 92 mode or not), nor do these limitations exist when submitting SQL code using VBA via CurrentProject.Connection.Execute.
Hopefully, this saves someone else some time in learning about these differences. And if anyone has found a workaround, that would be appreciated. Note that the above is in the context of using the JDBC/ODBC bridge (3rd of 3 illustrated configurations in the drivers documentation. The database toolbox documentation for directly connecting to an Access file using code (rather than setting up a data source using the GUI) only describes a code pattern that uses the JDBC/ODBC bridge. This is described in Example "Connect to Microsoft Access Using a File DSN" in the "Connect to database" page. I'd like to stick to this approach because I want to quickly be able to directly specify the source *.accdb file without jumping through GUI hoops of setting up a data source.
I've posted this to:
Stack overflow
Usenet
While I have not connected Matlab to Access, I have connected Access databases to PHP, Python, SAS, R, even Excel without any issues with quotes or asterisk. As for the needed source table in queries that is simply due to the Jet/ACE SQL dialect as each dialect have their particular rules. But your query above should work for one-row, one-column output of such date. SQL Server does not require a source table but Oracle does. Access dates are in the MM/DD/YYYY format, requiring # in conditional statements. MySQL use dates in YYYY-MM-DD format, requiring single quotes.
Are you using Windows installed Jet/ACE ODBC drivers? Do note these drivers are softwares external to any application (not part of Access) and can be used by any client interfacing to any data source. Aside - many are not aware that Access' backend database engine, Jet/ACE is actually a Windows technology and not restricted to Access; PC users without Access installed can still use this engine. Hence, connection strings are standard across ODBC calls. And as I see you can apply same principles to Matlab database connections.
With that said, my suggestions/workarounds:
properly learn the SQL dialect for connecting database and check if query runs; even optimize or re-write as needed
try escaping various punctuation not compliant in Matlab strings or use the ASCII counterparts: chr(34) for double quote; chr(39) for single quote; chr(42) for asterisk.
if select statements are limited in Matlab strings, create a temp table from query (use make-table query SELECT * INTO newtable FROM query outside of Matlab), then connect to this new table
use intermediary coding language (VBA, Python, C#) by user trigger or
command line to connect to database and export needed dataset, then
automate Matlab to import
export query to flatfile format (csv, txt, etc.) for Matlab import
Related
PostgreSQL has excellent support for evaluating JSONPath expressions against JSON data.
For example, this query returns true because the value of the nested field is indeed "foo".
select '{"header": {"nested": "foo"}}'::jsonb #? '$.header ? (#.nested == "foo")'
Notably this query does not reference any schemas or tables. Ideally, I would like to use this functionality of PostgreSQL without creating or connecting to a full database instance. Is it possible to run PostgreSQL in such a way that it doesn't have schemas or tables, but is still able to evaluate "standalone" queries?
Some other context on the project, we need to evaluate JSONPath expressions against JSON data in both a Postgres database and Python application. Unfortunately, Python does not have any JSONPath libraries that support enough of the spec to be useful to us.
Ideally, I would like to use this functionality of PostgreSQL without creating or connecting to a full database instance.
Well, it is open source. You can always pull out the source code for this functionality you want and adapt it to compile by itself. But that seems like a large and annoying undertaking, and I probably wouldn't do it. And short of that, no.
Why do you need this? Are you worried about scalability or ease of installation or performance or what? If you are already using PostgreSQL anyway, firing up a dummy connection to just fire some queries at the JSONB engine doesn't seem too hard.
We're using Tableau 10.5.6. I used a reporting tool years ago called Oracle Sales Analyzer. In that tool you could get to the queries generated by the reports and graphs you created through back-end catalogs using their command line.
There you could rewrite the query to be more efficient by fine-tuning the code if you needed. It was a very cool feature of that reporting tool for geeks like me who like to dive into the back end of the product and tune it at a very low level.
My question is, does Tableau have any of this type of facility? Is there a way to get to the queries that get stored once you create a report or a graph. Also is there command line where you can access these catalogs if they exist? Otherwise are these queries just stored in ASCII flat files that can be accessed by a user.
Thanks!
There are two ways that Tableau will query a database.
Option 1: Custom SQL
In your data source, you paste in the sql you have written and Tableau will pass that query through to the database. This gives you complete control over the sql, including adding any indexing hints you may want. See https://onlinehelp.tableau.com/current/pro/desktop/en-us/customsql.html
Option 2: Use the Tableau data source designer
This is what many people do. Here, you visually design your data source with the joins. Tableau translates that design into what the Hyper engine considers to be the most effective way to run the query. Sometimes, Hyper translates that into a regular sql statement. Sometimes it does some additional things to help boost performance, like breaking it up into different queries. A lot depends on the db engine you are connecting to. There is no "sql" stored in a flat file for this. Tableau just translates your design at run-time. The Hyper engine does a good job with fine-tuning, assuming you have an efficient database design with proper indexing and current table statistics.
There is a way to see the sql from option 2 at run-time using Performance Recording. Performance Recording keeps track of each step of the visualization process and will spit out the sql statement(s) that Tableau ran to generate your dataset. The sql is not stored in the twb file though, it's a run-time analysis.
I am trying to import to pgAdmin a big table with more than 100 columns. Is there any way to import the table without creating those 100 columns in a table within the pgAdmin? That would be a considerably time-consuming task.
You are not importing data into pgAdmin, you are importing it into Postgres, and using pgAdmin to help you in that task. Graphical tools like pgAdmin are, at heart, just convenience wrappers around the actual functionality of the database, and everything they do can be done in other ways.
In the case of a simple task like creating a table, the relevant SQL syntax is well worth learning. It will work in any database tool, even (with some minor changes) on other SQL databases (e.g. MySQL), can be saved in version control, and manipulated with an editor of your choice.
You could even go so far as to write a script in the language of your choice that generates the SQL for you based on some other data (e.g. the headings of the CSV file) - although make sure you don't run that with third-party data without checking the result or taking extreme care with code injection and other security concerns!
The Postgres manual has an introduction to tables and creating them which would be a good place to start.
I am trying to load text data into a postgresql database via COPY FROM. Data is definitely not clean CSV.
The input data isn't always consistent: sometimes there are excess fields (separator is part of a field's content) or there are nulls instead of 0's in integer fields.
The result is that PostgreSQL throws an error and stops loading.
Currently I am trying to massage the data into consistency via perl.
Is there a better strategy?
Can PostgreSQL be asked to be as tolerant as mysql or sqlite in that respect?
Thanks
PostgreSQL's COPY FROM isn't designed to handle bodgy data and is quite strict. There's little support for tolerance of dodgy data.
I thought there was little interest in adding any until I saw this proposed patch posted just a few days ago for possible inclusion in PostgreSQL 9.3. The patch has been resoundingly rejected, but shows that there's some interest in the idea; read the thread.
It's sometimes possible to COPY FROM into a staging TEMPORARY table that has all text fields with no constraints. Then you can massage the data using SQL from there. That'll only work if the SQL is at least well-formed and regular, though, and it doesn't sound like yours is.
If the data isn't clean, you need to pre-process it with a script in a suitable scripting language.
Have that script:
Connect to PostgreSQL and INSERT rows;
Connect to PostgreSQL and use the scripting language's Pg APIs to COPY rows in; or
Write out clean CSV that you can COPY FROM
Python's csv module can be handy for this. You can use any language you like; perl, python, php, Java, C, whatever.
If you were enthusiastic you could write it in PL/Perlu or PL/Pythonu, inserting the data as you read it and clean it up. I wouldn't bother.
I have a huge SQL script which i need to analyse. It would be really helpful if i could find a way which can generate a call tree; ie, to see which all procedures are called from a particular procedure. a perl based example is here, http://sqlblog.com/blogs/linchi_shea/archive/2009/10/23/find-the-complete-call-tree-for-a-stored-procedure.aspx
but i need a tool to analyse the text file (.sql file), not the procedure stored in the database. due to some reasons i will not be able to create the whole set of procedures in the database and use the above mentioned tool.
please respond if you have come across any ide/tool with this feature.
Probably not very helpful, as it violates your request for a "offline" sql file, text based parsing tool, but wanted to throw this redgate tool out there that I have used with great success in the past; RedGate Sql Dependency Tracker. It works very well and does a good job mapping out your objects and all their dependencies (definable as to what you want mapped). But it does require a database with all of the existing objects in place to work properly. :(
If you can't find one out there, I guess you could maybe do some script/macro text parsing if all the procedure calls are easily defined and predictable in the file. AutoHotKey is a great general purpose scripting tool/framework, and there are a few sql based scripts out there...just not one exactly like you are looking for that I have seen.