My database needs to mirror another, to which I have no access except for a nightly export of the sql file. I could script the import using psql.exe, but would prefer everything to be under the control of the dotnet core application.
I can't use the COPY command, because the file contains ALL the sql to set up the schemas and tables, as well as all the sql commands to insert/alter/copy the data.
I can't use \i because that is a postgresql console command, not something I can run through npgsql.
Is what I'm trying to do possible? Is it inherently a bad idea, and should I run a script to import it outside of the dotnet application? Should the dotnet application run and talk to the psql.exe program directly?
You could theoretically parse the SQL file in .NET and send it to PostgreSQL, but this is a very non-trivial thing to do, since you'd need to understand where statements end (identify semicolons) in order to send chunks.
You could, of course, send the entire file as a single chunk, but if it's huge, that may be a bad idea.
At the end of the day, I don't think there's any particular issue with launching psql.exe as an external process from .NET, and properly inspecting its exit code for error handling. Any reason you think you need to avoid that?
Related
I'm used to working with SQL Server and the SQL Server Management Studio has the option to automatically generate a script to drop and recreate everything in a database (tables/views/procedures/etc). I find that when developing a new application and writing a bunch of junk in a local database for basic testing it's very helpful to have the options to just nuke the whole thing and recreate it in a clean slate, so I'm looking for a similar functionality within postgres/pgadmin.
PGAdmin has an option to generate a create script for a specific table but right clicking each table would be very tedious and I'm wondering if there's another way to do it.
To recreate a clean schema only database you can use the pg_dump client included with a Postgres server install. The options to use are:
-c
--clean
Output commands to clean (drop) database objects prior to outputting the commands for creating them. (Unless --if-exists is also specified, restore might generate some harmless error messages, if any objects were not present in the destination database.)
This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.
and:
-s
--schema-only
Dump only the object definitions (schema), not data.
This option is the inverse of --data-only. It is similar to, but for historical reasons not identical to, specifying --section=pre-data --section=post-data.
(Do not confuse this with the --schema option, which uses the word “schema” in a different meaning.)
To exclude table data for only a subset of tables in the database, see --exclude-table-data.
clean in Flyway
The database migration tool Flyway offers a clean command that drops all objects in the configured schemas.
To quote the documentation:
Clean is a great help in development and test. It will effectively give you a fresh start, by wiping your configured schemas completely clean. All objects (tables, views, procedures, …) will be dropped.
Needless to say: do not use against your production DB!
I have a project with Spring, Hibernate and PostgreSQL and have to use ANT to create schema with data:
<sql driver="org.postgresql.Driver"
classpath="src/main/webapp/WEB-INF/lib/postgresql-9.1-901.jdbc4.jar"
url="jdbc:postgresql://localhost:5433/postgres"
userid="postgres"
password="pw123"
autocommit="true"
src="src/main/sql/dbbackup.sql">
</sql>
but I get this error:
C:\Users\<user>\<workspace>\<Project>\antdb.xml:22: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: The JDBC driver currently does not support COPY operations.
Don't know if somehow we could use postgresql.copy class here?
PgJDBC doesn't support COPY directly, but it does via the CopyManager API you can get from the PGConnection interface of the java.sql.Connection returned by PgJDBC.
Unfortunately, you can't use that from a plain SQL file where you mix COPY operations in with other commands.
Personally, I'd shell out to psql to run .sql files using the Ant <exec> task. That way you can include COPY data in-line in your SQL files.
It'd be nice to enable PgJDBC to handle COPY, but it's not easy. It's effectively a different protocol mode in PostgreSQL, and it doesn't make much sense to use the usual JDBC interfaces with prepared statements, execute, etc, for it. We could provide an execSQLScript on the custom PGconnection but that wouldn't help you out much because things like Ant's <sql> task wouldn't use it. You'd have to write a custom task.
Instead, PgJDBC would have to pretty much lie to clients - when it entered COPY mode after a COPY command, it'd have to ignore the JDBC spec and not really do what it was supposed to in response to JDBC statement executes. This would be likely to break all sorts of things.
So - for now, by far the easiest option is to just exec the psql command to do what you want.
I'm attempting to import an SQL dump in PgAdmin 4 using the psql client - However the error message returned is - The system cannnot find the file specified.
Here is a screenshot of my psql client -
The file films.sql is currently stored on my desktop, but I suspect the default location that the psql client accesses is not my desktop? Is there anyway to set the location that the client looks in order to resolve this?
The file SQL is viewable here: https://github.com/datacamp/courses-intro-to-sql/tree/master/datasets
I simply want to get the database on my local machine so that I don't need to store queries in an online learning platform. It would be best if this database is available locally to query and practice on.
I've attempted to execute the whole SQL file as a query on the films database but this does not seem to be working either and returns 'Asynchronous query execution/operation underway.
Query returned successfully in 388 msec.' - However it seems to be the case that the Asynchronous query never completes when I refresh the database.
Please can someone help?
Just give the path to your file:
psql -d my_database -f /path/to/the/file.sql
psql -d my_database -f C:/path/to/the/file.sql
Depending on whether you are on a unix/linux machine or Windows.
Oh, and if you aren't familiar with file paths you may want to take a step back and become more familiar with general computer terminology before diving into a RDBMS. Your learning will be much easier if you have a solid foundation to build upon.
I suspect this question might be moot for the asker at this point, but for anyone else stumbling upon it like I did: the interactive connection info prompts are provided by a batch script (in Windows, I'd guess there's an analogous shell script for Unix) called runpsql.bat, which then just passes your inputs as commandline arguments to the psql.exe executable. I was getting this error because I had migrated my Postgres installation and the batch script was calling a nonexistent path for psql.exe, hence The system cannot find the file specified. I edited runpsql.bat to point to the correct location of psql.exe and that resolved the issue. So for OP, I would look into PgAdmin4 and see where it's (presumably) calling runpsql.bat, then make sure that that calls psql.exe with the correct path.
I am using execute command tag from my liquibase changesets and this inturn is configured to run the sqls in oracle instant client sql plus.
when i run a liquibase update on my changelogxml everything works fine and the liquibase update is sucessfull.I can see the changes to the table also.
But when i try to fail the update process by giving a syntax error in my sql file refered in the changeset.Liquibase still returns liquibase update sucessfull.I expected it to throw sql errors.The sql when run seperately in toad throws syntax error.What should i do to get the error displayed out.?
Datical has created a custom Liquibase change tag that executes SQL using the sqlplus command line client. It was surprisingly much more complicated that you might think.
Some of the issues we had to deal with:
we had to do things to ensure that the sql files always had certain statements in place, and never had certain other statements. This might include things like setting the schema, ensuring that the only spool commands were ones we knew about, that the script had an 'EXIT' command, and ensuring that whenever there was a SQL error that the exit code was returned.
The sqlplus executable does not return an exit code (i.e. a non-zero exit code form the native process) in all cases, and instead will write errors to an error table in the database. The table where sqlplus writes errors is called sperrorlog, and this may be what you will need to look into.
I can't really go into all the details, but just know that what you are attempting to do is neither simple nor straightforward.
I am using Enterprise library semantic logging block (out of process) and using SQL Database sink to dump all the message. After putting everything in place and doing a test run, I am getting the following error - could not find stored procedure 'dbo.WriteTraces'.
Anybody faced similar issue ? Pl suggest.
Out of process semantic logging assembly comes with some powershell scripts and .sql files. We have to edit (to change DB name) and run these scripts. This will generate the stored procs and the associated table for us.
I encountered this same error but it was because we were trying to use a schema other than dbo for our logging database. Once we changed it back to dbo that resolved the problem. We were using the out of process SemanticLogging-svc.exe, which, from what I can tell, assumes that dbo is the schema name.