I have a question regarding the output of a job.
Would it be possible to make an output of a job dynamic?
What I mean is being able to configure a wanted output via a property or something? The user should be able to choose in what kind of database he wants the data to be imported by modifying a property.
Apparently, this can be done using a runIf - I unfortunately could not figure out how to do that.
The run if can only be used for booleans. For instance when a condition is met, the existence (or non-existence) of a file, if the number of rows returned is greater than 100, and countless other conditions.
How would it be possible the check if the context variable equals a specific String? For instance, if context.test = "postgres" then tpostgresqloutput, if context.test = "snowflake" then tsnowflakeoutput?
e.g.
job > tPostgresqlOutput > run if > context.test = "postgres"
job > tsnowflakeoutput > run if > context.test = "snowflake"
Thanks,
BR
For String comparison in RunIf, you can use:
context.test.equalsIgnoreCase("postgres")
You may want to connect the appropriate dataflow to the above RunIf depending on whether it is postgres or snowflake.
Related
In a parameterized query issued from c# code to PostgreSQL 10.14 via dotConnect 7.7.832 .NET connector, I select either a parameter value or the local timestamp, if the parameter is NULL:
using (var cmd = new PgSqlCommand("select COALESCE(#eventTime, LOCALTIMESTAMP)", connection)
When executed, this statement throws the error in subject. If I comment out the corresponding parameter
cmd.Parameters.Add("#eventTime", PgSqlType.TimeStamp).Value = DateTime.Now;
and hardcode
using (var cmd = new PgSqlCommand("select COALESCE('11/6/2020 2:36:58 PM', LOCALTIMESTAMP)", connection)
or if I cast the parameter
using (var cmd = new PgSqlCommand("select COALESCE(cast(#eventTime as timestamp without time zone), LOCALTIMESTAMP)", connection)
then it works. Can anyone explain what # operator in the error is referring to and why the error?
In the case that doesn't work, your .Net connection library seems to be passing an SQL command containing a literal # to the database, rather than substituting it. The database assumes you are trying to use # as a user defined operator, as it doesn't know what else it could possibly be. But no such operator has been defined.
Why is it doing that? I have no idea. That is a question about your .Net connection library, not about PostgreSQL itself, so you might want to add tag.
The error message you get from the database should include the text of the query it received (as opposed to the text you think it was sent) and it is often useful to see that in situations like this. If that text is not present in the client's error message (some connection libraries do not faithfully pass this info along) you should be able to pull it directly from the PostgreSQL server's log file.
I have many files in a blob container. However I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container. I used Get metadata and IF Condition on Data Factory. Can you please help me with the dynamic script for this. I tried this #bool(startswith(
activity('Get Metadata1').output.childitems.ItemName,
'SRManifest.csv')). It doesnt work.
Then I thought, what if i used #greaterOREquals(activity('Get Metadata1').output.LastModified,adddays(utcnow(),-2))But this checks the last modified within 2 days of the Bloob not the file exist. Thank you.
Please see below my diagram
I have understood your requirement differently I think.
I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container
1 Change your metadata activity to look for existence of sentinel file (SRManifest.csv)
2 Follow with an IF activity, use this condition:
3 Put your sp in the True part of the IF activity
If you also needed the file list passed to the sp then you'll need the GetMetadata with childitems option inside the IF-True activity
Based on your diagram, since you are looping over all the blob names already, you can add a Boolean variable to the pipeline and set its default value to false:
Inside the ForEach activity, you only want to attempt to set the variable if the value is still false, and if the blob name is found, set it to true. Since Set Variable cannot be self-referential, do this inside the False branch of an If activity:
This will only attempt to process if the value is false (so the file name has not been found yet), and will do nothing if the value is true. Now set the variable based on your file name:
[NOTE: This value can be hard coded, parameterized, or based on a variable]
When you execute the pipeline, you'll see the Set Variable stops attempting once the value is set to true:
In the main pipeline, after the ForEach activity has completed, you can use the variable to set the condition of your final If activity. If the blob is never found, it will still be false, so put the Stored Procedure activity inside the True branch.
I would like to know if there is a way to set db using a variable
For example: I am coding in Python, and I connect using client = MongoClient(uri). All goes fine.
There are 4 dbs: test1,test2,test3,test4.
I am able to list them all.
dblist = client.list_database_names()
print(dblist)
All goes fine.
Now, Instead of connecting/ using
db = client.test1
Is there a way to use a string rather than actual name of the db?
such as str = 'test1', and then db=client.str.
(this doesn't work)
In my program , I display the list of dbs first and then I am taking user input on the db , and proceed with further flow, but unable to do so.
Please help.
You cannot add string as an name when it comes to that. However there is another function that takes string of the name of certain database and gets the database.
db=client.get_database('test')
Here is the documentation: https://api.mongodb.com/python/current/api/pymongo/mongo_client.html
I would like to create sql server job using stored procedure and I can't seem to get it right.
Integration Service Catologs -> SSIDB -> Cat1 ->Projects->999->Packages->999.dtsx
In step 1 properties of below script on Package tab "Server: and Package:" are empty, I need to populate these as well as set 32bit to true
Below is what I got, thanks in advance
EXECUTE msdb..sp_add_job #job_name = 'Job 1', #owner_login_name = SUSER_NAME(), #job_id = #JobId OUTPUT
EXECUTE msdb..sp_add_jobserver #job_id = #JobId, #server_name = ##SERVERNAME
EXECUTE msdb..sp_add_jobstep #job_id = #JobId, #step_name = 'Step1',#database_name = DB_NAME(), #on_success_action = 3 ,#subsystem = N'ssis'
, #command = N' "\SSISDB\Cat1\999\999.dtsx" #SERVER=N"#ServerName"'
EXECUTE msdb..sp_add_jobstep #job_id = #JobId, #step_name = 'Step2', #command = 'execute msdb..sp_delete_job #job_name="Job 1"'
EXECUTE msdb..sp_start_job #job_id = #JobId
if anyone else comes across similar situation, easiest way to figure out how to create a job pragmatically is to create it using UI (Server Agent -> New Job). create everything you want to see, save it, then right click at the job Script Job As -> Create To -> New query and sql server will export the job as a query so you can see what you need to do.
While we wait for clarification on the existing syntax, the two arguments to msdb..sp_add_jobstep that you need to be concerned with are the #subsystem and #command.
, #subsystem = N'SSIS'
, #command = N'/ISSERVER "\"\SSISDB\POC\SSISConfigMixAndMatch\Package.dtsx\"" /SERVER "\".\dev2014\"" /X86 /Par "\"$ServerOption::LOGGING_LEVEL(Int16)\"";1 /Par "\"$ServerOption::SYNCHRONIZED(Boolean)\"";True /CALLERINFO SQLAGENT /REPORTING E'
The GUI will build out these options happily but you can read the dtexec documentation and come to the same script.
/ISSERVER This specifies that we're using the fancy new execution engine built into the SSISDB
We pass in the package we want to execute to this option
/SERVER where will these packages be found
Specify the server name and optional instance
/X86 As the fine documentation notes, this option only works for invocation from SQL Agent but this is how you specify you need to use the 32 bit dtexec.exe
/Par Specify parameter values as needed
Indicates our standard, Basic, level of logging
The next instance of /Par specifies whether the caller should wait for the process to complete (synchronous versus asynchronous process). Yes, the job steps should wait for the process to complete.
/Reporting What information should be reported. This is odd because the useful information you used to get in an SQL Agent job report is no longer there. It will just say Consult the SSISDB reports for more information
E, report Errors only.
I have this ADO.NET command object and I can set some parameters and execute it successfully.
_mergecommand.Parameters.Add(new SqlParameter("values", SqlDbType.Structured));
_mergecommand.Parameters["values"].TypeName = "strlist";
_mergecommand.Parameters["values"].Direction = ParameterDirection.Input;
_mergecommand.Parameters["values"].Value = valuelist;
_mergecommand.ExecuteNonQuery();
This works fine. But I want to prepare this command before executing it because I need to run this millions of times. I am using SQL Server 2008. I get this error if I try to prepare it
SqlCommand.Prepare method requires all variable length parameters to have an explicitly set non-zero Size.
Any idea how to do this?
This is old, but there does appear to be a correct answer which is to use -1 as the size, e.g.:
_mergecommand.Parameters.Add(new SqlParameter("values", SqlDbType.Structured, -1));
If you have to do it millions of times using a command like this is probably not a good strategy.
Can you serialize your data into an XML string and pass that as a single argument? That will be considerably less load on your network and SQL Server.... although it will probably hit your client a lot harder.
If you are dead set on doing it that way, maybe what you are looking for is an overload of the SqlCommand.Parameters.Add method:
_mergecommand.Parameters.Add("#values", System.Data.SqlDbType.NVarChar, 100).Value = foo;
is that more like what you wanted?