there is a Windows Server which runs an SQL Server 2019 instance at my company. I have deployed an Integration services solution (project deployment model) with some SSIS packages. I wondered if there is any way to get data flow task names from the packages, or to get the xml file body in order to extract names by using T-SQL. If none of the above is possible i would like to know in which directory the actual dtsx files are stored in Windows Server when you deploy a solution in SQL Server. I have searched a lot for the above but i cannot find any answer.
Thanks in advance.
How is the package deployed? If it's File System, you could likely do
something like Powershell and SQL. I can't, personally, remember if
packages deployed in msdb are encrypted (I haven't used the deployment
method since 2012), however, if they are deployed via SSISDB you won't
be able to query the packages stored in the database as they are all
encrypted. You'd need to inspect the source packages (in your source
controlled project).
From a comment by #Larnu
Related
Tools used
Visual Studio 2017 (with SSIS)
SQL Server Management Studio 17.9.1
Involved in the process
Two SSIS developers and SSMS with Integration Services Catalog which stores deployed projects.
Overview
I have a solution with projects inside created in SSIS. Each project has project parameters specifying for each database connection two different params: Connection string and a password. Password is marked sensitive.
Project and all it's packages have ProtectionLevel set to EncryptAllWithPassword. The project get's pushed to git repository and another developer downloads changes. Now, he needs to provide password in order to be able to work with the project (or multiple projects within solution). So far so good, we have a "master password" on project levels which protect access to parameters such as sensitive passwords. When a developer goes to Project.params and untick sensitive mark, the password is shown. All good for now as well, since he needed to know the password for the project first to see the passwords.
Here's the tricky part
When the project is being deployed do Integration Services Catalog, ProtectionLevel is being changed and the project which can be exported from Management Studio is no longer password protected. To export such a project one obviously needs ssis_admin permission, but that's out of scope for this issue. When the project was deployed and then imported back from SSMS to SSIS, a developer can open it without password and untick the sensitive mark for Project.params passwords. All passwords are visible for him now. This is wrong.
What am I trying to achieve
I want to mimic the same behaviour with sensitive values we have in SSMS. Whenever you untick a sensitive mark on an environment variable, the value is cleared - like below.
However, when I do the same in SSIS Project.params (untick sensitive mark), the value is still shown so I can see all the passwords - as presented below.
I'd like it to be stored as it is, but unable to see it's plain text value.
Is it possible at all? Or maybe there's a better way to organise this? I need to be able to execute packages from within SQL Server Agent (SSMS) providing environment variables as well as from my own computer under SSIS, which is why I need to store these passwords in order not to repeat them every time.
This problem that you described is a real issue for any team working collaboratively on SSIS. I'll describe the pattern that I've used to solve this, which might be helpful. First, I should state that I don't like storing passwords in source control, even if they're encrypted. Here is what I typically do:
Set all SSIS packages and projects to Don't Save Sensitive. This removes all passwords from the files and closes the source control loophole
When possible, all the developers should have a local set up of the ETL ecosystem - SQL databases (no data or just test data), file system, etc. All packages should be configured to work against this local environment. In this way, you can be an admin, connect with windows authentication and have full control over the test data. This also helps you avoid interfering with anyone else's development and testing.
For a SQL connection, set parameters for the connection string and password. The connection string can point to your local instance and use windows auth. The password can be blank and checked as sensitive. If everyone sets up their local system the same way, then nothing needs to change when another developer opens this up and begins work on the project.
For deployment, environments can be configured for each server. The password can optionally be used for SQL authentication and the connection string would change to include the username property and not windows auth.
The above pattern makes it really easy to develop as a team and pretty straight forward for deployment automation.
I would propose to use SSIS Project Catalog and Project Environments together with the following approach.
Think about SSIS packages as programs or runners, and databases - as resources. Thus, packages are independent from resources, and the resources are configured at package setup phase in specific environment.
In practice, this leads to the following configuration and activities:
Packages are created and developed in SSIS Project Mode. All connection manages are declared at Project level.
Do not save passwords in Packages or Projects.
Each environment Project is deployed to has defined Environment variable configuration where we store configuration about databases, namely:
Connection strings, could be cut and paste from the original package
DB name
Server name
User name if Windows Auth is not used
Password if Windows Auth is not used
After project deploy, one has to map all project connection params with environment variables. We created a simple C# program for that.
Values from environment variables are used at corresponding param values of connections. Moreover, you can store other configuration params there, not only connections.
You can have several sets of params at the same environment, and choose set when staring package.
Automated testing is done with scripted execution, and environment is specified in testing script.
So, every environment we deploy project to has configuration environment with all connection data. Connectivity params in QA environments are supplied by env engineer; developer does not need to worry about that.
what is difference between SSIS package configuration and package deployment? what is the difference between 2008 package configuration and 2012 package configuration and also deployment. After creating a package what steps should follow for deploy a SSIS package? what configuration company uses?
Package configuration - This is to make your package dynamic like instead of making your connection manager with static values you can pass them dynamically. You can get the dynamic values from a file or table or environment variables or registry entry. This is done during Development phase.
Package deployment - This is done once your package development is complete. You will not run your package always by opening the BIDS/Data Tool. For that, you need to deploy the packages to some location like to a File path or to a database. You can access/call the packages from the deployed location using a job or command.
2008 vs 2012+
Configuration: In 2012 or higher, you have extra feature like Project level Connection manger, parameters those can be accessed by all the packages with in that project. If any changes is required in connection, you can change it in one place instead of going each.
Deployment: 2012 or hogher, there is a new feature called project deployment model. A new database called SSIDB is created in database engine of the SQL Server. There are lot of feature available like Environments, Folder level Securiy, View logs through reports etc.
More information available in MSDN.
Refer:
SSIS Catalog
Does anyone have any tips to ensure the SSIS on a production server is in sync with source control?
The only way I can think of to do that is have a separate group that always deploys the SSIS packages, but that isn't feasible in my environment.
Thanks
You can check the version number of the package in source control and the version number on the server.
You can also export the package on the server to the file system of your development box and check to see if the .dtsx file is the same as the one you have in source control.
You could also compare/diff XMLs...
I'm trying to wrap my hand around DSC in PowerShell 4.0, what's possible, and what's not. I've installed xWebAdministration Module.
Could I create a website in IIS, then somehow export it to the a .ps1 file (I think in the MOF format). For example, txWebAdministration has a sample called xWebAdministration\Examples\Sample_xWebsite_NewWebsite.ps1. Could I use an existing website to build a file like this? Or do I have to type everything in manually?
I'm basically trying to jump in and try some things without reading from end-to-end. I was hoping Get-DscConfiguration would somehow read my IIS configuration and create such a file.
For example, in SQLServer, you can create a database and tables using the graphical user interface of SQL Studio Management Studio (SSMS), then export a script to re-create those same tables on another server.
Is such a concept possible with DSC, or am I missing the boat.
There is nothing like this at the moment.
Keep in mind DSC is very new and quite sparse. The DSC Resource Kit is created by people at Microsoft, but isn't supported or all that official.
You could write a script to parse an existing web site and export it to a configuration in a PS1, or even directly into an MOF (it's a text file too), but there is nothing close to this functionality existing at the moment.
Your best bet is probably to type it out manually at this time.
There is a tool that creates a DSC configuration from an existing machine called GuardRail but I think it will not support the experimental xWebAdministration Module. As briantist said, it wouldn't be too difficult to create a powershell script that uses WebAdministration commandlets to produce a DSC configuration.
In fact a workmate asked me the same question today and I'm considering writing one. If you I'll update my answer in a few days with a script that will do this.
In order to implement run and re-run my integration tests an indefinite number of times, I would like to make use of SSDT in VS2012 to publish to a LOCALDB file instance and run EF against that file during integration tests.
Few notes:
We are using EF Database first
We already have a SSDT project that we will use to deploy to a full
database in our different environments
I know that SSDT uses internally a LOCALDB instance to build/deploy/check for errors, so deploying to another custom localdb seems like it should make sense/be doable
Few questions:
Can I deploy to a specific LOCALDB file with SSDT?
Can I do this from the command line in order to automate it when I run integration tests?
Does this roughly seems like a good idea for integration tests with EF or is there a better way? ;-)
Thank you all
You can change the localdb for SSDT in the Debug options for the project. By default the debug options are set to the (localdb) instance and a DB name that corresponds to the project.
You may have more success with Publish Profiles if you're trying to push the project changes to a DB server. You can use those with SQLPackage to push the changes along with a known set of options to a pre-defined server/database.
You can definitely push the changes through a command line. We're doing it with MSBuild to generate a dacpac file, then SQLPackage to publish the changes from the dacpac to the appropriate server/database.
Can't say for sure on this one. If it works for you, it's likely a good start. We do DB development outside of EF and try to do that first rather than trust EF to generate a good relational model.
I have a handful of blog posts on SSDT SQL Projects at http://schottsql.blogspot.com/search/label/SSDT that might be helpful.