Tools used
Visual Studio 2017 (with SSIS)
SQL Server Management Studio 17.9.1
Involved in the process
Two SSIS developers and SSMS with Integration Services Catalog which stores deployed projects.
Overview
I have a solution with projects inside created in SSIS. Each project has project parameters specifying for each database connection two different params: Connection string and a password. Password is marked sensitive.
Project and all it's packages have ProtectionLevel set to EncryptAllWithPassword. The project get's pushed to git repository and another developer downloads changes. Now, he needs to provide password in order to be able to work with the project (or multiple projects within solution). So far so good, we have a "master password" on project levels which protect access to parameters such as sensitive passwords. When a developer goes to Project.params and untick sensitive mark, the password is shown. All good for now as well, since he needed to know the password for the project first to see the passwords.
Here's the tricky part
When the project is being deployed do Integration Services Catalog, ProtectionLevel is being changed and the project which can be exported from Management Studio is no longer password protected. To export such a project one obviously needs ssis_admin permission, but that's out of scope for this issue. When the project was deployed and then imported back from SSMS to SSIS, a developer can open it without password and untick the sensitive mark for Project.params passwords. All passwords are visible for him now. This is wrong.
What am I trying to achieve
I want to mimic the same behaviour with sensitive values we have in SSMS. Whenever you untick a sensitive mark on an environment variable, the value is cleared - like below.
However, when I do the same in SSIS Project.params (untick sensitive mark), the value is still shown so I can see all the passwords - as presented below.
I'd like it to be stored as it is, but unable to see it's plain text value.
Is it possible at all? Or maybe there's a better way to organise this? I need to be able to execute packages from within SQL Server Agent (SSMS) providing environment variables as well as from my own computer under SSIS, which is why I need to store these passwords in order not to repeat them every time.
This problem that you described is a real issue for any team working collaboratively on SSIS. I'll describe the pattern that I've used to solve this, which might be helpful. First, I should state that I don't like storing passwords in source control, even if they're encrypted. Here is what I typically do:
Set all SSIS packages and projects to Don't Save Sensitive. This removes all passwords from the files and closes the source control loophole
When possible, all the developers should have a local set up of the ETL ecosystem - SQL databases (no data or just test data), file system, etc. All packages should be configured to work against this local environment. In this way, you can be an admin, connect with windows authentication and have full control over the test data. This also helps you avoid interfering with anyone else's development and testing.
For a SQL connection, set parameters for the connection string and password. The connection string can point to your local instance and use windows auth. The password can be blank and checked as sensitive. If everyone sets up their local system the same way, then nothing needs to change when another developer opens this up and begins work on the project.
For deployment, environments can be configured for each server. The password can optionally be used for SQL authentication and the connection string would change to include the username property and not windows auth.
The above pattern makes it really easy to develop as a team and pretty straight forward for deployment automation.
I would propose to use SSIS Project Catalog and Project Environments together with the following approach.
Think about SSIS packages as programs or runners, and databases - as resources. Thus, packages are independent from resources, and the resources are configured at package setup phase in specific environment.
In practice, this leads to the following configuration and activities:
Packages are created and developed in SSIS Project Mode. All connection manages are declared at Project level.
Do not save passwords in Packages or Projects.
Each environment Project is deployed to has defined Environment variable configuration where we store configuration about databases, namely:
Connection strings, could be cut and paste from the original package
DB name
Server name
User name if Windows Auth is not used
Password if Windows Auth is not used
After project deploy, one has to map all project connection params with environment variables. We created a simple C# program for that.
Values from environment variables are used at corresponding param values of connections. Moreover, you can store other configuration params there, not only connections.
You can have several sets of params at the same environment, and choose set when staring package.
Automated testing is done with scripted execution, and environment is specified in testing script.
So, every environment we deploy project to has configuration environment with all connection data. Connectivity params in QA environments are supplied by env engineer; developer does not need to worry about that.
Related
I have a Websphere Application Server v8.0, and my job requires me to change the location of my JDBC data source to different values to test in different environments. I traditionally would do this via the admin console and change the settings via the Resources > JDBC > Data sources section, but I'd like to write a script to change these settings. When I run the admin console, where do the settings get stored? I can run the console vis-a-vis the Servers tab in Eclipse (Rational Application Developer) or by navigating to localhost:9044, but I don't know where the settings are stored - which I'd need to write said script.
Can anybody help me out?
From what I remember of WebSphere Application Server, the settings are ultimately persisted to the file system - however you shouldn't be changing them this way because application server config is a messy and complicated business and by directly changing settings you risk destroying your app server.
I'd recommend checking out this redbook, particularly Chapter 8 which describes how you can configure your app server with scripts. Also I seem to recall plans to display the equivalent scripting commands in the admin console.
If it helps, I had a quick look locally and found a reference to my JDBC data source in "resources.xml" located within the websphere directory at...
<server profile root>\config\cells\<aNodeCell>\nodes\<aNode>\servers\<aServer>
In the past I've used xml config to read values for convenience, but not often to update. Instead I have made use of some of the jython script options available and can echo Jim's response to check out the options there in case there is something that would be a viable alternative.
Edit:
There is another link that may be of interest Configuring data access with wsadmin scripting. I've not used this particular feature of wsadmin myself but it does appear to show promise at first glance.
If you want to write a script, then rather than looking at file system write a proper jython script, which will do your modifications in the similar way as you would do it via console.
To make writing script easier you can use:
Command assistance in the console - the Help portlet on the right shows last invoked command in jython
Script library, which already provides some scripts - Automating data access resource configuration using wsadmin scripting
And basic scripting commands - Configuring data access with wsadmin scripting
I'm trying to develop a little toy PHP project, and the most convenient location to run it is on a shared host I happen to have for my ill-maintained blog. The problem with this is that I have no way to run Subversion on this shared host, nor do I even have SSH access to be able to access an external repository from the host. Had I been thinking straight a few months ago when the hosting was up for renewal, I probably should have paid a couple extra bucks to switch to something a bit better, but for now I can't justify throwing money at having a second host just for side projects.
This means that a working copy of my project would need to be checked out to my laptop, while the project itself would need to be uploaded to the shared host to run. My best option seems to be creating a virtual machine running Linux and developing everything from in there, but I know from past experience that the extra barrier that creates, small though it may be, is enough that it puts me off firing the VM up just to do a couple minutes work to make some minor change I just thought up. I'd much prefer to just be able to fire up my editor and get to work.
While I'd imagine I'm not the first to encounter such a problem, I haven't had much success finding a solution online. Perhaps there isn't one beyond the VM or "manual mirroring" options, but if there is I'd expect StackOverflow to be the place to find it.
Edit: There's some confusion, it seems, so let me attempt to clarify. The shared host here is basically my dev server, but it has no svn or ssh. In other words, I can svn checkout to my laptop, but I can't run that on my shared host. Similarly, I can run/test my code on the shared host, but I can't do that on my laptop (well, I technically could, but it's Windows, and I don't want to worry about Win-vs.-Linux differences with PHP, since I do want this to become public at some point, and it will certainly be Linux-based at that point).
You might consider writing a post-commit hook to automatically upload the code to your host, so that any time you commit a change, a script executes that:
Checks out a copy of the code into a temporary directory
Uploads that code via FTP (or whatever your preferred method is) to the shared host
Cleans up after itself, optionally informing you via e.g. email when the transfer is successful
Subversion makes enough information available to these scripts at runtime that you could get more sophisticated and opt only to upload the files that changed or alter behavior based on specific property changes, for instance, but for a small project the brute force "copy it all" approach should be fine.
I have a ClickOnce deployment that has different web service endpoints and strings that need to be changed in Settings.Settings. Right now I am only having to deal with on localized development version being done in house and one version that i push out to the customer for their UAT. Now i need 4 versions of this application. in house dev and testing, customer testing and production. I also need these 4 deployments to be able to be installed along side each other. I have discovered that i can change the name (i.e. APP -- INTERNAL -- TEST, APP -- INTERNAL -- DEV, APP -- CUST -- TEST, APP -- CUST -- PROD) and that will allow them all to be installed alongside each other. But, having to remember every place a string needs changed in the various settings.setting of each build, swapping the end points, changing the application names, changing the certificate, changing the deploy addreess and the url for each different build is time consuming and cumbersome. Is there a way to just say "Publish internal test build" and have it do the right thing? I was going to just write various mage scripts but I dont thing that gets me around having to mess with the settings.settings stuff. i didnt write this application nor maintain it but I suppose i could go in and use some sort of conditional logic, but the connections strings for instance are wired to reports and table adapter etc... P.S. I hate ClickOnce
Ok, for a useful answer and not a critique of my writing style. mage.exe is severly lacking in options on what it can an cannot do, it is also poorly documented and does not work as advertised. In order to accomplish what I wanted, I had to download sed for windows and write .bat files to manually rename files to .deploy. I used sed to edit the manifest files and flip options on and off and keep track of the different deployments. So in short write a batch file using mage.exe and sed and have a very good understanding of the contents of a manifest file. Feel free to contact me and I can send scripts that will automate multiple ClickOnce deployments, add the .deploy extension, require a specific version number before start up etc... none of these are possible using the tools MSFT provides.
I'm using CassiniDevLib to host an MVC app for integration testing.
In order to do it I need to amend some config settings on the web server so they match the integration testing environment, first one being the connection string so it points to the test database.
I know I can have two copies of the web.config file and rename them but I was wondering if there was a more elegant way. ie a way to amend the settings in code as part of the Test Fixture setup. The challenge being that I need to access the web server process from my test ficture process
Would appreciate any thoughts on this.
I assume that you are using Visual Studio 2010. In that, you have a feature called as Config Transforms. Basically you can have multiple config file for each build environment. You can have your own custom build env. You have a new one by going to Configuration Manager and adding a new one.
http://blogs.msdn.com/b/webdevtools/archive/2009/05/04/web-deployment-web-config-transformation.aspx
you can search on the internet for Config Transforms, if you need more examples.
I've been trying to figure this out and so far haven't found a simple solution. Is it really that hard to deploy a database project (and a web site) using TFS 2010 as part of the build process?
I've found one example that involved lots of complicated checks and editing the workflow (which is a giant workflow btw).
I've even purchased the book "professional application lifecycle management with VS 2010", but apparently professionals don't deploy their applications since it isn't even mentioned in the book.
I know I'm retarded when it comes to TFS, but it seems like there should be any easy way to do this. Is there?
I can't speak for the database portion, but I just went through this on the web portion, the magic part is not very well documented component, namely the MSBuild Parameters.
In your build definition:
Process on the Left
Required > Items to Build > Configurations to Build
Edit, add a new one, for this example
Configuration: Dev (I cover how to create a configuration below)
Platform: Any CPU
Advanced > MSBuild Process
Use the following arguments (at least for me, your publish method may vary).
MsBuild Params:
/p:MSDeployServiceURL="http://myserver"
/p:MSDeployPublishMethod=RemoteAgent
/p:DeployOnBuild=True
/p:DeployTarget=MsDeployPublish
/p:CreatePackageOnPublish=True
/p:username=aduser
/p:password=adpassword
Requirements:
You need to install the MS Deploy Remote Agent Service on the destination web server, MSDeploy needs to be on the Build/Deployer server as well, but this should be the case by default.
The account you use in the params above needs admin access, at least to IIS...I'm not sure what the minimum permission requirements are.
You configure which WebSite/Virtual Directory the site goes to in the Web project you're deploying. Personally I have a build configuration for each environment, this makes the builds very easy to handle and organize. For example we have Release, Debug and Dev (there are more but for this example that's it). Only the Web project has a Dev configuration.
To do this, right click the solution, Configuration Manager..., On the web project click the configuration drop down, click New.... Give it a name, "Dev" for this example, copy settings from debug or release, whatever matches closest to what your deployment server environment should be. Make sure "Create new solution configurations" is checked, it is by default. After creating this, change the configuration dropdown on the solution to the new Dev one, and Any CPU...make sure your projects are all correct, I had some flipping to x86 and x64 randomly, not sure of the exact cause of that).
In your web project, right click, properties. On the left, click Package/Publish Web (you'll also want to mess with the other Package/Publish SQL tab, but I can't speak to that). In the options on the right click Create deployment package as a zip file. The default location is fine, the next textbox I didn't find documented anywhere. The format is this: WebSite/Virtual Directory, so if you have a site called "BuildSite" in IIS with no virtual directory (app == site root), you would have BuildSite only in this box. If it was in a virtual directory, you might have Default Web Site/BuildVirtualDirectory.
After you set all that, make sure to check-in the solution and web project so the build server has the configuration changes you made, then kick off a build :)
If you have more questions, I recommend you watch this video by Vishal Joshi, specifically around 22 and 59 minutes in, he covers the database portion as well...but I have no actual experience trying it since we're on top of a non MSSQL database.