Does anyone know of a paid service that allows you to load up queue of url based tasks that are then run in realtime? I'm looking at setting up a local task queue, but I'm willing to pay for it.
Beanstalkd is easy enough to install on Linux, or there is a SAAS compatible version - http://www.iron.io/products/mq
Beyond that, Amazon SQS has various libraries that make it quite easy to use, though it uses HTTP to get/set items in the queue, so it has an additional overhead.
Related
What is the best way to achieve DevOps with XPages.
Multiple Developers working as a team, On Premises Servers [Dev, QA,
Prod] can we replicate to Bluemix? Source Control Automated Testing UI
/ Application, Unit testing business logic with testing framework, Automated Deployment
IDE/Tools
Domino Designer; are there other ways?
Note: Use of Views when the data is in a NSF, otherwise data is in the cloud, or SQL. No Forms or other classic Notes design elements.
What are your approaches to this?
This is a high level overview of the topics required to attempt what you're describing. I'm breezing past lots of details, so please search them out; I've tried to reference what I'm currently aware of as far as supporting documentation and blog posts, etc. of others. If anyone has anything good to add, I'm happy to add it in.
There are several components involved with what you're describing, generally amounting to:
scm workflow
building the app (NSF)
deploying the built app to a Domino server
Everything else, such as release workflow through a QA/QC environment, is secondary to the primary steps above. I'll outline what I'm currently doing with, attempting to highlight where I'm working on improving the process.
1. SCM Workflow
This can be incredibly opinionated and will depend a lot on how your team does/wants to use source control with your deployment / release process. Below I'll touch on performing tests, conceptually, during/around the build step.
I've switched from a fairly generic scm server implementation to a GitLab instance. Even running a CE instance is pretty fantastic with their CI runner capabilities. Previously, I had a Jenkins CI instance performing about the same tasks, but had to bake more "workflow" into the Jenkins task, whereas now most of that logic is in a unified script, referenced from a config file (.gitlab-ci.yml). This is similar to how a Travis CI or other similar CI config file works.
This config calls some additional helper work, but ultimately revolves around an adapted version of Egor Margineanu's PowerShell script which invokes the headless DDE build task.
2. Building an NSF from Source
I've blogged about my general build process, with my previous Jenkins CI implementation. I followed the blogging of Cameron Gregor and Martin Pradny for this. Ultimately, you need to:
configure a Windows environment with Domino Designer
set up Domino Designer to import from ODP (disable export), ensuring Build Automatically is enabled
the notes.ini will need a flag of DESIGNER_AUTO_ENABLED=true
the Jenkins CI or GitLab CI runner (or other) will need to run as the logged in user, not a Windows service; this allows it to invoke the "headless dde" command correctly, since it runs in the background as opposed to a true headless invocation
ensure that Domino Designer can start without prompting for a user's password
My blog post covers additional topics such as flagging the build as success or failure, by scanning the output logs for being marked as a failed build. It also touches on how to submit the code to a SonarQube instance.
Ref: IBM Notes/Domino App Dev Wiki page on headless designer
Testing
Any additional testing or other workflow considerations (e.g.- QA/QC approval) should go around the build phase, depending on how you set up your SCM workflow. A lot of the implementation will revolve around the specifics of your setup. A general idea is to allow/prevent deployment based on the outcome of the build + test phase.
Bluemix Concerns
IBM Bluemix, the only PaaS that runs IBM XPages applications, will require some additional consideration, such as:
their Git deploy process will only accept a pre-built NSF
the NSF must be signed by the account owner's Bluemix ID
Ref:
- IBM XPages on Bluemix
- Bluemix Docs: Building XPages apps for the Bluemix Runtime
3. Deploy
To Bluemix
If you're looking to deploy an XPages app to run on Bluemix, you would want to either ensure your headless build runs with the Bluemix ID, or is at least signed with it, and then deploy it for a production push either via a git connection or the cf/bluemix command line utility. Bluemix's receive hooks handle all the rest of the deployment concerns, such as starting/stopping the server instance, etc.
To On-Premise Server
A user ID with appropriate level credentials needs to perform the work of either performing a design replace/refresh or stopping a dev/test/staging server, performing the file copy of the .nsf, then starting it back up. I've heard rumors of Cameron Gregor making use of a plugin to Domino Designer to perform the operations needed for OSGi plugin development, which sounds pretty useful. As most of my Domino application development is almost purely NSF based, I'm focusing more on an approach of deploying to a staging/dev/test server, which I can then perform a design task on to do the needed refresh/replace; closer to the "normal" Domino way of doing things.
Summary
Again, there are a lot of moving pieces involved here, some of which gets rather opinionated rather quickly. For example, I'm currently virtualizing my build machine, so I can spool up a couple virtual machines of it, allowing for more than one build at a time. If there are major gaps in the process, let me know and I'll fill it what I can.
Lets take the example of a heavily used cloud based software.
When a deployment happens, let's say users are online.
Won't the server require stop & start after deploy? How is the service continuity maintained?
How will the ongoing user sessions / unsaved data be continued post deploy?
How is the risk managed? (Lets say an issue comes up after deploy and you need to revert to the older version, now imagine a user has already worked on the new version and saved some data with it, which is not compatible with previous versions)
Whether the server require stop & start after deploy depends on the technology being used. Any state information that is kept within the server can be externalized (i.e. written to disk) for the purpose of updating the application. Whether this is neccessary and whether this is done depends on the technology being used.
I have a REST API endpoint written in Go and I am wondering what is the best way to deploy it. I know that using Google App Engine would probably make my life easier in terms of deployment. But, suppose that I want to deploy this on AWS. What options/process/procedures do I have. What are some of the best practices out there? Do I need to write my own task to build, SCP and run it?
One option that I am interested in trying is using Fabric to create deployment tasks.
Just got back from Mountain West DevOps today where we talked about this, a lot. (Not for Go specifically, but in general.)
To be concise, all I can say is: it depends.
For a simple application that doesn't receive high use, you might just manually spin up an instance, plop the binary onto it, run it, and then you're done. (You can cross-compile your Go binary if you're not developing on the production platform.)
For slightly more automation, you might write a Python script that uploads and runs the latest binary to an EC2 instance for you (using boto/ssh).
Even though Go programs are usually pretty safe (especially if you test), for more reliability, you might daemonize the binary (make it a service) so that it will run again if it crashes for some reason.
For even more autonomy, use a CI utility like Jenkins or Travis. These can be configured to run deployment scripts automatically when you commit code to a certain branch or apply tags.
For more powerful automation, you can take it up another notch and use tools like Packer or Chef. I'd start with Packer unless your needs are really intense. The Packer developer talked about it today and it looks simple and powerful. Chef serves several enterprises, but might be overkill for you.
Here's the short of it: the basic idea with Go programs is that you just need to copy the binary onto the production server and run it. It's that simple. How you automate that or do it reliably is up to you, depending on your needs and preferred workflow.
Further reading: http://www.reddit.com/r/golang/comments/1r0q79/pushing_and_building_code_to_production/ and specifically: https://medium.com/p/528af8ee1a58
I have an application running on a solaris machine with configuration modifications and deletions of configuration handled by multiple persons in the team....I would like to streamline this process to ensure no loss of configuration and make it easier to identify when/where the changes are done and by whom and retreive the configuration files as necessary.
I looked at svn and other repositories but they all seem to work with a repository stored on some machine and all the chnages have to be made then and there....
I am hoping to find a solution where a service would be running on solaris in teh background and monitors the changes and automatically creates the necessary versioning.
Am I asking for something that doesn't exist? or are they better approaches to solving this issue?
Thank you.....
A couple thoughts. If you mean system configuration then you can use SMF (Service Management Facility). This provides a service for managing the configuration, inter dependencies, and the running of services. Tracking of changes is easy but I'm not so sure if versioning would be easy to manage.
The next to last paragraph makes me think you want to allow the user to make ad hoc changes and have some daemon monitor and store those. That could probably be done with FAM and maybe use of ZFS versioning but that seems like overkill.
As far as a better approach. I would tend to use ZFS snapshots or a like mechanism to checkpoint my configuration. And then I would use SMF to manage my configuration. Maintaining versions of the configuration of individual subsystems doesn't quite do it for me. It is the configuration of the whole box that is interesting.
We will be embarking on an Application developement project (.NET 3.5) for a large organization. As we started thinking about the upgrades we would be giving across the machines, we are looking at options like ClickOnce.
What we need is a push model, as long as the client machine is connected to the network, the server can send updates. I believe ClickOnce is a pull model(although by specifying minimum version we can kind of push). Also ClickOnce downloads complete files only, it cannot download the change (byte difference) among the files.
Can anyone point me to a better tool that can be used here. Also better strategies, if any, are welcome, we are in a very early stage of the project.
I don't have a definitive answer on better options, but I've used ClickOnce and can offer some advice.
There are several update options with ClickOnce (before starting, after starting, check every time, check every X Hours/Days/Weeks, etc). You can also throw those out and write code to check for updates. It's not a "push" from the server, but your client could poll for updates which would be the next best thing. Just remember, the application is going to have to restart after the update to see changes.
ClickOnce only downloads changed files. However, the progress dialog always shows the entire size of the application even if it's only downloading a single file. Everyone worries about that, but it's just a bug with the progress dialog.
Finally, I'm a big fan of keeping it simple. It's really easy to over-think these things and create a monstrosity that was never needed. We went through something similar at my company. We were so worried about users downloading unnecessary bytes, we broke our apps up into more, smaller assemblies. It turned into a nightmare; apps were harder to maintain and performed worse on the client. We finally undid it all and wasted weeks just to end up where we started.
I'm not saying you don't need the features you're asking for, I don't know your scenario. Just educate yourself first and know what you're getting yourself into.
We use clickonce at my company (about few hundred users for the app geographically dispersed). By specifying the minimum version we can make sure that every app installation gets updated after deployment automatically. You are right that clickonce downloads full files only but only files that have changed since previous version. If that is still a concern you can break your application into more smaller assemblies. I think you can also use netmodules but then Visual Studio has not built in support for that.
In general clickonce has worked good for us.
I am just in the process of implementing such a service on top of my distributed application platform. In essence I have developed a "push" model for corporates that follows these basic principles:
Software upgrades are "managed" from the server, NOT from the client, which is in line with the deployment of corporate software as opposed to user software (this is a very important point)
Software upgrades can be customised per client application on the server, i.e. the server can deploy unique configurations to every client if required
Software upgrades can be deployed to clients at different times, or all at the same time, or any combination of the two
The software upgrade version can be specified per client, i.e. different versions can be deployed to different clients as required
All software upgrades for all clients can be "managed" from a single server, i.e. the software upgrading "service" is consistent across any application, and all applications can utilise the software upgrading "service"
Clients can implement a software upgrade policy of automatic (application restarts as soon as the upgrade has been downloaded and available at the client), manual (application needs to be "sent" a custom "force upgrade"
message"), or on restart (application upgrades on shutdown if an upgrade has been downloaded and is available)
All auto-upgrading functionality is transparent to any running applications as this is all performed in autonomous background threads and all inter-process communication and file transfer is handled by my framework
In essence this now allows me (or will allow me when I have tidied a few things up and thoroughly tested the implementation) to manage the version of any application developed by me from a central server after it has been initially installed, without any client intervention.