I asked question about how to code cost-efficient way with CDK in here.
https://stackoverflow.com/questions/71836274/what-is-the-cost-efficient-approach-for-aws-serverless-team-development?noredirect=1#comment127052075_71836274
But the question is closed with [Questions related to pricing or costs of services are not acceptable here].
https://stackoverflow.com/users/405555/elliveny and
https://stackoverflow.com/users/174777/john-rotenstein
advised me to share a single account for developers and I did.
But now my team says if they create 2 separate branches and deploy, they are overwriting each other's work on AWS.
developer A creates branchA from develop
developer B creates branchB from develop
developer A develops API (appsync + lambda)
developer b develops API (appsync + lambda)
developer A deploy branchA
developer B deploy branchB
and on aws developer A work is overwritten. only developer B work stays.
my question :
Is this how aws cloudformation work? or our team doing something wrong?
if yes, on what level does it overwrite? for example if it is stack level, we can make [2 developer chan't change same cdk stack same time] rule
Depending on what kind of recurses they are using in their branches. If the resources are regional such as lambda and API gateway, then they can depoly their branches to different regions. CloudFormation is regional, so they will not clash with each other.
But ideally, the developers A and B should have their own sandbox accounts for testing and development for maximum separation. Once their branches gets merged into master (or equivalent) then you, and only you, deploy these changes to a dedicated test/development account for further testing.
Related
I would like to ask what people use to provision an ephemeral preview environment in AWS EKS for your service under test. Also in addition, I am curious to know how you provision any dependent services (such as Database).
E.g. I am working on a back-end service and would like to deploy an isolated ephemeral version of this service packaged from my feature branch, including the database. Furthermore, I would also like copy of a front-end service in my isolated environment to test my back-end.
Any thoughts would be appreciated
Thanks
Sachin
You can roll your own solution: by wiring together your own CI/CD (Jenkins, CircleCI, BuildKite, Github Actions, etc) solution to trigger building and deploying of a preview environment by tying in to webhooks on your source repository. This would have to include your building of the modified code, then deploying that code to some staging environments, then of course seeding those environments with some type of data.
There is a bit of nuance to getting this right. You should check out https://ephemeralenvironments.io/ which is a good template of what needs to go in to these environments.
A lot of other folks use services that provide this as a SaaS platform, Shipyard.build, Release, and Velocity.tech are a few of your options.
Disclaimer: I'm on the Operations team at Shipyard
Hope this helps!
Currently, we have already implemented the SonarQube on Azure VM, Due to cost and maintenance activity, we are thinking about moving to SonarCloud. Basic differences we are having from the pricing point of view and LOC, by feature-wise, am looking for major differences
Question: We have observed that in SonarQube we can set new code analysis on any branch whereas I did not find the same thing in SonarCloud it only allows me to set new code only on the Master/Default branch, the rest of the branches are not having options to set, also If I set the previous version, specific analysis the build fails for other branches except for master/default.
How can we set a new code rule for branches?
For SonarCloud, the prime focus is on the developer’s workflow and bringing value to the development team within their existing ALM environment. Thus, the ‘Enterprise’ use-case is not currently addressed by SonarCloud.
SonarCloud is hosted by SonarSource in AWS and your code will be stored in a private subnet which is secured but again it's your choice..!! (We prefer to keep our code in our own infra)
For Enterprise use-cases, explore SonarQube
SonarQube Enterprise Edition (EE) includes a few management features that may be valuable to your organization. SonarQube along with a supported database is installed on your own on-site servers or in a self-managed cloud environment
SonarQube is enterprise ready application with all the required configurations whereas SonarCloud is missing enterprise level functions such as Authentication, Portfolios, Global settings, Branch & New code, and Housekeeping etc. that is why you are unable to set New code analysis on branch level.
SonarCloud meant for small projects that focuses on Master where your application will be built and deployed using only Master/release branch and rest of the branches are considered as a short-living branches in which we don't get much more options to play.
As part of Licensing and services both products cover essentially the same languages (SonarCloud doesn’t support PL/I, RPG or VB6) and support similar functions, SonarCloud additionally offers a SonarServer and Database to store the code and results with all the services like database and server SonarCloud still much more cheaper than the SonarQube but why? --
You can go through the below links for more details.
https://blog.sonarsource.com/sq-sc_guidance
https://sonarcloud.io/documentation/user-guide/new-code/
I'm not that familiar with Github Actions and how their runners work but my Devops team are adamant that we cannot run any github actions that require runners because the runners are public and it puts our code at risk of being accessed by 3rd parties.
I know there is an option to use hosted runners, but we're not there yet.
Looking for any advice/references that devs with experience can provide me to dispel these rumours
As per the articles Link1 and Link2, GitHub public hosted machines are more secured than self-hosting machines. (At least the articles claim so)
In my experience, big organizations that are very particular about security always go for self-hosting machines which require separate teams to maintain.
The choice should be completely based on what type of application you use and how well you are managing the secure details within the repo.
There are several ways for GitHub-hosted Actions runners to connect to resources on your private network, as described in this 2022 June GitHub blog post:
Use the GitHub Actions OpenID Connect (OIDC) token to authenticate through an API gateway (we open sourced a reference implementation as an example, but do note that it requires customization for your use case and is not ready-to-run as-is).
You would run the API gateway on your infrastructure, but as it is stateless, it can scale quite well for high-bandwidth use cases.
:
You can use WireGuard to create a temporary overlay network between the GitHub Actions runner and your private network.
You can use a commercial solution for an overlay network, like Tailscale.
Based on WireGuard, it has similar advantages and disadvantages, except that as a commercial product, it’s even easier to set up and includes NAT traversal.
Please note that it might require a paid plan for higher data volumes.
I'm a developer whose transitioning into Devops. By observation, I've noticed that a lot of dev shops have started using Octopus Deploy and Azure Devops Services (AzDo, formerly VSTS), or they are starting new projects to setup devops ci/cd pipelines AND they spec to use both tools.
I've been through some quick training for both tools and though they aren't perfectly the same, AzDo seems to offer all of the same features as Octopus Deploy.
So, my question is if a company is already using AzDo for much of their version control, or anything CI/CD pipeline-related, why would you use Octopus? What benefit does it offer to use Octopus for your build and deploys to AzDo?
Note, I am very, very new to Devops. I'm just asking because at the "10,000 feet view" there doesn't seem to be any reason for Octopus if you're already using AzDo. I mention Octopus Deploy by name because I see it come up frequently. However, I assume there could be other tools that serve the same purpose of automatic build and deploying that might also integrate with AzDo. However, AzDo offers build and deploy built in one. Why split out the work?
Let me preface why I like to both build and deploy with VSTS:
Same permissioning end to end
Line of sight from end to end build and deployment
Reasons I favor Octopus Deploy over VSTS Release:
Ability to upload packages/artifacts
External ones that are maybe one off packages to get deployed for a specific release
Target Definition
When you create Targets or servers you are deploying to, you are able to add a target to one or multiple environments and assign tags/roles to a target. What does this mean? More flexible server definition rather than defining strict Agents to a pool or servers to a Deployment Group, you can allow a target to span multiple (ie: a testing server that spans your Dev and Test environments and only gets triggered on steps that are defined for that role). I realize you can accomplish similar things to this in VSTS but in my opinion it's far more cumbersome.
Variable Definition
Variables can be grouped at a global level and grouped by a specific pipeline/process (that part is similar to VSTS). Variables can also be grouped or scoped by environments or roles (above) so you are able to have different variable values per role per environment; both super granular and flexible. Places this comes in handy is if you have a backend server with a connection string and maybe 2 content delivery nodes (role - content delivery) that get slightly different values than the backend server. At the moment, I do not know (other than creating new environments) how one would accomplish the same in VSTS.
Process Definition
All of the above comes together in the process definition of Octopus Deploy. The super flexible and granular variables and target definition allows you to focus on the actual deployment process rather than getting hung up on the nuances of the UI and its limitations. One example would be defining a process where the first step would be taking something out of a load balancer from a central server, step two deploy code to delivery server one, step three put back in lb, step 4 take out node two from lb called from a central server, step 5 deploy code to node two, and last step, back into load balancer. I realize it's a very simple hypothetical, but within Octopus Deploy, it's one steady process filtered to execute on specific roles, within VSTS you would have to break that down into different agent phases and probably pipelines.
Above are really the biggest points I see to use Octopus Deploy over VSTS Release. Now why would someone use VSTS to build and OD to release/deploy? There are a lot of different factors that go into it, some are corporate drivers like having an enterprise git client that has permissions handled thru MSDN. Sometimes it's a project management driver of having work items tied tightly to commit and builds, but with the added flexibility that OD brings to the table for free/minimal cost.
Hoping this help shine a little light into maybe why some people are crossing streams and using both VSTS and OD.
A lot of good points have been made already, but it really comes down to what you need. I would venture a lot of us started using Octopus before Release Management was really a thing.
We use VSTS for all our source control and builds and then all our deployments are handled through Octopus.
When we started evaluating tools, VSTS had nothing for deployments. Even now, they are still playing catch up to Octopus in feature set.
If you are doing true multi-tenanted and multi-environment deployments, I don't think VSTS really compares. We are using Octopus with around 30 tenants, some on Azure, some on premise. We deploy a mix of web and desktop apps. We are even using Octopus to deploy some legacy VB6 and winforms applications.
Multi-Tenancy (critical for us)
VSTS added Deployment Groups a while ago which sound pretty similar to Octopus Environments before multi-tenancy was implemented. Before Octopus had true multi-tenancy (it's been around a while now), people would work around it by creating different environments per tenant, like "CustomerA - Dev", "CustomerA - Prod", etc. Now you just have your Dev/Test/Prod environments and each tenant can have variables scoped to those individual environments.
Support
Documentation is excellent and it's really easy to get up and running.
The few times I've needed to contact someone at Octopus, they've answered very quickly and knowledgeably.
Usability
Having the Octopus dashboard giving us an overview of all our projects is amazing. I don't know of anyway to do this in VSTS, without going into each individual project.
Octopus works great on a mobile device for checking deployment status and even starting new deployments.
Community
Octopus works with their customers to understand what they want and they often release draft RFCs and have several times completely changed course based on customer feedback.
If we know what sort of applications you are deploying, and to what kinds of environments, we would be able to better tailor our responses.
The features you see today in VSTS weren't there a few years ago, so there might be an historical reason.
But I want to state here some non-opinionated reasons that may suggest an organization to opt for different tools instead of one.
Separate responsibility and access levels
Multiple CI tools in dev teams (orgs that are using also Jenkins or TeamCity or else) and need to standardize and control deployments
An org needs a feature available only in Octopus (maybe Multi-tenancy)
Octopus does a great job of focusing on deployments. Features reach octopus before vsts, support is local and responsive. That, and you never run out of build/release minutes!
Seriously though, I just like to support smaller companies where possible and if all features were equal, I'd still pick them.
The big reason in the past was that TFS On prem and early VSTS did NOT support non-Microsoft (.Net) code very well if at all. You could utilize the source control and work features of TFS and then use octopus/Jenkins etc... as the build release parts to cover code that TFS didn't really know what to do with.
Also the release pipelines used to be very simplistic and not that useful where the other products were all plugin based and could do (almost) anything you needed them to. Most of that has changed so that VSTS is much better at working with Non-Microsoft code bases then it used to be. Over time integrations get created inside a companies walls and undoing those decisions can be more painful then just having "too many" tools. Also I feel like there is just more people out there familiar with those tools since they have been mature longer and cover a larger part of the development world then VSTS has in the past.
To fully implement CD you need both. VSTS runs tests and is a build server. OD isn’t. VSTS is light on sophisticated application installations. And if you are provisioning environments, IaC style, you need Terraform in addition. Don’t try to shoehorn everything into a single tool. DevOps requires a whole ecosystem. The reasons are not historical.
Currently I am using sourcetree to manage my repo's. I've read about using different branches for 3 stages of website development. I've seen sites like deployhq.com and beanstalkapp.com. I am curious is there a "free" way to deploy/rollback and publish via sftp/shh to my web servers both staging and live production for each of my repo's. I need a way for junior level developers to easily be able to do it too, be great if there was a review process. If there is no free method, what;s the best paid method out there. I also use bitbucket.org to store my repos.
Thanks so much
You could use a lightweight deployment tool such as Kwatee (self-promotion). Kwatee is free and can deploy any files via a web interface or python scripts (also ant/maven if you use java). All you'd have to do is to automate the check-out of your branch and then trigger the deploy command in kwatee which will use ssh/scp or telnet/ftp behind the scenes to update your deployment (only changes will be deployed).