I've been digging the Kubernetes documentation to try to figure out what is the recommend approach for this case.
I have a private movie API with the follow microservices (pods).
- summary
- reviews
- popularity
Also I have accounts that can access these services.
How do restrict access to services per account e.g. account A can access all the services but account B can only access summary.
The account A could be doing 100x more requests than account B. It's possible to scale services for specific accounts?
Should I setup the accounts as Nodes?
I feel like I'm missing something basic here.
Any thoughts or animated gifs are very welcome.
It sounds like this is level of control should be implemented at the application level.
Access to particular parts of your application, in this case the services, should probably be controlled via user permissions. Similar line of thought for scaling out the services...allow everything to scale but rate limit up front, e.g., account A can get 10 requests per second and account B can do 100x. Designating accounts to nodes might also be possible, but should be avoided. You don't want to end up micromanaging the orchestration layer :)
Related
My current understanding is that if I were to set up a Multi Account Landing Zone ( MALZ) in one region , say for example Ireland, I will still be able to have accounts that can contain resources in other regions ( US , Frankfurt et al ) assuming the guardrails allows .
Is my understanding correct ? I am bit confused when I read this
Single AWS Region. AMS multi-account landing zone is restricted to a single AWS Region. To span multiple AWS Regions, use multiple multi-account landing zone.
https://docs.aws.amazon.com/managedservices/latest/userguide/single-or-multi-malz.html
AWS managed service is a bit of a white-glove service so I'm not familiar how standardised their offering and guard rails are. There's a few different parts that come into play
regions that host your landing zone shared infrastructure, e.g. logging account, control tower, AWS SSO etc.
regions that host shared infrastructure that you deploy into every account managed under the landing zone, e.g. a default VPC (peered to a TGW)
regions that are allowed to be used in managed accounts, e.g. because an SCP on the OU forbids everything else
From my understanding it seems that one AMS multi-account landing zone always operates in a single region for all three of those.
May be a fine restriction for starting out, but my experience with large landing zones (> 500 Accounts) is that you start keeping 1. and 2. locked to a single region, but keep 3. restricted only for governance/compliance reasons (e.g. EU only). That gives teams the freedom to leverage AWS regions the way that makes the most sense to their applications like lambda edge functions, regional s3 buckets etc.
Of course, applications that do need on-premise connectivity have a strong gravity to the region hosting transit gateway. Depending on how your on-prem looks like, larger orgs can later add multiple landing zones or even preferably use a modular landing zone approach with "TGW peerings as a service".
I am looking for a better solution for secrets rotation and found Vault dynamic secrets is a good one. By enabling secrets engine, say to database, the applications / services can lease dynamic secrets.
I noticed every time the application to lease a database secret, Vault creates a new user / account in the DB. I understand, each application / service needs to be a good citizen and uses the secret as per the lease time. However, in a microservices environment, an implementation bug may cause the services to request too many dynamic secrets thus triggering to create too many accounts in the DB.
Is there any way to prevent creating too many accounts? I am just worrying too many accounts may cause problem in the DB.
You could go down the static roles which would created one role with a fixed user name and then vault would just rotate that password when it needs to be rotated.
Here are some docs to get you started
https://www.vaultproject.io/api/secret/databases#create-static-role
https://www.vaultproject.io/docs/secrets/databases#static-roles
Also, a warning from the website:
Not all database types support static roles at this time. Please
consult the specific database documentation on the left navigation or
the table below under Database Capabilities to see if a given database
backend supports static roles.`
I must be able to disable a Kubernetes ingress at any point, and put it back on as easily. Is there a way to have a kind of "feature switch" to enable/disable an ingress at any moment, to stop forwarding traffic to the backend it's pointing to, and let us turn it on again?
I believe that what you are trying to achieve can be solved with Istio (I'm not familiar with other Service Mesh, but I think you can do the same with them as well).
If you take a look at this example, they deploy an app which has 3 different product review pages, and then using Traffic Management - Routing Rules they specify how the traffic will be handled in this product review page.
In the doc they go through some cases like round robin, percentage and user. So you could use the percentage to be your switch.
Lets say that you have backend v1 and v2, you could use traffic management to direct 100% of your requests to v1, change it to v2, and then change to v1 again, as much as you like.
You could also do something like 85% - 15%, and test your new backend just with a small percentage of your requests, evaluate and make the decision to move on, or roll back afterwards.
Is there a way how to apply upload limit for google storage bucket per day/month/year?
Is there is a way how to apply limit on amount of Network traffic?
Is there is a way how to apply limit on Class A operations?
Is there is a way how to apply limit on Class B operations?
I found only Queries per 100 seconds per user and Queries per day using
https://cloud.google.com/docs/quota instructions, but this is JSON Api quotas
(I even not sure what kind of api is used inside of StorageClient c# client class)
For defining Quotas, and by the way SLO, you need to have SLI: Service level indicator. that means to have metrics on what you want to observe.
Here, it's not the case. Cloud Storage haven't indicator on the volume of data per day. Thus, you don't have built in indicator and metrics, ... and quotas.
If you want it, you have to build something by your own. To wrap all the Cloud Storage call in a service that count the volume of blob per days and then you will be able to apply your own rules on this personal indicator.
Of course, for preventing any by pass, you have to deny direct access to the buckets and only grant your "indicator service" to access them. Same things for the bucket creation, to register the new buckets in your service.
Not an easy task...
In our company we are transitioning from a huge monolithic application to a micro-service architecture. The main technical drivers for this decision were the need to be able to scale services independently and the scalability of development - we've got ten scrum teams working in different projects (or 'micro-services').
The transition process is being smooth and we've already started to benefit from the advantages of this new technical and organizational structures. Now, on the other hand, there is a main point of pain that we are struggling with: how to manage the 'state' of the dependencies between these micro-services.
Let's put an example: one of the micro-services deals with users and registrations. This service (let's call it X) is responsible for maintaining identity information and thus is the main provider for user 'ids'. The rest of the micro-services have a strong dependency on this one. For example, there are some services responsible for user profile information (A), user permissions (B), user groups (C), etc. that rely on those user ids and thus there is a need for maintaining some data sync between these services (i.e. service A should not have info for a userId not registered in service X). We currently maintain this sync by notifying changes of state (new registrations, for example) using RabbitMQ.
As you can imagine, there are many Xs: many 'main' services and many more complicated dependencies between them.
The main issue comes when managing the different dev/testing environments. Every team (and thus, every service) needs to go through several environments in order to put some code live: continuous integration, team integration, acceptance test and live environments.
Obviously we need all services working in all these environments to check that the system is working as a whole. Now, this means that in order to test dependent services (A, B, C, ...) we must not only rely on service X, but also on its state. Thus, we need somehow to maintain system integrity and store a global & coherent state.
Our current approach for this is getting snapshots of all DBs from the live environment, making some transformations to shrink and protect data privacy and propagating it to all environments before testing in a particular environment. This is obviously a tremendous overhead, both organizationally and in computational resources: we have ten continuous integration environments, ten integration environments and one acceptance test environment that all need to be 'refreshed' with this shared data from live and the latest version of the code frequently.
We are struggling to find a better way to ease this pain. Currently we are evaluating two options:
using docker-like containers for all these services
having two versions of each service (one intended for development of that service and one another as a sandbox to be used by the rest of the teams in their development & integration testing)
None of these solutions ease the pain of shared data between services. We'd like to know how some other companies/developers are addressing this problem, as we think this must be common in a micro services architecture.
How are you guys doing it? Do you also have this problem? Any recommendation?
Sorry for the long explanation and thanks a lot!
This time I've read your question from different perspective, so here is a 'different opinion'. I know it may be too late but hope it helps with further development.
It looks like shared state is a result of wrong decoupling. In 'right' microservice architecture all microservices have to be isolated functionally rather than logically. I mean all three user profile information (A), user permissions (B), user groups (C) look functionally the same and more or less functionally coherent. They seem to be a single user service with a coherent storage, although it may not look as a micro-service. I don't see here any reasons of decoupling them (or at least you haven't told about them).
Starting from this point, splitting it into smaller independently deployable units may cause more cost and troubles than benefits. There should be a significant reason for that (sometimes political, sometimes just lack of product knowledge)
So the real problem is related to microservice isolation. Ideally each microservice can live as complete standalone product and deliver a well defined business value. When elaborating system architecture we break up it into tiny logical units (A, B, C, etc in your case, or even smaller) and then define functionally coherent subgroups. I can't tell you exact rules of how to do that, perhaps some examples. Complex communication/dependencies between the units, many common terms in their ubiquitous languages so it looks like such units belong to the same functional group and thus to a single service.
So from your example, since there is a single storage you have only way of managing its consistency as you did.
BTW I wonder what actual way you've solved your problem?
Let me try to reformulate the problem:
Actors:
X: UserIds (state of account)
provide service to get ID (based on credentials) and status of account
A: UserProfile
Using X to check status of a user account. Stores name along with link to account
provide service to get/edit name based on ID
B: UserBlogs
Using X in same way. Stores blog post along with link to account when user writes one
Using A to search blog post based on user name
provide service get/edit list of blog entries based on ID
provide service to search for blog post based on name (relies on A)
C: MobileApp
wraps features of X, A, B into a mobile app
provide all services above, relying on well-defined communication contract with all others (following #neleus statement)
Requirements:
Work of teams X, A, B, C need to be uncoupled
Integration environments for X, A, B, C need to be updated with latests features (in order to perform integration tests)
Integration environments for X, A, B, C need to have 'sufficient' set of data (in order to perform load tests, and to find edge cases)
Following #eugene idea: having mocks for each service provided by every team would allow 1) and 2)
cost is more development from the teams
also maintenance of the mocks as well as the main feature
impediment is the fact that you have a monolithic system (you do not have a set of clean well defined/isolated services yet)
Suggested solution:
What about having a shared environment with the set of master data to resolve 3)? Every 'delivered services' (i.e running in production) would be avalailable. Each teams could chose which services they would use from here and which one they would use from their own environment
One immediate drawback I can see is the shared states and consistency of data.
Let's consider automated tests ran against the master data, e.g:
B changes names (owned by A) in order to work on its blog service
might break A, or C
A changes the status of an account in order to work on some permission scenarios
might break X, B
C changes all of it on same accounts
breaks all others
The master set of data would quickly become inconsistent and lose its value for requirement 3) above.
We could therefore add a 'conventional' layer on the shared master data: anyone can read from the full set, but can only modify the objects that they have created ?
From my perspective only the objects uses the services should have the state. Let's consider your example: service X responsible for the user Id, service A responsible for the profile information, etc. Lets assume the user Y that has some security token (that may be created for example by using it's user name and password - should be unique) entries to the system. Then the client, contains the user information, sends the security token to the service X. The service X contains info about user ID linked to such token. In case the new user, the service X creates the new ID and stores it's token. Then the service X returns ID to the user object. The user object asks service A about the user profile by providing user ID. Service A takes the ID and asks service X if that ID exists. Service X sends the positive answer then service A may search the profile information by user ID or ask the user to provide such information in order to create it. The same logic should work with the B and C services. They have to talk each with other but they don't need to know about the user state.
Few words about the environments. I would suggest to use puppets. This is the way to automate the service deploy process. We are using the puppets to deploy the services on the different environments. The puppet script is reach and allows flexible configuration.