I can see storage service level and container level access control but Is it possible to assign folder level access control for users and service principals?
No, there isn't a folder level access control in ADLS. We can't assign it for users or service principals.
Azure Data Lake only suppurt: Service, Container, Object level access control.
HTH.
Related
I have 2 different storage accounts with same container name. Lets say tenant1 and tenant2 as storage account name with "appdata" as container name in both accounts. I can create and mount both containers to dbfs. But i am unable to read/write dynamically by passing storage account names to the mount point code. since dbfs has mnt/containername as mount point in dbfs, only latest or previously passed storage account's mount point is being referred in databricks. How to achieve my goal here?
Mount points should be static, so you just need to have two different mount points pointing to the correct container, something like this:
/mnt/storage1_appdata
/mnt/storage2_appdata
so if you want your code be dynamic, use the f"/mnt/{storage_name}_appdata".
It's not recommended to dynamically remount containers - you can get cryptic errors when you remount mount point while somebody is reading/writing data using it.
Also, you can access ADLS directly if you specify correct configuration for your cluster/job (see doc) - you can even access both containers at the same time, just need to setup configuration for both storage accounts:
spark.conf.set("fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net",
"OAuth")
spark.conf.set(
"fs.azure.account.oauth.provider.type.<storage-account-name>.dfs.core.windows.net",
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set(
"fs.azure.account.oauth2.client.id.<storage-account-name>.dfs.core.windows.net",
"<application-id>")
spark.conf.set(
"fs.azure.account.oauth2.client.secret.<storage-account-name>.dfs.core.windows.net",
dbutils.secrets.get(scope="<scope-name>",key="<service-credential-key-name>"))
spark.conf.set(
"fs.azure.account.oauth2.client.endpoint.<storage-account-name>.dfs.core.windows.net",
"https://login.microsoftonline.com/<directory-id>/oauth2/token")
I'm putting together a role/policy for running cloudformation/sam to limit access as much as I can. Is there a general set of policy actions that should be used to run create-stack?
This is for a codebuild which I'm using to create infrastructure using a cloudformation template during runtime of my application.
At the moment I've got a policy which allows full access, because it needs to create the infrastructure within the stack.
But there are only a subset of actions which cloudformation can actually perform and it doesn't need full access. For example, CF can't put items into a dynamodb table.
So this led me to think that maybe there's a basic role/policy that is limited to only the actions which cloudformation is able to perform.
If you're having to assign a role to a service (such as CodePipeline or CodeBuild) to deploy a stack, you do not only need to assign the necessary CloudFormation permissions (such as cloudformation:CreateStack or cloudformation:ExecuteChangeSet) but also permissions necessary for the deployment of the CloudFormation stack itself.
When you are deploying a stack manually, CloudFormation will use your user permissions to verify access to the services you are deploying/updating. When you're initiating the action from another AWS service, the same thing happens, but with the services from the service role. (Unless you are specifically assigning a role to the CloudFormation stack, documentation).
Keep in mind if you're constructing such a role, that CloudFormation might need more permissions than you think, such as extra read permissions, permissions to add tags to resources, permissions to delete and/or update those resources when you're deleting/updating the resources etc.
I am following Zero to JupyterHub with Kubernetes to create a jupyterHub environment for my team to use.
Using Google Kubernetes Engine and every user gets his/her own storage and files are stored - this setup works fine.
I am having trouble as how should I create a volume or shared database so that everyone in team can see each other's notebooks, share file's and data.
To explain more, in desired setup - when a user signs in and goes to his/her jupyter image - every user sees the same folder "shared" and one can create individual folders for themselves inside that folder but are able to reuse code that someone else has already written.
I looked into NFS with Firestore but that seems very expensive.
As in the documentation gcePersistenceDisk do not support multiple read and write.
There is alternative solution for the problem. Rook is a storage backend various storage provisioner available through it. One of them is Ceph which has shared filesystem solution on kubernetes.
It seems that by default a lambda function created by Pulumi has an AWSLambdaFullAccess permissions. This type of access is too wide and I'd like to replace it with fine-grained ACLs.
For instance, assuming I am creating a cloud.Table in my index.js file, I would like to specify that the lambda endpoint I am creating (in the same file) only has read access to that specific table.
Is there a way to do it without coding the IAM policy myself?
The #pulumi/cloud library currently runs all compute (lambdas and containerized services) with a single uniform set of IAM policies on AWS.
You can set the policies to use by running:
pulumi config set cloud-aws:computeIAMRolePolicyARNs "arn:aws:iam::aws:policy/AWSLambdaFullAccess,arn:aws:iam::aws:policy/AmazonEC2ContainerServiceFullAccess"
The values above are the defaults. See https://github.com/pulumi/pulumi-cloud/blob/master/aws/config/index.ts#L52-L56.
There are plans to support more fine-grained control over permissions and computing permissions directly from resources being used in #pulumi/cloud - see e.g. https://github.com/pulumi/pulumi-cloud/issues/145 and https://github.com/pulumi/pulumi-cloud/issues/168.
Lower level libraries (like #pulumi/aws and #pulumi/aws-serverless) provide complete control over the Role and/or Policies applied to Function objects.
I've got a Web Role on Azure with 2 Applications and 1 Virtual Directory.
1 Application is a backend, where admins can upload files, which are stored in the virtual directory (which is accessed by both applications).
Everytime I deploy a new version to Azure, all the uploaded content in the virtual directory is deleted - this is what I don't want!
So how is it possible to publish a new version without deleting all my user generated files?
I've already managed to update the application with WebDeploy. But this is only possible for the "main" application, and not the 2nd application (which is configured as a Virtual Application).
Thanks
You can't. The web role is recreated on deployment. It may also occur on hardware failure, azure redeploys your system if an instance fails. Redeploys a clean virtual machine and then deploys your app to it. You should never store data you want to keep on a web role. You need to use blob storage etc to store files you want to persist.
Virtual directories are stored on "Application" partition which is recreated on each upgrade - see this for more information. So the virtual directory folder is not the right place to store stuff you want preserved across upgrades. BTW the "Application" partition only has 1 gigabyte of space and some of that is used for storing your application binary code so you may find yourself in a "disk full" situation at some moment.
If you want to store some data which you don't mind sacrificing on rare occasions - like cached results - you may use "local resources" disk for that which will survive in-place upgrades and reboots. However it is not guaranteed to be preserved if your VM crashes - for such level of preservation you have to use persistent storage like blob storage for example.
Since you are talking about virtual directories and using web deploy to update application outside of the usual Azure package deployment mechanism, it sounds like your architecture/application might be more suited to a persistent VM role rather than a Web role. These are available on Azure in preview only at the moment.
http://www.windowsazure.com/en-us/home/scenarios/virtual-machines/
They let you have persistent storage that will survive a recycle. The storage is actually backed by blob storage, but it looks like a normal disk from the PVM.