How to set custom path when using New-AzureStorageContext - powershell

I am copying some files to azure blob storage. I am following Microsoft docs https://learn.microsoft.com/en-us/powershell/module/azure.storage/new-azurestoragecontainersastoken and https://learn.microsoft.com/en-us/powershell/module/azure.storage/new-azurestoragecontext to create a SAS token. I have the code working but I can't figure out how to set a custom path for URL that New-AzureStorageContext generates, because I would like to copy to a particular path and not the container root. Is there some sort of flag or something for New-AzureStorageContext that will allow me to set this?
# Set AzStorageContext
$destinationContext = New-AzureStorageContext -ConnectionString "DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xxx;EndpointSuffix=xxx;"
# Generate SAS URI
$containerSASURI = New-AzureStorageContainerSASToken -Context $destinationContext -ExpiryTime(get-date).AddSeconds(3600) -FullUri -Name "xxx" -Permission rw -Protocol HttpsOnly
My issue is that New-AzureStorageContainerSASToken generates a URL the I use for azure copy destination azcopy copy "xxx" $containerSASURI but it copies to the container root and I would like it to copy to a specific directory e.g \test\demo

Assuming you're manually copying the Container SAS URL from PowerShell and using it in azcopy, simplest would be to insert the path in the SAS URL.
For example, if your Container SAS URL looks something like:
https://account.blob.core.windows.net/container-name?sv=2020-08-04&se=2021-10-04T18%3A30%3A00Z&sr=c&sp=w&sig={signature}
and you want to upload the files in test/demo folder inside this container, you will change this SAS URL to something like:
https://account.blob.core.windows.net/container-name/test/demo?sv=2020-08-04&se=2021-10-04T18%3A30%3A00Z&sr=c&sp=w&sig={signature}
and use that in azcopy. All the files are then uploaded into test/demo folder inside container-name blob container.

Related

how to delete subfolder in ADLS gen2 via azure databricks

I need to delete ADLS Gen2 subfolder with specific name using databricks dbutils but not able to perform wild card recursion.
e.g.
adlsstorageaccount/container1/folder2/folder3/folderA/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderB/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderC/**abc**/.parquet
Need to delete subfolder name "abc" & its contents only, entire path is dynamic.
We have reproduced the same Folders in our environment and here are the commands that worked.
$FileSystemName="<Your Container Name>"
$dirname="folder2/folder3/folderA/abc/"
$ctx=New-AzStorageContext -StorageAccountName '<Your Storage Account>' -StorageAccountKey '<Your Access Key>'
Remove-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
and then confirm it with Y - 'yes'
Here are the screenshots for your reference
Before using the command execution
After the command execution
REFERENCES:
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell#delete-a-directory
you need just use magic command %sh in databricks and remove it with wilcard using standard linux commands, something like:
%sh
rm /dbfs/mnt/adlsstorageaccount/container1/folder2/folder3/folderA/*abc*/*.parquet

how do we copy an azure storage account table to another storage account?

I'm attempting to clone my storage tables into a different storage account. What is the best way to do this with powershell?
I've attempted this solution; however, at this point I'm getting this exception:
Copy-AzureStorageTable : The term 'Copy-AzureStorageTable' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the
name, or if a path was included, verify that the path is correct and try again.
How do we copy a table into another storage account?
AzCopy v10 doesn't support Azure Table Storage unfortunately. To export/import data from/to Azure Table Storage, you need to use AzCopy v7.3 instead.
Note that it doesn't support direct Table to Table copy, so you need to export the source table to local disk or Blob Storage at first, then import it to another destination table.
We have written the below PowerShell script that will download all the tables under the storage account to your local & it will upload to the destination storage account which is working fine.
Here is the PowerShell Script:
Connect-azaccount
$strgName='<storageAccountName>'
$stcontext=New-AzStorageContext -StorageAccountName $strgName -StorageAccountKey <StorageAccountKey>
$tablelist=Get-AzStorageTable -Context $stcontext | Select-Object -Property Uri,Name
foreach($table in $tablelist){
$Sourceuri=$table.Uri
cd "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy"
.\AzCopy /Source:$Sourceuri /Dest:C:\Users\Downloads\azcopy1 /SourceKey:<StorageAccountKey>
}
$localist=Get-ChildItem -Path C:\users\Downloads\azcopy1\ -Exclude *.json
foreach( $item in $localist){
$tbname=$item.Name.Replace('<storageaccountName>_','').Replace('.manifest','').Replace('_','').Replace('.','')
$manifest=$item.Name.Replace('C:\users\Downloads\azcopy1\','')
cd "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy" `
.\AzCopy /Source:C:\users\Downloads\azcopy\ /Dest:https://<DestinationStorageAccount>.table.core.windows.net/$tbname/ /DestKey:<DestinationAccountKey> /Manifest:$manifest /EntityOperation:InsertOrReplace
}
Here is the output for reference :

How to get the command line for AZCopy?

I want to send dump files to a storage container and for the copy to work we need to obtain a SAS key for the container we’re copying to.
When you use Azure Storage Explorer you can copy a file to a container and then copy the command it used to the clipboard which looks something like this:
$env:AZCOPY_CRED_TYPE = "Anonymous";
./azcopy.exe copy "C:\temp\test.txt" "https://dbbackups.blob.core.windows.net/memorydumps/test.txt?{SAS-TOKEN}" --overwrite=prompt --from-to=LocalBlob --blob-type Detect --follow-symlinks --put-md5 --follow-symlinks --recursive;
$env:AZCOPY_CRED_TYPE = "";
I copied this from AZ Storage Explorer when copying a file called test.txt from c:\temp to the memorydumps container in a Storage Account.
What I would need help with is creating a PowerShell script that generates the above command line so I can run it on azcopy-less nodes and have the dumps end up in the storage container. Any ideas?
You could use the Azure PowerShell equivalent to upload blobs to your container. The Set-AzStorageBlobContent uploads a local file to an Azure Storage blob.
Set-AzStorageBlobContent -File "C:\Temp\test.txt" `
-Container $containerName `
-Blob "Image001.jpg" `
-Context $ctx
Refer to this blog post for a detailed walkthough: File Transfers to Azure Blob Storage Using Azure PowerShell

Function to move/delete files within file share in Azure Storage Explorer?

I'm not proficient in Powershell yet, so please bear with me if I use the incorrect terminology.(And please correct me if I do.)
I have installed the Az and Azure.Storage modules.
I have also connected to my account using Connect-AZAccount (Is this the best way? Since you need to copy the URL and login via a browser)
Then I was just trying to view the files, to test the connection. Using Get-AzureStorageFile
This prompts me for a sharename - I used the name of the folder under File Shares in Azure Storage Explorer. But this failed, see failure below
cmdlet Get-AzureStorageFile at command pipeline position 1 Supply
values for the following parameters: (Type !? for Help.) ShareName:
bss get-azurestoragefile : Could not get the storage context. Please
pass in a storage context or set the current storage context.
Additional information to note, I do not have access to the Account Key, only the SAS Token.
Any help would be appreciated.
If you use Connect-AzAccount, you will use the Az module powershell Get-AzStorageFile instead of Get-AzureStorageFile. Before running the Get-AzStorageFile command, you need to pass the storage context with New-AzStorageContext to fix the error.
Sample:
$context = New-AzStorageContext -StorageAccountName "<StorageAccountName>" -StorageAccountKey "<StorageAccountKey>"
Get-AzStorageFile -ShareName "<ShareName>" -Path "<ContosoWorkingFolder>" -Context $context

Download Azure VHD to local use powershell

How can I download azure vhd with powershell to local machine?
I read the document, but I can't find the blob url like "https://XXX.blob.core.windows.net/vhds/XXX.vhd"
Anybody know that?
Thanks
According to your description, your VM uses managed disk not unmanaged disk. So, you could not find the VHD file in storage account. More information about managed disk please refer to this link.
If you want to download the VHD in managed disk, you should copy it to a storage account first.
##create $SAS
$sas = Grant-AzureRmDiskAccess -ResourceGroupName shui -DiskName shuitest -DurationInSecond 3600 -Access Read
$destContext = New-AzureStorageContext –StorageAccountName contosostorageav1 -StorageAccountKey 'YourStorageAccountKey'
Start-AzureStorageBlobCopy -AbsoluteUri $sas.AccessSAS -DestContainer 'vhds' -DestContext $destContext -DestBlob 'MyDestinationBlobName.vhd'
Then, you could use Save-AzureVhd or Azcopy to download the VHD to your local.
Please refer to the similar question.
You can use Save-AzureVhd cmdlet to download the vhd file to local machine.
The Save-AzureVhd cmdlet enables download of .vhd images from a blob where they are stored to a file. It has parameters to configure the download process by specifying the number of downloader threads that are used or overwriting the file which already exists in the specified file path.
Save-AzureVhd does not do any VHD format conversion and the blob is downloaded as it is.
Example PowerShell cmdlets:
Save-AzureVhd -Source "http://YourStorageaccountname.blob.core.windows.net/yourContainerName/win7baseimage.vhd" -LocalFilePath "C:\vhd\Win7Image.vhd"
To get the exact URL, Login to Azure Portal, Select Storage Account and the vhd file, then copy the path.
You can also use Azure Storage Explorer to download the vhd file easily.
https://storage-account-name.blob.core.windows.net/container-name/vhd-name.vhd
You can also find the URL in Azure Portal. Open your storage account, go into the container and click the VHD you want. You will see "URL" in "Blob properties" blade.
Try this.
Download, install, login and then browse to your container, select your .vhd and click download.