I have a script to copy files from local machine to Azure blob, but my new requirement is to copy half of source files into one blob container and another half into another blob container. Let me know if I can do so using parallel or one after the other. I am using azcopy for now to move these files without splitting and from only one source to one destination.
.\AzCopy.exe /Source:$localfilepath /Dest:$Destinationpath /DestKey:$key1 /S
As I known, if there is a pattern for filtering these file names, you can use the parameter Pattern of AzCopy tool to separately upload them in two times, such as the command as below from the section Upload blobs matching a specific pattern of the offical tutorial if they are named with the a prefix.
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /Pattern:a* /S
Here is the description of the parameter Pattern of AzCopy
/Pattern:"file-pattern"
Specifies a file pattern that indicates which files to copy. The behavior of the /Pattern parameter is determined by the location of the source data, and the presence of the recursive mode option. Recursive mode is specified via option /S.
If the specified source is a directory in the file system, then standard wildcards are in effect, and the file pattern provided is matched against files within the directory. If option /S is specified, then AzCopy also matches the specified pattern against all files in any subfolders beneath the directory.
If the specified source is a blob container or virtual directory, then wildcards are not applied. If option /S is specified, then AzCopy interprets the specified file pattern as a blob prefix. If option /S is not specified, then AzCopy matches the file pattern against exact blob names.
If the specified source is an Azure file share, then you must either specify the exact file name, (e.g. abc.txt) to copy a single file, or specify option /S to copy all files in the share recursively. Attempting to specify both a file pattern and option /S together results in an error.
AzCopy uses case-sensitive matching when the /Source is a blob container or blob virtual directory, and uses case-insensitive matching in all the other cases.
The default file pattern used when no file pattern is specified is . for a file system location or an empty prefix for an Azure Storage location. Specifying multiple file patterns is not supported.
Applicable to: Blobs, Files
If there is a simple pattern for files, you have to manually move them into the directories of their own category or write a simple script to filter them to generate the command strings for uploading. Then you can use Foreach-Parallel in PowerShell to realize the parallel upload workflow to satisfy your needs.
Related
I have some files that are always in use and I want to make a copy of them automatically on another disk. I have tried Shadow Copy, but it does not put them in a specific path so that I can access them and send them in an email automatically
Automatically copy files in use
One way to make a copy of files that are always in use is to use the robocopy command-line utility. robocopy is a robust file copying tool that is built into Windows. It can handle copying files that are in use, and it has a wide range of options that allow you to customize the copy process.
For example, you can use the '/mir' option to mirror the source folder to the destination folder, the /r:n option to retry the copy operation a specified number of times if errors occur, and the '/w:n' option to specify the wait time between retries.
You can also use the '/log' option to create a log file of the copy operation, and the '/np' option to suppress the display of the file-transfer progress.
robocopy C:\SourceFolder D:\DestinationFolder /mir /r:3 /w:3 /log:C:\Robocopy.log /np
You can schedule this command to run at specific intervals using the Task Scheduler. Then you can use this command to copy the files that are in use at specific intervals.
Additionally you can use any language or automation tool to send the files after the copy operation is done
The Goal:
How to use .gitignore to exclude all folders & files except PowerShell $Profile?
The answer should help expand the current exception file list to more files in other subfolders. If possible, reduce wildcards to minimum, and use as specific as possible (full/fuller file path). Why? For instance, Book1.xlsx may exist in multiple subfolders but I want to be able to choose only the specific desired subfolders.
Thanks in advance!
Current status:
On Windows 10 (not Linux Distros):
git init initiated on top level directory C:\. [Please don't suggest to start from other subfolders. Just remain with C:\, as I will include more files to the exception list]
C:\.gitignore containing the below:
# Ignore All
/*
# Exception List [eg. PowerShell's $Profile (please use FULL/FULLER FILE PATH, if possible)]
!.gitignore
!Microsoft.PowerShell_profile.ps1
!Users/John/Documents/WindowsPowerShell/Microsoft.PowerShell_profile.ps1
!Users\John\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1
!C:/Users/John/Documents/WindowsPowerShell/Microsoft.PowerShell_profile.ps1
!C:\Users\John\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1
With the above codes, git status successfully returned only .gitignore as 1 untracked file. Microsoft.PowerShell_profile.ps1 remained missing from untrack listing.
I've tried alternative ways (wildcard, partial subfolder name with pattern, etc) but all failed to return the fuller file path PowerShell $Profile.
Using AzCopy.exe, I am able to copy over an entire container successfully. However, I cannot figure out how to copy over a blob where the name includes a folder structure. I have tried the following:
.\AzCopy.exe /Source:https://sourceaccount.blob.core.windows.net/container /Dest:https://destaccount.blob.core.windows.net/container /SourceKey:sourceKey== /DestKey:destKey== /S /Pattern:CorruptZips/2013/6
While also changing the /Pattern: to things like:
/Pattern:CorruptZips/2013/6/*
/Pattern:CorruptZips/2013/6/.
/Pattern:CorruptZips/2013/6/
And everything just says that there are zero records copied. Can this be done or is it just for container/file copying? Thank you.
#naspinski, there is the other tool named Azure Data Factory which can help copying a folder from a blob storage account to another one. Please refer to the article Move data to and from Azure Blob using Azure Data Factory to know it and follow the steps below to do.
Create a Data Factory on Azure portal.
Click the Copy Data button as below to move to the powercopytool interface, and follow the tips to copy the folder step by step.
Took me a few tries to get this. Here is the key:
If the specified source is a blob container or virtual directory, then
wildcards are not applied.
In other words, you can't wildcard copy files nested in a folder structure in a container. You have two options:
Use /S WITHOUT a pattern to recursive copy everything
Use /S and specify the full file path in your pattern without a wildcard
Example:
C:\Users\myuser>azcopy /Source:https://source.blob.core.windows.net/system /Dest:https://dest.blob.core.windows.net/system /SourceKey:abc /DestKey:xyz /S /V /Pattern:"Microsoft.Compute/Images/vmimage/myimage.vhd"
EDIT: Oops, my answer was worded incorrectly!
Please specify the command without /S:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer1 /Dest:https://myaccount.blob.core.windows.net/mycontainer2 /SourceKey:key /DestKey:key /Pattern:abc.txt
You can find the information from "Copy single blob within Storage account" in http://aka.ms/azcopy .
I'm trying to use Robocopy as part of my backup procedures, whereby backup files are copied from one disk to another (for secondary and tertiary backups). I basically have one key external drive for primary backups, and then various of the backups get siphoned off to other disks, depending on which disk it is. I want to develop a sequence of Robocopy commands in a batch file that I can use in any situation, so that if the Robocopy command tries to copy a folder from the source to a destination drive where there is no matching folder, it automatically FAILS and moves on to the next command. Robocopy by default creates the destination folder if it does not already exist, and I want to switch this default OFF if at all possible.
I am using Windows 7 Enterprise and have approximately 20 files per day for the last 365 days that I need to sort.
All of the files are in the same directory. Each file name also contains the date of the file's creation. The date is in the format MM-DD-YYY and starts at the 29th character of the file name. The files have the .csv extension.
I need to create a batch file to move all of the files with the same date into their own folder and onto another drive on my computer.
#echo off
setlocal EnableDelayedExpansion
for %%a in (*.csv) do (
set fileName=%%~a
set datePart=!fileName:~28,9!
if not exist "D:\!datePart!" md "D:\!datePart!"
move "%%a" "D:\!datePart!"
)
This script extracts the date part of each file name and uses it as the name of the folder to move the file to. If the folder doesn't exist, it is created, then the file is moved to it.
As written, the script iterates over the .csv files in the current directory. This is specified by the mask in the for loop: *.csv. You can change the mask to include a specific path to process, like C:\path\to\*.csv.
The target drive is also hard-coded and assumed to be D:. Change the corresponding entries of the script if you need to use a different drive.
Detail information on every command used in this script can be obtained by calling the command's built-in help from the command prompt, using either of the below syntaxes:
command /?
help command