I have this little script running on a windows machine that syncs down files from google storage bucket and sends the new files to a printer hot folder. The file disappears from printers hot directory as soon as it is printed. As you can see the script uses a backup directory to compare each and every file to make sure only new files are sent to the printer. This solution works well, but clearly not very efficient for large volume of files. Just wondering if rsync has options to copy only new files from bucket since the last run.
#echo off
SET SOURCE=%1
SET DESTINATION=%2
SET HOTDIR=%3
SET BACKUPDIR=%4
SET GSUTIL_INST_DIR=C:\PRINT
SET PARENT_DIR=C:\PRINT
SET GSUTIL=%GSUTIL_INST_DIR%\google-cloud-sdk-386.0.0-windows-x86_64-bundled-python\google-cloud-sdk\bin\gsutil
SET GSUTIL=%GSUTIL% -m
call %GSUTIL% rsync -d -C %SOURCE%/ %DESTINATION%/
:: Compare each file before sending it to hot directory
for /f %%F in ('dir /b "%DESTINATION%"') do (
if not exist "%BACKUPDIR%\%%F" (
XCOPY /Y /F "%DESTINATION%\%%F" "%HOTDIR%\%%F*"
)
)
robocopy %DESTINATION% %BACKUPDIR% /MIR
SET CONFIRMATION= "%DATE% %TIME% %SOURCE% to %HOTDIR%"
Just wondering if rsync has options to copy only new files from bucket
since the last run.
Below solution may work for your scenario:
Maintain last script run timestamp history in some local file
List the GCS objects(files) in sorted order by date
gsutil ls -l gs://your-bucket/ | sort -k 2
The above command will give the timestamp of the objects added, so you can pipe the output and filter the object list based on the timestamp stored in history file.
Related
I am creating a batch file to be run later which will be used to copy files from one location to another while renaming the files, skipping any existing files, and without prompting the user. Examples of files to be copied:
00021001.txt
00021001.xyz
00021001.abc
00021001001.jpg
Copied files will have the names:
00022001.txt
00022001.xyz
00022001.abc
00022001001.jpg
Things I have tried:
xcopy C:\Testing\1000012\21\00021*.* C:\Testing\1000013\22\00022*.* /D
This almost does it. It copies all the files starting with "00021" in the first location into the second location while properly renaming them to start with "00022". It skips all the files with the same name and date stamp, but ends up prompting to copy any files from the source which are newer than the target.
robocopy C:\Testing\1000012\21\ C:\Testing\1000013\22\ 00021*.* /xo /xn /xc
I was hoping that by excluding older, newer, and same date files it would work (even if it doesn't rename - I would just do that in a separate step.) Unfortunately, this just ends up overwriting newer source files over existing target files if they are a different filesize.
I have even tried the Copy-Item command in PowerShell. But it doesn't do the renaming like Xcopy, and it doesn't skip existing files (although I can get it to confirm and say "No to All".)
Copy-Item -Path "C:\CWUImageCompare\Testing\1000012\CWU\chemistry\129\21\00021*.*" -Destination "C:\CWUImageCompare\Testing\1000013\CWU\chemistry\129\20\" -Confirm
If xcopy had the "Skip if existing" flag I'd be all set, but it doesn't.
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
rem The following settings for the source directory & destination directory are names
rem that I use for testing and deliberately includes spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
SET "destdir=u:\your results"
FOR /f "skip=1delims=" %%b IN ('xcopy /L /Y "%sourcedir%\00021*.*" "%destdir%\00022*.*" ^|sort') DO (
SET "oname=%%~nxb"
IF EXIST "%destdir%\00022!oname:~5!" (ECHO "%%b" skipped) ELSE (ECHO COPY /y "%%b" "%destdir%\00022!oname:~5!")
)
GOTO :EOF
Always verify against a test directory before applying to real data.
Seems a little complicated, but essentially, execute the xcopy (I omitted the /D for testing) with /L /Y to simply produce a list.
Since the list has a last line that begins with a numeric, whereas the other lines start with a drive-letter, sort the list and skip the first line.
This would implement the date-requirement.
Then grab the part after the first 5 characters of the name+extension, test whether the new name exists and either report or copy as appropriate (The copy command is disarmed for testing)
I would like to use a batch file to copy files in a folder to a backup folder (OLD) within, on a Windows server. And I want to do this for multiple folders.
So for example, I have multiple folders, named like this:
C:\A01
C:\A02
C:\A03
...
C:\A50
I will nickname them here as A##.
Each folder has hundreds of files.
Then each year before updating them, I copy those files into a backup folder called OLD, for each A##, so I have a backup of last year's files. Then I copy in new files into C:\A##, in another manual process, individually.
So there will be folders like:
C:\A01\OLD
C:\A02\OLD
C:\A03\OLD
...
C:\A50\OLD
When I do the copy, to the OLD folder, I keep the most recent version of each file. But then the files that I later manually copy in will overwrite in the C:\A## folder and be this year's files, while C:\A##\OLD will contain last year's files.
How can I script this so it will copy all the files for each C:\A## folder into their respective \OLD folder?
I assume I would use a variable for the path I want to copy from and to.
I could use Robocopy or Xcopy.
in a script file
#echo off
cd c:\
for /f "usebackq delims=:" %%a IN (`dir /AD /B ^| FINDSTR /I /R "A[0-9]*"`) DO (
#mkdir "%%a\OLD"
xcopy "%%a\*.*" "%%a\OLD\" /ECIFHRY /D
)
it copy only the updated files (/D) to the OLD subfolder.
if you want to mantain ACLs of the files, use /ECIFHRKOXY
I suggest you to test it in a test folder, that you can generate with this commands directly in the shell
#mkdir c:\test76
cd c:\test76
For /L %a IN (1,1,10) DO #mkdir A%a && echo "the quick brown fox jumps over the lazy dog">A%a\file.txt
I have a daily process which generates some zip files out of files that are being created by other processes. I need to create a daily log file which indicates the timestamps the contents of one specific file of each zip file that is found.
I created the following batch script which seemed to work yesterday on my test system, but not anymore today, no idea why...
set VersionDirectory=C:\Test\VersionX\
set ResultOutputFile=C:\Test\LogFile.txt
for /f %%f in ('dir /b %VersionDirectory%\Installable\Packages\pattern*.zip') do (
mkdir %temp%\%%f\
unzip -extract -dir %VersionDirectory%\Installable\Packages\%%f %temp%\%%f\ > nul
for %%a in (%temp%\%%f\InstallScript.xml) do set InstallScriptXMLDate=%%~ta
rmdir /s /q %temp%\%%f\
echo %%f [package from %InstallScriptXMLDate%] >> %ResultOutputFile%
)
Short summary of what this file is supposed to do:
Loop through each pattern*.zip file in C:\Test\VersionX\ directory
unzip this file to the %temp%\%%f directory (where %%f is the filename)
Get the timestamp of the %temp%\%%f\InstallScript.xml and put it in the %InstallScriptXMLDate% variable
Delete the %temp%\%%f directory
Echo the filename (%%f) and timestamp (%InstallScriptXMLDate%) into the log file
As of now the log file just contains the filenames, followed by the string '[package from ]' string, but missing the actual date timestamp
The unzipping and removing of the zip files is working flawlessly, it's just the timestamp that's not being set.
You are setting a variable and using it in the same block. This cannot work in cmd because environment variables are expanded when a statement is parsed not when it's executed. So when the loop is run all variables have already been replaced with the values they had before the loop.
Put
setlocal enabledelayedexpansion
at the start of your batch and use !InstallScriptXmlDate! instead of %InstallScriptXmlDate%.
Another note: for is perfectly capable of iterating over files by itself, you almost never need to iterate over dir /b output with for /f. In fact, it can introduce problems that can be avoided with
for %%f in (%VersionDirectory%\Installable\Packages\pattern*.zip)
I have a job that periodically runs and archives files into a folder structure that looks like this:
ArchiveFolder
TimestampFolder
JobnameFolder
job<timestamp>.xml
For a given job, I'd like to collect all xml files in the archive folder into a flat directory (no subdirectories, just all the files) without having to drill down into each one, examine for the proper job, then copy the file.
It seems there should be a fairly straigtforward way of doing this. Any suggestions?
EDIT: I guess I wasn't clear here. The TimeStampFolder will have a name of something like 2011-07-24, the JobnameFolder will have a name like FooFeed or BarFeed, and the job file will have a name like job2011-07-24.xml. There are hundreds to thousands of TimeStampFolders, and each one may have one or more job folders in it. Given a specific job name, I want to collect all the files in all the directories that match that job type, and dump them into the local folder, with no subdirectories.
EDIT1:
SET JOB=JobName
SET OF=OutputFolder
START /wait NET USE Z: "\\ServerName\Sharename\ArchiveFolder" password password_here /USER:domainname\username /P:NO
PUSHD Z:\
FOR /F "USEBACKQ tokens=*" %%A IN (`DIR /b /a:d /s ^| FIND /I "%JOB%"`) DO (
FOR /R %%F IN (%%A) DO (
COPY /Y "%%~fF" "%OF%"
)
)
POPD
It basically locates each subdirectory of ArchiveFolder that includes the JobName in it, then digs into each one that it finds to copy the files out of them.
EDIT2:
Added NET USE to access your network share to perform tasks on the files. If your local machine already has the UNC assigned to a driveletter, you can remove the NET USE command line and change Z: to the assigned driveletter.
#ECHO OFF
FOR /R %%v IN (job*.xml) DO COPY "%%v" c:\out\
I'm sending commands to a remote computer in order to have it copy a file.
I want the file to be copied, but not to overwrite the previous file with the same name (if it exists).
I also need the command to run without any prompts (xcopy likes to prompt whether the target name I've specified is file or directory, and it will also prompt about overwriting a file).
I have good results with xcopy /d.
It will copy NEWER files, and since we can assume that existing files have same time-stamp, you will copy only files that don't exist.
just in case anyone else finds this:
robocopy x:\sourcefolder Y:\destfolder /s /e /r:0 /z
much better than xcopy, even gives you a table at the end informing of any failed or skipped files. Doesn't prompt to not overwrite.
Well, there's a certain remedy! It has helped me with saving much of my effort and time on Win10 while writing a setup for our product demo.
Just try to use piping:
#ECHO N|COPY /-Y SourceFiles Destination
As an example I used this piece of code so that I would have a clean gentle quiet and safe copy!
#FOR /D %%F in ("FooPath") DO #(
#ECHO N|COPY /-Y ^"%%~npdxF\*.*^" ^"GooPath^" 3>NUL 2>NUL >NUL
)
where obviously FooPath is the source and GooPath is the destination.
Enjoy!
(main source: https://ss64.com/nt/copy.html)
Following command copy files and folder but not override file if already exist.
xcopy "*.*" "C:\test\" /s /y /d
No way to make it NOT overwrite as far as I know. but /Y will make it overwrite. and /I will get rid of the file/dict prompt. See xcopy /? for all options
You can also use the replace command. It has two modes: to add files that don't exist there or replace files that do exist. You want the previous mode:
replace <path1> <path2> /A
I had to copy AND rename files, so I got the prompt about creating a file or a directory.
This is the, rather "hackish" way I did it:
ECHO F | XCOPY /D "C:\install\dummy\dummy.pdf" "C:\Archive\fffc810e-f01a-47e8-a000-5903fc56f0ec.pdf"
XCOPY will use the "F" to indicate it should create the target as a file:
C:\install>ECHO F | XCOPY /D "C:\install\dummy\dummy.html" "C:\Archive\aa77cd6e-1d19-4eb4-b2a8-3f8fe60daf00.html"
Does C:\Archive\aa77cd6e-1d19-4eb4-b2a8-3f8fe60daf00.html specify a file name or directory name on the target
(F = file, D = directory)? F
C:\install\dummy\dummy.html
1 File(s) copied
I've also verified this command leaves existing files alone. (You should too :-)