There is a fast script or command in batch/powershell to analyse only the duplicate and variable segments in all lines of a txt file and remove them? Example:
input file1.txt:
abcde11234452232131
abcde6176413190830
abcde6278647822786
abcde676122249819113
output file1.txt:
11234452232131
6176413190830
6278647822786
676122249819113
input file2.txt:
11234452232131xyz
6176413190830xyz
6278647822786xyz
676122249819113xyz
output file2.txt:
11234452232131
6176413190830
6278647822786
676122249819113
My script:
#echo off & setlocal enabledelayedexpansion
:startline
set /p first=<#SHA1.txt
set status=notequal
for /f "delims=" %%a in (#SHA1.txt) do (
set second=%%a
if "!first:~0,1!"=="!second:~0,1!" (set status=equal) else (set status=notequal & goto break)
)
if "!status!"=="equal" (
for /f "delims=" %%a in (#SHA1.txt) do (
set second=%%a
echo !second:~1!>>#SHA1.tmp
)
if exist #SHA1.tmp (del #SHA1.txt & ren #SHA1.tmp #SHA1.txt)
goto startline
)
:break
:endline
set /p first=<#SHA1.txt
set status=notequal
for /f "delims=" %%a in (#SHA1.txt) do (
set second=%%a
if "!first:~-1!"=="!second:~-1!" (set status=equal) else (set status=notequal & goto break)
)
if "!status!"=="equal" (
for /f "delims=" %%a in (#SHA1.txt) do (
set second=%%a
echo !second:~0,-1!>>#SHA1.tmp
)
if exist #SHA1.tmp (del #SHA1.txt & ren #SHA1.tmp #SHA1.txt)
goto endline
)
:break
exit
I think that this script is slow to run in multiple files.
What about this (see the explanatory :: comment):
#echo off
::This script assumes that the lines of the input file (provided as command line argument)
::do not contain any of the characters `^`, `!`, and `"`. The lines may be of different
::lengths, empty lines are ignored though.
::The script processes the input file in two phase:
::1. let us call this the analysis phase, which consists of the following steps:
:: * read the first line of the file, store the string and determine its length;
:: * read the second line, walk through all characters beginning from the left and from
:: the right side within the same loop, find the character indexes that point to the
:: first left-most and the last right-most character that do not equal the respective
:: ones in the string from the first line, and store the retreived indexes;
:: * read the remaining lines, and for each one, extract the prefix and the suffix that
:: is indicated by the respective stored indexes and compare them with the respective
:: prefix and suffix from the first line; if both are equal, exit with the loop here
:: and continue with the next line; otherwise, walk through all characters beginning
:: before the previous left-most and after the previous right-most character indexes
:: towards the respective ends of the string, find the character indexes that again
:: point to the first left-most and the last right-most character that do not equal
:: the respective ones in the string from the first line, and update the previously
:: stored indexes accordingly;
::2. let us call this the execution phase, which reads the input file again, extracts the
:: portion of each line that is indicated by the two computed indexes and returns it;
::The output is displayed in the console; to write it to a file, use redirection (`>`).
setlocal EnableDelayedExpansion
set "MIN=" & set "MAX=" & set /A "ROW=0"
for /F usebackq^ delims^=^ eol^= %%L in ("%~1") do (
set /A "ROW+=1" & set "STR=%%L"
if !ROW! equ 1 (
call :LENGTH LEN "%%L"
set "SAV=%%L"
) else if !ROW! equ 2 (
set /A "IDX=LEN-1"
for /L %%I in (0,1,!IDX!) do (
if not defined MIN (
if not "!STR:~%%I,1!"=="!SAV:~%%I,1!" set /A "MIN=%%I"
)
if not defined MAX (
set /A "IDX=%%I+1"
for %%J in (!IDX!) do (
if not "!STR:~-%%J,1!"=="!SAV:~-%%J,1!" set /A "MAX=1-%%J"
)
)
)
if not defined MIN set /A "MIN=LEN, MAX=-LEN"
) else (
set "NXT=#"
if !MIN! gtr 0 for %%I in (!MIN!) do if not "!STR:~,%%I!"=="!SAV:~,%%I!" set "NXT="
if !MAX! lss 0 for %%J in (!MAX!) do if not "!STR:~%%J!"=="!SAV:~%%J!" set "NXT="
if not defined NXT (
if !MAX! lss -!MIN! (set /A "IDX=1-MAX") else (set /A "IDX=MIN-1")
for /L %%I in (!IDX!,-1,0) do (
if %%I lss !MIN! (
if not "!STR:~%%I,1!"=="!SAV:~%%I,1!" set /A "MIN=%%I"
)
if -%%I geq !MAX! (
set /A "IDX=%%I+1"
for %%J in (!IDX!) do (
if not "!STR:~-%%J,1!"=="!SAV:~-%%J,1!" set /A "MAX=1-%%J"
)
)
)
)
)
)
if defined MAX if !MAX! equ 0 set "MAX=8192"
for /F "tokens=1,2" %%I in ("%MIN% %MAX%") do (
for /F usebackq^ delims^=^ eol^= %%L in ("%~1") do (
set "STR=%%L"
echo(!STR:~%%I,%%J!
)
)
endlocal
exit /B
:LENGTH <rtn_length> <val_string>
::Function to determine the length of a string.
::PARAMETERS:
:: <rtn_length> variable to receive the resulting string length;
:: <val_string> string value to determine the length of;
set "STR=%~2"
setlocal EnableDelayedExpansion
set /A "LEN=1"
if defined STR (
for %%C in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
if not "!STR:~%%C!"=="" set /A "LEN+=%%C" & set "STR=!STR:~%%C!"
)
) else set /A "LEN=0"
endlocal & set "%~1=%LEN%"
exit /B
This could maybe be improved further, depending also on the data:
if the length of the first line is fixed, or the line lengths vary in a quite small range, you could avoid the :LENGTH sub-routine call and use a constant value instead; if there is a known maximum length of the common prefix/suffix, the line length is even not needed at all;
instead of reading the file twice (due to the two-pass algorithm), you could read it into memory at the beginning and use these data later; for huge files this might be a bad idea though;
I used several for /L loops to walk through certan character ranges, whose bodies are skipped by some if conditions due to lack of while loops or something like exit for; I could have left them using goto, but then I needed to put these loops in separate sub-routines to not break the outer loops; anyway, for [/L] loops finish iterating in the background even when broken by goto, although faster than executing the body; so together with the slow call and goto, I doublt that I would have gained much speed; depending on the data, pure goto loops could be more efficient as they can be left without any remaining background processing, but of course they also needed to be placed in their own sub-routines;
Remove common prefix and/or suffix of unknown length from a list of strings
This batch takes a quite simplistic (and probaply inefficient) approach
It reads the first line and iterates with a growing prefix over the first 30 chararcters
uses findstr to match the lines | pipes the result to find to get a line count
if the line count doesn't match the files total lines the prefix got to long and
the batch continues to the next step.
the same procedure is then used for the suffix
finally the lines are truncated (even prefix and suffix at the same time)
Pass the file name to process as argument, otherwise file1.txt is default.
:: Q:\Test\2018\06\29\SO_51093137.cmd
#echo off & setlocal enabledelayedexpansion
Set "File=%~1"
If not defined File Set "File=file1.txt"
Echo Processing %File%
:: get number of lines
for /f %%i in ('Find /V /C "" ^<"%File%"') Do Set Lines=%%i
Echo #Lines is %Lines%
:: get 1st line
Set /P "Line1=" < "%File%"
Echo Line1 is %Line1%
:: Iterate Prefixlength until Prefix doesn't match all lines
For /L %%i in (1,1,30) Do (
For /F %%A in ('
Findstr /B /L "!Line1:~0,%%i!" "%File%" ^|Find /C /V "" '
) Do Set "EQ=%%A"
If %Lines% neq !EQ! (Set /A "PrefixLength=%%i -1" & Goto :Break1)
)
:Break1
Echo PrefixLength is %PrefixLength%
:: Iterate Suffixlength until Suffix doesn't match all lines
For /L %%i in (-1,-1,-30) Do (
For /F %%A in ('
Findstr /E /L "!Line1:~%%i!" "%File%" ^|Find /C /V "" '
) Do Set "EQ=%%A"
If %Lines% neq !EQ! (Set /A "SuffixLength=%%i +1" & Goto :Break2)
)
:Break2
Echo SuffixLength is %SuffixLength%
Echo ============
For /f "usebackqDelims=" %%A in ("%File%") Do (
Set "Line=%%A"
If %SuffixLength%==0 (
Echo=!Line:~%PrefixLength%!
) Else (
Echo=!Line:~%PrefixLength%,%SuffixLength%!
)
)
Sample output:
> SO_51093137.cmd file2.txt
Processing file2.txt
#Lines is 4
Line1 is 11234452232131xyz
PrefixLength is 0
SuffixLength is -3
============
11234452232131
6176413190830
6278647822786
676122249819113
Following is probably overcomplicating things but it pushed my limit so a great learning experience for me.
$file1 = #(
,'abcde11234452232131'
,'abcde6176413190830'
,'abcde6278647822786'
,'abcde676122249819113'
)
function Test-EqualChar
{
param (
[Scriptblock] $Expression,
[Object[]] $Sequence,
[int] $i
)
!(($Sequence[1..($Sequence.Length -1)] | % {(&$Expression $_ $i) -eq ($Sequence[0][$i])}) -contains $False)
}
$OneChar = {param($x, $i) $x[$i]}
$start = for($i=0;$i -lt ($file1 | % {$_.Length} | Measure -Minimum | Select -ExpandProperty Minimum);$i++) {
if (!(Test-EqualChar $OneChar $file1 $i)) {$i; break}
}
$file1 | % {$_.Substring($start, $_.Length-$start)}
I'll leave it as an excercise to work out reversing (or padding) the strings to remove equal characters from the end of strings
This solution uses a different approach. IMHO this is the fastest way to process a file.
#echo off
setlocal EnableDelayedExpansion
if "%~1" equ "" echo Usage: %0 filename & goto :EOF
if not exist "%~1" echo File not found: "%~1" & goto :EOF
rem Read first two lines and get their base 0 lengths
( set /P "line1=" & set /P "line2=" ) < %1
call :StrLen0Var len1=line1
call :StrLen0Var len2=line2
rem Extract the largest *duplicate segment* from first two lines
set "maxDupSegLen=0"
for /L %%i in (0,1,%len1%) do (
for /L %%j in (0,1,%len2%) do (
if "!line1:~%%i,1!" equ "!line2:~%%j,1!" (
rem New duplicate segment, get its length and keep the largest one
set /A "maxLen=len1-%%i+1, maxLen2=len2-%%j+1"
if !maxLen2! gtr !maxLen! set "maxLen=!maxLen2!"
for /L %%l in (1,1,!maxLen!) do (
if "!line1:~%%i,%%l!" equ "!line2:~%%j,%%l!" set "dupSegLen=%%l"
)
if !dupSegLen! geq !maxDupSegLen! (
set /A "maxDupSegLen=dupSegLen, maxDupSegPos=%%i"
)
)
)
)
set "dupSeg=!line1:~%maxDupSegPos%,%maxDupSegLen%!"
rem Process the file removing duplicate segments
for /F "delims=" %%a in (%1) do (
set "line=%%a"
echo !line:%dupSeg%=!
)
goto :EOF
Get the length base 0 of a variable
:StrLen0Var len= var
setlocal EnableDelayedExpansion
set "str=!%2!"
set "len=0"
for /L %%a in (12,-1,0) do (
set /A "newLen=len+(1<<%%a)"
for %%b in (!newLen!) do if "!str:~%%b,1!" neq "" set "len=%%b"
)
endlocal & set "%1=%len%"
input1.txt:
abcde11234452232131
abcde6176413190830
abcde6278647822786
abcde676122249819113
output:
11234452232131
6176413190830
6278647822786
676122249819113
input2.txt:
11234452232131xyz
6176413190830xyz
6278647822786xyz
676122249819113xyz
output:
11234452232131
6176413190830
6278647822786
676122249819113
"The rows have variable length and multiple duplicate parts may occur".
input3.txt:
abcde11234452232131
6176abcde4131908abcde30
6278647abcde822786
676122249819113abcde
output:
11234452232131
6176413190830
6278647822786
676122249819113
I need to get the previous working day in a dos batch file, as the files to handle has the date in the filenames, under yyyymmdd and ddmmmyy format.
It seems, according to Dos to get next day date , this is rather complex in pure dos command.
So I turned to calling powershell in dos. so far I get till here:
FOR /f "usebackq" %%i IN (`PowerShell $date ^= Get-Date^; $date.ToString^('yyyyMMdd'^)`) DO SET yyyymmdd=%%i
FOR /f "usebackq" %%i IN (`PowerShell $date ^= Get-Date^; $date.ToString^('ddMMMyy'^)`) DO SET ddmmmyy=%%
FOR /f "usebackq" %%i IN (`PowerShell $date ^= Get-Date^; $date^=$date.AddDays^(-1^)^; $date.ToString^('yyyyMMdd'^)`) DO SET yyyymmddEOD=%%i
FOR /f "usebackq" %%i IN (`PowerShell $date ^= Get-Date^; $date^=$date.AddDays^(-1^)^; $date.ToString^('ddMMMyy'^)`) DO SET ddmmmyyEOD=%%
by doing so, yyyymmdd and ddmmmyy is today's date, while yyyymmddEOD and ddmmmyyEOD is the day before.
However, what shall I do, if i need to get the previous "working day"?
By working day, the best solution would probably like Excel WorkDay function, WORKDAY(start_date, days, [holidays]), but for now, for the sake of simplicity, let's just take Monday to Friday as Working Day. So that if today is Tuesday, then the previous working day is Monday, while if today is Monday, then the previous working day is previous Friday.
There's an example on how to get the "next working day" at Powershell Golf: Next business day , however, it seems run under powershell interface. I tried but did not succeed turn it into a dos batch file.
This might be a bit more readable. You can adjust as you need. No escaping characters are necessary.
#echo off & setlocal enabledelayedexpansion
set days=-0 -1 -3
set formats=yyyyMMdd ddMMMyy
set ps=[DateTime]::Now.AddDays(%%b).ToString('%%a')
REM Get today, yesterday, and three days ago in two date formats
for %%a in (%formats%) do (
for %%b in (%days%) do (
for /f %%p in ('"powershell %ps%"') do set date-%%a%%b=%%p
)
)
REM If today is Monday, subtract 3 for EOD, else subtract 1
for /f %%a in ('"powershell [DateTime]::Now.ToString('ddd')"') do (
if "%%a"=="Mon" (set sub=3) else (set sub=1)
)
REM Change the variable names if you want
set yyyymmdd=%date-yyyymmdd-0%
set ddmmmyy=%date-ddmmmyy-0%
set yyyymmddEOD=!date-yyyymmdd-%sub%!
set ddmmmyyEOD=!date-ddmmmyy-%sub%!
REM Output your variables
set yyyymmdd
set ddmmmyy
Output (for today, Wed, Oct 12):
yyyymmdd=20161012
yyyymmddEOD=20161011
ddmmmyy=12Oct16
ddmmmyyEOD=11Oct16
EDIT
Here is an even simpler way of doing it. It seemed more obvious to me after a night of sleep. =) I didn't replace what I wrote above because it would have invalidated your comment, but I thought it might be useful still, so I added it.
#echo off
REM If today is Monday, subtract 3 for EOD, else subtract 1
if %date:~0,3%==Mon (set EOD=-3) else (set EOD=-1)
REM Get today and last EOB in two date formats
for %%a in (yyyyMMdd ddMMMyy) do (
for /f %%p in ('"powershell [DateTime]::Now.ToString('%%a')"') do set %%a=%%p
for /f %%p in ('"powershell [DateTime]::Now.AddDays(%EOD%).ToString('%%a')"') do set %%aEOD=%%p
)
REM Output your variables
set yyyymmdd
set ddmmmyy
A small script, I made up would look like this:
for /f "usebackq" %%i in (`PowerShell $date ^= Get-Date^; $date.AddDays^(-1^)^; $date.getWeekDay^(^)^;`) DO (
SET weekday=%%i
Goto outer
)
:outer
set weekday=%weekday:,=%
"set weekend="
if "x%weekday%"=="xSunday" set weekend=1
if "x%weekday%"=="xSaturday" set weekend=1
if defined weekend (
echo Friday
) Else (
echo %weekday%
)
pause
This would echo the weekday of yesterday or if yesterday was either Saturday or Sunday it would echo "Friday"
EDIT:
Beeing curious for myself I wrote a version giving out the date if that is needed:
for /f "usebackq" %%i in (`PowerShell $date ^= Get-Date^; $date.AddDays^(-1^)^; $date.getWeekDay^(^)^;`) DO (
SET weekday=%%i
Goto:outer
)
:outer
set weekday=%weekday:,=%
if "x%weekday%"=="xSaturday" (
for /f "usebackq" %%i in (`PowerShell $date ^= Get-Date^; $date.AddDays^(-2^)^; $date.toString^('yyyyMMdd'^)^;`) DO (
SET weekday=%%i
)
Goto:echoWD
) Else (
if "x%weekday%"=="xSunday" (
for /f "usebackq" %%i in (`PowerShell $date ^= Get-Date^; $date.AddDays^(-3^)^; $date.toString^('yyyyMMdd'^)^;`) DO (
SET weekday=%%i
)
Goto:echoWD
) Else (
for /f "usebackq" %%i in (`PowerShell $date ^= Get-Date^; $date.AddDays^(-1^)^; $date.ToString^('yyyyMMdd'^)^;`) DO (
SET weekday=%%i
)
Goto:echoWD
)
)
:echoWD
echo %weekday%
pause
Notice: The output will be in your language, so you might need to change xSunday to xSonntag in Germany for example.
Is probably easier to do but never touch a running system ;)
On Windows, I want to execute something only when the script dies.
Below block didn't help; I think it's because Windows don't support signals.
$SIG{__DIE__} = sub {
qx(taskkill /F /IM telnet.exe);
CORE::die #_;
}
Then I tried this:
END {
qx(taskkill /F /IM telnet.exe);
exit $exit_code;
}
It performed taskkill, but exited with exit code 0. I need to propagate exit_code as we do further processing based on it.
END blocks can set $? to control the exit value.
END {
qx(taskkill /F /IM telnet.exe);
$? = $exit_code;
}
I'd like to redirect my program to a certain set of commands if the cmd window outputs: "Can't find ref_eng...", how can i accomplish this? In the codes below, at the 2nd line containing the For, is the location where the cmd.exe can output "can't find ref_eng" if !refLogPath! doesnt exist. At this point, I'd like to redirect my program elsewhere...
<!logPath! (
For /F "tokens=*" %%R in (!refLogPath!) DO (
if %ERRORLEVEL% NEQ 0 (
ECHO Check certain lines of code
)
set logLine=
set /p logLine=
set refLogLine=%%R
REM Check line by line of log against refLog
REM assume ALL times have been replaced with: "xx:xx:xx"
REM if corresponding lines mismatch
if NOT "!logLine!"=="!refLogLine!" (
Echo.
Echo line below is Incorrect:
set lnCorrect=false
REM output to command line: can be put into .log/.txt later
REM output ANY and ALL incorrect line in log file
ECHO !logLine!
)
)
)
Check refLogPath before the for loop.
Eg:
if !refLogPath!xx == xx (
REM refLogPath is empty or unset
goto :SomewhereElse
) else (
if not exist !refLogPath! (
REM refLogPath is missing
goto :SomewhereElse
)
)
For /F "tokens=*" %%R in (!refLogPath!) DO (
...etc...
I am trying to find way to break / exit from FOR loop, if there are any error occured.
Below is content of batch file.
#echo on
set myfile=D:\sample.txt
FOR /F "tokens=1,2 delims=," %%i in (%myfile%) do call :process "%%i"
:process
set recfile=%1%
echo %recfile%
echo "Step in Test1"
echo %errorlevel%
pause;
exit /B 0
If %errorlevel% NEQ 0 goto :fail1
:fail1
echo "Step in fail1"
pause;
exit /B 9993
:EOF
Sample.txt has multiple records. If there are any error occured then I am expecting to exit the batch file rather then checking the complete sample.txt file. e.g. on statement
echo %recfile%, If I place some wrong command ech %recfile% which is incorrect command then I am expecting that it should go to fail1 level and exit. It's catured the error code successfully and going to fail1 level however after this statment, it's checking the sample.txt file (next record) again. Is there any way, If I can break / exit FOR loop.
Please advice.
Thanks,
Joey's answer is great. I have used it with success. I discovered that you don't have to exit the script though. You can use goto :SomeLabel, where :SomeLabel is a label outside of the loop.
FOR /F "tokens=1,2 delims=," %%i in (%myfile%) do (
if defined exit goto :ParseError
call :process "%%i"
)
#echo SUCCESS: %myfile%
goto :RestOfScript
:ParseError
#echo FAILURE: cannot parse %myfile%
#echo Using defaults...
:RestOfScript
...
You can set a variable, meaning that the complete loop should be aborted and use it like this:
:fail1
echo "Step in fail1"
pause
set exit=1
And you'd change the loop like this:
FOR /F "tokens=1,2 delims=," %%i in (%myfile%) do (
if defined exit (
exit /b 9993
) else (
call :process "%%i"
)
)
(broken into multiple lines for readability).
Since you are just calling a subroutine from the for loop there is no way for this subroutine to exit the loop directly. Hence the workaround with a variable.
You don't need to call a label
set USBDRIVE=SETLOCAL
set exit=ENABLEDELAYEDEXPANSION
FOR %%D IN (C D E F G H I J K L M N O P Q R S T U V W X Y Z) DO (
DIR %%D:\SOURCES\INSTALL.WIM > nul 2>&1 && call set USBDRIVE=%%D: && call set exit=1
if defined exit goto :dd3
)
:dd3