I have directories that looks like this in my MacOs:
For example ???9-24_v_hMrgprx2 where ??? is actually
white spaces. What I want to do is to use rename to remove those leading white spaces.
I tried this but failed.
rename "s/\s*//g" *
What's the right way to do it?
Update
Hexdump looks like this:
ls | hexdump -C
00000000 e3 80 80 39 2d 32 34 5f 76 5f 68 4d 72 67 70 72 |...9-24_v_hMrgpr|
00000010 78 32 0a |x2.|
00000013
Verify what those characters are first, since macOS doesn't display ASCII whitespace characters in filenames as ? (unless you have some weird encoding issue going on). It would help if you added information like this to your question:
$ touch " touched"
$ ls -l *touched
-rw-rw-r--# 1 brian staff 0 Aug 18 13:52 touched
$ ls *touched | hexdump -C
00000000 20 20 20 74 6f 75 63 68 65 64 0a | touched.|
0000000b
For rename, you've almost got it right if those leading characters were whitespace. However, you want to anchor the pattern so you only match whitespace at the beginning of the name:
rename 's/\A\s+//' *
Now that we know your filenames start with U+3000 (which is whitespace), I can see what's going on.
There are various versions of rename. Larry Wall wrote one, #tchrist wrote one based on that (and I use that), and File::Rename is another modification of Larry's original. Then there is Aristotle's version.
The problem with my rename (from #tchrist) is that it doesn't interpret the filenames as UTF-8. So, U+3000, looks like the three bytes you see: e3 80 80. I'm guessing that your font might not support any of those. There could be all sorts of things going on. See #tchrist's Unicode answer.
I can create the file:
% perl -CS -le 'print qq(\x{3000}abc)' | xargs touch
I can easily see the file, but I have a font that can display that character:
$ ls -l *abc
-rw-rw-r-- 1 brian staff 0 Aug 22 02:44 abc
But, when I try to rename it, using the -n for a dry run, I get no output (so, no matching files to change):
$ rename -n 's/\A\s+//' *abc
If I run perl directly and give it -CSA to treat the standard file handles (-CS) and the command-line arguments (-CA) as UTF-8, the file matches and the replacement happens:
$ perl -CSA `which rename` -n 's/\A\s+//' *abc
rename abc abc
So, for my particular version, I can edit the shebang line to have the options I need. This works for me because I know my terminal settings, so it might not work everywhere for all settings:
#!/usr/bin/env perl -CSA
But the trick is how did I get that version of rename? I'm pretty sure I installed some module from CPAN that gave it to me, but what? I'd supply a patch if I could.
E3 80 80 is the UTF-8 encoding of U+3000 which is a CJK whitespace character.
rename is not a standard utility on MacOS, and there are several popular utilities with this name, so what exactly works will depend on which version you have installed. The syntax looks like you have this one, from the Perl distribution. Maybe try
rename 's/\xe3\x80\x80//' *
Related
I am looking for non printable characters into a file, and I found this web page.
It shows the following command:
sed "l" file
If I am not mistaken, according to man, this option is:
List out the current line in a ''visually unambiguous'' form.
Moreover, when I run this command on a fake file with one line, the output is as follow:
The line is displayed twice, but each displayed line (in the output) contains at most 69 bytes of the input line. The rest of the line is displayed at the next line.
The second time the line is displayed, it is in its full length.
fake file
toto, titi, tatafdsfdsfdgfgfdsgrgdfgzfdgzgffgerssssssssssssssssssssssssss
Command
sed "l" fake_file
output
$ sed "l" fake_file
toto, titi, tatafdsfdsfdgfgfdsgrgdfgzfdgzgffgerssssssssssssssssssssss\
ssss$
toto, titi, tatafdsfdsfdgfgfdsgrgdfgzfdgzgffgerssssssssssssssssssssssssss
Questions
What does ''visually unambiguous'' exactly mean ?
Why is the output like this ? I was expecting only one line with the $ sign at the end. I was also not expecting output to be displayed on 69 bytes max.
Environment
Tested with same output on:
sed (GNU sed) 4.7
sed (GNU sed) 4.2.2
By default, sed outputs the result after processing a line. If you handle the output yourself, tell sed not to output the line by the -n switch.
Lets say I have two files as below :
$ ll
total 8
-rw-rw-r--. 1 matias matias 6 Nov 27 20:25 ascii.txt
-rw-rw-r--. 1 matias matias 8 Nov 28 21:57 unicode.txt
Both contain a single line of text, but there is an extra character in the second file as shown here ( Greek letter Sigma ) :
$ cat ascii.txt
matias
$ cat unicode.txt
matiasΣ
If I pass them through file command this is the output :
$ file *
ascii.txt: ASCII text, with no line terminators
unicode.txt: UTF-8 Unicode text, with no line terminators
Which seems ok. Now If I make an hexdump of the file I get this :
$ hexdump -C ascii.txt
00000000 6d 61 74 69 61 73 |matias|
00000006
$ hexdump -C unicode.txt
00000000 6d 61 74 69 61 73 ce a3 |matias..|
00000008
So, my question is, how does an application as cat know that the last two bytes are actually a single Unicode character. If I print the last two bytes individually I get:
$ printf '%d' '0xce'
206
$ printf '%d' '0xa3'
163
Which in extended ASCII are :
$ py3 -c 'print(chr(206))'
Î
$ py3 -c 'print(chr(163))'
£
Is my logic flawed? What Am I missing here?
Command-line tools work with bytes – they receive bytes and send bytes.
The notion of a character – be it represented by a single or multiple bytes – is a task-specific interpretation of the raw bytes.
When you call cat on a UTF-8 file, I assume it just forwards the bytes it reads without caring about characters.
But your terminal, which has to display the output of cat, does take care to interpret the bytes as characters and show a single character for the byte sequence 206, 163.
From its configuration (locale env vars etc.), your terminal apparently assumes that text IO happens with UTF-8.
If this assumption is violated (eg. if a command sends the byte 206 in isolation, which is invalid UTF-8), you will see � symbols or other text garbage.
Since UTF-8 was designed to be backwards-compatible to ASCII, ASCII text files can be treated just like UTF-8 files (the are UTF-8).
While cat probably doesn't care about characters, many other commands do, eg. the wc -m command to count characters (not bytes!) in a text file.
Such commands all need to know how UTF-8 (or whatever your terminal encoding is) maps bytes to characters and vice versa.
For example, when you print(chr(206)) in Python, then it sends the bytes 195, 142 to STDOUT because:
(a) it has figured out your terminal expects UTF-8 and (b) the character "Î" (to which Unicode codepoint 206 corresponds) is represented with these two bytes in UTF-8.
Finally, the terminal displays "Î", because it decodes the two bytes to the corresponding character.
How do applications know character encoding?
Either:
(They guess—perhaps with heuristics. This isn't "knowing".)
They tell you exactly which one to use (via documentation, standard, convention, etc). (This isn't really "knowing" either.)
They allow you to tell them which one you are using.
It's your file; You have to know.
I need to generate an md5 hash for a list of tif and jpg images.
The hash must be inserted in an XML file with metadata about each image, which will be used for the digitalisation of the document.
The use of the md5 hash was not my decision, but a formal requirement of a standard based on the Dublin Core for the digitalisation of these kinds of documents. Xml file, md5 tag is underlined
I am currently generating each md5 hash using Windows built-in Certutil program from the command prompt.
My question is simple: am I doing this right?
I know the process is slow, but the list is short.
Certutil hash function
it looks OK, remember that specifying hash algorithm (MD5) works from windows 7 and up (older windows throw and error) and must be in uppercase.
You can also add find /v "hash" to get only hash itself
like this
certUtil -hashfile pathToFileToCheck MD5 | find /v "hash"
for example, running on windows 8, i got this output
C:\Users\xxxx\Documents>certutil -hashfile innfo MD5
MD5 hash of file innfo:
67 4b ba 79 42 32 d6 24 f0 56 91 b6 da 41 34 6d
CertUtil: -hashfile command completed successfully.
and with find /v "hash" i've got
C:\Users\xxxx\Documents>certutil -hashfile innfo MD5
67 4b ba 79 42 32 d6 24 f0 56 91 b6 da 41 34 6d
the find trick is to exclude (/v parameter) following string "hash" and you have to specify the string in double quotes.
As the first and last line around hash itself contain word hash, you have clean output
my version works from cmd not powershell
I often need to create a hash file to place on an FTP service.
These hash files must contain the name of the original file to allow automatic verification that is done by various tools.
For example, if you have a file named foo.txt, it is necessary to have a file foo.txt.md5 with the following content:
a3713593c5edb65c8287eb6ff9ec4bc0 *foo.txt
The following batch does the job
for %%a in (%1) do set filename=%%~nxa
#certutil -hashfile "%1" md5 | find /V "hash" >temp.txt
for /f "delims=" %%i in (temp.txt) do set hash=%%i
echo %hash% *%filename%>%1.md5
del temp.txt
You can replace md5 for sha256 or 512 if you need. The CERTUTIL supports.
how to use "grep" command to find a match and to print followup of 10 lines from the match. this i need to get some error statements from log files. (else need to download use match for log time and then copy the content). Instead of downloading bulk size files i need to run a command to get those number of lines.
A default install of Solaris 10 or 11 will have the /usr/sfw/bin file tree. Gnu grep - /usr/sfw/bin/ggrep is there. ggrep supports /usr/sfw/bin/ggrep -A 10 [pattern] [file] which does what you want.
Solaris 9 and older may not have it. Or your system may not have been a default install. Check.
Suppose, you have a file /etc/passwd and want to filter user "chetan"
Please try below command:
cat /etc/passwd | /usr/sfw/bin/ggrep -A 2 'chetan'
It will print the line with letter "chetan" and the next two lines as well.
-- Tested in Solaris 10 --
I'm playing with bash, experiencing with utf-8 encoding. I'm new to unicode.
The following command (well, their output) surprises me :
$ locale
LANG="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_ALL=
$ printf '1\né\n12\n123\n' | egrep '^(.|...)$'
1
é
12
$ touch 1 é 12 123
$ ls | egrep '^(.|...)$'
1
123
Ok. The two egrep filters lines with one or three characters. Their input is quite similar, but the output differs with the character é. Any explanation?
More details on my environment :
$ uname -a
Darwin macbook-pro-de-admin-6.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386
$ egrep -V
egrep (GNU grep) 2.5.1
Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Any variable length encoding can mess with tools that is not aware of the encoding, and considers bytes, not characters, when you use single-character wildcards (because the tool assumes that byte=character). If you use literal characters, then for UTF-8, it doesn't matter since the structure of UTF-8 prevents matches in the middle of a character (assuming proper encoding).
At least some versions of grep are supposed to be UTF-8 aware, according to http://mailman.uib.no/public/corpora/2006-December/003760.html, GNU grep 2.5.1 and later is included there as long as an appropriate LANG is set. If you use an older version, however, or something other than GNU grep, that would likely be the cause of your problem, since é is a two-byte character (0xC3 0xA9).
EDIT: Based on your recent comment, your grep is probably Unicode-aware, but it does not perform any sort of Unicode normalization (and I wouldn't really expect it to, to be honest).
0x65 0xCC 0x81 is an e, followed by COMBINING ACUTE ACCENT (U+0301). This is effectively two characters, but it's rendered as one due to the semantics of combining characters. This then causes grep to detect it as two characters; one for the e and one for the accent.
It seems likely that decomposed Unicode is how the file name is actually stored in your file system - otherwise, you could store files that, for all intent and purposes, have the exact same name, but only differ in their use of combining characters.