I've been working on a small project and came across some information that has some sort of encoding (I assume).
7C-FC-1B-C9-97-1B-A9-EB-2E-45-2A-73-CE-E3-17-F9
01-3E-6A-50-09-ED-1C-A1-80-A0-27-B9-0C-D3-C4-9D
89-4C-B3-52-4A-B8-93-CB-95-4F-E2-9A-0C-59-7C-FD
Does anyone know what sort of encoding this is? I looked into UTF-8 since this came from a SQL file. No luck there.
I think that is written in hexadecimal. Not encoded
Here's an interesting puzzle, and I'd welcome any thoughts that anyone has. I don't think that there's a right or wrong answer here.
My program is loading a file which contains (amongst other things) string data structures. It helpfully declares what type the structure is (UTF8, UTF16 etc), and its length (of course) before the structure, so my program knows how to handle the data. Up to now this has worked perfectly every time.
Now I have been given a data file to load which has rubbish on the end of it - and, when I say rubbish, I mean that it looks like UTF8 at the end of a structure which is declared as UTF16.
D·a·v·e···E·d·m·u·n·d·s·o·n·d`upbqp!c°rÞPrpupÎÐAgâh(28.RïSÿ
The Dave Edmundson part is fine - it's everything after that which needs to be trimmed in this case. The fly in the ointment is that I still need to be able to handle legitimate UTF16 extended characters (like Korean, Chinese etc).
I could just fling my hands up in the air and say, "this data is corrupt", and spit out an error. But I'd like to be able to clean it if at all possible. Any ideas that anyone might have would be very welcome!
This is a logical issue, hence no code - if anyone is interested, I'm using Objective-C, but all I'm really after is some intelligent conversation on how I might approach this issue. I don't need code to be written for me!
I am trying to intercept and replace emoji with a corresponding text. I left the default encoding on the Web API (UTF-8 / UTF-16 respectively).
How can I convert an emoji like 😉 to something like U+1F609?
Here is something that helped me out, although it is in Perl. But you can encode and decode. This should be what you're looking for: https://metacpan.org/pod/Encode::JP::Emoji
This is quite an old post and even though I'm not on the project anymore I want to still answer with my findings for future reference if someone else has the same problem.
What I ended up doing is to create a dictionary with key the UTF combination of the emoji and as a value the text. One thing as an advice: I made sure the longest UTF combination, some emoji have 4 or even 5, as the first ones as otherwise some emoji never got reached. Totally not a perfect and future proof solution that I was hoping for but it worked for us and it shipped into production where it has been working since 2017.
Can anyone help me to implement Base58 encoding stored procedure in PostgreSQL.
I've found answer for numbers but I'm looking for similar stored procedure that can accept TEXT or VARCHAR value.
On this very rare occasion I'm going to suggest you don't do this. It will be computationally possible but highly inadvisable.
https://en.wikipedia.org/wiki/Base58
In contrast to Base64, the digits of the encoding don't line up well
with byte boundaries of the original data. For this reason, the method
is well-suited to encode large integers, but not designed to encode
longer portions of binary data.
To put this another way, Base58 is not designed to encode strings / text. Your main alternatives are:
Base64 which if copied manually by a human, the human may make mistakes. Otherwise Base64 is safe to copy / paste
Hexadecimal which is easily copied by humans but significantly longer than Base64
If you feel you really need Base58 and not Base64 then it may be worth editing your requirements into your question. This may help someone give an answer more specific to your requiremnts:
What are these strings you need to convert (examples are preferable)?
Why do they need to be Base58 and not Base64 (what other system are you passing these to)?
If you have binary data that you need to encode, what encoding scheme do you use?
I know about:
Hex encoding. Very simple, but quite verbose, expands one byte to two.
Base 64. Most common, not so verbose, expands three bytes to four.
Base 85. Not common, less verbose again, expands four bytes to five.
Are there any other encoding schemes in common use? If so, what are there advantages and disadvantages?
Edit: This is useful, for example, when trying to store arbitrary data in a cookie. Cookies can only store text, not arbitrary data, so you need to convert it in some way, preferably with a way to convert it back. Further, assume that you are using a stateless server so that you cannot save the state on the server and just put an identifier into the cookie. Of course, if you do this you would also need some way of verifying that what the user is passing back to you is what you passed to the user, for example a signature.
Also, since the current consensus is that you should use base64 since it is widespread, I will also point out that this is what I use... I am just curious if anyone used anything else, and if so, why.
Edit: Just in case someone stumbles across this, if you do want to use Base64 to store data in a cookie, you need to use a modified Base64 implementation. See this answer for the reason why.
For encoding cookie values, you need to be careful. See this older answer:
With Version 0 cookies, values should
not contain white space, brackets,
parentheses, equals signs, commas,
double quotes, slashes, question
marks, at signs, colons, and
semicolons. Empty values may not
behave the same way on all browsers.
Base64 encoding can generate = symbols for certain inputs, and this technically is not permitted in cookies (version 0 cookies, anyway, which are the most widely supported). In practice, I suspect the = will actually work fine, but maybe not.
I would suggest that to be absolutely sure that your encoded binary is cookie-compatible, then basic hex encoding is safest (e.g. in java).
edit: As #Paul helpfully pointed out, there is a modified version of Base 64 that is "URL safe" (and, I assume, "cookie safe"). Using a modified version of a standard algorithm rather dilutes its charm, mind you.
edit: #shoosh pointed out that the = is only used to denote the end of the base64 string, so you could trim the =, set the cookie, then reattach the = again when you need to decode it.
Base64 wins because it's so common that I don't have to ever worry about rolling my own encoder/decoder. I haven't run into any applications where I've been worried about saving bandwidth or filespace in encoded binary data.
Once upon a time, there was UTF-7. It's officially deprecated, but it still works as an ACE (ASCII Compatible Encoding). Now there's IDN.
uuencode is popular is some circles
HTML and XML encode unicode using this syntax
Base64 is the de-facto standard. Using anything else is asking for trouble.