I am facing some trouble in properly visualizing emails extracted from GMail. I use the GMail API to retrieve the messages. This part seems to be working properly and I get the json with the entire message.
Here is a small part of one of the body parts
"mimeType": "multipart/alternative",
"filename": "",
"headers": [
{
"name": "Content-Type",
"value": "multipart/alternative; boundary=001a114710d029267205278f13b9"
}
],
"body": {
"size": 0
},
"parts": [
{
{
"partId": "0.0",
"mimeType": "text/plain",
"filename": "",
"headers": [
{
"name": "Content-Type",
"value": "text/plain; charset=UTF-8"
},
{
"name": "Content-Transfer-Encoding",
"value": "quoted-printable"
}
],
"body": {
"size": 549,
"data": "SGkgUGF1bCwNCg0KQXBvbG9naWVzLCBidXQgSSBmb3Jnb3QgdG8gbWVudGlvbiB0aGF0IHRoZSByZXN0IG9mIHlvdXIgb3JkZXIgaGFzIGJlZW4NCnNlbnQgb3V0IGluIHRoZSBtZWFudGltZQ0KDQpNYW55IHRoYW5rcw0KDQoqS2luZCBSZWdhcmRzKg0KKkJhcmJhcmEgSm9uZXMqDQoqSW50ZXJuZXQgU2FsZXMqDQoNCipPbGQgTWlsbCBTYWRkbGVyeSoNCg0KKnd3dy5zYWRkbGVyeS5iaXogPGh0dHA6Ly93d3cuc2FkZGxlcnkuYml6Lz4qDQoNCipUZWw6ICs0NCAoMCkyOCA5MzM1IDMyNjggPCUyQjQ0JTIwJTI4MCUyOTI4JTIwOTMzNSUyMDMyNjg-Kg0KDQoqVGFrZSBhIFZpcnR1YWwgdG91ciBvZiBvdXIgc2hvcCoNCkdvb2dsZSBwbGFjZXMgaHR0cDovL2dvby5nbC85Y1o5ZDANCipbaW1hZ2U6IElubGluZSBpbWFnZXMgNF1XaXNoaW5nIHlvdSBhIHZlcnnigItbaW1hZ2U6IElubGluZSBpbWFnZXMgM10qDQoqICAgICAgICAgICAgICAgICAgTWVycnkgQ2hyaXN0bWFzKg0KDQoq4oCLICAg4oCLICDigIsgICAgW2ltYWdlOiBJbmxpbmUgaW1hZ2VzIDJd4oCLKg0K"
}
}
So what is the data part encoded with? I am getting confused with the "Content-Transfer-Encoding" ->"quoted-printable". Should I decode the value of the data using a "quoted-printable" decoder or not?
Initially without noticing the "quoted-printable" value, I decoded the data value using DecodeBase64, here is how I am making it
function TViewEmailsForm.DecodeData(aStr: String): String;
var
aStrm: TBytesStream;
aStrStrm: TStringStream;
begin
Result := '';
if aStr = '' then
Exit;
aStrm := TBytesStream.Create(DecodeBase64(aStr));
aStrStrm := TStringStream.Create;
try
aStrm.Position := 0;
aStrStrm.LoadFromStream(aStrm);
Result := aStrStrm.DataString;
finally
aStrm.Free;
aStrStrm.Free;
end;
end;
Using that returns human readable text, however at the end something more is decoded and I don't get what it is. I presume it is some kind of formatting bold text, link, kind of signature but I don't succeed in anyway to show it properly (not sure what to use though as component - RichEdit, HTMLViewer)
The end of the decoded string looks like
.......
*Name of the company*
*website of the company <again the website of the company>*
*Tel: +44 (0)28 9335 3268 <%2B44%20%280%2928%209335%203268
ѓBѓBЉ•ZЩHHљ\ќX[Э\€Щ€Э\€ЪЬ
ѓB‘ЫЫЩЫHXЩ\И‹ЛЩЫЫЛ™ЫОXЦЋYBЉ–Ъ[XYЩN€[›[™H[XYЩ\И
UЪ\Ъ[™И[ЭHH™\ћx "ЦЪ[XYЩN€[›[™H[XYЩ\ИЧJѓBЉ€Y\њћHЪљ\ЭX\КѓBѓBЉё "И8 "И8 "ИЪ[XYЩN€[›[™H[XYЩ\И—x "КѓB›
I have some other messages which pretend to have html body, but again the data is seen in that way. I tried to load this string into the lines of TRichEdit, but had no luck, I tried to use TIdDecoderQuotedPrintable to decode this string, though I am not sure if I have to make it, but some of the characters got replaced by '?' (question marks)
What I am missing here and what is the proper way of visualizing the content of the messages?
After serious research and testing different encoders/decoders I finally managed to properly decode what was encoded in the message.
I used Indy's TIdEncoderMIME found in IdCoderMIME and used the DecodeString method. The HTML messages and bodies are properly decoded too with it.
Hope this will help other people not to spend two days in "fighting" with decoding messages!
EDIT: I noticed that the symbol > comes as ? Maybe there something else which has to be done?
EDIT2: It seems that the encoding of the data is not actually Base64 but Base64Url. On the following link http://blog.marcocantu.com/blog/delphi_facebook_base64_encoding.html you can find interesting post on that. The images are encoded in that way and the standard decoding doesn't work for extracting them.
Whoever knows French can read something here too http://codes-sources.commentcamarche.net/source/51156-base64-base64url-encode-decode
Related
I've been using Kafka Connect for the last couple of months
and recently I included the ActiveMQ source plugin in order to read some JMS topic messages that include a json file inside, put them in a kafka topic and then create a stream/table in Ksqldb that uses as columns some of the keys the json file has.
The thing is though that the plugin inserts the JMS message as text with double quotes so it's not recognized properly in Ksqldb.
I tried various things in configuration in order to fix it but nothing worked so far.
I also want to use json formatting and not Avro in kafka connect (no schema registry running too).
For testing purposes I also tried to send JMS messages by specifying the header content as "application/json" and still no luck.
Here's how my ActiveMQ plugin looks like
"config": {"connector.class":"ActiveMQSourceConnector", "tasks.max":"1", "kafka.topic":"activemq", "activemq.url":"tcp://localhost:61616","activemq.username":"admin","activemq.password":"admin","jms.destination.name":"topic.2","jms.destination.type":"topic","jms.message.format":"json","jms.message.converter":"org.apache.kafka.connect.json.JsonConverter","confluent.license":"","confluent.topic.bootstrap.servers":"localhost:9092"}}
and here's how my Kafka connect configuration looks like
bootstrap.servers=localhost:9092
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=1
config.storage.topic=connect-configs
config.storage.replication.factor=1
status.storage.topic=connect-status
status.storage.replication.factor=1
offset.flush.interval.ms=10000
plugin.path=/opt/kafka_2.13-2.5.0/plugins
Also here's a example of how Kafka consumes the messages
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": "{\"widget\": { \"debug\": \"on\", \"window\": { \"title\": \"Sample Konfabulator Widget\", \"name\": \"main_window\", \"width\": 500, \"height\": 500 }, \"image\": { \"src\": \"Images/Sun.png\", \"name\": \"sun1\", \"hOffset\": 250, \"vOffset\": 250, \"alignment\": \"center\" }, \"text\": { \"data\": \"Click Here\", \"size\": 36, \"style\": \"bold\", \"name\": \"text1\", \"hOffset\": 250, \"vOffset\": 100, \"alignment\": \"center\", \"onMouseUp\": \"sun1.opacity = 39\"} }}\n"
}
If any other info is needed please let me know
Any help would be much appreciated.
UPDATE: Ultimately the best solution would be to somehow be able to configure the connector to not escape the quotes in the payload.
Also unfortunately the escaped quotes are generated from activeMQ itself and are not part of the initial message
So the message would look like this
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": {
"widget": {
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment":
"center"
}
}
Welcome Elen1no1Yami!
Looks to me like the issue is that the text field of the message is a string containing the JSON payload you're interested in, but that payload has its double-quotes escaped with a \ char.
I'm assuming the data in ActiveMQ itself does not have the \ char, but it would be good if you could clarify this.
The approaches I see to solving this are to either:
be able to configure the connector to NOT escape the quotes in the payload. So that the message looks more like:
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": {
"widget": {
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment":
"center"
},
... etc
}
or somehow have ksqlDB handle the message as it is an still get access to the JSON within the text field.
Does that summarise what you're looking for? If so, please update your question to reflect this. (It's good to include such details in your question so that its clear what you're asking.
As for an answer...
I'm no Connect expert, so can't really comment and can't really see anything in the details of the connector's config that may allow you to change the contents of text. Others that know more about Connect may be able to help more.
To be able to access the embedded/escaped JSON in ksqlDB you would first need to remove the escaping. See below for ways to do this using ksqlDB
Using ksqlDB to access escaped JSON
Before we can access the JSON document in text we must remove the escaping.
I can think of two ways of the top of my head:
Write a custom UDF
The best way would be to write a custom UDF 'unescape_json` that could remove the escaping.
-- Import raw stream with value as simple STRING containing all the payload
CREATE STREAM RAW (
message STRING
) WITH (
KAFKA_TOPIC=<something>,
VALUE_FORMAT='KAFKA'
);
-- Use custom UDF to process this and write it back as a properly formatted JSON document:
CREATE STREAM JSONIFIED AS
SELECT MY_CUSTOM_UDF(message) FROM RAW;
If written correctly, the custom UDF approach would not suffer from the potential data corruption issues the REPLACE based solution suffers from.
Using REPLACE to remove escaping
NOTE: this solution is brittle: the character replacement can match and replace things it shouldn't, depending on the content of your message!
Let's work with more simple test data to explain what's needed, e.g we want to convert:
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": "{\"widget\": 10}"
}
To:
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": {"widget": 10}
}
This requires three things:
Replace opening "text": "{ with "text": {
Replace all \" with ".
Replace closing }" with }
We can use the REPLACE function to do this, or the REGEXP_REPLACE function:
-- Import raw stream with value as simple STRING containing all the payload
CREATE STREAM RAW (
message STRING
) WITH (
KAFKA_TOPIC=<something>,
VALUE_FORMAT='KAFKA'
);
-- Use REPLACE to remove reformat:
CREATE STREAM JSONIFIED AS
SELECT
REPLACE(
REPLACE(
REPLACE(message,
'"text": "{', '"text": {'),
'\"', '"'),
'"}', '}')
FROM RAW;
Of course this solution suffers from potentially corrupting your data if your data contains any of the search terms: "text": "{, \" or "} anywhere else in your data, e.g.
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": "{\"widget\": \"hello \\\"} world\"}"
}
Would incorrectly be converted to
{
"messageID": "ID:plato-46377-1596636746117-4:4:1:1:1",
"text": {"widget": "hello \\}world"}
}
This is why a custom UDF would be preferable.
Once you've corrected the contents of your input (and written it to a new topic), then you can import your data as normal:
CREATE STREAM DATA (
messageId STRING,
text STRUCT<Widget INT>
) WITH (
kafka_topic='JSONIFIED',
value_format='JSON'
);
i am trying to use the REST-API of my domino lotus notes server (9.0.1FP8)
It works for simple text-fields but if I try to add an rich-text-item as json, the domino server receives only the text-fields and the richt-text field is empty.
I try it with Postman and a Postrequest to http://localhost/Test/JSON_Vie…s/name/List?form=mainForm,
where I send the json
{
"title":"test rich text",
"artist":"rich texter",
"ntracks":1,
"attachments": {
"type": "multipart",
"content": [
{
"contentType": "multipart/alternative; Boundary=\"0__=4EBB0925DFBB40F18f9e8a93df938690918c4EBB0925DFBB40F1\"",
"contentDisposition": "inline"
},
{
"contentType": "text/plain; charset=US-ASCII",
"data": "test rich text",
"boundary": "--0__=4EBB0925DFBB40F18f9e8a93df938690918c4EBB0925DFBB40F1"
},
{
"contentType": "text/html; charset=US-ASCII",
"contentDisposition": "inline",
"data": "<html><body><font size=\"2\" face=\"sans-serif\"><b>test rich text</b></font></body></html>",
"boundary": "--0__=4EBB0925DFBB40F18f9e8a93df938690918c4EBB0925DFBB40F1"
}
]
}
}
The mask mainForm has the text-fields "title", "artist" , "ntracks" and the rich text-field "attachments".
How can I send an image as json and what is "boundary"?
Have you any idea , what's going wrong?
According to the documentation from IBM you need to use PUT (instead of POST) in order to update the fields.
After experimenting a little bit, I now know what was wrong. If I do the POST on the address http://localhost/Test/JSON_Views.nsf/api/data/documents?form=mainForm it works with the attachment. If I use http://localhost/Test/JSON_Views.nsf/api/data/collections/name/List?form=mainForm&computewithform=true the attachment field is ignored. But maybe you can explain me, what the extra field boundary is and if I can set the contentLocation by the name of my attachment?
I create a script which will embed all posts from my group on my website. To do that, I have to know all possible types of modifier "attachments".
Now I know, that type can be:
share,
video_share_youtube.
But I don't know where I can find all these types? I need this types to create good design in HTML, for example when attachment type is video_share_youtube, I know this attachment is youtube movie. If type is "share" it's just pasted a url into text message. But I know that not all types of attachments.
Here is my code:
$response = file_get_contents("https://graph.facebook.com/$group_id/feed?fields=videos,from,picture,message,full_picture&limit=$limit&access_token=$token");
$response = file_get_contents("https://graph.facebook.com/$group_id/feed?fields=attachments,description,full_picture,picture,story,source,created_time,from,message&locale=pl_PL&limit=$limit&access_token=$token");
$array = json_decode($response, true);
Thanks.
The question is, what are the types you are interested in for your website, as the file types are vast. Basically, I did this.
Created a file with possible formats (.doc, .txt, .pdf, .jpeg etc.,), attach it to the message and observe response from graph api.
The most possible types from mime_type field are,
application/pdf
application/msword
text/plain
image/jpeg
image/gif
"data": [
{
"id": "abcd",
"mime_type": "text/plain",
"name": "Sample 1.txt",
"size": 12,
"file_url": ""
},
{
"id": "abcd",
"mime_type": "text/plain",
"name": "Sample2.txt",
"size": 12,
"file_url": ""
}
]
I have setup a content type with a subject field that has pre assigned values in a dropdown field.
I am using the services api to post new content from a polymer app.
When I POST to the api I send the field structure and value in json but get and error.
"406 (Not Acceptable : An illegal choice has been detected. Please contact the site administrator.)"
Even though the object I am sending matches one of the required values in the field.
Do I need to prefix the value with something? I assume I'm posting to the right place to get that response but don't know why it would accept anything other than the string value.
Here is what I sent to the api which is picked up by my Charles proxy.
{
"node": {
"type": "case",
"title": "my case",
"language": "und",
"field_subject": {
"und": {
"0": {
"value": "subject1"
}
}
},
"body": {
"und": {
"0": {
"value": "my details of subject"
}
}
}
}
}
And here is an example of what I have setup in my Drupal field
subject1| first
subject2| second
subject3| third
subject4| forth
For anyone else with the same problem, this subject is poorly documented, but the answer is simple, my subject did not need the value key despite devel suggesting thats how it would be formatted.
"field_subject": {
"und": [
"subject1"
]
}
I could also shorten my code with "und" being an array.
I have been trying to find encoding for below json strings from a day.
I am getting jSon string like
[
{
"ParentId": "154",
"TopMenuId": "159",
"MainMenuText": "开放时间",
"Language": "6",
"MenuImage": ""
},
{
"ParentId": "154",
"TopMenuId": "166",
"MainMenuText": "СЕРТИФИКАЦИЯ ISO",
"Language": "8",
"MenuImage": ""
}
]
with browser it looks ok. but when i get NSData String Encoding in NSLog it shows,
[
{
"ParentId": "154",
"TopMenuId": "159",
"MainMenuText": "开放时间",
"Language": "6",
"MenuImage": ""
},
{
"ParentId": "154",
"TopMenuId": "166",
"MainMenuText": "СЕРТИФИКАЦИЯ ISO",
"Language": "8",
"MenuImage": ""
}
]
I used almost all CFString Encoding but still do not get success.
Note: When i put NSLog in Browser it looks OK. but when it stores it in xcdatamodeld with string it store as С format.
please help me..
Thanks, in advance.
OK, there are two different kinds of encoding we're talking about here. Any string is first and foremost encoded as byte sequence. I.e. the string "bits" is encoded as these bits in the ASCII encoding:
01100010 01101001 01110100 01110011
That's the kind of encoding we're talking about in code like:
[[NSString alloc] initWithData:... encoding:NSUTF8StringEncoding]
But that's not the problem you have. Your string is encoded in ASCII, but the characters in it are not represented as their actual characters, but as XML/HTML entities. I.e. instead of the letter С encoded in UTF-8, you have the HTML entity С, the byte encoding of which is pretty irrelevant.
You need to either HTML-entity-decode those characters, or not HTML encode them to begin with before sending them. Having HTML entity representations of characters in a JSON string is pretty unusual and superfluous.
Finally, i solved the problem. To convert HTML entity to original string as follows.
1) download content.
2) add GTMNSString+HTML.h .m and NSString+html.h .m files to your project.
3) assume your return string is 'allData' then convert it in original string by using
[allData stringByDecodingHTMLEntities] function.
That's it. got your original string!!