Does anyone know how to calculate the MD5 hash that is needed to be used with Amazon's SubmitFeed API? I am using ColdFusion and every time I calculate the MD5 hash on my end it never matches what Amazon calculates.
Amazon responds with this error:
ContentMD5DoesNotMatch
the Content-MD5 HTTP header you passed for your feed (C7EF1CADB27497B46FCD6F69516F96E0) did not match the Content-MD5 we calculated for your feed (x+8crbJ0l7RvzW9pUW+W4A==)
I am using the built-in function that ColdFusion uses for hashing (example hash(myStr)). Is there a step I am missing?
public any function EncryptSignature(required string argValue,required string publicKey) hint="I create my own signature that I will matching later." {
local.filters=StructNew();
local.filters["F:publicKey"]=arguments.publicKey;
var jMsg=JavaCast("string",arguments.argValue).getBytes("iso-8859-1");
var thisSecretKey = getDAO().getSecretKey(local.filters).apiSecretKey;
var jKey=JavaCast("string",thisSecretKey).getBytes("iso-8859-1");
var key=createObject("java","javax.crypto.spec.SecretKeySpec");
var mac=createObject("java","javax.crypto.Mac");
key=key.init(jKey,"HmacSHA1");
mac=mac.getInstance(key.getAlgorithm());
mac.init(key);
mac.update(jMsg);
return lCase(binaryEncode(mac.doFinal(),'Hex'));
//return Encrypt(arguments.argValue,getapiUsersDAO().getSecretKey(arguments.publicKey),'HMAC-SHA1');
}
The problem is that Feed must have preceding
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
Even adding it as a Declaration is not enough
var memoryStream = new MemoryStream();
XDocument doc = new XDocument(
new XDeclaration("1.0", "utf-8", "yes"),
new XElement("AmazonEnvelope",
...
if you are using XmlWriter:
using (var xmlWriter = XmlWriter.Create(memoryStream))
{
doc.WriteTo(xmlWriter);
}
You have to save XDocument to file and then get stream from file. Only in this case XDocument preserves declaration (designed behaviour of Save() and WriteTo() methods):
var memoryStream = new MemoryStream();
doc.Save(memoryStream);
var file = Path.GetTempFileName();
using (var fileStream = File.OpenWrite(file))
{
var buffer = memoryStream.GetBuffer();
fileStream.Write(buffer, 0, (int)memoryStream.Length);
}
return File.Open(file, FileMode.Open, FileAccess.Read);
I checked this online tool and you just need to send that MD5 in base64 encoding. It is currently just hexadecimal encoded.
I'm afraid I don't know what the ColdFusion way to do that is, maybe this:
SHA or MD5 Digests in ColdFusion
Here is what I did to get this to work:
<cfset getMD5 = ToBase64(binaryDecode(hash(xmlRequest),'hex'))>
And bang it matched Amazons MD5 hash.
Here is an alternative java-way
found on http://www.kba.suche-spezialwerkzeug.de/pdf/MWSDeveloperGuide.pdf
public static String computeContentMD5HeaderValue(FileInputStream fis)
throws IOException, NoSuchAlgorithmException {
DigestInputStream dis = new DigestInputStream(fis,
MessageDigest.getInstance("MD5"));
byte[] buffer = new byte[8192];
while (dis.read(buffer) > 0)
;
String md5Content = new String(
org.apache.commons.codec.binary.Base64.encodeBase64(dis.getMessageDigest().digest())
);
// Effectively resets the stream to be beginning of the file via a
// FileChannel.
fis.getChannel().position(0);
return md5Content;
}
This will give your desire output.
<cfset binaryValue = binaryDecode( 'C7EF1CADB27497B46FCD6F69516F96E0', "hex" )>
<cfset base64Value = binaryEncode( binaryValue, "base64" )>
<cfdump var="#base64Value#">
Related
My Web API returns a string looking like XML file below. I put all rows ending with \r\n to new line for readability.
{"DataTable.RemotingVersion":{"major":2,"minor":0,"build":-1,"revision":-1,"majorRevision":-1,"minorRevision":-1},"XmlSchema":" <?xml version=\"1.0\" encoding=\"utf-16\"?>\r\n
<xs:schema xmlns=\"\" xmlns:xs=\"http://www.w3.org/2001/XMLSchema\" xmlns:msdata=\"urn:schemas-microsoft-com:xml-msdata\">\r\n
<xs:element name=\"Test\">\r\n
<xs:complexType>\r\n
<xs:sequence>\r\n
<xs:element name=\"Col1\" type=\"xs:string\" msdata:targetNamespace=\"\" minOccurs=\"0\" />\r\n
<xs:element name=\"Col2\" type=\"xs:string\" msdata:targetNamespace=\"\" minOccurs=\"0\" />\r\n
</xs:sequence>\r\n
</xs:complexType>\r\n
</xs:element>\r\n
<xs:element name=\"tmpDataSet\" msdata:IsDataSet=\"true\" msdata:MainDataTable=\"Test\" msdata:UseCurrentLocale=\"true\">\r\n
<xs:complexType>\r\n
<xs:choice minOccurs=\"0\" maxOccurs=\"unbounded\" />\r\n
</xs:complexType>\r\n
</xs:element>\r\n
</xs:schema>","XmlDiffGram":"<diffgr:diffgram xmlns:msdata=\"urn:schemas-microsoft-com:xml-msdata\" xmlns:diffgr=\"urn:schemas-microsoft-com:xml-diffgram-v1\">\r\n
<tmpDataSet>\r\n
<Test diffgr:id=\"Test1\" msdata:rowOrder=\"0\">\r\n
<Col1>Column 1 value</Col1>\r\n
<Col2>Column 2 value</Col2>\r\n
</Test>\r\n
</tmpDataSet>\r\n
</diffgr:diffgram>"}
I wonder how I can feed this string to a datatable? I tried to feed the table like this
System.Data.DataTable dt = new System.Data.DataTable();
byte[] byteArray =Encoding.ASCII.GetBytes(<above string returned by Web API>);
MemoryStream stream = new MemoryStream( byteArray );
dt.ReadXml(stream);
However, the code failed with exception
System.Xml.XmlException: 'Data at the root level is invalid, position 1
Actually, your WebAPI didn't respond with a valid XML payload but return a JSON instead :
{
"DataTable.RemotingVersion": {...},
"XmlSchema":"... xml schema string ...",
"XmlDiffGram": " ... xml data string ..."
}
So you can't read the whole string by dt.ReadXml(stream);
Also note that the xml string begins with an empty string. It's better to invoke .Trim() before reading the XML string.
A working demo:
Let's create two helper methods that read the schema & data parts into DataTable:
// read the schema into DataTable
public static void ReadSchema(DataTable dt,string schema)
{
var stream = new MemoryStream();
ReadXMLToMemoryStream(schema, stream);
dt.ReadXmlSchema(stream);
}
// read the data into DataTable
public static void ReadData(DataTable dt,string xml)
{
var stream = new MemoryStream();
ReadXMLToMemoryStream(xml,stream);
dt.ReadXml(stream);
}
// read xml string into MemoryStream
private static void ReadXMLToMemoryStream(string xml, MemoryStream stream)
{
var doc = new XmlDocument();
doc.LoadXml(xml.Trim());
doc.Save(stream);
stream.Position = 0;
}
To fix the issue, we need parse the related json properties (I'm using Newtonsoft.Json) :
byte[] bytes = Encoding.ASCII.GetBytes(strReturnedFromWebApi);
MemoryStream ms = new MemoryStream(bytes);
using(var reader = new JsonTextReader(new StreamReader(ms))){
var json = JToken.Load(reader);
var version= json.SelectToken("$.['DataTable.RemotingVersion']");
var schema = (string)json.SelectToken("$.XmlSchema"); // the schema part
var data = (string)json.SelectToken("$.XmlDiffGram"); // the data part
System.Data.DataTable dt = new System.Data.DataTable();
ReadSchema(dt,schema); // read schema
ReadData(dt,data); // read data
}
I am trying to read a xml file from the web and parse it out using XDocument. It normally works fine but sometimes it gives me this error for day:
**' ', hexadecimal value 0x1F, is an invalid character. Line 1, position 1**
I have tried some solutions from Google but they aren't working for VS 2010 Express Windows Phone 7.
There is a solution which replace the 0x1F character to string.empty but my code return a stream which doesn't have replace method.
s = s.Replace(Convert.ToString((byte)0x1F), string.Empty);
Here is my code:
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
using (var reader = new StreamReader(e.Result))
{
int[] counter = { 1 };
string s = reader.ReadToEnd();
Stream str = e.Result;
// s = s.Replace(Convert.ToString((byte)0x1F), string.Empty);
// byte[] str = Convert.FromBase64String(s);
// Stream memStream = new MemoryStream(str);
str.Position = 0;
XDocument xdoc = XDocument.Load(str);
var data = from query in xdoc.Descendants("user")
select new mobion
{
index = counter[0]++,
avlink = (string)query.Element("user_info").Element("avlink"),
nickname = (string)query.Element("user_info").Element("nickname"),
track = (string)query.Element("track"),
artist = (string)query.Element("artist"),
};
listBox.ItemsSource = data;
}
}
XML file:
http://music.mobion.vn/api/v1/music/userstop?devid=
0x1f is a Windows control character. It is not valid XML. Your best bet is to replace it.
Instead of using reader.ReadToEnd() (which by the way - for a large file - can use up a lot of memory.. though you can definitely use it) why not try something like:
string input;
while ((input = sr.ReadLine()) != null)
{
string = string + input.Replace((char)(0x1F), ' ');
}
you can re-convert into a stream if you'd like, to then use as you please.
byte[] byteArray = Encoding.ASCII.GetBytes( input );
MemoryStream stream = new MemoryStream( byteArray );
Or else you could keep doing readToEnd() and then clean that string of illegal characters, and convert back to a stream.
Here's a good resource for cleaning illegal characters in your xml - chances are, youll have others as well...
https://seattlesoftware.wordpress.com/tag/hexadecimal-value-0x-is-an-invalid-character/
What could be happening is that the content is compressed in which case you need to decompress it.
With HttpHandler you can do this the following way:
var client = new HttpClient(new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip
| DecompressionMethods.Deflate
});
With the "old" WebClient you have to derive your own class to achieve the similar effect:
class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
Above taken from here
To use the two you would do something like this:
HttpClient
using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
{
using (var stream = client.GetStreamAsync(url))
{
using (var sr = new StreamReader(stream.Result))
{
using (var reader = XmlReader.Create(sr))
{
var feed = System.ServiceModel.Syndication.SyndicationFeed.Load(reader);
foreach (var item in feed.Items)
{
Console.WriteLine(item.Title.Text);
}
}
}
}
}
WebClient
using (var stream = new MyWebClient().OpenRead("http://myrss.url"))
{
using (var sr = new StreamReader(stream))
{
using (var reader = XmlReader.Create(sr))
{
var feed = System.ServiceModel.Syndication.SyndicationFeed.Load(reader);
foreach (var item in feed.Items)
{
Console.WriteLine(item.Title.Text);
}
}
}
}
This way you also recieve the benefit of not having to .ReadToEnd() since you are working with the stream instead.
Consider using System.Web.HttpUtility.HtmlDecode if you're decoding content read from the web.
If you are having issues replacing the character
For me there were some issues if you try to replace using the string instead of the char. I suggest trying some testing values using both to see what they turn up. Also how you reference it has some effect.
var a = x.IndexOf('\u001f'); // 513
var b = x.IndexOf(Convert.ToString((byte)0x1F)); // -1
x = x.Replace(Convert.ToChar((byte)0x1F), ' '); // Works
x = x.Replace(Convert.ToString((byte)0x1F), " "); // Fails
I blagged this
I had the same issue and found that the problem was a embedded in the xml.
The solution was:
s = s.Replace(" ", " ")
I'd guess it's probably an encoding issue but without seeing the XML I can't say for sure.
In terms of your plan to simply replace the character but not being able to, because you have a stream rather than a text, simply read the stream into a string and then remove the characters you don't want.
Works for me.........
string.Replace(Chr(31), "")
I used XmlSerializer to parse XML and faced the same exception.
The problem is that the XML string contains HTML codes of invalid characters
This method removes all invalid HTML codes from string (based on this thread - https://forums.asp.net/t/1483793.aspx?Need+a+method+that+removes+illegal+XML+characters+from+a+String):
public static string RemoveInvalidXmlSubstrs(string xmlStr)
{
string pattern = "&#((\\d+)|(x\\S+));";
Regex regex = new Regex(pattern, RegexOptions.IgnoreCase);
if (regex.IsMatch(xmlStr))
{
xmlStr = regex.Replace(xmlStr, new MatchEvaluator(m =>
{
string s = m.Value;
string unicodeNumStr = s.Substring(2, s.Length - 3);
int unicodeNum = unicodeNumStr.StartsWith("x") ?
Convert.ToInt32(unicodeNumStr.Substring(1), 16)
: Convert.ToInt32(unicodeNumStr);
//according to https://www.w3.org/TR/xml/#charsets
if ((unicodeNum == 0x9 || unicodeNum == 0xA || unicodeNum == 0xD) ||
((unicodeNum >= 0x20) && (unicodeNum <= 0xD7FF)) ||
((unicodeNum >= 0xE000) && (unicodeNum <= 0xFFFD)) ||
((unicodeNum >= 0x10000) && (unicodeNum <= 0x10FFFF)))
{
return s;
}
else
{
return String.Empty;
}
})
);
}
return xmlStr;
}
Nobody can answer if you don't show relevant info - I mean the Xml content.
As a general advice I would put a breakpoint after ReadToEnd() call. Now you can do a couple of things:
Reveal Xml content to this forum.
Test it using VS Xml visualizer.
Copy-paste the string into a txt file and investigate it offline.
Given two different revisions need to get the differences between them, I intend to use the method duvuelve Diff but I anything as a result, it could be? Thanks.
My code is as follows
using (SvnClient client = new SvnClient())
using (MemoryStream result = new MemoryStream())
{
client.Authentication.DefaultCredentials = new NetworkCredential("asdf", "asdf/*");
try
{
//SvnUriTarget is a wrapper class for SVN repository URIs
SvnUriTarget target = new SvnUriTarget(textBox1.Text);
if (client.Diff(target, rango, result))
MessageBox.Show("Successfully para" + rango.ToString() + ".");
StreamReader strReader = new StreamReader(result);
string str = strReader.ReadToEnd();
}
}
The stream that is returned from the Diff() function is positioned at the end of the stream, so before creating your stream reader, you need to reposition it at the beginning of the stream:
result.Position = 0;
StreamReader strReader = new StreamReader(result);
I have my passwords encoded in MD5 in C# and inserted in my DB.
MD5 MD5Hasher = MD5.Create();
byte[] PasswordHash = MD5Hasher.ComputeHash(Encoding.Unicode.GetBytes(PasswordText.Value));
PasswordHash is inserted as is and look like 0x09C09E5B52580E477514FA.......... for example.
In the blackberry app, I get the password, want to encode it to pass it to a web service that will compare both hashed password. The problem is my result is different from the MD5 I create in my Blackberry app.
password = Crypto.encodeStringMD5(password);
Then below my function:
public static String encodeStringMD5(String s) throws Exception {
byte[] bytes = s.getBytes();
MD5Digest digest = new MD5Digest();
digest.update(bytes, 0, bytes.length);
int length = digest.getDigestLength();
byte[] md5 = new byte[length];
digest.getDigest(md5, 0, true);
return convertToHex(md5);
}
private static String convertToHex(byte[] data) {
StringBuffer buf = new StringBuffer();
for (int i = 0; i < data.length; i++) {
int halfbyte = (data[i] >>> 4) & 0x0F;
int two_halfs = 0;
do {
if ((0 <= halfbyte) && (halfbyte <= 9))
buf.append((char) ('0' + halfbyte));
else
buf.append((char) ('a' + (halfbyte - 10)));
halfbyte = data[i] & 0x0F;
} while(two_halfs++ < 1);
}
return buf.toString();
}
So it returns something like this: 07054da3aea1cc98377fe0..........
Any idea how I can get the same hashed password that I create with my C# function in the Blackberry?
Thank you!
The getBytes() method of java String returns a different encoding than the Encoding.Unicode in .NET. You need to specify unambiguous encoding algorithms. Use UTF-8 for both platforms and you should be ok. You can also try providing a charset name to the getBytes method on the Java side; try getBytes("UTF-16")
GregS answered your question directly; but as an aside I would recommend against having the client create the MD5 sum. If the server manages creating the MD5sum, you can further ensure that the password can't be reverse engineered (eg rainbow table) by adding a "salt" value to the password before encoding it on the server. If you do that on the client, you must expose the salt to the client which is less secure.
Do you check the format? Many languages create the same hashes but in different formats.
For example:
5f45r5ssfds544g56fd4gfd56g4f6dgf
vs.
5f-45-r5-ss-fd-s5-44-g5-6f-d4-gf-d5-6g-4f-6d-gf
Try checking for both formats when converting to a string.
To load XML files with arbitrary encoding I have the following code:
Encoding encoding;
using (var reader = new XmlTextReader(filepath))
{
reader.MoveToContent();
encoding = reader.Encoding;
}
var settings = new XmlReaderSettings { NameTable = new NameTable() };
var xmlns = new XmlNamespaceManager(settings.NameTable);
var context = new XmlParserContext(null, xmlns, "", XmlSpace.Default,
encoding);
using (var reader = XmlReader.Create(filepath, settings, context))
{
return XElement.Load(reader);
}
This works, but it seems a bit inefficient to open the file twice. Is there a better way to detect the encoding such that I can do:
Open file
Detect encoding
Read XML into an XElement
Close file
Ok, I should have thought of this earlier. Both XmlTextReader (which gives us the Encoding) and XmlReader.Create (which allows us to specify encoding) accepts a Stream. So how about first opening a FileStream and then use this with both XmlTextReader and XmlReader, like this:
using (var txtreader = new FileStream(filepath, FileMode.Open))
{
using (var xmlreader = new XmlTextReader(txtreader))
{
// Read in the encoding info
xmlreader.MoveToContent();
var encoding = xmlreader.Encoding;
// Rewind to the beginning
txtreader.Seek(0, SeekOrigin.Begin);
var settings = new XmlReaderSettings { NameTable = new NameTable() };
var xmlns = new XmlNamespaceManager(settings.NameTable);
var context = new XmlParserContext(null, xmlns, "", XmlSpace.Default,
encoding);
using (var reader = XmlReader.Create(txtreader, settings, context))
{
return XElement.Load(reader);
}
}
}
This works like a charm. Reading XML files in an encoding independent way should have been more elegant but at least I'm getting away with only one file open.
Another option, quite simple, is to use Linq to XML. The Load method automatically reads the encoding from the xml file. You can then get the encoder value by using the XDeclaration.Encoding property.
An example from MSDN:
// Create the document
XDocument encodedDoc16 = new XDocument(
new XDeclaration("1.0", "utf-16", "yes"),
new XElement("Root", "Content")
);
encodedDoc16.Save("EncodedUtf16.xml");
Console.WriteLine("Encoding is:{0}", encodedDoc16.Declaration.Encoding);
Console.WriteLine();
// Read the document
XDocument newDoc16 = XDocument.Load("EncodedUtf16.xml");
Console.WriteLine("Encoded document:");
Console.WriteLine(File.ReadAllText("EncodedUtf16.xml"));
Console.WriteLine();
Console.WriteLine("Encoding of loaded document is:{0}", newDoc16.Declaration.Encoding);
While this may not server the original poster, as he would have to refactor a lot of code, it is useful for someone who has to write new code for their project, or if they think that refactoring is worth it.