" Message: Security error." While Requesting a Https Uri using WebClient OR HttpWebRequest using SilverLight Client - rest

Question

My Silverlight Client and REST Web Server(Implented using ASP.NET Handlers) both are on same domain.
Both HTTP and HTTPS protocol bindings are configured in IIS 7.0
When Request a URI using HTTPS , i Get Error " Message: "Security error."
But Same URI works just fine on HTTP request without any error
private void btnLoginWebRequest_Click(object sender, RoutedEventArgs e)
{
bool httpResult = WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
bool httpsResult = WebRequest.RegisterPrefix("https://", WebRequestCreator.ClientHttp);
this.Dispatcher.BeginInvoke(()=>
{
WebClient webClient = new WebClient();
webClient.Headers["CustomHeader"] = String.Format("myHeaders={0}", "Value");
webClient.OpenReadCompleted += new OpenReadCompletedEventHandler(webClient_OpenReadCompleted);
webClient.OpenReadAsync(new Uri(#"https://xxx.xx.xxx.xxx/MyRESTApi/UserGroups"));
});
}
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
if (e.Error == null)
{
}
}
I Get a Error and e.Error contains the following Error information
{System.Security.SecurityException: Security error.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
at System.Net.WebClient.OpenReadAsyncCallback(IAsyncResult result)}
[System.Security.SecurityException]: {System.Security.SecurityException: Security error.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
at System.Net.WebClient.OpenReadAsyncCallback(IAsyncResult result)}
Data: {System.Collections.ListDictionaryInternal}
InnerException: null
Message: "Security error."
StackTrace: " at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)\r\n at System.Net.WebClient.OpenReadAsyncCallback(IAsyncResult result)"
crossdomain.xml is as follows :-
<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
<site-control permitted-cross-domain-policies="all"/>
<allow-access-from domain="*" secure="true"/>
<allow-http-request-headers-from domain="*" headers="*" secure="true"/>
</cross-domain-policy>
clientaccesspolicy.xml is as follows :-
<?xml version="1.0" encoding="utf-8"?>
<access-policy>
<cross-domain-access>
<policy>
<allow-from http-request-headers="*">
<domain uri="https://"/>
<domain uri="http://"/>
</allow-from>
<grant-to>
<resource path="/" include-subpaths="true"/>
</grant-to>
</policy>
</cross-domain-access>
</access-policy>
I Have seen many threads related to similar error on stack overflow but not able to resolve this issue by any of the solutions provided in those threads. Am i missing something?