Large http payloads aren't getting sent to the Serilog Http Sink endpoint in .NET Core 3.1 - asp.net-core-3.1

Summary
I'm having trouble posting from Serilog (Http Sink) to my custom .NET Core 3.1 WebAPI endpoint when the logging data is large. If I remove some log data when I do the logging, then Serilog sinks properly with my WebAPI endpoint.
My Configuration
new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Http(httpPath, httpClient: new CustomHttpClient(), batchPostingLimit: int.MaxValue, queueLimit: int.MaxValue)
.CreateLogger();
My Custom Http Client
public class CustomHttpClient : IHttpClient
{
private readonly HttpClient c_httpClient;
public CustomHttpClient()
{
c_httpClient = new HttpClient
{
MaxResponseContentBufferSize = 2147483647L
};
}
public void Configure(IConfiguration configuration)
{
}
public Task<HttpResponseMessage> PostAsync(string requestUri, HttpContent content) => c_httpClient.PostAsync(requestUri, content);
public void Dispose() => c_httpClient?.Dispose();
}
What actually does the logging
var exceptionModel = new AppMonModel
{
Application = "SerilogMvc Sample Application",
Message = ex.Message,
Source = "SerilogMvc.HomeController.Index",
StackTrace = ex.StackTrace,
InnerException = ex.InnerException?.StackTrace,
Details = "Sample details here",
InsertDate = DateTime.Now,
Severity = 100,
UserDescription = "Keyvan User",
ScreenshotBase64String = Convert.ToBase64String(System.IO.File.ReadAllBytes("C:/SamplePath/Untitled.png"))
};
c_logger.LogError(ex, "{exceptionModel}", exceptionModel);
My Endpoint
[HttpPost("log")]
[DisableRequestSizeLimit]
public void Log([FromBody] object logEvents) { ... }
Serilog Error
Event JSON representation exceeds the byte size limit of 262144 set for this sink and will be dropped;
Issue
When I remove ScreenshotBase64String = Convert.ToBase64String(System.IO.File.ReadAllBytes("C:/SamplePath/Untitled.png")) from my exceptionModel object, I see the error in my WebAPI endpoint. As soon as I add it back in, it doesn't even hit the endpoint.
Please let me know if you need additional details. I'd be more than glad to provide them.

The answer was quite simple after turning on Self logging. This is the change I needed to make to increase the batch formatter size:
var defaultBatchFormatter = new DefaultBatchFormatter(batchFormatterSize);
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Error()
.Enrich.FromLogContext()
.WriteTo.Http(httpPath, batchFormatter: defaultBatchFormatter)
.CreateLogger();
The batch formatter size needed to be increased.

Need to add eventBodyLimitBytes
enter image description here
.WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl, eventBodyLimitBytes: 1048576)

Related

Encoded special characters in parameters causing spring boot contract testing query does not match problem

While writing the spring boot contract testing on consumer side, I having problem when request parameters contains special characters. They'll automatically encoding causing the test failed due to the spring consider that the "Query does not match"
"自动制动" has been encoded as "%E8%87%AA%E5%8A%A8%E5%88%B6%E5%8A%A8"
Check the log, i could see:
Query: word = 自动制动 | word: %E8%87%AA%E5%8A%A8%E5%88%B6%E5%8A%A8 <<<<< Query does not match
Here's my groovy file on producer side:
Contract.make {
description "Returns \"Auto hold\"'s canonical value_Mandarin"
name "getSynonym_AutoHold_canonical_Mandarin"
request {
urlPath( "/synonyms"){
headers {"accept: application/json;charset=UTF-8"}
queryParameters {
parameter("filter","canonical")
parameter("lang", "cmn-CHN")
parameter("word","自动制动")
}
}
method GET()
}
response {
status OK()
headers {
contentType applicationJson()
}
body '''
{
"canonical": "autohold",
"word": "自动制动"
}'''
}
}
And here's what I have in consumer side:
#Test
public void testSynonyms_Cmn(){
RestTemplate restTemplate = new RestTemplate();
String url = "http://localhost:8080/synonyms";
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.setContentType(MediaType.APPLICATION_JSON_UTF8);
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url)
.queryParam("filter","canonical")
.queryParam("lang","cmn-CHN")
.queryParam("word","自动制动");
HttpEntity<?> entity = new HttpEntity<>(httpHeaders);
CentralizedSynonyms centralizedSynonyms = restTemplate.exchange(builder.toUriString(), HttpMethod.GET, entity
, CentralizedSynonyms.class).getBody();
assertThat(centralizedSynonyms.getWord()).isEqualTo("自动制动");
assertThat(centralizedSynonyms.getCanonical()).isEqualTo("autohold");
}
I had something similar and fixed it with this:
url value(consumer("/path1/path2/something%3Dsomethingelse"), producer("/path1/path2/something=somethingelse"))

How to call Windows Azure Storage RestFul Service without SDK?

I try to use Windows Azure like a Storage fom Salesforce.com.
I cheked the documentation and I only can see call the calls to azure rest api from SDK (Java, .Net, JS, etc) examples.
I need integrate Salesforce with Windows Azure Storage but, Azure don't have a SDK for Salesforce.com
From Salesforce.com is allow the calls to rest services but the process to call Azure Rest Services require one o more librarys.
Exameple:
Authentication for the Azure Storage Services require of:
Headers: Date Header and Authorization Header
The Authorization Header require two elments
SharedKey
Account Name
Authorization="[SharedKey|SharedKeyLite] :"
SharedKey and Account Name give a conversion:
HMAC-SHA256 conversion
over UTF-8 encoded
For this convertion the documentation referes to SDK Librarys in others words Java Class or .Net Class type helper that in Salesforce.com not exist.
Please, I need a example to call the authentification service without sdk
Sorry for my bad English.
Visit: https://learn.microsoft.com/en-us/rest/api/storageservices/fileservices/authentication-for-the-azure-storage-services
I need a example to call the authentification service without sdk
We could generate signature string and specify Authorization header for the request of performing Azure storage services without installing SDK. Here is a simple working sample to list the containers, you could refer to my generateAuthorizationHeader function and Authentication for the Azure Storage Services to construct the signature string.
string StorageAccount = "mystorageaccount";
string StorageKey = "my storage key";
string requestMethod = "GET";
string mxdate = "";
string storageServiceVersion = "2014-02-14";
protected void btnlist_Click(object sender, EventArgs e)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(CultureInfo.InvariantCulture,
"https://{0}.blob.core.windows.net/?comp=list",
StorageAccount
));
req.Method = requestMethod;
//specify request header
string AuthorizationHeader = generateAuthorizationHeader();
req.Headers.Add("Authorization", AuthorizationHeader);
req.Headers.Add("x-ms-date", mxdate);
req.Headers.Add("x-ms-version", storageServiceVersion);
using (HttpWebResponse response = (HttpWebResponse)req.GetResponse())
{
var stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string content = reader.ReadToEnd();
StringReader theReader = new StringReader(content);
DataSet theDataSet = new DataSet();
theDataSet.ReadXml(theReader);
DataTable dt = theDataSet.Tables[2];
}
}
public string generateAuthorizationHeader()
{
mxdate = DateTime.UtcNow.ToString("R");
string canonicalizedHeaders = string.Format(
"x-ms-date:{0}\nx-ms-version:{1}",
mxdate,
storageServiceVersion);
string canonicalizedResource = string.Format("/{0}/\ncomp:list", StorageAccount);
string stringToSign = string.Format(
"{0}\n\n\n\n\n\n\n\n\n\n\n\n{1}\n{2}",
requestMethod,
canonicalizedHeaders,
canonicalizedResource);
HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(StorageKey));
string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
String authorization = String.Format("{0} {1}:{2}",
"SharedKey",
StorageAccount,
signature
);
return authorization;
}
Besides, please refer to Azure Storage Services REST API Reference to know more about programmatic access to Azure Storage Services via REST APIs.
I find a way to solve this.
You should use Shared Sing, here explain me:
Enter to Portal Azure
Open the Account Storage
In the General Information click on "Share sing access"
Enable all permissions that you need (In my case only Enable "File")
Enable all resources permission that you need (In my case onl Enable "Service, Container and Object")
Define and Start Date and End Date (This is the space of time that Shared Key will be valid)
Define protocol type (In my case use HTTPS)
Clic on "Generate SAS" button
After this process you will get a token like this:
?sv=2016-05-31&ss=f&srt=sco&sp=rwdlc&se=2017-11-28T04:29:49Z&st=2017-02-18T20:29:49Z&spr=https&sig=rt7Loxo1MHGJqp0F6ryLhYAmOdRreyiYT418ybDN2OI%3D
You have to use this Token like Autentication
Example Call Code List a Content:
public with sharing class CallAzureRestDemo {
public string token = '&sv=2016-05-31&ss=f&srt=sco&sp=rwdlc&se=2017-02-19T04:00:44Z&st=2017-02-18T20:00:44Z&spr=https&sig=GTWGQc5GOAvQ0BIMxMbwUpgag5AmUVjrfZc56nHkhjI%3D';
//public Integer batchSize;
public CallAzureRestDemo(){}
public void getlistcontent(String endpoint)
{
// Create HTTP GET request
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setEndpoint(endpoint+token);
Http http = new Http();
HTTPResponse res;
System.debug(LoggingLevel.INFO, '##RESPONSE: '+res);
// only do this if not running in a test method
if(!Test.isRunningTest())
{
System.debug(LoggingLevel.INFO, 'Sending the message to Azure');
res = http.send(req);
System.debug(LoggingLevel.INFO, 'http.send result status: ' + res.getStatus());
}
else
{
System.debug(LoggingLevel.INFO, 'Running in a test so not sending the message to Azure');
}
}
}
Example TestMethod:
#isTest
private class Test_CallAzureRestDemo {
static testMethod void myUnitTest() {
CallAzureRestDemo oRest = new CallAzureRestDemo();
try{
//Call the method and set endpoint
oRest.getlistcontent('https://accountstoragecomex.file.core.windows.net/?comp=list');
}catch(Exception e){
System.debug('##'+e);
}
}
}
Example to Response:
20:15:47.64 (79388244)|CALLOUT_REQUEST|[100]|System.HttpRequest[Endpoint=https://accountstoragecomex.file.core.windows.net/?comp=list&sv=2016-05-31&ss=f&srt=sco&sp=rwdlc&se=2017-02-19T04:00:44Z&st=2017-02-18T20:00:44Z&spr=https&sig=GTWGQc5GOAvQ0BIMxMbwUpgag5AmUVjrfZc56nHkhjI%3D, Method=GET]
20:15:47.64 (395755012)|CALLOUT_RESPONSE|[100]|System.HttpResponse[Status=OK, StatusCode=200]
Example Call Service "FILE - Get List Share"
Call To List Content
One more time, Sorry for my bad english.

HttpResponseException: Internal Server Error

Weirdest thing I have seen in a while. I run my API call through Postman and have no problems at all making a GET request. However, the groovy code below pulls groovyx.net.http.HttpResponseException: Internal Server Error. I am not able to pull even debug to understand if I am actually getting a 5xx error or my code is legitimately broken.
Additionally I have had code like this work in the past, I re-pulled that working code and have the same error. Curious if my Maven config settings would be causing the issue as well (Not sure where I would have to debug). I have also tried messing with the URIbuilder line to see if changing the endpoints would help.
Thanks for helping
abstract class HTTTPClient {
protected runGetRequest(String endpointPassedIn, RESTClient Client){
URIBuilder myEndpoint = new URIBuilder(new URI(Client.uri.toString() + endpointPassedIn))
//Error happens at the next Line
Client.get(uri: myEndpoint, contentType: ContentType.JSON)
LazyMap Response = unprocessedResponse.getData() as LazyMap
return Response
}
}
#Singleton(lazy = true)
class RequestService extends HTTTPClient {
private String auth = "myAuth"
private String baseURL = 'https://api.endpoint.net/'
private RESTClient client = setClient(baseURL, auth)
public buildResponseList(int pagesToPull) {
String endpoint = 'site/address.json?page='
ArrayList responseList = []
for (int i = 1; i <= pagesToPull; i++) {
LazyMap Response = runGetRequest(endpoint + i, client)
for (row in Response) {
responseList.add(row)
//TODO Add in items call here
}
}
return conversationList
}
The error was due to encoding in the Authorization, was on the server side, not the code side

Implement Custom Authentication In Windows Azure Mobile Services

Windows Azure Mobile Services currently doesn't have an option for custom authentication and looking at the feature request
http://feedback.azure.com/forums/216254-mobile-services/suggestions/3313778-custom-user-auth
It isn't coming anytime soon.
With a .NET backend and a .NET application how do you implement custom authentication, so that you don't have to use Facebook, Google or any of their other current providers?
There are plenty of partially completed tutorials on how this this is done with a JS backend and iOS and Android but where are the .NET examples?
I finally worked through the solution, with some help of the articles listed below, some intellisense and some trial and error.
How WAMS Works
First I wanted to describe what WAMS is in a very simple form as this part confused me for a while until it finally clicked. WAMS is just a collection of pre-existing technologies packaged up for rapid deployment. What you need to know for this scenario is:
As you can see WAMS is really just a container for a WebAPI and other things, which I won't go into detail here. When you create a new Mobile Service in Azure you get to download a project that contains the WebAPI. The example they use is the TodoItem, so you will see code for this scenario through the project.
Below is where you download this example from (I was just doing a Windows Phone 8 app)
I could go on further about this but this tutorial will get you started:
http://azure.microsoft.com/en-us/documentation/articles/mobile-services-dotnet-backend-windows-store-dotnet-get-started/
Setup WAMS Project
You will need your MasterKey and ApplicationKey. You can get them from the Azure Portal, clicking on your Mobile Services App and pressing Manage Keys at the bottom
The project you just downloaded, in the Controllers folder I just created a new controller called AccountController.cs and inside I put
public HttpResponseMessage GetLogin(String username, String password)
{
String masterKey = "[enter your master key here]";
bool isValidated = true;
if (isValidated)
return new HttpResponseMessage() { StatusCode = HttpStatusCode.OK, Content = new StringContent("{ 'UserId' : 'F907F58C-09FE-4F25-A26B-3248CD30F835', 'token' : '" + GetSecurityToken(new TimeSpan(1,0, 0), String.Empty, "F907F58C-09FE-4F25-A26B-3248CD30F835", masterKey) + "' }") };
else
return Request.CreateErrorResponse(HttpStatusCode.Unauthorized, "Username and password are incorrect");
}
private static string GetSecurityToken(TimeSpan periodBeforeExpires, string aud, string userId, string masterKey)
{
var now = DateTime.UtcNow;
var utc0 = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
var payload = new
{
exp = (int)now.Add(periodBeforeExpires).Subtract(utc0).TotalSeconds,
iss = "urn:microsoft:windows-azure:zumo",
ver = 2,
aud = "urn:microsoft:windows-azure:zumo",
uid = userId
};
var keyBytes = Encoding.UTF8.GetBytes(masterKey + "JWTSig");
var segments = new List<string>();
//kid changed to a string
var header = new { alg = "HS256", typ = "JWT", kid = "0" };
byte[] headerBytes = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(header, Formatting.None));
byte[] payloadBytes = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(payload, Formatting.None));
segments.Add(Base64UrlEncode(headerBytes));
segments.Add(Base64UrlEncode(payloadBytes));
var stringToSign = string.Join(".", segments.ToArray());
var bytesToSign = Encoding.UTF8.GetBytes(stringToSign);
SHA256Managed hash = new SHA256Managed();
byte[] signingBytes = hash.ComputeHash(keyBytes);
var sha = new HMACSHA256(signingBytes);
byte[] signature = sha.ComputeHash(bytesToSign);
segments.Add(Base64UrlEncode(signature));
return string.Join(".", segments.ToArray());
}
// from JWT spec
private static string Base64UrlEncode(byte[] input)
{
var output = Convert.ToBase64String(input);
output = output.Split('=')[0]; // Remove any trailing '='s
output = output.Replace('+', '-'); // 62nd char of encoding
output = output.Replace('/', '_'); // 63rd char of encoding
return output;
}
You can replace what is in GetLogin, with your own validation code. Once validated, it will return a security token (JWT) that is needed.
If you are testing on you localhost, remember to go into your web.config file and fill in the following keys
<add key="MS_MasterKey" value="Overridden by portal settings" />
<add key="MS_ApplicationKey" value="Overridden by portal settings" />
You need to enter in your Master and Application Keys here. They will be overridden when you upload them but they need to be entered if you are running everything locally.
At the top of the TodoItemController add the AuthorizeLevel attribute as shown below
[AuthorizeLevel(AuthorizationLevel.User)]
public class TodoItemController : TableController<TodoItem>
You will need to modify most of the functions in your TodoItemController but here is an example of the Get All function.
public IQueryable<TodoItem> GetAllTodoItems()
{
var currentUser = User as ServiceUser;
Guid id = new Guid(currentUser.Id);
return Query().Where(todo => todo.UserId == id);
}
Just a side note I am using UserId as Guid (uniqueidentifier) and you need to add this to the todo model definition. You can make the UserId as any type you want, e.g. Int32
Windows Phone/Store App
Please note that this is just an example and you should clean the code up in your main application once you have it working.
On your Client App
Install NuGet Package: Windows Azure Mobile Services
Go into App.xaml.cs and add this to the top
public static MobileServiceClient MobileService = new MobileServiceClient(
"http://localhost:50527/",
"[enter application key here]"
);
In the MainPage.xaml.cs I created
public class Token
{
public Guid UserId { get; set; }
public String token { get; set; }
}
In the main class add an Authenticate function
private bool Authenticate(String username, String password)
{
HttpClient client = new HttpClient();
// Enter your own localhost settings here
client.BaseAddress = new Uri("http://localhost:50527/");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage response = client.GetAsync(String.Format("api/Account/Login?username={0}&password={1}", username, password)).Result;
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
var token = Newtonsoft.Json.JsonConvert.DeserializeObject<Token>(response.Content.ReadAsStringAsync().Result);
App.MobileService.CurrentUser = new MobileServiceUser(token.UserId.ToString());
App.MobileService.CurrentUser.MobileServiceAuthenticationToken = token.token;
return true;
}
else
{
//Something has gone wrong, handle it here
return false;
}
}
Then in the Main_Loaded function
private void MainPage_Loaded(object sender, RoutedEventArgs e)
{
Authenticate("test", "test");
RefreshTodoItems();
}
If you have break points in the WebAPI, you will see it come in, get the token, then come back to the ToDoItemController and the currentUser will be filled with the UserId and token.
You will need to create your own login page as with this method you can't use the automatically created one with the other identity providers. However I much prefer creating my own login screen anyway.
Any other questions let me know in the comments and I will help if I can.
Security Note
Remember to use SSL.
References
[] http://www.thejoyofcode.com/Exploring_custom_identity_in_Mobile_Services_Day_12_.aspx
[] http://www.contentmaster.com/azure/creating-a-jwt-token-to-access-windows-azure-mobile-services/
[] http://chrisrisner.com/Custom-Authentication-with-Azure-Mobile-Services-and-LensRocket
This is exactly how you do it. This man needs 10 stars and a 5 crates of beer!
One thing, I used the mobile Service LoginResult for login like:
var token = Newtonsoft.Json.JsonConvert.DeserializeObject(response.Content.ReadAsStringAsync().Result);
Hope to get this into Android now!

GCM Cloud Connection Server doesn't respond to SASL on XMPP

Using agsXMPP to connect to Google Cloud Messaging XMPP API for the purpose of sending notification to Android devices.
The connection is established OK, but on SASL start, after sending the PLAIN auth element, the server stops responding, and closes the connection after a further 20 seconds.
Base64 decoding the auth example from the documentation page (http://developer.android.com/google/gcm/ccs.html) shows login values of:
126200347933#projects.gcm.android.com12620034793#projects-ga-.android.comAIzaSyB3rcZNkfnqKdFb9mhzCBiYpORDA2JWWtw
Where as agsXMPP is (correctly I think) encoding the string, to give something like:
[ProjectID]\40gcm.googleapis.com[**API*KEY*PASSWORD**]
Note the \40 in my version instead of the # in the Google example - could this make a difference?
I'm expecting either a success or failure message, no response at all is difficult to debug. Could this at character be responsible for some failure, or does Google's implementation of XMPP just not provide the correct responses.
UPDATED:
I answered below, essentially, yes, Google can't handled the encoded # character because it doesn't support that XMPP extension.
After some more testing, I added a new SaslFactory mechanism in agsXMPP and bound it to use the username without encoding (part of extension http://xmpp.org/extensions/xep-0106.html, which Google doesn't support), and then on SaslStartEvent - specify that I want to use that mechanism instead of the inbuilt plain one. - and now the connection will continue normally.
xmpp = new XmppClientConnection();
xmpp.UseSSL = true;
xmpp.UseStartTLS = false;
xmpp.Server = "gcm.googleapis.com";
xmpp.ConnectServer = "gcm.googleapis.com";
xmpp.Port = 5235;
/* Other connection settings /*
SaslFactory.AddMechanism("MyPLAINMechanism", typeof(MyPlainMechanismClass));
xmpp.OnSaslStart += (sender, args) =>
{
args.Auto = false;
args.Mechanism = "MyPLAINMechanism";
args.ExtentedData = new GcmPlainSaslExtendedData
{
Username = "MY UNENCODED USERNAME"
};
};
Then we define the MyPlainMechanismClass which inherits from the Mechanism in agsXMPP, the source code is the same as the original PlainSaslMechanism except the line where the username is input - you can pass in an unencoded username using the ExtendedData property on args.
public class MyPlainMechanismClass: Mechanism
{
private XmppClientConnection m_XmppClient = null;
public GcmPlainSaslMechanism()
{
}
public override void Init(XmppClientConnection con)
{
m_XmppClient = con;
// <auth mechanism="PLAIN" xmlns="urn:ietf:params:xml:ns:xmpp-sasl">$Message</auth>
m_XmppClient.Send(new agsXMPP.protocol.sasl.Auth(agsXMPP.protocol.sasl.MechanismType.PLAIN, Message()));
}
public override void Parse(Node e)
{
// not needed here in PLAIN mechanism
}
private string Message()
{
// NULL Username NULL Password
StringBuilder sb = new StringBuilder();
//sb.Append( (char) 0 );
//sb.Append(this.m_XmppClient.MyJID.Bare);
sb.Append((char)0);
//sb.Append(this.Username);
sb.Append(((GcmPlainSaslExtendedData) this.ExtentedData).Username);
sb.Append((char)0);
sb.Append(this.Password);
byte[] msg = Encoding.UTF8.GetBytes(sb.ToString());
return Convert.ToBase64String(msg, 0, msg.Length);
}
}
Our custom ExtendedData object which we use to pass in custom arguments, such as an unencoded username in this case.
public class GcmPlainSaslExtendedData : agsXMPP.Sasl.ExtendedData
{
public string Username { get; set; }
}