Weird issue while processing events coming from kinesis - postgresql

I setup amazon connect on aws and if I make a test call, it will put that call in a aws kinesis stream. I am trying to write a lambda that process this records and save them to database.
If I make a simple call (call the number - asnwer - hangup) it works just fine. However if I make a multipart call (call a number - answer - trasnfer to another number - hangup) this comes to kinesis as two separate records (CTR).
My lambda process the CTR (Contact Trace Records) one by one. First it saves the CTR to a table called call_segments and then it query this table to see if the other part of this call is already there. If it is, it merges the data and save to a table called completed_calls, otherwise skips it.
If a call has more than on segment (if it was transfered to another number) it brings it to you as two events.
My problem is that even though I am processing the events one after the other it seems that when the second event is processed (technically the call segment from first event is already in database), it can not see the first segment of the call.
here is my code:
const callRecordService = require("./call-records-service");
exports.handler = async (event) => {
await Promise.all(
event.Records.map(async (record) => {
return processRecord(record);
})
);
};
const processRecord = async function(record) {
try{
const payloadStr = new Buffer(record.kinesis.data, "base64").toString("ascii");
let payload = JSON.parse(payloadStr);
await callRecordService.processCTR(payload);
}
catch(err){
// console.error(err);
}
};
and here is the service file:
async function processCTR(ctr) {
let userId = "12"
let result = await saveCtrToContactDetails(ctr, userId);
let paramsForCallSegments = [ctr.InstanceARN.split("instance/").pop(), ctr.ContactId]
let currentCallSegements = await dbHelper.getAll(dbQueries.getAllCallSegmentsQuery, paramsForCallSegments)
let completedCall = checkIfCallIsComplete(currentCallSegements);
if (completedCall) {
console.log('call is complete')
let results = await saveCallToCompletedCalls(completedCall);
}
}
//------------- Private functions --------------------
const saveCtrToContactDetails = async (ctr, userId) => {
let params = [ctr.ContactId,userId,ctr.callDuration];
let results = await dbHelper.executeQuery(dbQueries.getInsertCallDetailsRecordsQuery, params);
return results;
}
const checkIfCallIsComplete = (currentCallSegements) => {
//This function checks if all callSegments are already in call_segments table.
}
const saveCallToCompletedCalls = async (completedCall) => {
let contact_id = completedCall[0].contact_id;
let user_id = completedCall[0].user_id;
let call_duration = completedCall[0] + completedCall[1]
completedCall.forEach(callSegment => {
call_duration += callSegment.call_duration;
});
let params = [contact_id, user_id, call_duration];
let results = await dbHelper.executeQuery(dbQueries.getInsertToCompletedCallQuery, params);
};

Related

how to update a collection if you already called it MongoDB Mongoos

Ok so I have a problem in which I use a collection to gather some ratings data and work with it, by the time I finish the rating update process, I have new ratings that I would like to update the collection with. However I can't call update because I get the error "Cannot overwrite model once compiled." I understand that I already called once the model to work with the data and that's why I get the error. is there any way I can update the collection? Or I will just have to workaround by creating a new collection with the latest rating, and then matching the latest ratings collection with the one I use to work with the data.
This is my code
let calculateRating = async () => {
const getData = await matchesCollection().find().lean();
const playerCollection = await playersCollection();
const getDataPlayer = await playerCollection.find().lean();
let gamesCounting = [];
getDataPlayer.forEach((player) => {
player.makePlayer = ranking.makePlayer(1500);
});
for (let i = 0; i < getData.length; i++) {
const resultA = getDataPlayer.findIndex(({ userId }, index) => {
if (userId === getData[i].userA) {
return index;
}
});
const resultB = getDataPlayer.findIndex(
({ userId }) => userId === getData[i].userB
);
const winner = getData[i].winner;
if (getDataPlayer[resultA] === undefined) {
continue;
} else if (getDataPlayer[resultB] === undefined) {
continue;
}
gamesCounting.push([
getDataPlayer[resultA].makePlayer,
getDataPlayer[resultB].makePlayer,
winner,
]);
}
ranking.updateRatings(gamesCounting);
let ratingsUpdate = [];
getDataPlayer.forEach((item) => {
let newRating = item.makePlayer.getRating();
let newDeviation = item.makePlayer.getRd();
let newVolatility = item.makePlayer.getVol();
item.rating = newRating;
item.rd = newDeviation;
item.vol = newVolatility;
ratingsUpdate.push(item);
});
};
I try the work around with creating the new collection

How to assign values from an API call to a variable in flutter

I have the following method which is use dto verify a ticket/token
var ticketArray = ticket.split('|');
//First check to verify token using simple versification algo
if (widget.eventID.toString() != (ticketArray[0])) {
setState(() {
ticketMainMsg = 'This QR code is NOT VALID';
ticketsubtitle = ticketArray.length != 2
? 'The QR code is fake'
: 'QR code could belong to another event';
ticketStatus = false;
return;
});
}
//Make API call
ticketModel = HttpVerifyTicketPost(
eventId: widget.eventID,
ticket: ticket,
scannerId: widget.scannerId,
).verifyTicket();
}
From above, you can see I do a very simple check on the qr code/token if this simple step fails, I don't bother making an API call and I set the state based on these values.
However if the check passes, then I proceed to make an API call to the server to fully verify the token/code.
My issue is I am struggling to now assign the values from the API call to the ticketStatus, ticketMainMsgand ticketsubtitle parameters. Can anyone helo shed some light. I am quite new to flutter but I am aware that the TicketModel will be a type of Future. My background is PHP so forgive me!
EDIT: The httpVerifyTicket Class
class HttpVerifyTicketPost {
String ticket;
int someId;
int anotherId;
HttpVerifyTicketPost(
{required this.ticket, required this.someId, required this.anotherId});
String verifyURL =
'https://api.com/api/vendors/scanner/native/verify/ticket';
Future<TicketModel> verifyTicket() async {
var storage = await SharedPreferences.getInstance();
var code= storage.getString('code');
var client = http.Client();
var ticketModel = null;
var body = {
'ticket': ticket,
'scanner': scannerCode,
'someId': someId,
'anotherId': anotherId
};
try {
var url = Uri.parse(verifyURL);
var res = await client.post(url, body: jsonEncode(body));
if (res.statusCode == 200) {
var jsonString = res.body;
var jsonMap = json.decode(jsonString);
ticketModel = TicketModel.fromJson(jsonMap);
}
return ticketModel;
} catch (Exception) {
return ticketModel;
}
}
}
Try this please
HttpVerifyTicketPost(
eventId: widget.eventID,
ticket: ticket,
scannerId: widget.scannerId,
).verifyTicket().then((value){setState(() {
ticketModel=value
});
});
I don't quite understand what you want to achieve, but maybe you need to add an asynchronous method like
ticketModel = await HttpVerifyTicketPost( //add await eventId: widget.eventID, ticket: ticket, scannerId: widget.scannerId, ).verifyTicket();
and you must add async like Future Foo() async {your code...}

My async metho didn't go after "for" iteration

My code like this:
Future<List<ListClass>> getIndexList() async {
return Future.delayed(delayTime).then((_) async {
//initialize
List<ListClass> resultListItems = new List<ListClass>();
await listCollection.get().then((value) async {
var v = value.data;
for (var data in v) {
String userId = data['userId'];
String authorName=await UserTable().getUserNameById(userId);
print("createUserName=" + authorName);
resultListItems.add(
ListClass(header, content, userId, wordCount, authorName));
}
print("resultListItems.length="+resultListItems.length.toString());
return resultListItems;
});
return resultListItems;
});
}
When I debug,it shows that this method return null,and after the for,the print("resultListItems"); doesn't run too. How can I fix this?Thanks!!
I have found the problem,the for clause doesn't run completely.One of my data in database is null,so it runs with bug.Sorry for my fault

Prefix text to ASP.NET Core response body

I'm trying to prepend the string )]}',\n to any response body that's JSON. I thought that an IAsyncResultFilter would be what I needed to use, but I'm not having luck. If I use the below code, it appends the text to the response since calling await next() writes to the response pipe. If I try and look at the context before that though, I can't tell what the response will actually be to know if it's JSON.
public class JsonPrefixFilter : IAsyncResultFilter
{
public async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
{
var executed = await next();
var response = executed.HttpContext.Response;
if (response.ContentType == null || !response.ContentType.StartsWith("application/json"))
return;
var prefix = Encoding.UTF8.GetBytes(")]}',\\n");
var bytes = new ReadOnlyMemory<byte>(prefix);
await response.BodyWriter.WriteAsync(bytes);
}
}
Thanks to timur's post I was able to come up with this working solution.
public class JsonPrefixFilter : IAsyncResultFilter
{
public async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
{
var response = context.HttpContext.Response;
// ASP.NET Core will always send the contents of the original Body stream back to the client.
var originalBody = response.Body;
// We want to write into a memory stream instead of the actual response body for now.
var ms = new MemoryStream();
response.Body = ms;
// After this call the body is written into the memory stream and the properties
// of the response object are populated.
await next();
if (response.ContentType != null && response.ContentType.StartsWith("application/json")) {
var prefix = Encoding.UTF8.GetBytes(")]}',\\n");
var prefixMemoryStream = new MemoryStream();
await prefixMemoryStream.WriteAsync(prefix);
await prefixMemoryStream.WriteAsync(ms.ToArray());
prefixMemoryStream.Seek(0, SeekOrigin.Begin);
// Now put the stream back that .NET wants to use and copy the memory stream to it.
response.Body = originalBody;
await prefixMemoryStream.CopyToAsync(response.Body);
} else {
// If it's not JSON, don't muck with the stream, so just put things back.
response.Body = originalBody;
ms.Seek(0, SeekOrigin.Begin);
await ms.CopyToAsync(response.Body);
}
}
}
Update:
I never liked the above, so I switched to this solution. Instead of calling AddJsonOptions, I took inspiration from ASP.NET's formatter to use this instead:
public class XssJsonOutputFormatter : TextOutputFormatter
{
private static readonly byte[] XssPrefix = Encoding.UTF8.GetBytes(")]}',\n");
public JsonSerializerOptions SerializerOptions { get; }
public XssJsonOutputFormatter()
{
SerializerOptions = new() {
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
ReferenceHandler = ReferenceHandler.IgnoreCycles
};
SupportedEncodings.Add(Encoding.UTF8);
SupportedMediaTypes.Add(MediaTypeHeaderValue.Parse("application/json"));
}
public override sealed async Task WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
{
ArgumentNullException.ThrowIfNull(context, nameof(context));
ArgumentNullException.ThrowIfNull(selectedEncoding, nameof(selectedEncoding));
var httpContext = context.HttpContext;
var objectType = context.Object?.GetType() ?? context.ObjectType ?? typeof(object);
var responseStream = httpContext.Response.Body;
try {
await responseStream.WriteAsync(XssPrefix);
await JsonSerializer.SerializeAsync(responseStream, context.Object, objectType, SerializerOptions, httpContext.RequestAborted);
await responseStream.FlushAsync(httpContext.RequestAborted);
} catch (OperationCanceledException) when (context.HttpContext.RequestAborted.IsCancellationRequested) {
}
}
}
Now, when you call .AddControllers() you just set that as the first output formatter:
services.AddControllers(options => {
options.Filters.Add(new ProducesAttribute("application/json"));
options.OutputFormatters.Insert(0, new XssJsonOutputFormatter());
});
Obviously you could improve this to take serialization options in the constructor, but all my project would work exactly like the above so I just hardcoded it right in.
You could've used Seek on a steam to rewind it. Issue is, you can only keep adding onto default HttpResponseStream, it does not support seeking.
So you can employ the technique from this SO answer and temporarily replace it with MemoryStream:
private Stream ReplaceBody(HttpResponse response)
{
var originBody = response.Body;
response.Body = new MemoryStream();
return originBody;
}
private async Task ReturnBodyAsync(HttpResponse response, Stream originalBody)
{
response.Body.Seek(0, SeekOrigin.Begin);
await response.Body.CopyToAsync(originalBody);
response.Body = originalBody;
}
public async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
{
var originalBody = ReplaceBody(context.HttpContext.Response); // replace the default stream with MemoryStream
await next(); // we probably dont care about the return of this call. it's all in the context
var response = context.HttpContext.Response;
if (response.ContentType == null || !response.ContentType.StartsWith("application/json"))
return;
var prefix = Encoding.UTF8.GetBytes(")]}',\\n");
var bytes = new ReadOnlyMemory<byte>(prefix);
response.Body.Seek(0, SeekOrigin.Begin); // now you can seek. but you will notice that it overwrites the response so you might need to make extra space in the buffer
await response.BodyWriter.WriteAsync(bytes);
await ReturnBodyAsync(context.HttpContext.Response, originalBody); // revert the reference, copy data into default stream and return it
}
this is further complicated by the fact that you need to restore reference to original stream, so you have to careful around that.
This SO answer has a bit more context.

CosmosDB Paging Return Value

I am trying to return paging results the request from CosmosDB. I saw this example from here but I am not sure what to do with the response variable.
// Fetch query results 10 at a time.
var queryable = client.CreateDocumentQuery<Book>(collectionLink, new FeedOptions { MaxItemCount = 10 });
while (queryable.HasResults)
{
FeedResponse<Book> response = await queryable.ExecuteNext<Book>();
}
Am I suppose to return it directly? Or do I have to do something further with the response variable? I tried to return the response variable directly and it's not working. Here's my code:
public async Task<IEnumerable<T>> RunQueryAsync(string queryString)
{
var feedOptions = new FeedOptions { MaxItemCount = 3 };
IQueryable<T> filter = _client.CreateDocumentQuery<T>(_collectionUri, queryString, feedOptions);
IDocumentQuery<T> query = filter.AsDocumentQuery();
var response = new FeedResponse<T>();
while (query.HasMoreResults)
{
response = await query.ExecuteNextAsync<T>();
}
return response;
}
Update:
After reading #Evandro Paula's answer, I followed the URL and changed my implementation to below. But it is still giving me 500 status code:
public async Task<IEnumerable<T>> RunQueryAsync(string queryString)
{
var feedOptions = new FeedOptions { MaxItemCount = 1 };
IQueryable<T> filter = _client.CreateDocumentQuery<T>(_collectionUri, queryString, feedOptions);
IDocumentQuery<T> query = filter.AsDocumentQuery();
List<T> results = new List<T>();
while (query.HasMoreResults)
{
foreach (T t in await query.ExecuteNextAsync())
{
results.Add(t);
}
}
return results;
}
And here's the exception message:
Cross partition query is required but disabled. Please set
x-ms-documentdb-query-enablecrosspartition to true, specify
x-ms-documentdb-partitionkey, or revise your query to avoid this
exception., Windows/10.0.17134 documentdb-netcore-sdk/1.9.1
Update 2:
I added the EnableCrossPartitionQuery to true and I am able to get the response from CosmosDB. But I am not able to get the 1 item that I defined. Instead, I got 11 items.
Find below a simple example on how to use the CosmosDB/SQL paged query:
private static async Task Query()
{
Uri uri = new Uri("https://{CosmosDB/SQL Account Name}.documents.azure.com:443/");
DocumentClient documentClient = new DocumentClient(uri, "{CosmosDB/SQL Account Key}");
int currentPageNumber = 1;
int documentNumber = 1;
IDocumentQuery<Book> query = documentClient.CreateDocumentQuery<Book>("dbs/{CosmoDB/SQL Database Name}/colls/{CosmoDB/SQL Collection Name}", new FeedOptions { MaxItemCount = 10 }).AsDocumentQuery();
while (query.HasMoreResults)
{
Console.WriteLine($"----- PAGE {currentPageNumber} -----");
foreach (Book book in await query.ExecuteNextAsync())
{
Console.WriteLine($"[{documentNumber}] {book.Id}");
documentNumber++;
}
currentPageNumber++;
}
}
Per exception described in your question Cross partition query is required but disabled, update the feed options as follows:
var feedOptions = new FeedOptions { MaxItemCount = 1, EnableCrossPartitionQuery = true};
Find a more comprehensive example at https://github.com/Azure/azure-documentdb-dotnet/blob/d17c0ca5be739a359d105cf4112443f65ca2cb72/samples/code-samples/Queries/Program.cs#L554-L576.
you are not specifying any where criteria for your specific item...so you are getting all results..try specifying criteria for the item (id , name etc) you are looking for. And keep in mind cross partition queries consume much more RUs n time, you can revisit architecture of your data model..Ideally always do queries with in same partition