hopefully someone can point me in the right direction on this, I'm trying to delete log streams from cloud watch and have the following function, which deletes the events of the stream, but not the stream itself. I don't want a massive page of empty logs as I'm looping through these to check for an event, and eventually the loop is going to take ages due to al the empty stream logs
private async Task DeleteLog(IAmazonCloudWatchLogs client, GetLogEventsRequest eventsRequest)
{
var request = new DeleteLogStreamRequest
{
LogGroupName = eventsRequest.LogGroupName,
LogStreamName = eventsRequest.LogStreamName,
};
var response = await client.DeleteLogStreamAsync(request);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
Console.WriteLine($"Successfully deleted CloudWatch log stream, {eventsRequest.LogStreamName}.");
}
}
Related
Is there are good way to find out what the total number of records to be synced will be before the records are actually synced via the datastore? This is refering to at the start of time when I am going to sync the datastore with what's in the cloud (so the downstream sync). I'm wanting to create an actual progress indicator for the user (since it takes about a minute for ~1500 records to sync), and don't want to just put up a CircleProgressIndicator().
All I'm currently able to do is:
hubSubscription = Amplify.Hub.listen([HubChannel.DataStore], (msg) {
if (msg.eventName == "ready") {
getAllDevicesInDataStore().then((value) => stopListeningToHub());
}
if (kDebugMode) {
if (msg.eventName == "modelSynced") {
final syncedModelPayload = msg.payload as ModelSyncedEvent;
print(
'Model: ${syncedModelPayload.modelName}, Delta? ${syncedModelPayload.isDeltaSync}');
print(
'${syncedModelPayload.added}, ${syncedModelPayload.updated}, ${syncedModelPayload.deleted}');
}
}
});
I can implement a CircleProgressIndicator() while this is happening, but I want something more definitive.
I have a Chat document that represents a chat between two users. It starts out empty, and eventually looks like this:
// chats/CHAT_ID
{
users: {
USER_ID1: true,
USER_ID2: true
},
lastAddedUser: USER_ID2
}
Each user is connected to a different Cloud Run container via websockets.
I would like to send a welcome message to both users once the second user connected. This message must be sent exactly once.
When a user sends a "connected" message to its websocket, the container performs something like the following:
// Return boolean reflecting whether the current container should emit the welcome message to both users
async addUserToChat(userId) {
// Write operation
await this.chatDocRef.set({ activeUsers: { [userId]: true }, lastAddedUser: userId, { merge: true })
// Read operation
const chatSnap = await this.chatDocRef.get();
const chatData = chatSnap.data();
return chatData.users.length === 2 && chatData.lastAddedUser === userId;
}
And there is a working mechanism that allows container A to send a message to a user connected to container B.
The issue is that sometimes, each container ends up concluding that it is the one that should send the welcome message to both users.
I am unclear as to why that would happen given Firestore's "immediately consistency model" (per this). The only explanation I can think of that allows racing condition is that write operations involving multiple fields are not guaranteed to be atomic. So this:
await this.chatDocRef.set({ activeUsers: { [userId]: true }, lastAddedUser: userId, { merge: true })
actually performs two separate updates for activeUsers and lastAddedUser, opening the possibility for a scenario where after partial update of activeUsers by container A, container B completes the write and read operations before container A overwrites lastAddedUser.
But this sounds wrong.
Can anyone shed light on why racing conditions might occur?
I no longer have racing conditions if I base the logic on the server timestamps instead of the lastAddedUser field.
The document is now simpler:
// chats/CHAT_ID
{
users: {
USER_ID1: true,
USER_ID2: true
}
}
And the function looks like this:
// Return boolean reflecting whether the current container should emit the welcome message to both users
async addUserToChat(userId) {
// Write operation
const writeResult = await this.chatDocRef.set({ activeUsers: { [userId]: true }, { merge: true })
// Read operation
const chatSnap = await this.chatDocRef.get();
const chatData = chatSnap.data();
return chatData.users.length === 2 && writeResult.writeTime.isEqual(chatSnap.updateTime);
}
In other words, the condition for sending the welcome message now becomes: the executing container is the container responsible for the update that resulted in having two users.
While the problem is solved, I am still unclear as to why relying on document data (instead of server metadata) opens up the possibility for racing conditions to occur. If anyone knows the explanation behind this phenomenon, please add an answer and I'll accept it as the solution to this question.
I coded the next Node/Express/Mongo script:
const { MongoClient } = require("mongodb");
const stream = require("stream");
async function main() {
// CONECTING TO LOCALHOST (REPLICA SET)
const client = new MongoClient("mongodb://localhost:27018");
try{
// CONECTION
await client.connect();
// EXECUTING MY WATCHER
console.log("Watching ...");
await myWatcher(client, 15000);
} catch (e) {
// ERROR MANAGEMENT
console.log(`Error > ${e}`);
} finally {
// CLOSING CLIENT CONECTION ???
await client.close(); << ????
}
}main().catch(console.error);
// MY WATCHER. LISTENING CHANGES FROM MY DATABASE
async function myWatcher(client, timeInMs, pipeline = []) {
// TARGET TO WATCH
const watching = client.db("myDatabase").collection("myCollection").watch(pipeline);
// WATCHING CHANGES ON TARGET
watching.on("change", (next) => {
console.log(JSON.stringify(next));
console.log(`Doing my things...`);
});
// CLOSING THE WATCHER ???
closeChangeStream(timeInMs, watching); << ????
}
// CHANGE STREAM CLOSER
function closeChangeStream(timeInMs = 60000, watching) {
return new Promise((resolve) => {
setTimeout(() => {
console.log("Closing the change stream");
watching.close();
resolve();
}, timeInMs);
});
}
So, the goal is to keep always myWatcher function in an active state, to watch any database changes and for example, send an user notification when is detected some updating. The closeChangeStream function close myWatcher function in X seconds after any database changes. So, to keep the myWatcher always active, do you recomment not to use the closeChangeStream function ??
Another thing. With this goal in mind, to keep always myWatcher function in an active state, if I keep the await client.close();, my code emits an error: Topology is closed, so when I ignore this await client.close(), my code works perfectly. Do you recomment not to use the await client.close() function to keep always myWatcher function in an active state ??
Im a newbee in this topics !
thanks for the advice !
Thanks for help !
MongoDB change streams are implemented in a pub/sub paradigm.
Send your application to a friend in the Sudan. Have both you and your friend run the application (that has the change stream implemented). If you open up mongosh and run db.getCollection('myCollection').updateOne({_id: ObjectId("6220ee09197c13d24a7997b7")}, {FirstName: Bob}); both you and your friend will get the console.log for the change stream.
This is assuming you're not running localhost, but you can simulate this with two copies of the applications locally.
The issue comes from going into production and suddenly you have 200 load bearers, 5 developers, etc. running and your watch fires a ton of writes around the globe.
I believe, the practice is to functionize it. Wrap your watch in a function and fire the function when you're about to do a write (and close after you do your associated writes).
My goal with this function is to return an integer representing all of the habits completed by the user. My databases structure is a collection of habits, each with a sub-collection containing its history. The function below utilizes a list of habits, and using the habit's ID it gets a stream of the history for that habit. My problem is that when I try to loop through the history for each habit it stalls at the await for. Any advice would be appreciated, I am still trying to fully understand streams/asynchronous functions.
Future<int> getCompleteHabits(User user, List<Habit> habits) async {
int completed = 0;
if(habits.isNotEmpty) {
for(Habit habit in habits) {
Stream<List<HabitHistory>> streamHistory = db.streamHabitHistory(user, habit.id);
await for(var h in streamHistory){
print(habit.title);
print(h);
}
print("test");
}
}
return completed;
}
This function will add to the variable completed, however, since I cannot access all the history I have yet to do so.
This function prints the following to the console, it doesn't get to the print("test").
flutter: Get in bed by 11:30 pm
flutter: [Instance of 'HabitHistory', Instance of 'HabitHistory', Instance of 'HabitHistory', Instance of 'HabitHistory', Instance of 'HabitHistory']
From the documentaion on Streams:
Streams are done when there are no more events in them, and the code receiving the events is notified of this just as it is notified that a new event arrives. When reading events using an await for loop, the loops stops when the stream is done.
The reason your loop is hanging is that the stream hasn't yet been closed. It will continue to hang until the stream gets closed or until you execute a break or return statement within the loop.
As Abion47 stated the stream has not been closed so the await for is never complete. It is waiting for new events.
Knowing this I can use the await streamHabitHistory(user, habit.id).first to get the list (so it is not waiting for events). The code is below.
Future<int> getCompleteHabits(User user, List<Habit> habits) async {
int completed = 0;
if(habits.isNotEmpty) {
for(Habit habit in habits) {
List<HabitHistory> history = await db.streamHabitHistory(user, habit.id).first;
for (HabitHistory h in history) {
completed = completed + h.completed.length;
}
}
}
return completed;
}
I have a Stateful service with 1000 partitions and 1 replica.
This service in the RunAsync method have an infinte while cycle where I call a Reliable Queue to get messages.
If there are no messages I wait 5 seconds, then retry.
I used to do exactly that with Azure Storage Queue with success.
But with Service Fabric I'm getting thousands of FabricNotReadableExceptions, the Service become unstable and I'm not able to update it or delete it, I need to cancel the entire cluster.
I tried to update it and after 18 hours it was still stuck, so there is something terribly wrong in what I'm doing.
This is the method code:
public async Task<QueueObject> DeQueueAsync(string queueName)
{
var q = await StateManager.GetOrAddAsync<IReliableQueue<string>>(queueName);
using (var tx = StateManager.CreateTransaction())
{
try
{
var dequeued = await q.TryDequeueAsync(tx);
if (dequeued.HasValue)
{
await tx.CommitAsync();
var result = dequeued.Value;
return JSON.Deserialize<QueueObject>(result);
}
else
{
return null;
}
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceMessage(this, $"!!ERROR!!: {e.Message} - Partition: {Partition.PartitionInfo.Id}");
return null;
}
}}
This is the RunAsync
protected override async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
var message = await DeQueueAsync("MyQueue");
if (message != null)
{
//process, takes around 500ms
}
else
{
Thread.Sleep(5000);
}
}
}
I also changed Thread.Sleep(5000) with Task.Delay and was having thousands of "A task was canceled" errors.
What I'm missing here?
It's the cycle too fast and SF cannot update the other replicas in time?
Should I remove all the replicas leaving just one?
Should I use the new ConcurrentQueue instead?
I have the problem in production and in local with 50 or 1000 partitions, doesn't matter.
I'm stuck and confused.
Thanks
You need to honor the cancellationToken that is passed in to your RunAsync implementation. Service Fabric will cancel the token when it wants to stop your service for any reason - including upgrades - and it will wait indefinitely for RunAsync to return after cancelling the token. This could explain why you couldn't upgrade your application.
I would suggest checking cancellationToken.IsCancelled inside your loop, and breaking out if it has been cancelled.
FabricNotReadableException can happen for a variety of reasons - the answer to this question has a comprehensive explanation, but the takeaway is
You can consider FabricNotReadableException retriable. If you see it, just try the call again and eventually it will resolve into either NotPrimary or Granted.