How to upload data in IReliable dictionary from IEnumerable<T> - azure-service-fabric

I am retrieving data from Azure Table Storage and storing them in IEnumerable, but I don't know how to convert this IEnumerable<T> into IReliableDictionary<string, Store>, so that I can save this store data into state manager of stateful service.
var storesRepo = await StateManager.GetOrAddAsync<IReliableDictionary<long, Store>>("");
So how can I insert this IEnumerable<Store> into storesRepo?
Store Class:
public class Store : TableEntity
{
public string Name { get; set; }
public double Longitude { get; set; }
public double Latitude { get; set; }
public string Address { get; set; }
public double Distance { get; set; }
public string Email { get; set; }
public string City { get; set; }
}
ReliableDictionary<string, Store>
where key is store name.

Create a dictionary, enumerate over all stores in the collection, add them one by one, inside a transaction.
Make sure the set of stores doesn't get too large to avoid a long running transactions & locks.
Make sure to think about partitioning first
Add retry support.
Apply best practices from here.
meta code below:
var storesRepo = await StateManager.GetOrAddAsync<IReliableDictionary<string, Store>>("someName");
IEnumerable<Store> stores = await GetFromTableStorage();
using (ITransaction tx = base.StateManager.CreateTransaction())
{
foreach(var store in stores)
{
await storesRepo.AddAsync(tx, store.Name, store, cancellationToken);
}
await tx.CommitAsync();
}

Related

Xamarin Forms Plugin.CloudFirestore - How to Handle Firestore Map Fields and Firestore Map Arrays with Reactive

I am currently using the Plugin.CloudFirestore.Sample project (link: https://github.com/f-miyu/Plugin.CloudFirestore) as a reference for creating my own application and would like to know how I should code my project to upload documents containing Firestore Maps and Firestore Arrays that have nested Firestore Maps then listen for document changes and display these changes using Xamarin Forms, Plugin.CloudFirestore, and Reactive. I would also like to know the maximum amount of Map conversions that can be handled at a time while converting from a Firestore Document to a .Net Model Class. To clarify, if the ExampleItem Class mentioned below contains another Model Class as a property, how would the class and reactive class viewmodel code look to upload/listen for changes. If anyone knows the answer to my question, it would be awesome if you could download the Plugin.CloudFirestore.Sample project, make the changes to the file, share a response explaining what you added, and share a download link.
public class TodoItem
{
public static string CollectionPath = "todoItems";
//Should I use DocumentConverters for converting to the "DataMap" and "DataMapArray" properties? If so what would the code for them look like?
[Id]
public string? Id { get; set; }
public string? Name { get; set; }
public string? Notes { get; set; }
public ExampleItem1? DataMap { get; set; }
public ObservableCollection<ExampleItem1>? DataMapArray { get; set; }
[ServerTimestamp(CanReplace = false)]
public Timestamp CreatedAt { get; set; }
[ServerTimestamp]
public Timestamp UpdatedAt { get; set; }
}
public class ExampleItem1
{
// Should I use [Id] on the Id property or any other Plugin.CloudFirestore.Attributes?
public string? Id { get; set; }
public string? Name { get; set; }
public ExampleItem2? DataMap { get; set; }
public ObservableCollection<ExampleItem2>? DataMapArray { get; set; }
}
public class TodoItemViewModel : BindableBase
{
public string? Id { get; }
public ReactivePropertySlim<string?> Name { get; set; } = new ReactivePropertySlim<string?>();
public ReactivePropertySlim<string?> Notes { get; set; } = new ReactivePropertySlim<string?>();
public ReactivePropertySlim<ExampleItem?> DataMap { get; set; } = new ReactivePropertySlim<ExampleItem?>();
public ReactivePropertySlim<ObservableCollection<ExampleItem>?> DataMapArray { get; set; } = new ReactivePropertySlim<ObservableCollection<ExampleItem>?>();
public TodoItemViewModel(TodoItem item)
{
Id = item.Id;
Name.Value = item.Name;
Notes.Value = item.Notes;
DataMap.Value = item.DataMap;
DataMapArray.Value = item.DataMapArray;
}
public void Update(string? name, string? notes, ExampleItem1? dataMap, ObservableCollection<ExampleItem1>? dataMapArray)
{
Name.Value = name;
Notes.Value = notes;
DataMap.Value = dataMap;
DataMapArray.Value = dataMapArray;
}
}

Entityframeworkcore Include method doesnt working. Eager Loading Lazy Loading doesn't working correctly

Relationship tables are not added to stations(Lists) when I use the Include () method. eager loading and lazy loading both do not work.the associated station list is always zero.I'm using aspnet core mvc 5.0. any library missing? It pulls data from the database but does not fetch any associated data.
[HttpGet]
public IActionResult Index(int? SayfaNo)
{
int _sayfaNo = SayfaNo ?? 1;
// Proplem is here. Networks.Stations.Count=0 always
var networks = _context.Networks.Include(x=>x.Stations).ToList();
//.ToPagedList<Network>(_sayfaNo, 9);
if (networks == null)
{
return View(networks);
}
var request = Request;
if (request.Headers!=null)
{
if (request.Headers["X-Requested-With"] == "XMLHttpRequest")
{
return PartialView("~/Views/Home/_BicycleListPartialView.cshtml", networks);
}
}
return View(networks);
}
public class Network:BaseEntity
{
public Network()
{
Stations = new HashSet<Station>();
}
public string NId { get; set; }
public string Name { get; set; }
[ForeignKey("Location")]
public int LocationId { get; set; }
public virtual Location Location { get; set; }
public virtual ICollection<Station> Stations { get; set; }
}
public class Location:BaseEntity
{
public string Country { get; set; }
public string City { get; set; }
public virtual Network Network { get; set; }
}
public class Station:BaseEntity
{
public string SId { get; set; }
public string Name { get; set; }
public int? FreeBikes { get; set; }
public int? EmptySlots { get; set; }
[ForeignKey("Network")]
public int NetworkId { get; set; }
public virtual Network Network { get; set; }
}
public class BaseEntity
{
[Key]
public int Id { get; set; }
}
I see a couple issues:
[ForeignKey] attributes can be put on either the navigation property or the FK field, however the attribute must point to the other property. I.e. if you put it on the FK field, it should point to the navigation property:
[ForeignKey("Network")]
public int NetworkId { get; set; }
public virtual Network Network { get; set; }
.. or this:
public int NetworkId { get; set; }
[ForeignKey("NetworkId")]
public virtual Network Network { get; set; }
There also looks to be a conflicting one to one relationship declared between Location and Network where Network is expected to have a single Location and Location has a single Network. By default one-to-one relationships expect to join on the PKs of the two tables. If this is a many-to-one relationship (Network has one location, location can be part of many networks, but can have a "default" network) then Location would need a FK for the Network. (I.e. DefaultNetworkId) and the relationship between network and location will need to be explicitly configured. The same if instead the many-to-one relationship goes from Location to Network. If instead it is a one-to-one relationship then I believe EF Core can configure 1-to-1 /w paired FKs, otherwise it would expect their ID's to match. Either way it would likely require some explicit configuration for the relationship.
The next thing is that I would avoid initializing reference navigation properties in a constructor. Initializing child objects or collections is fine, but avoid initializing references:
public Network()
{
// Location = new Location(); <- don't initialize as a *new* one is not a valid state.
Stations = new HashSet<Station>();
}

Entity Framework Core: How do I update a record with nested fields?

I've got a simple "ContactsList" ASP.Net Core Web (REST) application, .Net Core 3.0, an MSSQL LocalDB, using MSVS 2019.
My "Contact" entity contains a list of "Notes".
When I create a new contact that already contains one or more notes, everything works fine. EF automagically inserts the notes into the notes table.
But when I try to UPDATE a contact, EF seems to disregard "notes".
Q: For "Updates", do I need write code in my controller to explicitly update the notes myself?
Or am I doing something "wrong", such that EF can't "automagically" do the updates it's supposed to?
Models/Contact.cs:
public class Contact
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int ContactId { get; set; }
public string Name { get; set; }
public string EMail { get; set; }
public string Phone1 { get; set; }
public string Phone2 { get; set; }
public string Address1 { get; set; }
public string Address2 { get; set; }
public string City { get; set; }
public string State { get; set; }
public string Zip { get; set; }
public List<Note> Notes { get; set; }
}
Models/Note.cs:
public class Note
{
public Note()
{
this.Date = DateTime.Now; // Default value: local "now"
}
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int NoteId { get; set; }
public string Text { get; set; }
public DateTime Date { get; set; }
[ForeignKey("Contact")]
public int ContactId { get; set; }
}
Controllers/ContactsController.cs (POST works: if there are notes in the contacts list, it adds them):
[HttpPost]
public async Task<ActionResult<Contact>> PostContact(Contact contact)
{
_context.Contacts.Add(contact);
await _context.SaveChangesAsync();
//return CreatedAtAction("GetContact", new { id = contact.ContactId }, contact);
return CreatedAtAction(nameof(GetContact), new { id = contact.ContactId }, contact);
}
Controllers/ContactsController.cs (PUT seems to completely disregard any assocated notes):
[HttpPut("{id}")]
public async Task<IActionResult> PutContact(int id, Contact contact)
{
if (id != contact.ContactId)
{
return BadRequest();
}
_context.Entry(contact).State = EntityState.Modified;
try
{
await _context.SaveChangesAsync();
}
catch (DbUpdateConcurrencyException)
{
if (!ContactExists(id))
{
return NotFound();
}
else
{
throw;
}
}
return NoContent();
}
The SQL for POST shows four separate INSERTs: one for the contact, and one for each note.
The SQL for PUT only shows one UPDATE: just the contact; nothing else.
The debugger shows "notes" are clearly part of the "Contact" record that the controller received by PutContact().
Q: Should EF deal with "updating" notes automagically, or do I need to hand-code my updates in the controller?
Entity Framework Core ignores relationships unless you explicitly include them in queries.
_context.Entry(contact).State = EntityState.Modified;
The problem with the line above is that you did not specify that the related data has been modified, so it will be ignored in the query.
So you can either
attach all the related data
set the state of the related data to EntityState.Modified
or you can
query the object in the database and include the related data
and then assign the contact object to that queried object
var dbContactObj = _context.Contacts.Include(x => x.Notes).First(x => x.Id == contact.Id);
dbContactObj = contact;
_context.SaveChangesAsync();

Slow performance in reliable collections with relatively large dataset

I have a reliable service with 3 partitions. The data is partitioned reasonably equally between the 3 - it's almost exact. I have just under 3.95 million entries of the following contract across the 3 partitions
[DataContract]
public class Item
{
[DataMember]
public string StandardisedText { get; set; }
[DataMember]
public string Vendor{ get; set; }
[DataMember]
public bool Card{ get; set; }
[DataMember]
public string Suburb { get; set; }
[DataMember]
public string State { get; set; }
[DataMember]
public string PostCode { get; set; }
[DataMember]
public float Latitude { get; set; }
[DataMember]
public float Longitude { get; set; }
[DataMember]
public string Level1 { get; set; }
[DataMember]
public string Level2 { get; set; }
[DataMember]
public string Level3 { get; set; }
[DataMember]
public string Level4 { get; set; }
}
I have an upstream aggregate service which reports back a count across all partitions with the following code (please ignore the terribleness of this code, its a quick and dirty POC to see how viable reliable collections are for reasonably large data sets).
[HttpGet("")]
public IActionResult GetAll()
{
try
{
_logger.LogError("Getting all count");
ServiceEventSource.Current.Message("Getting all count");
var settings = new FabricTransportRemotingSettings { OperationTimeout = TimeSpan.FromMinutes(30) };
var factory =
new ServiceProxyFactory(h =>
new FabricTransportServiceRemotingClientFactory(settings));
int total = 0;
foreach (var servicePartitionKey in PartitionKeys)
{
var proxy = factory.CreateServiceProxy<ITermTextService>(
new Uri("fabric:/Acme.Fabric.Refinery/Acme.Fabric.Refinery.RefineryStore"),
servicePartitionKey);
var count = proxy.Count().Result;
ServiceEventSource.Current.Message($"Searched partition {servicePartitionKey.Value} and found {count} values" );
_logger.LogInformation($"Searched partition {servicePartitionKey.Value} and found {count} values");
total += count;
}
return Ok(total);
}
catch (Exception excep)
{
_logger.LogError($"Error in get all {excep.Message} {excep.StackTrace}");
ServiceEventSource.Current.Message($"Error {excep.Message} {excep.StackTrace}");
}
return null;
}
The count code is
public async Task<int> Count()
{
int i = 0;
var termTexts = await TermTexts;
using (var tx = StateManager.CreateTransaction())
{
var enumerable = await termTexts.CreateEnumerableAsync(tx);
var enumerator = enumerable.GetAsyncEnumerator();
while (await enumerator.MoveNextAsync(CancellationToken.None))
{
i++;
}
}
return i;
}
The total time of this operation is 747165 ms (12 minutes). This is an integer on the wire - no large packets.
My question is, is this performance in the expected region or should I investigate this further? It could be that using a RC for this volume of data is a misuse, so we would need to look elsewhere. These response times imply also that there are disk reads involved, so another question is at what point would this occur and can this be configured? A read from hot storage I would imagine to be in the region below 50ms per partition.
Thanks in advance.
If u enumerate millions of records asynchronously, this performance is expected. If u use IReliableDictionary2, you can use the Count property directly. No need to enumerate all records. If u want to search by key, use CreateAsyncEnumerable function.

RavenDb-Get list from a list of ids

I'm developing an events manager with RavenDb. The logged user can write a post in a place that follow and then see all posts in the user wall (home page). Every post can be mark as "Like" and "Favourite". A small structure like twitter or facebook.
Wich is the best way to get the posts by the places that I follow? I need to get the count of likes and favourites and if I already mark the post on viewmodel to send to the view and display the list. I'm trying like this, but throw the max requests exception.
var myPlaces = this.SessionDb.Query<EventFollow>()
.Where(ls => ls.UserId == User.Identity.Name)
.Select(l => l.PlaceId);
var listPosts = this.SessionDb
.Advanced.LuceneQuery<Post>().WhereIn("PlaceId", myPlaces)
.Skip(skip)
.Take(20)
.ToList();
List<PostViewModel> posts = new List<SpottedViewModel>();
foreach (var p in listPosts)
{
PostViewModel vm = new PostViewModel();
vm.Creator = p.UserCreatorId == User.Identity.Name;
vm.Like = this.SessionDb.Query<Lol>().Where(lol => lol.SpottedId == p.Id).Count();
vm.Favourites = this.SessionDb.Query<Favourite>().Where(fa => fa.SpottedId == p.Id).Count();
vm.Post= s;
vm.IsLike = this.SessionDb.Query<Like>().FirstOrDefault(l => l.PostId == p.Id) != null;
vm.IsFavourite = this.SessionDb.Query<Favourite>().FirstOrDefault(f => f.SpottedId == s.Id) != null;
posts.Add(vm);
}
I have these models:
public class Post
{
public string Id { get; set; }
public string UtenteCreatorId { get; set; }
public string PlaceId { get; set; }
public string Body{ get; set; }
}
public class Lol
{
public string Id { get; set; }
public string PostId { get; set; }
public string UserId { get; set; }
}
public class Place
{
public string Id { get; set; }
public string UserCreatorId { get; set; }
public string Name{ get; set; }
}
public class Favourite
{
public string Id { get; set; }
public string PostId { get; set; }
public string UserId { get; set; }
}
public class EventFollow
{
public string Id { get; set; }
public string PlaceId { get; set; }
public string UserId { get; set; }
}
Thank you and sorry for my english! ;D
Your problem here isn't really getting a list of documents from a list of IDs, you can do that by either using the IDocumentSession.Load method and sending in a list of document IDs, or by using WhereIn like you do. The problem is your model design.
It seems to me you have some serious re-design of your data models to do. It's strongly relational as it is now, it seems, and RavenDB isn't relational (or optimized towards relational models, anyway). It's hard for me to see exactly what your use cases are, but from what I can see I would probably save 'favourites' and 'likes' in the 'post' model, and put 'EventFollow' in the 'user' model.
Also, by RavenDB's "safe by default" principles, you're limited to 30 requests per session (per default, it's configurable). You have some select N+1 going on here so you'll most likely exceed those 30 requests, by far. That's why you get the exception.