I have the following endpoint
[HttpPost("export")]
public async Task<ActionResult<IEnumerable<Person>>> CreateCSV([FromBody] Request request)
{
var t = Stopwatch.StartNew();
List<AppDbContext> dbContexts = new List<AppDbContext>();
List<Task<List<Person>>> tasks = new List<Task<List<Person>>>();
for (int i = 0; i < request.ContextCount; i++) dbContexts.Add(_dbContextFactory.CreateDbContext());
List<int> batches = request.Count
.Batch(request.Count / request.ContextCount)
.ToList();
for (int page = 1; page < batches.Count() + 1; ++page)
{
tasks.Add(dbContexts[page - 1].Persons
.OrderBy(entry => entry.FirstName)
.AsNoTracking()
.Skip((page - 1) * (batches[page - 1]))
.Take(batches[page - 1])
.ToListAsync());
}
var data = (await Task.WhenAll(tasks)).SelectMany(list => list);
var builder = new StringBuilder();
builder.AppendLine("Id, FirstName, LastName, SecurrityCode");
Parallel.ForEach(
source: data.AsParallel().AsOrdered(),
parallelOptions: new ParallelOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount
},
body: (entry) =>
{
builder.AppendLine($"{entry.Id}, {entry.FirstName}, {entry.LastName}, {entry.SecurrityCode}");
});
t.Stop();
var seconds = t.ElapsedMilliseconds / 1000d;
return File(Encoding.UTF8.GetBytes(builder.ToString()), "text/xlsx", "users.xlsx");
}
What I want to do is query the same datasource and get the items in batches (if you have a better approch please make the suggestions)
What I've noticed is if I'm using orderBy as seen in the exemply, the more dbContext are created (e.g. 10) the more the time of computation increases as oposit to if I'm not using sorting the more dbContexts I have the better the performance.
I'm not sure why is this and what can I do to improve it.
Related
Ok so I have a problem in which I use a collection to gather some ratings data and work with it, by the time I finish the rating update process, I have new ratings that I would like to update the collection with. However I can't call update because I get the error "Cannot overwrite model once compiled." I understand that I already called once the model to work with the data and that's why I get the error. is there any way I can update the collection? Or I will just have to workaround by creating a new collection with the latest rating, and then matching the latest ratings collection with the one I use to work with the data.
This is my code
let calculateRating = async () => {
const getData = await matchesCollection().find().lean();
const playerCollection = await playersCollection();
const getDataPlayer = await playerCollection.find().lean();
let gamesCounting = [];
getDataPlayer.forEach((player) => {
player.makePlayer = ranking.makePlayer(1500);
});
for (let i = 0; i < getData.length; i++) {
const resultA = getDataPlayer.findIndex(({ userId }, index) => {
if (userId === getData[i].userA) {
return index;
}
});
const resultB = getDataPlayer.findIndex(
({ userId }) => userId === getData[i].userB
);
const winner = getData[i].winner;
if (getDataPlayer[resultA] === undefined) {
continue;
} else if (getDataPlayer[resultB] === undefined) {
continue;
}
gamesCounting.push([
getDataPlayer[resultA].makePlayer,
getDataPlayer[resultB].makePlayer,
winner,
]);
}
ranking.updateRatings(gamesCounting);
let ratingsUpdate = [];
getDataPlayer.forEach((item) => {
let newRating = item.makePlayer.getRating();
let newDeviation = item.makePlayer.getRd();
let newVolatility = item.makePlayer.getVol();
item.rating = newRating;
item.rd = newDeviation;
item.vol = newVolatility;
ratingsUpdate.push(item);
});
};
I try the work around with creating the new collection
I want to update multiple rows in ASP.NET Core, but I get an error:
InvalidOperationException: The entity type 'EntityQueryable' was not found. Ensure that the entity type has been added to the model.
This is my code:
var data = _db.UserTable.Where(a => a.CityIDn == selectedid);
foreach (var items in data)
{
await Task.Run(() =>
{
items.CityID = 2;
});
_db.Update(data);
await _db.SaveChangesAsync();
}
Try this for multiple rows:
var data = _db.UserTable.Where(a => a.CityIDn == selectedid).ToList();
foreach (var item in data)
{
item.CityID = 2;
_db.UserTable.Update(item);
}
await _db.SaveChangesAsync();
And for one record, try like this:
var data = _db.UserTable.Where(a => a.CityIDn == selectedid).FirstOrDefault();
if(data != null)
{
data.CityID = 2;
_db.UserTable.Update(data );
await _db.SaveChangesAsync();
}
The content of Update should be items, it worked for me, you can have a try.
var data = _db.UserTable.Where(a => a.CityIDn == selectedid).ToList();
foreach (var items in data)
{
await Task.Run(() =>
{
items.CityID = 2;
});
_db.Update(items);
await _db.SaveChangesAsync();
}
I have Flutter + Firestore app with a perfomance problem: large database query execution time (about a 5 sec.). I have a small database size, I think that if it increases, the query execution speed will be even greater. How can I improve application performance?
import 'package:carstat/models/entry.dart';
import 'package:carstat/models/operation.dart';
import 'package:carstat/services/data_service.dart';
class DashboardService {
DataService dataService = DataService();
getMarkers(List<Entry> entries, String carId) async {
var _markers = [];
for (int i = 0; i < entries.length; i++) {
List<Operation> _operations = [];
_operations =
await dataService.getEntryOperations(entries[i].entryId, carId);
_markers.add({'entry': entries[i], 'operations': _operations});
}
return _markers;
}
}
My data structure for example:
.document(docId)
.collection('cars')
.document(carId)
.collection('entries')
.document(entryId)
.collection('operations')
.document();
DataService code:
getEntryOperations(String entryId, String carId) async {
List<Operation> _operations = [];
Future<QuerySnapshot> _userDoc =
fs.where('userId', isEqualTo: _userId).getDocuments();
await _userDoc.then((res) {
docId = res.documents[0].documentID;
});
Future<QuerySnapshot> _entryOperations = fs
.document(docId)
.collection('cars')
.document(carId)
.collection('entries')
.document(entryId)
.collection('operations')
.getDocuments();
await _entryOperations.then((val) {
for (int i = 0; i < val.documents.length; i++) {
var _operation = Operation();
_operation.entryId = entryId;
_operation.operationNote = val.documents[i].data['operationNote'];
_operation.operationDate =
val.documents[i].data['operationDate'].toDate();
_operation.operationMileage = val.documents[i].data['operationMileage'];
_operation.operationPartName =
val.documents[i].data['operationPartName'];
_operation.operationPrice = val.documents[i].data['operationPrice'];
_operation.partPrice = val.documents[i]['partPrice'];
_operation.operationId = val.documents[i]['operationId'];
_operations.add(_operation);
}
});
return _operations;
}
The query you're showing is unconditionally getting all of the documents in a specific subcollection. Of course, that will take more time as the collection grows. There is no secret trick or special flag to pass to make this query happen any faster.
In fact, there is not much you can do about this at all, other than to limit the size of the collection, or limit the number of documents in the query. You might also want to reconsider your database structure to reduce the number of documents you're fetching.
My answer, much faster
class DashboardService {
DataService dataService = DataService();
getMarkers(List<Entry> entries, String carId) async {
var _marker = []; // коллекция списков операторов для каждого регламента ТО
final opsForEntries = await Future.wait(
entries.map((value) {
return dataService.getEntryOperations(value.entryId, carId);
})
);
for(int i = 0; i < entries.length; i++) {
_marker.add(
{
'entry': entries[i],
'operations': opsForEntries[i]
}
);
}
return _marker;
}
}
I have collection of data in Mongodb, i want to give best matches suggestion while user input query in our suggestion box,
when user start typing com suggestion should be:
Computer
Computer Science
something more alike
I am sorting in Node by getting all matched data from mongo first and then give a rank to each data
function rank(name, q) {
var len = name.length,
lastIndex = -1;
for(var i = 0; i < q.length; i++) {
var n = name.indexOf(q[i], (lastIndex + 1));
if(n !== -1) {
len--;
lastIndex = n;
}
}
return len;
}
var query = 'com';
// giving rank to data
data = data.map(function(v) {
v.rank = rank(v.value, query);
return v;
});
// sorting by rank
data = data.sort(function(a, b) {
return a.rank - b.rank
});
It is giving me satisfied result, but it will be too slow while dealing with large data.
I want let mongodb engine to deal with sorting and give me just limited best matches result.
Maybe you could do it through mapreduce. Map-reduce is a data processing paradigm for condensing large volumes of data into useful aggregated results.
var mapFn = function(){
var len = this.name.length,
lastIndex = -1;
var q = 'com';
for(var i = 0; i < q.length; i++) {
var n = this.name.indexOf(q[i], (lastIndex + 1));
if(n !== -1) {
len--;
lastIndex = n;
}
}
emit(len, this);
};
var reduceFn = function(key, values){
return values.sort(function(a,b){
return a.name - b.name;
});
};
db.collection.mapReduce(mapFn, reduceFn, { out: { reduce: 'result_collection'}});
I'm trying to get type fields for each attribute of my entities. Quering Orion and getting entities is not the problem (I do this through NGSI Source widget) but the way getting these parameters.
From NGSI Source (usual suscription to Orion instance):
var doInitialSubscription = function doInitialSubscription() {
this.subscriptionId = null;
this.ngsi_server = MashupPlatform.prefs.get('ngsi_server');
this.ngsi_proxy = MashupPlatform.prefs.get('ngsi_proxy');
this.connection = new NGSI.Connection(this.ngsi_server, {
ngsi_proxy_url: this.ngsi_proxy
});
var types = MashupPlatform.prefs.get('ngsi_entities').split(new RegExp(',\\s*'));
var entityIdList = [];
var entityId;
for (var i = 0; i < types.length; i++) {
entityId = {
id: '.*',
type: types[i],
isPattern: true
};
entityIdList.push(entityId);
}
var attributeList = null;
var duration = 'PT3H';
var throttling = null;
var notifyConditions = [{
'type': 'ONCHANGE',
'condValues': MashupPlatform.prefs.get('ngsi_update_attributes').split(new RegExp(',\\s*'))
}];
var options = {
flat: true,
onNotify: handlerReceiveEntity.bind(this),
onSuccess: function (data) {
this.subscriptionId = data.subscriptionId;
this.refresh_interval = setInterval(refreshNGSISubscription.bind(this), 1000 * 60 * 60 * 2); // each 2 hours
window.addEventListener("beforeunload", function () {
this.connection.cancelSubscription(this.subscriptionId);
}.bind(this));
}.bind(this)
};
this.connection.createSubscription(entityIdList, attributeList, duration, throttling, notifyConditions, options);
};
var handlerReceiveEntity = function handlerReceiveEntity(data) {
for (var entityId in data.elements) {
MashupPlatform.wiring.pushEvent("entityOutput", JSON.stringify(data.elements[entityId]));
}
};
To MyWidget:
MashupPlatform.wiring.registerCallback("entityInput", function (entityString) {
var entity;
entity = JSON.parse(entityString);
id = entity.id;
type = entity.type;
for(var attr in entity){
attribute = entity[attr];
}
I'm trying to code something similar to obtain the value of type fields. How can I do that? (I'm sure it's quite easy...)
You cannot make use of the current NGSI source operator implementation (at least v3.0.2) if you want to get the type metadata of attributes as the NGSI source makes use of the flat option (discarding that info).
We are studying updating this operator to allow creating subscriptions without using the flat option. The main problem here is that other components expect the data provided by this operator being provided in the format returned when using the flat option. I will update this answer after analysing deeper the issue.