Schematically, I spend my time doing things looking like the following.
public async Task<ItemA> GetItemsA(object someParams)
{
var res = new List<ItemA>();
var listOfItemAIds = await GetIdsFromServerAsync(someParams);
var tableAQuery = _tableA.Where(x => listOfItemAIds.Contains(x.Id));
await _tableA.PullAsync(null, tableAQuery);
var itemsA= await tableAQuery.ToListAsync();
var listOfItemBIds = itemsA.Select(x => x.bId).ToList();
await _tableB.PullAsync(null, _tableB.Where(x => listOfItemBIds .Contains(x.Id));
foreach(var itemA in itemsA)
{
itemA.ItemB = await _tableB.LookupAsync(itemA.itemBId);
}
return res;
}
There are several problems with that:
listOfTableAIds.Contains(x.Id) leads to errors due to URL length limitations
As a cannot represent the content of listOfItemAIds or listOfItemBIds with a queryId of less than 50 chars, I end up pulling data that I might already have
It's a shame that all my pulls are not batched into a single server call
I could directly get all I need from a single server query but then I wouldn't benefit from the Azure Mobile Sync framework
Any suggestions on how to improve that sequence?
I would suggest that you refine things so that you query is simpler. You may overall pull more data, but then that data will be available within tableB. For instance, you may want to say "where itemBId is not null".
Related
The code below is my member enrollment function.
I want to make sure the new document is successfully created in Firestore before increasing memberCount.
var db = FirebaseFirestore.instance;
await db.runTransaction((transaction) async {
final memberCountDoc = db.collection('variables').doc('memberCount');
final snapshot = await transaction.get(memberCountDoc); // get total num of members right now
int newMemberCount = snapshot.get('value') + 1;
Map<String, dynamic> userdata = {
'sequence': newMemberCount, // specify the value
'enrollTime': DateTime.now().toUtc(),
};
var user = FirebaseAuth.instance.currentUser;
db.collection('users').doc(user!.uid).set(userdata); // add a new doc to record userdata
transaction.update(memberCountDoc, {'value': newMemberCount}); // finish transaction
return newMemberCount;
}).then(
(value) => print('Now you have $value members!'),
onError: (e) => print(e),
);
Is it a good idea to add data within a transaction function?
I afraid there are bugs that I haven't noticed yet, or it's not efficient.
The coded looks fine at first glance to me.
One optimization was to use a batched write operation instead of a transaction, by using the atomic increment operator instead of a read-then-write operation. This will increase the scalability of the system, as the whole read-then-write now happens on the database server, rather than in your code.
I have QnA Maker chatbot. I want to do that: If bot gives the DefaultNoAnswer 3 times in a session, I want to show different DefaultNoAnswer. How can I count the DefaultNoAnswers in QnAMakerBaseDialog ?
ex:
Client: asdaaasd
Bot: Sorry, Could you phrase your question differently?
Client: dsjhdsgjdsa
Bot:Sorry, Could you phrase your question differently?
Client: aasdjhajds
Bot: Sorry, I couldn't get the question. Send an email for detailed information.
I find the best way to handle this is with a conversation state variable. I have my default message set up in my helper (i.e. I have a helper file that makes the call to QnA Maker, checks the confidence, and sends a default message in case of low confidence or no answer). If you are using a similar case, you can increment your state variable there. If you are using QnA Maker's default answer directly, you still need to do some check on every result before sending the response to user. I haven't used that method, but I would probably just check the result for the default answer and increment the variable accordingly.
Here is a sample for the first case. I am assuming here that you are already familiar with managing user and conversation state.
var qnaResult = await QnAServiceHelper.queryQnaService(query, oldState);
if (qnaResult[0].score > MINIMUM_SCORE) {
const conversationData = await this.dialogState.get(step.context, {});
conversationData.defaultAnswerCounter = 0;
await this.conversationState.saveChanges(step.context);
var outputActivity = MessageFactory.text(qnaResult[0].answer);
} else {
const conversationData = await this.dialogState.get(step.context, {});
conversationData.defaultAnswerCounter += 1;
if (conversationData.defaultAnswerCounter <= 2) {
var outputActivity = defaultAnswer;
} else {
var outputActivity = escalationAnswer;
}
await this.conversationState.saveChanges(step.context);
}
I'm having a huge issue that I've been trying for days to get through. I have a scenario in which I'm trying to handle an Insert Conflict in my Xamarin project. The issue is that the record in the Cloud DB doesn't exist because there was an issue with a foreign key constraint so I'm in a scenario in which the sync conflict handler needs to delete the local record along with the record in the __operations table in SQLite. I've tried everything. Purge with the override set to 'true' so that it should delete the local record and all operations associated. Doesn't work. I've been just trying to force delete it by accessing the SQL store manually:
var id = localItem[MobileServiceSystemColumns.Id];
var operationQuery = await store.ExecuteQueryAsync("__operations", $"SELECT * FROM __operations WHERE itemId = '{id}'", null).ConfigureAwait(false);
var syncOperation = operationQuery.FirstOrDefault();
var tableName = operation.Table.TableName;
await store.DeleteAsync(tableName, new List<string>(){ id.ToString() });
if (syncOperation != null)
{
await store.DeleteAsync("__operations", new List<string>() { syncOperation["id"].ToString() }).ConfigureAwait(false);
}
I am able to query the __operations table and I can see the ID of the item I want to delete. The DeleteAsync method runs without exception but no status is returned so I have no idea if this worked or not. When I try to sync again the operation stubbornly exists. This seems ridiculous. How do I just delete an operation without having to sync with the web service? I'm about to dig down further and try to force it even harder by using the SQLiteRaw library but I'm really really hoping I'm missing something obvious? Can anyone help? THANKS!
You need to have a subclass of the Microsoft.WindowsAzure.MobileServices.Sync.MobileServiceSyncHandler class, which overrides OnPushCompleteAsync() in order to handle conflicts and other errors. Let's call the class SyncHandler:
public class SyncHandler : MobileServiceSyncHandler
{
public override async Task OnPushCompleteAsync(MobileServicePushCompletionResult result)
{
foreach (var error in result.Errors)
{
await ResolveConflictAsync(error);
}
await base.OnPushCompleteAsync(result);
}
private static async Task ResolveConflictAsync(MobileServiceTableOperationError error)
{
Debug.WriteLine($"Resolve Conflict for Item: {error.Item} vs serverItem: {error.Result}");
var serverItem = error.Result;
var localItem = error.Item;
if (Equals(serverItem, localItem))
{
// Items are the same, so ignore the conflict
await error.CancelAndUpdateItemAsync(serverItem);
}
else // check server item and local item or the error for criteria you care about
{
// Cancels the table operation and discards the local instance of the item.
await error.CancelAndDiscardItemAsync();
}
}
}
Include an instance of this SyncHandler() when you initialize your MobileServiceClient:
await MobileServiceClient.SyncContext.InitializeAsync(store, new SyncHandler()).ConfigureAwait(false);
Read up on the MobileServiceTableOperationError to see other conflicts you can handle as well as its methods to allow resolving them.
In my table controller, I have:
public IQueryable<MyTable> GetAllMyTable()
I would like to replace the above with my own:
[HttpGet, Route("tables/MyTable")]
public IEnumerable<MyTable> GetAllMyTable()
But I get this response when I call it:
HTTP/1.1 405 Method Not Allowed
Somehow the Web API routing does not reach my method.
Why I'm doing this: the original method produces an inefficient Entity Framework SQL query that takes 3 seconds per call on my local test environment. This is running the query captured from SQL Profiler directly in SQL Mgt Studio. An equivalent query takes less than a second to run. Terrible.
Worse, the inefficient EF queries consumes lots of Azure SQL DTUs, tempting you to up your Azure subscription level if you want a quick fix.
Azure Mobile Apps is wonderful, but the multiple layers of abstraction makes it hard to really see what's going on under the hood, and therefore harder to tune.
Any help would be much appreciated.
HTTP/1.1 405 Method Not Allowed
Per my understanding, the error is obvious. You could send the GET HTTP verb to your endpoint tables/MyTable for retrieving the data. You need to check your request against your mobile app backend via fiddler.
Azure Mobile Apps is wonderful, but the multiple layers of abstraction makes it hard to really see what's going on under the hood, and therefore harder to tune.
For the common table controller, it would look like this:
public IQueryable<Message> GetAllMessage()
{
return Query();
}
The Query() method under EntityDomainManager.cs would equal as follows:
IQueryable<TData> query = this.Context.Set<TData>();
if (!includeDeleted)
{
query = query.Where(item => !item.Deleted);
}
return query;
If it deals with the ODATA queries (e.g. $top, $skip, $filter, etc.), the Nested SQL statement would be generated. We could modify the action to clarify it as follows:
public IEnumerable<Message> GetAllMessage(ODataQueryOptions opt)
{
var message = context.Set<Message>();
var query2=opt.ApplyTo(message, new ODataQuerySettings());
return query2.Cast<Message>().ToList();
}
Here's my rather crude attempt at bypassing the Entity Framework/OData plumbing and using direct SQL. (Wouldn't it be great if Dapper is supported!) This one works well, and is faster than the nested SQL that EF produces. The handling of OData is hacky; I have not had time to investigate using OData to extract the values for UpdatedAt, skip, and top.
I'm only using this approach for one method that needs optimisation. This is the method that the Azure Mobile App client calls when doing a pull.
public IEnumerable<MyTable> GetAllMyTable()
{
var qryValues = HttpUtility.ParseQueryString(Request.RequestUri.Query);
var updatedAtFilter = qryValues["$filter"];
var skip = qryValues["$skip"];
var top = qryValues["$top"];
if (updatedAtFilter != null)
{
var r = new Regex(#"^.+datetimeoffset'(?<time>.+)'.+$", RegexOptions.None);
var m = r.Match(updatedAtFilter);
if (m.Success)
{
var updatedAt = m.Groups["time"].Value.Replace("T", " ");
var sqlString = #"SELECT T0.*
FROM MyTable T0
WHERE T0.UpdatedAt >= #UpdatedAt
ORDER BY UpdatedAt, Id
OFFSET #Skip ROWS
FETCH NEXT #Top ROWS ONLY";
var updatedAtParam = new SqlParameter("UpdatedAt", SqlDbType.DateTimeOffset);
updatedAtParam.Value = updatedAt;
var skipParam = new SqlParameter("Skip", SqlDbType.Int);
skipParam.Value = int.Parse(skip);
var topParam = new SqlParameter("Top", SqlDbType.Int);
topParam.Value = int.Parse(top);
var data = _context.Database.SqlQuery<MyTable>(sqlString, new object[] { updatedAtParam, skipParam, topParam }).AsEnumerable<MyTable>();
return data;
}
}
return null;
}
I find transactions (https://www.firebase.com/docs/transactions.html) to be a cool way of handling concurrency, however it seems they can only be done from clients.
The way we use Firebase is mainly by writing data from our servers and observing them on clients. Is there a way to achieve optimistic concurrency model when writing data via REST API?
Thanks!
You could utilize an update counter to make write ops work in a similar way to transactions. (I'm going to use some pseudo-code below; sorry for that but I didn't want to write out a full REST API for an example.)
For example, if I have an object like this:
{
total: 100,
update_counter: 0
}
And a write rule like this:
{
".write": "newData.hasChild('update_counter')",
"update_counter": {
".validate": "newData.val() === data.val()+1"
}
}
I could now prevent concurrent modifications by simply passing in the update_counter with each operation. For example:
var url = 'https://<INSTANCE>.firebaseio.com/path/to/data.json';
addToTotal(url, 25, function(data) {
console.log('new total is '+data.total);
});
function addToTotal(url, amount, next) {
getCurrentValue(url, function(in) {
var data = { total: in.total+amount, update_counter: in.update_counter+1 };
setCurrentValue(ref, data, next, addToTotal.bind(null, ref, amount, next));
});
}
function getCurrentValue(url, next) {
// var data = (results of GET request to the URL)
next( data );
}
function setCurrentValue(url, data, next, retryMethod) {
// set the data with a PUT request to the URL
// if the PUT fails with 403 (permission denied) then
// we assume there was a concurrent edit and we need
// to try our pseudo-transaction again
// we have to make some assumptions that permission_denied does not
// occur for any other reasons, so we might want some extra checking, fallbacks,
// or a max number of retries here
// var statusCode = (server's response code to PUT request)
if( statusCode === 403 ) {
retryMethod();
}
else {
next(data);
}
}
FYI, Firebase Realtime Database officially supports this now.
Read the blog and the docs for more info.
check out Firebase-Transactions project: https://github.com/vacuumlabs/firebase-transactions
I believe, this may be quite handy for your case, especially if you do a lot of writes from the server.
(disclaimer: I'm one of the authors)