I want to create "invitations" based on email addresses so that when a user with that email would arrive to the system his invitations would already be waiting for him.
so I wanted to create an invitations collection with key based on groupId + '_' + invited_user_email but if the user is test#test.com this the .com is interpreted as a subkey :(
I tried groupId + '_' + invited_user_email.replace(/[.]/g,'\\.') but it did not help either.
What am I missing? Is there no way to include a '.' in a firestore document's ID ?
EDIT:
per Doug Stevenson's request here is the code I use now:
var invitedUserEmailAsKey=invitedUserEmail.replace(/[.]/g,'\\.')
var batch = db.batch();
var groupRef = db.collection("groups").doc(groupId);
var invitationRef = db.collection("invitations").doc( groupRef.id + '_' + invitedUserEmailAsKey);
var groupUpdates = {};
groupUpdates['invited.' + invitedUserEmailAsKey + '.user_name'] = invitedUserName;
groupUpdates['merge'] = true;
batch.update(groupRef, groupUpdates);
The problems are with the updated document on the groups collection.
It ends up looking like this:
instead of having someone#gmail.com as a key
Using update() with "dot" in a key is interpreted as a nested field as documented here.
Notice that you're updating a document fields (groupRef) and not really doing anything with the "invitations" collection. I don't think that creating a new field on the group doc for every invitation is what you really mean to do.
As Dror mentioned, adding users with update() and a dot in the key won't work.
I use the set method with merge true:
addUserToGroup(id,email,type){
this.log("id",id,"email",email,"type",type);
let obj = {};
obj[email] = type;
this.angularFireStore.collection("Users").doc(id).set(obj, { merge: true }).then(() => {
this.log("addedUser",email);
}).catch((error) => {
this.log("failed to add user",error)
})
}
From the documentation: https://firebase.google.com/docs/firestore/manage-data/add-data#web-v8_1
"To create or overwrite a single document, use the set() method"
"If you're not sure whether the document exists, pass the option to merge the new data with any existing document to avoid overwriting entire documents."
Related
Hi I am newbie to dynamoDB. Below is the schema of the dynamo table
{
"user_id":1, // partition key
"dob":"1991-09-12", // sort key
"movies_watched":{
"1":{
"movie_name":"twilight",
"movie_released_year":"1990",
"movie_genre":"action"
},
"2":{
"movie_name":"harry potter",
"movie_released_year":"1996",
"movie_genre":"action"
},
"3":{
"movie_name":"lalaland",
"movie_released_year":"1998",
"movie_genre":"action"
},
"4":{
"movie_name":"serendipity",
"movie_released_year":"1999",
"movie_genre":"action"
}
}
..... 6 more attributes
}
I want to insert a new item if the item(that user id with dob) did not exist, otherwise add the movies to existing movies_watched map by checking if the movie is not already available the movies_watched map .
Currently, I am trying to use update(params) method.
Below is my approach:
function getInsertQuery (item) {
const exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(item).forEach(([key, item]) => {
if (key !== 'user_id' && key !== 'dob' && key !== 'movies_watched') {
exp.UpdateExpression += ` #${key} = :${key},`
exp.ExpressionAttributeNames[`#${key}`] = key
exp.ExpressionAttributeValues[`:${key}`] = item
}
})
let i = 0
Object.entries(item. movies_watched).forEach(([key, item]) => {
exp.UpdateExpression += ` movies_watched.#uniqueID${i} = :uniqueID${i},`
exp.ExpressionAttributeNames[`#uniqueID${i}`] = key
exp.ExpressionAttributeValues[`:uniqueID${i}`] = item
i++
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1)
return exp
}
The above method just creates update expression with expression names and values for all top level attributes as well as nested attributes (with document path).
It works well if the item is already available by updating movies_watched map. But throws exception if the item is not available and while inserting. Below is exception:
The document path provided in the update expression is invalid for update
However, I am still not sure how to check for duplicate movies in movies_watched map
Could someone guide me in right direction, any help is highly appreciated!
Thanks in advance
There is no way to do this, given your model, without reading an item from DDB before an update (at that point the process is trivial). If you don't want to impose this additional read capacity on your table for update, then you would need to re-design your data model:
You can change movies_watched to be a Set and hold references to movies. Caveat is that Set can contain only Numbers or Strings, thus you would have movie id or name or keep the data but as JSON Strings in your Set and then parse it back into JSON on read. With SET you can perform ADD operation on the movies_watched attribute. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html#Expressions.UpdateExpressions.ADD
You can go with single table design approach and have these movies watched as separate items with (PK:userId and SK:movie_id). To get a user you would perform a query and specify only PK=userId -> you will get a collection where one item is your user record and others are movies_watched. If you are new to DynamoDB and are learning the ropes, then I would suggest go with this approach. https://www.alexdebrie.com/posts/dynamodb-single-table/
The Problem
I have seen this question several times (also in the context of the Firebase Real-Time Database), but I haven't seen a convincing answer to it. The problem statement is fairly simple:
How can (authenticated) users choose a username that hasn't been taken yet?
First of all, the why: After a user authenticates, they have a unique user ID. Many web-apps, however, let the user choose a "display name" (how the user wants to appear on the website), in order to protect the users personal data (like real name).
The Users Collection
Given a data structure like the following it is possible to store a username along with other data for each user:
/users (collection)
/{uid} (document)
- name: "<the username>"
- foo: "<other data>"
However, nothing prevents another user (with a different {uid}) to store the same name in their record. As far as I know, there is no "security rule" that allows us to check if the name has already been by another user.
Note: A client side check is possible, but unsafe as a malicious client could omit the check.
The Reverse Mapping
Popular solutions are creating a collection with a reverse mapping:
/usernames (collection)
/{name} (document)
- uid: "<the auth {uid} field>"
Given this reverse mapping, it is possible to write a security rule to enforce that a username is not already taken:
match /users/{userId} {
allow read: if true;
allow create, update: if
request.auth.uid == userId &&
request.resource.data.name is string &&
request.resource.data.name.size() >= 3 &&
get(/PATH/usernames/$(request.resource.data.name)).data.uid == userId;
}
and to force a user to create a usernames document first:
match /usernames/{name} {
allow read: if true;
allow create: if
request.resource.data.size() == 1 &&
request.resource.data.uid is string &&
request.resource.data.uid == request.auth.uid;
}
I believe the solution is half-way there. However, there are still a few unsolved issues.
Remaining Issues / Questions
This implementation is quite involved already but it doesn't even solve the problem of users that want to change their user name (requires record deletion or update rules, etc.)
Another issue is, nothing prevents a user from adding multiple records in the usernames collection, effectively snatching all good usernames to sabotage the system.
So to the questions:
Is there a simpler solution to enforce unique usernames?
How can spamming the usernames collection be prevented?
How can the username checks be made case-insensitive?
I tried also enforcing existence of the users, with another exists() rule for the /usernames collection and then committing a batch write operation, however, this doesn't seem to work ("Missing or insufficient permissions" error).
Another note: I have seen solutions with client-side checks. BUT THESE ARE UNSAFE. Any malicious client can modify the code, and omit checks.
#asciimike on twitter is a firebase security rules developer.
He says there is currently no way to enforce uniqueness on a key on a document. https://twitter.com/asciimike/status/937032291511025664
Since firestore is based on Google Cloud datastore it inherits this issue. It's been a long standing request since 2008.
https://issuetracker.google.com/issues/35875869#c14
However, you can achieve your goal by using firebase functions and some strict security rules.
You can view my entire proposed solution on medium.
https://medium.com/#jqualls/firebase-firestore-unique-constraints-d0673b7a4952
Created another, pretty simple solution for me.
I have usernames collection to storing unique values. username is available if the document doesn't exist, so it is easy to check on front-end.
Also, I added the pattern ^([a-z0-9_.]){5,30}$ to valide a key value.
Checking everything with Firestore rules:
function isValidUserName(username){
return username.matches('^([a-z0-9_.]){5,30}$');
}
function isUserNameAvailable(username){
return isValidUserName(username) && !exists(/databases/$(database)/documents/usernames/$(username));
}
match /users/{userID} {
allow update: if request.auth.uid == userID
&& (request.resource.data.username == resource.data.username
|| isUserNameAvailable(request.resource.data.username)
);
}
match /usernames/{username} {
allow get: if isValidUserName(username);
}
Firestore rules will not allow updating user's document in case if the username already exists or have an invalid value.
So, Cloud Functions will be handling only in case if the username has a valid value and doesn't exist yet. So, your server will have much less work.
Everything you need with cloud functions is to update usernames collection:
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp(functions.config().firebase);
exports.onUserUpdate = functions.firestore
.document("users/{userID}")
.onUpdate((change, context) => {
const { before, after } = change;
const { userID } = context.params;
const db = admin.firestore();
if (before.get("username") !== after.get('username')) {
const batch = db.batch()
// delete the old username document from the `usernames` collection
if (before.get('username')) {
// new users may not have a username value
batch.delete(db.collection('usernames')
.doc(before.get('username')));
}
// add a new username document
batch.set(db.collection('usernames')
.doc(after.get('username')), { userID });
return batch.commit();
}
return true;
});
Create a series of cloud functions that are triggered whenever a document is added, updated, or deleted in the users table. The cloud functions will maintain a separate lookup table named usernames, with document ids set to the usernames. Your front-end app can then query the usernames collection to see if a username is available.
Here is TypeScript code for the cloud functions:
/* Whenever a user document is added, if it contains a username, add that
to the usernames collection. */
export const userCreated = functions.firestore
.document('users/{userId}')
.onCreate((event) => {
const data = event.data();
const username = data.username.toLowerCase().trim();
if (username !== '') {
const db = admin.firestore();
/* just create an empty doc. We don't need any data - just the presence
or absence of the document is all we need */
return db.doc(`/usernames/${username}`).set({});
} else {
return true;
}
});
/* Whenever a user document is deleted, if it contained a username, delete
that from the usernames collection. */
export const userDeleted = functions.firestore
.document('users/{userId}')
.onDelete((event) => {
const data = event.data();
const username = data.username.toLowerCase().trim();
if (username !== '') {
const db = admin.firestore();
return db.doc(`/usernames/${username}`).delete();
}
return true;
});
/* Whenever a user document is modified, if the username changed, set and
delete documents to change it in the usernames collection. */
export const userUpdated = functions.firestore
.document('users/{userId}')
.onUpdate((event, context) => {
const oldData = event.before.data();
const newData = event.after.data();
if ( oldData.username === newData.username ) {
// if the username didn't change, we don't need to do anything
return true;
}
const oldUsername = oldData.username.toLowerCase().trim();
const newUsername = newData.username.toLowerCase().trim();
const db = admin.firestore();
const batch = db.batch();
if ( oldUsername !== '' ) {
const oldRef = db.collection("usernames").doc(oldUsername);
batch.delete(oldRef);
}
if ( newUsername !== '' ) {
const newRef = db.collection("usernames").doc(newUsername);
batch.set(newRef,{});
}
return batch.commit();
});
This works for me efficiently whereby username must be unique. I am able to add and edit usernames without duplicates.
NOTE: username must be in lowercase always, this eliminates duplicates caused by case sensitivity.
Create users collection:
/users (collection)
/{uid} (document)
- name "the username"
Create usernames collection:
/usernames (collection)
/{name} (document)
- uid "the auth {uid} field"
Then in firestore use the following rules:
match /databases/{database}/documents {
match /usernames/{name} {
allow read,create: if request.auth != null;
allow update: if
request.auth.uid == resource.data.uid;
}
match /users/{userId}{
allow read: if true;
allow create, update: if
request.auth.uid == userId &&
request.resource.data.name is string &&
request.resource.data.name.size() >=3 &&
get(/databases/$(database)/documents/usernames/$(request.resource.data.name)).data.uid == userId;
}
}
I store the usernames in the same collection where each username occupies a unique document ID. That way the username which already exists will not be created in the database.
One possible solution is to store all usernames in a single document's usernames field and then permit only additions to that document using sets in Rules:
match /users/allUsernames {
function validateNewUsername() {
// Variables in functions are allowed.
let existingUsernames = resource.data.usernames;
let newUsernames = request.resource.data.usernames;
let usernameToAdd = newUsernames[newUsernames.size() - 1];
// Sets are a thing too.
let noRemovals = existingUsernames.toSet().difference(newUsernames.toSet()).size() == 0;
let usernameDoesntExistYet = !(usernameToAdd in existingUsernames.toSet());
let exactlyOneAddition = newUsernames.size() == existingUsernames.size() + 1;
return noRemovals && usernameDoesntExistYet && exactlyOneAddition;
}
allow update: if request.resource.data.keys().hasOnly(['usernames']) && validateNewUsername();
}
If you wanted to make a mapping from username -> uid (for validating other parts of the ruleset) this is possible in a single document too. You can just take the keyset of the document and do the same set operations as above.
This answer addresses your second concern about adding multiple records in the usernames collection. I'm not sure if this is the best method, but I believe a possible approach to prevent a given user from creating multiple username documents is writing an onCreate cloud function which checks if the user has an existing username document when a new username document is created. If the user does, then the cloud function can delete this document to prevent any malicious username parking.
Store the max integer user id used in the database in another collection. Query that collection everytime to find the max user id. You can even store other max ids in this collection. It can look something like this:
MaxIDCollection:
maxStudentIDDocument={ maxID: 55 } //lets say the max user id in db is 55
maxCourseIDDocument={ maxID: 77 }
Make sure to update the maxIDs everytime you add a new Student or Course.
If in future you add a new Student then by querying this collection you can know "if 55 is max then the new Student should get 56 as id."
How can I set id of post that I'm adding? I thought that getItemNextKey() returns id that will be assigned for the post, but it's not.
AddItem(data, downloadURLs) {
data.id= this.getItemNextKey(); // Persist a document id
data.upload = downloadURLs;
// console.log('this.uploadService.downloadURLs: ' + downloadURLs);
// console.log('data.upload: ' + data.upload);
this.db.collection('items').add(data);
}
I did this and it works now.
// Add a new document with a generated id
var addDoc = this.db.collection('items').add(data).then(ref => {
var updateNested = this.db.collection('items').doc(ref.id).update({
id: ref.id
});
});
As stated in the official docs
When you use set() to create a document, you must specify an ID for
the document to create. For example:
db.collection("cities").doc("new-city-id").set(data);
If you dont want to set an ID yourself you can use add
But sometimes there isn't a meaningful ID for the document, and it's
more convenient to let Cloud Firestore auto-generate an ID for you.
You can do this by calling add():
That is meant to be read as a dual upsert operation, upsert the document then the array element.
So MongoDB is a denormalized store for me (we're event sourced) and one of the things I'm trying to deal with is the concurrent nature of that. The problem is this:
Events can come in out of order, so each update to the database need to be an upsert.
I need to be able to not only upsert the parent document but an element in an array property of that document.
For example:
If the document doesn't exist, create it. All events in this stream have the document's ID but only part of the information depending on the event.
If the document does exist, then update it. This is the easy part. The update command is just written as UpdateOneAsync and as an upsert.
If the event is actually to update a list, then that list element needs to be upserted. So if the document doesn't exist, it needs to be created and the list item will be upserted (resulting in an insert); if the document does exist, then we need to find the element and update it as an upsert, so if the element exists then it is updated otherwise it is inserted.
If at all possible, having it execute as a single atomic operation would be ideal, but if it can only be done in multiple steps, then so be it. I'm getting a number of mixed examples on the net due to the large change in the 2.x driver. Not sure what I'm looking for beyond the UpdateOneAsync. Currently using 2.4.x. Explained examples would be appreciated. TIA
Note:
Reiterating that this is a question regarding the MongoDB C# driver 2.4.x
Took some tinkering, but I got it.
var notificationData = new NotificationData
{
ReferenceId = e.ReferenceId,
NotificationId = e.NotificationId,
DeliveredDateUtc = e.SentDate.DateTime
};
var matchDocument = Builders<SurveyData>.Filter.Eq(s => s.SurveyId, e.EntityId);
// first upsert the document to make sure that you have a collection to write to
var surveyUpsert = new UpdateOneModel<SurveyData>(
matchDocument,
Builders<SurveyData>.Update
.SetOnInsert(f => f.SurveyId, e.EntityId)
.SetOnInsert(f => f.Notifications, new List<NotificationData>())){ IsUpsert = true};
// then push a new element if none of the existing elements match
var noMatchReferenceId = Builders<SurveyData>.Filter
.Not(Builders<SurveyData>.Filter.ElemMatch(s => s.Notifications, n => n.ReferenceId.Equals(e.ReferenceId)));
var insertNewNotification = new UpdateOneModel<SurveyData>(
matchDocument & noMatchReferenceId,
Builders<SurveyData>.Update
.Push(s => s.Notifications, notificationData));
// then update the element that does match the reference ID (if any)
var matchReferenceId = Builders<SurveyData>.Filter
.ElemMatch(s => s.Notifications, Builders<NotificationData>.Filter.Eq(n => n.ReferenceId, notificationData.ReferenceId));
var updateExistingNotification = new UpdateOneModel<SurveyData>(
matchDocument & matchReferenceId,
Builders<SurveyData>.Update
// apparently the mongo C# driver will convert any negative index into an index symbol ('$')
.Set(s => s.Notifications[-1].NotificationId, e.NotificationId)
.Set(s => s.Notifications[-1].DeliveredDateUtc, notificationData.DeliveredDateUtc));
// execute these as a batch and in order
var result = await _surveyRepository.DatabaseCollection
.BulkWriteAsync(
new []{ surveyUpsert, insertNewNotification, updateExistingNotification },
new BulkWriteOptions { IsOrdered = true })
.ConfigureAwait(false);
The post linked as being a dupe was absolutely helpful, but it was not the answer. There were a few things that needed to be discovered.
The 'second statement' in the linked example didn't work
correctly, at least when translated literally. To get it to work, I had to match on the
element and then invert the logic by wrapping it in the Not() filter.
In order to use 'this index' on the match, you have to use a
negative index on the array. As it turns out, the C# driver will
convert any negative index to the '$' character when the query is
rendered.
In order to ensure they are run in order, you must include bulk write
options with IsOrdered set to true.
I'm using elasticabundle for symfony2 and I want to sort results I'm seeking by distance
I'm new to elasticsearch and I don't know how I can start
the query I'm using :
$c = $this->container->get('fos_elastica.finder.cities_index.cities');
$nameQuery = new \Elastica\Query\Match();
$nameQuery->setFieldQuery('tokens', $city);
$nameQuery->setSort(array("distance" => "asc"));// I want to achieve this
Thanks for your time
here is what I did following the example as provided by Dimitris Tsarouhas
The following setup allows to filter by keyword, order by id, and add all sort of other filtering using the $boolQuery ->addMust($yourfilter) formula.
We fetch the keyword as posted by a form. This is $data['query'] and by default we use it to perform a _all search, thus searching throughout all the fields listed in our config.yml file. To perform this search, we use the QueryString object, as this allows us to use wildcards.
We then look if exists the variable $data['status'], which comes through a dropdown select box (please note the use of strtolower(), without which the query wouldn't work - either that or you set up your own case-insensitive analyzer). If so, we use it to filter our results.
By default, we also want to narrow down our search and pick up only active users.
$data = $form->getData();
$finder = $this->container->get('fos_elastica.finder.search.user');
$keyword = $data['query'];
$status= $data['status'];
$keywordQuery = new QueryString;
$keywordQuery->setQuery('*' . $keyword . '*');
$query = new Query();
$query->setQuery($keywordQuery);
$query->setSort(array('id' => array('order' => 'asc')));
$boolQuery = new Bool();
$active= new Term();
$active->setTerm('active', true);
$boolQuery ->addMust($active);
if (!empty($status)) {
$statusQuery = new Term();
$statusQuery->setTerm('status', strtolower($status->getName()));
$boolQuery ->addMust($typeQuery);
}
$query->setFilter($boolQuery);
$entities = $finder->find($query);
And of course do not forget to import the necessary libraries:
use
Elastica\Query\QueryString,
Elastica\Query,
Elastica\Filter\Bool,
Elastica\Filter\Term
;
Remember that to be able to perform actions on fields (searching, sorting, etc) these haveto be included in your config.yml. Of course, this would cause the field to be automatically picked up when searching generally onto a certain entity. So, if you need to avoid this, but you still need certain fields to be available to elastica. just define it as showed below:
user:
mappings:
name: ~
surname: ~
status: ~
active: { include_in_all: false }
id: { include_in_all: false }