Fast Searching Based on "custom fields" along with other fields - mongodb

Question

We have a legacy implementation of User Groups, which is way more than it implies. Users can be assigned to a group and you can create a hierarchy of groups. Groups can also have system wide permissions assigned to them, or a group can be used on some other module for permissions. You can even do a permission where it's something complicated like
((In Group1 or Group2) and (In Group3 and Group4)) or (In Group5 and (not IN Group1 or Group2))
When a permission like this is created, it will actually select all the users that match this, create a "derived group" and then assign those users to the new group.
In our new application, we have a completely different permissions system that handles these sorts of use cases pretty well, with it also being attribute based, rather than group/role based.
That being said, groups are still used for other things, other than permissions. We might build a report based upon a group, or send out emails to a group, etc. We still need this functionality.
It also looks like we're moving our current user information, into Mongo DB due to the fact that each client can customize the fields that are available for a user to populate, so a user might have a "job title" for one client, but another they would have a "designation", or "position". We call these "custom fields". The client can create as many of these fields as they want.
So that's the back story. My issue is that I don't really want to create a new "groups" implementation, since all we really need is a way to create and save filters for users, so when we need to send out an email to a specific subset of users, it will either use a filter that has already been saved, or create a new one.
So this is the original format for the user document in MongoDB:
{
"id": 123456,
"username": "john.smith#domain.com",
"first_name": "John",
"last_name": "Smith",
"email": "john.smith#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": {
"job_title": "Cashier",
"branch_code": "000123",
"social_team_name": "The Terrible Trolls"
}
},
{
"id": 123457,
"username": "jane.smith#domain.com",
"first_name": "Jane",
"last_name": "Smith",
"email": "john.smith#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": {
"job_title": "Mortgage Consultant",
"branch_code": "000123",
"social_team_name": "Team Savage"
}
},
{
"id": 123458,
"username": "morgan.jones#domain.com",
"first_name": "Morgan",
"last_name": "Jones",
"email": "morgan.jones#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": {
"job_title": "Regional Manager",
"branch_code": "000124",
"social_team_name": "The Terrible Trolls"
}
}
So we might want to create a filter where account.enabled = true AND employee_type='permanent' AND custom_fields.branch_code=000124. The filter could be any combination of fields in anyway.
Ultimately I'm wondering if this sort of structure is the best way to do this, I know I can use wildcard indexes to index the custom fields, but I'm still limited with regards to the amount of indexes I can create, so if a field is used in a query that isn't indexed, or we've hit our limit for creating indexes, then it's going to start slowing things down.
Another structure I saw is as follows:
{
"id": 123456,
"username": "john.smith#domain.com",
"first_name": "John",
"last_name": "Smith",
"email": "john.smith#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": [
{
"k": "Job Title",
"v": "Cashier"
},
{
"k": "Branch Code",
"v": "000123"
},
{
"k": "Social Team Name",
"v": "The Terrible Trolls"
}
]
},
{
"id": 123457,
"username": "jane.smith#domain.com",
"first_name": "Jane",
"last_name": "Smith",
"email": "john.smith#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": [
{
"k": "Job Title",
"v": "Mortgage Consultant"
},
{
"k": "Branch Code",
"v": "000123"
},
{
"k": "Social Team Name",
"v": "Team Savage"
}
]
},
{
"id": 123458,
"username": "morgan.jones#domain.com",
"first_name": "Morgan",
"last_name": "Jones",
"email": "morgan.jones#domain.com",
"employee_type": "permanent",
"account": {
"enabled": true,
"locked": false,
"redeem_only": false
},
"custom_fields": [
{
"k": "Job Title",
"v": "Regional Manager"
},
{
"k": "Branch Code",
"v": "000124"
},
{
"k": "Social Team Name",
"v": "The Terrible Trolls"
}
]
}
However, I'm not really sure if this would be better or not, as the problem still remains that we are limited by the amount of indexes we can create.
Is there a viable solution for this (links to articles/resources would be great), or am I going to end up saving a "filter", selecting all the users that apply to the filter and then assigning them to "filter" for easy lookup, but then have to rebuild every time a user updates their information, gets promoted, or anything else that changes the field values?