In Rx.NET, how to enrich values from a (async/observable) lookup - system.reactive

Lets say I have a stream of bids - and i want to enrich it with the bidders names:
[
{ bidder: 'user/7', bet: 20 },
{ bidder: 'user/8', bet: 21 },
{ bidder: 'user/7', bet: 25 },
/*...., 2 seconds later */
{ bidder: 'user/8', bet: 25 },
{ bidder: 'user/9', bet: 30 },
...
Bidder names come from a webservice:
GET '/users?id=7&id=8' =>
[{ user: 'user/7', name: 'Hugo Boss'}, { user: 'user/8', name: "Karl Lagerfeld"}
Which i wrap into a reactive read-through-cache:
IObservable<User> Users(IObservable<string> userIds);
Compose
Now I want to compose that into following output:
[
{ bidder: 'user/7', bet: 20, name: 'Hugo Boss' },
{ bidder: 'user/8', bet: 21, name: 'Karl Lagerfeld' },
{ bidder: 'user/7', bet: 25, name: 'Hugo Boss' },
/*...., 2 seconds later */
{ bidder: 'user/8', bet: 25, name: 'Karl Lagerfeld' },
{ bidder: 'user/9', bet: 30, name: 'Somebody else' },
...
Step 1
I guess I need to project the stream of bids to a stream of user ids. Simple Select. Then inside the read-throug-cache I split it up in chunks with Buffer(TimeSpan, int).
Now I have a stream of bids and one of users.
Step 2
But now, how to combine the two?
Hint: For the BIDs it would make sense to keep the order - but in my real code I don't care about the order. So I'd like to have a solution that does not rely on that the user cache returns users in the right order. Then a Zip could do the job.
I rather want to release all bids into my result stream as soon as I have the user information available.
Solution?
I'm pretty sure I need to maintain some temporary state somehow (window/buffer/...). But I don't know where and how.
Maybe this should be implemented as a custom operator; or maybe there is one out there already?
Any ideas?
Edit: It seems like it isn't possible to actually compose this on top of streams. Instead I need to get a promise (either Task or IObservable) for the userid->user function and leave it up to the promise to bulk load and/or cache users, if appropriate.

Something like the following should work. Represent the cache as an ISubject and you have an async cache. Internally in the cache you could buffer queries over a time window and batch request them to the server.
Some dummy classes
public struct User
{
public string id;
}
public struct Bid
{
public string userId;
public int bid;
}
The cache object itself
/// <summary>
/// Represent the cache by a subject that get's notified of
/// user requests and produces users. Internally this can
/// be buffered and chunked to the server.
/// </summary>
public static ISubject<string, User> UsersCacheSubject;
Method for dispatching and async queries to the DB
public static Task<User> UsersCache(string id)
{
var r = UsersCacheSubject
.Where(user=>user.id==id)
.Replay(1)
.Take(1);
UsersCacheSubject.OnNext(id);
return r.FirstAsync().ToTask();
}
try it out
public void TryItOut()
{
IObservable<Bid> bidderObservable = Observable.Repeat(new Bid());
var foo = from bidder in bidderObservable
from user in UsersCache(bidder.userId)
where bidder.userId == user.id
select new {bidder, user};
}

Isn't this just as simple as doing the following?
var query =
from b in bids
from u in Users(Observable.Return(b.Bidder))
select new
{
b.Bidder,
b.Bet,
u.Name
};

Related

try_join to make mongodb transactions sent at the same time

I'm new to Rust and I'm using the default MongoDB driver
https://docs.rs/mongodb/2.0.0/mongodb/
I remember when coding with Node.js, there was a possibility to send transactions with some Promise.all() in order to execute all transactions at the same time for optimization purposes, and if there are no errors, to make a commit to the transaction.
(Node.js example here: https://medium.com/#alkor_shikyaro/transactions-and-promises-in-node-js-ca5a3aeb6b74)
I'm trying to implement the same logic in Rust now, using try_join! but I'm always opposed to the problem:
error: cannot borrow session as mutable more than once at a time;
label: first mutable borrow occurs here
use mongodb::{bson::oid::ObjectId, Client, Database, options};
use async_graphql::{
validators::{Email, StringMaxLength, StringMinLength},
Context, ErrorExtensions, Object, Result,
};
use futures::try_join;
//use tokio::try_join; -> same thing
#[derive(Default)]
pub struct UserMutations;
#[Object]
impl UserMutations {
async fn user_followed<'ctx>(
&self,
ctx: &Context<'ctx>,
other_user_id: ObjectId,
current_user_id: ObjectId,
) -> Result<bool> {
let mut session = Client::with_uri_str(dotenv!("URI"))
.await
.expect("DB not accessible!")
.start_session(Some(session_options))
.await?;
session.start_transaction(Some(options::TransactionOptions::builder()
.read_concern(Some(options::ReadConcern::majority()))
.write_concern(Some(
options::WriteConcern::builder()
.w(Some(options::Acknowledgment::Majority))
.w_timeout(Some(Duration::new(3, 0)))
.journal(Some(false))
.build(),
))
.selection_criteria(Some(options::SelectionCriteria::ReadPreference(
options::ReadPreference::Primary
)))
.max_commit_time(Some(Duration::new(3, 0)))
.build())).await?;
let db = Client::with_uri_str(dotenv!("URI"))
.await
.expect("DB not accessible!").database("database").collection::<Document>("collection");
try_join!(
db.update_one_with_session(
doc! {
"_id": other_user_id
},
doc! {
"$inc": { "following_number": -1 }
},
None,
&mut session,
),
db.update_one_with_session(
doc! {
"_id": current_user_id
},
doc! {
"$inc": { "followers_number": -1 }
},
None,
&mut session,
)
)?;
Ok(true)
}
}
849 | | &mut session,
| | ------------ first mutable borrow occurs here
... |
859 | | &mut session,
| | ^^^^^^^^^^^^ second mutable borrow occurs here
860 | | )
861 | | )?;
| |_____________- first borrow later captured here by closure
Is there any way to send transaction functions sync to not lose any time on independent mutations? Does anyone have any ideas?
Thanks in advance!
Thanks, Patrick and Zeppi for your answers, I did some more research on this topic and also did my own testing. So, let's start.
First, my desire was to optimize transactional writes as much as possible, since I wanted the complete rollback possibility required by code logic.
In case you missed my comments to Patrick, I'll restate them here to better reflect what was my way of thinking about this:
I understand why this would be a limitation for multiple reads, but if
all actions are on separate collections (or are independent atomic
writes to multiple documents with different payloads) I don't see why
it's impossible to retain casual consistency while executing them
concurrently. This kind of transaction should never create race
conditions / conflicts / weird lock behaviour, and in case of error
the entire transaction is rolled back before being committed anyways.
Making an analogy with Git (which might be wrong), no merge conflicts
are created when separate files / folders are updated. Sorry for being
meticulous, this just sounds like a major speed boost opportunity.
But, after lookups I was opposed to this documentation:
https://github.com/mongodb/specifications/blob/master/source/sessions/driver-sessions.rst#why-does-a-network-error-cause-the-serversession-to-be-discarded-from-the-pool
An otherwise unrelated operation that just happens to use that same
server session will potentially block waiting for the previous
operation to complete. For example, a transactional write will block a
subsequent transactional write.
Basically, this means that even if you will send transaction writes concurrently, you won't gain much efficiency because MongoDB itself is a blocker. I decided to check if this was true, and since NodeJS driver setup allows to send transactions concurrently (as per: https://medium.com/#alkor_shikyaro/transactions-and-promises-in-node-js-ca5a3aeb6b74) I did a quick setup with NodeJS pointing to the same database hosted by Atlas in the free tier.
Second, statistics and code: That's the NodeJS mutation I will be using for tests (each test has 4 transactional writes). I enabled GraphQL tracing to benchmark this, and here are the results of my tests...
export const testMutFollowUser = async (_parent, _args, _context, _info) => {
try {
const { user, dbClient } = _context;
isLoggedIn(user);
const { _id } = _args;
const session = dbClient.startSession();
const db = dbClient.db("DB");
await verifyObjectId().required().validateAsync(_id);
//making sure asked user exists
const otherUser = await db.collection("users").findOne(
{ _id: _id },
{
projection: { _id: 1 }
});
if (!otherUser)
throw new Error("User was not found");
const transactionResult = session.withTransaction(async () => {
//-----using this part when doing concurrency test------
await Promise.all([
await createObjectIdLink({ db_name: 'links', from: user._id, to: _id, db }),
await db.collection('users').updateOne(
{ _id: user._id },
{ $inc: { following_number: 1 } },
),
await db.collection('users').updateOne(
{ _id },
{
$inc: { followers_number: 1, unread_notifications_number: 1 }
},
),
await createNotification({
action: 'USER_FOLLOWED',
to: _id
}, _context)
]);
//-----------end of concurrency part--------------------
//------using this part when doing sync test--------
//this as a helper for db.insertOne(...)
const insertedId = await createObjectIdLink({ db_name: 'links', from: user._id, to: _id, db });
const updDocMe = await db.collection('users').updateOne(
{ _id: user._id },
{ $inc: { following_number: 1 } },
);
const updDocOther = await db.collection('users').updateOne(
{ _id },
{
$inc: { followers_number: 1, unread_notifications_number: 1 }
},
);
//this as another helper for db.insertOne(...)
await createNotification({
action: 'USER_FOLLOWED',
to: _id
}, _context);
//-----------end of sync part---------------------------
return true;
}, transactionOptions);
if (transactionResult) {
console.log("The reservation was successfully created.");
} else {
console.log("The transaction was intentionally aborted.");
}
await session.endSession();
return true;
}
And related performance results:
format:
Request/Mutation/Response = Total (all in ms)
1) For sync writes in the transaction:
4/91/32 = 127
4/77/30 = 111
7/71/7 = 85
6/66/8 = 80
2/74/9 = 85
4/70/8 = 82
4/70/11 = 85
--waiting more time (~10secs)
9/73/34 = 116
totals/8 = **96.375 ms in average**
//---------------------------------
2) For concurrent writes in transaction:
3/85/7 = 95
2/81/14 = 97
2/70/10 = 82
5/81/11 = 97
5/73/15 = 93
2/82/27 = 111
5/69/7 = 81
--waiting more time (~10secs)
6/80/32 = 118
totals/8 = ** 96.75 ms ms in average **
Conclusion: the difference between the two is within the margin of error (but still on the sync side).
My assumption is with the sync way, you're spending time to wait for DB request/response, while in a concurrent way, you're waiting for MongoDB to order the requests, and then execute them all, which at the end of the day will cost the same time.
So with current MongoDB policies, I guess, the answer to my question will be "there is no need for concurrency because it won't affect the performance anyway." However, it would be incredible if MongoDB would allow parallelization of writes in transactions in future releases with locks on document level (at least for WiredTiger engine) instead of database level, as it is currently for transactions (because you're waiting for the whole write to finish until next one).
Feel free to correct me if I missed/misinterpreted something. Thanks!
This limitation is actually by design. In MongoDB, client sessions cannot be used concurrently (see here and here), and so the Rust driver accepts them as &mut to prevent this from happening at compile time. The Node example is only working by chance and is definitely not recommended or supported behavior. If you would like to perform both updates as part of a transaction, you'll have to run one update after the other. If you'd like to run them concurrently, you'll need to execute them without a session or transaction.
As a side note, a client session can only be used with the client that it was created from. In the provided example, the session is being used with a different one, which will cause an error.

Can you update existing data with an Array of objects using a mutation serverside with Apollo GraphQL?

Little background:
I'm building a Fantasy FPL App for my colleagues at work so we can have our own app with our own data and league. I'm using Vue.js on the FE and GraphQL + Apollo + MongoDB for the BE.
To clarify the question: I want to update my DB with data that is fetched from an external REST API using a mutation on the backend if possible.
So I've manually added some initial data to the DB(some of it does not exist in the external API like player avatars), and I want to now update some of the existing data fields through a Query to an external REST API. Namely points, rank, total points etc.
I've successfully made the Query to get the data from the REST API, and reduced it using Apollos RESTDataSource that I found in the Apollo docs here:
class LeagueAPI extends RESTDataSource {
constructor() {
super();
this.baseURL = 'https://fantasy.premierleague.com/api/leagues-classic/xxxxxx/standings/';
}
// Makes sure we can call the API(necessary headers)
willSendRequest(request) {
request.headers.set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36");
}
// Reduces data
leagueReducer(league) {
return {
player_id: league.entry || 0,
total: league.total,
player_name: league.player_name,
rank: league.rank,
previous_rank: league.last_rank,
points: league.event_total,
team_name: league.entry_name
}
}
// Get's total league data standings for current season
async getLeagueData() {
try {
const response = await this.get('');
const extractedResponse = response.standings.results
return Array.isArray(extractedResponse)
? extractedResponse.map(league => this.leagueReducer(league))
: [];
}
catch(error) {
console.log('Error in getLeagueData', error)
}
}
}
module.exports = LeagueAPI
In my response I get an Array with objects that I want to iterate over and update the DB accordingly so that the newest data is always displayed after a match.
Example of an object in the array:
{
"id": 57643299,
"event_total": 71,
"player_name": "John Doe",
"rank": 1,
"last_rank": 1,
"rank_sort": 1,
"total": 580,
"entry": 10432744,
"entry_name": "Amazing team name"
},
I have an updateGameWeek mutation that works when I use it in the Graphiql playground, but I'm wondering if there is a way to call a mutation on the backend?
And how would I do this since I need to iterate over it several times as there are many objects that need to be updated according to their corresponding ids on the backend.
Here's an example of an updateQuery that works:
mutation updateGameWeek{
updateGameWeek(player_id: 29729805, input: {
avatar:"somestring"
player_id: 29324805
})
This mutation correctly updates the players avatar by id, and it's basically the blueprint for what I want to use when I am updating the whole DB by player id's.
Here is also what a GameWeek Object looks like/defined as:
type GameWeek {
avatar: String
team_id: Int
points: Float
player_name: String
rank: Int
previous_rank: Int
total: Float
player_id: ID!
team_name: String
}
And also what the actual resolver for the mutation above looks like:
module.exports = async (_, { id, input }, { models }) => {
try {
const gameWeekToUpdate = await models.GameWeek.findOne({ _player_id: id });
Object.keys(input).forEach(value => {
gameWeekToUpdate[value] = input[value];
});
const updatedGameWeek = await gameWeekToUpdate.save();
return updatedGameWeek;
} catch (error) {
console.log('Error in updateGameWeek Mutation', error)
}
}

mongodb need to populate a new field with an old fields value, without destroying other data

I have a situation where a model changed at some point in time and I am faced with (for argument sake) half my data liks like this
{
_id: OID,
things: [{
_id:OID,
arm: string,
body: string
}],
other: string
}
and the other half of my data look like this
{
_id: OID,
things: [{
_id:OID,
upper_appendage: string,
body: string
}],
other: string
}
I would like to 'correct' half of the data - so that I DON'T have to accommodate both names for 'arm' in my application code.
I have tried a couple different things:
The first errors
db.getCollection('x')
.find({things:{$exists:true}})
.forEach(function (record) {
record.things.arm = record.things.upper_appendage;
db.users.save(record);
});
and this - which destroys all the other data in
db.getCollection('x')
.find({things:{$exists:true}})
.forEach(function (record) {
record.things = {
upper_appendage.arm = record.things.upper_appendage
};
db.users.save(record);
});
Keeping in mind that there is other data I want to maintain...
How can I do this???
the $rename operator should have worked for this job but unfortunately it doesn't seem to support nested array fields (as of mongodb server 4.2). instead you'd need a forEach like the following:
db.items.find({
things: {
$elemMatch: {
arm: {
$exists: true
}
}
}
}).forEach(function(item) {
for (i = 0; i != item.things.length; ++i)
{
item.things[i].upper_appendage = item.things[i].arm;
delete item.things[i].arm; ;
}
db.items.update({
_id: item._id
}, item);
})
note: i've assumed you want to make all records have upper_appendageand get rid of 'arm' field. if it's the other way you want, just switch things around.

Inconsistent results with Meteor's pub/sub feature

I'm experiencing inconsistent results with Meteor's pub/sub feature, and I suspect it's a source of confusion for a lot of developers hitting the threshold of an MVP built in Meteor becoming a production app.
Maybe this is a limitation of MergeBox:
Let's say I have a collection called Events, in which I have document-oriented structures, ie, nested Array, Objects. An Events document might look like so:
// an Events document //
{
_id: 'abc',
name: 'Some Event',
participation: {
'userOneId': {
games: {
'gameOneId': {
score: 100,
bonus: 10
}
},
{
'gameTwoId': : {
score: 100,
bonus: 10
}
}
}
},
'userTwoId': {
games: {
'gameOneId': {
score: 70,
bonus: 15
}
},
contests: {
'contestOneId': [2, 3, 6, 7, 4],
'contestTwoId': [9, 3, 7, 2, 1],
}
}
},
}
}
So at these events, users can optionally participate in games of certain types and contests of certain types.
Now, I want to restrict subscriptions to the Events collection based on the user (show only this user's participation), and, sometimes I'm only interested in changes to one subset of the data (like, show only the user's scores on 'gameOneId').
So I've created a publication like so:
Meteor.publish("events.participant", function(eventId, userId) {
if(!Meteor.users.findOne({_id: this.userId})) return this.ready();
check(eventId, String);
check(userId, String);
const includeFields = {
name: 1,
[`participation.${userId}`]: 1
};
return Events.find({_id: eventId}, {fields: includeFields});
});
This publication seems to work fine on the client if I do:
// in onCreated of events template //
template.autorun(function() {
const
eventId = FlowRouter.getParam('id'),
userId = Meteor.userId(),
subscription = template.subscribe('events.participant', eventId, userId);
if (subscription.ready()) {
const event = Events.findOne({_id: eventId}, parseIncludeFields(['name', `participation.${userId}`]));
template.props.event.set(event);
}
});
Happily, I can use the Event document returned that includes only the name field and all of the user's participation data.
But, later, in another template if I do:
// in onCreated of games template //
template.autorun(function() {
const
eventId = FlowRouter.getParam('id')
gameId = FlowRouter.getParam('gameId'),
userId = Meteor.userId(),
subscription = template.subscribe('events.participant', eventId, userId);
if(subscription.ready()) {
const event = Events.findOne({_id: eventId}, {fields: {[`participation.${userId}.games.${gameId}`]: 1}});
template.props.event.set(event);
}
});
I sometimes get back the data at event.participation[userId].games[gameId], and sometimes I don't - the Object that's suppose to be at gameId is non-existent, even the it exists in the Mongo document, and the subscription should include it. Why?
The only difference is between the two calls to Events.findOne() is that in the latter, I'm not requesting the name field. But, if this is a problem, why?. If minimongo already has the document, who cares if I request parts of it?
The subscriptions in both templates are identical - I'm doing this because the games template is available at a route, so the user could go straight to the games url, by-passing the events template altogether, so I want to be sure the client has the document it needs to render correctly.
The only way I've gotten around this is to make a straight Meteor method call to the server in the games template to fetch the subset of interest, but this seems like a cop-out.
If you've read this far, you're a champ!

Buffer grouped observables

I have the following class and array
class Hero {
id: number;
name: string;
}
const HEROES: Hero[] = [
{ id: 11, name: 'Narco' },
{ id: 12, name: 'Narco' },
{ id: 13, name: 'Bombasto' },
{ id: 14, name: 'Bombasto' },
{ id: 15, name: 'Bombasto' },
{ id: 16, name: 'Dynama' },
{ id: 17, name: 'Dynama' },
{ id: 18, name: 'Dynama' },
{ id: 19, name: 'Dynama' },
{ id: 20, name: 'Dynama' }
];
I want to create an observable that treats the HEROES array as the source, groups the array by name and emits the result as a single array, i.e., I should end up with three arrays, one for Narco, one for Bombasto and one for Dynama.
I can create my source observable as follows
var heroSource = Observable.create((observer:any) => {
HEROES.forEach((hero: Hero) => {
observer.next(hero);
})
});
I can then group heros by using groupBy, i.e.,
var groupHeroSource = heroSource
.groupBy((hero: Hero) => {return hero.name});
This effectively gives me as many observables as there are different names in my HEROES array (three in this case). However, these will be emitted as a sequence and ideally I'd like to buffer them until heroSource is complete. How would I use the buffer operator in rxjs on my grouped observables so that emit a single collection?
First of all you can create your initial observable much simpler:
var heroSource = Observable.from(HEROES);
Next, your groupBy mapper/selector can be abbreviated to:
var groupHeroSource = heroSource.groupBy((hero: Hero): string => hero.name);
To solve the original problem I needed to buffer the streams, since they are ready a buffer with time of 0 would do the work (I guess there should be a more elegant solution out there), use take(1) to take only the first result (and avoid a repeating buffer) and then merge all:
var finalGroup = groupHeroSource.map((ob) => ob.bufferWithTime(0).take(1)).mergeAll();
Note that since that since your array is actually static, putting it through a stream and then mapping it might not be the simplest solution, you can simply reduce it:
var grouped = HEROES.reduce((acc: any, hero: Hero) => {
acc[hero.name] = acc[hero.name] || [];
acc[hero.name].push(hero);
return acc;
}, {});
Since Object.values is not standard, you'll have to iterate the keys to get an array, yet, it might be a better fit for your need