Hi I am working in a NestJS project using mongodb and mongoose.
I have to create transactions with a lot of promises inside, so i think it was a good idea to use Promise.all() inside my transaction for performace issues.
Unfortunately when i started working with my transactions i have a first issue, i was using
session.startTransaction(); and my code was throwing the following error:
Given transaction number 2 does not match any in-progress transactions. The active transaction number is 1, the error was thrown sometimes, not always but it was a problem
So i read the following question Mongoose `Promise.all()` Transaction Error, and i started to use withTransaction(), this solved the problem, but now mi code does not work propertly.
the code basically takes an array of bookings and then creates them, also needs to create combos of the bookings, what I need is that if a creation of a booking or a combo fail nothing should be inserted, for perfomance I use Promise.all().
But when i execute the function sometimes it creates more bookings than expected, if bookingsArray is from size 2, some times it creates 3 bookings and i just don't know why, this occurs very rarely but it is a big issue.
If i remove the Promise.all() from the transaction it works perfectly, but without Promise.all() the query is slow, so I wanted to know if there is any error in my code, or if you just cannot use Promise.all() inside a mongodb transaction in Nestjs
Main function with the transaction and Promise.all(), this one sometimes create the wrong number of bookings
async createMultipleBookings(
userId: string,
bookingsArray: CreateBookingDto[],
): Promise<void> {
const session = await this.connection.startSession();
await session.withTransaction(async () => {
const promiseArray = [];
for (let i = 0; i < bookingsArray.length; i++) {
promiseArray.push(
this.bookingRepository.createSingleBooking(
userId,
bookingsArray[i],
session,
),
);
}
promiseArray.push(
this.bookingRepository.createCombosBookings(bookingsArray, session),
);
await Promise.all(promiseArray);
});
session.endSession();
}
Main function with the transaction and withot Promise.all(), works fine but slow
async createMultipleBookings(
userId: string,
bookingsArray: CreateBookingDto[],
): Promise<void> {
const session = await this.connection.startSession();
await session.withTransaction(async () => {
for (let i = 0; i < bookingsArray.length; i++) {
await this.bookingRepository.createSingleBooking(
userId,
bookingsArray[i],
session,
);
}
await this.bookingRepository.createCombosBookings(bookingsArray, session);
});
session.endSession();
}
Functions called inside the main function
async createSingleBooking(
userId: string,
createBookingDto: CreateBookingDto,
session: mongoose.ClientSession | null = null,
) {
const product = await this.productsService.getProductById(
createBookingDto.productId,
session,
);
const user = await this.authService.getUserByIdcustomAttributes(
userId,
['profile', 'name'],
session,
);
const laboratory = await this.laboratoryService.getLaboratoryById(
product.laboratoryId,
session,
);
if (product.state !== State.published)
throw new BadRequestException(
`product ${createBookingDto.productId} is not published`,
);
const bookingTracking = this.createBookingTraking();
const value = product.prices.find(
(price) => price.user === user.profile.role,
);
const bookingPrice: Price = !value
? {
user: user.profile.role,
measure: Measure.valorACotizar,
price: null,
}
: value;
await new this.model({
...createBookingDto,
userId,
canceled: false,
productType: product.productType,
bookingTracking,
bookingPrice,
laboratoryId: product.laboratoryId,
userName: user.name,
productName: product.name,
laboratoryName: laboratory.name,
facultyName: laboratory.faculty,
createdAt: new Date(),
}).save({ session });
await this.productsService.updateProductOutstanding(
createBookingDto.productId,
session,
);
}
async createCombosBookings(
bookingsArray: CreateBookingDto[],
session: mongoose.ClientSession,
): Promise<void> {
const promiseArray = [];
for (let i = 1; i < bookingsArray.length; i++) {
promiseArray.push(
this.combosService.createCombo(
{
productId1: bookingsArray[0].productId,
productId2: bookingsArray[i].productId,
},
session,
),
);
}
await Promise.all(promiseArray);
}
also this is how i create the connection element:
export class BookingService {
constructor(
#InjectModel(Booking.name) private readonly model: Model<BookingDocument>,
private readonly authService: AuthService,
private readonly bookingRepository: BookingRepository,
#InjectConnection()
private readonly connection: mongoose.Connection,
) {}
When we get data from useQuery hook, I need to parse the data a specific type before it return to user. I want data which return from useQuery hook should be of "MyType" using the parsing function i created below. I am unable to find method to use my parsing function. Is there any way to do it? I don't want to rely on schema structure for data type.
type MyType = {
id: number;
//some more properties
}
function parseData(arr: any[]): MyType[]{
return arr.map((obj, index)=>{
return {
id: arr.id,
//some more properties
}
})
}
const {data} = await useQuery('fetchMyData', async ()=>{
return await axios.get('https://fake-domain.com')
}
)
I would take the response from the api and transform it inside the queryFn, before you return it to react-query. Whatever you return winds up in the query cache, so:
const { data } = await useQuery('fetchMyData', async () => {
const response = await axios.get('https://fake-domain.com')
return parseData(response.data)
}
)
data returned from useQuery should then be of type MyType[] | undefined
There are a bunch of other options to do data transformation as well, and I've written about them here:
https://tkdodo.eu/blog/react-query-data-transformations
I think you should create your own hook and perform normalisation there:
const useParseData = () => {
const { data } = await useQuery('fetchMyData', async () => {
return await axios.get('https://fake-domain.com')
}
return parseData(data)
}
And where you need this data you could just call const parsedData = useParseData()
This question already has an answer here:
How to access route parameter inside getServerSideProps in Next.js?
(1 answer)
Closed 1 year ago.
I am tryin to fetch user data from MongoDB database using getServerSideProps with dynamic path. Here is my code.
import dbConnect from 'lib/dbConnect'
import User from 'models/User'
export default function refID({user}){
return(
<>
<p>USERID:{user.userID}</p>
<p>USERNAME:{user.userName}</p>
</>
);
}
export async function getServerSideProps({ params }) {
await dbConnect()
const user = await User.findOne({userID}).lean()
user._id = user._id.toString()
return { props: { user } }
}
I have tried using hardcoded data.ie 'userID:S7L4SU' which works fine except that for only that one user.
How can I define the userID such that it fetches data for that ID ?I have tried couple of methods which resulted to errors..
Sample path:http://localhost:3000/p/[userID]
How will i get around for dynamic path to work for all users in the DATABASE??Help here
Try this:
export async function getServerSideProps(ctx) {
const { userID } = ctx.query;
await dbConnect()
const user = await User.findOne({userID}).lean()
if (user !== null) {
user._id = user._id.toString()
}
return { props: { user } }
}
I have this update query in mongoose. It's 1600 posts and takes like 5 min to run.
What's the bottleneck? Am I using the wrong approach?
export const getAndStoreLatestKPI = async () => {
console.log("start kpi");
try {
const marketCaps = await getKPI();
const stocks = await mongoose.model("stock").find().exec();
for (const stock of stocks) {
const marketCap = marketCaps.find(
(marketCap) => marketCap.i === stock.insId
);
if (marketCap != null) {
const marketCapAdjustedVal =
stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;
const update = {
marketCap: marketCapAdjustedVal,
};
console.log(marketCapAdjustedVal);
await mongoose
.model("stock")
.findOneAndUpdate({ insId: stock.insId }, { update });
}
}
console.log("done");
return Promise.resolve();
} catch (err) {
return Promise.reject(err);
}
};
export const getKPI = async (kpiId: number) => {
try {
const kpiFetch = await Axios.get(someurl);
return Promise.resolve(kpiFetch.data.values);
} catch (err) {
return Promise.reject(err);
}
};
So the main bottle neck is your for loop. for each stock item you perform several "expensive" actions such as data fetching from external API + a single update, and you're doing them 1 by 1.
What I would recommend you doing is looping on several items at once. similar to the idea multithreading.
There are several different solutions on how to do it in nodejs for example nodejs worker threads
However I personally use and recommend using bluebird which gives you this ability and many others straight out of the box.
Some sample code:
import Bluebird = require('bluebird');
const stocks = await mongoose.model("stock").find().exec();
await Bluebird.map(stocks, async (stock) => {
const marketCap = marketCaps.find(
(marketCap) => marketCap.i === stock.insId
);
if (marketCap != null) {
const marketCapAdjustedVal =
stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;
const update = {
marketCap: marketCapAdjustedVal,
};
console.log(marketCapAdjustedVal);
await mongoose
.model("stock")
.findOneAndUpdate({ insId: stock.insId }, { update });
}
}, {concurrency: 25})
// concurrency details how many concurrent process run parallel. the heavier they are the less you want concurrent for obvious reasons.
Following is the code provided in Collections in AngularFirestore.
export class AppComponent {
private shirtCollection: AngularFirestoreCollection<Shirt>;
shirts: Observable<ShirtId[]>;
constructor(private readonly afs: AngularFirestore) {
this.shirtCollection = afs.collection<Shirt>('shirts');
// .snapshotChanges() returns a DocumentChangeAction[], which contains
// a lot of information about "what happened" with each change. If you want to
// get the data and the id use the map operator.
this.shirts = this.shirtCollection.snapshotChanges().map(actions => {
return actions.map(a => {
const data = a.payload.doc.data() as Shirt;
const id = a.payload.doc.id;
return { id, ...data };
});
});
}
}
Here method snapshotChanges() returns observable of DocumentChangeAction[]. So why using a map to read it when it has only one array and it will loop only one time?