Is there any way with FeathersJS service to create a compound index for a MongoDB DB?
Find the place where you create MongoDB client. If you used FeatherJS generator, - it is probably at src/mongodb.js. In the promise result where client is already available, - get the collection and add indexes:
// src/app.ts
import mongodb from './mongodb';
...
app.configure(mongodb);
...
// src/mongodb.ts
export default function (app: Application): void {
const connection = app.get(`mongodb`);
const database = connection.substr(connection.lastIndexOf(`/`) + 1);
const mongoClient:Promise<Db> = MongoClient.connect(connection, {
useNewUrlParser: true, useUnifiedTopology: true
}).then((client:MongoClient) => {
const db:Db = client.db(database);
try {
db.collection(`users`).createIndex({ name: 1 });
} catch(error:any) {
console.error(`mongodb: failed to created indexes, error: `, error);
}
return db;
});
app.set(`mongoClient`, mongoClient);
}
Related
i posted this question some time ago at the FilesCollection Github Repo (https://github.com/veliovgroup/Meteor-Files), but I'll hope to reach out here anyone who is familiar with Meteor and React Native.
My problem is, that I'm not sure how to use a FilesCollection with React Native (RN) and Meteor.
I used the official Meteor guide to set up a RN app: https://guide.meteor.com/react-native.html
So i have access to #meteorrn/core but how is it now possible for me to access a FileCollection.
Usually you would do this with (on a non-RN-web app):
import { FilesCollection } from "meteor/ostrio:files";
export const ImageDB = new FilesCollection(chosenSettings);
where chosenSettings are some settings might be e.g.
const chosenSettings = {
collectionName: "Images",
allowClientCode: false,
onBeforeUpload(file) {
if (file.size <= 10485760 && /png|jpg|jpeg/i.test(file.extension)) {
return true;
}
return "Please upload image, with size equal or less than 10MB";
},
storagePath: `${process.env.PWD}/images`,
};
However, with RN it is not possible to access FilesCollection from meteor/ostrio:files as I don't have any meteor relation here
With other collections I can use Mongo.Collection from #meteorrn/core, however this is not possible with FilesCollection:
const collection = new Mongo.Collection("myCollection");
Any suggestions?
There is no "out-of-the-box" solution, as there is no FilesCollection on the RN client available. However, you could create a workaround strategy using a Meteor method.
Create a publication for the FilesCollection's underlying collection:
server
Meteor.publish('imagesForRN', function () {
return ImageDB.collection.find({}) // note the Mongo.collection is accessible here
})
Create a Method to resolve links:
server
Meteor.methods({
resolveRNLinks ({ ids }) {
return ImageDB.collection
.find({ _id: { $in: ids}})
.map(linkImage);
}
})
const linkImage = function (fileRef) {
const url = Images.link(fileRef, 'original', Meteor.absoluteUrl())
return { _id: fileRef._id, url }
}
Get the URLS of the subscribed docs
This assumes you have subscribed to the Image Collection as shown in the guide.
client
import Meteor from '#meteorrn/core'
const MyComponent = props => {
const { loading, myTodoTasks } = this.props;
useEffect(() => {
const ids = myTodoTasks.map(doc => doc._id)
// note, you should filter those ids, which you have already
// resolved so you only call for the new ones
Meteor.call('resolveRNLinks', { ids }, (err, urls) => {
// store urls in state or ref, depending on your usecase
})
})
}
Resources
https://github.com/TheRealNate/meteor-react-native
https://github.com/veliovgroup/Meteor-Files/blob/master/docs/collection.md
https://github.com/veliovgroup/Meteor-Files/blob/master/docs/link.md
I have API, where I get datas. I use mongoose to save it into my local MongoDB. Each time when I save, I create dynamically new model and use insertMany on it:
const data = result.Data;
const newCollectionName = `tab_${name.toLowerCase()}`;
const Schema = new mongoose.Schema({}, { strict: false });
const TAB = mongoose.model(newCollectionName, Schema);
return TAB.collection.drop(() => {
// eslint-disable-next-line
const clearJSONs = Object.values(data)
.filter(x => typeof x === 'object');
return TAB.insertMany(clearJSONs, (err, result2) => {
// console.log(result2);
res.json({ success: true });
});
});
But... later, when all almost complete, my mongoose service falls down ... And... I even dont know what to do.
upd. mongo log:
2018-06-17T13:43:09.883+0300 E STORAGE [conn58] WiredTiger error (24) [1529232189:883394][4799:0x7f9fe1d30700], WT_SESSION.create: /var/lib/mongodb/: directory-sync: open: Too many open files
How to solve this?
The problem was in ulimits of the system. The best solution was described by this guy
I was using sailsjs 0.12. It supported index attributes on models, also
i was using npm package Sails-hooks-mongoat to create inverse indexes and so.
It wasn't ideal, but it worked. Right now they dropped the index attribute and mongoat is currently unsafe and pending updates to work on Sails.js 1.0.
I would like to know the best approach to:
Create Indexes on new deployments.
Migrate (ensure?) indexes on deployment updates.
Since you are not allowed to run 'migrate: alter' in production (even if you try), one option is to create those index in bootstrap file ('config/bootstrap.js').
Imagine you have an User model like this:
var User = {
attributes: {
email : { type: 'string', unique: true, required: true },
username : { type: 'string', unique: true, required: true },
pin: { type: 'string'}
}
};
module.exports = User;
Then you can manually create the missing indexes like this in bootstrap file:
module.exports.bootstrap = async function(done) {
console.log("Loading bootstrap...")
if (process.env.NODE_ENV === 'test') {
}
if (process.env.NODE_ENV === 'production') {
console.log("CREATING DATABASE INDEX BY HAND")
// USER MODEL
var db = User.getDatastore().manager;
await db.collection(User.tableName).createIndex( { email: 1 }, {unique: true} );
await db.collection(User.tableName).createIndex( { username: 1 }, {unique: true} );
// PANDA MODEL
// db = Panda.getDatastore().manager;
// await db.collection(Panda.tableName).createIndex( { name: 1 }, {unique: true} );
}
// await initializeDatabase() // custom DB initialization...
return done();
};
The index are created only once, subsequent runs will not recreate those indexes.
The ensureIndex was an alias to createIndex function and it has been deprecated.
References:
Waterline manager reference
MongoDB create index reference
In development mode you can specify custom indexes in the model within Sails, or it will remove them when lifting the server and performing migration.
My (preferred) alternative approach is to manage all indexes on my own within the DB. In order to do so, you have to change the "migrate" attribute from "alter" to "safe" in models.js config file.
Note that in production mode "migrate" attribute is always set to "safe".
I'm new to MongoDB and I'm looking for a way to do the following:
I have a collection of a number of available "things" to be used.
The user can "save" a "thing" and decrement the number of available things.
But he has a time to use it before it expires.
If it expires, the thing has to go back to the collection, incrementing it again.
It would be ideal if there was a way to monitor "expiring dates" in Mongo. But in my searches I've only found a TTL (time to live) for automatically deleting entire documents.
However, what I need is the "event" of the expiration... Than I was wondering if it would be possible to capture this event with Change Streams. Then I could use the event to increment "things" again.
Is it possible or not? Or would there be a better way of doing what I want?
I was able to use Change Streams and TTL to emulate a cronjob. I've published a post explaining what I did in details and gave credits at:
https://www.patreon.com/posts/17697287
But, basically, anytime I need to schedule an "event" for a document, when I'm creating the document I also create an event document in parallel. This event document will have as its _id the same id of the first document.
Also, for this event document I will set a TTL.
When the TTL expires I will capture its "delete" change with Change Streams. And then I'll use the documentKey of the change (since it's the same id as the document I want to trigger) to find the target document in the first collection, and do anything I want with the document.
I'm using Node.js with Express and Mongoose to access MongoDB.
Here is the relevant part to be added in the App.js:
const { ReplSet } = require('mongodb-topology-manager');
run().catch(error => console.error(error));
async function run() {
console.log(new Date(), 'start');
const bind_ip = 'localhost';
// Starts a 3-node replica set on ports 31000, 31001, 31002, replica set
// name is "rs0".
const replSet = new ReplSet('mongod', [
{ options: { port: 31000, dbpath: `${__dirname}/data/db/31000`, bind_ip } },
{ options: { port: 31001, dbpath: `${__dirname}/data/db/31001`, bind_ip } },
{ options: { port: 31002, dbpath: `${__dirname}/data/db/31002`, bind_ip } }
], { replSet: 'rs0' });
// Initialize the replica set
await replSet.purge();
await replSet.start();
console.log(new Date(), 'Replica set started...');
// Connect to the replica set
const uri = 'mongodb://localhost:31000,localhost:31001,localhost:31002/' + 'test?replicaSet=rs0';
await mongoose.connect(uri);
var db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function () {
console.log("Connected correctly to server");
});
// To work around "MongoError: cannot open $changeStream for non-existent database: test" for this example
await mongoose.connection.createCollection('test');
// *** we will add our scheduler here *** //
var Item = require('./models/item');
var ItemExpiredEvent = require('./models/scheduledWithin');
let deleteOps = {
$match: {
operationType: "delete"
}
};
ItemExpiredEvent.watch([deleteOps]).
on('change', data => {
// *** treat the event here *** //
console.log(new Date(), data.documentKey);
Item.findById(data.documentKey, function(err, item) {
console.log(item);
});
});
// The TTL set in ItemExpiredEvent will trigger the change stream handler above
console.log(new Date(), 'Inserting item');
Item.create({foo:"foo", bar: "bar"}, function(err, cupom) {
ItemExpiredEvent.create({_id : item._id}, function(err, event) {
if (err) console.log("error: " + err);
console.log('event inserted');
});
});
}
And here is the code for model/ScheduledWithin:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var ScheduledWithin = new Schema({
_id: mongoose.Schema.Types.ObjectId,
}, {timestamps: true});
// timestamps: true will automatically create a "createdAt" Date field
ScheduledWithin.index({createdAt: 1}, {expireAfterSeconds: 90});
module.exports = mongoose.model('ScheduledWithin', ScheduledWithin);
Thanks for the detailed code.
I have two partial alternatives, just to give some ideas.
1.
Given we at least get the _id back, if you only need a specific key from your deleted document, you can manually specify _id when you create it and you'll at least have this information.
(mongodb 4.0)
A bit more involved, this method is to take advantage of the oplog history and open a watch stream at the moment of creation (if you can calculate it), via the startAtOperationTime option.
You'll need to check how far back your oplog history goes, to see if you can use this method:
https://docs.mongodb.com/manual/reference/method/rs.printReplicationInfo/#rs.printReplicationInfo
Note: I'm using the mongodb library, not mongoose
// https://mongodb.github.io/node-mongodb-native/api-bson-generated/timestamp.html
const { Timestamp } = require('mongodb');
const MAX_TIME_SPENT_SINCE_CREATION = 1000 * 60 * 10; // 10mn, depends on your situation
const cursor = db.collection('items')
.watch([{
$match: {
operationType: 'delete'
}
}]);
cursor.on('change', function(change) {
// create another cursor, back in time
const subCursor = db.collection('items')
.watch([{
$match: {
operationType: 'insert'
}
}], {
fullDocument : 'updateLookup',
startAtOperationTime: Timestamp.fromString(change.clusterTime - MAX_TIME_SPENT_SINCE_CREATION)
});
subCursor.on('change', function(creationChange) {
// filter the insert event, until we find the creation event for our document
if (creationChange.documentKey._id === change.documentKey._id) {
console.log('item', JSON.stringify(creationChange.fullDocument, false, 2));
subCursor.close();
}
});
});
I am trying to make db.collection.renameCollection with mongoose, but i can't find that function anywhere. Did they miss to add it or i am looking at wrong place?
As quick example of what i am doing is:
var conn = mongoose.createConnection('localhost',"dbname");
var Collection = conn.model(collectionName, Schema, collectionName);
console.log(typeof Collection.renameCollection);
Which show undefined.
var con = mongoose.createConnection('localhost',"dbname");
con.once('open', function() {
console.log(typeof con.db[obj.name]);
});
This give also undefined.
Here's an example that will perform a rename operation using Mongoose.
const mongoose = require('mongoose');
mongoose.Promise = Promise;
mongoose.connect('mongodb://localhost/test').then(() => {
console.log('connected');
// Access the underlying database object provided by the MongoDB driver.
let db = mongoose.connection.db;
// Rename the `test` collection to `foobar`
return db.collection('test').rename('foobar');
}).then(() => {
console.log('rename successful');
}).catch(e => {
console.log('rename failed:', e.message);
}).then(() => {
console.log('disconnecting');
mongoose.disconnect();
});
As you can see, the MongoDB driver exposes the renameCollection() method as rename(), which is documented here: http://mongodb.github.io/node-mongodb-native/2.2/api/Collection.html#rename