I'd need to disable index refreshing for the course of a batch indexing (of gigabytes) and set it back again after it's done. But from the source code of elastic4s I can't find a way to do it other than at index creation time... Is it possible? Or is there a workaround for this?
In java client :
client
.admin
.indices()
.prepareUpdateSettings()
.setSettings(settings)
.setIndices(indexName)
.execute()
.actionGet()
Natively :
curl -XPUT 'localhost:9200/my_index/_settings' -d '
{
"index" : {
"refresh_interval" : -1
}
}
'
This is how you do it in elastic4s (for example setting the refresh interval property).
client.execute {
update settings "myindex" set Map("index.refresh_interval" -> "10s")
}
Note: Not all settings can be changed at runtime or after the index is created.
Note 2: I have added this API in response to your question, and is only available in version 1.5.1 onwards.
Note 3: I could back port it to 1.4.x or 1.3.x if required by anyone.
Related
I recently updated from Wagtail 2.13.5 to 3.0.3. After the update, the search() method on Wagtail's PageQuerySet returns no results for search terms that clearly should return results (and which do when using the older version).
For example, under 2.13, where search_backend is an instance of wagtail.contrib.postgres_search.backend.PostgresSearchBackend, and qs is an instance of wagtail.core.query.PageQuerySet the following returns lots of results:
search_backend.search('popular-search-term', qs, fields=None, operator=None, order_by_relevance=True, partial_match=True)
But under 3.0.3, where search_backend is now an instance of wagtail.search.backends.database.postgres.postgres.PostgresSearchBackend and qs is an instance of wagtail.query.PageQuerySet, the same call to search() will return nothing (an empty queryset).
The data in the qs queryset is the same in both cases, so maybe I'm missing something in my configuration of the search backend? My "settings.py" file has:
WAGTAILSEARCH_BACKENDS = {
'default': {
'BACKEND': 'wagtail.search.backends.database',
'SEARCH_CONFIG': 'english',
},
}
and
INSTALLED_APPS = [
...
'wagtail.search',
'wagtail.search.backends.database.postgres',
...
]
I had to guess at the value for 'wagtail.search.backends.database.postgres'. AFAICT, Wagtail's docs don't mention what should go into INSTALLED_APPS. But the pre-upgrade value of 'wagtail.contrib.postgres_search' would clearly be wrong, as that module has been removed.
Anyone got an idea why calling search() on a PageQuerySet would incorrectly return no results?
The steps for changing the search backend are documented at https://docs.wagtail.org/en/stable/releases/2.15.html#database-search-backends-replaced. In particular:
You should remove wagtail.contrib.postgres_search from INSTALLED_APPS, and do not need to add anything in its place - the existing wagtail.search app is sufficient
After changing over, you need to re-run the ./manage.py update_index command to ensure that the new search index is populated.
I am trying to do as in title with the following code:
resp = requests.post("http://es_7:9200/my-index/_clone/new-my-index")
And I get this error:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://es_7:9200/my-index/_clone/new-my-index
Seems like it is not existing query, but as per docs it should work fine.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clone-index.html
What may be the issue? Thanks in advance
This route was added in the release 7.4
Could it be you are using a prior release ?
I am using version 7.0.1 how to clone the index within this version?
To perform a clone like operation:
In the case you wish to perform a _clone artificially.
Say you have index1 as the original index and index2 being the clone,
I believe you have to :
Get the mappings and settings of index1
Create index2 with the said setting/mapping
_reindex the data from index1 to index2
I need to execute a command that is part of my migration scripts (mongock java library) conditionally based on Mongo version
In case Mongo version is lower then 4.2 I need to execute this command. For Mongo 4.2 and higher it should not be executed.
db.getSiblingDB('admin').runCommand( { setParameter: 1, failIndexKeyTooLong: false } )
Is there a way how to do it?
Thank you,
Lukas
don't know if you already found a solution for this.
The idea is to figure out the Mongo's version inside the changeSet and then perform the required actions.
I will assume you are using the last Mongock's release for version 3.x.x, which is 3.3.2 or 4.0.x.alpha, so you should use MongockTemplate, instead of MongoTemplate. Don't worry, it's just a wrapper/decorator, you won't miss anything from the original API.
So you can do something like this:
#ChangeSet(id = "changeset-1", order = "001", author = "mongock")
public void changeSet(MongockTemplate mongockTemplate, ClientRepository clientRepository) {
String version = mongockTemplate.executeCommand(new Document("buildinfo", new BsonString("")))
.get("version")
.toString();
if(getNumericalVersion(version) < 42) {
//do your actions
} else {
//do you other actions
}
}
NOTE 1: We don't recommend using executeCommand. In this case, for taking the version it's fine, but in general we recommend not using it, specially for writes.
NOTE 2: Currently we are working on version 4, which will allow to scan multiple packages and even isolated ChangeLog classes. So another approach would be to take the version in the bean creation and inject the change or not depending on the MongoDB's version
I am trying to speed up "SELECT * FROM WHERE name=?" kind of queries in Play! + Scala app. I am using Play 2.4 + Scala 2.11 + play-slick-1.1.1 package. This package uses Slick-3.1 version.
My hypothesis was that slick generates Prepared statements from DBIO actions and they get executed. So I tried to cache them buy turning on flag cachePrepStmts=true
However I still see "Preparing statement..." messages in the log which means that PS are not getting cached! How should one instructs slick to cache them?
If I run following code shouldn't the PS be cached at some point?
for (i <- 1 until 100) {
Await.result(db.run(doctorsTable.filter(_.userName === name).result), 10 seconds)
}
Slick config is as follows:
slick.dbs.default {
driver="slick.driver.MySQLDriver$"
db {
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/staging_db?useSSL=false&cachePrepStmts=true"
user = "user"
password = "passwd"
numThreads = 1 // For not just one thread in HikariCP
properties = {
cachePrepStmts = true
prepStmtCacheSize = 250
prepStmtCacheSqlLimit = 2048
}
}
}
Update 1
I tried following as per #pawel's suggestion of using compiled queries:
val compiledQuery = Compiled { name: Rep[String] =>
doctorsTable.filter(_.userName === name)
}
val stTime = TimeUtil.getUtcTime
for (i <- 1 until 100) {
FutureUtils.blockFuture(db.compiledQuery(name).result), 10)
}
val endTime = TimeUtil.getUtcTime - stTime
Logger.info(s"Time Taken HERE $endTime")
In my logs I still see statement like:
2017-01-16 21:34:00,510 DEBUG [db-1] s.j.J.statement [?:?] Preparing statement: select ...
Also timing of this is also remains the same. What is the desired output? Should I not see these statements anymore? How can I verify if Prepared statements are indeed reused.
You need to use Compiled queries - which are exactly doing what you want.
Just change above code to:
val compiledQuery = Compiled { name: Rep[String] =>
doctorsTable.filter(_.userName === name)
}
for (i <- 1 until 100) {
Await.result(db.run(compiledQuery(name).result), 10 seconds)
}
I extracted above name as a parameter (because you usually want to change some parameters in your PreparedStatements) but that's definitely an optional part.
For further information you can refer to: http://slick.lightbend.com/doc/3.1.0/queries.html#compiled-queries
For MySQL you need to set an additional jdbc flag, useServerPrepStmts=true
HikariCP's MySQL configuration page links to a quite useful document that provides some simple performance tuning configuration options for MySQL jdbc.
Here are a few that I've found useful (you'll need to & append them to jdbc url for options not exposed by Hikari's API). Be sure to read through linked document and/or MySQL documentation for each option; should be mostly safe to use.
zeroDateTimeBehavior=convertToNull&characterEncoding=UTF-8
rewriteBatchedStatements=true
maintainTimeStats=false
cacheServerConfiguration=true
avoidCheckOnDuplicateKeyUpdateInSQL=true
dontTrackOpenResources=true
useLocalSessionState=true
cachePrepStmts=true
useServerPrepStmts=true
prepStmtCacheSize=500
prepStmtCacheSqlLimit=2048
Also, note that statements are cached per thread; depending on what you set for Hikari connection maxLifetime and what server load is, memory usage will increase accordingly on both server and client (e.g. if you set connection max lifetime to just under MySQL default of 8 hours, both server and client will keep N prepared statements alive in memory for the life of each connection).
p.s. curious if bottleneck is indeed statement caching or something specific to Slick.
EDIT
to log statements enable the query log. On MySQL 5.7 you would add to your my.cnf:
general-log=1
general-log-file=/var/log/mysqlgeneral.log
and then sudo touch /var/log/mysqlgeneral.log followed by a restart of mysqld. Comment out above config lines and restart to turn off query logging.
I'm in a meteor/mongodb environment and I'm trying to add a key/value to the users collection and check that the updated data now exists on the client.
In the cloud9 ide, on the command line, I enter
db.users.find({emails : { $elemMatch : { "address":"colm#1.com" }}});
This finds the corresponding entry.
Then,in cloud9 ide, I do
db.users.findAndModify({ query:{emails : { $elemMatch : { "address":"colm#1.com" }}}, update:{$set:{ "roles":"VIP"}}});
This works to the extent that when I repeat the db.user.find.. command above I now see the "roles":"VIP" at the same level as the "emails" key.
I restart the server and login to the website as colm#1.com.
In the console on the browser I type in
Meteor.user().emails[0].address and it replies colm#1.com
However when I type in
Meteor.user().roles it tells me undefined.
it looks as though the db change I make on the console in cloud9-ide, doesn't get propagated to the client ? Is this normal, or what am I doing wrong ?
P.S. I'm also confused by the existence of local and meteor databases. I believe that I am in meteor db by default but I'm not entirely sure about this.
best regards / Colm