I noticed my delta index wasn't being merged into the main index and after doing some investigating discovered this happens when I try to merge the delta index to the main index immediately after rotating the delta index.
This is the command I was running on my dev computer (Windows):
C:\Sphinx\bin\indexer.exe --config C:\wamp\www\path\to\sphinx\etc\sphinx_windows.conf my_delta_index --rotate && C:\Sphinx\bin\indexer.exe --config C:\wamp\www\path\to\sphinx\etc\sphinx_windows.conf --merge my_index my_delta_index --rotate
The output showed that both commands executed successfully, but items didn't show up in search that should have.
When I tried running the first part of that command:
C:\Sphinx\bin\indexer.exe --config C:\wamp\www\path\to\sphinx\etc\sphinx_windows.conf my_delta_index --rotate
Followed a few seconds later by:
C:\Sphinx\bin\indexer.exe --config C:\wamp\www\path\to\sphinx\etc\sphinx_windows.conf --merge my_index my_delta_index --rotate
Everything worked.
I haven't been able to find anything suggesting this in the documentation, but do you have to wait a certain amount of time after rotating a delta index before merging it into the main index?
--rotate by its nature is asynchronous - indexer builds the new version of the index, but just 'tells' searchd to 'activate' it. searchd can do this at its leasure. It may take effect right away, or may take many seconds.
So when the 'merge' happens, its quite possible that the new index hasnt finished being 'activated'; so the merge process, just remerges the old version of delta. (the old version is still in use until rotation completes)
But its also possible that its the second rotation that hasnt completed, searchd hadnt finished loading the new merged main yet, so you still searching the old version.
... so yes, you do need to wait after running a --rotate style reindex before doing anything more with the index. There is no built in commands to monitor the process, all can do is either watch the searchd log file, or watch the filenames of the indexes, as the old and new are switched around.
Related
I'm using mongodump to migrate a 200+ GB database to directoryPerDB. In the process of trying to test a smaller dataset for finding the optimal number of parallel dumps I could run (per this answer), my mongodump keeps resuming from where I left off.
So if I cancel it at 10% and then re-run it, it picks right back at 10%.
I've tried deleting the dump folder, exporting to an archive file, changing my target and current folders, and restarting mongod, but I can't find any way to force mongodump to start over from the beginning on the collection.
Thanks in advance for your help!
I recently upgraded sphinx to version 2.2.11 on Ubuntu.
THen I started getting daily emails where a process is attempting to connect and generating this error:
ERROR: index 'test1stemmed': sql_connect: Access denied for user 'test'#'localhost'
ERROR: index 'test1': sql_connect: Access denied for user 'test'#'localhost'
The email warning has a topic which I assume is the info regarding the root of the problem
. /etc/default/sphinxsearch && if [ "$START" = "yes" ] && [ -x /usr/bin/indexer ]; then /usr/bin/indexer --quiet --rotate --all; fi
so /etc/default/sphinxsearch does have the start variable as yes.
but the /usr/bin/indexer is total gibberish.
Such a user never existed on the system AFAIK.
It would be interesting to know how this process got generated, but more importantly
How can this process be safely stopped?
I've seen that happen, it comes from the Sphinx install 'package'. Whoever setup that package, created a cron task that does that indexer --all command, that just tries to reindex every index (once a day IIRC). The package maintainer thought they being helpful :)
From https://packages.ubuntu.com/bionic/ppc64el/sphinxsearch/filelist
looks like it might be in
/etc/cron.d/sphinxsearch
You could remove that cron task, if dont want it.
Presumably you already have some other process for actually updating your actual real 'live' indexes. (either dedicated cron tasks, or maybe use RT indexes or whatever)
Also it seems you still have these 'test' indexes in your sphinx.conf. Maybe left over from the initial installation. Installing a new package I dont think would overwrite sphinx.conf to add them later?
May want to clear them out of your sphinx.conf if don't use them, could simplify the file.
(although possibly still want to get of the --all cron, which just blindly reindexes everything daily!)
What is the best method to transfer Sphinx real time index from one machine to another. If it was disk index, I could just move the database and re-index it again, but the index is RT. Thanks in advance!
Stop searchd on source gracefully. (ie searchd --stopwait, rather than just forceibly killing it, or crashing etc)
Copy /var/folder/indexname* to destination machine. (where using the prefix as noted in the index definition)
Copy the index definition to the destination.
Start up searchd on destination.
Most likly to work successfully if both machines have the same version of sphinx installed.
Is it possible to modify the MongoDB oplog and replay it?
A bug caused an update to be applied to more documents than it was supposed to be, overwriting some data. Data was recovered from backup and reintegrated, so nothing was actually lost, but I was wondering if there was a way to modify the oplog to remove or modify the offending update and replay it.
I don't have in depth knowledge of MongoDB internals, so informative answers along the lines of, "you don't understand how it works, it's like this" will also be considered for acceptance.
One of the big issues in application or human error data corruption is that the offending write to the primary will immediately be replicated to the secondary.
This is one of the reasons that users take advantage of "slaveDelay" - an option to run one of your secondary nodes with a fixed time delay (of course that only helps you if you discover the error or bug during the time period that's shorter than the delay on that secondary).
In case you don't have such a set-up, you have to rely on a backup to recreate the state of the records you need to restore to their pre-bug state.
Perform all the operations on a separate stand-alone copy of your data - only after verifying that everything was properly recreated should you then move the corrected data into your production system.
What is required to be able to do this is a recent copy of the backup (let's say the backup is X hours old) and the oplog on your cluster must hold more than X hours worth of data. I didn't specify which node's oplog because (a) every member of the replica set has the same contents in the oplog and (b) it is possible that your oplog size is different on different node members, in which case you want to check the "largest" one.
So let's say your most recent backup is 52 hours old, but luckily you have an oplog that holds 75 hours worth of data (yay).
You already realized that all of your nodes (primary and secondaries) have the "bad" data, so what you would do is restore this most recent backup into a new mongod. This is where you will restore these records to what they were right before the offending update - and then you can just move them into the current primary from where they will get replicated to all the secondaries.
While restoring your backup, create a mongodump of your oplog collection via this command:
mongodump -d local -c oplog.rs -o oplogD
Move the oplog to its own directory renaming it to oplog.bson:
mkdir oplogR
mv oplogD/local/oplog.rs.bson oplogR/oplog.bson
Now you need to find the "offending" operation. You can dump out the oplog to human readable form, using the bsondump command on oplogR/oplog.bson file (and then use grep or what-not to find the "bad" update). Alternatively you can query against the original oplog in the replica set via use local and db.oplog.rs.find() commands in the shell.
Your goal is to find this entry and note its ts field.
It might look like this:
"ts" : Timestamp( 1361497305, 2789 )
Note that the mongorestore command has two options, one called --oplogReplay and the other called oplogLimit. You will now replay this oplog on the restored stand-alone server BUT you will stop before this offending update operation.
The command would be (host and port are where your newly restored backup is):
mongorestore -h host --port NNNN --oplogReplay --oplogLimit 1361497305:2789 oplogR
This will restore each operation from the oplog.bson file in oplogR directory stopping right before the entry with ts value Timestamp(1361497305, 2789).
Recall that the reason you were doing this on a separate instance is so you can verify the restore and replay created correct data - once you have verified it then you can write the restored records to the appropriate place in the real primary (and allow replication propagate the corrected records to the secondaries).
If I update a resource in my sphinx.conf file I can reindex with --rotate and everything works fine. If I update an index in my sphinx.conf or add a new index --rotate has no effect and I have to restart searchd.
Am I doing this correctly, I feel like --rotate should correctly index the new or modified index configurations.
It depends on your sphinx version. In the latest versions just about anything (except maybe the searchd config section) will work with changing the config file.
Just changing the settings on an individual index, a --rotate indexing of the particular index is enough. If you change the settings of particular index, and dont actully reindex it, searchd probably wont pickup the changes. (because it reads stuff from the index header, not direct from conf file)
I just tested adding a index, and removing a index. both happened with a seemless rotate.
Sphinx 2.0.1-beta (r2792)
Prior to 0.9.9-rc1 - a restart would be required for most config file changes.
You have to restart searchd when modifying the sphinx.conf file.
Rotate will not effect new index additions to your sphinx.conf file - it reindexes an analogous index of the original. Kind of like having a file and file-copy(1) then swapping them over.
If you modify the .conf file its sort of like declaring a brand new index.
Thus --rotate does not work if the exact index does not previously exist.
See; http://sphinxsearch.com/docs/2.0.1/ref-indexer.html