skipping non-plain index rt (sphinx 2.1.6) - sphinx

There is the question. Sphinx, version 2.1.6. I used to rt(real time) index, but when indexing display message in koncole:
using config file 'sphinx.conf'...
skipping non-plain index 'rt'...
But at a connection to sphinxbase and write query mysql> desc rt - displays:
+------------+--------+
| Field | Type |
+------------+--------+
| id | bigint |
| id | field |
| first_name | field |
| last_name | field |
+------------+--------+
This is default data?? They do not meet my request. How to work with index rt?
Sphinx.conf.
source database
{
type = mysql
sql_host = 127.0.0.1
sql_user = test
sql_pass = test
sql_db = community
sql_port = 3306
mysql_connect_flags = 32 # enable compression
sql_query_pre = SET NAMES utf8
sql_query_pre = SET SESSION query_cache_type=OFF
}
source rt : database
{
sql_query_range = SELECT MIN(id),MAX(id) FROM mbt_accounts
sql_query = SELECT id AS 'accountId', first_name AS 'fname', last_name AS 'lname' FROM mbt_accounts WHERE id >= 0 AND id<= 1000
sql_range_step = 1000
sql_ranged_throttle = 1000 # milliseconds
}
index rt
{
source = rt
type = rt
path = /etc/sphinxsearch/rtindex
rt_mem_limit = 700M
rt_field = accountId
rt_field = fname
rt_field = lname
rt_attr_string = fname
rt_attr_string = lname
charset_type = utf-8
charset_table = 0..9, A..Z->a..z, _, -, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451
}
searchd
{
listen = localhost:9312 # port for API
listen = localhost:9306:mysql41 #port for a SphinxQL
log = /var/log/sphinxsearch/searchd.log
binlog_path = /var/log/sphinxsearch/
query_log = /var/log/sphinxsearch/query.log
query_log_format = sphinxql
pid_file = /var/run/sphinxsearch/searchd.pid
workers = threads
max_matches = 1000
read_timeout = 5
client_timeout = 300
max_children = 30
max_packet_size = 8M
binlog_flush = 2
binlog_max_log_size = 90M
thread_stack = 8M
expansion_limit = 500
rt_flush_period = 1800
collation_server = utf8_general_ci
compat_sphinxql_magics = 0
prefork_rotation_throttle = 100
}
Thanks.

indexer only works with indexes that have a 'source' - ie plain disk indexesd. ie indexer does the stuff in the source to get the data to create the index.
RT (Real Time) indexes work very differently. indexer is not involved with RT indexes at all. They are handled totally by searchd.
To add data to a RT index, you need to run a bunch of SphinxQL commands (INSERT, UPDATE etc) that actually add the data to the index.
(DESCRIBE works, because searchd knows the 'structure' of the index (you told it via the rt_field etc) - even if never inserted any data)
Ah, I think you are asking why the structure is different. That's probably because the index was probably created before, you modified sphinx.conf. If you change the definiton of a RT index, you need to 'destroy' the index, to allow it be recreated again.
The simplest way is to shutdown searchd, delete the index files, delete the binlog (it no longer relevent) and then restart searchd.
searchd --stopwait
rm /etc/sphinxsearch/rtindex*
rm /path/to/binlog* #(you dont define a path, so it must be the default, which varies)
searchd #(starts searchd again)

Related

sphinxsearch does not return me content and title fields

I am using sphinx search with document table. I want to fetch all the fields but it doesn't return me all the fields. Please check my config file and let me know where to make change? using test index
table : documents
Fields: id, group_id, group_id2, date_added, content, title
mysql> select * from test1 where match ('my document');
+------+----------+------------+
| id | group_id | date_added |
+------+----------+------------+
| 1 | 1 | 1461672351 |
| 2 | 1 | 1461672351 |
+------+----------+------------+
2 rows in set (0.00 sec)
mysql>
source src1
{
type = mysql
sql_host = localhost
sql_user = root
sql_pass = india#123
sql_db = test
sql_port = 3306
sql_query = \
SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \
FROM documents
sql_attr_uint = group_id
sql_attr_timestamp = date_added
sql_ranged_throttle = 0
sql_query_info = SELECT * FROM documents WHERE id=$id
}
source src1throttled : src1
{
sql_ranged_throttle = 100
}
index test1
{
source = src1
path = /var/lib/sphinxsearch/data/test1
docinfo = extern
dict = keywords
mlock = 0
morphology = none
min_word_len = 1
html_strip = 0
}
index test1stemmed : test1
{
path = /var/lib/sphinxsearch/data/test1stemmed
morphology = stem_en
}
index dist1
{
type = distributed
local = test1
local = test1stemmed
agent = localhost:9313:remote1
agent = localhost:9314:remote2,remote3
agent_connect_timeout = 1000
agent_query_timeout = 3000
}
index rt
{
type = rt
path = /var/lib/sphinxsearch/data/rt
rt_field = title
rt_field = content
rt_attr_uint = gid
}
indexer
{
mem_limit = 128M
}
searchd
{
listen = 9312
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
query_log = /var/log/sphinxsearch/query.log
read_timeout = 5
client_timeout = 300
max_children = 30
persistent_connections_limit = 30
pid_file = /var/run/sphinxsearch/searchd.pid
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
mva_updates_pool = 1M
max_packet_size = 8M
max_filters = 256
max_filter_values = 4096
max_batch_queries = 32
workers = threads # for RT to work
}
common
{
}
No. sphinx does not 'store' Fields. They can't be returned.
Sphinx 'indexes' the fields, but in its internal Inverse-Index format.
... to get data back out of sphinx - use Attributes. In your example index, you've made group_id and date_added as attributes via the sql_attr_*directives.
sql_field_string could be useful, as it makes a column BOTH a Field, AND a string attribute.
I play a lot with Sphinxsearch and I get same result (no title, content).
If you add this rows to the config file, the SELECT * FROM table1; query will return the title and content fields, too:
sql_field_string = title
sql_field_string = content

Sphinx stopped indexing

After doing rotate and re-generating index, Sphinx doesn't include new records from database. It doesn't give any error and also it includes old index data.
I removed data files and tried again but the result was same.
I also applied range query but the result was same.
So, I cannot update my search index now :(
Here I'm giving my configuration, thank you for your tips :)
source search_song
{
type = mysql
sql_host = localhost
sql_user = myusername
sql_pass = mypass
sql_db = mydb
sql_port = 3306 # optional, default is 3306
sql_query_pre = SET NAMES utf8
sql_query_pre = SET NAMES utf8 COLLATE utf8_turkish_ci
sql_query_pre = SET CHARACTER SET utf8
sql_query_pre = SET COLLATION_CONNECTION = utf8_turkish_ci
sql_query_range = SELECT MIN(song_ID), MAX(song_ID) FROM song
sql_range_step = 20000
sql_query = SELECT song.song_ID, artist.artist_ID, song.title, song_stats.total_read, IF(artist.flag_The = 1, CONCAT("The ", artist.name), artist.name) AS fullname \
FROM song \
INNER JOIN artist ON artist.artist_ID = song.artist_ID \
LEFT JOIN song_stats ON song_stats.song_ID = song.song_ID \
WHERE song.song_ID >= $start AND song.song_ID <= $end;
sql_attr_uint = total_read
}
index search_song
{
source = search_song
path = /var/lib/sphinxsearch/data/search_song
morphology = metaphone
min_word_len = 1
min_prefix_len = 2
enable_star = 1
charset_type = utf-8
# exceptions = /var/lib/sphinxsearch/exceptions.txt
charset_table = A->a, B->b, C->c, U+C7->c, U+E7->c, D..G->d..g, U+11E->g, U+11F->g, H->h, I->i, U+131->i, U+130->i, J..O->j..o, U+D6->o, U+F6->o, P..S->p..s, U+15E->s, U+15F->s, T..U->t..u, U+DC->u, U+FC->u, V..Z->v..z, _, a..z,[,],0..9
}
Does it work without the range ? just the sql_query , no range and step ?
I suspect the SQL query might be limiting results.

Error after converting Sphinx original indexes to real-time indexes

I used this tutorial to convert my original sphinx indexes to real-time indexes: http://www.ivinco.com/blog/converting-sphinx-original-indexes-to-real-time-indexes/
I changed my sphinx.conf:
source movies_dev
{
type = mysql
sql_host = localhost
sql_user = ********
sql_pass = ********
sql_db = ********
sql_sock = /var/run/mysqld/mysqld.sock
sql_port = 3306
sql_query = \
SELECT \
CRC32(movie_id) AS id, movie_id, format_id, active, year, title \
FROM \
movie;
sql_attr_uint = format_id
sql_attr_uint = active
sql_attr_uint = year
sql_field_string = movie_id
sql_field_string = title
sql_query_info = SELECT * FROM movie WHERE CRC32(movie_id)=$id
sql_query_pre = SET NAMES utf8
}
index movies_dev
{
source = movies_dev
path = /var/data/sphinx/movies_dev
morphology = stem_en
enable_star = 1
min_word_len = 3
min_prefix_len = 0
min_infix_len = 3
charset_type = utf-8
charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+DC->U+FC, U+C4->U+E4, U+D6->U+F6, U+DF, U+E4, U+F6, U+FC
}
index rt_movies_dev
{
type = rt
rt_mem_limit = 32M
path = /var/data/sphinx/rt_movies_dev
charset_type = utf-8
rt_field = movie_id
rt_field = title
rt_attr_uint = format_id
rt_attr_uint = year
rt_attr_uint = active
}
source attach_movies_dev
{
type = mysql
sql_host = localhost
sql_user = ********
sql_pass = ********
sql_db = ********
sql_query = SELECT 1 FROM rt_movies_dev
sql_query_post = ATTACH INDEX movies_dev TO RTINDEX rt_movies_dev
}
index attach_movies_dev
{
source = attach_movies_dev
path = /var/data/sphinx/attach_movies_dev
docinfo = extern
charset_type = utf-8
}
I created the "rt_movies_dev" table:
SET NAMES utf8;
SET foreign_key_checks = 0;
SET time_zone = '+01:00';
SET sql_mode = 'NO_AUTO_VALUE_ON_ZERO';
DROP TABLE IF EXISTS `rt_movies_dev`;
CREATE TABLE `rt_movies_dev` (
`movie_id` varchar(20) NOT NULL,
`format_id` int(10) NOT NULL,
`title` varchar(255) NOT NULL,
`year` int(20) DEFAULT NULL,
`active` tinyint(1) NOT NULL,
PRIMARY KEY (`movie_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
After that, I run these three commands:
root#server:~# /usr/local/sphinx/bin/searchd --config /usr/local/sphinx/etc/sphinx.conf;
root#server:~# /usr/local/sphinx/bin/indexer --config /usr/local/sphinx/etc/sphinx.conf movies_dev --rotate;
root#server:~# /usr/local/sphinx/bin/indexer --config /usr/local/sphinx/etc/sphinx.conf attach_movies_dev;
No errors after the first two commands (except the warnings like in the tutorial).
But the last command throws this:
ERROR: index 'attach_movies_dev': No fields in schema - will not index.
I do not know exactly what the error says and I could find nothing useful. Can you say what's wrong? I'm stuck here.
Firstly attach_movies_dev source, connects to SPHINX, not to mysql. So no mysql table is required.
You are just using indexer to invoke SphinxQL commands.
But from what I can see trying to index the attach index, will always result in an error, because the RT index itself must be empty (so can attach a disk index to it!)
So change your attach index to connect to searchd instead. And it should work better. Probably an empty RT index, is ok, indexer will just create an empty index, but importantly it will still run the _post command. Which is the whole reason the index exists!
Also beware that your disk index and RT index have different fields, in your disk index, you have two sql_field_string, which create both attributes AND fields. So your RT index, should to contain two string attributes to match (rather than just fields).

SphinxQL infix search

I am trying to use Sphinx RT index for my website. Everything works except infix searching.
index rt
{
type = rt
path = /var/lib/sphinxsearch/data/rt.sph
rt_field = name
rt_field = address
rt_field = keyword
rt_attr_uint = type
min_word_len = 1
min_infix_len = 3
enable_star = 0
html_strip = 0
inplace_enable = 0
charset_type = utf-8
}
I inserted some values to my rt index.
Example:
Insert into rt(id,name,address,keyword,type) values(100,'JohnRambo','Newyork','assassin',1);
Now, when I search for 'JohnRambo', it returns me the correct result. But when I search for 'John' or 'Rambo', it gives me an empty result set.
mysql> select * from rt where match('John');
Empty set (0.00 sec)
Is there anything that I am missing? Any help is much appreciated!
Infix/prefix searching requires dict=keywords on a RT index.

Sphinx sort problem?

i am using 1.10 sphinx.
source tags
{
type = mysql
sql_host = localhost
sql_user = abc
sql_pass = 123456
sql_db = company
sql_sock = /var/lib/mysql/mysql.sock
sql_port = 3306
sql_query_pre = SET NAMES utf8
sql_query_range = SELECT 1,max(id) FROM companies
sql_query = \
SELECT id,name FROM companies;
;
sql_attr_str2ordinal = name
}
this is sorted by name but i am retriving data.
the result set is [name] => 954 ...... like this coming .
sql_attr_str2ordinal = name before use this i am using sql_field_string = name that name is retrive but not sort by name.
what is the solution for this?
sql_attr_str2ordinal :
Does not stores the value only stores the sort order (ordinal) after indexing. so you can not the value from it.
sql_field_string :
full-text field but lacks sorting (since its not an attribute).
sql_attr_string :
lacks full-text index.
So what I would do is :
source tags
{
type = mysql
sql_host = localhost
sql_user = abc
sql_pass = 123456
sql_db = company
sql_sock = /var/lib/mysql/mysql.sock
sql_port = 3306
sql_query_pre = SET NAMES utf8
sql_query_range = SELECT 1,max(id) FROM companies
sql_query = \
SELECT id,name as name_ordinal, name as name FROM companies;
sql_attr_str2ordinal = name_ordinal
sql_field_string = name
}
Then sort by name_ordinal and query with name