I'm getting the following error while trying a wildcard(*) enabled search in Sphinx 2.0.6
index products: syntax error, unexpected $undefined near '*'
My search term is iphone 4s*
It's using the products index as defined below.
index users
{
enable_star = 1
docinfo = extern
morphology = stem_en
charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F
ignore_chars = U+0021..U+002F,U+003A..U+003F,U+0060
charset_type = utf-8
html_strip = 0
source = gdgt_user
path = /var/lib/sphinxsearch/data/gdgt/users
min_infix_len = 3
min_word_len = 3
}
index products : users
{
enable_star = 1
min_infix_len = 1
min_word_len = 1
source = gdgt_products
path = /var/lib/sphinxsearch/data/gdgt/products
}
I am using the php api that can be found in the source tar ball.
I am able to see the error when using search CLI.
search -c app/config/sphinx.compiled.conf -i products -e "ipho*"
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file 'app/config/sphinx.compiled.conf'...
index 'products': search error: .
My php code looks like
$client = new SphinxClient();
$client->SetServer($serverIp, $serverPort);
$client->SetMaxQueryTime(5000);
$client->SetSortMode(SPH_SORT_RELEVANCE);
$client->SetMatchMode(SPH_MATCH_EXTENDED);
$res = $client->query('ipho*', 'products');
var_dump($res, $client->getLastError(), $client->getLastWarning());
The issue is that star(*) for wildcard is also in your ignore_chars (U+002A).
Update it to:
ignore_chars = U+0021..U+0029,U+002B..U+002F,U+003A..U+003F,U+0060
Related
I using sphinx search to create indexes and search data in my PostgreSQL database.
I have 2 questions about it.
If I run command
/usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate --all
I get output from 'show tables;'
Index
Type
dist_title_de
distributed
word_title_de
local
word_titlestemmed_de
local
rt_title_de
rt
But If I run command
/usr/bin/indexer --config /etc/sphinxsearch/sphinx_another_conf_file.conf --rotate --all
Then I get the same output on terminal, but I dont see new indexes on 'show tables;'. It seems like '--config' option in indexer not working and only properly name is sphinx.conf. It's problematic, because if I want reindex sphinxsearch I have to changing file sphinx.conf.
Second question is it possible to 'add' new index without delete old ones? Currently I using sphinx like (everyday):
Get new data (datasource1, datasource2, ..., datasource8)
Index --rotate --all (index data from 8 datasources)
Search some info on indexes
Write it to db
But now, I want sth like:
Get new data from datasource1
Index datasource1
Get new data from datasource2
Index datasource2 (without delete index datasource1)
Search something in index datasource1
....
Get new data form datasource8 (without deleteing indexes)
Index datasource8
etc
On 'without delete index' I mean, now if I use command from top of topic, I 'lost' my indexes and get only new ones (from sphinx.conf).
My sphinx.conf (only 1 datasource):
source src_title_de
{
type = pgsql
sql_host = #######
sql_user = #######
sql_pass = #######
sql_db = #######
sql_port = 3306 # optional, default is 3306
sql_query = \
SELECT id, group_id, (date_extraction::TIMESTAMP) AS date_extraction, title \
FROM sphinx_test
sql_ranged_throttle = 0
}
index word_title_de
{
source = src_title_de
path = /var/lib/sphinxsearch/data/word_title_de
docinfo = extern
dict = keywords
mlock = 0
morphology = none
stopwords = /var/lib/sphinxsearch/data/stopwords.txt
wordforms = /var/lib/sphinxsearch/data/wordforms_de.txt
min_word_len = 1
}
index word_titlestemmed_de : word_title_de
{
path = /var/lib/sphinxsearch/data/word_titlestemmed_de
morphology = stem_en
}
index dist_title_de
{
type = distributed
local = word_title_de
local = word_titlestemmed_de
agent = localhost:9313:remote1
agent = localhost:9314:remote2,remote3
agent_connect_timeout = 1000
agent_query_timeout = 3000
}
index rt_title_de
{
type = rt
path = /var/lib/sphinxsearch/data/rt_title_de
rt_field = title
rt_field = content
rt_attr_uint = gid
}
indexer
{
mem_limit = 128M
}
searchd
{
listen = 9312:sphinx
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
query_log = /var/log/sphinxsearch/query.log
read_timeout = 5
client_timeout = 300
max_children = 30
persistent_connections_limit = 30
pid_file = /var/run/sphinxsearch/searchd.pid
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
mva_updates_pool = 1M
max_packet_size = 8M
max_filters = 256
max_filter_values = 4096
max_batch_queries = 32
workers = threads # for RT to work
}
My second file for 8 datasources like the same like above with CTRL+C CTRL+V on 'source src_title_de', 'index word_title_de', 'index word_titlestemmed_de', 'index rt_title_de' with another countries and change table with data in 'sql_query'.
On your first question, the --config option only applies to that indexer run. Ie the --all should cause it index (or try to ) index all the plain indexes mentioned in that file.
... but when it sends the signal to reload (what the --rotate) does, searchd just reloads its CURRENT config file, NOT the one you told indexer about.
To get serachd to use a new config file would have to stop searchd, and start it again with new config file.
So change sphinx.conf directly, rather than a 'second' file.
Acully the second question is the same answer...
So change sphinx.conf directly, rather than a 'second' file.
Ie add your new index to sphinx.conf, use indexer to 'build' it. When indexer has finished, it will tell 'reload' whcih will cause searchd to load the new config file AND the new index just built.
My system is
% uname -or
FreeBSD 11.0-RELEASE-p2
Sphinx version is
% searchd --help
Sphinx 2.2.11-id64-release (95ae9a6)
Sphinx configuration:
index content_rt_template : common_template
{
type = rt
rt_mem_limit = 128M # 128M only...
rt_field = text
rt_attr_string = text
rt_field = title
rt_attr_string = title
rt_field = url
rt_attr_string = url
rt_attr_bigint = item_id
rt_attr_uint = source_id
rt_attr_timestamp = published_date
rt_attr_timestamp = created_date
}
common {
lemmatizer_base = /path/to/sphinx/
}
indexer
{
mem_limit = 128M # 128M only...
}
index content_rt_from_20170501_to_20170601 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20170501_to_20170601
}
index content_rt_from_20170601_to_20170701 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20170601_to_20170701
}
index content_rt_from_20171201_to_20180101 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20171201_to_20180101
}
index content2017
{
type = distributed
local = content_rt_from_20170501_to_20170601
local = content_rt_from_20170601_to_20170701
local = content_rt_from_20171201_to_20180101
}
searchd
{
listen = 127.0.0.1:9417
listen = 9317:mysql41
log = /path/to/sphinx/log/searchd_2017.log
query_log = /path/to/sphinx/log/query_2017.log
read_timeout = 60
max_children = 30
pid_file = /path/to/sphinx/pid/searchd2017.pid
seamless_rotate = 0
preopen_indexes = 0
unlink_old = 1
workers = threads # for RT to work
binlog_path = /path/to/sphinx/data/2017/
}
Used memory before starting Sphinx:
Mem[||||||||| 5.33G/40.0G]
Swp[||||||||||||||||||||||||||||||3.35G/4.00G]
Log on Sphinx start:
% ./start.sh
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/path/to/sphinx/conf/content2017.conf'...
listening on 127.0.0.1:9417
listening on all interfaces, port=9317
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170501_to_20170601'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170601_to_20170701'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20171201_to_20180101'
precached 3 indexes in 6.520 sec
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/path/to/sphinx/conf/content_dist.conf'...
listening on 127.0.0.1:9312
listening on all interfaces, port=9306
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_snippet'
precached 1 indexes in 0.064 sec
Used memory after Sphinx started:
Mem[||||||||||||||| 11.6G/40.0G]
Swp[||||||||||||||||||||||||||||||3.34G/4.00G]
Sphinx use about 6G. But according to mem_limit and rt_mem_limit must use not more than 128 * 3 = 384M.
What may be reason of using lot of memory?
Maybe reason in this warning?
WARNING: failed to init process shared rwlock: process shared rwlock
UPD
I has tryed it on Ubuntu 16.04 and situation is same.
The rt_mem_limit only limits the size of the ram chunk itself. Any disk chunks will use their own memory.
Typically its attributes that compose the biggest part , as by default, all are held in memory. Can cut down memory with http://sphinxsearch.com/docs/current.html#conf-ondisk-attrs
With option
ondisk_attrs = pool
RT index use not many RAM.
I get a conflict in EXT:realUrl because of a page with the same name in both domains.
domain.com/contact.html
mobile.domain.com/contact.html
I got this setup:
The realurl configuration in PHP both with their own rootpage_id:
['EXTCONF']['realurl']['_DEFAULT']
['EXTCONF']['realurl']['mobile.domain.com']
TypoScript:
config.baseURL = http://mobile.domain.com/
config.tx_realurl_enable = 1
config.typolinkCheckRootline = 1
config.typolinkEnableLinksAcrossDomains = 1
config.typolinkLinkAccessRestrictedPages = NONE
config.prefixLocalAnchors = all
config.content_from_pid_allowOutsideDomain = 1
Is there something missing in my configuration?
What could I do to solve the conflict?
When you have following steps
Step 1
$TYPO3_CONF_VARS['EXTCONF']['realurl']['mobile.example.com'] =
$TYPO3_CONF_VARS['EXTCONF']['realurl']['_DEFAULT'];
$TYPO3_CONF_VARS['EXTCONF']['realurl']['mobile.example.com']['pagePath']['rootpage_id'] = 999;
Step 2
Add Domain Entries on the rootpages
Step 3
Add setup condition
[globalString = IENV:HTTP_HOST=www.example.com]
config.baseURL = http://www.example.com/
[end]
Step 4
Clear all caches (realUrl caches inkl.)
Greetings
I am trying to get the date datatype from SQLserver 2014 as native python date through pymssql. Based on pymssql doc, I need to have 0.95+ freetds lib and 7.3 freetds ver.
# tsql -C
Compile-time settings (established with the "configure" script)
Version: freetds v0.95.8
freetds.conf directory: /etc
MS db-lib source compatibility: no
Sybase binary compatibility: no
Thread safety: yes
iconv library: yes
TDS version: 4.2
iODBC: no
unixodbc: yes
SSPI "trusted" logins: no
Kerberos: no
OpenSSL: no
GnuTLS: no
And I made following config:
#/etc/freetds.conf
[sqlsvr1]
host = sqlsvr1
port = 1433
tds version = 7.3
And I still get python string instead of date. This is from the TDSDUMPCONFIG. Is it because freetds 0.95.8 does not support tds 7.3? (The major/minor version seem to have been replace to 7/1).
config.c:620:IP addr is 172.16.12.26.
config.c:580: port = '1433'
config.c:580: tds version = '7.3'
config.c:886:Setting tds version to 7.3 (0x703).
config.c:568: Reached EOF
config.c:300:Success: [sqlsvr1] defined in /etc/freetds.conf.
config.c:765:Setting 'dump_file' to '/tmp/freetds.log' from $TDSDUMP.
config.c:689:tds_config_login: client_charset is UTF-8.
config.c:213:Final connection parameters:
config.c:214: server_name = sqlsvr1:1433
config.c:215: server_host_name = sqlsvr1
config.c:218: ip_addr = 172.16.12.26
config.c:218: ip_addr = 172.16.12.26
config.c:218: ip_addr = 172.16.12.26
config.c:223: instance_name =
config.c:224: port = 1433
config.c:225: major_version = 7
config.c:226: minor_version = 1
config.c:227: block_size = 0
config.c:228: language = us_english
config.c:229: server_charset =
config.c:230: connect_timeout = 0
config.c:231: client_host_name = rh1.int
config.c:232: client_charset = UTF-8
config.c:233: use_utf16 = 0
config.c:234: app_name = pymssql
config.c:235: user_name = USER
config.c:238: library = DB-Library
config.c:239: bulk_copy = 0
config.c:240: suppress_language = 0
config.c:241: encrypt level = 0
config.c:242: query_timeout = 0
config.c:245: database =
config.c:246: dump_file = /tmp/freetds.log
config.c:247: debug_flags = 0
config.c:248: text_size = 64512
config.c:249: emul_little_endian = 0
config.c:250: server_realm_name =
config.c:251: server_spn =
config.c:252: cafile =
config.c:253: crlfile =
If I set the tds version to 7.4 in the config, then I notice this error in the log:
config.c:580: tds version = '7.4'
config.c:881:error: no such version: 7.4
config.c:568: Reached EOF
config.c:213:Final connection parameters:
... ...
config.c:225: major_version = 7
config.c:226: minor_version = 1
Env: Linux/RH 6 + Python3.4.3 + pymssql(2.1.1) all x64
Any suggestions?
You have most of the required pieces.
But the one missing is the fact you are using pymssql 2.1.1.
Support for the DATE and TIME data types is implemented but targeted for pymssql 2.2.0 which isn't released yet. If you need it now you'll need to build it yourself from the 'master' Git branch first.
I am trying to run the indexer of my sphinx server.
This is the command I use (through root access) to start the indexing:
indexer --all
When I use the command, this is the reponse I get:
Sphinx 2.1.9-id64-release (rel21-r4761)
Copyright (c) 2001-2014, Andrew Aksyonoff
Copyright (c) 2008-2014, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/etc/sphinxsearch/sphinx.conf'...
FATAL: no indexes found in config file '/etc/sphinxsearch/sphinx.conf'
This is the sphinx.conf file that is located in /etc/sphinxsearch/
#############################################################################
## indexer settings
#############################################################################
indexer
{
# memory limit, in bytes, kiloytes (16384K) or megabytes (256M)
# optional, default is 32M, max is 2047M, recommended is 256M to 1024M
mem_limit = 1024M
}
#############################################################################
## searchd settings
#############################################################################
searchd
{
listen = 127.0.0.1:9312
listen = 127.0.0.1:9306:mysql41
log = /var/log/sphinxsearch/searchd.log
query_log = /var/log/sphinxsearch/query.log
read_timeout = 5
client_timeout = 300
max_children = 30
pid_file = /var/log/sphinxsearch/searchd.pid
max_matches = 1000
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
mva_updates_pool = 1M
max_packet_size = 8M
max_filters = 256
max_filter_values = 4096
workers = threads # for RT to work
}
index myindex
{
type = rt
path = /var/www/vhosts/user/sphinx/myindex
rt_field = description
rt_field = searchcode
rt_field = weight
rt_field = productid
rt_attr_uint = stockproduct
rt_attr_uint = instock
charset_type = utf-8
min_infix_len = 3
enable_star = 1
expand_keywords = 1
dict = keywords
}
# --eof--
Can someone help me with resolving this error?
FATAL: no indexes found in config file '/etc/sphinxsearch/sphinx.conf
the indexer command only works on traditional disk-index. Not real time indexes.
Because indexer doesn't do anything with type=rt it doesnt 'see' them, hence your config file has no indexes to index.
I guess in an ideal world it would say 'no plain indexes found' or similar to clarify its ignoring rt (same as it ignores distributed)
All you have to do is put sphinx.conf file inside bin folder, which mean it will be inside this path for example "etc/sphinxsearch/bin/sphinx.conf".