Rails5, i have it installed on database
pg_trgm | 1.1 | public | text similarity measurement and index searching based on trigrams)
and in the initializer :
PgSearch.multisearch_options = {
:using => [:tsearch, :trigram],
}
i've tried it with only :trigram (not :tsearch), doesn't work, even after db:reset and rake pg_search:multisearch:rebuild[AllModels].
Am I missing a step?
I'm the author of pg_search.
Trigram search requires the pg_trgm extension.
See the pg_search README and the wiki page for Installing PostgreSQL Extensions for more details.
Related
After parsing many documents, I have a lot of rows/columns with Ukrainian text that should be indexed for full-text search in Postgres.
I've found that Postgres 14 supports by default 29 languages, but unfortunately not the Ukrainian one.
After subsequent digging, I've found that it allows adding an external dictionary:
CREATE TEXT SEARCH DICTIONARY my_lang_ispell (
TEMPLATE = ispell,
DictFile = path_to_my_lang_dict_file,
AffFile = path_to_my_lang_affixes_file,
StopWords = path_to_my_lang_astop_words_file
);
But how to find the most relevant DictFile, AffFile, and StopWords files? For example, snowball source doesn't contain this language.
So, could anyone help me find the best way to obtain ispell, aspell, snowball, or another dictionary for the Ukrainian language?
Thanks!
After more deep exploring, found the solution on this resource dict_uk
Compile files manually by this guide:
sudo snap install gradle
$ cd dict_uk
$ ./gradlew expand
$ cd distr/hunspell/
$ ../../gradlew hunspell
$ sudo cp build/hunspell/uk_UA.aff /usr/share/postgresql/12/tsearch_data/uk_ua.affix
$ sudo cp build/hunspell/uk_UA.dic /usr/share/postgresql/12/tsearch_data/uk_ua.dict
$ sudo cp ../postgresql/ukrainian.stop /usr/share/postgresql/12/tsearch_data/ukrainian.stop
Or just download all files from here
Follow the guide of setting up ukrainian language in the Postgres:
$ sudo cp uk_UA.affix /usr/share/postgresql/12/tsearch_data/uk_ua.affix
$ sudo cp uk_UA.dic /usr/share/postgresql/12/tsearch_data/uk_ua.dict
$ sudo cp ukrainian.stop /usr/share/postgresql/12/tsearch_data/ukrainian.stop
$ sudo su postgres
$ psql
CREATE TEXT SEARCH DICTIONARY ukrainian_huns (TEMPLATE = ispell, DictFile = uk_ua, AffFile = uk_ua, StopWords = ukrainian);
CREATE TEXT SEARCH DICTIONARY ukrainian_stem (template = simple, stopwords = ukrainian);
CREATE TEXT SEARCH CONFIGURATION ukrainian (PARSER=default);
ALTER TEXT SEARCH CONFIGURATION ukrainian ALTER MAPPING FOR hword, hword_part, word WITH ukrainian_huns, ukrainian_stem;
ALTER TEXT SEARCH CONFIGURATION ukrainian ALTER MAPPING FOR int, uint, numhword, numword, hword_numpart, email, float, file, url, url_path, version, host, sfloat WITH simple;
ALTER TEXT SEARCH CONFIGURATION ukrainian ALTER MAPPING FOR asciihword, asciiword, hword_asciipart WITH english_stem;
# \dFd
...
pg_catalog | english_stem | snowball stemmer for english language
...
public | ukrainian_huns |
public | ukrainian_stem |
Now it is available for creating a searchable column with help of to_tsvector:
ALTER TABLE extracted_pages
ADD COLUMN tsvector_uk tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('ukrainian', coalesce(column_with_text, '')), 'A')
) STORED;
This example shows the correct stemming for the Ukrainian language:
SELECT to_tsvector('ukrainian', 'солодко дзюрчить джерело і хочеться жити, любити, творити... ');
=> [{"to_tsvector"=>"'джерело':3 'дзюрчати':2 'жити':6 'любити':7 'солодко':1 'творити':8 'хочеться':5"}]
Results
The Postgres full-text search works as well as similar search text engine SphinxSearch in terms of quality, but it is a bit slower.
On the same query from the huge amount of records (278_000) it returns the same results:
Postgres - ActiveRecord: 67.6ms
SphinxSearch - ActiveRecord: 10.9ms
OS: Ubuntu 20.04
Thank you very much, dict_uk support team!
I'm trying to convert geometries to images, and the functions to do so don't seem to exist.
The following example is from the ST_AsRaster Docs WHich specify the requirements are Availability: 2.0.0 - requires GDAL >= 1.6.0.
SELECT ST_AsPNG(ST_AsRaster(ST_Buffer(ST_Point(1,5),10),150, 150));
This results in:
ERROR: function st_asraster(geometry, integer, integer) does not exist
LINE 1: SELECT ST_AsPNG(ST_AsRaster(ST_Buffer(ST_Point(1,5),10),150,...
I found some info that points towards needing GDAL drivers, however, when I try:
SELECT short_name, long_name FROM ST_GdalDrivers();
I get:
ERROR: function st_gdaldrivers() does not exist
LINE 1: SELECT short_name, long_name FROM ST_GdalDrivers();
I have no idea where to even go to try solving this, why don't the functions exist, was there some config I needed to add, some doc I didn't read?
Even the https://postgis.net/docs/RT_reference.html seems to suggest that it should "just work"?
This is installed from the package manager on Ubuntu 20.0.4.
Version Info SELECT PostGIS_Full_Version();:
POSTGIS="3.0.0 r17983" [EXTENSION]
PGSQL="120"
GEOS="3.8.0-CAPI-1.13.1 "
PROJ="6.3.1"
LIBXML="2.9.4"
LIBJSON="0.13.1"
LIBPROTOBUF="1.3.3"
WAGYU="0.4.3 (Internal)"
You must have forgotten to install the postgis_raster extension:
CREATE EXTENSION postgis_raster;
This extension is new in PostGIS 3.0; before that, its objects were part of the postgis extension.
The documentation mentions that:
Once postgis is installed, it needs to be enabled in each individual database you want to use it in.
psql -d yourdatabase -c "CREATE EXTENSION postgis;"
-- if you built with raster support and want to install it --
psql -d yourdatabase -c "CREATE EXTENSION postgis_raster;"
I'm using Timescale and today I faced on a problem:
I`m creating a simple table with any of bellow ways:
1-
> create table if not exists "NewTable" as (select * from "OldTable");
SELECT 6
2-
create table "NewTable" ("eventTime" timestamptz, name varchar);
after the table successfully created. I write \d and the result for both tables are same:
Column | Type | Collation | Nullable | Default
-----------+--------------------------+-----------+----------+---------
eventTime | timestamp with time zone | | |
name | character varying | | |
but the problem starts here...
> SELECT create_hypertable('"NewTable"', '"eventTime"' ,migrate_data => true);
ERROR: tried calling catalog_get when extension isn't loaded
so after I googled my issue I found nothing usefull insted of everyone told to create timescaledb extention that I had it before
> CREATE EXTENSION timescaledb CASCADE; //OR CREATE IF NOT EXISTS EXTENSION timescaledb CASCADE;
ERROR: extension "timescaledb" has already been loaded with another version
and
> \dx
Name | Version | Schema | Description
-------------+---------+------------+--------------------------------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
timescaledb | 1.5.1 | public | Enables scalable inserts and complex queries for time-series data
so what should I do?
Question : why this happened? how should I create a hyper table now?
before these operations, I tried to take a dump from my database and before that, I have 20 main_hyper_tables.
why this happened?
I guess TimescaleDB was installed in the standard way through apt. Since new version of TimescaleDB (v.1.6) was released recently, apt automatically updated installation and copied the binary of 1.6 into the shared library of PostgreSQL installation. Thus new extension version was loaded, which is different from the extension used to create the database (v.1.5.1).
how should I create a hyper table now?
I see two options:
Load the extension version used to create the database, by explicitly specifying in the postgres config.
Update the extension to the latest version by
ALTER EXTENSION timescaledb UPDATE
See using ALTER EXTENSION section in Update TimescaleDB doc
Hi i am new logstash and as well as mongodb. I am not really sure if this is the right place to ask question or not, but i really need help. I have posted the same question in logstash-users google group as well, but i havent received and any answer. Further i have gone through all the posts at the google group but none of the solutions seem to be working.
i need to use mongodb for output so i ran the following command:
./plugin install contrib
After that i ran this command
./logstash agent --config logstash-shipper.conf
my logstash-shipper.conf is
input {
file {
path => "/var/log/alternatives.log"
type => "syslog"
}
}
output {
mongodb {
codec => "plain"
collection => "logstash"
database => "mydb"
generateId => true
isodate => true
uri => "mongodb://localhost:27017/"
} }
i get the error
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.0/plugin-milestones {:level=>:warn}
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
| |
| * chat: #logstash IRC channel on freenode irc. |
| IRC via the web: http://goo.gl/TI4Ro |
| * email: logstash-users#googlegroups.com |
| * bug system: https://logstash.jira.com/ |
| |
+---------------------------------------------------------+
The error reported is:
Couldn't find any output plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb output plugin resulted in this error: no such file to load -- logstash/outputs/mongodb
As far as I can tell, the 1.4.2 distribution is slightly broken. The version name of the release is "1.4.2-modified" but the contrib version is still "1.4.2".
My workaround was to find ./lib/logstash/version.rb and change
LOGSTASH_VERSION = "1.4.2-modified"
to
LOGSTASH_VERSION = "1.4.2"
This allowed
./plugin install contrib
to work (i.e. my mongodb connection works now), and so far nothing else has broken. Interestingly enough, there's a note right in the version.rb file that says
# Note to authors: this should not include dashes because 'gem' barfs if
# you include a dash in the version string.
Oops.
I'm designing a fulltext search function in postgresql for my current project.
It works ok with ispell/myspell dictionaries so far.
Now I need to add support for chinese, japanese and arabic search.
Where do I start?
There are no templates or dictionaries available for those languages
as far as I can see.
Will it work with pg_catalog.simple configuration?
Just a hint from the manual: A large list of dictionaries is available on the OpenOffice Wiki.
Dictionaries won't help you too much with Chinese - you'll need to look in to NGRAM tokenising...
The similar solution of link at stackoverflow.com is How do I implement full text search in Chinese on PostgreSQL? .
Although that, I would provide a solution below in detail based on my experience and a solution on Internet. I use both tools of SCWS and zhparser as the solution of Chinese full-text search in postgres.
20160131 Update:
You must check whether you have installed postgresql-server-devel-{number version} because we will use pgxs function from it for creating extension in postgresql.
Step1: install SCWS.
It's remarkable that --prefix=/usr/local/scws follows ./configure . Not just has ./configure along in below 4th line.
wget http://www.xunsearch.com/scws/down/scws-1.2.2.tar.bz2
tar xvjf scws-1.2.2.tar.bz2
cd scws-1.2.2
./configure --prefix=/usr/local/scws
make
make install
To check whether it installed successfully, please enter below command:
ls -al /usr/local/scws/lib/libscws.la
Step2: Install zhparser
git clone https://github.com/amutu/zhparser.git
cd zhparser
SCWS_HOME=/usr/local/scws/include make && make install
20160131 Update:
If you use Mac OS X Yosemite, aboved value of SCWS_HOME is same. But if you use Ubuntu 14.04 LTS, please change value of SCWS_HOME to /usr/local/scws .
Step3: Configure a new extension using zhparser in Postres
Step3.1: Login your postgres database through terminal/commandline
psql yourdatabasename
Step3.2: Create extension in Postgres. You could specify what dictionary name you want.
CREATE EXTENSION zhparser;
CREATE TEXT SEARCH CONFIGURATION dictionarynameyouwant (PARSER = zhparser);
ALTER TEXT SEARCH CONFIGURATION dictionarynameyouwant ADD MAPPING FOR n,v,a,i,e,l WITH simple;
If you follow above steps, you can use the function of Postgres full-text searching in Chinese/Mandarin words.
Extra step(not necessary) in Rails for using pg_search gem: Step4. Configure the dictionary name at :dictionary attribute of :tsearch in app/models/yourmodel.rb
class YourOwnClass < ActiveRecord::Base
...
include PgSearch
pg_search_scope :functionnameyoulike, :against => [columnsyoulike1, columnsyoulike2, ...,etc], :using => { :tsearch => {:dictionary => "dictionary name you just specified in creating a extension in postgres", blah blah blah, ..., etc} }
end
Reference:
1. SCWS install tutorial
2. Zhparser#github.com
3. Francs' Post - Postgres full-text search in Chinese with zhparser and SCWS
4. Rails365.net's Post - Postgres full-text search in Chinese with pg_search gem with zhparser
5. My Post at xuite.net - Make Postgres support full text search in Mandarin/Chinese