I'm trying to use sphinx 3.1.1 with plain index for autocomplete function. The problem is that I get out of memory error each time I run indexer despite there is mem_limit option in my config:
indexer
{
mem_limit = 2048M
}
If it makes sense I'm using tsvpipe as data source:
source autocomplete
{
type = tsvpipe
tsvpipe_command = cat /var/lib/sphinx/tmp/*.tsv
tsvpipe_field = value
}
The error message:
FATAL: out of memory (unable to allocate 10737418248 bytes)
Related
Context - I am trying out Postgres' Geographic Information System extension PostGis that enables stories latitude and longitudes as Point and operations on it.
If I understand correctly then I need to add a custom converter that can convert the point between JOOQ and PostGis and add it to the gradle file.
Problem - When I generate the jooq-code, few files are generated incorrectly and have the fields defined twice which fail compilation. These are:
<configured-generation-dir>/tables/StValuecount.java
<configured-generation-dir>/tables/records/StValuecountRecord.java
<configured-generation-dir>/tables/records/StValuepercentRecord.java
<configured-generation-dir>/tables/_StValuecount.java
<configured-generation-dir>/tables/records/_StValuecountRecord.java
<configured-generation-dir>/tables/_StHistogram.java
<configured-generation-dir>/tables/records/_StHistogramRecord.java
<configured-generation-dir>/tables/_StQuantile.java
Gradle config =>
jooq{
myAwesomeApp(sourceSets.main){
logging = 'WARN'
jdbc {
driver = 'org.postgresql.Driver'
url = db_url
user = db_user
password = db_password
}
generator {
name = 'org.jooq.codegen.DefaultGenerator'
strategy {
name = 'org.jooq.codegen.DefaultGeneratorStrategy'
}
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
inputSchema = 'public'
forcedTypes {
forcedType {
userType = 'org.postgis.Point'
converter = 'com.example.JooqBreaksWithPostGis.jooq.converters.PostgresPointJooqConverter'
expression = '.*\\.point'
types = '.*'
}
}
}
generate {
routines = false
relations = true
deprecated = false
records = true
immutablePojos = false
fluentSetters = true
}
target {
packageName = 'jooq.fancy.app'
directory = 'src/main/java/generated'
}
}
}
}
What am I doing wrong?
I have also created a minimal project where I have reproduced the problem in case someone wants to quickly try it.
Steps to reproduce
Checkout project
git clone git#github.com:raj-saxena/JooqBreaksWithPostGis.git
Go to the project directory and start postgis docker container with
docker-compose up
Similarly, to remove postgis docker container run
docker-compose down
Run migrations that add a simple City table containing Point type with
./gradlew flywayMigrate
I have added few rows in a second migration to verify if the DB structure was working. Details to connect to Postgres instance in the build.gradle file.
Generate jooq files with
./gradlew generateMyAwesomeAppJooqSchemaSource
Verify that the files are generated in the configured src/main/java/generated directory.
Verify that the files mentioned above fail to compile.
Taking Lukas' advice, I added the exclude configuration to the jooq config as below:
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
...
excludes = '.*ST_ValueCount' +
'|.*St_Valuepercent' +
'|.*St_Histogram' +
'|.*St_Quantile' +
'|.*St_Approxhistogram' +
'|.*St_PixelOfValue' +
'|.*St_Approxquantile' +
'|.*ST_Tile'
}
This allowed the code to compile.
This sounds a lot like https://github.com/jOOQ/jOOQ/issues/4055. jOOQ 3.11 currently cannot handle overloaded table valued functions in any RDBMS that supports table valued functions. Your best option here is to exclude all the affected functions from the code generation, using <excludes>:
https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-includes-excludes/
I am using this fork of LMDBjni:https://github.com/deephacks/lmdbjni to
form the backend of a medium-sized databases project in scala.
I've been hitting a EXCEPTION_ACCESS_VIOLATION (0xc0000005) in the JNI code for this LMDB, and would like to know whether there is anything obvious I'm doing wrong or what the next steps for debugging should be. I'm not exactly sure what I'm looking for, so I'm going to list as much information about what's happening as I can and hope the symptoms make sense to somebody.
The access violation occurs in a Database with a single key mapping to a single 8-byte value, on approximately the 4000th access to that database (the exact number appears to be the same with each run), suggesting that this is a deterministic problem.
I believe I only have one thread accessing the database at a time, and regardless, in my understanding, as the operation is wrapped in a transaction, concurrent accesses should not matter anyway.
By looking through stack traces and printing values the issue comes from this generic construction I wrote for building transactions.
My code that causes the issue is here, the crash occurs in the marked db.get() call:
def transactionalGetAndSet[A](
key: Key,
db: Database
)(
compute: A => LMDBEither[A]
)(
implicit sa: Storeable[A],
env: Env
): LMDBEither[A] = {
import org.fusesource.lmdbjni.Transaction
// get a new transaction
val tx: Transaction = instance.env.createWriteTransaction()
println("tx = " + tx + " id = " + tx.getId)
// get the key as an Array[Byte]. This is done by converting the key to a base64 string then converting that to bytes (so arbitary objects can be made into keys)
val k = key.render
println("Key = " + key + " Rendered = " + new String(k))
// instantiate a result value, so there is something if it fails
var res: LMDBEither[A] = NoResult.left // initialise the result as a failure to begin with
try {
res = for { // This for construction chains together operations that return LMDBEithers into one LMDBEither
bytes <- LMDBEither(db.get(tx, k)) // error occurs in this Database.get() call
_ = println("bytes = " + bytes)
a <- sa.fromBytes(safeRetrieve(bytes)) // sa is effectively an unmarshaller/unmarshaller object which converts Vector[Byte] => LMDBEither[A]
_ = println("a = " + a)
res <- compute(a) // get the next value for the value at the key
_ = println("res = " + res)
_ <- LMDBEither(db.put(tx, k, sa.toBytes(res).toArray))
} yield a // effectively, if all these steps worked, res == Right(a)
res // return the result
} finally {
// Make sure you either commit or rollback to avoid resource leaks.
if (res.isRight) tx.commit() // if the result is not an error (ie Either.isRight is true)
else tx.abort()
tx.close()
}
}
Where LMDBEither[A] is an alias for Either[E, A] for a specific error type E, and LMDBEither(x) is a function that lifts an expression that might throw exceptions during execution into an LMDBEither, catching any exceptions.
the function safeRetrieve converts a possibly null Array[Byte] into a definitely not null Vector[Byte], as follows:
private def safeRetrieve(bytes: Array[Byte]): Vector[Byte] =
Option(bytes).fold(Vector[Byte]()){ // if the array is null, convert to an empty vector, otherwise call the array's wrapper's vector
arr =>
println("Vector = " + arr.toVector)
arr.toVector
}
To the best of my knowledge, this does not modify the memory where the array is stored (LMDB's protected memory)
The values printed up to and including the crash are as follows:
tx = org.fusesource.lmdbjni.Transaction#391cec1f id = 15104
Key = Vector(Objects) Rendered = 84507411390877848991196161
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000018002453f, pid=10220, tid=7268
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [lmdbjni-64-0-7710432736670562378.4+0x2453f]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\dev\PartIIProject\hs_err_pid10220.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Which is more evidence that the mdb_get() fails.
the full contents of the file referenced are here: https://pastebin.com/v6AmFBjq
Again, I would be extremely grateful for the small chance that anyone could point me in the right direction. What next steps should I be taking?
HOW does one set the language on org.moxieapps.gwt.highcharts.client.Chart
setting the Highcharts.options (in Java) as:
Highcharts.Options options = new Highcharts.Options();
options.setGlobal(new Global().setUseUTC(false));
String[] cat = new String[] {"Jan.", "Fev.", "Março", "Abril", "Maio", "Junho", "Julho", "Agosto", "Set.", "Out.", "Nov.", "Dez."} ;
options.setLang(new Lang().setMonths(cat));
Highcharts.setOptions(options);
has NO effect. Remains in English.
trying to set:
chart.getXAxis()
.setCategories("Jan.", "Fev.", "Março", "Abril", "Maio", "Junho", "Julho", "Agosto", "Set.", "Out.", "Nov.", "Dez.");
fails:
--> Chromium fails:
V8 error: Allocation failed - process out of memory (invalid array length). Current memory usage: 890 MB
--> Firefox complains of a script error
Fails even using documented example of:
chart.getXAxis().setCategories(....)
HELP please!
Lol!
I needed to
options.setLang(new Lang().setShortMonths(cat));
!!!
So here is a snippet from the config file I'd like to parse (It is an LVM2 Config):
VolGroup00 {
id = "vyllep-rfI6-LCvO-h6mN-zYZu-hiAN-QShmG6"
seqno = 3
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 65536 # 32 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0
physical_volumes {
pv0 {
id = "1yLiSl-x0fp-ZkyU-HMQl-eTVt-xiId-cFnih0"
device = "/dev/xvda2" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 31246425 # 14.8995 Gigabytes
pe_start = 384
pe_count = 476 # 14.875 Gigabytes
}
}
}
I would like to parse this into a Perl data structure. What format is this config in? My guess is it looks likes a python data structure.
Any thoughts the format, or better yet, an existent module to parse it with?
The config uses a custom config language specifically for LVM. The lvm userspace tools include code to parse this language.
You could grab the userspace code for lvm2 and attempt to replicate its parser, maybe using Parse::RecDescent.
Or maybe the Perl Linux::LVM module in CPAN provides the functionality to extract the information you need.
This is what I am trying to do:
Read an image from the oracle BLOB
Resize it
Write the resized image back to the table(update the blob itself).
My table looks like this:
TECHID NOT NULL NUMBER(12)
MEDIADATA NOT NULL BLOB()
INSERTEDDATE NOT NULL DATE
MODIFIEDDATE NOT NULL DATE
Steps 1 and 2 work perfectly. The code for step 3 is as follows (this is a PoC type spike solution - not final production):
File resized = get the resized image
FileInputStream fis = new FileInputStream(resized)
PreparedStatement p = db.connection.prepareStatement("update mymediadata set mediadata = ? where TECHID=142")
if (fis != null)
{
println("Available: ${fis.available()}"); // this works - shows 117K bytes available.
}
p.setBinaryStream (1, fis, resized?.length()?.intValue())
try
{
p.executeUpdate()
}
catch (Exception e)
{
e.printStackTrace()
}
finally
{
p.close()
fis.close()
}
When I get to step 3, I get the following error:
SQLException: ORA-01407: cannot update ("OWNER"."MEDIADATA"."MEDIADATA") to NULL
I explicitly checked (in debugger) that the FileInputStream (fis) is not null. I also checked that the resized?.length()?.intValue() value is also > 0. So I am struggling to see what I might be doing wrong.
Technology stack:
Groovy GDK 1.7
Java 1.5
Oracle 10g
Running on 32 bit Windows XP.
OK, I finally managed to resolve this. It turns out it was a dodgy JDBC oracle driver (apparently an earlier version) which was causing the errors.
As soon as I switched to the proper version, the error vanished!