Grouping fields within a Lucene.Net document - lucene.net

Question

Within our application, we've been working with Lucene.Net to index large numbers of data. The fields themselves are configurable, so the name and type of the fields can change with each rebuild. Within each document, we can have multiple fields having the same name and a various number of Numeric and text fields. Since we've put a lot of work in the current development, changing to a different search engine is a no go a.t.m.
The fact is, that for the most part, it is working as a charm, but we do have one difficulty which we do not seem to get around.
Suppose we want to index document "X" containing:
Row A - Field1: 4 + Field2: a
Row B - Field1: 8 + Field2: b
The index we would make would contain 4 fields:
Document X:
Field1: 4 (Numeric)
Field2: a (Text)
Field1: 8 (Numeric)
Field2: b (Text)
(The row Ids are not important)
Doing a search for Field1:[3 TO 6] AND Field2:b would have a hit on this document.
However, the link between the fields represented by the row (linking 4 and 'a') is gone.
We can concatenate the values like 4_a, but that would crush our numeric search and would require the clients to know which fields are concatenated for proper results. It would also increase difficulty with our analyzer, as for each field, we can add a different analyzer (mostly for language purposes).
Also, we can create a separate document for each row with the same key and add a distinct to the search results, but that doesn't sound like the way to go, does it? It would seriously multiply the number of documents as we would create between the 20 - 100 documents for each document we would create now. I haven't tested this on performance or usability, as the current implementation doesn't allow me to try this out very easily :-)
Does anyone know how I can force a link between certain fields within Lucene.Net, but still keep a way to search for each field individually?

Answer 1

I personally don't see why increased number of documents would affect the performance. At least in Java version of Lucene, the bulk of the memory is used for the terms cache - which is per term and has no relationship to document count (providing term count doesn't change). Can't elaborate on usability though as this is specific to your app.
The main point is that once you group the rows into documents, you lose row relationship info. You can fix that by adding extra fields (something like rowInfoA:4_a, rowInfoB:8_b) but this seems too cumbersome and will actually require far more memory. Yes, you can select not to index but only store these auxiliary fields but I (having given information) would still prefer the 1:1 row:document mapping.

Answer 2

One kludge is to add another field for linkages:
Document X:
Field1: 4 (Numeric)
Field2: a (Text)
Field1: 8 (Numeric)
Field2: b (Text)
Link: 4_a
Link: 8_b
Another kludge is to add a field like MyDocument:X and index each row separately, with each row containing a MyDocument field for its Document. This would let you filter by document later in your process.