Lucene Numeric Range Query Returning No Results / Unexpected Results - lucene.net

I'm using Lucene .NET v3 on .NET 4.5. I'm not sure why I'm getting incorrect results in my query. Here's a simple test app:
static class Program
{
static Document MakeDocument(String rcaName, int statusCode)
{
var doc = new Document();
doc.Add(new Field("RCAName", rcaName, Field.Store.YES, Field.Index.NO));
doc.Add(new Field("ActionId", Guid.NewGuid().ToString(), Field.Store.YES, Field.Index.NOT_ANALYZED));
var status = new NumericField("Status", Field.Store.YES, true);
status.SetLongValue(statusCode);
doc.Add(status);
return doc;
}
static void Main()
{
var dir = new SimpleFSDirectory(new DirectoryInfo("C:/Sologic/temp/bsindex"));
// var analyzer = new SologicAnalyzer() { StopWords = StopAnalyzer.ENGLISH_STOP_WORDS_SET };
var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
using (var writer = new Lucene.Net.Index.IndexWriter(dir, analyzer, true, Lucene.Net.Index.IndexWriter.MaxFieldLength.UNLIMITED))
{
writer.AddDocument(MakeDocument("RCA 1", 2));
writer.AddDocument(MakeDocument("RCA 2", 4));
writer.AddDocument(MakeDocument("RCA 3", 2));
writer.AddDocument(MakeDocument("RCA 4", 4));
}
var searcher = new Lucene.Net.Search.IndexSearcher(IndexReader.Open(dir, true));
var query = new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Status", analyzer).Parse("2");
var hits = searcher.Search(query, 100);
Console.WriteLine(hits.TotalHits);
Console.Read();
}
}
When I open the index with luke, I run the following queries:
Status:[2 TO 2] - 0
Status:[4 TO 4] - 0
Status:[0 TO 4] - 4
Status:[0 TO 2] - 4
Status:[0 TO 0] - 0
I'm a bit baffled. Anyone see what I'm doing wrong?

Two things:
A.) You're setting a long value with an int. I'm guessing this probably is leading to some wonky mishap a level below. Stop doing that.
status.SetLongValue(statusCode);
becomes
status.SetIntValue(statusCode);
B.) Don't parse a string to get an int. Basically QueryParser is the wrong choice here. Where you have:
var query = new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Status", analyzer).Parse("2");
you need
var query = NumericRangeQuery.NewIntRange("Status", 4, 4, true, true);
So now your problem is solved, but I still get no results in luke. Perhaps it's due to a misunderstanding of the tool itself. Anyhow, happy trails buddy.

Related

English stemming or lemmatization in Lucene.NET without SnowBall Analyzer or a custom analyzer

Is there a non-obsolete Lucene.NET Analyzer that can do english language stemming or lemmatization or do I need to write a custom Analyzer?
I can't seem to find an Analyzer that includes PorterStemFilter or EnglishMinimalStemFilter in the source code. I could write my own Analyzer, but it feels like that shouldn't be required, and I'd be reinventing the wrong wheel.
I'm trying to do Stemming of english words in Lucene.NET. As far as I can tell, this does not work out of the box. I tried using the EnglishAnalizer like so:
[TestFixture]
public class TestAnalyzers
{
private const string FieldName = "CustomFieldName";
public Directory CreateDirectory(IEnumerable<string> documents, Analyzer analyzer)
{
var directory = new RAMDirectory();
var iwc = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer)
{
OpenMode = OpenMode.CREATE_OR_APPEND,
};
var writer = new IndexWriter(directory, iwc);
writer.Commit();
foreach(var doc in documents) {
var document = new Document();
document.AddTextField(FieldName, doc, StoredField.Store.YES);
writer.AddDocument(document);
}
writer.Flush(true, true);
writer.Commit();
return directory;
}
private QueryParser CreateQueryParser(Analyzer analyzer)
=> new MultiFieldQueryParser(
LuceneVersion.LUCENE_48,
GetSearchFields(),
analyzer);
private string[] GetSearchFields() => new [] { FieldName };
[TestCase("for", "for")]
[TestCase("for", "forward")]
[TestCase("forward", "for")]
//[TestCase("retire", "retirement")]
[TestCase("retirement", "retire")]
[Test]
public void TestPartialWordsStandard(string fieldValue, string query)
{
var analyzer = new EnglishAnalyzer(LuceneVersion.LUCENE_48);
var directory = CreateDirectory(new [] { fieldValue }, analyzer);
var indexReader = DirectoryReader.Open(directory);
Assert.AreEqual(1, indexReader.NumDocs);
var doc = indexReader.Document(0);
Assert.NotNull(doc);
Assert.AreEqual(fieldValue, doc.GetField(FieldName).GetStringValue());
var searcher = new IndexSearcher(indexReader);
var queryObj = CreateQueryParser(analyzer).Parse(query);
var results = searcher.Search(queryObj, 2);
Assert.AreEqual(1, results.TotalHits);
doc = indexReader.Document(results.ScoreDocs.First().Doc);
Assert.AreEqual(fieldValue, doc.GetField(FieldName).GetStringValue());
}
}
It did no stemming. From reading the code it using a possessive filter to remove 's and s, but not the english stemming filter or thePorterStemFilter`.
I was able to get some stemming to happen with var analyzer = new SnowballAnalyzer(LuceneVersion.LUCENE_48, "English");. Its an adequate amount of stemming , but the class is obsolete.
The Lucene.Net EnglishAnalyzer does include porter stemming. In line 117 of the source code for the class is this line:
result = new PorterStemFilter(result);
I also ran a test in my system using the EnglishAnalyzer and confirmed that it is in fact stemming. So for example my indexed text contained the word "walking" and when I searched on "walked" I got a hit on the record.

How to read the unit price of an Item with IPP DevKit

I've seen elsewhere how to set the UnitPrice on an Item, using the wiley and elusive Item1 field as Intuit.Ipp.Data.Qbd.Money. But how do I READ the unit price from the Item1 field? I can't cast it. The new operator doesn't work ("new ...Money(myItem.Item1)"). So how do I get the price?
I realize the DevKit will probably never be changed so this makes sense. But can we at least get some doc explaining all those strange "xxxItemxxx" fields?
ServiceContext context = new ServiceContext(oauthValidator, realmId, intuitServiceType);
DataServices commonService = new DataServices(context);
Intuit.Ipp.Data.Qbd.Item qbdItem = new Intuit.Ipp.Data.Qbd.Item();
Intuit.Ipp.Data.Qbd.Money unitPrice = new Intuit.Ipp.Data.Qbd.Money();
unitPrice.Amount = 22;
unitPrice.AmountSpecified = true;
qbdItem.Item1 = unitPrice;
IEnumerable<Intuit.Ipp.Data.Qbd.Item> qbdItemsResult = commonService.FindAll(qbdItem, 1, 10) as IEnumerable<Intuit.Ipp.Data.Qbd.Item>;
foreach (var itemResult in qbdItemsResult)
{
Intuit.Ipp.Data.Qbd.Money test1UnitPrice = itemResult.Item1 as Intuit.Ipp.Data.Qbd.Money;
}
You can use the above code for .Net.
Response XML of Item entity suggests that 'UntiPrice' is a top level tag.
I tried this usecase using java. PFB code.
QBItemService itemService = QBServiceFactory.getService(context,QBItemService.class);
items = itemService.findAll(context,1, 100);
for (QBItem item : items) {
System.out.println("Name - " + item.getName() + " UnitPrice - " + item.getUnitPrice().getAmount());
Can you please try the same in .Net and let me know if it works in the same way.
Intuit.Ipp.Data.Qbd.Money [ getAmount() ]
Thanks

Testing With A Fake DbContext and Autofixture and Moq

SO follow this example
example and how make a fake DBContex For test my test using just this work fine
[Test]
public void CiudadIndex()
{
var ciudades = new FakeDbSet<Ciudad>
{
new Ciudad {CiudadId = 1, EmpresaId =1, Descripcion ="Santa Cruz", FechaProceso = DateTime.Now, MarcaBaja = null, UsuarioId = 1},
new Ciudad {CiudadId = 2, EmpresaId =1, Descripcion ="La Paz", FechaProceso = DateTime.Now, MarcaBaja = null, UsuarioId = 1},
new Ciudad {CiudadId = 3, EmpresaId =1, Descripcion ="Cochabamba", FechaProceso = DateTime.Now, MarcaBaja = null, UsuarioId = 1}
};
//// Create mock unit of work
var mockData = new Mock<IContext>();
mockData.Setup(m => m.Ciudades).Returns(ciudades);
// Setup controller
var homeController = new CiudadController(mockData.Object);
// Invoke
var viewResult = homeController.Index();
var ciudades_de_la_vista = (IEnumerable<Ciudad>)viewResult.Model;
// Assert..
}
Iam tryign now to use Autofixture-Moq
to create "ciudades" but I cant. I try this
var fixture = new Fixture();
var ciudades = fixture.Build<FakeDbSet<Ciudad>>().CreateMany<FakeDbSet<Ciudad>>();
var mockData = new Mock<IContext>();
mockData.Setup(m => m.Ciudades).Returns(ciudades);
I get this error
Cant convert System.Collections.Generic.IEnumerable(FakeDbSet(Ciudad)) to System.Data.Entity.IDbSet(Ciudad)
cant put "<>" so I replace with "()" in the error message
Implementation of IContext and FakeDbSet
public interface IContext
{
IDbSet<Ciudad> Ciudades { get; }
}
public class FakeDbSet<T> : IDbSet<T> where T : class
how can make this to work?
A minor point... In stuff like:
var ciudades_fixture = fixture.Build<Ciudad>().CreateMany<Ciudad>();
The second type arg is unnecessary and should be:
var ciudades_fixture = fixture.Build<Ciudad>().CreateMany();
I really understand why you need a FakeDbSet and the article is a bit TL;DR... In general, I try to avoid faking and mucking with ORM bits and instead dealing with interfaces returning POCOs to the max degree possible.
That aside... The reason the normal syntax for initialising the list works is that there is an Add (and IEnumerable) in DBFixture. AutoFixture doesn't have a story for that pattern directly (after all it is compiler syntactic sugar and not particularly amenable to reflection or in line with any other conventions) but you can use AddManyTo as long as there is an ICollection in play. Luckily, within the impl of FakeDbSet as in the article, the following gives us an in:-
public ObservableCollection<T> Local
{
get { return _data; }
}
As ObservableCollection<T> derives from ICollection<T>, you should be able to:
var ciudades = new FakeDbSet<Cuidad>();
fixture.AddManyTo(ciudades.Local);
var mockData = new Mock<IContext>();
mockData.Setup(m => m.Ciudades).Returns(ciudades);
It's possible to wire up a customization to make this prettier, but at least you have a way to manage it. The other option is to have something implement ICollection (or add a prop with a setter taking IEnumerable<T> and have AF generate the parent object, causing said collection to be filled in.
Long superseded side note: In your initial question, you effectively have:
fixture.Build<FakeDbSet<Ciudad>>().CreateMany()
The problem becomes clearer then - you are asking AF to generate Many FakeDbSet<Ciudad>s, which is not what you want.
I haven't used AutoFixture in a while, but shouldn't it be:
var ciudades = new FakeDbSet<Ciudad>();
fixture.AddManyTo(ciudades);
for the moment I end doing this, I will keep reading about how use automoq, cause I'm new in this
var fixture = new Fixture();
var ciudades_fixture = fixture.Build<Ciudad>().CreateMany<Ciudad>();
var ciudades = new FakeDbSet<Ciudad>();
foreach (var item in ciudades_fixture)
{
ciudades.Add(item);
}
var mockData = new Mock<IContext>();
fixture.Create<Mock<IContext>>();
mockData.Setup(r => r.Ciudades).Returns(ciudades);

Extending TokenStream

I am trying to index into a document a field with one term that has a payload.
Since the only constructor of Field that can work for me takes a TokenStream, I decided to inherit from this class and give the most basic implementation for what I need:
public class MyTokenStream : TokenStream
{
TermAttribute termAtt;
PayloadAttribute payloadAtt;
bool moreTokens = true;
public MyTokenStream()
{
termAtt = (TermAttribute)GetAttribute(typeof(TermAttribute));
payloadAtt = (PayloadAttribute)GetAttribute(typeof(PayloadAttribute));
}
public override bool IncrementToken()
{
if (moreTokens)
{
termAtt.SetTermBuffer("my_val");
payloadAtt.SetPayload(new Payload(/*bye[] data*/));
moreTokens = false;
}
return false;
}
}
The code which was used while indexing:
IndexWriter writer = //init tndex writer...
Document d = new Document();
d.Add(new Field("field_name", new MyTokenStream()));
writer.AddDocument(d);
writer.Commit();
And the code that was used during the search:
IndexSearcher searcher = //init index searcher
Query query = new TermQuery(new Term("field_name", "my_val"));
TopDocs result = searcher.Search(query, null, 10);
I used the debugger to verify that call to IncrementToken() actually sets the TermBuffer.
My problem is that the returned TopDocs instance returns no documents, and I cant understand why... Actually I started from TermPositions (which gives me approach to the Payload...), but it also gave me no results.
Can someone explain to me what am I doing wrong?
I am currently using Lucene .NET 2.9.2
After you set the TermBuffer you need to return true from IncrementToken, you return false when you have nothing to feed the TermBuffer with anymore

How do you increment a field in mongodb using c#

Thought this would be pretty straight forward, but my value is remaining the same (0).
What I'd like to do is increment my UnreadMessages field when the user receives a message they haven't read and then decrement it when they have. So I thought code like this would work:
var userHelper = new MongoHelper<User>();
//increment
userHelper.Collection.Update(Query.EQ("Id", userId.ToId()), Update.Inc("UnreadMessages", 1));
//decrement
userHelper.Collection.Update(Query.EQ("Id", userId.ToId()), Update.Inc("UnreadMessages", -1));
After running these no errors are thrown but the value doesn't change either. And no I'm not running one after the other as the code above could be interpreted :)
Update
Here's my helper class:
public class MongoHelper<T> : Sandbox.Services.IMongoHelper<T> where T : class
{
public MongoCollection<T> Collection { get; private set; }
public MongoHelper()
{
var con = new MongoConnectionStringBuilder(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString);
var server = MongoServer.Create(con);
var db = server.GetDatabase(con.DatabaseName);
Collection = db.GetCollection<T>(typeof(T).Name.ToLower());
}
}
and thanks to Travis' answer I was able to pull this off:
MongoHelper<UserDocument> userHelper = new MongoHelper<UserDocument>();
var user = userHelper.Collection.FindAndModify(Query.EQ("Username", "a"), SortBy.Null, Update.Inc("MessageCount", 1), true).GetModifiedDocumentAs<UserDocument>();
Not sure what your helper does. Here is a working snippet I use:
var query = Query.And(Query.EQ("_id", keyName));
var sortBy = SortBy.Null;
var update = Update.Inc("KeyValue", adjustmentAmount);
var result = collection.FindAndModify(query, sortBy, update, true);
So, "query" finds the document, update does the increment, and FindAndModify puts them together and actually hits the database.