I have some class:
class SomeClass(val client: ElasticClient, val config: Config, val configName: String)(implicit val ec: ExecutionContext)
extends ElasticSearchRepositoryWrapper[AnotherClass]{
override def mapping: Option[MappingDefinition] = Some(
properties(
KeywordField("id"),
TextField("name").fielddata(true).analyzer("ngram_analyzer"),
KeywordField("lang"),
BasicField("order", "long"),
...
)
)
I'm creating an index with repository.createIndexIfNotExists() using this mapping.
Now I must create ngram_analyzer in my index settings:
"settings": {
"index": {
"analysis": {
"analyzer": {
"ngram_analyzer": {
"filter": [
"lowercase"
],
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"token_chars": [
"letter",
"digit"
],
"min_gram": "3",
"type": "ngram",
"max_gram": "3"
}
}
}
}
How can I do that using elastic4s?
OK. A lot of functions of createIndexIfNotExists() were deprecated. So, I used CreateIndexRequest where I put my analyzer:
CreateIndexRequest(repository.indexName, analysis = Option(ngramAnalyzer), mapping = repository.mapping)
.shards(repository.shards)
.replicas(repository.replicas)
And I initialized my analyzer like this:
val ngramAnalyzer = Analysis(
List(CustomAnalyzer(
name = "ngram_analyzer",
tokenizer = "ngram",
charFilters = Nil,
tokenFilters = List("lowercase")
))
)
Related
I am trying to extract the following avro record
{
"StateName": "Alabama",
"Capital": "Montgomery",
"Counties": [{
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}, {
"CountyName": "Calhoun",
"CountyPopulation": 100000,
"Cities": [{
"CityName": "Anniston",
"CityPopulation": 23000
},
{
"CityName": "Glencoe",
"CityPopulation": 5000
}
]
}]
}
and modify them and create new individual record like this(Extract each county and create new records based on county like this)
{
"StateName": "Alabama",
"Capital": "Montgomery",
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}
I am trying to extract the records using the json4s. Taken the reference from https://nmatpt.com/blog/2017/01/29/json4s-custom-serializer/
val StateName = avroRecord.get("StateName").asInstanceOf[Utf8].toString
val Capital = avroRecord.get("Capital").asInstanceOf[Utf8].toString
val CountyArray = avroRecord.get("Counties").toString
val jsonData = parse(CountyArray, useBigDecimalForDouble = true)
val CountyList = jsonData match {
case JArray(_) =>
jsonData.extract[List[CountyArrayRecord]]
case JObject(_) =>
List(jsonData.extract[CountyArrayRecord])
List()
}
Custom serializer
implicit val formats: Formats = Serialization.formats(NoTypeHints) + new TestSerializer
class TestSerializer extends CustomSerializer[CountyArrayRecord](format => (
{ case jsonObj: JObject =>
val countyName = (jsonObj \ "CountyName").extract[String]
val countyPopulation = (jsonObj \ "CountyPopulation").extract[Int]
val cities = (jsonObj \ "Cities").extract[List[GenericRecord]]
CountyArrayRecord(countyName, countyPopulation, cities)
}
)
)
Once extracted trying to create list new records using avro4s.Taken reference from this https://github.com/sksamuel/avro4s#avro-records
val returnList = CountyList.map { CountyListRecord =>
val record = FinalCountyRecord (StateName, Capital, CountyListRecord.CountyName, CountyListRecord.CountyPopulation, CountyListRecord.Cities)
val format = RecordFormat[FinalCountyRecord]
format.to(record)
}
returnList
But this does not seem to work since county list has another list(Cities) inside.
I am trying to parse list of dictionaries(which is in string) inside scala. Basically i want to build another list so that i can traverse through the list using a for loop.
When i have one single list of dictionaries it works fine.
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]
val jsonString =
"""
{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
}
""".stripMargin
//println(List(JSON.parseFull(jsonString)))
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
//L(text) = map("text")
//M(texts) <- text
I(index) = map("index")
S(source) = map("source")
N(name) = map("name")
A(s3q)=map("s3")
G(s3data) <- s3q
F(path) = s3data("path")
} yield {
(index.toInt,source,name,path)
}
But when i aded another list, it gives error stating "java.lang.ClassCastException: scala.collection.immutable.$colon$colon cannot be cast to scala.collection.immutable.Map"
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]
val jsonString =
"""
[{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
},{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
}]
""".stripMargin
//println(List(JSON.parseFull(jsonString)))
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
//L(text) = map("text")
//M(texts) <- text
I(index) = map("index")
S(source) = map("source")
N(name) = map("name")
A(s3q)=map("s3")
G(s3data) <- s3q
F(path) = s3data("path")
} yield {
(index.toInt,source,name,path)
}
I need modify .block() repository database operations to without .block() -ยป just put request to stream not to memory.
When the results return simple entity or just something list is ok!
But when I return combined dto I get this:
{
"resultsLength": 1,
"results": {
"scanAvailable": true
}
}
Mapped dto to response
data class CustomObjectResultDto(
val resultsLength: Long?,
#JsonInclude
val results: Mono<MutableList<CustomObjectDto>>
)
JsonFormat:
Want:
{
"resultsLength": 1,
"results": {
"id": "61953fb7e85fe0605c00bdd1",
"cardRequestDate": "2029-02-05T23:00:00Z",
"userName": "XYZ",
"applicationType": {
"name": "plastic",
"label": "plastic_label"
},
"status": {
"name": "accepted",
"label": "accepted_label"
},
"statusChangeDate": "2019-02-10T23:00:00Z",
"virtualCardNumber": 157426399
}
}
Get:
{
"resultsLength": 1,
"results": {
"scanAvailable": true
}
}
Repository:
override fun listing(ctx: context, filterParams: FilterParams, filterConfig: FilterConfig):
Mono<MutableList<CustomObject>> {
val sort = Sort.by(Sort.Direction.fromString(filterConfig.order), filterConfig.orderBy)
val pageRequest = PageRequest.of(filterConfig.pageIndex, filterConfig.pageLimit, sort)
val userInfoWithoutUser = context(
partnerId = ctx.partnerId,
correlationId = ctx.correlationId
)
val query =
createQuery(userInfoWithoutUser, filterParams).collation(Collation.of(DEFAULT_COLLATION)).with(pageRequest)
return reactiveMongoTemplate.find(query, CustomObject::class.java).collectList()
}
Service:
override fun listing(
ctx: context,
filterConfig: FilterConfig,
filterParams: FilterParams
): CustomObjectResultDto {
val result = repository.listing(ctx, filterParams, filterConfig)
.flatMapIterable { list: List<CostumObject> ->
list.map { it.toItemDto() }
}.collectList()
val count = repository.count(ctx, filterParams)
return CustomObjectResultDto(count,result)
}
Controller:
...
): MutableHttpResponse<CustomObjectResultDto> {
val ctx = context(partnerId, userId, correlationId)
...log...
val filterParams = FilterParams(userName, applicationType, status, wifiStatus, cardNumber)
val filterConfig = FilterConfig(pageLimit, pageIndex, PAGE_ORDER_BY_VALUES.getValue(orderBy), order)
return HttpResponse.ok(service.listing(ctx, filterConfig, filterParams))
}
I solved
override fun listing(
ctx: context,
filterConfig: FilterConfig,
filterParams: FilterParams
): Mono<CustomObjectResultDto> {
val count = repository.count(ctx, filterParams)
return repository.listing(ctx, filterParams, filterConfig)
.flatMapIterable { list: List<CustomObject> ->
list.map { it.toItemDto() }
}.collectList()
.map { listOfDocuments ->
CustomObjectResultDto(count, listOfDocuments)
}
}
I have the JSon response as given below.
If metadata's Organic=true then label='true-Organic', else label='non-Organic'
in the end => return List or Map[modelId,label]
import net.liftweb.json.{DefaultFormats, _}
object test1 extends App {
val json_response =
"""{
"requestId": "91ee60d5f1b45e#316",
"error": null,
"errorMessages": [
],
"entries": [
{
"modelId":"RT001",
"sku": "SKU-ASC001",
"store": "New Jersey",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-01-22T01:25Z",
"organic": "true"
}
},
{
"modelId":"RT002",
"sku": "SKU-ASC002",
"store": "livingstone",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-10-03T01:25Z",
"organic": "false"
}
}
] }"""
tried like this :
val json = parse(json_response)
implicit val formats = DefaultFormats
var map = Map[String, String]()
case class Sales(modelId: String, sku: String, store: String, ttlInSeconds: Int, metadata:
Map[String, String])
case class Response(entries: List[Sales])
val response = json.extract[Response]
After this, not sure how to proceed.
This is a straightforward map operation on the entries field:
response.entries.map{ e =>
e.modelId ->
if (e.metadata.get("organic").contains("true")) {
"true-Organic"
} else {
"non-Organic"
}
}
This will return List[(String, String)], but you can call toMap to turn this into a Map if required.
How do you create a jackson custom serializer and use it in your program? The serializer is used to serialize data from a kafka stream, because my job fails if it encounters a null.
I tried the following to create a serializer.
import org.json4s._
import org.json4s.jackson.JsonMethods._
case class Person(
val user: Option[String]
)
object PersonSerializer extends CustomSerializer[Person](formats => ( {
case JObject(JField("user", JString(user)) :: Nil) => Person(Some(user))
case JObject(JField("user", null) :: Nil) => Person(None)
},
{
case Person(Some(user)) => JObject(JField("user", JString(user)) :: Nil)
case Person(None) => JObject(JField("user", JString(null)) :: Nil)
}))
I am trying to use it this way.
object ConvertJsonTOASTDeSerializer extends App
{
case class Address(street : String, city : String)
case class PersonAddress(name : String, address : Address)
val testJson1 =
"""
{ "user": null,
"address": {
"street": "Bulevard",
"city": "Helsinki",
"country": {
"code": "CD" }
},
"children": [
{
"name": "Mary",
"age": 5,
"birthdate": "2004-09-04T18:06:22Z"
},
{
"name": "Mazy",
"age": 3
}
]
}
"""
implicit var formats : Formats = DefaultFormats + PersonSerializer
val output = parse(testJson1).as[Person]
println(output.user)
}
I am getting an error saying that
Error:(50, 35) No JSON deserializer found for type com.examples.json4s.Person. Try to implement an implicit Reader or JsonFormat for this type.
val output = parse(testJson1).as[Person]
Not sure if I answer your question. I provide the runnable code:
import org.json4s._
import org.json4s.jackson.JsonMethods._
case class Person(
user: Option[String],
address: Address,
children: List[Child]
)
case class Address(
street: String,
city: String,
country: Country
)
case class Country(
code: String
)
case class Child(
name: String,
age: Int
)
val s =
"""
{ "user": null,
"address": {
"street": "Bulevard",
"city": "Helsinki",
"country": {
"code": "CD" }
},
"children": [
{
"name": "Mary",
"age": 5,
"birthdate": "2004-09-04T18:06:22Z"
},
{
"name": "Mazy",
"age": 3
}
]
}
"""
implicit val formats : Formats = DefaultFormats
parse(s).extract[Person] // Person(None,Address(Bulevard,Helsinki,Country(CD)),List(Child(Mary,5), Child(Mazy,3)))