Kotlin : Mono<MutableList<Object>> don't return as dto when i put to ResultDto - mongodb

I need modify .block() repository database operations to without .block() -ยป just put request to stream not to memory.
When the results return simple entity or just something list is ok!
But when I return combined dto I get this:
{
"resultsLength": 1,
"results": {
"scanAvailable": true
}
}
Mapped dto to response
data class CustomObjectResultDto(
val resultsLength: Long?,
#JsonInclude
val results: Mono<MutableList<CustomObjectDto>>
)
JsonFormat:
Want:
{
"resultsLength": 1,
"results": {
"id": "61953fb7e85fe0605c00bdd1",
"cardRequestDate": "2029-02-05T23:00:00Z",
"userName": "XYZ",
"applicationType": {
"name": "plastic",
"label": "plastic_label"
},
"status": {
"name": "accepted",
"label": "accepted_label"
},
"statusChangeDate": "2019-02-10T23:00:00Z",
"virtualCardNumber": 157426399
}
}
Get:
{
"resultsLength": 1,
"results": {
"scanAvailable": true
}
}
Repository:
override fun listing(ctx: context, filterParams: FilterParams, filterConfig: FilterConfig):
Mono<MutableList<CustomObject>> {
val sort = Sort.by(Sort.Direction.fromString(filterConfig.order), filterConfig.orderBy)
val pageRequest = PageRequest.of(filterConfig.pageIndex, filterConfig.pageLimit, sort)
val userInfoWithoutUser = context(
partnerId = ctx.partnerId,
correlationId = ctx.correlationId
)
val query =
createQuery(userInfoWithoutUser, filterParams).collation(Collation.of(DEFAULT_COLLATION)).with(pageRequest)
return reactiveMongoTemplate.find(query, CustomObject::class.java).collectList()
}
Service:
override fun listing(
ctx: context,
filterConfig: FilterConfig,
filterParams: FilterParams
): CustomObjectResultDto {
val result = repository.listing(ctx, filterParams, filterConfig)
.flatMapIterable { list: List<CostumObject> ->
list.map { it.toItemDto() }
}.collectList()
val count = repository.count(ctx, filterParams)
return CustomObjectResultDto(count,result)
}
Controller:
...
): MutableHttpResponse<CustomObjectResultDto> {
val ctx = context(partnerId, userId, correlationId)
...log...
val filterParams = FilterParams(userName, applicationType, status, wifiStatus, cardNumber)
val filterConfig = FilterConfig(pageLimit, pageIndex, PAGE_ORDER_BY_VALUES.getValue(orderBy), order)
return HttpResponse.ok(service.listing(ctx, filterConfig, filterParams))
}

I solved
override fun listing(
ctx: context,
filterConfig: FilterConfig,
filterParams: FilterParams
): Mono<CustomObjectResultDto> {
val count = repository.count(ctx, filterParams)
return repository.listing(ctx, filterParams, filterConfig)
.flatMapIterable { list: List<CustomObject> ->
list.map { it.toItemDto() }
}.collectList()
.map { listOfDocuments ->
CustomObjectResultDto(count, listOfDocuments)
}
}

Related

How to complete restful api query with rust and actix-web?

I have some data, from the request's querystring, the structure like this:
Crates: serde_qs serde mongodb and its serde_helpers
filter: {
linkOperator: "and", // maybe 'or'
items: [
{
field: "parent",
operator: "is", // maybe 'isNot', 'isAnyOf'
value: "id"
},
{
field: "createdAt",
operator: "is", // maybe 'before', 'after', .etc
value: "RFC 3339 string"
}
]
}
I use the serde crate to handle it, the linkOperator as enum, items as its type. I use the mongodb as the database.
#[derive(Debug, Deserialize, Serialize, PartialEq)]
#[serde(tag = "linkOperator", content = "items", rename_all = "camelCase")]
pub enum Filter<T> {
And(Vec<T>),
Or(Vec<T>),
}
pub trait ToFilter {
fn to_filter(self) -> Document;
}
impl<T> Filter<T> {
pub fn to_filter(self) -> Document
where
T: ToFilter,
{
match self {
Filter::And(items) => {
let mut filter = vec![];
for item in items {
filter.push(item.to_filter());
}
doc! {
"$and": filter
}
}
Filter::Or(items) => {
let mut filter = vec![];
for item in items {
filter.push(item.to_filter());
}
doc! {
"$or": filter
}
}
}
}
}
Then, I further process it in the next progress. This enum is used to handle datetime. Use the content of the vec above. operator field as enum, value field as enum's inner type.
#[derive(Debug, Deserialize, Serialize, PartialEq)]
#[serde(rename_all = "camelCase", tag = "operator", content = "value")]
pub enum DateTimeFilter {
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
Is(DateTime),
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
Not(DateTime),
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
After(DateTime),
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
OnOrAfter(DateTime),
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
Before(DateTime),
#[serde(deserialize_with = "deserialize_bson_datetime_from_rfc3339_string")]
OnOrBefore(DateTime),
IsEmpty,
IsNotEmpty,
}
impl DateTimeFilter {
pub fn to_filter(self, field: &'static str) -> Document {
match self {
DateTimeFilter::Is(value) => doc! { field: value },
DateTimeFilter::Not(value) => doc! { field: { "$ne": value } },
DateTimeFilter::After(value) => doc! { field: { "$gt": value } },
DateTimeFilter::OnOrAfter(value) => doc! { field: { "$gte": value } },
DateTimeFilter::Before(value) => doc! { field: { "$lt": value } },
DateTimeFilter::OnOrBefore(value) => doc! { field: { "$lte": value } },
DateTimeFilter::IsEmpty => doc! { field: { "$exists": false } },
DateTimeFilter::IsNotEmpty => doc! { field: { "$exists": true } },
}
}
}
There are some other enums which have a similar effect.
#[derive(Debug, Deserialize, Serialize, PartialEq)]
#[serde(rename_all = "camelCase", tag = "operator", content = "value")]
pub enum SingleSelectFilter<T> {
Is(T),
Not(T),
IsAnyOf(Vec<T>),
}
impl<T> SingleSelectFilter<T> {
pub fn to_filter(self, field: &'static str) -> Document
where
Bson: From<T> + From<Vec<T>>,
{
match self {
SingleSelectFilter::Is(value) => doc! { field: value },
SingleSelectFilter::Not(value) => doc! { field: { "$ne": value } },
SingleSelectFilter::IsAnyOf(values) => doc! { field: { "$in": values } },
}
}
}
I combine them.
#[derive(Debug, Deserialize, Serialize, PartialEq)]
pub struct PostQuery {
filter: Option<Filter<FilterField>>,
#[serde(flatten)]
sort: Option<Sort<SortField>>,
#[serde(flatten)]
pagination: Pagination,
}
/// add the new field here
#[derive(Debug, Deserialize, Serialize, PartialEq)]
#[serde(tag = "field", rename_all = "camelCase")]
pub enum FilterField {
Published(SingleSelectFilter<bool>),
PublishedAt(DateTimeFilter),
CreatedAt(DateTimeFilter),
Creator(SingleSelectFilter<ObjectId>),
}
impl ToFilter for FilterField {
fn to_filter(self) -> Document {
match self {
FilterField::Published(filter) => filter.to_filter("published"),
FilterField::PublishedAt(filter) => filter.to_filter("publishedAt"),
FilterField::CreatedAt(filter) => filter.to_filter("createdAt"),
FilterField::Creator(filter) => filter.to_filter("creator"),
}
}
}
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct AggregatedPost {
#[serde(rename = "_id")]
id: ObjectId,
title: String,
slug: String,
content: String,
categories: Vec<Category>,
tags: Vec<Tag>,
#[serde(rename = "publishedAt")]
published_at: DateTime,
published: bool,
#[serde(rename = "createdAt")]
created_at: DateTime,
pub creator: User,
}
pub async fn aggregate_posts(
query: PostQuery,
service: &Service,
) -> Result<(u64, Vec<Self>), AppError> {
let filter = match query.filter {
Some(filter) => filter.to_filter(),
None => doc! {},
};
let skip = query.pagination.skip();
let limit = query.pagination.limit();
let options = CountOptions::builder().skip(skip).limit(limit).build();
let count = service
.count::<Self>(filter.clone(), options, "posts")
.await?;
let mut pipeline = vec![
doc! {
"$match": filter
},
doc! {
"$skip": skip as u32
},
doc! {
"$limit": limit as u32
},
doc! {
"$lookup": {
"from": "categories",
"localField": "categories",
"foreignField": "_id",
"as": "categories"
}
},
doc! {
"$lookup": {
"from": "tags",
"localField": "tags",
"foreignField": "_id",
"as": "tags"
}
},
doc! {
"$lookup": {
"from": "users",
"localField": "creator",
"foreignField": "_id",
"as": "creator"
}
},
doc! {
"$unwind": {
"path": "$creator",
"preserveNullAndEmptyArrays": true
}
},
];
if query.sort.is_some() {
let sort = query.sort.unwrap().to_sort();
pipeline.insert(1, doc! { "$sort": sort });
}
match service
.aggregate_docs::<Self>(pipeline, None, "posts")
.await
{
Ok(posts) => Ok((count, posts)),
Err(error) => Err(error),
}
}
Use this is a router handler with serde_qs
pub async fn aggregate_posts(
session: Session,
service: Data<Service>,
query: QsQuery<PostQuery>,
) -> Result<HttpResponse, AppError> {
let user = protect(&session, &service).await?;
user.has_permission(
Permission::ReadAggregatedPosts,
"You don't have permission to get aggregated posts.",
)?;
let (count, posts) = AggregatedPost::aggregate_posts(query.into_inner(), &service).await?;
let posts_json = posts
.into_iter()
.map(|post| post.into_json(&user))
.collect::<Vec<Value>>();
Ok(HttpResponse::Ok().json(json!({ "count": count, "posts": posts_json })))
}
Finally, the problem arises. When I receive a request with a query like this.
filter: {
linkOperator: "and", // maybe 'or'
items: [
{
field: "published",
operator: "isAnyOf", // maybe 'before', 'after', .etc
value: [true]
}
]
}
My query error handler will generate the message like this. Custom("invalid type: string \"true\", expected a boolean")
pub fn query_error_handler(error: serde_qs::Error, _req: &HttpRequest) -> Error {
let app_error = match &error {
serde_qs::Error::Custom(message) => BadRequest(format!("Query params: {}.", message)),
_ => BadRequest("Invalid query params.".to_string()),
};
InternalError::from_response(error, app_error.error_response()).into()
}
My api test result
How can i convert the string to a bool value here? Or, have any other options to complete a query in restful api with actix-web?

Extracting list from avro record and converting to new record

I am trying to extract the following avro record
{
"StateName": "Alabama",
"Capital": "Montgomery",
"Counties": [{
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}, {
"CountyName": "Calhoun",
"CountyPopulation": 100000,
"Cities": [{
"CityName": "Anniston",
"CityPopulation": 23000
},
{
"CityName": "Glencoe",
"CityPopulation": 5000
}
]
}]
}
and modify them and create new individual record like this(Extract each county and create new records based on county like this)
{
"StateName": "Alabama",
"Capital": "Montgomery",
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}
I am trying to extract the records using the json4s. Taken the reference from https://nmatpt.com/blog/2017/01/29/json4s-custom-serializer/
val StateName = avroRecord.get("StateName").asInstanceOf[Utf8].toString
val Capital = avroRecord.get("Capital").asInstanceOf[Utf8].toString
val CountyArray = avroRecord.get("Counties").toString
val jsonData = parse(CountyArray, useBigDecimalForDouble = true)
val CountyList = jsonData match {
case JArray(_) =>
jsonData.extract[List[CountyArrayRecord]]
case JObject(_) =>
List(jsonData.extract[CountyArrayRecord])
List()
}
Custom serializer
implicit val formats: Formats = Serialization.formats(NoTypeHints) + new TestSerializer
class TestSerializer extends CustomSerializer[CountyArrayRecord](format => (
{ case jsonObj: JObject =>
val countyName = (jsonObj \ "CountyName").extract[String]
val countyPopulation = (jsonObj \ "CountyPopulation").extract[Int]
val cities = (jsonObj \ "Cities").extract[List[GenericRecord]]
CountyArrayRecord(countyName, countyPopulation, cities)
}
)
)
Once extracted trying to create list new records using avro4s.Taken reference from this https://github.com/sksamuel/avro4s#avro-records
val returnList = CountyList.map { CountyListRecord =>
val record = FinalCountyRecord (StateName, Capital, CountyListRecord.CountyName, CountyListRecord.CountyPopulation, CountyListRecord.Cities)
val format = RecordFormat[FinalCountyRecord]
format.to(record)
}
returnList
But this does not seem to work since county list has another list(Cities) inside.

Multiplying event case class depending on the list based on nested IDs

I am processing a dataframe and converting into Dataset[Event] using Event case class.How ever there are nested Ids for which i need to multiply the events based on the flattening of nested device:os.
I am able to return the case class Event at the Kafka event level. But not sure how to multiply events .
Kafka incoming Event:
{
"partition": 1,
"key": "34768_20220203_MFETP501",
"offset": 1841543,
"createTime": 1646041475348,
"topic": "topic_int",
"publishTime": 1646041475344,
"errorCode": 0,
"userActions": {
"productId": "3MFETP501",
"createdDate": "2022-02-26T11:19:35.786Z",
"events": [
{
"GUID": "dbb1-f38b-f7f0-44af-90da-80179412f89c",
"eventDate": "2022-02-26T11:19:35.786Z",
"familyId": 2010,
"productTypeId": 1004678,
"serialID": "890479804",
"productName": "MFE Total Protection 2021 Family Pack",
"features": {
"mapping": [
{
"deviceId": 999795,
"osId": [
100
]
},
{
"deviceId": 987875
"osId": [
101
]
}
]
}
}
]
}
}
The expected output case classes for Event
Event("3MFETP501","1004678","2010","3MFETP501:890479804","MFE Total Protection 2021 Family Pack","999795_100", Map("targetId"->"999795_100") )
Event("3MFETP501","1004678","2010","3MFETP501:890479804","MFE Total Protection 2021 Family Pack","987875_100", Map("targetId"->"987875_100") )
case class Event(
productId: String,
familyId: String,
productTypeId: String,
key: String,
productName: String,
deviceOS:String,
var featureMap: mutable.Map[String, String])
val finalDataset:Dataset[Event] = inputDataFrame.flatMap(
row=> {
val productId = row.getAs[String]("productId")
val userActions = row.getAs[Row]("userActions")
val userEvents:mutable.Seq[Row] = userActions.getAs[mutable.WrappedArray[Row]]("events")
val processedEvents:mutable.Seq[Row]= userEvents.map(
event=>
val productTypeId = event.getAs[Int]("productTypeId")
val familyId = event.getAs[String]("familyId")
val features = activity.getAs[mutable.WrappedArray[Row]]("features")
val serialId = activity.getAs[String]("serialId")
val key = productId+":"+serialId
val features = mutable.Map[String, String]().withDefaultValue(null)
val device_os_list=List("999795_100","987875_101")
//Feature Map is for every device_os ( example "targetId"->"999795_100") for 999795_100
if (familyId == 2010 )
{
val a: Option[List[String]] = flatten the deviceId,osId ..
a.get.map(i=>{
val key: String = methodToCombinedeviceIdAndosId
val featureMapping: mutable.Map[String, String] = getfeatureMapForInvidualKey
Event(productId,productTypeId,familyId,key,productName,device_os,feature) ---> This is returning **List[Event]**
})
}
else{
Event(productId,productTypeId,familyId,key,productName,device_os,feature) --> This is returning **Event**. THIS WORKS
}
)
}
)
I do not implement it fully the same but I think it will be possible to understand logic and apply it on your case.
I created json file like kafka.json and put there code like this(your event):
[{
"partition": 1,
"key": "34768_20220203_MFETP501",
"offset": 1841543,
"createTime": 1646041475348,
"topic": "topic_int",
"publishTime": 1646041475344,
"errorCode": 0,
"userActions": {
"productId": "3MFETP501",
"createdDate": "2022-02-26T11:19:35.786Z",
"events": [
{
"GUID": "dbb1-f38b-f7f0-44af-90da-80179412f89c",
"eventDate": "2022-02-26T11:19:35.786Z",
"familyId": 2010,
"productTypeId": 1004678,
"serialID": "890479804",
"productName": "MFE Total Protection 2021 Family Pack",
"features": {
"mapping": [
{
"deviceId": 999795,
"osId": [
100
]
},
{
"deviceId": 987875,
"osId": [
101
]
}
]
}
}
]
}
}]
Please find below first solution that is based on flatMap and for loop.
case class Event(
productId: String,
familyId: String,
productTypeId: String,
key: String,
productName: String,
deviceOS: String,
featureMap: Map[String, String])
import org.apache.spark.sql.{Dataset, Row, SparkSession}
import scala.collection.mutable
val spark = SparkSession
.builder
.appName("StructuredStreaming")
.master("local[*]")
.getOrCreate()
private val inputDataFrame = spark.read.option("multiline", "true").format("json").load("/absolute_path_to_kafka.json")
import spark.implicits._
val finalDataset: Dataset[Event] = inputDataFrame.flatMap(
row => {
val userActions = row.getAs[Row]("userActions")
val productId = userActions.getAs[String]("productId")
val userEvents = userActions.getAs[mutable.WrappedArray[Row]]("events")
for (event <- userEvents;
familyId = event.getAs[Int]("familyId").toString;
productTypeId = event.getAs[Int]("productTypeId").toString;
serialId = event.getAs[String]("serialID");
productName = event.getAs[String]("productName");
key = s"$productId:$serialId";
features = event.getAs[Row]("features");
mappings = features.getAs[mutable.WrappedArray[Row]]("mapping");
mappingRow <- mappings;
deviceId = mappingRow.getAs[Long]("deviceId");
osIds = mappingRow.getAs[mutable.WrappedArray[Long]]("osId");
osId <- osIds;
deviseOs = deviceId + "_" + osId
) yield Event(productId, familyId, productTypeId, key, productName, deviseOs, Map("target" -> (deviseOs)))
}
)
finalDataset.foreach(e => println(e))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,999795_100,Map(target -> 999795_100))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,987875_101,Map(target -> 987875_101))
Also, you can solve this task using select, withColumn, explode, concat functions.
case class Event(
productId: String,
familyId: String,
productTypeId: String,
key: String,
productName: String,
deviceOS: String,
featureMap: Map[String, String])
import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.functions.{col, explode, concat, lit, map}
val spark = SparkSession
.builder
.appName("StructuredStreaming")
.master("local[*]")
.getOrCreate()
private val inputDataFrame = spark.read.option("multiline", "true").format("json").load("/absolute_path_to_kafka.json")
val transformedDataFrame = inputDataFrame
.select(col("userActions.productId").as("productId"),
explode(col("userActions.events")).as("event"))
.select(col("productId"),
col("event.familyId").as("familyId"),
col("event.productTypeId").as("productTypeId"),
col("event.serialID").as("serialID"),
col("event.productName").as("productName"),
explode(col("event.features.mapping")).as("features")
)
.select(
col("productId"),
col("familyId"),
col("productTypeId"),
col("serialID"),
col("productName"),
col("features.deviceId").as("deviceId"),
explode(col("features.osId")).as("osId")
)
.withColumn("key", concat(col("productId"), lit(":"), col("serialID")))
.withColumn("deviceOS", concat(col("deviceId"), lit("_"), col("osId")))
.withColumn("featureMap", map(lit("target"), col("deviceOS")))
import spark.implicits._
private val result: Dataset[Event] = transformedDataFrame.as[Event]
result.foreach(e => println(e))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,999795_100,Map(target -> 999795_100))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,987875_101,Map(target -> 987875_101))
Add option to customize response based on the value one of the field. I replace here for comprehension to map/flatmap, so you can return as response one or several events based on the type. Also, I customized json a little bit to show more examples in the result.
New json:
[{
"partition": 1,
"key": "34768_20220203_MFETP501",
"offset": 1841543,
"createTime": 1646041475348,
"topic": "topic_int",
"publishTime": 1646041475344,
"errorCode": 0,
"userActions": {
"productId": "3MFETP501",
"createdDate": "2022-02-26T11:19:35.786Z",
"events": [
{
"GUID": "dbb1-f38b-f7f0-44af-90da-80179412f89c",
"eventDate": "2022-02-26T11:19:35.786Z",
"familyId": 2010,
"productTypeId": 1004678,
"serialID": "890479804",
"productName": "MFE Total Protection 2021 Family Pack",
"features": {
"mapping": [
{
"deviceId": 999795,
"osId": [
100,
110
]
},
{
"deviceId": 987875,
"osId": [
101
]
}
]
}
},
{
"GUID": "1111-2222-f7f0-44af-90da-80179412f89c",
"eventDate": "2022-03-26T11:19:35.786Z",
"familyId": 2011,
"productTypeId": 1004679,
"serialID": "890479805",
"productName": "Product name",
"features": {
"mapping": [
{
"deviceId": 999796,
"osId": [
103
]
},
{
"deviceId": 987877,
"osId": [
104
]
}
]
}
}
]
}
}]
Please find code below:
case class Event(
productId: String,
familyId: String,
productTypeId: String,
key: String,
productName: String,
deviceOS: String,
featureMap: Map[String, String])
import org.apache.spark.sql.{Dataset, SparkSession}
val spark = SparkSession
.builder
.appName("StructuredStreaming")
.master("local[*]")
.getOrCreate()
private val inputDataFrame = spark.read.option("multiline", "true").format("json").load("/absolute_path_to_kafka.json")
import spark.implicits._
val finalDataset: Dataset[Event] = inputDataFrame.flatMap(
row => {
val userActions = row.getAs[Row]("userActions")
val productId = userActions.getAs[String]("productId")
val userEvents = userActions.getAs[mutable.WrappedArray[Row]]("events")
for (event <- userEvents;
productTypeId = event.getAs[Int]("productTypeId").toString;
serialId = event.getAs[String]("serialID");
productName = event.getAs[String]("productName");
key = s"$productId:$serialId";
familyId = event.getAs[Int]("familyId").toString;
features = event.getAs[Row]("features");
mappings = features.getAs[mutable.WrappedArray[Row]]("mapping");
mappingRow <- mappings;
deviceId = mappingRow.getAs[Long]("deviceId");
osIds = mappingRow.getAs[mutable.WrappedArray[Long]]("osId");
osId <- osIds;
deviseOs = deviceId + "_" + osId
) yield Event(productId, familyId, productTypeId, key, productName, deviseOs, Map("target" -> deviseOs))
userEvents.flatMap(event => {
val productTypeId = event.getAs[Int]("productTypeId").toString
val serialId = event.getAs[String]("serialID")
val productName = event.getAs[String]("productName")
val key = s"$productId:$serialId"
val familyId = event.getAs[Long]("familyId")
if(familyId == 2010) {
val features = event.getAs[Row]("features")
val mappings = features.getAs[mutable.WrappedArray[Row]]("mapping")
mappings.flatMap(mappingRow => {
val deviceId = mappingRow.getAs[Long]("deviceId")
val osIds = mappingRow.getAs[mutable.WrappedArray[Long]]("osId")
osIds.map(osId => {
val devise_os = deviceId + "_" + osId
Event(productId, familyId.toString, productTypeId, key, productName, devise_os, Map("target" -> devise_os))
})
})
} else {
Seq(Event(productId, familyId.toString, productTypeId, key, productName, "default_defice_os", Map("target" -> "default_defice_os")))
}
})
}
)
finalDataset.foreach(e => println(e))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,999795_100,Map(target -> 999795_100))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,999795_110,Map(target -> 999795_110))
// Event(3MFETP501,2010,1004678,3MFETP501:890479804,MFE Total Protection 2021 Family Pack,987875_101,Map(target -> 987875_101))
// Event(3MFETP501,2011,1004679,3MFETP501:890479805,Product name,default_defice_os,Map(target -> default_defice_os))
As this is under a Row of DataFrame, returning Event case class , converts into DataSet.Issue here is for one condition ,i am getting List[Event] and rest type , i am getting only Event class
FYI :This is not an answer. But my further attempt to solve.
if (familyId == 2010 )
{
val a: Option[List[String]] = flatten the deviceId,osId ..
a.get.map(i=>{
val key: String = methodToCombinedeviceIdAndosId
val featureMapping: mutable.Map[String, String] = getfeatureMapForInvidualKey
Event(productId,productTypeId,familyId,key,productName,device_os,feature) ---> This is returning List[Event]
})
}
else{
Event(productId,productTypeId,familyId,key,productName,device_os,feature) --> This is returning Event
}

Scala JSON If key matches value return string

I have the JSon response as given below.
If metadata's Organic=true then label='true-Organic', else label='non-Organic'
in the end => return List or Map[modelId,label]
import net.liftweb.json.{DefaultFormats, _}
object test1 extends App {
val json_response =
"""{
"requestId": "91ee60d5f1b45e#316",
"error": null,
"errorMessages": [
],
"entries": [
{
"modelId":"RT001",
"sku": "SKU-ASC001",
"store": "New Jersey",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-01-22T01:25Z",
"organic": "true"
}
},
{
"modelId":"RT002",
"sku": "SKU-ASC002",
"store": "livingstone",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-10-03T01:25Z",
"organic": "false"
}
}
] }"""
tried like this :
val json = parse(json_response)
implicit val formats = DefaultFormats
var map = Map[String, String]()
case class Sales(modelId: String, sku: String, store: String, ttlInSeconds: Int, metadata:
Map[String, String])
case class Response(entries: List[Sales])
val response = json.extract[Response]
After this, not sure how to proceed.
This is a straightforward map operation on the entries field:
response.entries.map{ e =>
e.modelId ->
if (e.metadata.get("organic").contains("true")) {
"true-Organic"
} else {
"non-Organic"
}
}
This will return List[(String, String)], but you can call toMap to turn this into a Map if required.

creating settings of index using elastic4s

I have some class:
class SomeClass(val client: ElasticClient, val config: Config, val configName: String)(implicit val ec: ExecutionContext)
extends ElasticSearchRepositoryWrapper[AnotherClass]{
override def mapping: Option[MappingDefinition] = Some(
properties(
KeywordField("id"),
TextField("name").fielddata(true).analyzer("ngram_analyzer"),
KeywordField("lang"),
BasicField("order", "long"),
...
)
)
I'm creating an index with repository.createIndexIfNotExists() using this mapping.
Now I must create ngram_analyzer in my index settings:
"settings": {
"index": {
"analysis": {
"analyzer": {
"ngram_analyzer": {
"filter": [
"lowercase"
],
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"token_chars": [
"letter",
"digit"
],
"min_gram": "3",
"type": "ngram",
"max_gram": "3"
}
}
}
}
How can I do that using elastic4s?
OK. A lot of functions of createIndexIfNotExists() were deprecated. So, I used CreateIndexRequest where I put my analyzer:
CreateIndexRequest(repository.indexName, analysis = Option(ngramAnalyzer), mapping = repository.mapping)
.shards(repository.shards)
.replicas(repository.replicas)
And I initialized my analyzer like this:
val ngramAnalyzer = Analysis(
List(CustomAnalyzer(
name = "ngram_analyzer",
tokenizer = "ngram",
charFilters = Nil,
tokenFilters = List("lowercase")
))
)