spring batch jpa with many to one relation - spring-data-jpa

I'm trying to import csv file to populate my database.
City {
String name;
#ManyToOne
Country country;
String State;
double area;
}
Country {
String name;
String code;
}
Here is a csv record example:
city_name state country_name country_code area garbage_data
Toronto Ontario Canada CA 205.3 .....
I use spring data JPA to handle database transactions.
Here is my spring batch code:
#Bean
public FlatFileItemReader<City> reader() {
FlatFileItemReader<City> reader = new FlatFileItemReader<>();
reader.setResource(new ClassPathResource("Hotel-Cities.csv"));
reader.setLineMapper(new DefaultLineMapper<City>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setIncludedFields(new int[]{0,1,2,3,4});
}
});
**//what to do here to map city that includes country?**
setFieldSetMapper(new BeanWrapperFieldSetMapper<City>() {
{
setTargetType(City.class);
}
});
}
});
return reader;
}
Since city has many-to-one relationship with country, where should I do something like dao.save(country);city.set(country); dao.save(city);
From what I learned, reader reads csv columns and map value to object, writer writes object to database. I still don't know how to resolve entity reference.
PS. I get type mismatch error when convert string to double, how to resolve it?

Related

Query dynamic schema using a JSON object - GraphQL .NET 6 PostgreSQL and HotChocolate

I need to develop a graphql server to query data directly from a JSON object. this JSON object is stored in a postgres database table as below.
This value field can be any JSON object. I don't have control over that. it's directly coming from a SharePoint server. I need to query this JSON object dynamically using graphql.
What I want here is to query JSON object and get only what I need instead of getting all the JSON data. for example like below
query {
allBookings {
id,
listId
widget {
text {
name
style
onMouseUp
}
}
}
}
currently, my result is this.
Technologies I am using
.NET 6
Postgresql
HotChocolate
this is my code so far.
[Table("bookings")]
public class Booking
{
[Column(name: "id")]
public int Id { get; set; }
[Column(name: "list_id")]
public Guid ListId { get; set; }
[Column(name: "value", TypeName = "jsonb")]
[GraphQLType(typeof(AnyType))]
public string Value { get; set; }
}
public class BookingType : ObjectType<Booking>
{
private readonly IDbContextFactory<DemoDBContext> _factory;
public BookingType(IDbContextFactory<DemoDBContext> factory)
{
_factory = factory;
}
[Obsolete]
protected override void Configure(IObjectTypeDescriptor<Booking> descriptor)
{
descriptor.Field(t => t.Id)
.Type<NonNullType<IntType>>();
descriptor.Field(t => t.ListId)
.Type<NonNullType<UuidType>>();
descriptor.Field(t => t.Value)
.Type<AnyType>()
.Resolver(context =>
{
var db = _factory.CreateDbContext();
var value = context.Parent<Booking>().Value;
return value;
});
}
}
public class Query
{
private readonly IDbContextFactory<DemoDBContext> _factory;
public Query(IDbContextFactory<DemoDBContext> factory)
{
_factory = factory;
}
public async Task<IEnumerable<Booking>> GetAllBookings()
{
using var context = await _factory.CreateDbContextAsync();
var bookings = await context.Bookings.ToListAsync();
return bookings;
}
public async Task<Booking> GetBooking(Guid id)
{
using var context = await _factory.CreateDbContextAsync();
var booking = await context.Bookings.Where(x => x.ListId == id).FirstOrDefaultAsync();
return booking;
}
}
I've tried different methods but I don't have a clear idea to implement this kind of behavior or even is this possible to do.
if there's a better way of doing this please suggest me. Thanks all.
GraphQL will automatically filter out fields that are either not requested or not part of a model.
If you define your types as:
type Booking {
id: ID!
listID: String
value: Widget
}
type Widget {
text: SubWidget
}
type SubWidget {
name: String
style: String
onMouseUp: String
}
query allBookings: [Booking]
In your resolver you're going to return an array of JSON objects corresponding to each Booking. If that JSON has fields that are not part of your type definitions they will not be returned. If some of the fields you ask for are missing then they will come back as undefined unless you make them non-nullable (ex: name: String!)
So you're most of the way there.

Postgres to bring list all of table fields for particular Employee row?

Taking a reference from link: Postgres to fetch the list having comma separated values, I want to write a query which somhow brings Employee email Table fields as a list for a particular Empployee. This is needed for Spring Batch to Simply match it from the Resultset and create a POJO/Model class like List emails for Employee class?
Can this be possible ?
select c.*, ce.*, string_agg(ce.email, ',') as emails
from root.employee c
full outer join root.employee_email ce
on c.employee_id = ce.employee_id
group by
c.employee_id, ce.employee_email_id
order by
c.employee_id
limit 1000
offset 0;
Your problem is a common one in the batch processing realm and with Spring Batch it is called "Driving Query Based ItemReaders", you can find more about that in here.
Basically you retrieve the Contacts in your reader, and in your processor you add the list of Emails to them.
#Bean(destroyMethod = "")
public JdbcCursorItemReader<Employee> employeeReader(DataSource dataSource) {
JdbcCursorItemReader<Employee> ItemReader = new JdbcCursorItemReader<>();
ItemReader.setDataSource(dataSource);
ItemReader.setSql("SELECT * FROM employee.employee C ");
ItemReader.setRowMapper(new EmployeeRowMapper());
return ItemReader;
}
#Bean
public ItemProcessor<Employee, Employee> settlementHeaderProcessor(JdbcTemplate jdbcTemplate){
return item -> {
root.EMPLOYEE_EMAIL CE WHERE ce.employee_id = ? ",
new Object[]{item.getId()},
new RowMapper<String>() {
#Override
public String mapRow(ResultSet resultSet, int i) throws SQLException {
return resultSet.getString("EMAIL");
}
});
item.setEmails(emails);
return item;
};
}
PS : this could have some performance issues if you have lots of contacts, because for each contact Item you will hit the database to retrieve Emails.
There is another optimized way, by creating a custom reader that will return a List of Contacts (For example 1000 by 1000), and a processor that will enrich them with their emails. This way you will hit the database again for each 1000 Contact Item.
In your reader your retrieve a list of unique Employees page per page (Say your page is 1000 long).
And in your processor for the 1000 employees you retrieve all their emails in one query.
Then for each employee you set the emails retrieved in the last query.
An example might like the following:
public interface EmployeeRepository extends PagingAndSortingRepository<Employee, Integer> {
}
#Getter
class EmployeeVO {
private Long employeeId;
private String email;
EmployeeVO(Long employeeId, String email) {
this.employeeId= employeeId;
this.email = email;
}
}
public class EmployeeListReader implements ItemReader<List<Employee>> {
private final static int PAGE_SIZE = 1000;
private final EmployeeRepository employeeRepository;
private int page = 0;
public EmployeeListReader(EmployeeRepository employeeRepository) {
this.employeeRepository = employeeRepository;
}
public List<Employee> read() throws Exception {
Page<Employee> employees = employeeRepository.findAll(PageRequest.of(page, PAGE_SIZE));
page++;
return employees.getContent();
}
}
#Bean
EmployeeListReader reader(){
return new EmployeeListReader(this.employeeRepository);
}
#Bean
public ItemProcessor<List<Employee>, List<Employee>> settlementHeaderProcessor(NamedParameterJdbcTemplate namedParameterJdbcTemplate) {
return item -> {
List<Long> employeesIds = item.stream().map(Employee::getId).collect(Collectors.toList());
SqlParameterSource parameters = new MapSqlParameterSource("ids", employeesIds);
List<ContactVO> emails = namedParameterJdbcTemplate
.query("SELECT CE.employeeId, CE.EMAIL FROM employee_EMAIL CE WHERE ce.contact_id IN (:ids) ",
parameters,
new RowMapper<ContactVO>() {
#Override
public ContactVO mapRow(ResultSet resultSet, int i) throws SQLException {
return new ContactVO(
resultSet.getLong("EMPLOYEE_ID"),
resultSet.getString("EMAIL"));
}
});
Map<Long, List<ContactVO>> emailsByContactId = emails.stream().collect(Collectors.groupingBy(ContactVO::getContactId));
List<Employee> newEmployeesWithEmails = Collections.unmodifiableList(item);
newEmployeesWithEmails.forEach(employee -> {
employee.setEmails(emailsByContactId.get(employee.getId()).stream().map(ContactVO::getEmail).collect(Collectors.toList()));
});
return newEmployeesWithEmails;
};
}
Hope this helps

Entity framework relations

im trying to make an app and got a slight problem. my structure looks like this:
public class BaseModel
{
[Key]
private int _Id;
public int Id {
get { return _Id; }
set { _Id = value; }
}
}
public class SupplierModel : BaseModel
{
[ForeginKey("CountryCode")] // This should map to say "se" or "no" or whatever in the CountryModel table
public virtual CountryModel Country;
}
public class CountryModel : BaseModel
{
private string _CountryCode;
[Key] // This should be another key in the table to get the actual country.
public string CountryCode {
get { return _CountryCode; }
set { _CountryCode = value; }
}
private string _CountryName;
public string CountryName {
get { return _CountryName; }
set { _CountryName = value; }
}
}
Now i want SupplierModel to link to CountryModel (Works fine by the Id) but i want it to be the country code to be the relationship not the Id between the Entities.
So accessing CountryModel.Country should map to the CountryModel table and pull out the one that matches the country model.
Hope i didnt mess it up totaly for you, hard to explain when i do not fully understand Entity framework and database relations .. trying to learn =)

How can i design a Model of a document contains more than one collection in Mongodb

I am novice in mongodb.
I have a mongodb and i have a table / document(Ex: sampleTable).
I know how to retrieve data from this single table.
I used below code to retrieve data and working fine.
#Document(collection="sampleTable")
public class sampleMapping {
private String id;
private String strClassName;
public sampleMapping () {}
public sampleMapping (String strClassName) {
super();
this.strClassName = strClassName;
}
public String getStrClassName() {
return strClassName;
}
public void setStrClassName(String strClassName) {
this.strClassName = strClassName;
}
}
But now, i wanted to retrieve data from more than one table. How can i do that?
Say, i need to retrieve table1 and table2.
How can i update the above code to get data from multiple collections?

Failed to decode property of type BasicDBList with PojoCodecProvider in mongo db

I am tying to load an instance of the class "DataTable" from a mongo database by using the default codec registry (MongoClient.getDefaultCodecRegistry()) and the builder provided by the PojoCodecProvider. I have registered the DataTable class in the codec provider and the object is properly mapped from the database when the records field is null. Nevertheless, I get an error when the records property contains data. Furthermore, I need to have the records field defined as a list of objects with arbitrary attributes. Is it possible to use the default PojoCodecProvider for this purpose? Is there any other alternative?
import com.mongodb.BasicDBList;
import org.bson.types.ObjectId;
import java.util.List;
public class DataTable {
private ObjectId id;
private List<String> fields;
private BasicDBList records;
public ObjectId getId() {
return id;
}
public void setId(ObjectId id) {
this.id = id;
}
public List<String> getFields() {
return fields;
}
public void setFields(List<String> fields) {
this.fields = fields;
}
public BasicDBList getRecords() {
return records;
}
public void setRecords(BasicDBList records) {
this.records = records;
}
}
The exception that I get when load an instance of the DataTable class is the following.
2018-03-21T16:32:04,526 [http-bio-8081-exec-4] ERROR ...service.controllers.BaseController - Failed to decode 'records'. Unable to set value for property 'records' in DataTable
org.bson.codecs.configuration.CodecConfigurationException: Failed to decode 'records'. Unable to set value for property 'records' in DataTable
at org.bson.codecs.pojo.PojoCodecImpl.decodePropertyModel(PojoCodecImpl.java:192) ~[bson-3.6.3.jar:?]
at org.bson.codecs.pojo.PojoCodecImpl.decodeProperties(PojoCodecImpl.java:168) ~[bson-3.6.3.jar:?]
at org.bson.codecs.pojo.PojoCodecImpl.decode(PojoCodecImpl.java:122) ~[bson-3.6.3.jar:?]
at org.bson.codecs.pojo.PojoCodecImpl.decode(PojoCodecImpl.java:126) ~[bson-3.6.3.jar:?]
I get this exception when I try to load an item with the following code
DataTable item = collection.find(eq(new ObjectId(id))).first();
Well, one alternative you can use is Jackson Serialization.
I think something like this would suit you just fine
Document document = collection
.find(eq(new ObjectId(id)))
.first();
String json = document.toJson();
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
DataTable dataTable = mapper.readValue(json, DataTable.class);
See this question converting Document objects in MongoDB 3 to POJOS for reference