Importing data as dynamically as possible - scala

Question

Im looking for some mini pattern:
The program should be able to support various formats as Input and then apply a transformation and in the last step load them into a database.
Its main purpose is to provide test data.
My initial idea was to "glue" different components together like this:
We have an extractor that extracts from a generic datasource [A] to an iterator of [B]
and then a transformator that maps [B] to [C] and finally a step that loads [C] into a database. I'm sure there must be a better way of approaching this. Is there a better , possibly more generic way of achieving this?
trait Importer[A, B, C] {
val extractor: Extractor[A, B]
val transformer: Transformator[B, C]
val loader: Loader[C]
/**
* this is the method call for chaining all events together
*/
def importAndTransformData(dataSource: A): Unit =
{
/**
* extraction step
*/
val output = extractor.extract(dataSource: A)
/**
* conversion method
*/
val transformed = output map (transformer.transform(_))
/**
* loading step
*/
transformed.foreach(loader.load(_))
}
}
with best regards,
Stefan

Answer 1

One common approach used in Scala is self-typing (especially as used in the Cake Pattern). In your case that would look something like:
trait Importer[A, B, C] {
self: Extractor[A, B] with Transformator[B, C] with Loader[C] =>
/**
* this is the method call for chaining all events together
*/
def importAndTransformData(dataSource: A): Unit =
{
/**
* extraction step
*/
val output = extract(dataSource: A)
/**
* conversion method
*/
val transformed = output map (transform(_))
/**
* loading step
*/
transformed.foreach(load(_))
}
}
You can then build your Importer with code such as:
val importer = new Importer with FooExtractor with FooBarTransformer with BarLoader {}
or
val testImporter = Importer with MockExtractor with TestTransformer with MockLoader {}
or similar for your test cases.