Use TableProvider to generate a table and run an SQL query in Apache Beam - apache-beam

Question

I want to generate an unbounded collection of rows and run an SQL query on it using the Apache Beam Calcite SQL dialect and the Apache Flink runner. Based on the source code and documentation of Apache Beam, one can do something like this using a table provider: GenerateSequenceTableProvider. But I don't understand how to use it outside of the Beam SQL CLI. I'd like to use it in my regular Java code.
I was trying to do something like this:
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);
GenerateSequenceTableProvider tableProvider = new GenerateSequenceTableProvider();
tableProvider.createTable(Table.builder()
.name("sequence")
.schema(Schema.of(Schema.Field.of("sequence", Schema.FieldType.INT64), Schema.Field.of("event_time", Schema.FieldType.DATETIME)))
.type(tableProvider.getTableType())
.build()
);
PCollection<Row> res = PCollectionTuple.empty(pipeline).apply(SqlTransform.query("select * from sequenceSchema.sequence limit 5").withTableProvider("sequenceSchema", tableProvider));
pipeline.run().waitUntilFinish();
But I'm getting Object 'sequence' not found within 'sequenceSchema' errors, so I guess I'm not actually creating the table. So how do I create the table? If I understand correctly, the values should be provided automatically by the table provider.
Basically, how to use Beam SQL table providers if I want to execute queries on tables that these providers are supposed (I think?) to generate?

Answer 1

The TableProvider interface is a bit difficult to work with directly. The problem you're running into is that the GenerateSquenceTableProvider, like many other TableProviders, doesn't have any way to store table metadata on its own. So calling its createTable method is actually a no-op! What you'll want to do is wrap it in an InMemoryMetaStore, something like this:
GenerateSequenceTableProvider tableProvider = new GenerateSequenceTableProvider();
InMemoryMetaStore metaStore = new InMemoryMetaStore();
metaStore.registerProvider(tableProvider);
metaStore.createTable(Table.builder()
.name("sequence")
.schema(Schema.of(Schema.Field.of("sequence", Schema.FieldType.INT64), Schema.Field.of("event_time", Schema.FieldType.DATETIME)))
.type(tableProvider.getTableType())
.build()
);
PCollection<Row> res = PCollectionTuple.empty(pipeline)
.apply(SqlTransform.query("select * from sequenceSchema.sequence limit 5")
.withTableProvider("sequenceSchema", metaStore));
(Note I haven't tested this, but I think something like it should work)
As robertwb pointed out, another option would be to just avoid the TableProvider interface and use GenerateSequence directly. You'd just need to make sure that your PCollection has a schema. Then you could process it with SqlTransform, like this:
pc.apply(SqlTransform.query("select * from PCOLLECTION limit 5"))

Answer 2

If you can't get TableProviders to work, you could read this as an ordinary PCollection and then apply a SqlTransform to the result.