Apache Camel - Getting a list of files from FTP as a result of a GET request - rest

Question

As the title suggests I'm trying to get a list of files from an FTP directory to send as a response of a GET request.
I have current rest route implementation:
rest().get("/files")
.produces(MediaType.APPLICATION_JSON_VALUE)
.route()
.routeId("restRouteId")
.to("direct:getAllFiles");
On the other side of the direct route I have the following routes:
from("direct:getAllFiles")
.routeId("filesDirectId")
.to("controlbus:route" +
"?action=start" +
"&routeId=ftpRoute");
from([ftpurl])
.noAutoStartup()
.routeId("ftpRoute")
.aggregate(constant(true), new FileAggregationStrategy())
.completionFromBatchConsumer()
.process(filesProcessor)
.to("controlbus:route" +
"?action=stop" +
"&routeId=" + BESTANDEN_ROUTE_ID);
The issue at hand is that with this method the request does not wait for the complete process to finish, it almost instantly returns an empty response with StatusCode 200.
I've tried multiple solutions but they all fail in either of two ways: either the request gets a response even though the route hasn't finished yet OR the route gets stuck waiting for inflight exchanges at some point and waits for the 5 minute timeout to continue.
Thanks in advance for your advice and/or help!
Note: I'm working in a Spring Boot application (2.0.5) and Apache Camel (2.22.1).

Answer 1

I think the problem here is that your two routes are not connected. You are using the control bus to start the second route but it doesn't return the value back to the first route - it just completes, as you've noted.
What I think you need (I've not tested it) is something like:
from("direct:getAllFiles")
.routeId("filesDirectId")
.pollEnrich( [ftpurl], new FileAggregationStrategy() )
.process( filesProcessor );
as this will synchronously consume your ftp consumer, and do the post processing and return the values to your rest route.

Answer 2

With the help of #Screwtape's answer i managed to get it working for my specific issue. A few adjustments were needed, here is a list of what you need:
Add the option "sendEmptyMessageWhenIdle=true" to the ftp url
In the AggregationStrategy add an if (exchange == null) clause
In the clause set a property "finished" to true
Wrap the pollEnrich with a loopDoWhile that checks the finished property
In its entirety it looks something like:
from("direct:ftp")
.routeId("ftpRoute")
.loopDoWhile(!finished)
.pollEnrich("ftpurl...&sendEmptyMessageWhenIdle=true", new FileAggregationStrategy())
.choice()
.when(finished)
.process(filesProcessor)
.end()
.end();
In the AggregationStrategy the aggregate method looks something like:
#Override
public Exchange aggregate(Exchange currentExchange, Exchange newExchange) {
if (currentExchange == null)
return init(newExchange);
else {
if (newExchange == null) {
currentExchange.setProperty("finished", true);
return currentExchange;
}
return update(currentExchange, newExchange);
}
}