how to automate druid batch ingestion job? - druid

Question

I’m setting up Druid on an EMR cluster and successfully ingesting parquet files from an gcs bucket using Batch Ingestion.
Now, I wanted to automate this ingestion process to run everyday for a different file (in the same s3 bucket) with the same configuration. Is this possible? Or would I need to manually do it everyday.

Answer 1

It is not possible to create a job which is scheduled daily within druid. However, you can create your task and submit it daily using (for example) a cronjob.
Sending a query to druid is nothing more then executing an HTTP request. So if you make sure that the files are located at the same location every day, you should be able to automate this very easily.
curl -X 'POST' -H 'Content-Type:application/json' -d #/path/to/your/task.json https://url-to-druid.com/druid/indexer/v1/task

Answer 2

Working example for ingestion.
add the below in a php file and schedule the file to call the file every X hours
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'https://<druid_public_url>/druid/indexer/v1/task',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_POSTFIELDS =>file_get_contents($json_file_path),
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
?>