I'm trying to ingest data of my Google Ads account into a Cloud Data Fusion pipeline, but I just see that are only available 12 sources (BigQuery, Amazon S3, File, Excel, Kafka Consumer, etc)
Does anybody know if there are a way to connect directly via API? Or need I a paying solution as extractor of the data?
Many thanks!
Are you looking to ingest data from Analytics 360? https://marketingplatform.google.com/about/analytics-360/
Cloud Data Fusion does not have this connector but we will have this available in the future.
An update, now you can go to HUB on the top right corner and choose your data source such as Google ads or Google analyticsenter image description here
Related
I am getting near-real time data from 100s of customers. I need to store this data in Google Cloud Storage buckets created for each customer i.e. /gcs/customer_id/yy/mm/day/hhhh/
My data is in Avro. I guess I can use Pub/Sub to Avro Files on Cloud Storage template.
However, I'm not sure if Google Pub/Sub can accept data from multiple customers.
Appreciate any help here, thanks!
The template is quite simple: it takes all the data of PubSub and store them in an avro file on GCS.
However, it's a good starting point and you can make evolutions on that base to add a split per customer, and the file path that you want.
You can find the template in Java format on GitHub
Is there any way to search text in json files that Google vision api created from pdf.
searching of text should be happen over Google cloud storage only
Google Cloud Storage is an Object based storage solution that does not provide processing features. In order to perform any process job over the Cloud Storage data you would need a computing/processing solution, and I’d opt for a serverless option such as Cloud Functions.
I’ve found at the Cloud Functions Docs a sample application that integrates several APIs with Cloud Functions and Cloud Storage, I think you can use it as a guideline to develop your own setup.
Once you have the mentioned setup you could apply a regex implementation to search for the desired data, how to implement it will depend on the runtime, libraries and technologies that you choose to use.
I'm creating a pipeline in Google Data Fusion that allows me to export my bing-ads data into Bigquery using my bing-ads developer token. I couldn't find any data sources that should be added to my pipeline in data fusion. Is fetching data from API calls even supported on Google Data Fusion and if it is, how can it be done?
HTTP based sources for Cloud Data Fusion are currently in development and will be released by Q3. Could you elaborate on your use case a little more, so we can make sure that your requirements will be covered by those plugins? For example, are you looking to build a batch or real-time pipeline?
In the meantime, you have the following two, more immediate options/workarounds:
If you are ok with storing the data in a staging area in GCS before loading it into BigQuery, you can use the HTTPToHDFS plugin that is available in the Hub. Use a path that starts with gs:///path/to/file
Alternatively, we also welcome contributions, so you can also build the plugin using the Cloud Data Fusion APIs. We are happy to guide you, and can point you to documentation and samples.
We are looking for any open source tool to get the analytics data from Recurly to grafana. Is there any such tool please suggest us.
Regards,
Sreenath
At this time, there is not a way to export the data feeding Recurly Analytics directly. Many merchants consume webhooks (https://docs.recurly.com/docs/webhooks) and/or Automated Exports (https://docs.recurly.com/docs#section-automated-exports) in order to build their own external analytics.
I am building a application where I am trying to use IBM Watson question & answer API.
Currently I see only corpus for Healthcare and Travel, but I would like to ingest Custom dataset suiting my needs. Can anyone please point me to right direction or exact API which does that or IBM already built explorer which I can use to upload the data files directly.
Thanks for the help
At this time you can not ingest corpus data into IBM Watson. That is coming in the future.