I have an external REST based API that I need to create a connection to in order to retrieve data on a regularly scheduled basis (for BI purposes). This API is fairly robust, and supports around 60 distinct endpoints. Also, this same API is used to access information across multiple client sub-domains (e.g. client1.apisource.com, client2.apisource.com, client3.apisource.com, etc.). In other words, the API endpoints are the same for each client subdomain.
So what I'm trying to figure out is whether it's possible to create a single ADF that contains a complete set of pipeline actions for each endpoint THAT uses a "dynamic" URL based on the client subdomains? In other words...what I'm trying to see if its possible to create a single ADF that can manage a dynamic list of base URLs.
I tried to parameterize the HTTP and REST connections, but this doesn't appear to Is this possible yet. Any thoughts? Thanks!
Here is an example of a Web Activity to call a REST API using parameters and expressions. The URL can be an expression like:
#concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')
Related
I have to send data to a rest API via data factory.
I get batches of records from a database, send them in the body of the REST API call, which provides a response that informs the action performed on each record.
I created a linked service to the base API URL, and this linked service does the authentication to the API.
My question is how I use this linked service along with a web activity in a pipeline?
The web activity requires me to enter a full URL, which feels redundant as the base URL is already in the linked service.
The web activity does let me add multiple linked services but I'm unsure why it allows multiple linked services and how this is supposed to work.
I would appreciate expertise regarding how the web activity works with a linked service.
Thanks!
Can the namespace preferences and program preferences be set via REST API calls? If yes, what is the syntax for it?
Generally in Cloud Data Fusion, when we intend to perform the action on GCP side, like create/delete/restart etc. instance, it's feasible to use domestic Google Cloud API, giving the opportunity to interact with a service endpoint via JSON/HTTP calls interface as described in Google Cloud API design document.
Dedicated to Data Fusion you can follow the Cloud Data Fusion REST API reference document, nicely explaining the methods for composing REST API HTTP calls to manage Data Fusion instances, moreover every method description from the documentation contains Google API Explorer sub-panel, to get handy experience building JSON request on a live data.
Said above, I assume your initial question is related more to CDAP REST API, as it includes the methods for pure CDAP instance metadata/namespaces/application configuration.
From the user perspective your workflow might be the following:
Identify the CDAP API endpoint as explained in this guideline;
Compose an HTTP PUT/GET request relevant to Data Fusion
Namespace/Metadata/Preferences/Configuration
object via CDAP RESTful API.
Yes of course! You have two methods.
The first method is creating it from the platform. Follow the steps below:
Open your data fusion instance
Go to System Admin => Configuration => Make HTTP calls
To create a namespace, submit an HTTP PUT request:
PUT /v3/namespaces/<namespace-id>
Link of CDAP: CDAP
The second method is using terraform.
I have written some reports that contain data I'd like to programmatically access via API from an external system. I can't find any documentation or sample code to do so via any one of the 3 APIs. Has anyone used one of Archer's APIs to pull data from a report externally?
Unfortunately, you cannot pull reports via the REST API (yet), you'll have to use the Web Services API via the SearchRecordsbyReport method.
My goal is to create a REST API Integration from Salesforce to SAP application.
SUCCESS Through Chrome APP
1. All I need to do is retrieve values from sap application through the REST API. When I tried to use the Chrome APP 'Advanced Rest Client' and have passed the appropriate URL and Content with POST method I was able to retrieve the values from local server database.
For EG : If I pass request 92126 then I was able to get response 'SAN DIEGO' which is correct.
Here is the link (https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo?hl=en-US) for Advanced REST Client.
PROBLEM from Salesforce :
I had created a remotesite setting
When I created this REST class in SAlesforce and tried invoking the End Point then it's throwing this error.
System.HttpResponse[Status=Service Unavailable, StatusCode=503]
As the web api url which is provided to us is in local sql server i.e hosted in private, as we know in Salesforce for making callouts the URLs must be in public. But the URL is in private only for the security reasons not hosted in public. We should achieve it, any way is there to achieve it? What change should be done in Salesforce or server to communicate to each other, and allows to make the callout?
It is most likely that you endpoint does not allow access from outside some ip range which you indicated by saying it's not public. Salesforce is a SaaS application hosted outside the domain that your service is on. In order for Salesforce to access that endpoint resource you need to whitelist Salesforce IP ranges, which can be found here.
Whitelisting allows Salesforce to access the resource. The only caveat is that because Salesforce is multi-tenant it means that any instance of Salesforce on the range that you whitelist would have access to your endpoint. If this is not ok, you might want to add some sort of header or sign the request to the call to that identifies your Salesforce instance uniquely from any other instance to validate that the call originated from your Salesforce org.
(I am linking to the article instead of pasting the IP ranges here because these may change in the future).
I do have some data stored in my Real-Time Firebase database. I am willing to expose some of this data via a REST API to my B2B customers.
I know that Firebase is itself a REST API but its authentication mechanisms don't fit my needs. I am willing my customers to access the API with a simple API Key passed in the HTTP request headers.
To summarize, I need an API layer sitting on top of my Firebase real-time database with the following properties:
Basic Authentication via an API key passed in the HTTP request headers
Some custom logic that makes sure customers respect the API limits (maximum requests per day for example)
The only thing I can think of is implementing this layer in AWS lambda but that also sounds a bit off. From the lambda, I would have to access my Firebase database and serve that data. That seems too many network requests; something native to Firebase would be great.
Thanks,
Guven.
Why not have a simple API which provides them an Oauth token for the original firebase REST API if they have the correct Api Key
It'll be more secure as only you'll be able to make the tokens as only you'll have the service account private key. Also saves you the headache of making a whole REST API. Also the Oauth tokens expire relatively quickly so it's less of a risk than a normal key that you furnish
I personally have created my own Servlets where a user posts their data if they are authenticated using an id pass combo.
In the Servlets i use the default REST API provided by Firebase with the Oauth generated in my servlet. This way, i can have the DB security rules set to false for all writes from any client api. And the REST API and their admin sdk on my server ignore the security rules by default.
After some research, I have decided that AWS is the best platform such API related features.
Gateway API lets you setup your API interface in a matter of seconds
DynamoDB stores your API data; you can easily populate the data here
AWS Lambda lets you write the integration code between Gateway API and DynamoDB
On top of these, the platform offers these features out of the box:
Creation & handling and verification of API keys for authentication
Usage plans to make sure that API consumers don't exceed your API usage limits
Most of what I was looking for is offered in these AWS services.