I am having a trouble with Power BI dataset refresh via REST API.
I get this error whenever I try to execute the dataset refresh API:
Last refresh failed: ...
There is no available gateway.
I'm testing on two accounts, and this happens only on one of them.
What is more confusing is that the storage I'm using is cloud based (Azure Data lake). So it doesn't need a gateway connection. And when I refresh the datasets manually it works.
When I get the refresh history for further investigationI get this:
serviceExceptionJson={"errorCode":"DMTS_MonikerWithUnboundDataSources"}
I could use any given help.
I have found a workaround solution, although I'm not very convinced this is the source of the problem (maybe the why is still very blurry to me)
First for more context, this is what I'm doing (using the APIs) :
1-Upload a dataset to workspace (import a pbix and skipping report, I only need the parameters and power query within it)
2- Update the parameters
3- Refresh dataset
The thing is, I have a parameter for the url of the datalake data. There are two values of the url, depending on the use case/user.
So what I did, is that I removed that parameter and made two different pbix, each one with its specific url. So then, I import a pbix depending on the use case.
I still don't understand why I get a gateway error, while I should get "error while processing data" message (seeing that it seems to be a data access issue).
Related
My first post here and I'm new to Data Fusion and I'm with low to no coding skills.
I want to get data from ZohoCRM to BigQuery. Module from ZohoCRM (e.g. accounts, contacts...) to be a separate table in BigQuery.
To connect to Zoho CRM I obtained a code, token, refresh token and everything needed as described here https://www.zoho.com/crm/developer/docs/api/v2/get-records.html. Then I ran a successful get records request as described here via Postman and it returned the records from Zoho CRM Accounts module as JSON file.
I thought it will be all fine and set the parameters in Data Fusion
DataFusion_settings_1 and DataFusion_settings_2 it validated fine. Then I previewed and ran the pipeline without deploying it. It failed with the following info from the logs logs_screenshot. I tried to manually enter a few fields in the schema when the format was JSON. I tried changing the format to csv, nether worked. I tried switching the Verify HTTPS Trust Certificates on and off. It did not help.
I'd be really thankful for some help. Thanks.
Update, 2020-12-03
I got in touch with Google Cloud Account Manager, who then took my question to their engineers and here is the info
The HTTP plugin can be used to "fetch Atom or RSS feeds regularly, or to fetch the status of an external system" it does not seems to be designed for APIs
At the moment a more suitable tool for data collected via APIs is Dataflow https://cloud.google.com/dataflow
"Google Cloud Dataflow is used as the primary ETL mechanism, extracting the data from the API Endpoints specified by the customer, which is then transformed into the required format and pushed into BigQuery, Cloud Storage and Pub/Sub."
https://www.onixnet.com/insights/gcp-101-an-introduction-to-google-cloud-platform
So in the next weeks I'll be looking at Data Flow.
Can you please attach the complete logs of the preview run? Make sure to redact any PII data. Also what is the version of CDF you are using? Is CDF instance private or public?
Thanks and Regards,
Sagar
Did you end up using Dataflow?
I am also experiencing the same issue with the HTTP plugin, but my temporary way to go around it was to use a cloud scheduler to periodically trigger a cloud function that fetches my data from the API and exports them as a JSON to GCS, which can then be accessed by Data Fusion.
My solution is of course non-ideal, so I am still looking for a way to use the Data Fusion HTTP plugin. I was able to make it work to get sample data from public API end-points, but for a reason still unknown to me I can't get it to work for my actual API.
I'm trying to display incoming streamingdata in a Pentaho dashboard. The incoming data are simple strings, which I would just like to display at the dashboard for now.
I created a kettle transformation, in which I bound a dataservice to the last step(MQTT-Producer).
Within spoon, I tested the service and it seems to work fine.
After uploading the kettle file, the service showed up in the service list (http://localhost:9090/pentaho/kettle/listServices).
Working with the dashboard editor, I use 'streaming over dataservices' from the 'DATASERVICES Queries' as my datasource.
At this point I didn't seem to have any success an was just trying out different panel options and dataservice properties.
I was following those tutorials:
https://help.pentaho.com/Documentation/8.2/Products/Data_Integration/Data_Services
https://help.pentaho.com/Documentation/8.2/Products/CTools/Create_Streaming_Service_Dashboard
What is it that I'm doing wrong?
Any help is appreciated.
cheers
update:
I changed the incoming streaming data to be two doubles.
after some more playing around, I did connect to the data service, using an external tool. I did see the expected values within the database. My dashboard, however, still shows this error message:
Error processing component (ccclinechart)
The same kind of error occurs, when I try to view the sample real time dashboard. It can't process the chartComponent. Maybe I need to reconfigure some other things?
Found the mistake.
Something went wrong with the Ports. After switching back to the default(8080) it worked just fine.
There might be another way to adjust your ports-settings to the problem, but the easiest way to deal with this sort of thing is to switch back to default settings.
Is there a way to get all activities from the Google fitness store via the REST API?
My current assumption is that other apps store their activities in sessions and I can retrieve them using Users.sessions.list. However, the information there, does not really include all the information that was stored or I would like to see: when I manually add a short run via the Fit Android app, I expect this information to be somehow accessible via the sessions API. This should at least include the information I have provided, such as distance or time.
Looking at the same information via the app or the web interface, I can see all the details I have previously entered plus the approximate number of steps and calories.
How do I get this information via the API?
I am currently mainly interested in activities of type running or jogging (8, 56-58) and would like to read the distance in addition to the time information already provided in the session.
Not sure, if this is the right way, but I get all the information I need, if I follow these steps
Find the correct session via Users.sessions.list
Query all data via Users.datasets.aggregate:
Set startTimeMillis and endTimeMillis to the values from the session in question
Set bucketBySession to group results by sessions.
I explicitly query all data sources: For every data source id I add a { "dataSourceId": <id>}to theaggregateBy` array. Not sure, if this is necessary
The resulting bucket has all information related to the session. For my use case I need to clean up overloaded data: some data sources return the distance as steps (derived) while I need the physical length in meters.
This seems to work for my Fit data with the additional cleaning, but I will need to check, if this works for other user's data too.
I am using a data factory pipeline with a custom activity (configured to run on Azure Batch) that has a data lake store input dataset and output dataset. The data lake store linked service is using service to service auth (service principal) and is working fine while being used in a Copy activity through Copy Wizard. But when used with a custom activity that tries to check if a file is present in the data lake, the activity fails with an error "Authorization is required". Upon using a Azure Blob Store as the input and output datasets, the same custom activity works fine.
Seems like an issue with Azure Batch (Compute node) not able to authorize Data Lake Store. Please help if you have solved the above mentioned problem.
I had this exact same issue about 3 weeks ago. I feel your pain!
This is a Microsoft bug!
After much trial and error and redeployments I raised a support ticket with Microsoft who confirmed that service principal authentication for data lake store currently only works with copy activities. Not with custom activities.
This is the official response I got on Monday 10th April.
The issue happen because of a bug that custom activity’s connector
schema doesn’t match the latest published connector schema. Actually,
we notice the issue on custom activity and have plan to fix & deploy
to prod in next 2 weeks.
Be aware that if you change your linked service back to use a session token etc you'll also need to redeploy your pipelines that contain the custom activities. Otherwise you'll get another error something like the following...
Access is forbidden, please check credentials and try again. Code:
'AuthenticationFailed' Message: 'Server failed to authenticate the
request. Make sure the value of Authorization header is formed
correctly including the signature.
Hope this helps.
I am using Gatling and am running into an issue implementing an integration test. The test I am running has two parts (scenarios):
The first part is a simple GET request to verify expected server responses. In this step, it returns a form's configuration ID as well as other properties of the form.
In the next step, I am hitting PUT route which updates various properties of this form's configuration for testing of this route and other functionality through the back ends of an application.
The issue with these steps is that after running the second step, if there are any forms saved under that configuration, it will clone a copy for security purposes and changes the ID. Long story short, the same ID does not persist from test to test.
My question is: Is there a way within Gatling to run SQL Queries to get values and store them and then pass them? OR is there a way to store a variable from the GET method and pass it in in the second scenario's PUT method. I know there is Gatling's .saveAs() method, but I believe the scope of this function is only scenario.
Please notify me for further clarification and details; thanks in advance.
The Google group link only shows you how to share (&cache) data between different user-sessions.
If you are looking for sharing data between "scenarios" rather than "user sessions", you should check out Peter's answer: https://stackoverflow.com/a/43061278/1395126
It's the only information I can find on the internet today that really shows you how to share data between scenarios in Gatling.
IMHO, it's super important to look out for the line of "// Set it here" comment. This is THE secret about sharing data among scenarios. Without setting session attribute first or accessing the "CreateId" directly in another scenario could not work.
This answer: https://groups.google.com/forum/#!topic/gatling/bfpKd_9d6gk can probably be used to handle your use case.