I'm having a go at creating my first data fusion pipeline.
The data is going from Google Cloud Storage csv file to Big Query.
I have created the pipeline and carried out a preview run which was successful but after deployment trying to run resulted in error.
I pretty much accepted all the default settings apart from obviously configuring my source and destination.
Error from Log ...
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403
Forbidden
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "Required 'compute.firewalls.list' permission for
'projects/xxxxxxxxxxx'",
"reason" : "forbidden"
} ],
"message" : "Required 'compute.firewalls.list' permission for
'projects/xxxxxxxxxx'"
}
After deployment run fails
Do note that as a part of creating an instance, you must set up permissions [0]. The role "Cloud Data Fusion API Service Agent" must be granted to the exact service account, as specified in that document, which has an email address that begins with "cloud-datafusion-management-sa#...".
Doing so should resolve your issue.
[0] : https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions
Related
thanks in advance.
I am facing an issue while adding gate (query work item) to control the azure pipeline flow.
Error message i am getting from logs :)
==============================================================================
Task : Query Work Items
Description : Execute a work item query and check the number of items returned
Version : 0.0.16
Author : Microsoft Corporation
Help URL : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/work-item-query
==============================================================================
Response Code: 0
Response: An error was encountered while processing request. Exception: {"$id":"*","innerException":null,"message":"TF*****: The query 1****-****-****-****-******* does not exist, or you do not have permission to read it.","typeName":"Microsoft.TeamFoundation.WorkItemTracking.Server.QueryItems.QueryItemNotFoundException, Microsoft.TeamFoundation.WorkItemTracking.Server","typeKey":"QueryItemNotFoundException","errorCode":****,"eventId":****}
Exception Message: The remote server returned an error: (404) Not Found. (type WebException)
Evaluation of expression 'xor(and(or(eq(root['queryType'], 'oneHop'), eq(root['queryType'], 'tree')), and(le(count(root['workItemRelations']), 5), ge(count(root['workItemRelations']), 0))), and(eq(root['queryType'], 'flat'), and(le(count(root['workItems']), 5), ge(count(root['workItems']), 0))))' failed.
Exception Message: Error parsing API response into a JSON object. (type DistributedTaskException)
What I have tried already :)
I have added user to all the groups like , build administrator, contributer , project administrators.
Already tried updating query permission, but no luck.
I am unable to find the issue.
The issue is the account {project name} Build Service ({Org name}) does not have enough permission to access the query.
Steps:
Open Queries->select ... on your query and add that account {project name} Build Service ({Org name}) with READ access.
Note: We should provide the permission for the account {project name} Build Service ({Org name}) to read under Shared Queries page and select query permission page.
Result:
When granting permission, search for the project name, as it does not appear in the initial list, only after performing the search. The project will only appear in the list after you perform the search.
enter image description here
I have 2 requirements:
1:I have a clusterID. I need to start the cluster from a "Wb Activity" in ADF. The activity parameters look like this:
url:https://XXXX..azuredatabricks.net/api/2.0/clusters/start
body: {"cluster_id":"0311-004310-cars577"}
Authentication: Azure Key Vault Client Certificate
Upon running this activity I am encountering with below error:
"errorCode": "2108",
"message": "Error calling the endpoint
'https://xxxxx.azuredatabricks.net/api/2.0/clusters/start'. Response status code: ''. More
details:Exception message: 'Cannot find the requested object.\r\n'.\r\nNo response from the
endpoint. Possible causes: network connectivity, DNS failure, server certificate validation or
timeout.",
"failureType": "UserError",
"target": "GetADBToken",
"GetADBToken" is my activity name.
The above security mechanism is working for other Databricks related activity such a running jar which is already installed on my databricks cluster.
2: I want to create a new cluster with the below settings:
url:https://XXXX..azuredatabricks.net/api/2.0/clusters/create
body:{
"cluster_name": "my-cluster",
"spark_version": "5.3.x-scala2.11",
"node_type_id": "i3.xlarge",
"spark_conf": {
"spark.speculation": true
},
"num_workers": 2
}
Upon calling this api, if a cluster creation is successful I would like to capture the cluster id in the next activity.
So what would be the output of the above activity and how can I access them in an immediate ADF activity?
For #2 ) Can you please check if you change the version
"spark_version": "5.3.x-scala2.11"
to
"spark_version": "6.4.x-scala2.11"
if that helps
I want to install Apero Cas Management (verison 6.0) and integrate it with Cas Server (version 6.0).
I have installed following these step:
Step 1: I installed Cas Server
I checked it with REST API. It worked.
My server stays at http://203.162.141.7:8080
And this is configuration of my Cas server. I put this config at /etc/cas/config. Here is my file cas.properties file
cas.server.name=http://203.162.141.7:8080
cas.server.prefix=${cas.server.name}/cas
logging.config: file:/etc/cas/config/log4j2.xml
server.port=8080
server.ssl.enabled=false
cas.serviceRegistry.initFromJson=false
cas.serviceRegistry.json.location=file:/etc/cas/services-repo
cas.authn.oauth.grants.resourceOwner.requireServiceHeader=true
cas.authn.oauth.userProfileViewType=NESTED
cas.authn.policy.requiredHandlerAuthenticationPolicyEnabled=false
cas.authn.attributeRepository.stub.attributes.email=casuser#example.org
#REST API JSON
cas.rest.attributeName=email
cas.rest.attributeValue=.+example.*
Step 2: I installed Cas-management-overlay
I put my cas-management-overlay's config file a /etc/cas/config too. Here is my management.properties file
cas.server.name=http://203.162.141.7:8080
cas.server.prefix=${cas.server.name}/cas
mgmt.serverName=http://203.162.141.7:8088
mgmt.adminRoles[0]=ROLE_ADMIN
mgmt.userPropertiesFile=file:/etc/cas/config/users.json
server.port=8088
server.ssl.enabled=false
logging.config=file:/etc/cas/config/log4j2-management.xml
And my here is users.json file
{
"casuser" : {
"#class" : "org.apereo.cas.mgmt.authz.json.UserAuthorizationDefinition",
"roles" : [ "ROLE_ADMIN" ]
}
}
Then I run ./build.sh, and it shows me that
Finally, I access this link to open cas-management http://203.162.141.7:8088/cas-management, but the it redirects to this url http://203.162.141.7:8080/cas/login?service=http%3A%2F%2F203.162.141.7%3A8088%2Fcas-management%2F and shows this error below
I don't know where I have gone wrong.
I think since you haven't told the management webapp about the location of the service registry, it can't add itself as a registered service.
Manually add a registered service for http://203.162.141.7:8088/cas-management and you should be able to log in to the management app at that point.
Here is my answer for cas-management register file name /etc/cas/services-repo/casManagement-1.json
{
"#class" : "org.apereo.cas.services.RegexRegisteredService",
"serviceId":"^https://domain:8088/cas-management.+",
"name" : "casManagement",
"id" : 1,
"evaluationOrder" : 1,
"allowedAttributes":["cn","mail"]
}
I have project with XS project, I already shared to HANA packages but failed when show to browser, the error show:
404 - Not found
We could not find the resource you're trying to access.
It might be misspelled or currently unavailable.
My .xsaccess:
{
"exposed" : true,
"authentication" : [{"method":"Basic"}],
"cache_control" : "no-cache, no-store",
"cors" : {
"enabled" : false
}
}
.xsapp:
{}
xsprivileges:
{
"privileges" : [
{ "name" : "ProfileOwner", "description" : "Profile Ownership" }
]
}
and one question, is it possible the problem because the role user or privileges user, about authorization? How to fix this issue? thanks
The .xsapp should be a empty file with no content in it. The exposed parameter in the .xsaccess should be enough to expose your project. Make sure that all files are activated in the HANA repository.
If the error was authorization specific you would get a 503 error. If the 404 error is a XSEngine page, either your code isn't activated or the package path is incorrect.
When trying to restore a backup to a new cloud sql instance I get the following message when using curl:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalidOperation",
"message": "This operation isn\"t valid for this instance."
}
],
"code": 400,
"message": "This operation isn\"t valid for this instance."
}
}
When trying via google cloud console, after clicking 'ok' in the 'restore instance from backup' menu nothing happens.
I'll answer even thought this is a very old question, maybe useful for someone else (would have been for me).
I just had the same exact same error, my problem was that the storage capacity for the target instance was different than the one for the source instance. My source instance was accidentally deleted so this was a bit troublesome to figure out. This check list helped me https://cloud.google.com/sql/docs/postgres/backup-recovery/restore#tips-restore-different-instance