i have a hourly data migration process which copies data from appnexus to Azure using sas key. But suddenly on 6/27/2020, 3:20:11 AM its stopped and started giving error "java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details.". Could someone help me to understand what has actually gone wrong.
You have not mentioned what is that you are using to copy the data , let me assume that its some kind of SPARK based system may azure databricks etc . I were you I could have started from the source side and if all looked good could have visted the drivers setup .
Please do share more info on this and we will try to get help.
Related
I'm trying to copy files from Az Storage blob to ADX using Data factory, but I can't find a solution to do this using json datasets (not binary), so I can map schema. I would love to have a template to do this.
I have tried to follow the resolution mentioned here (Get metadata from Blob storage with "folder like structure" using Azure Data Factory pipeline), but I'm lacking some more guidance (this is my first project using ADF)
enter image description here
enter image description here
enter image description here
Now I am getting another error as shown below
I'm actually also looking for a COMPLETE guide setting this up.
Here is my overall use-case target https://learn.microsoft.com/en-us/azure/sentinel/store-logs-in-azure-data-explorer?tabs=azure-storage-azure-data-factory - but the documentation is missing the detailed steps - in step 6 Create a data pipeline with a copy activity, based on when the blob properties were last modified. This step requires an extra understanding of Azure Data Factory. For more information, see Copy activity in Azure Data Factory and Azure Synapse Analytics.
enter image description here
enter image description here
It seems as I was going in a wrong direction.
I have actually found a simple solution for setting this up. I have tried the copy data tool, and it seems to be doing what I want. So case closed :-)
From the error message you have shared, it seems like your dynamic expression for passing the childItems of your get metadata activity to ForEach activity items is causing this problem.
You might be using Items = #activity('GetMetadataActivity').output instead of #activity('GetMetadataActivity').output.childItems.
Please use Items = #activity('GetMetadataActivity').output.childItems in your ForEach activity which should help resolve your error.
Here is a video demonstration by a community volunteer where this error has been explained in detail: ADF Error Function Length expects its parameter to be array or string
I'm using Copy Data.
When there is some data error. I would export them to a blob.
But in this case, the Pipelines's status is still Succeeded. I want to set it to false. Is it possible?
When there is some data error.
It depends on what error you mentioned here.
1.If you mean it's common incompatibility or mismatch error, ADF supports built-in feature named Fault tolerance in Copy Activity which supports below 3 scenarios:
Incompatibility between the source data type and the sink native
type.
Mismatch in the number of columns between the source and the sink.
Primary key violation when writing to SQL Server/Azure SQL
Database/Azure Cosmos DB.
If you configure to log the incompatible rows, you can find the log file at this path: https://[your-blob-account].blob.core.windows.net/[path-if-configured]/[copy-activity-run-id]/[auto-generated-GUID].csv.
If you want to abort the job as soon as any error occurs,you could set as below:
Please see this case: Fault tolerance and log the incompatible rows in Azure Blob storage
2.If you are talking about your own logic for the data error,may some business logic. I'm afraid that ADF can't detect that for you, though it's also a common requirement I think. However,you could follow this case (How to control data failures in Azure Data Factory Pipelines?) to do a workaround. The main idea is using custom activity to divert the bad rows before the execution of copy activity. In custom activity, you could upload the bad rows into Azure Blob Storage with .net SDK as you want.
Update:
Since you want to log all incompatible rows and enforce the job failed at the same time, I'm afraid that it can not be implemented in the copy activity directly.
However, I came up with an idea that you could use If Condition activity after Copy Activity to judge if the output contains rowsSkipped. If so, output False,then you will know there are some skip data so that you could check them in the blob storage.
I am trying to one of my BigQuery tables but I get this error:
Errors:
Something went wrong with the table you queried. Contact the table owner for assistance. (error code: tableUnavailable)
I used to be able to run the query until couple of hour ago without a problem and I still can query another table in a different data set.
I was also running a Python application to stream data into the this table, but now on:
table.insert_data(rows)
Causes this erro:
google.cloud.exceptions.ServiceUnavailable:
503 Error encountered during execution. Retrying may solve the problem.
(POST https://www.googleapis.com/bigquery/v2/projects/#myProjectId#/datasets/#myDatasetName#/tables/#myTableName#/insertAll)
I have read similar questions here but can't find any solution to this problem.
Here is the latest error jobId
"jobId": "bquijob_6491e032_15846c06c18"
This has been going for a while now, can anybody help?
This was a hopefully rare BigQuery incident:
https://status.cloud.google.com/incident/bigquery/18022
I am working on an application with ASP.Net/MVC/EF/Breeze/Angular and it is working in my development environment. I just deployed it to my test environment and I am getting an error when I try to query breeze entity data. I don't think it's a DB connection problem, because I can query the breeze metadata successfully.
However, when I try to query for entity data, for example:
http://server/path/breeze/data/Cities
I get the following error:
{"$id":"1","$type":"System.Web.Http.HttpError, System.Web.Http","Message":"An error has occurred."}
This shows up as a 500 error in the browser network history. Is there any way to get Breeze to return a more detailed error message? I am not getting this error in development so I can't debug it there. Or any suggestions as to what I should look for if I can get metadata but not data?
Well, it turns out it was a connection string problem! I have no idea how breeze was generating metadata without a DB connection. Maybe it caches it in a local file?
Anyways, it would be nice to be able to get more detailed messages in cases like this. Since breeze returns IQueryable<T> I couldn't see any obvious way to catch the exception and send back a message myself...
I feel like this should be a lot easier than it's been on me.
copy table
from 's3://s3-us-west-1.amazonaws.com/bucketname/filename.csv'
CREDENTIALS 'aws_access_key_id=my-access;aws_secret_access_key=my-secret'
REGION 'us-west-1';
Note I added the REGION section after having a problem but did nothing.
Where I am confused though is that in the bucket properties there is only the https://path/to/the/file.csv. I can only assume that all the documentation that I have read calling for the path to start with s3://... that I would just change https to s3 like shown in my example.
However I get this error:
"Error : ERROR: S3ServiceException:
The bucket you are attempting to access must be addressed using the specified endpoint.
Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect,Rid"
I am using navicat for PostgreSQL to connect to Redshift and Im running on a mac.
The S3 path should be 's3://bucketname/filename.csv'. Try this.
Yes, It should be a lot easier :-)
I have only seen this error when your S3 bucket is not in US Standard. In such cases you need to use endpoint based address e.g. http://s3-eu-west-1.amazonaws.com/mybucket/myfile.txt.
You can find the endpoints for your region in this documentation page,
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region