Hi I'm trying to gather entities from Purview. I only know their assetType, ie. i do not have guids or such. My query runs into problems when the offset hits the 100000 limit. My idea was to paginate with entity creation date, but I've been unable to make it work for some reason.
My query is made with PyApacheAtlas' discovery query function. The search_filter arguments I've tried passing have been something like the following snippets:
{"attributeName":"createTime",
"operator":"lt",
"attributeValue": 1616071832353}
{"attributeName":"createTime",
"operator":"GreaterThan",
"attributeValue": 1616071832353}
{"createTime": {"lt": 1616071832353}}
{"createTime":{"lt":"2021-12-15"}}
{"lt": {"createTime":"2021-12-15"}}
"Illegal argument: wrong parameters for system time filter"
{"createTime":{"operator":"le","attributeValue":"2021-12-15"}}
{"lt": {"createTime":1616071832353}}
{"createTime": 1616071832353}
"errorCode":"InvalidSearchFilter","errorMessage":"The search filter is invalid: wrong contract for system time filter"
{"createTime": {"and": [{"le":"2021-12-15"},{"ge":"2021-12-10"}] }}
"errorCode":"RequestIllegalArgument","errorMessage":"Illegal argument: wrong parameters for system time filter"}
{"createTime":"2021-12-15"}
"errorCode":"InvalidSearchFilter","errorMessage":"The search filter is invalid: wrong contract for system time filter"}
{"createTime": {"le":"2021-12-15T04:25:03Z"}}
"errorCode":"RequestIllegalArgument","errorMessage":"Illegal argument: wrong parameters for system time filter"
Some error messages below the given search_filters.
Any tips for the correct syntax, or idea whether such date based requests are possible at all? Thanks.
Related
I use datagrip as client to connect redshift and encounter a stranger issue which exhaust my whole day.
When I run my query sql the datagrip complains
[XX000] ERROR: invalid string enlargement request size 1073741823
It seemed that there dont exist a place that I can check more detail error log. And I google this error it also have very little similar question and it seemed maybe due to my field is too long which exceed the max length that redshift can accept. But actually, the story is not such for me I dont have long field, then I comment all my sql statement and re-add them incrementally to locate this issue statement.
Finally, I find the error-msg-triggered statement as below:
(
case when trunc(request_date_skip_weekend_tmp) = to_date('2022-03-21', 'YYYY-MM-DD')
then dateadd(day, 1, trunc(request_date_skip_weekend_tmp))
else request_date_skip_weekend_tmp end
)
request_date_skip_weekend,
After I change it with:
dateadd(day, 1, trunc(request_date_skip_weekend_tmp)) request_date_skip_weekend,
the error complain disappear, it is very hard for me to accept the relationship error message and the sql change, I dont know why my the former statement will trigger error complain.
I will appreciate if you can spot why the former expression error or share some knowledge about where can I fetch more detail error message to know what happened.
Your code snippet is dates and timestamps but the error is for strings. So it is likely you have identified a "trigger" and not root cause. Also since you report that the SQL is very long you could be dealing with compiler optimization changes, moving the failure. Removing a CASE can cause the compiler/optimizer to choose different structures for the query.
One experiment to try is to change the to_date() to a cast to timestamp so there are no implicit casts ('2022-03-21'::timestamp). This is unlikely the cause but it may help.
I expect you will need to post the query to get more help. How large is it? This error could be related to building a large string in the query OR could be related to the text of the query OR creating the output. This isn't a standard "string too long" message so this is something more implicit. You could post to a google doc or some other file sharing service. Just link in the question.
I am trying to fetch data from SAP BW system into Azure Data Lake using MDX Query in SAP BW connector. But I am getting exception message in Azure is following :
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column '[Measures].[SomeMeasure]' contains an invalid value '4.000-2'. Cannot convert '4.000-2' to type 'Decimal'.,Source=Microsoft.DataTransfer.Common,''Type=System.InvalidCastException,Message=Specified cast is not valid.,Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "Copy1"
}
From the error, I can understand there are some values in Measures which is not actually numeric value. Changing or correcting value in the SAP system is not in my scope.
Is there any option in the Data Factory V2 for SAP BW connection so that I can define the data type of the Measure for input and/or output. or there is any fine-tuning in the MDX query so that I can fetch the data without any error?
This is my MDX Query :
SELECT
{[Measures].[SomeMeasure]} ON COLUMNS,
NON EMPTY
{ [0COMP_CODE].[LEVEL01].MEMBERS *
[0COSTELMNT].[LEVEL01].MEMBERS }
ON ROWS
FROM SomeQube
WHERE {[0FISCPER].[K42015008]}
If you could skip that incorrect rows, you could set enableSkipIncompatibleRow in as true. Please reference this doc.
Your filter is incorrect. SAP 0FISCPER dimension is in YYYYMMM format. You need to enter WHERE {[0FISCPER].[2015008]}. I am almost certain the "K4" you entered there is your fiscal year variant, which you should not put there. Objects like this K4 are called compounding infoobject (which your fiscal year/period belongs to if you envision it as hierarchical parent/child structure). But in this specific case, you don't need to specify fiscal year variant. Just remove K4.
I highly advise against skipping "incorrect rows". My guidance to you is that you should always fail the job whenever exception happens and investigate the data quality issue. Do not ever put enterprise data integrity at risk. SAP data has to be 100% accurate, not 99.9999%.
I am trying to use the VSTS.Feed() function in Power BI to read WorkItemSnapshot data. There are multiple problems. If I build the entire URL into a single string and call VSTS.Feed () with that, I get the correct information in Power BI desktop, but it will not refresh in Power BI online. I have been told to use the (undocumented) Query parameter, as shown below, but it is clear that this parameter is ignored. I can see that the select parameter is ignored on smaller projects, because all columns are returned. I can see that the filter parameter is ignored because the query fails on larger projects.
Does anyone have a working example of using the Query parameter with VSTS.Feed()?
let
BaseURL = "https://server.analytics.visualstudio.com/DefaultCollection/project/_odata/WorkItemSnapshot",
Select = "DateSK,WorkItemId,State,WorkItemType",
Filter = "WorkItemType eq Bug and State ne Closed and State ne Removed and DateSK ge 20180517 and DateSK le 20180615",
Source = VSTS.Feed(BaseURL, [Query=[select=#"Select",filter=#"Filter"]])
in
Source
Update:
With the query above, the message I get is shown below. As I said earlier, it is clearly not using the Filter parameter, and I'm assuming it is not using the Select parameter, either. I can't query everything because there is too much data, and I can't use a filter because I can't figure out a way to get the Options parameter to work. With VSTS.AccountContents, the options parameter works well, but those API endpoints don't use $ in parameter names.
Error: Query result contains 36,788,023 rows and it exceeds maximum allowed size of 300,000. Please reduce the number of records by applying additional filters
Details:
DataSourceKind=Visual Studio Team Services
ActivityId=881f7988-9863-4e03-8375-0489028f28f3
Url=https://server.analytics.visualstudio.com/DefaultCollection/Project/_odata/WorkItemSnapshot
error=Record
The query that started this whole line of questioning is simply one with a variable for a start date.
let
startDate = DateTimeZone.ToText (Date.AddDays(DateTimeZone.UtcNow(), -45), "yyyyMMdd"),
URL = "https://server.analytics.visualstudio.com/DefaultCollection/project/_odata/WorkItemSnapshot?$select=DateSK,WorkItemId,State,WorkItemType&$filter=WorkItemType eq 'Bug' and State ne 'Closed' and State ne 'Removed' and DateSK gt " & startDate,
Source = VSTS.Feed(URL)
in
Source
While this query mostly works in Power BI desktop (the select clause is ignored), the message I get when the data source is refreshed online is:
You can't schedule refresh for this dataset because one or more sources currently don't support refresh.
Discover Data Sources
Query contains unknown or unsupported data sources.
The documentation for VSTS.Feed() contradicts itself, saying both
The VSTS.Feed function has the same arguments, options and return value format as OData.Feed.
and
'VSTS.Feed' provides a subset of the Arguments and Options available through 'OData.Feed'.
To to summarize, I know that I can't combine data sources in Power BI. Does VSTS.Feed() support the options parameter? If so, how do I pass a Filter and Select clause to it?
To get WorkItemSnapshot by vsts.feed, please refer below query:
let
Source = OData.Feed("https://account.analytics.visualstudio.com/project/_odata/v1.0-preview", null, [Implementation="2.0"]),
WorkItemSnapshot_table = Source{[Name="WorkItemSnapshot",Signature="table"]}[Data]
in
WorkItemSnapshot_table
Note: the URL format should be https://account.analytics.visualstudio.com/project/_odata/v1.0-preview, or https://account.analytics.visualstudio.com/_odata/v1.0-preview.
And you can refer below documents:
Connect to VSTS using the Power BI OData feed
Connect using Power Query and Visual Studio Team Services (VSTS) functions
I want to get some statistics about the job I'm running on my pool, and for that I am trying to use the JobStatistics class, but I have been getting job.Statistics as null in most of my runs except for few where the result was magically not null. I read in a documentation (https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.batch.cloudjob.statistics?view=azurebatch-6.1.0#Microsoft_Azure_Batch_CloudJob_Statistics) that for the statistics results not to be null, I need to use an expand clause with DetailLevel, but each time I do, I get the error: "operation returned an invalid status code 'badrequest' ". This is what I have for that.
ODATADetailLevel detailExJob = new ODATADetailLevel();
detailExJob.SelectClause = "id,executionInfo,stats";
detailExJob.ExpandClause = "id,executionInfo,stats";
await job.RefreshAsync(detailExJob);
What am I missing here? How can I get job.Statistics not to be null?
Thanks!
I'll try to answer your question, but it looks like you have two separate issues.
Job lifetime statistics may not be immediately available. The Batch service
performs periodic roll-up of statistics. I believe the typical delay is about 30minutes, but this is not documented.
The expand clause currently only supports stats. If you modify your detailExJob.ExpandClause statement to be assigned just "stats", then your job query should work. Moreover, you can simplify your detail level object to omit the expand clause altogether since you specified stats in the select clause.
Using the the following CQL query:
SELECT NAMESPACES WHERE NameLike "Test$" ORDER BY NbLinesOfCode DESC
I am getting some results that show "N/A" instead of a number for NbLinesOfCode. Anyone know why this is happening and how to resolve it?
Note: I tried changing NbLinesOfCode to NbILInstructions, and none of the result records showed N/A.
There can be two things here:
If both NbLinesOfCode and NbILInstructions shows N/A, it means that the namespace doesnt have any code and contains only types without code (like interfaces, delegates or enumerations)
If only NbLinesOfCode shows N/A but NbILInstructions shows something, then it means NDepend cannot have access to the assemblies PDB. More information in: Understanding NDepend Analysis Inputs