As Earlier Posted a thread for syncing Data from Premises Mysql to Azure SQL over here referring this article, and found that lookup component for watermark detection is only available for SQL Server Only.
So tried a work Around, that while using "Copy" Data Flow task ,will pick data greater than last watermark stored from Mysql.
Issue:
Able to validate package successfully but not able to publish same.
Question :
In Copy Data Flow Task i'm using below query to get data from MySql greater than watermark available.
Can't we use Query like below on other relational sources like Mysql
select * from #{item().TABLE_NAME} where #{item().WaterMark_Column} > '#{activity('LookupOldWaterMark').output.firstRow.WatermarkValue}'
CopyTask SQL Query Preview
Validate Successfully
Error With no Details
Debug Successfully
Error After following steps mentioned by Franky
Azure SQL Linked Service Error (Resolved by re configuring connection /edit credentials in connection tab)
Source Query got blank (resolved by re-selection source type and rewriting query)
Could you verify if you have access to create a template deployment in the azure portal?
1) Export the ARM Template: int he top-right of the ADFv2 portal, click on ARM Template -> Export ARM Template, extract the zip file and copy the content of the "arm_template.json" file.
2) Create ARM Template deployment: Go to https://portal.azure.com/#create/Microsoft.Template and log in with the same credentials you use in the ADFv2 portal (you can also get to this page going in the Azure portal, click on "Create a resource" and search for "Template deployment"). Now click on "Build your own template in editor" and paste the ARM template from the previous step in the editor and Save.
3) Deploy template: Click on existing resource group and select the same resource group as the one where your Data Factory is. Fill out the parameters that are missing (for this testing it doesn't really matter if the values are valid); Factory name should already be there. Agree the terms and click purchase.
4) Verify the deployment succeeded. If not let me know the error, it might be an access issue which would explain why your publish fails. (ADF team is working on giving a better error for this issue).
Did any of the objects publish into your Data Factory?
Related
All, when running a build pipeline using Azure Devops with ARM template, the process is consistently failing when trying to deploy a dataset or a reference to a dataset with this error:
ARM Template deployment: Resource Group scope (AzureResourceManagerTemplateDeployment)
BadRequest: The document creation or update failed because of invalid reference 'dataset_1'.
I've tried renaming the dataset and also recreating it to see if that would help.
I then deleted the dataset_1.json file from the repo and still get the same message so it's some reference to this dataset and not the dataset itself I think. I've looked through all the other files for references to this but they all look fine.
Any ideas on how to troubleshoot this?
thanks
try this
Looks like you have created 'myTestLinkedService' linked service, tested connection but haven't published it yet and trying to reference that linked service in the new dataset that you are trying to create using Powershell.
In order to reference any data factory entity from Powershell, please make sure those entities are published first. Please try publishing the linked service first from the portal and then try to run your Powershell script to create the new dataset/actvitiy.
I think I found the issue. When I went into the detailed logs I found that in addition to this error there was an error message about an invalid SQL connection string, so I though it may be related since the dataset in question uses Azure SQL database linked service.
I adjusted the connection string and this seems to have solved the issue.
I am aware of Qlik sense serialize app where we generate a JSON object containing metadata information of a .qvf file using Qlik sense API.
I want to do a reverse operation of this i.e generate .qvf file back from json metadata.
After many research just found this link github and it doesnot have a complete information.
Any solution would be helpfull.
Technically you cant create qvf directly from json. You'll have to create an empty qvf and then use various api to import the json.
Qlik have a very nice tool for un-build/build apps (and more). qlik-cli have dedicated commands for un-build/build:
If you are looking for something more "programmable" then ive create some enigma.js mixin for the same purpouse - enigma-mixin. I still need to perform more detailed testing there but it was working ok with simpler tests
Update 08/10/2021
Using qlik-cli
setup context
first unbuild an app:
qlik app unbuild --app 11111111-2222-3333-4444-555555555555
This will create new folder in the current folder named <app_name>-unbuild. The folder will contain all info about the app in json and/or yaml files
once these files are available then you can use them to build another app. Just to mention that the target app should exists before the build is ran:
qlik.exe app build --config ./config.yml --app 55555555-4444-3333-2222-111111111111
The above command will use all available files (specified in config.yml) and update the target app
If you dont want all files to be used and only want to update the data connections, for example, then the build command can be ran with different arguments:
qlik.exe app build --connections ./connections.yml --app 55555555-4444-3333-2222-111111111111
This command will only update the data connections in the target app and will not update anything else
I'm new to using Azure release pipelines and have been fighting issues trying to deploy a database project to a new Azure SQL database. Currently the pipeline is giving me the following error...
TargetConnectionString argument cannot be used in conjunction with any other Target database arguments
I've tried deploying with and without the TargetConnectionString included in my publish profile. Any suggestions or something else to try? I'm out of ideas.
TargetConnectionString
Specifies a valid SQL Server/Azure connection string to the target database. If this parameter is specified it shall be used exclusively of all other target parameters. (short form /tcs)
So please remove all other TargetXXX arguments.
(if you don't have them can you show what arguments you have inline and in publish profile - of course without data)
I've created a process to be able send email to the user on order confirmation.
The problem is that on the DEV environment everything goes well but when I did a deploy to UAT server
I got an exception during the task execution ( " Media not found (requested media location: hf0/h27/8861015965726.bin) ").
Any Ideas what could be happening ?
How can this issue be resolved and what causes this issue.
hybris creates emails using Velocity templates. Those Velocity Templates are stored as Medias on the hybris Servers. hybris Medias consist of two parts: an entry in the respective table in the database and a file on the hard drive. The database entry stores metadata about that media while the file stores the actual content.
Now what hybris is telling you, is that the file on the hard drive is missing. The database entry directs to a file that is not existing. There could be a lot of reasons why that file is missing:
It was deleted during deployment.
It wasn't created during deployment.
The hybris server has no access/access rights to that directory.
In a clustered environment the file could have been stored on another node and is not accessible on the current node.
Media could be the email itself as Johannes stated, but it can also be a part of the email, an image set from the CMS cockpit for example.
To fix this issue you have to master your impex flows.
First be sure that impex contain all the data needed to create properly the email.
Then know what is imported when you deploy and update your system.
Be sure that mandatory files are imported during initialization.
Be sure that data that can be managed by webmasters are not reset by impex during update.
If a data is created during the update because init is already done then be sure that is won't be played after each update.
As the media file is not found, you can
1. go to hmc-->Multimedia-->Media, in search panel,
2. click "search additional attributes" dropdown box, select "PK of file"
3. use "8861015965726" as PK of file to search
Then you can find out what file is missing and you can import impex or upload using hmc to fix this problem.
When I try to upload the zip file to an azure function app using kudu REST API, it throws an error while I try to view the c# code in Function App editor in the browser. The error is:
"Error:
Function ($Source) Error: Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.Source'. Microsoft.Azure.WebJobs.Host: Value cannot be null.
Parameter name: hostAccount.
Session Id: xxxxxxxxxxx
Timestamp: 2016-12-02T18:35:00.083Z"
Please note that I have automated end to end of Application Insights starting from creation of a resource group till exporting the multi-setep web test results to our Splunk - All using Powershell.
In this process of automation, I am forming a storage connection string and setting it to the app settings of the function app and then providing that key in my function.json binding.
But still I get this error.
Here is the issue I created in the Azure Function App - Git: https://github.com/Azure/azure-webjobs-sdk-script/issues/1020
The error points to missing host configuration (e.g. the host storage account).
If you're automating the creation process, you need to make sure the required environment variables are properly set. The recommended way would be to use an ARM template manage that for you.
I have some details on how you can get the ARM template for a resource (which you could use to look at the required settings for your Function App) here.
We also have some sample templates you can use linked here
I hope this helps!