Compilation of Elyra-Pipelines to Tekton based Kubeflow fails - kubernetes

I've installed a kubernetes cluster running kubeflow pipelines based on tekton on top of KIND using the following instructions
Now I'm getting the following error message from the Elyra pipelines editor. Running against an argo based kfp cluster works fine.
Is the kfp compiler somehow not supporting tekton? Can someone please shine some light on this?
HTTP response body:
{"error_message":"Error creating pipeline: Create pipeline failed:
Failed to get parameters from the pipelineRun: Invalid input error:
Unsupported argo version.
Expected: tekton.dev/v1beta1.
Received: argoproj.io/v1alpha1",
"error_details":"Error creating pipeline: Create pipeline failed:
Failed to get parameters from the pipelineRun: Invalid input error:
Unsupported argo version. Expected: tekton.dev/v1beta1.
Received: argoproj.io/v1alpha1"}

There are different aspects to consider here:
There is a need to perform a custom deployment to enable Kubeflow Pipelines to integrate/use a Tekton environment. As you mentioned the steps are described here.
As for Elyra support, Elyra recent releases starting with Elyra 2.x has incorporated support for KFP using Tekton and further documentation is available in the Elyra user guide
If you are still using Elyra 1.x or lower, the below still applies:
Now, related to compiling and executing the Kubeflow Pipeline on a Tekton environment, Elyra is currently using the KFP SDK python package and the regular compiler to compile and generate the default ARGO YAML (see code here). In order to support Tekton, we will need to use a different python package (kfp-tekton==0.3.0) and there is also a need for a different code path during compilation (see example).
Another aspect is that it seems that there are some incompatibilities between the two pipelines as described in the migration path.
Regarding support, the Elyra project would welcome contributions.

As of now the Tekton compiler is in a separate package. You can install it with pip install kfp-tekton==0.3.0 for kubeflow 1.2 . Here is the user guide
Currently, Elyra doesn't support compiling for kfp-tekton, only kfp-argo
There is an open Issue on that with the Elyra team

Related

Testing of ARM Templates using ARM TTK Tool in a DevOps Pipeline

I am trying to use ARM TTK and i am selecting my tests.
One of the tests that i want to run is apiVersions Should Be Recent
Now in my ARM template i have a resource block of type "Microsoft.DevTestLab/schedules".
The latest API version for this resource as per the link API Versions is 2018-09-15.
This is more than 2 years old and my test is failing as it is not fulfilling one of the conditions.
So how to tackle such kind of API versions if there is no current version available which is within 2 years. Any help is appreciated
learn.microsoft.comlearn.microsoft.com
Template test cases for test toolkit - Azure Resource Manager
Describes the template tests that are run by the Azure Resource Manager template test toolkit.
learn.microsoft.comlearn.microsoft.com
Microsoft.DevTestLab/schedules - Bicep & ARM template reference
Azure Microsoft.DevTestLab/schedules syntax and properties to use in Azure Resource Manager templates for deploying the resource. API version latest
The issue still available in a arm ttk issue & arm ttk issue2
To get a API Version for the specific Resource type use below PowerShell script.
((Get-AzureRmResourceProvider -ProviderNamespace Microsoft.Web).ResourceTypes | Where-Object ResourceTypeName -eq sites).ApiVersions
Refer for azure ARM schema generation azure-resource-manager-schemas
Refer for additional Steps https://stackoverflow.com/a/45066194/15997690
Solution is available if the latest version of API available check here in your case No latest version available for DevTestLab.
Please raise a support ticket and MS Q&A here

ERROR: (gcloud.app.deploy) Error Response: [9] Flex operation projects/.../regions/us-central1/operations/... error [FAILED_PRECONDITION]

I'm pretty new on Google Cloud, and I just wanted to deploy my first streamlit webapp. I'm on Windows in command line. I already did the Google Cloud "Hello World" Example, which worked without any error.
When I deploy the streamlit webapp, I got after 3-4 minutes waiting "Updating Server" the following error:
ERROR: (gcloud.app.deploy) Error Response: [9] Flex operation projects/XXXX/regions/us-central1/operations/f0c89d22-2d09-410d-bf99-fc49ad337800 error [FAILED_PRECONDITION]: An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2021-05-27T06:13:50.278Z10796.jc.0: 2021-05-27 06:15:32.787 An update to the [server] config option section was detected. To have these changes be reflected, please restart streamlit.
That's my app.yaml file:
service: default
runtime: custom
env: flex
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
Posting my comment as an answer for better visibility and to summarize.
In this particular case, the error was caused by a mistake in the Dockerfile.
Here are some steps you can follow to fix or narrow down the error:
Try to deploy a test app to see the differences in configuration. Example.
Try deploying your app after updating the gcloud with gcloud components update command.
Make sure you run the SDK as an Admin.
If the error recurs, run the gcloud app deploy app.yaml --verbosity=debug to try getting more specified error.
It's good practice to include references in questions for folks who aren't familiar with e.g. Streamlit. I assume it's this: https://streamlit.io/
I suspect (!) that Streamlit does not (by default) satisfy App Engine's requirements:
A web app on port 8080
No additional (apt get) dependencies
No C-based dependencies
The Streamlit wiki references various deployment alternatives and includes Google Kubernetes Engine (aka GKE) (see below) but not App Engine.
This doesn't mean that it won't work on App Engine (standard) just that it may not be trivial.
The GKE instructions reference installing Cython an optimizing c-compiler and that gives me pause about using App Engine standard. Unless you're familiar with Kubernetes, I'd discourage you from trying GKE as there's more complexity.
So, it would be helpful if others with experience with Streamlit weigh in but, until then, you may wish to consider using Streamlit sharing.
It would be helpful if someone who has deployed Streamlit to App Engine (flexible?) or perhaps Cloud Run can provide an overview.

MsBuild fails on Azure thus failing deployments on App Service using GitHub/Azure DevOps

I have an app that runs locally using the test-webserver (xsp) in visual studio 2019 for mac.
I have setup the azure app service to deploy from a branch in the repository in github (via deployment center).
When it runs the deployment I get the following error:
D:\home\site\repository\packages\Microsoft.Net.Compilers.3.6.0\tools\Microsoft.Managed.Core.targets(60,27): error MSB4184: The expression ""AllowCrossSiteAttribute.cs".GetPathsOfAllDirectoriesAbove()" cannot be evaluated. Method 'System.String.GetPathsOfAllDirectoriesAbove' not found. [D:\home\site\repository\Fx\Fx.csproj]
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe"
I have tried to remove the Microsoft.Compiler package as per another stackoverflow question GetPathsOfAllDirectoriesAbove() cannot be evaluated after updating .Net Framework version (4.6.2 to 4.7.2) but to no avail.
Is there a way to replicate the failing msbuild step locally?
Or another workaround for the error?

Red gate DLM Automation binding error using VSTS

I'm using Red gate DLM Automation version 2 on VSTS. I installed DLM on the build server, it's fully licensed, and I have noth the build and release VSTS plugins installed. But now, when I perform a build on VSTS, I receive the following error message:
System.Management.Automation.CmdletInvocationException: A parameter cannot be found that matches parameter name 'transactionIsolationLevel'. ---> System.Management.Automation.ParameterBindingException: A parameter cannot be found that matches parameter name 'transactionIsolationLevel'.
My initial guess is that I have a version conflict between some Red Gate PowerShell libraries, but I cannot find any further information on this error.The build server is running PowerShell v4. The SQL code being built does not have any references to "transaction isolation" in it, but I don't think that's the problem. I have completed successful builds on this VSTS server in the past and am now confused what caused this error to start appearing.
Thank you!
You need at least 2.0.3 of the DLMA install on the local agent to work with the VSTS plugin - we added the Transaction Isolation Level option very recently, and VSTS auto-updates, but the DLMA install doesn't.
Sorry about that - we are looking into better update / communication mechanisms to keep these things in sync in future (or at least tell you what the problem is) but aren't quite there yet.
If you're still having trouble after updating the DLMA install on the local agent, please do get in touch via support#red-gate.com and we'll sort it out for you.

Scala application on CF

I tried to launch our Scala application on the Swisscom Cloud Foundry (CF) infrastructure. To do so, the matching Heroku buildpack was used:
https://github.com/heroku/heroku-buildpack-scala
As this did not work, I tried to deploy the 'hello-scala' example using this buildpack.
My fork to be able to build the slightly outdated example:
https://github.com/AlwinEgger/hello-scala
I have to underline that I am fetching the port I have to use as env variable 'PORT'.
Unfortunately, there is not much on the log. "failed to accept connections within health check timeout" message indicates that there is no one listening...
My questions: Did anyone succeed in deploying Scala apps on CF infrastructures (# Swisscom)?
A workaround I found:
I'm not using the scala- but the java-buildback. This with the major advantage and inconvenience that the project is not any more build on the instance.
Advantage: It speeds up the whole process considerably
Inconvenience: A build server is needed
So what do we have to do?
An example may be found here (this is the actual application):
https://github.com/OpenOlitor/openolitor-server
Add the sbt-native-packager to your project
Execute the action 'universal:packageBin' building by hand or configure your build server to do so
Change the buildpack in the manifest.yml and add some parameters, if necessary. Configure the path of the artifact to deploy.
Run cf push or let the build server do so.