We've just upgraded from Airflow 2.2.4 to 2.4.2, which we deploy via k8s and Helm, using the base apache-airflow==2.4.2 Docker image with some other Python packages installed with pip. Part of Airflow 2.4.2 upgrade replaces Tree View with Grid View in the web UI. When we go to Grid View, we see the following:
The marked up output is your generic Airflow error:
Ooops!
Something bad has happened.
Airflow is used by many users, and it is very likely that others had
similar problems and you can easily find a solution to your problem.
Consider following these steps:
gather the relevant information (detailed logs with errors,
reproduction steps, details of your deployment)
find similar issues using:
GitHub Discussions
GitHub Issues
Stack Overflow
the usual search engine you use on a daily basis
if you run Airflow on a Managed Service, consider opening an issue
using the service support channels
if you tried and have difficulty with diagnosing and fixing the
problem yourself, consider creating a bug report.
Make sure however, to include all relevant details and results of your investigation so far.
Python version: 3.9.15 Airflow version: 2.4.2 Node:
{{deprecated by author}}
------------------------------------------------------------------------------- Error! > Please contact server admin.
Looking in our web logs only yields the following text:
[[34m2022-12-15 01:31:52,943[0m] {[34mapp.py:[0m1741} ERROR[0m - Exception on /object/grid_data [GET][0m
I can't find any other instance online of another Airflow user experiencing this problem, despite searching for hours.
What could be the problem here? It appears to be an internal Airflow UI issue, so may be related to our infrastructure or setup perhaps, but I can't see why.
Also of note, some users had trouble with Grid View when the Operators in their DAGs had a parameters named params. I can rule this out as a cause, as our DAGs and Operators definitely do not have that.
How exactly did you make the update? Did you run helm uninstall airflow and then helm install apache/airflow?
Did you check that run migration job succeed or even if it was executed?
Looks like the issue was to do with dbt-snowflake==1.0.0 being installed as well, and it had some sort of Python package constraint conflict with apache-airflow==2.4.2. Upgrading dbt-snowflake to v1.3.0 solved the issue.
Related
I encountered the same issue as in https://github.com/kubeflow/kubeflow/issues/6014 with my Kubeflow app. The fix is very simple (just a type casting), then I would like to fix it myself and redeploy Kubeflow.
The problem is that I am running a k3s cluster on my local machine where I have installed Kubeflow bundle via Juju. Then, I cannot change the source code.
How to modify Kubeflow source code before deploying it with Kubernetes?
Should I use the manifest installation https://github.com/kubeflow/manifests#installation ? or a totally different method?
Thank you.
The bug was fixed in the last version of the manifests, then I have finally installed kubeflow directly from the manifests.
But still I am in touch with one Kubeflow developer, I will post here the right way to do modify/deploy if interested.
You got to check out their Github repo. Make changes and use kustomize to install like explained in their wiki. If you check the example folder you can see that it points to all other component folders.
https://github.com/kubeflow/manifests#install-with-a-single-command
One another hack could be, just look for the controllers in Kubernetes eg., deployments created for kubeflow, then modify them; works only if your changes are only related to Kubernetes resource definitions. I suggest going with the first option above for a clean development experience, and hey, that way can you contribute back to the kubeflow project as well, if you changes will benefit others.
I'm pretty new on Google Cloud, and I just wanted to deploy my first streamlit webapp. I'm on Windows in command line. I already did the Google Cloud "Hello World" Example, which worked without any error.
When I deploy the streamlit webapp, I got after 3-4 minutes waiting "Updating Server" the following error:
ERROR: (gcloud.app.deploy) Error Response: [9] Flex operation projects/XXXX/regions/us-central1/operations/f0c89d22-2d09-410d-bf99-fc49ad337800 error [FAILED_PRECONDITION]: An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2021-05-27T06:13:50.278Z10796.jc.0: 2021-05-27 06:15:32.787 An update to the [server] config option section was detected. To have these changes be reflected, please restart streamlit.
That's my app.yaml file:
service: default
runtime: custom
env: flex
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
Posting my comment as an answer for better visibility and to summarize.
In this particular case, the error was caused by a mistake in the Dockerfile.
Here are some steps you can follow to fix or narrow down the error:
Try to deploy a test app to see the differences in configuration. Example.
Try deploying your app after updating the gcloud with gcloud components update command.
Make sure you run the SDK as an Admin.
If the error recurs, run the gcloud app deploy app.yaml --verbosity=debug to try getting more specified error.
It's good practice to include references in questions for folks who aren't familiar with e.g. Streamlit. I assume it's this: https://streamlit.io/
I suspect (!) that Streamlit does not (by default) satisfy App Engine's requirements:
A web app on port 8080
No additional (apt get) dependencies
No C-based dependencies
The Streamlit wiki references various deployment alternatives and includes Google Kubernetes Engine (aka GKE) (see below) but not App Engine.
This doesn't mean that it won't work on App Engine (standard) just that it may not be trivial.
The GKE instructions reference installing Cython an optimizing c-compiler and that gives me pause about using App Engine standard. Unless you're familiar with Kubernetes, I'd discourage you from trying GKE as there's more complexity.
So, it would be helpful if others with experience with Streamlit weigh in but, until then, you may wish to consider using Streamlit sharing.
It would be helpful if someone who has deployed Streamlit to App Engine (flexible?) or perhaps Cloud Run can provide an overview.
I'm running a Neo4j instance (version 4.2.2) in a pod within a Kubernetes cluster, in standalone mode. The server starts, I can create, find and update nodes and relationships, however, when trying to install a UUID using apoc.uuid.install, the procedure hangs and never seems to finish.
I'd also like to mention that apoc.uuid.enabled=true is set in neo4j.conf, I've set a constraint on the designated UUID field before running install and I can't find any errors in the logs. I've also tried this functionality in non-K8s environments and I have no problem using it there.
The helm charts used for this deployment are taken from https://github.com/neo4j-contrib/neo4j-helm.
Did anyone experience this behavior? If yes, how did you manage to solve it?
I am trying to use KubeFlow on GCP and I am following this codelab, but "click-to-deploy" is no longer supported so I followed the documentation of "kubectl and kpt". However, I keep getting this "You cannot perform this action because the Cloud SDK component manager is disabled for this installation." error and none of the solutions I found worked. I have 2 other friends told me they tried to make KubeFlow work since last year, it never worked, but I did see people post question about KubeFlow on Stackoverflow still, so I want to ask if it is still working, if so, where can I find a decent guide to follow?
Thanks!
I finally got it working. For that error message, it turned out that I just didn't install the Cloud SDK properly. There will be a lot of other issues too down the road, but at least the KubeFlow web UI is working for me now.
yes, as the kubectl and kpt says, the first step in getting prepared to install cluster is installing gcloud that is CLI that manages authentication, local configuration, developer workflow, interactions with Google Cloud APIs.
Without is you simply cant work with objects(in your case you need to enable kpt anthoscli beta) and perform tasks like
creating a Compute Engine VM instance, managing a Google Kubernetes
Engine cluster, and deploying an App Engine application, either from
the command line or in scripts and other automations..
I just started playing around with Docker.io. Its a great platform for sure. I have an issue i need some help with. I ran a medium instance on ec2 setup docker. Now i want to run 2 wordpress blog independent of each other using docker.io on top of the medium instance.
Please if someone can kindly guide me to resolve this issue i will extremely grateful
Many Thanks Indeed
Hareem Haque
Updated:
Basically, what i am trying to do is run two nodes for docker (node 1 & node 2). I run another node (node3: private repo for docker). What i am looking to accomplish is i run two blogs (wordpress on node1). I export the docker images to node3 (updates/exports are done very rarely)
Since i am going to run wordpress i was hoping to run wordpress within Nginx and since node1/node2 will run 80 web i can put a physical node (nginx reverse proxy) in front of the two nodes and have the blogs run in ha mode.
I am hoping that this experiment work so i that i can get rid of the xen cloud platform we have in office. Its to bulky and I have to manage alot of components.
I would rather export/backup docker image with my live data once in a blue moon and not have to worry about failover and vm management.
The problem is that i have a novice when it comes to running docker and thus i am currently running around like a head less chicken with no idea where to properly begin.
I would be extremely grateful if you can provide any guidance/assistance indeed.
Best Regards
Hareem Haque
Hareem asked his question a while back, and there don't seem to be any good answers yet. I'm a noobie as well, and I too want to learn how to use a generic wordpress container that I can push to Amazon or test locally. I'm very new to docker, so this seems like a tall order!
Goal
For now, I'll start collecting some resources here. Maybe they will help Hareem, and others like myself. This document will turn into a complete answer, or prompt someone else to give their version of an answer (which I'm sure is not quite so complex.)
The Docker.io Index
First, the Docker index is a repository of already existing Docker.io components. Of these, there is a wordpress unit that seems relevant here:
jbfink - Wordpress 3.5.2.
Docker on EC2
There is as yet no official Docker support for Ec2. However, the Docker community suggests an install path using a tool called Vagrant. The instructions for this live here:
Docker Doc - Installing on Amazon EC2
Work In Progress
This is not a complete answer to the question. As of right now this only presents a couple of easy to locate resources, and perhaps goes against guidelines. Please bear with this!
Things that need to be answered:
How do we run / test the wordpress container(s) locally?
How do we push the container(s) up to the EC2 instance?
How do we wire the EC2 wordpress containers up to their own domains?
Hopefully I will answer these questions - contributions and forks are welcome. I think Hareem's question is worth answering!