pg_upgrade tool failed: invalid "unknown" user columns - postgresql

Question

Postgresql update from 9.6 to 10.4 (on Fedora 28) has me stuck: one table in one database has a column of data type "unknown". I would gladly remove the column, but since I cannot get postgresql service to start (because "An old version of the database format was found"), I have no access to the database. In more detail:
postgresql-setup --upgrade fails.
/var/lib/pgsql/upgrade_postgresql.log attributes this failure to column with data type "unknown": "...Checking for invalid 'unknown' user columns: fatal .... check tables_using_unknown.txt". And "tables_using_unknown.txt" specifies one column in one table that I wish I could drop, but can't, because I can't get the server to start:
systemctl start postgresql.service fails, and
systemctl status postgresql.service complains about the "old version of the database"
I have found no obvious way to install postgresql 9.6 on Fedora 28.
Is there a way to drop the column without a running server? Or at least produce a dump of the database? Or can I force the upgrade tool to drop columns with data type "unknown"? Or is there any other obvious solution that I'm missing?

Answer 1

Here's what finally worked for me:
I used a docker container (on the same machine) with postgres 9.6 to access the "old" database directory,
converted the problematic column from "unknown" to "text" in the container,
dumped the relevant database to a file on the container's host, and then
loaded the dumped db into the postgres 10.4 environment.
Not pretty, but worked. In more detail:
I copied postgresql's data directory (/var/lib/pgsql/data/ in Fedora) -- containing the database that could not be converted -- to a new, empty directory /home/hj/pg-problem/.
I created a Dockerfile (text file) called "Docker-pg-problem" reading
FROM postgres:9.6
# my databases need German locale;
# if you just need en_US, comment the next two lines out.
RUN localedef -i de_DE -c -f UTF-8 -A /usr/share/locale/locale.alias de_DE.UTF-8
ENV LANG de_DE.utf8
and saved it as the only file in the new, empty folder /home/hj/pg-problem/docker/.
I started the docker daemon and ran a container that uses the data from my copy of the problematic data (in /home/hj/pg-problem/data/) as data directory for the postgres 9.6 server in the container. (NB: the "docker build" command in line three needs a working internet connection, takes a while, and should finish saying "Successfully built").
root#host: cd /home/hj/pg-problem/docker
root#host: service docker start
root#host: docker build -t hj/failed-update -f Dockerfile .
root#host: docker run -it --rm -p 5431:5432 -v /home/hj/pg-problem/data:/var/lib/postgresql/data:z --name failed-update -e POSTGRES_PASSWORD=secret hj/failed-update
Then, I opened a terminal in the container to fix the database:
hj#host: docker exec -it failed-update bash
Inside the container, I fixed and dumped the database:
root#container: su postgres
postgres#container: psql <DB-name>
postgres#container: alter table <Table-name> alter column <Col-Name> type text;
postgres#container: \q
postgres#container: dump_db <DB-name> /var/lib/postgresql/data/dbREPAIRED.sql
I dumped the db right into the data directory so I could easily access the dumped file from the docker host.
On the docker host, the dumped database was, obviously, in /home/hj/pg-problem/data/dbREPAIRED.sql, and from there I could load it into postgresql 10:
postgres#host: createdb <DB-name>
postgres#host: psql <DB-name> < /home/hj/pg-problem/data/dbREPAIRED.sql
Since I was on a laptop with limited disk space, I deleted the docker stuff:
root#host: docker rm $(docker ps -a -q)
root#host: docker rmi $(docker images -q)