How to self-host Read the Docs using GitHub Pages - github

How can I setup a CI/CD workflow with gitlab (or GitHub Actions) that generates my own Read the Docs site and is hosted for free using gitlab pages?
Is there a fork-ready example repo on gitlab or github that I can use to self-generate and self-host my own Read the Docs site?

You can host a sphinx-powered site (optionally using the Read the Docs theme) on GitHub Pages using GitHub Actions to wrap sphinx-build and push your html static assets to your GitHub Pages source, such as the gh-pages branch..
You need to define a GitHub Actions workflow to execute a build script.
Here's an example workflow that will execute buildDocs.sh every time there's a push to master
name: docs_pages_workflow
# execute this workflow automatically when a we push to master
on:
push:
branches: [ master ]
jobs:
build_docs_job:
runs-on: ubuntu-latest
container: debian:buster-slim
steps:
- name: Prereqs
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
apt-get update
apt-get install -y git
git clone --depth 1 "https://token:${GITHUB_TOKEN}#github.com/${GITHUB_REPOSITORY}.git" .
shell: bash
- name: Execute script to build our documentation and update pages
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: "docs/buildDocs.sh"
shell: bash
And here's an example buildDocs.sh script that's executed by the workflow above:
#!/bin/bash
################################################################################
# File: buildDocs.sh
# Purpose: Script that builds our documentation using sphinx and updates GitHub
# Pages. This script is executed by:
# .github/workflows/docs_pages_workflow.yml
#
# Authors: Michael Altfield <michael#michaelaltfield.net>
# Created: 2020-07-17
# Updated: 2020-07-17
# Version: 0.1
################################################################################
###################
# INSTALL DEPENDS #
###################
apt-get update
apt-get -y install git rsync python3-sphinx python3-sphinx-rtd-theme
#####################
# DECLARE VARIABLES #
#####################
pwd
ls -lah
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
##############
# BUILD DOCS #
##############
# build our documentation with sphinx (see docs/conf.py)
# * https://www.sphinx-doc.org/en/master/usage/quickstart.html#running-the-build
make -C docs clean
make -C docs html
#######################
# Update GitHub Pages #
#######################
git config --global user.name "${GITHUB_ACTOR}"
git config --global user.email "${GITHUB_ACTOR}#users.noreply.github.com"
docroot=`mktemp -d`
rsync -av "docs/_build/html/" "${docroot}/"
pushd "${docroot}"
# don't bother maintaining history; just generate fresh
git init
git remote add deploy "https://token:${GITHUB_TOKEN}#github.com/${GITHUB_REPOSITORY}.git"
git checkout -b gh-pages
# add .nojekyll to the root so that github won't 404 on content added to dirs
# that start with an underscore (_), such as our "_content" dir..
touch .nojekyll
# Add README
cat > README.md <<EOF
# GitHub Pages Cache
Nothing to see here. The contents of this branch are essentially a cache that's not intended to be viewed on github.com.
If you're looking to update our documentation, check the relevant development branch's 'docs/' dir.
For more information on how this documentation is built using Sphinx, Read the Docs, and GitHub Actions/Pages, see:
* https://tech.michaelaltfield.net/2020/07/18/sphinx-rtd-github-pages-1
EOF
# copy the resulting html pages built from sphinx above to our new git repo
git add .
# commit all the new files
msg="Updating Docs for commit ${GITHUB_SHA} made on `date -d"#${SOURCE_DATE_EPOCH}" --iso-8601=seconds` from ${GITHUB_REF} by ${GITHUB_ACTOR}"
git commit -am "${msg}"
# overwrite the contents of the gh-pages branch on our github.com repo
git push deploy gh-pages --force
popd # return to main repo sandbox root
I wrote an article that describes how to run your own Read the Docs site on GitHub Pages that describes the above files in more detail.

I adapted #Michael Altfield's solution into a single GitHub Action:
name: docs_pages_workflow
on:
push:
branches: [ main ]
jobs:
build_docs_job:
runs-on: ubuntu-latest
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout
uses: actions/checkout#v2.3.4
- name: Set up Python
uses: actions/setup-python#v2.2.1
with:
python-version: 3.9
- name: Install dependencies
run: |
python -m pip install -U sphinx
python -m pip install sphinx-rtd-theme
- name: make the sphinx docs
run: |
make -C docs clean
make -C docs html
- name: Init new repo in dist folder and commit
run: |
cd docs/build/html/
git init
touch .nojekyll
git add -A
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git commit -m 'deploy'
- name: Force push to destination branch
uses: ad-m/github-push-action#v0.5.0
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: gh-pages
force: true
directory: ./docs/build/html
Note, my Makefile builds to build not _build directory. The last line with directory: is saying to push from the .docs/build/html directory where we just created the new Git repo. This avoids his rsync and pushd commands. Otherwise the logic follows #Michael Altfield's solution.

Related

Github Actions: Cache Error and end action without fail message

I have a github action that transforms my Readme from one format to the other and which will then push the new Readme to the repository. For the pushing I have defined this job:
push_readme:
name: Push new Readme
needs: generate_readme
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Download readme result from job 1 generate_readme
uses: actions/download-artifact#v3
with:
name: readme
- name: Commit files
run: |
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git status
git add READMEmd.md
git commit -m "Actions Generated Readme"
- name: Push changes
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
The commit returns an error when there is nothing to commit - which happens whenever the readme was not updated in the recent push. This is expected and fine. However, I would like to handle this error properly s.t. the action simply ends when it occurs WITHOUT telling me it failed. Instead I'd like something in the sense of "There is no new README to commit. Ending the action".
Could anyone point me to how to do that? I failed to find the solution yet.
You can utilize Bash and check git diff for the README file and set an output parameter with GITHUB_OUTPUT for the next step to check if there indeed is a commit.
Here is an example:
- name: Commit files
id: commit
run: |
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git status
if [[ -n $(git diff README.md) ]]; then
git add README.md
git commit -m "Actions Generated Readme"
echo "DONE=true" >> $GITHUB_OUTPUT
else
echo "README is the same. Nothing to commit."
fi
- name: Push changes
if: ${{ steps.commit.DONE }}
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}

Update dependabots pull request with version increment for other files for an Eclipse plug-in project

I use dependabots with Maven environment to receive pull requests on pom.xml files for version upgrades on all dependencies.
However, for an Eclipse plug-in project, more files need update:
MANIFEST.MF
build.properties
How do I configure dependabots in an Eclipse plugin project so that not only the pom.xml is updated but also MANIFEST.MF and build.properties?
An open feature request exists on dependabots
https://github.com/dependabot/dependabot-core/issues/5676
Meanwhile, I'm trying to tweak this with a Github Action that will checkout the branch from dependabots pull request and replace the version number in the files.
I've built this workflow file:
name: "Update dependabots"
on:
push:
branches: [ 'dependabot/**' ]
# Set the access for individual scopes, or use permissions: write-all
permissions:
pull-requests: write
issues: write
repository-projects: write
jobs:
build:
runs-on: ubuntu-latest
if: "contains(github.event.head_commit.message, 'Bump')"
steps:
- name: Checkout repository
uses: actions/checkout#v3
with:
ref: ${{ github.ref }}
- name: Setup git user
run: |
git config --global user.email "action#github.com"
git config --global user.name "GitHub Action"
- name: Update Version
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
git log | grep -w Bump | cut -d' ' -f6,8,10 | xargs -n3 bash -c 'shopt -s globstar && sed -bi "s/$1-$2/$1-$3/g" **/MANIFEST.MF **/build.properties' sh-replace-in-files
git commit -a -m "Updated versions in **/MANIFEST.MF **/build.properties"
git push
But when the workflow runs, it complains that user github-actions[bot] is not allowed for my repository :
remote: Permission to myusername/myrepo.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/myusername/myrepo/': The requested URL returned error: 403
Error: Process completed with exit code 128.
I tried to replace GITHUB_TOKEN with a PAT_TOKEN but no luck with that.
What should I do ?

GitHub Action incapable of pushing due to "unsafe repository" error

I have a private GitHub repository that has a GitHub Action that pushes files which are created at the Action's runtime to the repository. Since yesterday (2022-04-12), the Action fails at the following step:
- name: Push data to repo
uses: github-actions-x/commit#v2.8
with:
push-branch: 'main'
commit-message: 'Add current data'
force-add: 'true'
files: *.zip
name: autoupdate
Running this step triggers the following error message:
Command line: | /usr/bin/git checkout -B main
Stderr: | fatal: unsafe repository ('/github/workspace' is owned by someone else)
| To add an exception for this directory, call:
|
| git config --global --add safe.directory /github/workspace
Based on the error message I added the following to my GitHub Action:
- name: Fix issue with repository ownership
run: |
git config --global --add safe.directory /home/runner/work/vksm/vksm
git config --global --add safe.directory /github/workspace
I have also added /home/runner/work/vksm/vksm as I was not sure if /github/workspace in the error message is meant as a generic path or not. /home/runner/work/vksm/vksm is where the checkout step puts the repository when the Action runs: /usr/bin/git init /home/runner/work/vksm/vksm
The whole sequence of steps is as follows:
steps:
- name: Checkout the repository
uses: actions/checkout#v2
- name: Fix issue with repository ownership
run: |
git config --global --add safe.directory /home/runner/work/vksm/vksm
git config --global --add safe.directory /github/workspace
- name: Set up Python 3.9
uses: actions/setup-python#v2
...
- name: Install dependencies
run: |
pip install requests
- name: Run Python script
run: |
python script.py
- name: Push data to repo
uses: github-actions-x/commit#v2.8
...
However, the error still occurs.
This questions is possibly related to Cannot add parent directory to safe.directory on git.
Windows 10
In my case "Unsafe repository is owned by someone else" resolved by command in mention folder
Use takeown from the command prompt to take ownership a folder, all its subfolders and files recursively.
takeown.exe /f . /r
This works well, but if you don't run your command line console as administrator it may fail for files you don't own.
This is happening because of a security vulnerability. The error is thrown inside the docker container before you can execute the git config commands to fix the unsafe repository problem. You need to modify the entrypoint of the docker container to execute the git command. You can check this link for details about the vulnerability.
A temporary workaround until git/action owners make a change could be to fork/clone the action that uses docker and modify it with something like this.
#!/bin/bash
set -o pipefail
# config
# ...
# Fix the unsafe repo error which was introduced by the CVE-2022-24765 git patches
git config --global --add safe.directory /github/workspace
#...
You can take a look at the comments in this issue for some ideas about a workaround.
I added the whole path to the current project:
git config --global --add safe.directory '/home/user/AndroidStudioProjects/MyRepoNameInc'
Then performed the push
git commit -m "first commit"
I had this issue with GitLab runner. spent hours doing everything. At last, it started to work after updating .gtlab-ci.yml .
Folder permissions were as below
/var/www/html/project - www-data:www-data
/var/www/html/projcect/.git - gitlab-runner:www-data
Updated script as below.
script:
- git config --global --add safe.directory /var/www/html/phase-3/public-ui

GitHub Actions with multiple private submodules

I'm trying to create a GH Actions job, which will download two submodules from private repositories. I want them to be downloaded with SSH keys which I have already generated.
I've been trying to it as so:
- uses: actions/checkout#v2
with:
submodules: repo_1
ssh-key: ${{ secrets.REPO_1 }}
- uses: actions/checkout#v2
with:
submodules: repo_2
ssh-key: ${{ secrets.REPO_2 }}
This code will create the folders of repo_1 and repo_2, but will be empty.
I have not found a possible solution. Does anyone know how to download multiple private submodules with separate SSH keys?
The documentation mentions:
# Whether to checkout submodules: `true` to checkout submodules or `recursive` to
# recursively checkout submodules.
#
# When the `ssh-key` input is not provided, SSH URLs beginning with
# `git#github.com:` are converted to HTTPS.
#
# Default: false
submodules: ''
So submodules: repo_2 should not be correct.
For instance, this is a workflow with a recursive checkout of submodules (inside an existing repository reference)
# Submodules recursive
- name: Checkout submodules recursive
uses: ./
with:
ref: test-data/v2/submodule-ssh-url
path: submodules-recursive
submodules: recursive
- name: Verify submodules recursive
run: __test__/verify-submodules-recursive.sh
It will checkout the repo github.com/actions/checkout branch test-data/v2%2Fsubmodule-ssh-url, which includes a .gitmodules with the names and SSH URL of the submodules.
To answer your original question:
change your .gitmodules URL with
repo1:org1/repo1
repo2:org2/repo2
Add GIT_SSH_COMMAND environment variable to ssh -F config, with config being a file with:
Host repo2
Hostname github.com
User git
IdentityFile key2
Host repo2
Hostname github.com
User git
IdentityFile key2
I don't know if it is possible to reference that file, generated with the right secrets.REPO_x, but what I can see from the checkout action is that you won"t have a native way to specify multiple keys for multiple submodule repositories.
Found a workaround:
steps:
- name: Switch to HTTPS git
run: |
rm -f ~/.gitconfig
git config --local --unset url."git#github.com".insteadOf https://github.com || echo "OK"
git config --local --unset url."git://".insteadOf https:// || echo "OK"
- uses: actions/checkout#v2
- name: Switch to SSH git
run: |
git config --local --replace-all url."git#github.com".insteadOf https://github.com
git config --local --add url."git://".insteadOf https://
- name: Checkout submodules
env:
GIT_SSH_COMMAND: "ssh -o StrictHostKeyChecking=no"
run: |
eval `ssh-agent -s`
echo "${{secrets.REPO_1}}" | ssh-add -
git submodule update --init repo_1
ssh-add -D
echo "${{secrets.REPO_2}}" | ssh-add -
git submodule update --init repo_2
ssh-add -D
eval `ssh-agent -k`

GitHub Action using ImageMagick: Create a PNG and commit it to the repo

Background
I'm maintaining a forked repository that contains a LaTeX project. The README.md contains a .png preview of the .pdf compiled from the example .tex file included in the repo. I often forget to update the .png version using the following ImageMagick command:
magick convert -density 1200 -background white -alpha off Twenty-Seconds-Icons_cv.pdf -quality 90 Twenty-Seconds-Icons_cv.png
Thus I would like to automate this process using GitHub Actions.
Workflow
I think the main.yml file should look something like this, however, I don't fully understand what I am doing.
name: PDF to PNG
on:
push:
branches:
- kaspar
pull_request:
branches:
- kaspar
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Install ImageMagick
run: sudo apt install imagemagick # Seems to work and already be included with ubuntu-latest...
- name: Convert PDF to PNG
run: # How?
- name: Commit the updated PNG
run: # How?
Output:
Run sudo apt install imagemagick
sudo apt install imagemagick
shell: /usr/bin/bash -e {0}
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
imagemagick is already the newest version (8:6.9.10.23+dfsg-2.1ubuntu11.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Questions
It seems like I can just run bash shell commands with run: ; is this true?
How/where can I access the .pdf file in the repo from inside the GitHub Actions Ubuntu machine?
Where should I save the .png file, is there a ~ directory?
How can I commit the .png file generated by ImageMagick to the repo?
Would git commit <file> -m "updated png version of the CV" work?
I figured out a solution to my problem:
I forgot to checkout the repository
ghostscript needed to be installed explicitly
The security policy for ImageMagic needs to be edited for it to process PDFs
On Ubuntu the ImageMagick command doesn’t work for 1200 dpi (900 is ok in my case), on Windows, this works just fine
I found out how to commit and push files
name: PDF to PNG
on:
push:
branches:
- kaspar
pull_request:
branches:
- kaspar
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Install ghostscript
run: sudo apt install ghostscript
- name: Change ImageMagick security policy
run: |
DQT='"'
SRC="rights=${DQT}none${DQT} pattern=${DQT}PDF${DQT}"
RPL="rights=${DQT}read\|write${DQT} pattern=${DQT}PDF${DQT}"
sudo sed -i "s/$SRC/$RPL/" /etc/ImageMagick-6/policy.xml
- name: Convert PDF to PNG
run: convert -density 900 -background white -alpha off Twenty-Seconds-Icons_cv.pdf -quality 90 Twenty-Seconds-Icons_cv.png
- name: Commit PNG
id: commit
run: |
git config --local user.email "action[bot]#github.com"
git config --local user.name "github-actions[bot]"
git add Twenty-Seconds-Icons_cv.png
if [-z "$(git status --porcelain)"]; then
echo "::set-output name=push::false"
else
git commit -m "[bot] updated Twenty-Seconds-Icons_cv.png"
echo "::set-output name=push::true"
fi
shell: bash
- name: Push Commit
if: steps.commit.outputs.push == 'true'
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.SECRET_TOKEN }}