I maintain a public repository on GitHub where changes are only made to a single YAML file. I'm looking for a solution to process that file on every push and generate files based on it. Essentially, a pipeline or CI should parse the file and create many different markdown files. These files (or more specifically, the changes to these files) should then be pushed back to the repository.
Requirements:
Manual changes to the YAML file and automatic changes to the markdown files should both be pushed to the master branch.
The version history should be kept (e.g. forced push might not work).
There is an arbitrary number of files that are generated.
There are Travis providers for GitHub Pages and GitHub Releases. But both have limitations that make them unsuitable for my requirements.
Using what tool/CI/pipeline can I achieve that on GitHub? I would prefer a service over a self-hosted CI.
Assuming that you already have the program/script to parse the YAML file and to generate the Markdown files, I can give you some insights on how I would do this from Jenkins CI. While I draw my experience from running my own instance, there are also hosted options such as CloudBees that you can explore.
Create a new Jenkins Freestyle project.
Under the Source Code Management section, configure your GitHub project coordinates.
Under Build Triggers section, activate the 'Build when a change is pushed to GitHub' option. That would launch the CI job at the moment you push a new version of the YAML file into the repository.
Under the build section, add an Execute shell build step.
In the shell step, launch the program or script that processes the YAML file/generates the .md files. End the script by adding the git add ., git commit -m "message", git pull and git push commands (assumes git is in the path).
Enable the new job to make it active in Jenkins.
You can do this now with the free GitHub Actions option for the repositories.
You need to put this step into your YAML file.
- name: Commit back to GitHub
run: |
git config --global user.name "github-actiuons[bot]"
git config --global user.email "41898282+github-actions[bot]#users.noreply.github.com"
git add -A
git commit -m "Updating some file"
git push
There are some items in the marketplace, but they didn't work for me.
The email of the bot is based on this thread:
https://github.community/t/github-actions-bot-email-address/17204
Update the commit message.
Be careful with the folder paths if you decide to push a specific file in a folder.
Related
I want to deploy my application in github page. I want the build job to be automated. My build commands basically include running unit tests,building a production version, gh-deploy command to deploy to github page. Can i achieve this using code ship. Do i require a actual vm like ec2 to do this?
The official automation process with codeship involves pushing to GitHub.
If your build process can populate the content you want to publish, then you can simply push it from the codeship job itself.
As an example, see "Auto-deploying to gh-pages with Codeship " (Martyn Chamberlin), which is about publishing a Gulp build:
git clone <the ssh link to your repo> dist
cd dist
git checkout gh-pages || git checkout --orphan gh-pages
cd ..
rm -rf dist/**/* || exit 0
npm install
gulp deploy
cd dist
git config user.email "<the email address you used for your machine user GitHub account>"
git config user.name "<the username you used for your machine user GitHub account>"
git add .
git commit -m "Deploy to GitHub Pages: ${CI_COMMIT_ID} --skip-ci"
git push origin gh-pages
This is a clever little set of commands, really.
It’s creating a git repo inside of another git repo in order to handle the build correctly.
I wish I could say I was the one who came up with this idea but the hat tip belongs to Domenic who gave me the idea with his Travis gist.
A few things to note about this:
This assumes that npm install; gulp deploy is the series of commands you use to combine and minify your assets.
That’s true of my project but yours may be different (particularly the second one). Be sure to replace lines 6 and 7 with your actual ones (as well as the values, of course).
This also assumes your output directory is dist. Hopefully it is, because that’s the convention, but if you’ve got something else, you’ll need to update lines 1, 2, and 8.
I’m completely bypassing Codeship’s recommendation of handling the final git push via their own provided code (“include the commands from continuous-deployments/git-push.sh”).
Not a big deal either way, I just don’t see the need for an extra HTTP request.
I am new to GitHub and I have to submit one assignment through GitHub.
I have been reading this blog for uploading files through github, but am not able to see any button labeled Upload files on my home screen of the repository
What am I missing?
See the "or push an existing repository from the command line" to push existant files to your repository. You won't have to load them by a form but with your command line from your project.
Github does not work like cloud storage platform. You actually need to have git installed on your computer in order to be able to push code to your repositories.
First of all, you need to install git( download links here ).
Then you need to init your local repositories with git init. Caution you need to run this command inside the directory your code is placed.
Run git add --all to stage the changes
Run git commit -m "your message" to commit your changes
Run git add remote origin (repo_link or ssh) to add your Github repo as a remote repository
Run git push -u origin master to push your master branch to remote origin
This might look a lot at first place but, you 'll get used to it really soon
Alternative
You can also the git and GitHub GUI which are much friendlier for a beginner.
I'm trying to write a job in Jenkins that pulls my latest code from a Github repo, zips it all up, and then pushes it to another builds Github repo. The Publisher event in post-build seems to only push back to the repo you specify in source code management. Is there a way I can push to a different/separate repo?
You can create the zip and than use a bash command with git push ...
Another option is to keep the zip in artifacts repository - Nexus or artifactory ...
How can I upload folders to GitHub? I have all of my code in a folder, containing 98 files, on my desktop. I know how to upload files, but it there a way to upload the entire folder?
Thanks!
This is Web GUI of a GitHub repository:
Drag and drop your folder to the above area. When you upload too much folder/files, GitHub will notice you:
Yowza, that’s a lot of files. Try again with fewer than 100 files.
and add commit message
And press button Commit changes is the last step.
You can also use the command line, Change directory where your folder is located then type the following :
git init
git add <folder1> <folder2> <etc.>
git commit -m "Your message about the commit"
git remote add origin https://github.com/yourUsername/yourRepository.git
git push -u origin master
git push origin master
I Understand where you are coming from.
The solution provided by #James Graham may not work in certain cases. The Drag and Drop Functionality may cease to exist. See below link when that happens:
https://www.reddit.com/r/github/comments/meuxtg/github_drag_and_drop_not_working/
If somebody wants to avoid the shell and all the commands and wants to have a UI to do that,Github Desktop is one of the way to go forward.
Steps to follow to install and use Github Desktop:
I am assuming you know the difference between local repo and remote repo
Install Github Desktop
Create a repository locally on your hard drive by using github desktop. This will automatically create files like .git and .gitattributes. It also asks to create a README.md file, always best practice is to create it and edit it informing readers about your project overview, installation steps etc. README.md is rendered in Markdown and can also render HTML. See more about Markdown here: Markdown Cheatsheet guide
Copy and Paste all the folders and files that you want to upload(basically the right terminology is "Push" ) into this newly created local repository. Be aware of the directory structure as the exact same directory structure will be replicated on your remote repository.
Go to github desktop, as soon as you paste files in the local repo, you will see them as changes here. All you need to do is commit these changes with a comment. This will be your "First or Initial Commit" to the repo.
Next Github repo will ask whether you want to publish these to its remote repository. Click "Publish" Note Publish is just a one time operations. Going forward any further changes you make to local repo will be seen in github desktop and you need to again follow the loop of "Commit local->Fetch from remote->Push to Remote. As long as you are the only developer working on a project you need not go into other mechanics of git branches etc.
To verify if your repo is published remotely login to your github profile on the web and see your repository sitting there. This your remote repo which you effectively created from your local repo by using Github desktop.
I've just gone through that process again. Always end up cloning the repo locally, upload the folder I want to have in that repo to that cloned location, commit the changes and then push it.
Note that if you're dealing with large files, you'll need to consider using something like Git LFS.
For those still using master, this was changed to main
echo "# repo-name" >> README.md
git init
git add README.md
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/username/repo-name.git
git push -u origin main
In an attempt to make my workflow a bit neater, I came across the source tag, which allows me to host my "dev work" in a separate folder than my _site.
This is awesome, but it does not appear to be honoured by Github pages? I feel like I am doing something wrong though, so just wanted to check. I couldn't find much about it online.
In this case, I don't explicitly need github to do it, but it would be great to have this consistent workflow for projects that do rely on github pages.
Thanks!
GH Pages overrides the source setting in the Jekyll config file:
https://help.github.com/articles/troubleshooting-github-pages-build-failures#source-setting
http://jekyllrb.com/docs/github-pages/#project-pages
You don't need to change source or destination parameters to version them separately. You can just version them in two different branches from the same repository.
Why versioning in different branches ?
If you need some plugins (generator, tag, ...) or build tasks (gulp, grunt, ...) that will not run on gh-pages, you will have to publish your Jekyll sources in a branch and your build results in a separate branch.
For a User / Organization site, it will be master for the site build, and sources (or any name you like) for the code. A User / Organization site will be versioned at github.com/userName/userName.github.io and hosted at http://userName.github.io (or with a custom domain)
For a Project site, it will be gh-pages for the site build, and master (or any name you like) for the code.
A Project site will be versioned at github.com/userName/projectName and hosted at http://userName.github.io/projectName.
Steps for User\Organization site
create a repository on github (eg: https://github.com/userName/userName.github.io)
go to command line and cd pathTo/yourJekyllSource
git init
git remote add origin git#github.com:userName/userName.github.io.git
jekyll new . creates your code base
in _config.yml, set the baseurl parameter to baseurl: ''
in .gitignore add _site, it will be versioned in the gh-pages branch
jekyll build will create the _site destination folder and build site into it.
git checkout -b sources
git add -A && git commit -m "jekyll base sources" commit your source code
git push origin sources push your sources in the sources branch
cd _site
touch .nojekyll, this file tells gh-pages that there is no need to process files
git init
git remote add origin git#github.com:userName/userName.github.io.git
git checkout master
git add -A && git commit -m "jekyll first build" commit your site code
git push origin master
And your good to go !
Your deploy can be made like this :
cd pathTo/yourJekyllSource
jekyll build
git add -A
git commit -m "your commit message"
cd _site
git add -A
git commit -m "your commit message"
Steps for a Project site are described here and I've made an automation Rakefile, it can do everything for you from setup to deploy.
Enjoy !