How to export log files from Travis CI to GitHUB? - github

I am using Travis CI.org (Public repo) to execute my Builds and log is printing on Travis Home Page (Log File). I want to extract the log file/ send the log file to Git HUB or to any other external open source tool to access it.
Could you please let us know how to achieve this?

We can deploy build artifacts to S3 : Paste the below code in .travis.yml file if you are using github and S3.
after_failure:
addons:
artifacts:
paths:
- $(git ls-files -o | tr "\n" ":")
deploy:
- provider: s3
- access_key_id: $ARTIFACTS_KEY
- secret_access_key: $ARTIFACTS_SECRET
- bucket: $ARTIFACTS_BUCKET
- skip_cleanup: true
- acl: public_read
Also, if you want to send it free open source tool, you can use chunk.io. Place the below code in a shell script and call this from after_failure section from .travis.yml file:
cd path/to/directory/where/untracked files store/
count=$(git ls-files -o | wc -l)
git ls-files -o
echo ">>>>>>>>> CONTAINERS LOG FILES <<<<<<<<<<<<"
for (( i=1; i<"$count";i++ ))
do
file=$(echo $(git ls-files -o | sed "${i}q;d"))
echo "$file"
cat $file | curl -sT - chunk.io
done
echo " >>>>> testsummary log file <<<< "
cat testsummary.log | curl -sT - chunk.io

Related

Bitbucket REST API to do search in remote private master repository

Is there any way I can search for a specific string in a private master repositories in butbucket.
Below is the code I use to get the clone command and then I download the files and do a grep on them now. But is it possible to search directlt? I need the output like
Full file Path, Search result (Line with the word I am searching for)
set -e
echo -n '' > clone-repos.sh
chmod +x clone-repos.sh
ONPREM_USER=user1
ONPREM_PASS=pass1
ONPREM_PROJECT=project1
curl -s -u "$ONPREM_USER:$ONPREM_PASS" https://bitbucket.bmogc.net/rest/api/1.0/projects/$ONPREM_PROJECT/repos/\?limit=1000 | ./jq-win64.exe -r '.values[] | {slug:.slug, links:.links.clone[] } | select(.links.name=="http") | "git clone \(.links.href) \(.slug)"' >> clone-repos.sh

Github actions/cache#v2: unrecognized option: posix in the post job

I'm using Github Actions to implement a CI pipeline in my project. Currently, I'm trying to use actions/cache#v2 to cache yarn cache dir to improve the pipeline time. Unfortunately, always that the actions/cache#v2 runs I'm getting an error in the post-job saying: /bin/tar: unrecognized option: posix. The complete log is:
Post job cleanup.
/usr/bin/docker exec 4decc52e7744d9ab2e81bb24c99a830acc848912515ef1e86fbb9b8d5049c9cf sh -c "cat /etc/*release | grep ^ID"
/bin/tar --posix -z -cf cache.tgz -P -C /__w/open-tuna-api/open-tuna-api --files-from manifest.txt
/bin/tar: unrecognized option: posix
BusyBox v1.31.1 () multi-call binary.
Usage: tar c|x|t [-ZzJjahmvokO] [-f TARFILE] [-C DIR] [-T FILE] [-X FILE] [--exclude PATTERN]... [FILE]...
Create, extract, or list files from a tar file
c Create
x Extract
t List
-f FILE Name of TARFILE ('-' for stdin/out)
-C DIR Change to DIR before operation
-v Verbose
-O Extract to stdout
-m Don't restore mtime
-o Don't restore user:group
-k Don't replace existing files
-Z (De)compress using compress
-z (De)compress using gzip
-J (De)compress using xz
-j (De)compress using bzip2
-a (De)compress using lzma
-h Follow symlinks
-T FILE File with names to include
-X FILE File with glob patterns to exclude
--exclude PATTERN Glob pattern to exclude
Warning: Tar failed with error: The process '/bin/tar' failed with exit code 1
I'm following the example of the official action cache repository. Here a snippet of my CI.yml
# Configure cache
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- uses: actions/cache#v2
id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
Because of the above error, the cache is not created and the pipeline time is not improved. I've tried changing the hasFiles expression and the entire key, but no success.
My question is: Am I making some mistake in the use of Action Cache? Can anyone help me with this issue? Thanks.
Your problem is that you're running inside an Alpine Linux-based container. Alpine Linux is designed for small size, and as a result it replaces many of the standard GNU utilities with those from busybox, a multi-call binary. Your version of tar is one of those.
The actions/cache#v2 action uses tar --posix, which tells tar to create a standard pax-format archive. pax archives are a form of tar archive that can handle arbitrarily long filenames, huge file sizes, and other types of metadata that tar archives cannot. This format is specified by POSIX and is a better choice than GNU tar-style archives because it works across a variety of systems and isn't specified by what one implementation does, in addition to being more featureful.
However, the version of tar shipped as part of busybox doesn't support the --posix option, and as a result this command fails. If you want to use the actions/cache#v2 GitHub Action, then you need to provide a version of GNU or BSD (libarchive) tar earlier in your PATH before running it so that that command can be used instead of busybox's.

There is a way to batch archive GitHub repositories based off of a search?

From the answer to a related question I know it's possible to batch clone repositories based on a GitHub search result:
# cheating knowing we currently have 9 pages
for i in {1..9}
do
curl "https://api.github.com/search/repositories?q=blazor+language:C%23&per_page=100&page=$i" \
| jq -r '.items[].ssh_url' >> urls.txt
done
cat urls.txt | xargs -P8 -L1 git clone
I also know that the Hub client allows me to make API calls.
hub api [-it] [-X METHOD] [-H HEADER] [--cache TTL] ENDPOINT [-F FIELD|--input FILE]
I guess the last step is, how do I archive a repository with Hub?
You can update a repository using the Update a Repository API call.
I put all my repositories in a TMP variable in the following way, and ran the following:
echo $TMP | xargs -P8 -L1 hub api -X PATCH -F archived=true
Here is a sample of what the $TMP variable looked like:
echo $TMP
/repos/amingilani/9bot
/repos/amingilani/advent-of-code-2019
/repos/amingilani/alan
/repos/amingilani/annotate_models

Initialise and pull terraform public modules using GitHub SSH private key

Context:
I have gitlab runners which are executing terraform init command which is pulling all necessary terraform modules. Recently, I started hitting github throttling issues (60 calls to github api per hour). So I am trying to reconfigure my pipeline so it uses Github user's private key.
Currently, I have the following in my pipeline but it still doesn't seem to work and private key isn't used to pull the terraform modules.
- GITHUB_SECRET=$(aws --region ${REGION} ssm get-parameters-by-path --path /github/umotifdev --with-decryption --query 'Parameters[*].{Name:Name,Value:Value}' --output json);
- PRIVATE_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/private_key").Value' | base64 -d);
- PUBLIC_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/public_key").Value' | base64 -d);
- mkdir -p ~/.ssh;
- echo "${PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa;
- chmod 700 ~/.ssh/id_rsa;
- eval $(ssh-agent -s);
- ssh-add ~/.ssh/id_rsa;
- ssh-keyscan -H 'github.com' >> ~/.ssh/known_hosts;
- ssh-keyscan github.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_host;
- echo -e "Host github.com\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config;
- echo ${PUBLIC_KEY} >> ~/.ssh/authorized_keys
The error I am seeing in my pipeline is something like (which is basically throttling from github):
Error: Failed to download module
Could not download module "vpc" (vpc.tf:17) source code from
"https://api.github.com/repos/terraform-aws-modules/terraform-aws-vpc/tarball/v2.21.0//*?archive=tar.gz":
bad response code: 403.
Anyone can advise how to resolve an issue where private key isn't used to pull terraform modules?

Pushing on a Git repository with Travis-CI

I want to execute a script every time I push into the master, this script will create some files that I want to commit and push. The log of the Travis build seems to be adding the files, committing and pushing, but nothing happens.
My .travis.yml is this one:
before_install:
- openssl aes-256-cbc -K $encrypted_290c3b2c061d_key -iv $encrypted_290c3b2c061d_iv -in id_rsa.enc -out /tmp/id_rsa -d
- eval "$(ssh-agent -s)" # Start the ssh agent
- chmod 600 /tmp/id_rsa
- ssh-add /tmp/id_rsa
install:
- wget --user $docencia_ac_username --password $docencia_ac_password http://docencia.ac.upc.edu/FIB/grau/PEC/Protegido/Documentacion/eines-sisa-64BITS.tgz
- tar -xf eines-sisa-64BITS.tgz
- export PATH=$PATH:$(pwd)/eines-sisa/bin
script:
- chmod +x ./compile_to_hex.sh
- "./compile_to_hex.sh"
after_success:
- rm -rf $TRAVIS_BUILD_DIR/tmp/
- git config --local user.name "Marc43"
- git config --local user.email "my mail"
- git add $TRAVIS_BUILD_DIR/hex/*
- git commit -m "At $(date) hex files builded - travis-ci [skip ci]"
- git push git#github.com:Marc43/sisa_hexbuilder.git master > /dev/null 2>&1
before_deploy:
- rm -rf eines-sisa*
in the git user.email it really goes my email but I decided to delete it for the question.
And the travis log for the build is:
$ git add $TRAVIS_BUILD_DIR/hex/*
$ git commit -m "At $(date) hex files builded - travis-ci [skip ci]"
[detached HEAD 10e7e48] At Sun Apr 15 08:06:17 UTC 2018 hex files builded - travis-ci [skip ci]
4 files changed, 21 insertions(+)
create mode 100644 hex/exemple.hex
create mode 100644 hex/joc_io.hex
create mode 100644 hex/tb_branch.hex
create mode 100644 hex/tb_sum.hex
I know that there is another way to do this via GitHub tokens or something like that, anyway I don't know how to do it one way or the other. I've tried to do it with the deploy too but it never uploaded my files, just tagged the same commit I did. Any ideas?
Thank you,
Marc
I run several repositories that need documentation compiling to distributable formats (e.g. AsciiDoc to HTML, MD to PDF), rather than having to build and commit every time I want to update the distributable, I’d like to automate this process. This is where I use TravisCI as a build server.
before_install:
- sudo apt-get install pandoc
- gem install asciidoctor
script:
- make
after_success:
- .travis/push.sh
env:
global:
secure: hZJlqgOzA2zIUJSWIka0PylqNaTkfHq+kS48RrHmocrK0vLyCW7ECWrzez2f2RVdTNzPi0b+yJq2uCbFfWjImZqg+XY1I75/CVVdSYMk7PJkYZ/iBDixMYY8CAkRRd5yZft9uZAdZzR4KLCPN18n7qfISv/M9VA8989NKcVyiEU=
push.sh
#!/bin/sh
setup_git() {
git config --global user.email "travis#travis-ci.org"
git config --global user.name "Travis CI"
}
commit_website_files() {
git checkout -b gh-pages
git add . *.html
git commit --message "Travis build: $TRAVIS_BUILD_NUMBER"
}
upload_files() {
git remote add origin-pages https://${GH_TOKEN}#github.com/MVSE-outreach/resources.git > /dev/null 2>&1
git push --quiet --set-upstream origin-pages gh-pages
}
setup_git
commit_website_files
upload_files
Reference - https://gist.github.com/willprice/e07efd73fb7f13f917ea