Code coverage for Swift 3.0 project in Gitlab CI - swift

I have an iOS project written by Swift 3.0 with CocoaPods. I've configured Gitlab CI for this project and it works perfectly. This is my .gitlab-ci.yml file:
stages:
- build
build_project:
stage: build
script:
- rm -rf Pods;rm -rf MyProject.xcworkspace;rm -rf MyProject.xcodeproj/project.xcworkspace
- pod install
- xcodebuild clean -workspace MyProject.xcworkspace -scheme MyProject | xcpretty
- xcodebuild test -workspace MyProject.xcworkspace -scheme MyProject -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' | xcpretty -s
tags:
- ios
- lunchdot
I can't see code coverage for this project in my Gitlab repo. At the moment a coverage column for all my builds is empty. I tried to set Test coverage parsing in CI/CD Pipelines Gitlab settings, but it hadn't any effect because I don't know regex for Swift. Is it possible to set up code coverage for Swift project in Gitlab CI? How can I do this?

So, I tried a million ways to do it and the issue turned out to be xcpretty. After I removed it I started getting consistent results with my gitlab ci coverage data. If you still do not want to get tons of data you do not care about, you can use -quiet in your gitlab yams file. I will post it all though, so keep reading.
One still needs an external tool for coverage analysis - xcov seems to not be available anymore so I used slather. Worked like a charm.
These are the steps:
1) Get slather.
2) Get your .slather.yml file straightened out. Mine looks like the following (YourApp is the name of your app obviously):
# .slather.yml
coverage_service: cobertura_xml
xcodeproj: ./YourApp.xcodeproj
workspace: ./YourApp.xcworkspace
scheme: YourApp
source_directory: ./YourApp
output_directory: path/to/xml_report
ignore:
- "**/Pods/*"
- "thirdparty/*"
- "./Scripts/*"
- "**/3rdParty/*"
- "../**/Developer/*"
- "**/libraries/*"
- "Module-Core/*"
- "Module-Components/*"
- "**/AppUnitTests/*"
You can get the test output as html, xml, in codecov.io, etc, totally up to you. Check out the slather GitHub page to see the possible ways of doing that. But for the current issue, all we need is slather reporting in the command line so gitlab can pick it up. That is where properly setting up the gitlab yaml file comes in.
3) Set up the .gitlab-ci.yml file. Mine looks like this:
stages:
- build
build_project_1:
stage: build
script:
- xcodebuild clean -workspace YourApp.xcworkspace -scheme YourApp -quiet
- xcodebuild test -workspace YourApp.xcworkspace -scheme YourApp -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.3.1' -enableCodeCoverage YES -quiet
after_script:
- slather coverage -s
only:
- master
4) Next step is to:
Go to your gitlab page/profile or whatever you call it
Go to Settings and then Pipelines
Scroll down to Test coverage parsing and add this regex expression there: \d+\%\s*$
And that is it. All you need to do is invoke a build.

Since Xcode 9.3 Apple provides xccov, so here is a solution using this, that does not rely on third party dependencies (and does not care if you use xcpretty, -quiet etc.)
Overview of xccovreport and xccov
When your Scheme has test coverage enabled (or when you pass -enableCodeCoverage YES to xcodebuild), an .xccovreport file is created. It contains the coverage percentages that you can see in Xcode UI.
The file is located in:
(for Xcode 9)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/E387E6E7-0AE8-4424-AFBA-EF9FX71A7E46.xccovreport
(for Xcode 10)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/Test-MyApp-2018.10.12_20-13-43-+0100.xcresult/action.xccovreport
(unless you specify a different folder in xcodebuild via -derivedDataPath)
Note: In Xcode 10, the .xccovreport file location is printed out in console after test finishes, but Xcode 9 does not do this. In any case, relying on this is probably not a good idea, as it might be silenced (e.g. by xcpretty)
The file isn't a plain text and to view it you have to call:
xcrun xccov view <path_to_xccovreport_file>
Which will output the report.
(You can pass --json for JSON report)
What we need to do
We want to be able to parse the file and print out the total percentage (so then GitLab can pick this up and use it in the dashboard).
There are a couple of challenges:
- Firstly we need to find out the file path of the xccovreport which contains random strings (in two places)
- Then we need to parse the file (using some regular expressions) and extract the total percentage.
The script
Here's what I am using (any improvement suggestions welcome, as I am not a bash expert):
#!/bin/bash
#1
# Xcode 10
TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
TEST_RESULTS_DIR=`ls -t $TEST_LOGS_DIR | grep "xcresult" | head -1`
TEST_COV_REPORT_FILENAME=`ls "$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test" | grep "xccovreport"`
TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test/$TEST_COV_REPORT_FILENAME"
# Xcode 9
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FILENAME=`ls $TEST_LOGS_DIR | grep "xccovreport"`
# TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_COV_REPORT_FILENAME"
# More general recursive search. Perhaps less likely to fail on new Xcode versions. Relies on filepaths containing timestamps that sort alphabetically correctly in time
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FULL_PATH=`find $TEST_LOGS_DIR -name '*.xccovreport' | sort -r | head -1`
#2
TOTAL_XCTEST_COVERAGE=`xcrun xccov view $TEST_COV_REPORT_FULL_PATH | grep '.app' | head -1 | perl -pe 's/.+?(\d+\.\d+%).+/\1/'`
#3
echo "TOTAL_XCTEST_COVERAGE=$TOTAL_XCTEST_COVERAGE"
What it does
#1 - gets the BUILD_DIR and then manipulates the path to get to the xccovreport file. Comment / uncomment the block for your version of Xcode.
#2 - We start with the full report as text. grep '.app' takes only the lines that contain .app. This is guaranteed to exist, because there is a line that reports the total coverage and contains MyApp.app. There will be multiple matches, but the first match will always be the overall total codecov score. So we use head -1 to take that first line of the grep result.
Now we have a line that looks like this:
MyApp.app 12.34% (8/65)
We use a perl regex to take only the “12.34%” part.
#3 - We simply print out the result (together with the variable name to make it easier to locate later in GitLab CI)
How to use
Replace MyApp.xcodeproj with your correct value
Make sure the correct logic is applied in step #1 (Xcode 9 / Xcode 10 / Generalized recursive search)
Save to a printCodeCov.sh file in the root of your project.
Make the file executable chmod +x printCodeCov.sh
In your .gitlab-ci.yml file, add a line to the script that says - ./printCodeCov.sh
In your GitLab Pipeline Settings, set the Test coverage parsing to TOTAL_XCTEST_COVERAGE=(.+)%
Notes
This does not use the --json format of xccov. For that version, see below.
This solution might be fragile, because of multiple assumptions about folder locations and report format
I use perl instead of sed because the latter was too difficult (BSD/GNU differences, regex limitations etc).
JSON version
If you'd rather use the JSON report (from xccov), then in the script you need something like this:
# Xcode 10
COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+?\.app.+?"lineCoverage":([^,]+).+/\1/'`
# Xcode 9
# COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+"targets"[^l]+lineCoverage":([^,]+),.+/\1/'`
COV_NUMBER_PERCENTAGE=`bc <<< $COV_NUMBER_FRACTION*100`
TOTAL_XCTEST_COVERAGE=`printf "%0.2f%%\n" $COV_NUMBER_PERCENTAGE`

Related

How to access Bitrise iOS test results from Swift DangerFile?

I'm trying to implement danger-swift but I don't know where I can access test results and flag failing ones from Bitrise.
I'm using a plugin for danger-swift called DangerXCodeSummary but I don't know where Bitrise stores test results from xcode-test#2.
DangerFile:
import Danger
import DangerXCodeSummary
import Foundation
let danger = Danger()
let testReportPath = "??" // What's the path for the test results?
XCodeSummary(filePath: testReportPath).report()
Bitrise script:
...
UnitTests:
before_run:
- _ensure_dependencies
after_run:
- _add_build_result_to_pr
steps:
- xcode-test#2: {} # What's the file path for the test results?
- deploy-to-bitrise-io#1: {}
envs:
- ots:
is_expand: false
BITRISE_PROJECT_PATH: MyApp.xcodeproj
- opts:
is_expand: false
BITRISE_SCHEME: AppTests
description: Unit Tests running at every commit.
...
_add_build_result_to_pr:
steps:
- script#1:
title: Commenting on the PR
is_always_run: true
inputs:
- content: |-
#!/usr/bin/env bash
# fail if any commands fails
set -e
echo "################### DANGER ######################"
echo "Install Danger"
brew install danger/tap/danger-swift
echo "Run danger"
danger-swift ci
echo "#################################################"
You can see what outputs the step generates on the Workflow Editor UI, and alternatively in the Step's repository in step.yml. For Xcode Test step specifically: https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L287
Am I right that you're looking for the xcresult test output? If you are, you can read it from the BITRISE_XCRESULT_PATH environment variable (https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L296), which is the output of Xcode Test (once Xcode Test is finished, it sets this environment variable to the xcresult's path). Keep in mind this is in .xcresult format (the official Xcode test result format).

How to force Devel::Cover to ignore a folder when using perl-helpers via Travis CI

The MetaCPAN Travis CI coverage builds are quite slow. See https://travis-ci.org/metacpan/metacpan-web/builds/238884497 This is likely in part because we're not successfully ignoring the /local folder that gets created by Carton as part of our build. See https://coveralls.io/builds/11809290
We're using perl-helpers to help with our Travis configuration. I thought I should be able to use the DEVEL_COVER_OPTIONS environment variable in order to fix this, but I guess I don't have the correct incantation. I've included the entire config below because a few snippets out of context seemed misleading.
language: perl
perl:
- "5.22"
matrix:
fast_finish: true
allow_failures:
- env: COVERAGE=1 USE_CPANFILE_SNAPSHOT=true
- env: USE_CPANFILE_SNAPSHOT=false HARNESS_VERBOSE=1
env:
global:
# Carton --deployment only works on the same version of perl
# that the snapshot was built from.
- DEPLOYMENT_PERL_VERSION=5.22
- DEVEL_COVER_OPTIONS="-ignore ^local/"
matrix:
# Get one passing run with coverage and one passing run with Test::Vars
# checks. If run together they more than double the build time.
- COVERAGE=1 USE_CPANFILE_SNAPSHOT=true
- USE_CPANFILE_SNAPSHOT=false HARNESS_VERBOSE=1
- USE_CPANFILE_SNAPSHOT=true
before_install:
- git clone git://github.com/travis-perl/helpers ~/travis-perl-helpers
- source ~/travis-perl-helpers/init
- npm install -g less js-beautify
# Pre-install from backpan to avoid upgrade breakage.
- cpanm -n http://cpan.metacpan.org/authors/id/M/ML/MLEHMANN/common-sense-3.6.tar.gz
- cpanm -n App::cpm Carton
install:
- cpan-install --coverage # installs converage prereqs, if enabled
- 'cpm install `test "${USE_CPANFILE_SNAPSHOT}" = "false" && echo " --resolver metadb" || echo " --resolver snapshot"`'
before_script:
- coverage-setup
script:
# Devel::Cover isn't in the cpanfile
# but if it's installed into the global dirs this should work.
- carton exec prove -lr -j$(test-jobs) t
after_success:
- coverage-report
notifications:
email:
recipients:
- olaf#seekrit.com
on_success: change
on_failure: always
irc: "irc.perl.org#metacpan-travis"
# Use newer travis infrastructure.
sudo: false
cache:
directories:
- local
The syntax for the Devel::Cover options on the command line is weird. You need to put stuff comma-separated. At least when you use PERL5OPT.
DEVEL_COVER_OPTIONS="-ignore,^local/"
See for example https://github.com/simbabque/AWS-S3/blob/master/.travis.yml#L26, where it's a whole lot of stuff with commas.
PERL5OPT=-MDevel::Cover=-ignore,"t/",+ignore,"prove",-coverage,statement,branch,condition,path,subroutine prove -lrs t

Spring Restdocs source gradle build fails during test

I have downloaded a copy of the 1.1.0-RELEASE tagged source code for Spring RESTdocs, but "gradlew build" is failing during the test phase. 273 of 502 tests are failing with variations on this error:
org.springframework.restdocs.request.RequestPartsSnippetTests > requestPartsWithOptionalColumn[Markdown] FAILED
java.lang.AssertionError:
Expected: is adoc snippetPart | Optional | Description
---- | -------- | -----------
a | true | one
b | false | two
but: was:Part | Optional | Description
---- | -------- | -----------
a | true | one
b | false | two
The problem looks to be that the string "adoc snippet" is prefixed to the start
of the expected output. I don't think that's right, although I can see in the AbstractContentSnippetMatcher.describeTo() why it's happening and it doesn't look very conditional so maybe it's the test's actual result that's wrong?
I have made no changes to the source code* but I don't see other people reporting this problem, so I'm mystified. I'm entirely new to gradle. Is there some kind of config I need to set up to make the tests pass? Should I be using a different target?
(OK... 1 teensy change: I removed the new-line-at-end-of-file check from the checkStyle - I'm downloading from Github onto a Windows PC.)
The problem is that the files in the zip have Unix-style line endings but, when run on Windows, Checkstyle and the tests expect Windows-style line endings.
Typically a Windows Git client will take care of this for you by converting the line endings when you check out the code. For example, the default configuration of Git for Windows is to check code out with Windows-style line endings but commit changes with Windows-style line endings.
You may be able to find a Windows utility that will batch-convert the line endings from LF to CRLF. Failing that, it's probably easiest to install a Git client (such as Git for Windows that I linked to above), ensure it's configure to perform line ending conversion, and then:
> git clone https://github.com/spring-projects/spring-restdocs
> cd spring-restdocs
> gradlew build

Can you get the number of lines of code from a GitHub repository?

In a GitHub repository you can see “language statistics”, which displays the percentage of the project that’s written in a language. It doesn’t, however, display how many lines of code the project consists of. Often, I want to quickly get an impression of the scale and complexity of a project, and the count of lines of code can give a good first impression. 500 lines of code implies a relatively simple project, 100,000 lines of code implies a very large/complicated project.
So, is it possible to get the lines of code written in the various languages from a GitHub repository, preferably without cloning it?
The question “Count number of lines in a git repository” asks how to count the lines of code in a local Git repository, but:
You have to clone the project, which could be massive. Cloning a project like Wine, for example, takes ages.
You would count lines in files that wouldn’t necessarily be code, like i13n files.
If you count just (for example) Ruby files, you’d potentially miss massive amount of code in other languages, like JavaScript. You’d have to know beforehand which languages the project uses. You’d also have to repeat the count for every language the project uses.
All in all, this is potentially far too time-intensive for “quickly checking the scale of a project”.
You can run something like
git ls-files | xargs wc -l
Which will give you the total count →
You can also add more instructions. Like just looking at the JavaScript files.
git ls-files | grep '\.js' | xargs wc -l
Or use this handy little tool → https://line-count.herokuapp.com/
A shell script, cloc-git
You can use this shell script to count the number of lines in a remote Git repository with one command:
#!/usr/bin/env bash
git clone --depth 1 "$1" temp-linecount-repo &&
printf "('temp-linecount-repo' will be deleted automatically)\n\n\n" &&
cloc temp-linecount-repo &&
rm -rf temp-linecount-repo
Installation
This script requires CLOC (“Count Lines of Code”) to be installed. cloc can probably be installed with your package manager – for example, brew install cloc with Homebrew. There is also a docker image published under mribeiro/cloc.
You can install the script by saving its code to a file cloc-git, running chmod +x cloc-git, and then moving the file to a folder in your $PATH such as /usr/local/bin.
Usage
The script takes one argument, which is any URL that git clone will accept. Examples are https://github.com/evalEmpire/perl5i.git (HTTPS) or git#github.com:evalEmpire/perl5i.git (SSH). You can get this URL from any GitHub project page by clicking “Clone or download”.
Example output:
$ cloc-git https://github.com/evalEmpire/perl5i.git
Cloning into 'temp-linecount-repo'...
remote: Counting objects: 200, done.
remote: Compressing objects: 100% (182/182), done.
remote: Total 200 (delta 13), reused 158 (delta 9), pack-reused 0
Receiving objects: 100% (200/200), 296.52 KiB | 110.00 KiB/s, done.
Resolving deltas: 100% (13/13), done.
Checking connectivity... done.
('temp-linecount-repo' will be deleted automatically)
171 text files.
166 unique files.
17 files ignored.
http://cloc.sourceforge.net v 1.62 T=1.13 s (134.1 files/s, 9764.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Perl 149 2795 1425 6382
JSON 1 0 0 270
YAML 2 0 0 198
-------------------------------------------------------------------------------
SUM: 152 2795 1425 6850
-------------------------------------------------------------------------------
Alternatives
Run the commands manually
If you don’t want to bother saving and installing the shell script, you can run the commands manually. An example:
$ git clone --depth 1 https://github.com/evalEmpire/perl5i.git
$ cloc perl5i
$ rm -rf perl5i
Linguist
If you want the results to match GitHub’s language percentages exactly, you can try installing Linguist instead of CLOC. According to its README, you need to gem install linguist and then run linguist. I couldn’t get it to work (issue #2223).
I created an extension for Google Chrome browser - GLOC which works for public and private repos.
Counts the number of lines of code of a project from:
project detail page
user's repositories
organization page
search results page
trending page
explore page
If you go to the graphs/contributors page, you can see a list of all the contributors to the repo and how many lines they've added and removed.
Unless I'm missing something, subtracting the aggregate number of lines deleted from the aggregate number of lines added among all contributors should yield the total number of lines of code in the repo. (EDIT: it turns out I was missing something after all. Take a look at orbitbot's comment for details.)
UPDATE:
This data is also available in GitHub's API. So I wrote a quick script to fetch the data and do the calculation:
'use strict';
async function countGithub(repo) {
const response = await fetch(`https://api.github.com/repos/${repo}/stats/contributors`)
const contributors = await response.json();
const lineCounts = contributors.map(contributor => (
contributor.weeks.reduce((lineCount, week) => lineCount + week.a - week.d, 0)
));
const lines = lineCounts.reduce((lineTotal, lineCount) => lineTotal + lineCount);
window.alert(lines);
}
countGithub('jquery/jquery'); // or count anything you like
Just paste it in a Chrome DevTools snippet, change the repo and click run.
Disclaimer (thanks to lovasoa):
Take the results of this method with a grain of salt, because for some repos (sorich87/bootstrap-tour) it results in negative values, which might indicate there's something wrong with the data returned from GitHub's API.
UPDATE:
Looks like this method to calculate total line numbers isn't entirely reliable. Take a look at orbitbot's comment for details.
You can clone just the latest commit using git clone --depth 1 <url> and then perform your own analysis using Linguist, the same software Github uses. That's the only way I know you're going to get lines of code.
Another option is to use the API to list the languages the project uses. It doesn't give them in lines but in bytes. For example...
$ curl https://api.github.com/repos/evalEmpire/perl5i/languages
{
"Perl": 274835
}
Though take that with a grain of salt, that project includes YAML and JSON which the web site acknowledges but the API does not.
Finally, you can use code search to ask which files match a given language. This example asks which files in perl5i are Perl. https://api.github.com/search/code?q=language:perl+repo:evalEmpire/perl5i. It will not give you lines, and you have to ask for the file size separately using the returned url for each file.
Not currently possible on Github.com or their API-s
I have talked to customer support and confirmed that this can not be done on github.com. They have passed the suggestion along to the Github team though, so hopefully it will be possible in the future. If so, I'll be sure to edit this answer.
Meanwhile, Rory O'Kane's answer is a brilliant alternative based on cloc and a shallow repo clone.
From the #Tgr's comment, there is an online tool :
https://codetabs.com/count-loc/count-loc-online.html
You can use tokei:
cargo install tokei
git clone --depth 1 https://github.com/XAMPPRocky/tokei
tokei tokei/
Output:
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
BASH 4 48 30 10 8
JSON 1 1430 1430 0 0
Shell 1 49 38 1 10
TOML 2 78 65 4 9
-------------------------------------------------------------------------------
Markdown 4 1410 0 1121 289
|- JSON 1 41 41 0 0
|- Rust 1 47 38 5 4
|- Shell 1 19 16 0 3
(Total) 1517 95 1126 296
-------------------------------------------------------------------------------
Rust 19 3750 3123 119 508
|- Markdown 12 358 5 302 51
(Total) 4108 3128 421 559
===============================================================================
Total 31 6765 4686 1255 824
===============================================================================
Tokei has support for badges:
Count Lines
[![](https://tokei.rs/b1/github/XAMPPRocky/tokei)](https://github.com/XAMPPRocky/tokei)
By default the badge will show the repo's LoC(Lines of Code), you can also specify for it to show a different category, by using the ?category= query string. It can be either code, blanks, files, lines, comments.
Count Files
[![](https://tokei.rs/b1/github/XAMPPRocky/tokei?category=files)](https://github.com/XAMPPRocky/tokei)
You can use GitHub API to get the sloc like the following function
function getSloc(repo, tries) {
//repo is the repo's path
if (!repo) {
return Promise.reject(new Error("No repo provided"));
}
//GitHub's API may return an empty object the first time it is accessed
//We can try several times then stop
if (tries === 0) {
return Promise.reject(new Error("Too many tries"));
}
let url = "https://api.github.com/repos" + repo + "/stats/code_frequency";
return fetch(url)
.then(x => x.json())
.then(x => x.reduce((total, changes) => total + changes[1] + changes[2], 0))
.catch(err => getSloc(repo, tries - 1));
}
Personally I made an chrome extension which shows the number of SLOC on both github project list and project detail page. You can also set your personal access token to access private repositories and bypass the api rate limit.
You can download from here https://chrome.google.com/webstore/detail/github-sloc/fkjjjamhihnjmihibcmdnianbcbccpnn
Source code is available here https://github.com/martianyi/github-sloc
Hey all this is ridiculously easy...
Create a new branch from your first commit
When you want to find out your stats, create a new PR from main
The PR will show you the number of changed lines - as you're doing a PR from the first commit all your code will be counted as new lines
And the added benefit is that if you don't approve the PR and just leave it in place, the stats (No of commits, files changed and total lines of code) will simply keep up-to-date as you merge changes into main. :) Enjoy.
Firefox add-on Github SLOC
I wrote a small firefox addon that prints the number of lines of code on github project pages: Github SLOC
npm install sloc -g
git clone --depth 1 https://github.com/vuejs/vue/
sloc ".\vue\src" --format cli-table
rm -rf ".\vue\"
Instructions and Explanation
Install sloc from npm, a command line tool (Node.js needs to be installed).
npm install sloc -g
Clone shallow repository (faster download than full clone).
git clone --depth 1 https://github.com/facebook/react/
Run sloc and specifiy the path that should be analyzed.
sloc ".\react\src" --format cli-table
sloc supports formatting the output as a cli-table, as json or csv. Regular expressions can be used to exclude files and folders (Further information on npm).
Delete repository folder (optional)
Powershell: rm -r -force ".\react\" or on Mac/Unix: rm -rf ".\react\"
Screenshots of the executed steps (cli-table):
sloc output (no arguments):
It is also possible to get details for every file with the --details option:
sloc ".\react\src" --format cli-table --details
Open terminal and run the following:
curl -L "https://api.codetabs.com/v1/loc?github=username/reponame"
If the question is "can you quickly get NUMBER OF LINES of a github repo", the answer is no as stated by the other answers.
However, if the question is "can you quickly check the SCALE of a project", I usually gauge a project by looking at its size. Of course the size will include deltas from all active commits, but it is a good metric as the order of magnitude is quite close.
E.g.
How big is the "docker" project?
In your browser, enter api.github.com/repos/ORG_NAME/PROJECT_NAME
i.e. api.github.com/repos/docker/docker
In the response hash, you can find the size attribute:
{
...
size: 161432,
...
}
This should give you an idea of the relative scale of the project. The number seems to be in KB, but when I checked it on my computer it's actually smaller, even though the order of magnitude is consistent. (161432KB = 161MB, du -s -h docker = 65MB)
Pipe the output from the number of lines in each file to sort to organize files by line count.
git ls-files | xargs wc -l |sort -n
This is so easy if you are using Vscode and you clone the project first. Just install the Lines of Code (LOC) Vscode extension and then run LineCount: Count Workspace Files from the Command Pallete.
The extension shows summary statistics by file type and it also outputs result files with detailed information by each folder.
There in another online tool that counts lines of code for public and private repos without having to clone/download them - https://klock.herokuapp.com/
None of the answers here satisfied my requirements. I only wanted to use existing utilities. The following script will use basic utilities:
Git
GNU or BSD awk
GNU or BSD sed
Bash
Get total lines added to a repository (subtracts lines deleted from lines added).
#!/bin/bash
git diff --shortstat 4b825dc642cb6eb9a060e54bf8d69288fbee4904 HEAD | \
sed 's/[^0-9,]*//g' | \
awk -F, '!($2 > 0) {$2="0"};!($3 > 0) {$3="0"}; {print $2-$3}'
Get lines of code filtered by specified file types of known source code (e.g. *.py files or add more extensions, etc).
#!/bin/bash
git diff --shortstat 4b825dc642cb6eb9a060e54bf8d69288fbee4904 HEAD -- *.{py,java,js} | \
sed 's/[^0-9,]*//g' | \
awk -F, '!($2 > 0) {$2="0"};!($3 > 0) {$3="0"}; {print $2-$3}'
4b825dc642cb6eb9a060e54bf8d69288fbee4904 is the id of the "empty tree" in Git and it's always available in every repository.
Sources:
My own scripting
How to get Git diff of the first commit?
Is there a way of having git show lines added, lines changed and lines removed?
shields.io has a badge that can count up all the lines for you here. Here is an example of what it looks like counting the Raycast extensions repo:
You can use sourcegraph, an open source search engine for code. It can connect to your GitHub account, index the content, and then on the admin section you would see the number of lines of code indexed.
I made an NPM package specifically for this usage, which allows you to call a CLI tool and providing the directory path and the folders/files to ignore
it goes like this:
npm i -g #quasimodo147/countlines
to get the $ countlines command in your terminal
then you can do
countlines . node_modules build dist

Download latest GitHub release

I'd like to have "Download Latest Version" button on my website which would represent the link to the latest release (stored at GitHub Releases). I tried to create release tag named "latest", but it became complicated when I tried to load new release (confusion with tag creation date, tag interchanging, etc.). Updating download links on my website manually is also a time-consuming and scrupulous task. I see the only way - redirect all download buttons to some html, which in turn will redirect to the actual latest release.
Note that my website is hosted at GitHub Pages (static hosting), so I simply can't use server-side scripting to generate links. Any ideas?
You don't need any scripting to generate a download link for the latest release. Simply use this format:
https://github.com/:owner/:repo/zipball/:branch
Examples:
https://github.com/webix-hub/tracker/zipball/master
https://github.com/iDoRecall/selection-menu/zipball/gh-pages
If for some reason you want to obtain a link to the latest release download, including its version number, you can obtain that from the get latest release API:
GET /repos/:owner/:repo/releases/latest
Example:
$.get('https://api.github.com/repos/idorecall/selection-menu/releases/latest', function (data) {
$('#result').attr('href', data.zipball_url);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<a id="result">Download latest release (.ZIP)</a>
Github now provides a "Latest release" button on the release page of a project, after you have created your first release.
In the example you gave, this button links to https://github.com/reactiveui/ReactiveUI/releases/latest
You can use the following where:
${Organization} as the GitHub user or organization
${Repository} is the repository name
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball > ${Repository}.tar.gz
The top level directory in the .tar.gz file has the sha hash of the commit in the directory name which can be a problem if you need an automated way to change into the resulting directory and do something.
The method below will strip this out, and leave the files in a folder with a predictable name.
mkdir ${Repository}
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball | tar -zxv -C ${Repository} --strip-components=1
Since February 18th, 2015, the GitHUb V3 release API has a get latest release API.
GET /repos/:owner/:repo/releases/latest
See also "Linking to releases".
Still, the name of the asset can be tricky.
Git-for-Windows, for instance, requires a command like:
curl -IkLs -o NUL -w %{url_effective} \
https://github.com/git-for-windows/git/releases/latest|\
grep -o "[^/]*$"| sed "s/v//g"|\
xargs -I T echo \
https://github.com/git-for-windows/git/releases/download/vT/PortableGit-T-64-bit.7z.exe \
-o PortableGit-T-64-bit.7z.exe| \
sed "s/.windows.1-64/-64/g"|sed "s/.windows.\(.\)-64/.\1-64/g"|\
xargs curl -kL
The first 3 lines extract the latest version 2.35.1.windows.2
The rest will build the right URL
https://github.com/git-for-windows/git/releases/download/
v2.35.1.windows.2/PortableGit-2.35.1.2-64-bit.7z.exe
^^^^^^^^^^^^^^^^^ ^^^^^^^^^
Maybe could you use some client-side scripting and dynamically generate the target of the link by invoking the GitHub api, through some JQuery magic?
The Releases API exposes a way to retrieve the list of all the releases from a repository. For instance, this link return a Json formatted list of all the releases of the ReactiveUI project.
Extracting the first one would return the latest release.
Within this payload:
The html_url attribute will hold the first part of the url to build (ie. https://github.com/{owner}/{repository}/releases/{version}).
The assets array will list of the downloadable archives. Each asset will bear a name attribute
Building the target download url is only a few string operations away.
Insert the download/ keyword between the releases/ segment from the html_url and the version number
Append the name of the asset to download
Resulting url will be of the following format: https://github.com/{owner}/{repository}/releases/download/{version}/name_of_asset
For instance, regarding the Json payload from the link ReactiveUI link above, we've got html_url: "https://github.com/reactiveui/ReactiveUI/releases/5.99.0" and one asset with name: "ReactiveUI.6.0.Preview.1.zip".
As such, the download url is https://github.com/reactiveui/ReactiveUI/releases/download/5.99.0/ReactiveUI.6.0.Preview.1.zip
If you using PHP try follow code:
function getLatestTagUrl($repository, $default = 'master') {
$file = #json_decode(#file_get_contents("https://api.github.com/repos/$repository/tags", false,
stream_context_create(['http' => ['header' => "User-Agent: Vestibulum\r\n"]])
));
return sprintf("https://github.com/$repository/archive/%s.zip", $file ? reset($file)->name : $default);
}
Function usage example
echo 'Download';
As I didn't see the answer here, but it was quite helpful for me while running continuous integration tests, this one-liner that only requires you to have curl will allow to search the Github repo's releases to download the latest version
https://gist.github.com/steinwaywhw/a4cd19cda655b8249d908261a62687f8
I use it to run PHPSTan on our repository using the following script
https://gist.github.com/rvanlaak/7491f2c4f0c456a93f90e31774300b62
If you are trying to download form any linux — even old or tiny versions — or are trying to download from a bash script then the failproof way is using this command:
wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}' | xargs wget
do not forget to replace $OWNER and $REPO with the right owner and repository names. The command downloads a json page with the data of the latest release. then awk gets the value from the browser_download_url key.
If you are in a really old linux or a tiny embedded system with a small wget, the download name can be a problem. In such case you can always use the ultra-reliable:
URL=$(wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}'); wget $URL -O $(basename "$URL")
As noted by #Dan Dascalescu in a comment to accepted answer, there are some projects (roughly 30%) which do not bother to file formal releases, so neither "Latest release" button nor /releases/latest API call would return useful data.
To reliably fetch the latest release for a GitHub project, you can use lastversion.