I put a file that was previously being tracked by Git onto the .gitignore list. However, the file still shows up in git status after it is edited. How do I force Git to completely forget the file?
.gitignore will prevent untracked files from being added (without an add -f) to the set of files tracked by Git. However, Git will continue to track any files that are already being tracked.
To stop tracking a file, we must remove it from the index:
git rm --cached <file>
To remove a folder and all files in the folder recursively:
git rm -r --cached <folder>
The removal of the file from the head revision will happen on the next commit.
WARNING: While this will not remove the physical file from your local machine, it will remove the files from other developers' machines on their next git pull.
The series of commands below will remove all of the items from the Git index (not from the working directory or local repository), and then will update the Git index, while respecting Git ignores. PS. Index = Cache
First:
git rm -r --cached .
git add .
Then:
git commit -am "Remove ignored files"
Or as a one-liner:
git rm -r --cached . && git add . && git commit -am "Remove ignored files"
git update-index does the job for me:
git update-index --assume-unchanged <file>
Note: This solution is actually independent of .gitignore as gitignore is only for untracked files.
Update, a better option
Since this answer was posted, a new option has been created and that should be preferred. You should use --skip-worktree which is for modified tracked files that the user don't want to commit anymore and keep --assume-unchanged for performance to prevent git to check status of big tracked files. See https://stackoverflow.com/a/13631525/717372 for more details...
git update-index --skip-worktree <file>
To cancel
git update-index --no-skip-worktree <file>
git ls-files -c --ignored --exclude-standard -z | xargs -0 git rm --cached
git commit -am "Remove ignored files"
This takes the list of the ignored files, removes them from the index, and commits the changes.
Move it out, commit, and then move it back in.
This has worked for me in the past, but there is probably a 'gittier' way to accomplish this.
I always use this command to remove those untracked files.
One-line, Unix-style, clean output:
git ls-files --ignored --exclude-standard | sed 's/.*/"&"/' | xargs git rm -r --cached
It lists all your ignored files, replaces every output line with a quoted line instead to handle paths with spaces inside, and passes everything to git rm -r --cached to remove the paths/files/directories from the index.
The copy/paste (one-liner) answer is:
git rm --cached -r .; git add .; git status; git commit -m "Ignore unwanted files"
This command will NOT change the content of the .gitignore file. It will just ignore the files that have already been committed to a Git repository, but now we have added them to .gitignore.
The command git status; is to review the changes and could be dropped.
Ultimately, it will immediately commit the changes with the message "Ignore unwanted files".
If you don't want to commit the changes, drop the last part of the command (git commit -m "Ignore unwanted files")
Use this when:
You want to untrack a lot of files, or
You updated your .gitignore file
Source: Untrack files already added to Git repository based on .gitignore
Let’s say you have already added/committed some files to your Git repository and you then add them to your .gitignore file; these files will still be present in your repository index. This article we will see how to get rid of them.
Step 1: Commit all your changes
Before proceeding, make sure all your changes are committed, including your .gitignore file.
Step 2: Remove everything from the repository
To clear your repository, use:
git rm -r --cached .
rm is the remove command
-r will allow recursive removal
–cached will only remove files from the index. Your files will still be there.
The rm command can be unforgiving. If you wish to try what it does beforehand, add the -n or --dry-run flag to test things out.
Step 3: Readd everything
git add .
Step 4: Commit
git commit -m ".gitignore fix"
Your repository is clean :)
Push the changes to your remote to see the changes effective there as well.
If you cannot git rm a tracked file because other people might need it (warning, even if you git rm --cached, when someone else gets this change, their files will be deleted in their filesystem). These are often done due to config file overrides, authentication credentials, etc. Please look at https://gist.github.com/1423106 for ways people have worked around the problem.
To summarize:
Have your application look for an ignored file config-overide.ini and use that over the committed file config.ini (or alternately, look for ~/.config/myapp.ini, or $MYCONFIGFILE)
Commit file config-sample.ini and ignore file config.ini, have a script or similar copy the file as necessary if necessary.
Try to use gitattributes clean/smudge magic to apply and remove the changes for you, for instance smudge the config file as a checkout from an alternate branch and clean the config file as a checkout from HEAD. This is tricky stuff, I don't recommend it for the novice user.
Keep the config file on a deploy branch dedicated to it that is never merged to master. When you want to deploy/compile/test you merge to that branch and get that file. This is essentially the smudge/clean approach except using human merge policies and extra-git modules.
Anti-recommentation: Don't use assume-unchanged, it will only end in tears (because having git lie to itself will cause bad things to happen, like your change being lost forever).
I accomplished this by using git filter-branch. The exact command I used was taken from the man page:
WARNING: this will delete the file from your entire history
git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD
This command will recreate the entire commit history, executing git rm before each commit and so will get rid of the specified file. Don't forget to back it up before running the command as it will be lost.
What didn't work for me
(Under Linux), I wanted to use the posts here suggesting the ls-files --ignored --exclude-standard | xargs git rm -r --cached approach. However, (some of) the files to be removed had an embedded newline/LF/\n in their names. Neither of the solutions:
git ls-files --ignored --exclude-standard | xargs -d"\n" git rm --cached
git ls-files --ignored --exclude-standard | sed 's/.*/"&"/' | xargs git rm -r --cached
cope with this situation (get errors about files not found).
So I offer
git ls-files -z --ignored --exclude-standard | xargs -0 git rm -r --cached
git commit -am "Remove ignored files"
This uses the -z argument to ls-files, and the -0 argument to xargs to cater safely/correctly for "nasty" characters in filenames.
In the manual page git-ls-files(1), it states:
When -z option is not used, TAB, LF, and backslash characters in
pathnames are represented as \t, \n, and \\, respectively.
so I think my solution is needed if filenames have any of these characters in them.
Do the following steps for a file/folder:
Remove a File:
need to add that file to .gitignore.
need to remove that file using the command (git rm --cached file name).
need to run (git add .).
need to (commit -m) "file removed".
and finally, (git push).
For example:
I want to delete the test.txt file. I accidentally pushed to GitHub and want to remove it. Commands will be as follows:
First, add "test.txt" in file .gitignore
git rm --cached test.txt
git add .
git commit -m "test.txt removed"
git push
Remove Folder:
need to add that folder to file .gitignore.
need to remove that folder using the command (git rm -r --cached folder name).
need to run (git add .).
need to (commit -m) "folder removed".
and finally, (git push).
For example:
I want to delete the .idea folder/directory. I accidentally pushed to GitHub and want to remove it. The commands will be as follows:
First, add .idea in file .gitignore
git rm -r --cached .idea
git add .
git commit -m ".idea removed"
git push
Update your .gitignore file – for instance, add a folder you don't want to track to .gitignore.
git rm -r --cached . – Remove all tracked files, including wanted and unwanted. Your code will be safe as long as you have saved locally.
git add . – All files will be added back in, except those in .gitignore.
Hat tip to #AkiraYamamoto for pointing us in the right direction.
Do the following steps serially, and you will be fine.
Remove the mistakenly added files from the directory/storage. You can use the "rm -r" (for Linux) command or delete them by browsing the directories. Or move them to another location on your PC. (You maybe need to close the IDE if running for moving/removing.)
Add the files / directories to the .gitignore file now and save it.
Now remove them from the Git cache by using these commands (if there is more than one directory, remove them one by one by repeatedly issuing this command)
git rm -r --cached path-to-those-files
Now do a commit and push by using the following commands. This will remove those files from Git remote and make Git stop tracking those files.
git add .
git commit -m "removed unnecessary files from Git"
git push origin
I think, that maybe Git can't totally forget about a file because of its conception (section "Snapshots, Not Differences").
This problem is absent, for example, when using CVS. CVS stores information as a list of file-based changes. Information for CVS is a set of files and the changes made to each file over time.
But in Git every time you commit, or save the state of your project, it basically takes a picture of what all your files look like at that moment and stores a reference to that snapshot. So, if you added file once, it will always be present in that snapshot.
These two articles were helpful for me:
git assume-unchanged vs skip-worktree and How to ignore changes in tracked files with Git
Basing on it I do the following, if the file is already tracked:
git update-index --skip-worktree <file>
From this moment all local changes in this file will be ignored and will not go to remote. If the file is changed on remote, conflict will occur, when git pull. Stash won't work. To resolve it, copy the file content to the safe place and follow these steps:
git update-index --no-skip-worktree <file>
git stash
git pull
The file content will be replaced by the remote content. Paste your changes from the safe place to the file and perform again:
git update-index --skip-worktree <file>
If everyone, who works with the project, will perform git update-index --skip-worktree <file>, problems with pull should be absent. This solution is OK for configurations files, when every developer has their own project configuration.
It is not very convenient to do this every time, when the file has been changed on remote, but it can protect it from overwriting by remote content.
Using the git rm --cached command does not answer the original question:
How do you force git to completely forget about [a file]?
In fact, this solution will cause the file to be deleted in every other instance of the repository when executing a git pull!
The correct way to force Git to forget about a file is documented by GitHub here.
I recommend reading the documentation, but basically:
git fetch --all
git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch full/path/to/file' --prune-empty --tag-name-filter cat -- --all
git push origin --force --all
git push origin --force --tags
git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
Just replace full/path/to/file with the full path of the file. Make sure you've added the file to your .gitignore file.
You'll also need to (temporarily) allow non-fast-forward pushes to your repository, since you're changing your Git history.
Move or copy the file to a safe location, so you don't lose it. Then 'git rm' the file and commit.
The file will still show up if you revert to one of those earlier commits, or another branch where it has not been removed. However, in all future commits, you will not see the file again. If the file is in the Git ignore, then you can move it back into the folder, and Git won't see it.
The answer from Matt Frear was the most effective IMHO. The following is just a PowerShell script for those on Windows to only remove files from their Git repository that matches their exclusion list.
# Get files matching exclusionsfrom .gitignore
# Excluding comments and empty lines
$ignoreFiles = gc .gitignore | ?{$_ -notmatch "#"} | ?{$_ -match "\S"} | % {
$ignore = "*" + $_ + "*"
(gci -r -i $ignore).FullName
}
$ignoreFiles = $ignoreFiles| ?{$_ -match "\S"}
# Remove each of these file from Git
$ignoreFiles | % { git rm $_}
git add .
The accepted answer does not "make Git "forget" about a file..." (historically). It only makes Git ignore the file in the present/future.
This method makes Git completely forget ignored files (past/present/future), but it does not delete anything from the working directory (even when re-pulled from remote).
This method requires usage of file /.git/info/exclude (preferred) or a pre-existing .gitignore in all the commits that have files to be ignored/forgotten. 1
All methods of enforcing Git ignore behavior after-the-fact effectively rewrite history and thus have significant ramifications for any public/shared/collaborative repositories that might be pulled after this process. 2
General advice: start with a clean repository - everything committed, nothing pending in working directory or index, and make a backup!
Also, the comments/revision history of this answer (and revision history of this question) may be useful/enlightening.
#Commit up-to-date .gitignore (if not already existing)
#This command must be run on each branch
git add .gitignore
git commit -m "Create .gitignore"
#Apply standard Git ignore behavior only to the current index, not the working directory (--cached)
#If this command returns nothing, ensure /.git/info/exclude AND/OR .gitignore exist
#This command must be run on each branch
git ls-files -z --ignored --exclude-standard | xargs -0 git rm --cached
#Commit to prevent working directory data loss!
#This commit will be automatically deleted by the --prune-empty flag in the following command
#This command must be run on each branch
git commit -m "ignored index"
#Apply standard git ignore behavior RETROACTIVELY to all commits from all branches (--all)
#This step WILL delete ignored files from working directory UNLESS they have been dereferenced from the index by the commit above
#This step will also delete any "empty" commits. If deliberate "empty" commits should be kept, remove --prune-empty and instead run git reset HEAD^ immediately after this command
git filter-branch --tree-filter 'git ls-files -z --ignored --exclude-standard | xargs -0 git rm -f --ignore-unmatch' --prune-empty --tag-name-filter cat -- --all
#List all still-existing files that are now ignored properly
#If this command returns nothing, it's time to restore from backup and start over
#This command must be run on each branch
git ls-files --other --ignored --exclude-standard
Finally, follow the rest of this GitHub guide (starting at step 6) which includes important warnings/information about the commands below.
git push origin --force --all
git push origin --force --tags
git for-each-ref --format="delete %(refname)" refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
Other developers that pull from the now-modified remote repository should make a backup and then:
#fetch modified remote
git fetch --all
#"Pull" changes WITHOUT deleting newly-ignored files from working directory
#This will overwrite local tracked files with remote - ensure any local modifications are backed-up/stashed
git reset FETCH_HEAD
Footnotes
1 Because /.git/info/exclude can be applied to all historical commits using the instructions above, perhaps details about getting a .gitignore file into the historical commit(s) that need it is beyond the scope of this answer. I wanted a proper .gitignore file to be in the root commit, as if it was the first thing I did. Others may not care since /.git/info/exclude can accomplish the same thing regardless where the .gitignore file exists in the commit history, and clearly rewriting history is a very touchy subject, even when aware of the ramifications.
FWIW, potential methods may include git rebase or a git filter-branch that copies an external .gitignore into each commit, like the answers to this question.
2 Enforcing Git ignore behavior after-the-fact by committing the results of a stand-alone git rm --cached command may result in newly-ignored file deletion in future pulls from the force-pushed remote. The --prune-empty flag in the following git filter-branch command avoids this problem by automatically removing the previous "delete all ignored files" index-only commit. Rewriting Git history also changes commit hashes, which will wreak havoc on future pulls from public/shared/collaborative repositories. Please understand the ramifications fully before doing this to such a repository. This GitHub guide specifies the following:
Tell your collaborators to rebase, not merge, any branches they created off of your old (tainted) repository history. One merge commit could reintroduce some or all of the tainted history that you just went to the trouble of purging.
Alternative solutions that do not affect the remote repository are git update-index --assume-unchanged </path/file> or git update-index --skip-worktree <file>, examples of which can be found here.
In my case I needed to put ".envrc" in the .gitignore file.
And then I used:
git update-index --skip-worktree .envrc
git rm --cached .envrc
And the file was removed.
Then I committed again, telling that the file was removed.
But when I used the command git log -p, the content of the file (which was secret credentials of the Amazon S3) was showing the content which was removed and I don't want to show this content ever on the history of the Git repository.
Then I used this command:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch .envrc' HEAD
And I don't see the content again.
I liked JonBrave's answer, but I have messy enough working directories that commit -a scares me a bit, so here's what I've done:
git config --global alias.exclude-ignored '!git ls-files -z --ignored --exclude-standard | xargs -0 git rm -r --cached && git ls-files -z --ignored --exclude-standard | xargs -0 git stage && git stage .gitignore && git commit -m "new gitignore and remove ignored files from index"'
Breaking it down:
git ls-files -z --ignored --exclude-standard | xargs -0 git rm -r --cached
git ls-files -z --ignored --exclude-standard | xargs -0 git stage
git stage .gitignore
git commit -m "new gitignore and remove ignored files from index"
remove ignored files from the index
stage .gitignore and the files you just removed
commit
The BFG is specifically designed for removing unwanted data like big files or passwords from Git repositories, so it has a simple flag that will remove any large historical (not-in-your-current-commit) files: '--strip-blobs-bigger-than'
java -jar bfg.jar --strip-blobs-bigger-than 100M
If you'd like to specify files by name, you can do that too:
java -jar bfg.jar --delete-files *.mp4
The BFG is 10-1000x faster than git filter-branch and is generally much easier to use - check the full usage instructions and examples for more details.
Source: Reduce repository size
If you don't want to use the CLI and are working on Windows, a very simple solution is to use TortoiseGit. It has the "Delete (keep local)" Action in the menu which works fine.
This is no longer an issue in the latest Git (v2.17.1 at the time of writing).
The .gitignore file finally ignores tracked-but-deleted files. You can test this for yourself by running the following script. The final git status statement should report "nothing to commit".
# Create an empty repository
mkdir gitignore-test
cd gitignore-test
git init
# Create a file and commit it
echo "hello" > file
git add file
git commit -m initial
# Add the file to gitignore and commit
echo "file" > .gitignore
git add .gitignore
git commit -m gitignore
# Remove the file and commit
git rm file
git commit -m "removed file"
# Reintroduce the file and check status.
# .gitignore is now respected - status reports "nothing to commit".
echo "hello" > file
git status
This is how I solved my issue:
git filter-branch --tree-filter 'rm -rf path/to/your/file' HEAD
git push
In this, we are basically trying to rewrite the history of that particular file in previous commits also.
For more information, you can refer to the man page of filter-branch here.
Source: Removing sensitive data from a repository - using filter-branch
Source: Git: How to remove a big file wrongly committed
In case of already committed DS_Store:
find . -name .DS_Store -print0 | xargs -0 git rm --ignore-unmatch
Ignore them by:
echo ".DS_Store" >> ~/.gitignore_global
echo "._.DS_Store" >> ~/.gitignore_global
echo "**/.DS_Store" >> ~/.gitignore_global
echo "**/._.DS_Store" >> ~/.gitignore_global
git config --global core.excludesfile ~/.gitignore_global
Finally, make a commit!
Especially for the IDE-based files, I use this:
For instance, for the slnx.sqlite file, I just got rid off it completely like the following:
git rm {PATH_OF_THE_FILE}/slnx.sqlite -f
git commit -m "remove slnx.sqlite"
Just keep that in mind that some of those files store some local user settings and preferences for projects (like what files you had open). So every time you navigate or do some changes in your IDE, that file is changed and therefore it checks it out and show as uncommitted changes.
If anyone is having a hard time on Windows and you want to ignore the entire folder, go to the desired 'folder' on file explorer, right click and do 'Git Bash Here' (Git for Windows should have been installed).
Run this command:
git ls-files -z | xargs -0 git update-index --assume-unchanged
For me, the file was still available in the history and I first needed to squash the commits that added the removed files: https://gist.github.com/patik/b8a9dc5cd356f9f6f980
Combine the commits. The example below combines the last 3 commits
git reset --soft HEAD~3
git commit -m "New message for the combined commit"
Push the squashed commit
If the commits have been pushed to the remote:
git push origin +name-of-branch
In my case here, I had several .lock files in several directories that I needed to remove. I ran the following and it worked without having to go into each directory to remove them:
git rm -r --cached **/*.lock
Doing this went into each folder under the 'root' of where I was at and excluded all files that matched the pattern.
Is there any way of simulating a git merge between two branches, the current working branch and the master, but without making any changes?
I often have conflicts when I have to make a git merge. Is there any way of simulating the merge first?
You can use git merge --no-commit to prevent the merge from actually being committed, and if you don't like how the merge works out, just reset to the original head.
If you definitely don't want to finalize the merge, even if it's a fast-forward (and thus has no conflicts, by definition), you could add --no-ff as well.
I don't think there is a way of simulating what will happen until you try the merge. However, if you make sure that the output of git status is empty before you do the merge, it is quite safe to just go ahead and try it. If you get conflicts, you can immediately get back to the state you were at before with:
git reset --merge
Since git 1.7.4, you can also abort the merge by doing:
git merge --abort
(As the commit message that added that option explains, this was added for consistency with git rebase --abort and so on.)
If I want to compare changes on a topic branch to master, I find it easiest and safest to do the following:
git checkout master
git checkout -b trial_merge
git merge topic_branch
After completing the merge, it is easy to see the consolidated change from master
git diff master
When done, simply delete the trial_merge branch
git checkout master
git branch -D trial_merge
This way, the master branch never changes.
Here is the solution that I have found: git merge-tree does merging "in memory" and prints the diff without touching your working directory. You can even test a branch without checking it out.
Get the merge diff
First, do this to make sure your repository knows about all the remote branches:
$ git fetch --all
Now use this bash snippet to see how branch $branch would merge into $master:
$ branch='feature'
$ git merge-tree $(git merge-base $branch master) master $branch
No changes are made to your workdir or index. It's a dry-run merge.
Pick information from the output
The output is a diff.
In case the branch has been merged, it will be empty.
To find whether there are conflicts, grep it for <<<:
$ git merge-tree $(git merge-base $branch master) master $branch | fgrep '<<<'
To extract conflict diffs, use sed to extract lines between <<< and >>>:
$ git merge-tree $(git merge-base $branch master) master $branch | \
sed -ne '/^\+<<</,/^\+>>>/ p'
Features
The diff will be empty if a branch is already merged
Use grep/sed to extract conflicts information
Use origin/feature to test branches you've never worked with
Can be used to see how 2 branches have diverged
Add it to your favorites
Get the diff of the merge:
git config --global alias.mergediff '!f(){ branch="$1" ; into="$2" ; git merge-tree $(git merge-base "$branch" "$into") "$into" "$branch" ; };f '
Usage:
$ git mergediff <feature-branch> <merge-into>
$ git mergediff feature master
Get merge conflicts:
git config --global alias.mergetest '!f(){ git mergediff $# | sed -ne "/^+<<</,/^+>>>/ p" ; };f '
Usage:
$ git mergetest <feature-branch> <merge-into>
$ git mergetest feature master
Why not just create a throwaway branch (git checkout -b), and do a test merge there?
I use :
git merge --ff-only
according to documentation:
Refuse to merge and exit with a non-zero status unless the current HEAD is already up-to-date or the merge can be resolved as a fast-forward.
It's not really a simulation because there will be a fast-forward merge in case of no conflicts between the two branches. But in case of conflicts, you will be informed and nothing will happens.
I've been able to use git merge --abort, recently. However, this can only be used if there is a merge conflict. If you are sure that you will not want to commit, then use the other mentioned methods above.
I don't know exactly if it is your case, but your question remember me that sometimes I start a feature, I commit over the days and I merge the develop on it many times.
On this point I lose the control over the exact files I changed and I will only know it when my feature were closed and my code go to develop.
In this case, a good way to know what modifications you did (not other from the merges) is using Sourcetree.
You must click with the right button on the base branch and select Diff Against Current:
Then sourcetree will show you all the modifications that will be merged if you merge your branch into base branch.
Of course, it will not show you the conflicts, but it is a useful tool in merges.
I would like to put a Git project on GitHub but it contains certain files with sensitive data (usernames and passwords, like /config/deploy.rb for capistrano).
I know I can add these filenames to .gitignore, but this would not remove their history within Git.
I also don't want to start over again by deleting the /.git directory.
Is there a way to remove all traces of a particular file in your Git history?
For all practical purposes, the first thing you should be worried about is CHANGING YOUR PASSWORDS! It's not clear from your question whether your git repository is entirely local or whether you have a remote repository elsewhere yet; if it is remote and not secured from others you have a problem. If anyone has cloned that repository before you fix this, they'll have a copy of your passwords on their local machine, and there's no way you can force them to update to your "fixed" version with it gone from history. The only safe thing you can do is change your password to something else everywhere you've used it.
With that out of the way, here's how to fix it. GitHub answered exactly that question as an FAQ:
Note for Windows users: use double quotes (") instead of singles in this command
git filter-branch --index-filter \
'git update-index --remove PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA' <introduction-revision-sha1>..HEAD
git push --force --verbose --dry-run
git push --force
Update 2019:
This is the current code from the FAQ:
git filter-branch --force --index-filter \
"git rm --cached --ignore-unmatch PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA" \
--prune-empty --tag-name-filter cat -- --all
git push --force --verbose --dry-run
git push --force
Keep in mind that once you've pushed this code to a remote repository like GitHub and others have cloned that remote repository, you're now in a situation where you're rewriting history. When others try pull down your latest changes after this, they'll get a message indicating that the changes can't be applied because it's not a fast-forward.
To fix this, they'll have to either delete their existing repository and re-clone it, or follow the instructions under "RECOVERING FROM UPSTREAM REBASE" in the git-rebase manpage.
Tip: Execute git rebase --interactive
In the future, if you accidentally commit some changes with sensitive information but you notice before pushing to a remote repository, there are some easier fixes. If you last commit is the one to add the sensitive information, you can simply remove the sensitive information, then run:
git commit -a --amend
That will amend the previous commit with any new changes you've made, including entire file removals done with a git rm. If the changes are further back in history but still not pushed to a remote repository, you can do an interactive rebase:
git rebase -i origin/master
That opens an editor with the commits you've made since your last common ancestor with the remote repository. Change "pick" to "edit" on any lines representing a commit with sensitive information, and save and quit. Git will walk through the changes, and leave you at a spot where you can:
$EDITOR file-to-fix
git commit -a --amend
git rebase --continue
For each change with sensitive information. Eventually, you'll end up back on your branch, and you can safely push the new changes.
Changing your passwords is a good idea, but for the process of removing password's from your repo's history, I recommend the BFG Repo-Cleaner, a faster, simpler alternative to git-filter-branch explicitly designed for removing private data from Git repos.
Create a private.txt file listing the passwords, etc, that you want to remove (one entry per line) and then run this command:
$ java -jar bfg.jar --replace-text private.txt my-repo.git
All files under a threshold size (1MB by default) in your repo's history will be scanned, and any matching string (that isn't in your latest commit) will be replaced with the string "***REMOVED***". You can then use git gc to clean away the dead data:
$ git gc --prune=now --aggressive
The BFG is typically 10-50x faster than running git-filter-branch and the options are simplified and tailored around these two common use-cases:
Removing Crazy Big Files
Removing Passwords, Credentials & other Private data
Full disclosure: I'm the author of the BFG Repo-Cleaner.
git filter-repo is now officially recommended over git filter-branch
This is mentioned in the manpage of git filter-branch in Git 2.5 itself.
With git filter repo, you could either remove certain files with: Remove folder and its contents from git/GitHub's history
pip install git-filter-repo
git filter-repo --path path/to/remove1 --path path/to/remove2 --invert-paths
This automatically removes empty commits.
Or you can replace certain strings with: How to replace a string in whole Git history?
git filter-repo --replace-text <(echo 'my_password==>xxxxxxxx')
If you pushed to GitHub, force pushing is not enough, delete the repository or contact support
Even if you force push one second afterwards, it is not enough as explained below.
The only valid courses of action are:
is what leaked a changeable credential like a password?
yes: modify your passwords immediately, and consider using more OAuth and API keys!
no (naked pics):
do you care if all issues in the repository get nuked?
no: delete the repository
yes:
contact support
if the leak is very critical to you, to the point that you are willing to get some repository downtime to make it less likely to leak, make it private while you wait for GitHub support to reply to you
Force pushing a second later is not enough because:
GitHub keeps dangling commits for a long time.
GitHub staff does have the power to delete such dangling commits if you contact them however.
I experienced this first hand when I uploaded all GitHub commit emails to a repo they asked me to take it down, so I did, and they did a gc. Pull requests that contain the data have to be deleted however: that repo data remained accessible up to one year after initial takedown due to this.
Dangling commits can be seen either through:
the commit web UI: https://github.com/cirosantilli/test-dangling/commit/53df36c09f092bbb59f2faa34eba15cd89ef8e83 (Wayback machine)
the API: https://api.github.com/repos/cirosantilli/test-dangling/commits/53df36c09f092bbb59f2faa34eba15cd89ef8e83 (Wayback machine)
One convenient way to get the source at that commit then is to use the download zip method, which can accept any reference, e.g.: https://github.com/cirosantilli/myrepo/archive/SHA.zip
It is possible to fetch the missing SHAs either by:
listing API events with type": "PushEvent". E.g. mine: https://api.github.com/users/cirosantilli/events/public (Wayback machine)
more conveniently sometimes, by looking at the SHAs of pull requests that attempted to remove the content
There are scrappers like http://ghtorrent.org/ and https://www.githubarchive.org/ that regularly pool GitHub data and store it elsewhere.
I could not find if they scrape the actual commit diff, and that is unlikely because there would be too much data, but it is technically possible, and the NSA and friends likely have filters to archive only stuff linked to people or commits of interest.
If you delete the repository instead of just force pushing however, commits do disappear even from the API immediately and give 404, e.g. https://api.github.com/repos/cirosantilli/test-dangling-delete/commits/8c08448b5fbf0f891696819f3b2b2d653f7a3824 This works even if you recreate another repository with the same name.
To test this out, I have created a repo: https://github.com/cirosantilli/test-dangling and did:
git init
git remote add origin git#github.com:cirosantilli/test-dangling.git
touch a
git add .
git commit -m 0
git push
touch b
git add .
git commit -m 1
git push
touch c
git rm b
git add .
git commit --amend --no-edit
git push -f
See also: How to remove a dangling commit from GitHub?
I recommend this script by David Underhill, worked like a charm for me.
It adds these commands in addition natacado's filter-branch to clean up the mess it leaves behind:
rm -rf .git/refs/original/
git reflog expire --all
git gc --aggressive --prune
Full script (all credit to David Underhill)
#!/bin/bash
set -o errexit
# Author: David Underhill
# Script to permanently delete files/folders from your git repository. To use
# it, cd to your repository's root and then run the script with a list of paths
# you want to delete, e.g., git-delete-history path1 path2
if [ $# -eq 0 ]; then
exit 0
fi
# make sure we're at the root of git repo
if [ ! -d .git ]; then
echo "Error: must run this script from the root of a git repository"
exit 1
fi
# remove all paths passed as arguments from the history of the repo
files=$#
git filter-branch --index-filter \
"git rm -rf --cached --ignore-unmatch $files" HEAD
# remove the temporary history git-filter-branch
# otherwise leaves behind for a long time
rm -rf .git/refs/original/ && \
git reflog expire --all && \
git gc --aggressive --prune
The last two commands may work better if changed to the following:
git reflog expire --expire=now --all && \
git gc --aggressive --prune=now
You can use git forget-blob.
The usage is pretty simple git forget-blob file-to-forget. You can get more info here
https://ownyourbits.com/2017/01/18/completely-remove-a-file-from-a-git-repository-with-git-forget-blob/
It will disappear from all the commits in your history, reflog, tags and so on
I run into the same problem every now and then, and everytime I have to come back to this post and others, that's why I automated the process.
Credits to contributors from Stack Overflow that allowed me to put this together
Here is my solution in windows
git filter-branch --tree-filter "rm -f 'filedir/filename'" HEAD
git push --force
make sure that the path is correct
otherwise it won't work
I hope it helps
Use filter-branch:
git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch *file_path_relative_to_git_repo*' --prune-empty --tag-name-filter cat -- --all
git push origin *branch_name* -f
To be clear: The accepted answer is correct. Try it first. However, it may be unnecessarily complex for some use cases, particularly if you encounter obnoxious errors such as 'fatal: bad revision --prune-empty', or really don't care about the history of your repo.
An alternative would be:
cd to project's base branch
Remove the sensitive code / file
rm -rf .git/ # Remove all git info from
your code
Go to github and delete your repository
Follow this guide to push your code to a new repository as you normally would -
https://help.github.com/articles/adding-an-existing-project-to-github-using-the-command-line/
This will of course remove all commit history branches, and issues from both your github repo, and your local git repo. If this is unacceptable you will have to use an alternate approach.
Call this the nuclear option.
In my android project I had admob_keys.xml as separated xml file in app/src/main/res/values/ folder. To remove this sensitive file I used below script and worked perfectly.
git filter-branch --force --index-filter \
'git rm --cached --ignore-unmatch app/src/main/res/values/admob_keys.xml' \
--prune-empty --tag-name-filter cat -- --all
I've had to do this a few times to-date. Note that this only works on 1 file at a time.
Get a list of all commits that modified a file. The one at the bottom will the the first commit:
git log --pretty=oneline --branches -- pathToFile
To remove the file from history use the first commit sha1 and the path to file from the previous command, and fill them into this command:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch <path-to-file>' -- <sha1-where-the-file-was-first-added>..
So, It looks something like this:
git rm --cached /config/deploy.rb
echo /config/deploy.rb >> .gitignore
Remove cache for tracked file from git and add that file to .gitignore list
Considering that OP is using GitHub, if one commits sensitive data into a Git repo, one can remove it entirely from the history by using one of the previous options (read more about them below):
The git filter-repo tool (view source on GitHub).
The BFG Repo-Cleaner tool (it is open source - view source on GitHub).
After one of the previous options, there are additional steps to follow. Check the section Additional below.
If the goal is to remove a file that was added in the most recent unpushed commit, read the section Alternative below.
For future considerations, to prevent similar situations, check the For the Future section below.
Option 1
Using git filter-repo. Before moving forward, note that
If you run git filter-repo after stashing changes, you won't be able to retrieve your changes with other stash commands. Before running git filter-repo, we recommend unstashing any changes you've made. To unstash the last set of changes you've stashed, run git stash show -p | git apply -R. For more information, see Git Tools - Stashing and Cleaning.
Let us now remove one file from the history of one's repo and add it to .gitignore (to prevent re-committing it again).
Before moving forward, make sure that one has git filter-repo installed (read here how to install it), and that one has a local copy of one's repo (if that is not the case, see here how to clone a repository).
Open GitBash and access the repository.
cd YOUR-REPOSITORY
(Optional) Backup the .git/config file.
Run
git filter-repo --invert-paths --path PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA
replace PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA with the path to the file you want to remove, not just its filename to:
Force Git to process, but not check out the entire history of every branch and tag.
Remove the specified file (as well as empty commits generated as a result)
Remove some configs (such as remote URL stored in the .git/config file)
Overwrite one's existing tags.
Add the file with sensitive data to .gitignore
echo "YOUR-FILE-WITH-SENSITIVE-DATA" >> .gitignore
git add .gitignore
git commit -m "Add YOUR-FILE-WITH-SENSITIVE-DATA to .gitignore"
Check if everything was removed from one's repository history, and that all branches are checked out. Only then move to the next step.
Force-push the local changes to overwrite your repository on GitHub.com, as well as all the branches you've pushed up. A force push is required to remove sensitive data from your commit history. Read the first note at the bottom of this answer for more details one this.
git push origin --force --all
Option 2
Using BFG Repo-Cleaner. This is faster and simpler than git filter-branch.
For example, to remove one's file with sensitive data and leave your latest commit untouched, run
bfg --delete-files YOUR-FILE-WITH-SENSITIVE-DATA
To replace all text listed in passwords.txt wherever it can be found in your repository's history, run
bfg --replace-text passwords.txt
After the sensitive data is removed, one must force push one's changes to GitHub.
git push --force
Additional
After using one of the options above:
Contact GitHub Support.
(If working with a team) Tell them to rebase, not merge, any branches they created off of one's old (tainted) repository history. One merge commit could reintroduce some or all of the tainted history that one just went to the trouble of purging.
After some time has passed and you're confident that one had no unintended side effects, one can force all objects in one's local repository to be dereferenced and garbage collected with the following commands (using Git 1.8.5 or newer):
git for-each-ref --format="delete %(refname)" refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
Alternative
If the file was added with the most recent commit, and one has not pushed to GitHub.com, one can delete the file and amend the commit:
Open GitBash and access the repository.
cd YOUR-REPOSITORY.l
To remove the file, enter git rm --cached:
git rm --cached GIANT_FILE
# Stage our giant file for removal, but leave it on disk
Commit this change using --amend -CHEAD:
git commit --amend -CHEAD
# Amend the previous commit with your change
# Simply making a new commit won't work, as you need
# to remove the file from the unpushed history as well
Push one's commits to GitHub.com:
git push
# Push our rewritten, smaller commit
For the Future
In order to prevent sensitive data to be exposed, other good practices include:
Use a visual program to commit the changes. There are various alternatives (such as GitHub Desktop, GitKraken, gitk, ...) and it could be easier to track the changes.
Avoid the catch-all commands git add . and git commit -a. Instead, use git add filename and git rm filename to individually stage files.
Use git add --interactive to individually review and stage changes within each file.
Use git diff --cached to review the changes that one has staged for commit. This is the exact diff that git commit will produce as long as one doesn't use the -a flag.
Generate Secret Keys in secure hardware (HSM boxes, hardware keys - like Yubikey / Solokey), that never leaves it.
Train the team on x508.
Notes:
When one force pushes, it rewrites the repository history, which removes sensitive data from the commit history. That may overwrite commits that other people have based their work on.
For this answer one used content from some GitHub posts:
Removing sensitive data from a repository
About large files on GitHub