Switching from git-bigfile to git-lfs

In 2012, I was looking for a way to store big files in git. git-annex was already around, but I found it a bit too complex for my use case. I discovered git-media from Scott Chacon and it looked like what I was looking for. It was in Ruby which made it not super easy to install on some machines at work. I thought it was a good exercise to port it to Python. That's how git-bigfile was born. It was simple and was doing the job.

Last year, I was thinking about giving it some love: port it to Python 3, add some unittests... That's about when I switched from Gogs to Gitlab and read that Gitlab was about to support git-lfs.

Being developed by GitHub and with Gitlab support, git-lfs was an obvious option to replace git-bigfile.

Here is how to switch a project using git-bigfile to git-lfs:

  1. Make a list of all files tracked by git-bigfile:

    $ git bigfile status | awk '/pushed/ {print $NF}' > /tmp/list
  2. Edit .gitattributes to replace the filter. Replace filter=bigfile -crlf with filter=lfs diff=lfs merge=lfs -text:

    $ cat .gitattributes
    *.tar.bz2 filter=lfs diff=lfs merge=lfs -text
    *.iso filter=lfs diff=lfs merge=lfs -text
    *.img filter=lfs diff=lfs merge=lfs -text
  3. Remove all big files from the staging area and add them back with git-lfs:

    $ git rm --cached $(cat /tmp/list)
    $ git add .
    $ git commit -m "Switch to git-lfs"
  4. Check that the files were added using git-lfs. You should see something like that:

    $ git show HEAD
    diff --git a/CentOS_6.4/images/install.img
    b/CentOS_6.4/images/install.img
    index 227ea55..a9cc6a8 100644
    --- a/CentOS_6.4/images/install.img
    +++ b/CentOS_6.4/images/install.img
    @@ -1 +1,3 @@
    -5d243948497ceb9f07b033da62498e52269f4b83
    +version https://git-lfs.github.com/spec/v1
    +oid
    sha256:6fcaac620b82e38e2092a6353ca766a3b01fba7f3fd6a0397c57e979aa293db0
    +size 133255168
  5. Remove git-bigfile cache directory:

    $ rm -rf .git/bigfile

Note: to push files larger than 2.1GB to your gitlab server, wait for this fix. Hopefully it will be in 8.4.3.

Comments

Comments powered by Disqus