Issue cloning large repositories

[jantman]

GitPython seems to break on large repositories. I've hacked a bit of additional output into the assertion at https://github.com/gitpython-developers/GitPython/blob/master/git/remote.py#L576 :

        # NOTE: We assume to fetch at least enough progress lines to allow matching each fetch head line with it.
        assert len(fetch_info_lines) >= len(fetch_head_info), "%d <= %d - len(fetch_head_info) <= len(fetch_info_lines) : len(%s) <= len(%s)" % (
            len(fetch_head_info),
            len(fetch_info_lines),
            fetch_head_info,
            fetch_info_lines)

When I run git.Repo.clone_from() or .fetch(), it produces:

[DEBUG reconcile_git_repos.py:95 -       clone_or_fetch() ] Fetching origin in /localfs/users/jantman/B-07815/storyville2.git
Traceback (most recent call last):
  File "./reconcile_git_repos.py", line 168, in <module>
    args = parse_args(sys.argv[1:])
  File "./reconcile_git_repos.py", line 75, in run
    self.clone_or_fetch(path, url, fetch=fetch)
  File "./reconcile_git_repos.py", line 96, in clone_or_fetch
    except git.exc.NoSuchPathError:
  File "/home/jantman/tickets/B-07815/lib/python2.7/site-packages/git/remote.py", line 654, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress or RemoteProgress())
  File "/home/jantman/tickets/B-07815/lib/python2.7/site-packages/git/remote.py", line 588, in _get_fetch_info_from_stderr
    fetch_info_lines)
AssertionError: 1944 <= 1716 : len([u"c415f018

... and then an amazingly long line of output.

This appears to happen with very large repositories; I honestly can't tell if the issue here is commit messages or the repository size - the one I used for testing has (yeah..... :( ) 4,789 branches.

[jantman]

Updated issue - fixed formatting, and corrected description, this does not get better after multiple fetches. I have at least one repo that GitPython simply cannot fetch.

[Byron]

The issue described looks very much like #232, which was fixed back in the days. Maybe, the fix didn't work after all.

What happens is that GitPython receives output from stderr, line by line, and from the FETCH_INFO file git leaves on disk. The information of both data sources is merged to obtain all the branch information we seek to provide.

If these data sources do not match up for some reason, the assertion will fail.

Assuming you are already using the latest version on pypi or GitHub, we could try to allow me to reproduce the issue. To do that, you can run your code with GitPython on this branch which leaves debug files in the .git/ directory of the respective repo. These you could provide by some means so I can integrate them into the test-suite and fix the issue.

Thank you.

[Byron]

By the way: a quick workaround should be to leave GitPython out of the equation and just run repo.git.fetch(). That way, you will not have any branch information, but the fetch will definitely work.
The Repo.clone_from() operation can also be approximated by using the Git type directly.

[jantman]

@Byron thanks so much for the quick reply on this. This is happening with 1.0.1 from pypi (I wish I could say I was running one of the old RCs and the upgrade fixed it, but no such luck).

So unfortunately this is a private, internal repo, and given some of the commit messages that I've seen, I don't feel comfortable providing... anything... from this repo externally.

However, I've created a test repository at https://github.com/jantman/gitpython_issue_301 with 5,000 randomly-generated commits in it, each in its own branch. Very interestingly, I've been able to reproduce the issue, but only when using SSH as a transport; when using HTTPS, it works fine.

Code to reproduce:

#!/usr/bin/env python

import git
"""
# this works
url = 'https://github.com/jantman/gitpython_issue_301.git'
"""
# this doesn't
url = 'git@github.com:jantman/gitpython_issue_301.git'
path = 'testrepo'
print("cloning")
repo = git.Repo.clone_from(url, path)
print("fetching")
repo.remotes.origin.fetch()
print("done")

The failure happens during the fetch. I've added you as a collaborator on the repo so you can test if you want.

[Byron]

You can watch the development stream on youtube.

GitPython #12 [issue 301 - investigate fetching repos with huge amount of branches]

[jantman]

@Byron That was actually really interesting to watch... it's the first time I've seen a livestream of development work, let alone the thought process of working through an issue I opened.

[Byron]

Thank you ! And wait until you have seen a good one of that kind !

PS: I have the issue in my inbox and will get to it eventually. A few things piled up.

[Byron]

Alright, a fix was pushed, and I could verify that the provided means of reproduction do not fail anymore.
Travis is now testing for it as well.

Please let me know if it works for you as well.

[Byron]

You can watch the development stream on youtube.

GitPython #15 [issue 301 finally fixed]