Merge branch 'remotes' into improvements

* remotes:
  Finished all push tests I could think of so far. More error cases should be studied, but they would be hard to 'produce'
  Intermediate commit with a few added and improved tests as well as many fixes
  Implemented PushProgress and PushInfo class including basic test cases. Now many more test-cases need to be added to be sure we can truly deal with everything git throws at us
  Added frame for push testing and push implemenation
  Another attempt to make fetch emit progress information, but in fact its proven now that this is not happening if stderr is being redirected. A test is in place that will most likely fail in case this ever changes
  Added repo.refs for completeness (as remote.refs is there as well and quite nice to use)
  Tried to use shallow repository - this works in case it is remote, but unfortunately, deepening the repository fails if the server is used. This is bad, but a workaround is to create another shared repo which pushes a changes that we fetch into our given repo. This should provide more output to properly test the fetch handling. Harder than I thought
  Fixed bug when listing remotes - it was based on references which is incorrect as it cannot always work
  FetchInfo class is not a subclass of Remote class anymore, as more classes are to be added it cluttered up the view and made things more complex as well
  IterableList: added support for prefix allowing remote.refs.master constructs, previously it was remote.refs['%s/master'%remote]
  tag handling tests finished, unfortunately there is not yet a rejected case, but it will assuambly follow with the push tests
  Implemented handling of FETCH_HEAD and tags, some test cases still missing dealing with deletion and movements of remote tags ( which in fact is discouraged, but we should be able to deal with it, shouldnt we ;)
  Added special cases to test that shows we cannot yet:
  Added remote stale_refs property including test, tested new remote branch handling and deletion of stale remote branches
  renamed remote_branch to remote_head, improved errror message
  Added non-fast forward test case, fixed parsing issue caused by initial line stripping
  implemented test for rejection handling and fixed a bug when parsing remote reference paths
  Added testing frame for proper fetch testing to be very sure this works as expected. Plenty of cases still to be tested
  Reference._from_string will now create the appropriate type, not just the type of the actual class. This could result in a symbolic reference returned even though you technically requested a reference - this issue must still be addressed.
  put _make_file helper method into TestBase class

10 files changed, 886 insertions, 59 deletions

diff --git a/TODO b/TODO
index d841f774..147eb02d 100644
--- a/TODO
+++ b/TODO
@@ -43,6 +43,8 @@ Config
   it will be returned instead of raising. This way the class will be much more usable, 
   and ... I truly hate this config reader as it is so 'old' style. Its not even a new-style
   class yet showing that it must be ten years old.
+  - If you are at it, why not start a new project that reimplements the ConfigWriter
+  properly, honestly. Tune it for usability ... .
 
 Diff
 ----
@@ -96,14 +98,19 @@ Refs
   - NO: The reference dosnt need to know - in fact it does not know about the 
   main HEAD, so it may not use it. This is to be done in client code only. 
   Remove me
+* Reference.from_path may return a symbolic reference although it is not related 
+  to the reference type. Split that up into two from_path on each of the types, 
+  and provide a general method outside of the type  that tries both.
 
 Remote
 ------
-* 'push' method needs a test, a true test repository is required though, a fork 
-  of a fork would do :)!
-* Fetch should return heads that where updated, pull as well.
+* iter_items should parse the configuration file manually - currently a command 
+  is issued which is much slower than it has to be ( compared to manual parsing )
 * Creation and deletion methods for references should be part of the interface, allowing
   repo.create_head(...) instaed of Head.create(repo, ...). Its a convenience thing, clearly
+* When parsing fetch-info, the regex will not allow spaces in the target remote ref as 
+  I couldn't properly parse the optional space separated note in that case. Probably 
+  the regex should be improved to handle this gracefully.
   
 Repo
 ----
@@ -111,11 +118,26 @@ Repo
   currently regex are used a lot although we can deduct what will be next.
   - Read data from a stream directly from git command
 * Figure out how to implement a proper merge API
+* There should be a way to create refs and delete them, instead of having to use
+  the awkward Head.create( repo, ... ) way
+* repo.checkout should be added that does everything HEAD.reset does, but in addition
+  it allows to checkout heads beforehand, hence its more like a repo.head.reference = other_head.
+  
   
 Submodules
 ----------
 * add submodule support
 
+TestSystem
+----------
+* Figure out a good way to indicate the required presense of a git-daemon to host
+ a specific path. Ideally, the system would detect the missing daemon and inform 
+ the user about the required command-line to start the daemon where we need it.
+ Reason for us being unable to start a daemon is that it will always fork - we can 
+ only kill itself, but not its children. Even if we would a pgrep like match, we still 
+ would not know whether it truly is our daemons - in that case user permissions should
+ stop us though.
+
 Tree
 ----
 * Should return submodules during iteration ( identifies as commit )
diff --git a/lib/git/__init__.py b/lib/git/__init__.py
index e3043dc9..c7efe5ea 100644
--- a/lib/git/__init__.py
+++ b/lib/git/__init__.py
@@ -18,7 +18,7 @@ from git.errors import InvalidGitRepositoryError, NoSuchPathError, GitCommandErr
 from git.cmd import Git
 from git.repo import Repo
 from git.stats import Stats
-from git.remote import Remote
+from git.remote import *
 from git.index import *
 
 __all__ = [ name for name, obj in locals().items()
diff --git a/lib/git/refs.py b/lib/git/refs.py
index 26e7c09e..cea3e720 100644
--- a/lib/git/refs.py
+++ b/lib/git/refs.py
@@ -31,7 +31,7 @@ class Reference(LazyMixin, Iterable):
 			
 		"""
 		if not path.startswith(self._common_path_default):
-			raise ValueError("Cannot instantiate %s Reference from path %s" % ( self.__class__.__name__, path ))
+			raise ValueError("Cannot instantiate %s from path %s" % ( self.__class__.__name__, path ))
 			
 		self.repo = repo
 		self.path = path
@@ -167,6 +167,9 @@ class Reference(LazyMixin, Iterable):
 			Instance of type Reference, Head, Tag, SymbolicReference or HEAD
 			depending on the given path
 		"""
+		if not path:
+			raise ValueError("Cannot create Reference from %r" % path)
+		
 		if path == 'HEAD':
 			return HEAD(repo, path)
 		
@@ -208,7 +211,7 @@ class Reference(LazyMixin, Iterable):
 		# our path on demand - due to perstent commands it is fast.
 		# This reduces the risk that the object does not match 
 		# the changed ref anymore in case it changes in the meanwhile
-		return cls(repo, full_path)
+		return cls.from_path(repo, full_path)
 		
 		# obj = get_object_type_by_name(type_name)(repo, hexsha)
 		# obj.size = object_size
@@ -263,11 +266,13 @@ class SymbolicReference(object):
 		tokens = value.split(" ")
 		
 		# it is a detached reference
-		if len(tokens) == 1 and len(tokens[0]) == 40:
+		if self.repo.re_hexsha_only.match(tokens[0]):
 			return Commit(self.repo, tokens[0])
 		
 		# must be a head ! Git does not allow symbol refs to other things than heads
 		# Otherwise it would have detached it
+		if tokens[0] != "ref:":
+			raise ValueError("Failed to parse symbolic refernce: wanted 'ref: <hexsha>', got %r" % value)
 		return Head(self.repo, tokens[1]).commit
 		
 	def _set_commit(self, commit):
@@ -620,10 +625,10 @@ class RemoteReference(Head):
 		return tokens[2]
 		
 	@property
-	def remote_branch(self):
+	def remote_head(self):
 		"""
 		Returns
-			Name of the remote branch itself, i.e. master.
+			Name of the remote head itself, i.e. master.
 			
 		NOTE: The returned name is usually not qualified enough to uniquely identify
 		a branch
diff --git a/lib/git/remote.py b/lib/git/remote.py
index 7febf2ee..1b9c5360 100644
--- a/lib/git/remote.py
+++ b/lib/git/remote.py
@@ -7,8 +7,13 @@
 Module implementing a remote object allowing easy access to git remotes
 """
 
+from errors import GitCommandError
 from git.utils import LazyMixin, Iterable, IterableList
-from refs import RemoteReference
+from objects import Commit
+from refs import Reference, RemoteReference, SymbolicReference, TagReference
+
+import re
+import os
 
 class _SectionConstraint(object):
 	"""
@@ -34,6 +39,342 @@ class _SectionConstraint(object):
 		as first argument"""
 		return getattr(self._config, method)(self._section_name, *args)
 		
+		
+class PushProgress(object):
+	"""
+	Handler providing an interface to parse progress information emitted by git-push
+	and to dispatch callbacks allowing subclasses to react to the progress.
+	"""
+	BEGIN, END, COUNTING, COMPRESSING, WRITING =  [ 1 << x for x in range(5) ]
+	STAGE_MASK = BEGIN|END
+	OP_MASK = COUNTING|COMPRESSING|WRITING
+	
+	__slots__ = ("_cur_line", "_seen_ops")
+	re_op_absolute = re.compile("([\w\s]+):\s+()(\d+)()(, done\.)?\s*")
+	re_op_relative = re.compile("([\w\s]+):\s+(\d+)% \((\d+)/(\d+)\)(,.* done\.)?$")
+	
+	def __init__(self):
+		self._seen_ops = list()
+	
+	def _parse_progress_line(self, line):
+		"""
+		Parse progress information from the given line as retrieved by git-push
+		"""
+		# handle
+		# Counting objects: 4, done. 
+		# Compressing objects:  50% (1/2)   \rCompressing objects: 100% (2/2)   \rCompressing objects: 100% (2/2), done.
+		self._cur_line = line
+		sub_lines = line.split('\r')
+		for sline in sub_lines:
+			sline = sline.rstrip()
+			
+			cur_count, max_count = None, None
+			match = self.re_op_relative.match(sline)
+			if match is None:
+				match = self.re_op_absolute.match(sline)
+				
+			if not match:
+				self.line_dropped(sline)
+				continue
+			# END could not get match
+			
+			op_code = 0
+			op_name, percent, cur_count, max_count, done = match.groups()
+			# get operation id
+			if op_name == "Counting objects":
+				op_code |= self.COUNTING
+			elif op_name == "Compressing objects":
+				op_code |= self.COMPRESSING
+			elif op_name == "Writing objects":
+				op_code |= self.WRITING
+			else:
+				raise ValueError("Operation name %r unknown" % op_name)
+			
+			# figure out stage
+			if op_code not in self._seen_ops:
+				self._seen_ops.append(op_code)
+				op_code |= self.BEGIN
+			# END begin opcode
+			
+			message = ''
+			if done is not None and 'done.' in done:
+				op_code |= self.END
+				message = done.replace( ", done.", "")[2:]
+			# END end flag handling 
+			
+			self.update(op_code, cur_count, max_count, message)
+			
+		# END for each sub line
+	
+	def line_dropped(self, line):
+		"""
+		Called whenever a line could not be understood and was therefore dropped.
+		"""
+		pass
+	
+	def update(self, op_code, cur_count, max_count=None, message=''):
+		"""
+		Called whenever the progress changes
+		
+		``op_code``
+			Integer allowing to be compared against Operation IDs and stage IDs.
+			
+			Stage IDs are BEGIN and END. BEGIN will only be set once for each Operation 
+			ID as well as END. It may be that BEGIN and END are set at once in case only
+			one progress message was emitted due to the speed of the operation.
+			Between BEGIN and END, none of these flags will be set
+			
+			Operation IDs are all held within the OP_MASK. Only one Operation ID will 
+			be active per call.
+			
+		``cur_count``
+			Current absolute count of items
+			
+		``max_count``
+			The maximum count of items we expect. It may be None in case there is 
+			no maximum number of items or if it is (yet) unknown.
+		
+		``message``
+			In case of the 'WRITING' operation, it contains the amount of bytes
+			transferred. It may possibly be used for other purposes as well.
+		You may read the contents of the current line in self._cur_line
+		"""
+		pass
+		
+		
+class PushInfo(object):
+	"""
+	Carries information about the result of a push operation of a single head::
+	 info = remote.push()[0]
+	 info.flags			# bitflags providing more information about the result
+	 info.local_ref		# Reference pointing to the local reference that was pushed
+	 					# It is None if the ref was deleted.
+	 info.remote_ref_string # path to the remote reference located on the remote side
+	 info.remote_ref	# Remote Reference on the local side corresponding to 
+	 					# the remote_ref_string. It can be a TagReference as well.
+	 info.old_commit	# commit at which the remote_ref was standing before we pushed
+	 					# it to local_ref.commit. Will be None if an error was indicated
+	"""
+	__slots__ = ('local_ref', 'remote_ref_string', 'flags', 'old_commit', '_remote')
+	
+	NEW_TAG, NEW_HEAD, NO_MATCH, REJECTED, REMOTE_REJECTED, REMOTE_FAILURE, DELETED, \
+	FORCED_UPDATE, FAST_FORWARD, UP_TO_DATE, ERROR = [ 1 << x for x in range(11) ]
+
+	_flag_map = { 	'X' : NO_MATCH, '-' : DELETED, '*' : 0,
+					'+' : FORCED_UPDATE, ' ' : FAST_FORWARD, 
+					'=' : UP_TO_DATE, '!' : ERROR }
+	
+	def __init__(self, flags, local_ref, remote_ref_string, remote, old_commit=None):
+		"""
+		Initialize a new instance
+		"""
+		self.flags = flags
+		self.local_ref = local_ref
+		self.remote_ref_string = remote_ref_string
+		self._remote = remote
+		self.old_commit = old_commit
+		
+	@property
+	def remote_ref(self):
+		"""
+		Returns
+			Remote Reference or TagReference in the local repository corresponding 
+			to the remote_ref_string kept in this instance.
+		"""
+		# translate heads to a local remote, tags stay as they are
+		if self.remote_ref_string.startswith("refs/tags"):
+			return TagReference(self._remote.repo, self.remote_ref_string)
+		elif self.remote_ref_string.startswith("refs/heads"):
+			remote_ref = Reference(self._remote.repo, self.remote_ref_string)
+			return RemoteReference(self._remote.repo, "refs/remotes/%s/%s" % (str(self._remote), remote_ref.name))
+		else:
+			raise ValueError("Could not handle remote ref: %r" % self.remote_ref_string)
+		# END 
+		
+	@classmethod
+	def _from_line(cls, remote, line):
+		"""
+		Create a new PushInfo instance as parsed from line which is expected to be like
+		c	refs/heads/master:refs/heads/master	05d2687..1d0568e
+		"""
+		control_character, from_to, summary = line.split('\t', 3)
+		flags = 0
+		
+		# control character handling
+		try:
+			flags |= cls._flag_map[ control_character ]
+		except KeyError:
+			raise ValueError("Control Character %r unknown as parsed from line %r" % (control_character, line)) 
+		# END handle control character
+		
+		# from_to handling
+		from_ref_string, to_ref_string = from_to.split(':')
+		if flags & cls.DELETED:
+			from_ref = None
+		else:
+			from_ref = Reference.from_path(remote.repo, from_ref_string)
+		
+		# commit handling, could be message or commit info
+		old_commit = None
+		if summary.startswith('['):
+			if "[rejected]" in summary:
+				flags |= cls.REJECTED
+			elif "[remote rejected]" in summary:
+				flags |= cls.REMOTE_REJECTED
+			elif "[remote failure]" in summary:
+				flags |= cls.REMOTE_FAILURE
+			elif "[no match]" in summary:
+				flags |= cls.ERROR
+			elif "[new tag]" in summary:
+				flags |= cls.NEW_TAG
+			elif "[new branch]" in summary:
+				flags |= cls.NEW_HEAD
+			# uptodate encoded in control character
+		else:
+			# fast-forward or forced update - was encoded in control character, 
+			# but we parse the old and new commit
+			split_token = "..."
+			if control_character == " ":
+				split_token = ".."
+			old_sha, new_sha = summary.split(' ')[0].split(split_token)
+			old_commit = Commit(remote.repo, old_sha)
+		# END message handling
+		
+		return PushInfo(flags, from_ref, to_ref_string, remote, old_commit)
+		
+
+class FetchInfo(object):
+	"""
+	Carries information about the results of a fetch operation of a single head::
+	
+	 info = remote.fetch()[0]
+	 info.ref			# Symbolic Reference or RemoteReference to the changed 
+						# remote head or FETCH_HEAD
+	 info.flags 		# additional flags to be & with enumeration members, 
+						# i.e. info.flags & info.REJECTED 
+						# is 0 if ref is SymbolicReference
+	 info.note			# additional notes given by git-fetch intended for the user
+	 info.commit_before_forced_update	# if info.flags & info.FORCED_UPDATE, 
+						# field is set to the previous location of ref, otherwise None
+	"""
+	__slots__ = ('ref','commit_before_forced_update', 'flags', 'note')
+	
+	NEW_TAG, NEW_HEAD, HEAD_UPTODATE, TAG_UPDATE, REJECTED, FORCED_UPDATE, \
+	FAST_FORWARD, ERROR = [ 1 << x for x in range(8) ]
+	
+	#                             %c    %-*s %-*s             -> %s       (%s)
+	re_fetch_result = re.compile("^\s*(.) (\[?[\w\s\.]+\]?)\s+(.+) -> ([/\w_\.-]+)(  \(.*\)?$)?")
+	
+	_flag_map = { 	'!' : ERROR, '+' : FORCED_UPDATE, '-' : TAG_UPDATE, '*' : 0,
+					'=' : HEAD_UPTODATE, ' ' : FAST_FORWARD } 
+	
+	def __init__(self, ref, flags, note = '', old_commit = None):
+		"""
+		Initialize a new instance
+		"""
+		self.ref = ref
+		self.flags = flags
+		self.note = note
+		self.commit_before_forced_update = old_commit
+		
+	def __str__(self):
+		return self.name
+		
+	@property
+	def name(self):
+		"""
+		Returns
+			Name of our remote ref
+		"""
+		return self.ref.name
+		
+	@property
+	def commit(self):
+		"""
+		Returns
+			Commit of our remote ref
+		"""
+		return self.ref.commit
+		
+	@classmethod
+	def _from_line(cls, repo, line, fetch_line):
+		"""
+		Parse information from the given line as returned by git-fetch -v
+		and return a new FetchInfo object representing this information.
+		
+		We can handle a line as follows
+		"%c %-*s %-*s -> %s%s"
+		
+		Where c is either ' ', !, +, -, *, or =
+		! means error
+		+ means success forcing update
+		- means a tag was updated
+		* means birth of new branch or tag
+		= means the head was up to date ( and not moved )
+		' ' means a fast-forward
+		
+		fetch line is the corresponding line from FETCH_HEAD, like
+		acb0fa8b94ef421ad60c8507b634759a472cd56c	not-for-merge	branch '0.1.7RC' of /tmp/tmpya0vairemote_repo
+		"""
+		match = cls.re_fetch_result.match(line)
+		if match is None:
+			raise ValueError("Failed to parse line: %r" % line)
+			
+		# parse lines
+		control_character, operation, local_remote_ref, remote_local_ref, note = match.groups()
+		try:
+			new_hex_sha, fetch_operation, fetch_note = fetch_line.split("\t")
+			ref_type_name, fetch_note = fetch_note.split(' ', 1)
+		except ValueError:	# unpack error
+			raise ValueError("Failed to parse FETCH__HEAD line: %r" % fetch_line)
+		
+		# handle FETCH_HEAD and figure out ref type
+		# If we do not specify a target branch like master:refs/remotes/origin/master, 
+		# the fetch result is stored in FETCH_HEAD which destroys the rule we usually
+		# have. In that case we use a symbolic reference which is detached 
+		ref_type = None
+		if remote_local_ref == "FETCH_HEAD":
+			ref_type = SymbolicReference
+		elif ref_type_name  == "branch":
+			ref_type = RemoteReference
+		elif ref_type_name == "tag":
+			ref_type = TagReference
+		else:
+			raise TypeError("Cannot handle reference type: %r" % ref_type_name)
+			
+		# create ref instance
+		if ref_type is SymbolicReference:
+			remote_local_ref = ref_type(repo, "FETCH_HEAD") 
+		else:
+			remote_local_ref = Reference.from_path(repo, os.path.join(ref_type._common_path_default, remote_local_ref.strip()))
+		# END create ref instance 
+		
+		note = ( note and note.strip() ) or ''
+		
+		# parse flags from control_character
+		flags = 0
+		try:
+			flags |= cls._flag_map[control_character]
+		except KeyError:
+			raise ValueError("Control character %r unknown as parsed from line %r" % (control_character, line))
+		# END control char exception hanlding 
+		
+		# parse operation string for more info - makes no sense for symbolic refs
+		old_commit = None
+		if isinstance(remote_local_ref, Reference):
+			if 'rejected' in operation:
+				flags |= cls.REJECTED
+			if 'new tag' in operation:
+				flags |= cls.NEW_TAG
+			if 'new branch' in operation:
+				flags |= cls.NEW_HEAD
+			if '...' in operation:
+				old_commit = Commit(repo, operation.split('...')[0])
+			# END handle refspec
+		# END reference flag handling
+		
+		return cls(remote_local_ref, flags, note, old_commit)
+	
 
 class Remote(LazyMixin, Iterable):
 	"""
@@ -103,25 +444,20 @@ class Remote(LazyMixin, Iterable):
 		Returns
 			Iterator yielding Remote objects of the given repository
 		"""
-		# parse them using refs, as their query can be faster as it is 
-		# purely based on the file system
 		seen_remotes = set()
-		for ref in RemoteReference.iter_items(repo):
-			remote_name = ref.remote_name
-			if remote_name in seen_remotes:
-				continue
-			# END if remote done already
-			seen_remotes.add(remote_name)
-			yield Remote(repo, remote_name)
+		for name in repo.git.remote().splitlines():
+			yield Remote(repo, name)
 		# END for each ref
 		
 	@property
 	def refs(self):
 		"""
 		Returns
-			List of RemoteRef objects
+			IterableList of RemoteReference objects. It is prefixed, allowing 
+			you to omit the remote path portion, i.e.::
+			 remote.refs.master # yields RemoteReference('/refs/remotes/origin/master')
 		"""
-		out_refs = IterableList(RemoteReference._id_attribute_)
+		out_refs = IterableList(RemoteReference._id_attribute_, "%s/" % self.name)
 		for ref in RemoteReference.list_items(self.repo):
 			if ref.remote_name == self.name:
 				out_refs.append(ref)
@@ -129,6 +465,29 @@ class Remote(LazyMixin, Iterable):
 		# END for each ref
 		assert out_refs, "Remote %s did not have any references" % self.name
 		return out_refs
+		
+	@property
+	def stale_refs(self):
+		"""
+		Returns 
+			IterableList RemoteReference objects that do not have a corresponding 
+			head in the remote reference anymore as they have been deleted on the 
+			remote side, but are still available locally.
+			
+			The IterableList is prefixed, hence the 'origin' must be omitted. See
+			'refs' property for an example.
+		"""
+		out_refs = IterableList(RemoteReference._id_attribute_, "%s/" % self.name)
+		for line in self.repo.git.remote("prune", "--dry-run", self).splitlines()[2:]:
+			# expecting 
+			# * [would prune] origin/new_branch
+			token = " * [would prune] " 
+			if not line.startswith(token):
+				raise ValueError("Could not parse git-remote prune result: %r" % line)
+			fqhn = "%s/%s" % (RemoteReference._common_path_default,line.replace(token, ""))
+			out_refs.append(RemoteReference(self.repo, fqhn))
+		# END for each line 
+		return out_refs
 	
 	@classmethod
 	def create(cls, repo, name, url, **kwargs):
@@ -198,6 +557,45 @@ class Remote(LazyMixin, Iterable):
 		self.repo.git.remote("update", self.name)
 		return self
 	
+	def _get_fetch_info_from_stderr(self, stderr):
+		# skip first line as it is some remote info we are not interested in
+		output = IterableList('name')
+		err_info = stderr.splitlines()[1:]
+		
+		# read head information 
+		fp = open(os.path.join(self.repo.path, 'FETCH_HEAD'),'r')
+		fetch_head_info = fp.readlines()
+		fp.close()
+		
+		output.extend(FetchInfo._from_line(self.repo, err_line, fetch_line) 
+						for err_line,fetch_line in zip(err_info, fetch_head_info))
+		return output
+	
+	def _get_push_info(self, proc, progress):
+		# read progress information from stderr
+		# we hope stdout can hold all the data, it should ... 
+		for line in proc.stderr.readlines():
+			progress._parse_progress_line(line.rstrip())
+		# END for each progress line
+		
+		output = IterableList('name')
+		for line in proc.stdout.readlines():
+			try:
+				output.append(PushInfo._from_line(self, line))
+			except ValueError:
+				# if an error happens, additional info is given which we cannot parse
+				pass
+			# END exception handling 
+		# END for each line
+		try:
+			proc.wait()
+		except GitCommandError:
+			# if a push has rejected items, the command has non-zero return status
+			pass
+		# END exception handling 
+		return output
+		
+	
 	def fetch(self, refspec=None, **kwargs):
 		"""
 		Fetch the latest changes for this remote
@@ -218,10 +616,15 @@ class Remote(LazyMixin, Iterable):
 			Additional arguments to be passed to git-fetch
 			
 		Returns
-			self
+			IterableList(FetchInfo, ...) list of FetchInfo instances providing detailed 
+			information about the fetch results
+			
+		Note
+			As fetch does not provide progress information to non-ttys, we cannot make 
+			it available here unfortunately as in the 'push' method.
 		"""
-		self.repo.git.fetch(self, refspec, **kwargs)
-		return self
+		status, stdout, stderr = self.repo.git.fetch(self, refspec, with_extended_output=True, v=True, **kwargs)
+		return self._get_fetch_info_from_stderr(stderr)
 		
 	def pull(self, refspec=None, **kwargs):
 		"""
@@ -235,26 +638,37 @@ class Remote(LazyMixin, Iterable):
 			Additional arguments to be passed to git-pull
 			
 		Returns
-			self
+			Please see 'fetch' method
 		"""
-		self.repo.git.pull(self, refspec, **kwargs)
-		return self
+		status, stdout, stderr = self.repo.git.pull(self, refspec, with_extended_output=True, v=True, **kwargs)
+		return self._get_fetch_info_from_stderr(stderr)
 		
-	def push(self, refspec=None, **kwargs):
+	def push(self, refspec=None, progress=None, **kwargs):
 		"""
 		Push changes from source branch in refspec to target branch in refspec.
 		
 		``refspec``
 			see 'fetch' method
 		
+		``progress``
+			Instance of type PushProgress allowing the caller to receive 
+			progress information until the method returns.
+			If None, progress information will be discarded
+		
 		``**kwargs``
 			Additional arguments to be passed to git-push
 			
 		Returns
-			self
-		"""	
-		self.repo.git.push(self, refspec, **kwargs)
-		return self
+			IterableList(PushInfo, ...) iterable list of PushInfo instances, each 
+			one informing about an individual head which had been updated on the remote 
+			side.
+			If the push contains rejected heads, these will have the PushInfo.ERROR bit set
+			in their flags.
+			If the operation fails completely, the length of the returned IterableList will
+			be null.
+		"""
+		proc = self.repo.git.push(self, refspec, porcelain=True, as_process=True, **kwargs)
+		return self._get_push_info(proc, progress or PushProgress())
 		
 	@property
 	def config_reader(self):
diff --git a/lib/git/repo.py b/lib/git/repo.py
index 569d6f1b..d81106c0 100644
--- a/lib/git/repo.py
+++ b/lib/git/repo.py
@@ -143,6 +143,16 @@ class Repo(object):
 			``git.IterableList(Head, ...)``
 		"""
 		return Head.list_items(self)
+		
+	@property
+	def refs(self):
+		"""
+		A list of Reference objects representing tags, heads and remote references.
+		
+		Returns
+			IterableList(Reference, ...)
+		"""
+		return Reference.list_items(self)
 
 	# alias heads
 	branches = heads
@@ -615,7 +625,7 @@ class Repo(object):
 		Create a clone from this repository.
 
 		``path``
-			is the full path of the new repo (traditionally ends with /<name>.git)
+			is the full path of the new repo (traditionally ends with ./<name>.git).
 
 		``kwargs``
 			keyword arguments to be given to the git-clone command
diff --git a/lib/git/utils.py b/lib/git/utils.py
index 8cdb4804..48427ff2 100644
--- a/lib/git/utils.py
+++ b/lib/git/utils.py
@@ -260,16 +260,25 @@ class IterableList(list):
 	 heads.master
 	 heads['master']
 	 heads[0]
+	 
+	It requires an id_attribute name to be set which will be queried from its 
+	contained items to have a means for comparison.
+	
+	A prefix can be specified which is to be used in case the id returned by the 
+	items always contains a prefix that does not matter to the user, so it 
+	can be left out.
 	"""
-	__slots__ = '_id_attr'
+	__slots__ = ('_id_attr', '_prefix')
 	
-	def __new__(cls, id_attr):
+	def __new__(cls, id_attr, prefix=''):
 		return super(IterableList,cls).__new__(cls)
 		
-	def __init__(self, id_attr):
+	def __init__(self, id_attr, prefix=''):
 		self._id_attr = id_attr
+		self._prefix = prefix
 		
 	def __getattr__(self, attr):
+		attr = self._prefix + attr
 		for item in self:
 			if getattr(item, self._id_attr) == attr:
 				return item
@@ -283,7 +292,7 @@ class IterableList(list):
 		try:
 			return getattr(self, index)
 		except AttributeError:
-			raise IndexError( "No item found with id %r" % index )
+			raise IndexError( "No item found with id %r" % self._prefix + index )
 
 class Iterable(object):
 	"""
diff --git a/test/git/test_index.py b/test/git/test_index.py
index 3312abe1..3345949b 100644
--- a/test/git/test_index.py
+++ b/test/git/test_index.py
@@ -199,6 +199,9 @@ class TestTree(TestBase):
 		
 	
 	def _count_existing(self, repo, files):
+		"""
+		Returns count of files that actually exist in the repository directory.
+		"""
 		existing = 0
 		basedir = repo.git.git_dir
 		for f in files:
@@ -207,19 +210,6 @@ class TestTree(TestBase):
 		return existing
 	# END num existing helper
 	
-		
-	def _make_file(self, rela_path, data, repo=None):
-		"""
-		Create a file at the given path relative to our repository, filled
-		with the given data. Returns absolute path to created file.
-		"""
-		repo = repo or self.rorepo
-		abs_path = os.path.join(repo.git.git_dir, rela_path)
-		fp = open(abs_path, "w")
-		fp.write(data)
-		fp.close()
-		return abs_path
-		
 	@with_rw_repo('0.1.6')
 	def test_index_mutation(self, rw_repo):
 		index = rw_repo.index
diff --git a/test/git/test_refs.py b/test/git/test_refs.py
index 979165ef..0a70af1f 100644
--- a/test/git/test_refs.py
+++ b/test/git/test_refs.py
@@ -68,6 +68,12 @@ class TestRefs(TestBase):
 			assert prev_object is not cur_object	# but are different instances
 		# END for each head
 		
+	def test_refs(self):
+		types_found = set()
+		for ref in self.rorepo.refs:
+			types_found.add(type(ref))
+		assert len(types_found) == 3 
+		
 	@with_rw_repo('0.1.6')
 	def test_head_reset(self, rw_repo):
 		cur_head = rw_repo.head
diff --git a/test/git/test_remote.py b/test/git/test_remote.py
index ef00056d..156e7764 100644
--- a/test/git/test_remote.py
+++ b/test/git/test_remote.py
@@ -6,13 +6,334 @@
 
 from test.testlib import *
 from git import *
+import tempfile
+import shutil
+import os
+import random
+
+# assure we have repeatable results 
+random.seed(0)
+
+class TestPushProgress(PushProgress):
+	__slots__ = ( "_seen_lines", "_stages_per_op" )
+	def __init__(self):
+		super(TestPushProgress, self).__init__()
+		self._seen_lines = list()
+		self._stages_per_op = dict()
+		
+	def _parse_progress_line(self, line):
+		# we may remove the line later if it is dropped
+		# Keep it for debugging
+		self._seen_lines.append(line)
+		super(TestPushProgress, self)._parse_progress_line(line)
+		assert len(line) > 1, "line %r too short" % line
+		
+	def line_dropped(self, line):
+		try:
+			self._seen_lines.remove(line)
+		except ValueError:
+			pass
+		
+	def update(self, op_code, cur_count, max_count=None, message=''):
+		# check each stage only comes once
+		op_id = op_code & self.OP_MASK
+		assert op_id in (self.COUNTING, self.COMPRESSING, self.WRITING)
+		
+		self._stages_per_op.setdefault(op_id, 0)
+		self._stages_per_op[ op_id ] = self._stages_per_op[ op_id ] | (op_code & self.STAGE_MASK)
+		
+		if op_code & (self.WRITING|self.END) == (self.WRITING|self.END):
+			assert message
+		# END check we get message 
+		
+	def make_assertion(self):
+		if not self._seen_lines:
+			return 
+		
+		# sometimes objects are not compressed which is okay
+		assert len(self._seen_ops) in (2,3)
+		assert self._stages_per_op
+		
+		# must have seen all stages
+		for op, stages in self._stages_per_op.items():
+			assert stages & self.STAGE_MASK == self.STAGE_MASK
+		# END for each op/stage
 
 class TestRemote(TestBase):
 	
+	def _print_fetchhead(self, repo):
+		fp = open(os.path.join(repo.path, "FETCH_HEAD"))
+		print fp.read()
+		fp.close()
+		
+		
+	def _test_fetch_result(self, results, remote):
+		# self._print_fetchhead(remote.repo)
+		assert len(results) > 0 and isinstance(results[0], FetchInfo)
+		for info in results:
+			if isinstance(info.ref, Reference):
+				assert info.flags != 0
+			# END reference type flags handling 
+			assert isinstance(info.ref, (SymbolicReference, Reference))
+			if info.flags & info.FORCED_UPDATE:
+				assert isinstance(info.commit_before_forced_update, Commit)
+			else:
+				assert info.commit_before_forced_update is None
+			# END forced update checking  
+		# END for each info
+		
+	def _test_push_result(self, results, remote):
+		assert len(results) > 0 and isinstance(results[0], PushInfo)
+		for info in results:
+			assert info.flags
+			if info.old_commit is not None:
+				assert isinstance(info.old_commit, Commit)
+			if info.flags & info.ERROR:
+				has_one = False
+				for bitflag in (info.REJECTED, info.REMOTE_REJECTED, info.REMOTE_FAILURE):
+					has_one |= bool(info.flags & bitflag)
+				# END for each bitflag
+				assert has_one
+			else:
+				# there must be a remote commit
+				if info.flags & info.DELETED == 0: 
+					assert isinstance(info.local_ref, Reference)
+				else:
+					assert info.local_ref is None
+				assert type(info.remote_ref) in (TagReference, RemoteReference)
+			# END error checking
+		# END for each info 
+		
+		
+	def _test_fetch_info(self, repo):
+		self.failUnlessRaises(ValueError, FetchInfo._from_line, repo, "nonsense", '')
+		self.failUnlessRaises(ValueError, FetchInfo._from_line, repo, "? [up to date]      0.1.7RC    -> origin/0.1.7RC", '')
+		
+	def _commit_random_file(self, repo):
+		#Create a file with a random name and random data and commit it to  repo.
+		# Return the commited absolute file path
+		index = repo.index
+		new_file = self._make_file(os.path.basename(tempfile.mktemp()),str(random.random()), repo)
+		index.add([new_file])
+		index.commit("Committing %s" % new_file)
+		return new_file
+		
+	def _test_fetch(self,remote, rw_repo, remote_repo):
+		# specialized fetch testing to de-clutter the main test
+		self._test_fetch_info(rw_repo)
+		
+		def fetch_and_test(remote, **kwargs):
+			res = remote.fetch(**kwargs)
+			self._test_fetch_result(res, remote)
+			return res
+		# END fetch and check
+		
+		def get_info(res, remote, name):
+			return res["%s/%s"%(remote,name)]
+		
+		# put remote head to master as it is garantueed to exist
+		remote_repo.head.reference = remote_repo.heads.master
+		
+		res = fetch_and_test(remote)
+		# all uptodate
+		for info in res:
+			assert info.flags & info.HEAD_UPTODATE
+		
+		# rewind remote head to trigger rejection
+		# index must be false as remote is a bare repo
+		rhead = remote_repo.head
+		remote_commit = rhead.commit
+		rhead.reset("HEAD~2", index=False)
+		res = fetch_and_test(remote)
+		mkey = "%s/%s"%(remote,'master')
+		master_info = res[mkey]
+		assert master_info.flags & FetchInfo.FORCED_UPDATE and master_info.note is not None
+		
+		# normal fast forward - set head back to previous one
+		rhead.commit = remote_commit
+		res = fetch_and_test(remote)
+		assert res[mkey].flags & FetchInfo.FAST_FORWARD
+		
+		# new remote branch
+		new_remote_branch = Head.create(remote_repo, "new_branch")
+		res = fetch_and_test(remote)
+		new_branch_info = get_info(res, remote, new_remote_branch)
+		assert new_branch_info.flags & FetchInfo.NEW_HEAD
+		
+		# remote branch rename ( causes creation of a new one locally )
+		new_remote_branch.rename("other_branch_name")
+		res = fetch_and_test(remote)
+		other_branch_info = get_info(res, remote, new_remote_branch)
+		assert other_branch_info.ref.commit == new_branch_info.ref.commit
+		
+		# remove new branch
+		Head.delete(new_remote_branch.repo, new_remote_branch)
+		res = fetch_and_test(remote)
+		# deleted remote will not be fetched
+		self.failUnlessRaises(IndexError, get_info, res, remote, new_remote_branch)
+		
+		# prune stale tracking branches
+		stale_refs = remote.stale_refs
+		assert len(stale_refs) == 2 and isinstance(stale_refs[0], RemoteReference)
+		RemoteReference.delete(rw_repo, *stale_refs)
+		
+		# test single branch fetch with refspec including target remote
+		res = fetch_and_test(remote, refspec="master:refs/remotes/%s/master"%remote)
+		assert len(res) == 1 and get_info(res, remote, 'master')
+		
+		# ... with respec and no target
+		res = fetch_and_test(remote, refspec='master')
+		assert len(res) == 1
+		
+		# add new tag reference
+		rtag = TagReference.create(remote_repo, "1.0-RV_hello.there")
+		res = fetch_and_test(remote, tags=True)
+		tinfo = res[str(rtag)]
+		assert isinstance(tinfo.ref, TagReference) and tinfo.ref.commit == rtag.commit
+		assert tinfo.flags & tinfo.NEW_TAG
+		
+		# adjust tag commit
+		rtag.object = rhead.commit.parents[0].parents[0]
+		res = fetch_and_test(remote, tags=True)
+		tinfo = res[str(rtag)]
+		assert tinfo.commit == rtag.commit
+		assert tinfo.flags & tinfo.TAG_UPDATE
+		
+		# delete remote tag - local one will stay
+		TagReference.delete(remote_repo, rtag)
+		res = fetch_and_test(remote, tags=True)
+		self.failUnlessRaises(IndexError, get_info, res, remote, str(rtag))
+		
+		# provoke to receive actual objects to see what kind of output we have to 
+		# expect. For that we need a remote transport protocol
+		# Create a new UN-shared repo and fetch into it after we pushed a change
+		# to the shared repo
+		other_repo_dir = tempfile.mktemp("other_repo")
+		# must clone with a local path for the repo implementation not to freak out
+		# as it wants local paths only ( which I can understand )
+		other_repo = remote_repo.clone(other_repo_dir, shared=False)
+		remote_repo_url = "git://localhost%s"%remote_repo.path
+		
+		# put origin to git-url
+		other_origin = other_repo.remotes.origin 
+		other_origin.config_writer.set("url", remote_repo_url)
+		# it automatically creates alternates as remote_repo is shared as well.
+		# It will use the transport though and ignore alternates when fetching
+		# assert not other_repo.alternates	# this would fail
+		
+		# assure we are in the right state
+		rw_repo.head.reset(remote.refs.master, working_tree=True)
+		try:
+			self._commit_random_file(rw_repo)
+			remote.push(rw_repo.head.reference)
+			
+			# here I would expect to see remote-information about packing 
+			# objects and so on. Unfortunately, this does not happen 
+			# if we are redirecting the output - git explicitly checks for this
+			# and only provides progress information to ttys
+			res = fetch_and_test(other_origin)
+		finally:
+			shutil.rmtree(other_repo_dir)
+		# END test and cleanup
+		
+	def _test_push_and_pull(self,remote, rw_repo, remote_repo):
+		# push our changes
+		lhead = rw_repo.head
+		lindex = rw_repo.index
+		# assure we are on master and it is checked out where the remote is
+		lhead.reference = rw_repo.heads.master
+		lhead.reset(remote.refs.master, working_tree=True)
+		
+		# push without spec should fail ( without further configuration )
+		# well, works nicely
+		# self.failUnlessRaises(GitCommandError, remote.push)
+		
+		# simple file push
+		self._commit_random_file(rw_repo)
+		progress = TestPushProgress()
+		res = remote.push(lhead.reference, progress)
+		assert isinstance(res, IterableList)
+		self._test_push_result(res, remote)
+		progress.make_assertion()
+		
+		# rejected - undo last commit
+		lhead.reset("HEAD~1")
+		res = remote.push(lhead.reference)
+		assert res[0].flags & PushInfo.ERROR 
+		assert res[0].flags & PushInfo.REJECTED
+		self._test_push_result(res, remote)
+		
+		# force rejected pull
+		res = remote.push('+%s' % lhead.reference)
+		assert res[0].flags & PushInfo.ERROR == 0 
+		assert res[0].flags & PushInfo.FORCED_UPDATE
+		self._test_push_result(res, remote)
+		
+		# invalid refspec
+		res = remote.push("hellothere")
+		assert len(res) == 0
+		
+		# push new tags 
+		progress = TestPushProgress()
+		to_be_updated = "my_tag.1.0RV"
+		new_tag = TagReference.create(rw_repo, to_be_updated)
+		other_tag = TagReference.create(rw_repo, "my_obj_tag.2.1aRV", message="my message")
+		res = remote.push(progress=progress, tags=True)
+		assert res[-1].flags & PushInfo.NEW_TAG
+		progress.make_assertion()
+		self._test_push_result(res, remote)
+		
+		# update push new tags
+		# Rejection is default
+		new_tag = TagReference.create(rw_repo, to_be_updated, ref='HEAD~1', force=True)
+		res = remote.push(tags=True)
+		self._test_push_result(res, remote)
+		assert res[-1].flags & PushInfo.REJECTED and res[-1].flags & PushInfo.ERROR
+		
+		# push force this tag
+		res = remote.push("+%s" % new_tag.path)
+		assert res[-1].flags & PushInfo.ERROR == 0 and res[-1].flags & PushInfo.FORCED_UPDATE
+		
+		# delete tag - have to do it using refspec
+		res = remote.push(":%s" % new_tag.path)
+		self._test_push_result(res, remote)
+		assert res[0].flags & PushInfo.DELETED
+		
+		# push new branch
+		new_head = Head.create(rw_repo, "my_new_branch")
+		progress = TestPushProgress()
+		res = remote.push(new_head, progress)
+		assert res[0].flags & PushInfo.NEW_HEAD
+		progress.make_assertion()
+		self._test_push_result(res, remote)
+		
+		# delete new branch on the remote end and locally
+		res = remote.push(":%s" % new_head.path)
+		self._test_push_result(res, remote)
+		Head.delete(rw_repo, new_head)
+		assert res[-1].flags & PushInfo.DELETED
+		
+		# --all
+		res = remote.push(all=True)
+		self._test_push_result(res, remote)
+		
+		# pull is essentially a fetch + merge, hence we just do a light 
+		# test here, leave the reset to the actual merge testing
+		# fails as we did not specify a branch and there is no configuration for it
+		self.failUnlessRaises(GitCommandError, remote.pull)
+		remote.pull('master')
+		
+		# cleanup - delete created tags and branches as we are in an innerloop on 
+		# the same repository
+		TagReference.delete(rw_repo, new_tag, other_tag)
+		remote.push(":%s" % other_tag.path)
+	
 	@with_rw_and_rw_remote_repo('0.1.6')
 	def test_base(self, rw_repo, remote_repo):
 		num_remotes = 0
 		remote_set = set()
+		ran_fetch_test = False
+		
 		for remote in rw_repo.remotes:
 			num_remotes += 1
 			assert remote == remote
@@ -25,7 +346,7 @@ class TestRemote(TestBase):
 			assert refs
 			for ref in refs:
 				assert ref.remote_name == remote.name
-				assert ref.remote_branch
+				assert ref.remote_head
 			# END for each ref
 			
 			# OPTIONS
@@ -59,12 +380,21 @@ class TestRemote(TestBase):
 				assert remote.rename(prev_name).name == prev_name
 			# END for each rename ( back to prev_name )
 			
-			remote.fetch()
-			self.failUnlessRaises(GitCommandError, remote.pull)
-			remote.pull('master')
+			# PUSH/PULL TESTING
+			self._test_push_and_pull(remote, rw_repo, remote_repo)
+			
+			# FETCH TESTING
+			# Only for remotes - local cases are the same or less complicated 
+			# as additional progress information will never be emitted
+			if remote.name == "daemon_origin":
+				self._test_fetch(remote, rw_repo, remote_repo)
+				ran_fetch_test = True
+			# END fetch test  
+			
 			remote.update()
-			self.fail("test push once there is a test-repo")
 		# END for each remote
+		
+		assert ran_fetch_test
 		assert num_remotes
 		assert num_remotes == len(remote_set)
 		
@@ -77,6 +407,7 @@ class TestRemote(TestBase):
 		arg_list = (new_name, "git@server:hello.git")
 		remote = Remote.create(bare_rw_repo, *arg_list )
 		assert remote.name == "test_new_one"
+		assert remote in bare_rw_repo.remotes
 		
 		# create same one again
 		self.failUnlessRaises(GitCommandError, Remote.create, bare_rw_repo, *arg_list)
diff --git a/test/testlib/helper.py b/test/testlib/helper.py
index ab4b9f4e..081299be 100644
--- a/test/testlib/helper.py
+++ b/test/testlib/helper.py
@@ -5,7 +5,7 @@
 # the BSD License: http://www.opensource.org/licenses/bsd-license.php
 
 import os
-from git import Repo
+from git import Repo, Remote
 from unittest import TestCase
 import tempfile
 import shutil
@@ -115,8 +115,15 @@ def with_rw_repo(working_tree_ref):
 def with_rw_and_rw_remote_repo(working_tree_ref):
 	"""
 	Same as with_rw_repo, but also provides a writable remote repository from which the
-	rw_repo has been forked. The remote repository was cloned as bare repository from 
-	the rorepo, wheras the rw repo has a working tree and was cloned from the remote repository.
+	rw_repo has been forked as well as a handle for a git-daemon that may be started to 
+	run the remote_repo.
+	The remote repository was cloned as bare repository from the rorepo, wheras 
+	the rw repo has a working tree and was cloned from the remote repository.
+	
+	remote_repo has two remotes: origin and daemon_origin. One uses a local url, 
+	the other uses a server url. The daemon setup must be done on system level 
+	and should be an inetd service that serves tempdir.gettempdir() and all 
+	directories in it.
 	
 	The following scetch demonstrates this::
 	 rorepo ---<bare clone>---> rw_remote_repo ---<clone>---> rw_repo
@@ -135,6 +142,28 @@ def with_rw_and_rw_remote_repo(working_tree_ref):
 			rw_remote_repo = self.rorepo.clone(remote_repo_dir, shared=True, bare=True)
 			rw_repo = rw_remote_repo.clone(repo_dir, shared=True, bare=False, n=True)		# recursive alternates info ?
 			rw_repo.git.checkout("-b", "master", working_tree_ref)
+			
+			# prepare for git-daemon
+			rw_remote_repo.daemon_export = True
+			
+			# this thing is just annoying !
+			crw = rw_remote_repo.config_writer()
+			section = "daemon"
+			try:
+				crw.add_section(section)
+			except Exception:
+				pass
+			crw.set(section, "receivepack", True)
+			# release lock
+			del(crw)
+			
+			# initialize the remote - first do it as local remote and pull, then 
+			# we change the url to point to the daemon. The daemon should be started
+			# by the user, not by us
+			d_remote = Remote.create(rw_repo, "daemon_origin", remote_repo_dir)
+			d_remote.fetch()
+			d_remote.config_writer.set('url', "git://localhost%s" % remote_repo_dir)
+			
 			try:
 				return func(self, rw_repo, rw_remote_repo)
 			finally:
@@ -175,4 +204,15 @@ class TestBase(TestCase):
 		each test type has its own repository
 		"""
 		cls.rorepo = Repo(GIT_REPO)
-		
+	
+	def _make_file(self, rela_path, data, repo=None):
+		"""
+		Create a file at the given path relative to our repository, filled
+		with the given data. Returns absolute path to created file.
+		"""
+		repo = repo or self.rorepo
+		abs_path = os.path.join(repo.git.git_dir, rela_path)
+		fp = open(abs_path, "w")
+		fp.write(data)
+		fp.close()
+		return abs_path