| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
| |
Add a configuration variable, indexhelper.exitAfter, which provides a
default time to keep the index-helper alive. This is useful with
indexhelper.autorun; some users will want to keep the
automatically-run index-helper alive across their lunch break and will
thus set indexhelper.exitAfter to a high value.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All the known heavy code blocks are measured (except object database
access). This should help identify if an optimization is effective or
not. An unoptimized git-status would give something like below (92% of
time is accounted). To sum up the effort of making git scale better:
- read cache line is being addressed by index-helper
- preload/refresh index by watchman
- read packed refs by lmdb backend
- diff-index by rebuilding cache-tree
- read directory by untracked cache and watchman
- write index by split index
- name hash potientially by index-helper
read-cache.c:2075 performance: 0.004058570 s: read cache .../index
preload-index.c:104 performance: 0.004419864 s: preload index
read-cache.c:1265 performance: 0.000185224 s: refresh index
refs/files-backend.c:1100 performance: 0.001935788 s: read packed refs
diff-lib.c:240 performance: 0.000144132 s: diff-files
diff-lib.c:506 performance: 0.013592000 s: diff-index
name-hash.c:128 performance: 0.000614177 s: initialize name hash
dir.c:2030 performance: 0.015814103 s: read directory
read-cache.c:2565 performance: 0.004052343 s: write index, changed mask = 2
trace.c:420 performance: 0.048365509 s: git command: './git' 'status'
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
| |
Introduce a new config option, indexhelper.autorun, to automatically
run git index-helper before starting up a builtin git command. This
enables users to keep index-helper running without manual
intervention.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
| |
Soon, we'll want to automatically start index-helper, so we need
a mode that silently exits if it can't start up (either because
it's not in a git dir, or because another one is already running).
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
| |
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
| |
Add a new command (and command-line arg) to allow index-helpers to
exit cleanly.
This is mainly useful for tests.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
| |
For installations that have centrally-managed configuration, it's
easier to set a config once than to run update-index on every
repository.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Make git checkout (and other unpack_tree operations) preserve the
untracked cache and watchman status. This is valuable for two reasons:
1. Often, an unpack_tree operation will not touch large parts of the
working tree, and thus most of the untracked cache will continue to be
valid.
2. Even if the untracked cache were entirely invalidated by such an
operation, the user has signaled their intention to have such a cache,
and we don't want to throw it away.
The same logic applies to the watchman state.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
| |
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Watchman is hidden behind index-helper. Before git tries to read the
index from shm, it notifies index-helper through the socket and waits
for index-helper to prepare a file for sharing memory (with
MAP_SHARED). index-helper then contacts watchman, updates 'WAMA'
extension and put it in a separate file and wakes git up with a reply
to git's socket.
Git uses this extension to not lstat unchanged entries. Git only
trusts the 'WAMA' extension when it's received from the separate file,
not from disk. Unmarked entries are "clean". Marked entries are dirty
from watchman point of view. If it finds out some entries are
'watchman-dirty', but are really unchanged (e.g. the file was changed,
then reverted back), then Git will clear the marking in 'WAMA' before
writing it down.
Hiding watchman behind index-helper means you need both daemons. You
can't run watchman alone. Not so good. But on the other hand, 'git'
binary is not linked to watchman/json libraries, which is good for
packaging. Core git package will run fine without watchman-related
packages. If they need watchman, they can install git-index-helper and
dependencies.
This also lets us trust anything in the untracked cache that we haven't
marked invalid, saving those stat() calls.
Another reason for tying watchman to index-helper is, when used with
untracked cache, we need to keep track of $GIT_WORK_TREE file
listing. That kind of list can be kept in index-helper.
Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous patch has the logic to clear bits in 'WAMA' bitmap. This
patch has logic to set bits as told by watchman. The missing bit,
_using_ these bits, are not here yet.
A lot of this code is written by David Turner originally, mostly from
[1]. I'm just copying and polishing it a bit.
[1] http://article.gmane.org/gmane.comp.version-control.git/248006
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The extension contains a bitmap, one bit for each entry in the
index. If the n-th bit is zero, the n-th entry is considered
unchanged, we can ce_mark_uptodate() it without refreshing. If the bit
is non-zero and we found out the corresponding file is clean after
refresh, we can clear the bit.
In addition, there's a list of directories in the untracked-cache
to invalidate (because they have new or modified entries).
The 'skipping refresh' bit is not in this patch yet as we would need
watchman. More details in later patches.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
| |
Instead of writing warnings to stderr, write them to a log. Later, we'll
probably be daemonized, so writing to stderr will be pointless.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
| |
We detach after creating and opening the socket, because otherwise
we might return control to the shell before index-helper is ready to
accept commands. This might lead to flaky tests.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
| |
This allows signal handlers and atexit functions to realize this
situation and not clean up.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There are "holes" in the index-helper approach because the shared
memory is not verified again by git. If $USER is compromised, shared
memory could be modified. But anyone who could do this could already
modify $GIT_DIR/index. A more realistic risk is some bugs in
index-helper that produce corrupt shared memory. --strict is added to
avoid that.
Strictly speaking there's still a very small gap where corrupt shared
memory could still be read by git: after we write the trailing SHA-1 in
the shared memory (thus signaling "this shm is ready") and before
verify_shm() detects an error.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of reading the index from disk and worrying about disk
corruption, the index is cached in memory (memory bit-flips happen
too, but hopefully less often). The result is faster read. Read time
is reduced by 70%.
The biggest gain is not having to verify the trailing SHA-1, which
takes lots of time especially on large index files. But this also
opens doors for further optimizations:
- we could create an in-memory format that's essentially the memory
dump of the index to eliminate most of parsing/allocation
overhead. The mmap'd memory can be used straight away. Experiment
[1] shows we could reduce read time by 88%.
- we could cache non-index info such as name hash
Shared memory is done by storing files in a per-repository temporary
directory. This is more portable than shm (which requires
posix-realtime and has various quirks on OS X). It might even work on
Windows, although this has not been tested. The shared memory file's
name follows the template "shm-<object>-<SHA1>" where <SHA1> is the
trailing SHA-1 of the index file. <object> is "index" for cached index
files (and might later be "name-hash" for name-hash cache). If such
shared memory exists, it contains the same index content as on
disk. The content is already validated by the daemon and git won't
validate it again (except comparing the trailing SHA-1s).
We also add some bits to the index (to_shm and from_shm) to track
when an index came from shared memory or is going to shared memory.
We keep this daemon's logic as thin as possible. The "brain" stays in
git. So the daemon can read and validate stuff, but that's all it's
allowed to do. It does not add/create new information. It doesn't even
accept direct updates from git.
Git can poke the daemon via unix domain sockets to tell it to refresh
the index cache, or to keep it alive some more minutes. It can't give
any real index data directly to the daemon. Real data goes to disk
first, then the daemon reads and verifies it from there. The daemon
only handles $GIT_DIR/index, not temporary index files; it only gets
poked for the former.
$GIT_DIR/index-helper.sock is the socket for the daemon process. The
daemon reads from the socket and executes commands.
Named pipes were considered for portability reasons, but then commands
that need replies from the daemon would have to open their own pipes,
since a named pipe should only have one reader. Unix domain sockets
don't have this problem.
On webkit.git with index format v2, duplicating 8 times to 1.5m
entries and 236MB in size:
(vanilla) 0.50 s: read_index_from .git/index
(index-helper) 0.18 s: read_index_from .git/index
Interestingly with index v4, we get less out of index-helper. It makes
sense as v4 requires more processing after loading the index:
(vanilla) 0.37 s: read_index_from .git/index
(index-helper) 0.22 s: read_index_from .git/index
[1] http://thread.gmane.org/gmane.comp.version-control.git/247268/focus=248771
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Helped-by: Keith McGuigan <kmaggg@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This keeps #ifdef at the callee instead of caller, it's less messier.
The caller in question is in read-cache.c which, unlike other
unix-socket callers so far, is always built regardless of unix socket
support. No extra handling (for ENOSYS) is needed because in this
build, index-helper does not exist, $GIT_DIR/index-helper.sock does
not exist, so no unix socket call is made by read-cache.c in the first
place.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
packet_write calls write_or_die, which dies with a sigpipe even if
calling code has explicitly blocked that signal.
Add packet_write_gently and packet_flush_gently, which don't. Soon,
we will use this for communication with git index-helper, which, being
merely an optimization, should be permitted to die without disrupting
clients.
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Later, we will introduce git index-helper to share this memory with
other git processes.
We only unmap it when we discard the index (although the kernel may of
course choose to page it out).
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
|
|
| |
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |
|
|
| |
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |\
| |
| |
| |
| |
| | |
* maint:
Documentation: fix broken linkgit to git-config
git-compat-util: st_add4: work around gcc 4.2.x compiler crash
|
| | |\
| | |
| | |
| | |
| | | |
* mm/doc-hooks-linkgit-fix:
Documentation: fix broken linkgit to git-config
|
| | | |
| | |
| | |
| | |
| | | |
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| | |\ \
| | | |
| | | |
| | | |
| | | | |
* es/st-add4-gcc-4.2-workaround:
git-compat-util: st_add4: work around gcc 4.2.x compiler crash
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Although changes by 5b442c4 (tree-diff: catch integer overflow in
combine_diff_path allocation, 2016-02-19) are perfectly valid, they
unfortunately trigger an internal compiler error in gcc 4.2.x:
combine-diff.c: In function 'diff_tree_combined':
combine-diff.c:1391: internal compiler error: Segmentation fault: 11
Experimentation reveals that changing st_add4()'s argument evaluation
order is sufficient to sidestep this problem.
Although st_add3() does not trigger the compiler bug, for style
consistency, change its argument evaluation order to match.
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |\ \ \ \
| | | | |
| | | | |
| | | | |
| | | | | |
* tb/avoid-gcc-on-darwin-10-6:
Revert "config.mak.uname: use clang for Mac OS X 10.6"
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This reverts commit 7b6daf8d2fee1a9866b1d4eddbfaa5dbc42c5dbb.
Now that st_add4() has been patched to work around the gcc 4.2.x
compiler crash, revert the sledge-hammer approach of forcing Mac OS X
10.6 to unconditionally use 'clang' rather than the default compiler
(gcc).
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| |\ \ \ \ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
l10n-2.8.0-rnd3
* tag 'l10n-2.8.0-rnd3' of git://github.com/git-l10n/git-po:
l10n: zh_CN: review for git v2.8.0 l10n round 2
l10n: de.po: add missing newlines
l10n: de.po: translate 22 new messages
l10n: ko.po: Update Korean translation
l10n: fr.po v2.8.0 round 3
l10n: sv.po: Update Swedish translation (2530t0f0u)
l10n: ru.po: update Russian translation
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | | |
Signed-off-by: Ray Chen <oldsharp@gmail.com>
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | | |
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Translate 22 new messages came from git.pot update in f1522b2
(l10n: git.pot: v2.8.0 round 2 (21 new, 1 removed)) and a5a4168
(l10n: git.pot: Add one new message for Git 2.8.0).
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Acked-by: Matthias Rüster <matthias.ruester@gmail.com>
|
| | |\ \ \ \ \
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
* 'fr_v2.8.0_r3' of git://github.com/jnavila/git:
l10n: fr.po v2.8.0 round 3
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Jean-Noel Avila <jn.avila@free.fr>
|
| | |\ \ \ \ \ \
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
* 'ko/merge-l10n' of https://github.com/changwoo/git-l10n-ko:
l10n: ko.po: Update Korean translation
|
| | | |/ / / / /
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Changwoo Ryu <cwryu@debian.org>
|
| | |\ \ \ \ \ \
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
* 'master' of git://github.com/nafmo/git-l10n-sv:
l10n: sv.po: Update Swedish translation (2530t0f0u)
|
| | | |/ / / / /
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Peter Krefting <peter@softwolves.pp.se>
|
| | |/ / / / /
| | | | | |
| | | | | |
| | | | | | |
Signed-off-by: Dimitriy Ryazantcev <dimitriy.ryazantcev@gmail.com>
|
| |\ \ \ \ \ \
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
* 'master' of git://ozlabs.org/~paulus/gitk:
gitk: Follow themed bgcolor in help dialogs
gitk: fr.po: Sync translations with git
gitk: Update French translation (311t)
gitk: Update German translation
gitk: Update Bulgarian translation (311t)
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Make Help > About & Key bindings dialogs readable if theme
has changed font color to something incompatible with white.
Signed-off-by: Guillermo S. Romero <gsromero@infernal-iceberg.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Jean-Noel Avila <jn.avila@free.fr>
Signed-off-by: Paul Mackerras <paulus@samba.org>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Jean-Noel Avila <jn.avila@free.fr>
Signed-off-by: Paul Mackerras <paulus@samba.org>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Signed-off-by: Alexander Shopov <ash@kambanaria.org>
Signed-off-by: Paul Mackerras <paulus@samba.org>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
We will be postponing this to a later cycle.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This reverts commit 5e57f9c3dfe7dd44a1b56bb5b3327d7a1356ec7c, reversing
changes made to e79112d21024beb997951381db21a70b087d459d.
We will be postponing nd/exclusion-regression-fix topic to later
cycle.
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This reverts commit e80aae51f2be908e37fca47ea0dff6d7861c8497, reversing
changes made to 68846a92eafa6b2bfae778d0a656443a9fa61e59.
We will be postponing nd/exclusion-regression-fix topic to later
cycle.
|
| |\ \ \ \ \ \ \
| | |_|_|/ / /
| |/| | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
* maint:
Git 2.7.4
Git 2.6.6
Git 2.5.5
Git 2.4.11
|
