From c6b948ccab7ce67a60ed2e6360811888c579fdf7 Mon Sep 17 00:00:00 2001 From: Talon <77453497+Talonkinkade@users.noreply.github.com> Date: Wed, 13 Nov 2024 04:20:05 -0600 Subject: [PATCH] Squashed commit of the following: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit commit 2996b56fa7470c29e418e4e7249629ea74cdfdca Author: Johannes Schindelin Date: Fri Oct 11 16:55:26 2024 +0200 amend! Add experimental 'git survey' builtin (#5174) Add experimental 'git survey' builtin (#5174) This introduces `git survey` to Git for Windows ahead of upstream for the express purpose of getting the path-based analysis in the hands of more folks. The inspiration of this builtin is [`git-sizer`](https://github.com/github/git-sizer), but since that command relies on `git cat-file --batch` to get the contents of objects, it has limits to how much information it can provide. This is mostly a rewrite of the `git survey` builtin that was introduced into the `microsoft/git` fork in microsoft/git#667. That version had a lot more bells and whistles, including an analysis much closer to what `git-sizer` provides. The biggest difference in this version is that this one is focused on using the path-walk API in order to visit batches of objects based on a common path. This allows identifying, for instance, the path that is contributing the most to the on-disk size across all versions at that path. For example, here are the top ten paths contributing to my local Git repository (which includes `microsoft/git` and `gitster/git`): ``` TOP FILES BY DISK SIZE ============================================================================ Path | Count | Disk Size | Inflated Size -----------------------------------------+-------+-----------+-------------- whats-cooking.txt | 1373 | 11637459 | 37226854 t/helper/test-gvfs-protocol | 2 | 6847105 | 17233072 git-rebase--helper | 1 | 6027849 | 15269664 compat/mingw.c | 6111 | 5194453 | 463466970 t/helper/test-parse-options | 1 | 3420385 | 8807968 t/helper/test-pkt-line | 1 | 3408661 | 8778960 t/helper/test-dump-untracked-cache | 1 | 3408645 | 8780816 t/helper/test-dump-fsmonitor | 1 | 3406639 | 8776656 po/vi.po | 104 | 1376337 | 51441603 po/de.po | 210 | 1360112 | 71198603 ``` This kind of analysis has been helpful in identifying the reasons for growth in a few internal monorepos. Those findings motivated the changes in #5157 and #5171. With this early version in Git for Windows, we can expand the reach of the experimental tool in advance of it being contributed to the upstream project. Unfortunately, this will mean that in the next `microsoft/git` rebase, Jeff Hostetler's version will need to be pulled out since there are enough conflicts. These conflicts include how tables are stored and generated, as the version in this PR is slightly more general to allow for different kinds of data. Signed-off-by: Johannes Schindelin commit 5f1db046948aa39d2a28ddc9fba5a8975df40fa3 Author: Johannes Schindelin Date: Fri Oct 11 16:54:53 2024 +0200 amend! path-walk: improve path-walk speed with many tags (#5205) path-walk: improve path-walk speed with many tags (#5205) In the presence of many tags, the use of oid_array_lookup() can become extremely slow. We should rely upon the SEEN bit instead. This affects the tag-peeling walk as well as the switch statement for adding the peeled object to the correct oid_array. ---- Derrick Stolee found this while testing the 2.47.0.vfs.0.0 pre-release against a repo with many annotated tags. This is a backport of https://github.com/microsoft/git/pull/695. Signed-off-by: Johannes Schindelin commit f380030a00c15fa00d66799f875cf2c986e1fc97 Merge: 61ec9331b6 c33368b771 Author: Johannes Schindelin Date: Fri Oct 11 13:58:53 2024 +0200 path-walk: improve path-walk speed with many tags (#5205) In the presence of many tags, the use of oid_array_lookup() can become extremely slow. We should rely upon the SEEN bit instead. This affects the tag-peeling walk as well as the switch statement for adding the peeled object to the correct oid_array. ---- @derrickstolee found this while testing the 2.47.0.vfs.0.0 pre-release against a repo with many annotated tags. This is a backport of https://github.com/microsoft/git/pull/695. commit 61ec9331b61ab857a259d6e1c4c4f86775b34f26 Merge: 12031c299c 3ead00a02c Author: Johannes Schindelin Date: Thu Jun 7 08:45:01 2018 +0200 Merge 'readme' into HEAD Add a README.md for GitHub goodness. Signed-off-by: Johannes Schindelin commit c33368b7717c90296ad2ffac3b8ffb364e6173ef Author: Derrick Stolee Date: Wed Oct 9 09:57:32 2024 -0400 path-walk: improve path-walk speed with many tags In the presence of many tags, the use of oid_array_lookup() can become extremely slow. We should rely upon the SEEN bit instead. This affects the tag-peeling walk as well as the switch statement for adding the peeled object to the correct oid_array. Signed-off-by: Derrick Stolee Signed-off-by: Johannes Schindelin commit 12031c299c10eeb8af636303901db371b931b272 Merge: 3c2f5aa314 740b27f844 Author: Johannes Schindelin Date: Wed Oct 7 16:13:40 2020 +0200 Merge pull request #2837 from dscho/monitor-component-updates Start monitoring updates of Git for Windows' component in the open commit 3c2f5aa3148b8487f15ab828482ddc4e222c5262 Merge: fe2b01e513 8c90275e38 Author: Johannes Schindelin Date: Tue Jan 24 11:46:21 2023 +0100 Merge branch 'deprecate-core.useBuiltinFSMonitor' Originally introduced as `core.useBuiltinFSMonitor` in Git for Windows and developed, improved and stabilized there, the built-in FSMonitor only made it into upstream Git (after unnecessarily long hemming and hawing and throwing overly perfectionist style review sticks into the spokes) as `core.fsmonitor = true`. In Git for Windows, with this topic branch, we re-introduce the now-obsolete config setting, with warnings suggesting to existing users how to switch to the new config setting, with the intention to ultimately drop the patch at some stage. Signed-off-by: Johannes Schindelin commit fe2b01e51341088486df2b3747c0a40b365af1a5 Merge: c36a4deb88 aa062e96ec Author: Johannes Schindelin Date: Fri Jun 8 18:34:51 2018 +0200 Merge branch 'phase-out-reset-stdin' This topic branch re-adds the deprecated --stdin/-z options to `git reset`. Those patches were overridden by a different set of options in the upstream Git project before we could propose `--stdin`. We offered this in MinGit to applications that wanted a safer way to pass lots of pathspecs to Git, and these applications will need to be adjusted. Instead of `--stdin`, `--pathspec-from-file=-` should be used, and instead of `-z`, `--pathspec-file-nul`. Signed-off-by: Johannes Schindelin commit c36a4deb885361c37441564b1c62731809aa3917 Merge: 2fdfc3089a 2711b9ca0a Author: Johannes Schindelin Date: Sat Jan 15 11:37:56 2022 +0100 Merge branch 'un-revert-editor-save-and-reset' A fix for calling `vim` in Windows Terminal caused a regression and was reverted. We partially un-revert this, to get the fix again. Signed-off-by: Johannes Schindelin commit 2fdfc3089a6e7dd44a8c373f68bfaf6950c4da12 Merge: 42778ded1e a5cc82fd83 Author: Victoria Dye Date: Thu Oct 28 15:16:10 2021 -0400 Merge pull request #3492 from dscho/ns/batched-fsync Switch to batched fsync by default commit 42778ded1e9b9bac269a37c3e4c163dbb200a853 Merge: 0d956e0879 7e12ac9200 Author: Johannes Schindelin Date: Thu Oct 11 23:29:20 2018 +0200 Merge pull request #1170 from dscho/mingw-kill-process Handle Ctrl+C in Git Bash nicely Signed-off-by: Johannes Schindelin commit 0d956e0879144c6c6d736dde7ac80c3bf30c73a6 Merge: 7773e3c9fa 0a57a784e6 Author: Johannes Schindelin Date: Mon Aug 7 22:24:29 2023 +0200 Merge branch 'wsl-file-mode-bits' This patch introduces support to set special NTFS attributes that are interpreted by the Windows Subsystem for Linux as file mode bits, UID and GID. Signed-off-by: Johannes Schindelin commit 7773e3c9fac969b26c6c0238053fb9f7e511d147 Merge: fd8673cf9c c576da7398 Author: Johannes Schindelin Date: Thu Feb 7 14:35:33 2019 +0100 Merge branch 'busybox-w32' Signed-off-by: Johannes Schindelin commit fd8673cf9cf25119fbc242960b08fa9c4c0412fb Merge: 2a3d2866f2 3feb8f7dfb Author: Johannes Schindelin Date: Wed Oct 31 15:08:16 2018 +0100 Merge pull request #1897 from piscisaureus/symlink-attr Specify symlink type in .gitattributes commit 2a3d2866f2186843b09b922c25f6ca9c750e0770 Merge: 4725764578 f93344df80 Author: Johannes Schindelin Date: Mon Aug 7 16:12:10 2023 +0200 mingw: try resetting the read-only bit if rename fails (#4527) With this patch, Git for Windows works as intended on mounted APFS volumes (where renaming read-only files would fail). Signed-off-by: Johannes Schindelin commit 4725764578421fc40b2de1b230f31e7af6d62055 Merge: a4ba13da06 426566b1e4 Author: Johannes Schindelin Date: Fri Oct 12 23:04:03 2018 +0200 Merge 'docker-volumes-are-no-symlinks' This was pull request #1645 from ZCube/master Support windows container. Signed-off-by: Johannes Schindelin commit a4ba13da0615d74ad7c8b4ed9488bee4edd91e2b Merge: f860813f4e bd2c03e214 Author: Johannes Schindelin Date: Thu Nov 15 12:23:43 2018 +0100 Merge branch 'kblees/kb/symlinks' Signed-off-by: Johannes Schindelin commit f860813f4e58b4050eec6c2cd4a896aad7ea6cc5 Merge: 343c75d471 b9a9681993 Author: Johannes Schindelin Date: Thu Nov 15 12:23:43 2018 +0100 Merge branch 'msys2' commit 343c75d4716087b9a4a787a400c9391ba4248460 Merge: 86a198c2e2 0987f685d1 Author: Johannes Schindelin Date: Fri Apr 22 23:46:50 2022 +0200 Merge pull request #3817 from mathstuf/name-too-long-advice clean: suggest using `core.longPaths` if paths are too long to remove commit 86a198c2e27492928a052f924509c5ec5d4cc03c Merge: c7917ce9d3 29f7afc8f8 Author: Jeff Hostetler Date: Wed Sep 29 17:58:38 2021 -0400 Merge branch 'fix-v4-fsmonitor-long-paths' into try-v4-fsmonitor commit c7917ce9d3aaa7a78b42a2eb59519472b7582971 Merge: fa336b3e18 b8923c8fba Author: Johannes Schindelin Date: Thu Nov 15 12:23:43 2018 +0100 Merge branch 'long-paths' commit fa336b3e18bd1ae110429fbfffa61e42cc6cd665 Merge: b885fdedff f210ba75b1 Author: Johannes Schindelin Date: Thu Oct 11 13:38:58 2018 +0200 Merge branch 'gitk-and-git-gui-patches' These are Git for Windows' Git GUI and gitk patches. We will have to decide at some point what to do about them, but that's a little lower priority (as Git GUI seems to be unmaintained for the time being, and the gitk maintainer keeps a very low profile on the Git mailing list, too). Signed-off-by: Johannes Schindelin commit 3ead00a02cbbe54627fd9a2f0a64d9cc19167aa6 Author: Johannes Schindelin Date: Fri Aug 23 14:14:42 2019 +0200 SECURITY.md: document Git for Windows' policies This is the recommended way on GitHub to describe policies revolving around security issues and about supported versions. Signed-off-by: Johannes Schindelin commit 740b27f84487f3459c3be078bba05587b6038a09 Author: Johannes Schindelin Date: Tue Feb 6 18:45:35 2024 +0100 dependabot: help keeping GitHub Actions versions up to date See https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot#enabling-dependabot-version-updates-for-actions for details. Signed-off-by: Johannes Schindelin commit 3092e2ace9b944fe13be59dbe5839ab5bc4e9a4e Author: Alejandro Barreto Date: Fri Mar 9 14:17:54 2018 -0600 Document how $HOME is set on Windows Git documentation refers to $HOME and $XDG_CONFIG_HOME often, but does not specify how or where these values come from on Windows where neither is set by default. The new documentation reflects the behavior of setup_windows_environment() in compat/mingw.c. Signed-off-by: Alejandro Barreto commit 43ce7da7f3866891bfddef68d1e4353838711f9d Author: Johannes Schindelin Date: Tue Feb 20 15:44:57 2018 +0100 .github: Add configuration for the Sentiment Bot The sentiment bot will help detect when things get too heated. Hopefully. Signed-off-by: Johannes Schindelin commit b1fc6f3c7b58dab4a2e0e35a9b980f9feaa11efe Author: Philip Oakley Date: Fri Dec 22 17:15:50 2017 +0000 Modify the GitHub Pull Request template (to reflect Git for Windows) Git for Windows accepts pull requests; Core Git does not. Therefore we need to adjust the template (because it only matches core Git's project management style, not ours). Also: direct Git for Windows enhancements to their contributions page, space out the text for easy reading, and clarify that the mailing list is plain text, not HTML. Signed-off-by: Philip Oakley Signed-off-by: Johannes Schindelin commit f95739fa44d273f84a4801aab986f83d5cac3406 Author: Brendan Forster Date: Thu Feb 18 21:29:50 2016 +1100 Add an issue template With improvements by Clive Chan, Adric Norris, Ben Bodenmiller and Philip Oakley. Helped-by: Clive Chan Helped-by: Adric Norris Helped-by: Ben Bodenmiller Helped-by: Philip Oakley Signed-off-by: Brendan Forster Signed-off-by: Johannes Schindelin commit 862899ec93dc1cb1d91e0eeb31d0c3c6b2c57211 Author: Johannes Schindelin Date: Fri Jan 10 16:16:03 2014 -0600 README.md: Add a Windows-specific preamble Includes touch-ups by 마누엘, Philip Oakley and 孙卓识. Signed-off-by: Johannes Schindelin commit 58997bb46572ba290809301db1b547223004af42 Author: Derrick Stolee Date: Thu Mar 1 12:10:14 2018 -0500 CONTRIBUTING.md: add guide for first-time contributors Getting started contributing to Git can be difficult on a Windows machine. CONTRIBUTING.md contains a guide to getting started, including detailed steps for setting up build tools, running tests, and submitting patches to upstream. [includes an example by Pratik Karki how to submit v2, v3, v4, etc.] Signed-off-by: Derrick Stolee commit 12e0f90bea754eebe6ddf4c5b2af79572a891a63 Author: Johannes Schindelin Date: Fri Oct 11 13:22:24 2019 +0200 Modify the Code of Conduct for Git for Windows The Git project followed Git for Windows' lead and added their Code of Conduct, based on the Contributor Covenant v1.4, later updated to v2.0. We adapt it slightly to Git for Windows. Signed-off-by: Johannes Schindelin commit 898aba7180f0de1d7a26a12ce810788614c4f7b1 Author: Johannes Schindelin Date: Mon Feb 13 13:31:35 2023 +0100 Describe Git for Windows' architecture [no ci] The Git for Windows project has grown quite complex over the years, certainly much more complex than during the first years where the `msysgit.git` repository was abusing Git for package management purposes and the `git/git` fork was called `4msysgit.git`. Let's describe the status quo in a thorough way. Signed-off-by: Johannes Schindelin commit 8c90275e38a704e321af7f7cd1c42911526f6fff Author: Victoria Dye Date: Mon Apr 4 15:38:58 2022 -0700 fsmonitor: reintroduce core.useBuiltinFSMonitor Reintroduce the 'core.useBuiltinFSMonitor' config setting (originally added in 0a756b2a25 (fsmonitor: config settings are repository-specific, 2021-03-05)) after its removal from the upstream version of FSMonitor. Upstream, the 'core.useBuiltinFSMonitor' setting was rendered obsolete by "overloading" the 'core.fsmonitor' setting to take a boolean value. However, several applications (e.g., 'scalar') utilize the original config setting, so it should be preserved for a deprecation period before complete removal: * if 'core.fsmonitor' is a boolean, the user is correctly using the new config syntax; do not use 'core.useBuiltinFSMonitor'. * if 'core.fsmonitor' is unspecified, use 'core.useBuiltinFSMonitor'. * if 'core.fsmonitor' is a path, override and use the builtin FSMonitor if 'core.useBuiltinFSMonitor' is 'true'; otherwise, use the FSMonitor hook indicated by the path. Additionally, for this deprecation period, advise users to switch to using 'core.fsmonitor' to specify their use of the builtin FSMonitor. Signed-off-by: Victoria Dye commit aa062e96ec4348844793382b3eb1cb14c777372e Author: Johannes Schindelin Date: Tue Dec 10 21:41:57 2019 +0100 reset: reinstate support for the deprecated --stdin option The `--stdin` option was a well-established paradigm in other commands, therefore we implemented it in `git reset` for use by Visual Studio. Unfortunately, upstream Git decided that it is time to introduce `--pathspec-from-file` instead. To keep backwards-compatibility for some grace period, we therefore reinstate the `--stdin` option on top of the `--pathspec-from-file` option, but mark it firmly as deprecated. Helped-by: Victoria Dye Helped-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin commit 2711b9ca0a7639de281c8f289c52083823728e03 Author: Johannes Schindelin Date: Thu Nov 25 11:26:41 2021 +0100 Partially un-revert "editor: save and reset terminal after calling EDITOR" In e3f7e01b50be (Revert "editor: save and reset terminal after calling EDITOR", 2021-11-22), we reverted the commit wholesale where the terminal state would be saved and restored before/after calling an editor. The reverted commit was intended to fix a problem with Windows Terminal where simply calling `vi` would cause problems afterwards. To fix the problem addressed by the revert, but _still_ keep the problem with Windows Terminal fixed, let's revert the revert, with a twist: we restrict the save/restore _specifically_ to the case where `vi` (or `vim`) is called, and do not do the same for any other editor. This should still catch the majority of the cases, and will bridge the time until the original patch is re-done in a way that addresses all concerns. Signed-off-by: Johannes Schindelin commit 281b4c972ab365c75653b88c4628698f8bc27858 Author: Johannes Schindelin Date: Tue Sep 29 13:50:59 2020 +0200 Add a GitHub workflow to monitor component updates Rather than using private IFTTT Applets that send mails to this maintainer whenever a new version of a Git for Windows component was released, let's use the power of GitHub workflows to make this process publicly visible. This workflow monitors the Atom/RSS feeds, and opens a ticket whenever a new version was released. Note: Bash sometimes releases multiple patched versions within a few minutes of each other (i.e. 5.1p1 through 5.1p4, 5.0p15 and 5.0p16). The MSYS2 runtime also has a similar system. We can address those patches as a group, so we shouldn't get multiple issues about them. Note further: We're not acting on newlib releases, OpenSSL alphas, Perl release candidates or non-stable Perl releases. There's no need to open issues about them. Co-authored-by: Matthias Aßhauer Signed-off-by: Johannes Schindelin commit a5cc82fd8332925e77b1acf9714959e47ff67932 Author: Neeraj K. Singh Date: Wed Oct 27 14:22:42 2021 -0700 mingw: do not call xutftowcs_path in mingw_mktemp The `xutftowcs_path` function canonicalizes absolute paths using GetFullPathNameW. This canonicalization may change the length of the string (e.g. getting rid of \.\), which breaks callers that pass the template string in a strbuf and expect the length of the string to remain the same. In my particular case, the tmp-objdir code is passing a strbuf to mkdtemp and is breaking since the strbuf.len is no longer synchronized with strlen(strbuf.buf). Signed-off-by: Neeraj K. Singh Signed-off-by: Johannes Schindelin commit 7e12ac9200f6795d579a77b01d866a9123a81b74 Author: Johannes Schindelin Date: Mon Apr 23 00:24:29 2018 +0200 mingw: really handle SIGINT Previously, we did not install any handler for Ctrl+C, but now we really want to because the MSYS2 runtime learned the trick to call the ConsoleCtrlHandler when Ctrl+C was pressed. With this, hitting Ctrl+C while `git log` is running will only terminate the Git process, but not the pager. This finally matches the behavior on Linux and on macOS. Signed-off-by: Johannes Schindelin commit 1c88e62100f1cdf513bb74e9bbe7bbf018f1d6c6 Author: Johannes Schindelin Date: Wed May 17 17:05:09 2017 +0200 mingw: kill child processes in a gentler way The TerminateProcess() function does not actually leave the child processes any chance to perform any cleanup operations. This is bad insofar as Git itself expects its signal handlers to run. A symptom is e.g. a left-behind .lock file that would not be left behind if the same operation was run, say, on Linux. To remedy this situation, we use an obscure trick: we inject a thread into the process that needs to be killed and to let that thread run the ExitProcess() function with the desired exit status. Thanks J Wyman for describing this trick. The advantage is that the ExitProcess() function lets the atexit handlers run. While this is still different from what Git expects (i.e. running a signal handler), in practice Git sets up signal handlers and atexit handlers that call the same code to clean up after itself. In case that the gentle method to terminate the process failed, we still fall back to calling TerminateProcess(), but in that case we now also make sure that processes spawned by the spawned process are terminated; TerminateProcess() does not give the spawned process a chance to do so itself. Please note that this change only affects how Git for Windows tries to terminate processes spawned by Git's own executables. Third-party software that *calls* Git and wants to terminate it *still* need to make sure to imitate this gentle method, otherwise this patch will not have any effect. Signed-off-by: Johannes Schindelin commit 0a57a784e6a1b684ed4f5f6bf2b4e96e62261e3f Author: xungeng li Date: Wed Jun 7 20:26:33 2023 +0800 mingw: optionally enable wsl compability file mode bits The Windows Subsystem for Linux (WSL) version 2 allows to use `chmod` on NTFS volumes provided that they are mounted with metadata enabled (see https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/ for details), for example: $ chmod 0755 /mnt/d/test/a.sh In order to facilitate better collaboration between the Windows version of Git and the WSL version of Git, we can make the Windows version of Git also support reading and writing NTFS file modes in a manner compatible with WSL. Since this slightly slows down operations where lots of files are created (such as an initial checkout), this feature is only enabled when `core.WSLCompat` is set to true. Note that you also have to set `core.fileMode=true` in repositories that have been initialized without enabling WSL compatibility. There are several ways to enable metadata loading for NTFS volumes in WSL, one of which is to modify `/etc/wsl.conf` by adding: ``` [automount] enabled = true options = "metadata,umask=027,fmask=117" ``` And reboot WSL. It can also be enabled temporarily by this incantation: $ sudo umount /mnt/c && sudo mount -t drvfs C: /mnt/c -o metadata,uid=1000,gid=1000,umask=22,fmask=111 It's important to note that this modification is compatible with, but does not depend on WSL. The helper functions in this commit can operate independently and functions normally on devices where WSL is not installed or properly configured. Signed-off-by: xungeng li Signed-off-by: Johannes Schindelin commit c576da7398572a272d490aea4d42f783130529dc Author: Johannes Schindelin Date: Thu Jul 20 00:23:26 2017 +0200 mingw: add a Makefile target to copy test artifacts The Makefile target `install-mingit-test-artifacts` simply copies stuff and things directly into a MinGit directory, including an init.bat script to set everything up so that the tests can be run in a cmd window. Sadly, Git's test suite still relies on a Perl interpreter even if compiled with NO_PERL=YesPlease. We punt for now, installing a small script into /usr/bin/perl that hands off to an existing Perl of a Git for Windows SDK. Signed-off-by: Johannes Schindelin commit 68a9e1049d8af655ffdc19513654f4cb79116825 Author: Johannes Schindelin Date: Fri Jul 7 10:15:36 2017 +0200 t9200: skip tests when $PWD contains a colon On Windows, the current working directory is pretty much guaranteed to contain a colon. If we feed that path to CVS, it mistakes it for a separator between host and port, though. This has not been a problem so far because Git for Windows uses MSYS2's Bash using a POSIX emulation layer that also pretends that the current directory is a Unix path (at least as long as we're in a shell script). However, that is rather limiting, as Git for Windows also explores other ports of other Unix shells. One of those is BusyBox-w32's ash, which is a native port (i.e. *not* using any POSIX emulation layer, and certainly not emulating Unix paths). So let's just detect if there is a colon in $PWD and punt in that case. Signed-off-by: Johannes Schindelin commit 895c312461bb353a31cc6b1bb8460cb65a75fe86 Author: Johannes Schindelin Date: Wed Jul 5 15:14:50 2017 +0200 t5813: allow for $PWD to be a Windows path Git for Windows uses MSYS2's Bash to run the test suite, which comes with benefits but also at a heavy price: on the plus side, MSYS2's POSIX emulation layer allows us to continue pretending that we are on a Unix system, e.g. use Unix paths instead of Windows ones, yet this is bought at a rather noticeable performance penalty. There *are* some more native ports of Unix shells out there, though, most notably BusyBox-w32's ash. These native ports do not use any POSIX emulation layer (or at most a *very* thin one, choosing to avoid features such as fork() that are expensive to emulate on Windows), and they use native Windows paths (usually with forward slashes instead of backslashes, which is perfectly legal in almost all use cases). And here comes the problem: with a $PWD looking like, say, C:/git-sdk-64/usr/src/git/t/trash directory.t5813-proto-disable-ssh Git's test scripts get quite a bit confused, as their assumptions have been shattered. Not only does this path contain a colon (oh no!), it also does not start with a slash. This is a problem e.g. when constructing a URL as t5813 does it: ssh://remote$PWD. Not only is it impossible to separate the "host" from the path with a $PWD as above, even prefixing $PWD by a slash won't work, as /C:/git-sdk-64/... is not a valid path. As a workaround, detect when $PWD does not start with a slash on Windows, and simply strip the drive prefix, using an obscure feature of Windows paths: if an absolute Windows path starts with a slash, it is implicitly prefixed by the drive prefix of the current directory. As we are talking about the current directory here, anyway, that strategy works. Signed-off-by: Johannes Schindelin commit 9cc63e93ae3d0bd959c09c2ef321777d71d60497 Author: Johannes Schindelin Date: Fri Jul 21 13:24:55 2017 +0200 t5605: special-case hardlink test for BusyBox-w32 When t5605 tries to verify that files are hardlinked (or that they are not), it uses the `-links` option of the `find` utility. BusyBox' implementation does not support that option, and BusyBox-w32's lstat() does not even report the number of hard links correctly (for performance reasons). So let's just switch to a different method that actually works on Windows. Signed-off-by: Johannes Schindelin commit 40c1fdd50bc71a91773792a86a16034e2f880bd2 Author: Johannes Schindelin Date: Fri Jul 21 12:48:33 2017 +0200 t5532: workaround for BusyBox on Windows While it may seem super convenient to some old Unix hands to simpy require Perl to be available when running the test suite, this is a major hassle on Windows, where we want to verify that Perl is not, actually, required in a NO_PERL build. As a super ugly workaround, we "install" a script into /usr/bin/perl reading like this: #!/bin/sh # We'd much rather avoid requiring Perl altogether when testing # an installed Git. Oh well, that's why we cannot have nice # things. exec c:/git-sdk-64/usr/bin/perl.exe "$@" The problem with that is that BusyBox assumes that the #! line in a script refers to an executable, not to a script. So when it encounters the line #!/usr/bin/perl in t5532's proxy-get-cmd, it barfs. Let's help this situation by simply executing the Perl script with the "interpreter" specified explicitly. Signed-off-by: Johannes Schindelin commit cc77bda048f544e7469f7b39ea584431faef7475 Author: Johannes Schindelin Date: Sat Aug 5 21:36:01 2017 +0200 t5003: use binary file from t/lib-diff/ At some stage, t5003-archive-zip wants to add a file that is not ASCII. To that end, it uses /bin/sh. But that file may actually not exist (it is too easy to forget that not all the world is Unix/Linux...)! Besides, we already have perfectly fine binary files intended for use solely by the tests. So let's use one of them instead. Signed-off-by: Johannes Schindelin commit e31674004b7bbc4205ab017bdfa0912d8500612b Author: Johannes Schindelin Date: Wed Jul 19 17:07:56 2017 +0200 test-lib: add BUSYBOX prerequisite When running with BusyBox, we will want to avoid calling executables on the PATH that are implemented in BusyBox itself. Signed-off-by: Johannes Schindelin commit ee93b74edbb7baafc3d937a29e70b69c6ea54056 Author: Johannes Schindelin Date: Fri Jun 30 22:32:33 2017 +0200 tests (mingw): remove Bash-specific pwd option The -W option is only understood by MSYS2 Bash's pwd command. We already make sure to override `pwd` by `builtin pwd -W` for MINGW, so let's not double the effort here. This will also help when switching the shell to another one (such as BusyBox' ash) whose pwd does *not* understand the -W option. Signed-off-by: Johannes Schindelin commit c242e5bae9f7ba61d0c5a40a94a85240f925be23 Author: Johannes Schindelin Date: Fri Jun 30 00:35:40 2017 +0200 mingw: only use Bash-ism `builtin pwd -W` when available Traditionally, Git for Windows' SDK uses Bash as its default shell. However, other Unix shells are available, too. Most notably, the Win32 port of BusyBox comes with `ash` whose `pwd` command already prints Windows paths as Git for Windows wants them, while there is not even a `builtin` command. Therefore, let's be careful not to override `pwd` unless we know that the `builtin` command is available. Signed-off-by: Johannes Schindelin commit 416f9924f5b2d9dc491b1e18d3df868136c6bafd Author: Johannes Schindelin Date: Mon Nov 19 20:34:13 2018 +0100 tests: use the correct path separator with BusyBox BusyBox-w32 is a true Win32 application, i.e. it does not come with a POSIX emulation layer. That also means that it does *not* use the Unix convention of separating the entries in the PATH variable using colons, but semicolons. However, there are also BusyBox ports to Windows which use a POSIX emulation layer such as Cygwin's or MSYS2's runtime, i.e. using colons as PATH separators. As a tell-tale, let's use the presence of semicolons in the PATH variable: on Unix, it is highly unlikely that it contains semicolons, and on Windows (without POSIX emulation), it is virtually guaranteed, as everybody should have both $SYSTEMROOT and $SYSTEMROOT/system32 in their PATH. Signed-off-by: Johannes Schindelin commit ece253d9dab1d398b9a897caf0324dcd64c0a662 Author: Johannes Schindelin Date: Tue Jul 18 01:15:40 2017 +0200 tests: only override sort & find if there are usable ones in /usr/bin/ The idea is to allow running the test suite on MinGit with BusyBox installed in /mingw64/bin/sh.exe. In that case, we will want to exclude sort & find (and other Unix utilities) from being bundled. Signed-off-by: Johannes Schindelin commit 87fcc0611ca22bbad574c939ab32f955fb0e6507 Author: Johannes Schindelin Date: Sat Aug 5 20:28:37 2017 +0200 tests: move test PNGs into t/lib-diff/ We already have a directory where we store files intended for use by multiple test scripts. The same directory is a better home for the test-binary-*.png files than t/. Signed-off-by: Johannes Schindelin commit 9679df2f5509f8b988b79812a1712ecd2c531916 Author: Johannes Schindelin Date: Thu Oct 11 23:55:44 2018 +0200 gitattributes: mark .png files as binary Signed-off-by: Johannes Schindelin commit 271dace1e6bc34defdda72628732733cbd82d888 Author: Johannes Schindelin Date: Thu Jul 20 22:25:21 2017 +0200 tests(mingw): if `iconv` is unavailable, use `test-helper --iconv` Signed-off-by: Johannes Schindelin commit 61271ae6f4a22d6b4bdee5f2cc5359c4dbc37e6e Author: Johannes Schindelin Date: Thu Jul 20 22:18:56 2017 +0200 test-tool: learn to act as a drop-in replacement for `iconv` It is convenient to assume that everybody who wants to build & test Git has access to a working `iconv` executable (after all, we already pretty much require libiconv). However, that limits esoteric test scenarios such as Git for Windows', where an end user installation has to ship with `iconv` for the sole purpose of being testable. That payload serves no other purpose. So let's just have a test helper (to be able to test Git, the test helpers have to be available, after all) to act as `iconv` replacement. Signed-off-by: Johannes Schindelin commit fb11e63d6d3430e62a8fc2372736034a8af929ed Author: Johannes Schindelin Date: Sat Aug 5 22:23:36 2017 +0200 test-lib: avoid unnecessary Perl invocation It is a bit strange, and even undesirable, to require Perl just to run the test suite even when NO_PERL was set. This patch does not fix this problem by any stretch of imagination. However, it fixes *the* Perl invocation that *every single* test script has to run. While at it, it makes the source code also more grep'able, as the code that unsets some, but not all, GIT_* environment variables just became a *lot* more explicit. And all that while still reducing the total number of lines. Signed-off-by: Johannes Schindelin commit 69165a9a4807e855b634f96e573ba8f120fc6ee4 Author: Johannes Schindelin Date: Thu Jul 20 20:41:29 2017 +0200 mingw: when path_lookup() failed, try BusyBox BusyBox comes with a ton of applets ("applet" being the identical concept to Git's "builtins"). And similar to Git's builtins, the applets can be called via `busybox `, or the BusyBox executable can be copied/hard-linked to the command name. The similarities do not end here. Just as with Git's builtins, it is problematic that BusyBox' hard-linked applets cannot easily be put into a .zip file: .zip archives have no concept of hard-links and therefore would store identical copies (and also extract identical copies, "inflating" the archive unnecessarily). To counteract that issue, MinGit already ships without hard-linked copies of the builtins, and the plan is to do the same with BusyBox' applets: simply ship busybox.exe as single executable, without hard-linked applets. To accommodate that, Git is being taught by this commit a very special trick, exploiting the fact that it is possible to call an executable with a command-line whose argv[0] is different from the executable's name: when `sh` is to be spawned, and no `sh` is found in the PATH, but busybox.exe is, use that executable (with unchanged argv). Likewise, if any executable to be spawned is not on the PATH, but busybox.exe is found, parse the output of `busybox.exe --help` to find out what applets are included, and if the command matches an included applet name, use busybox.exe to execute it. Signed-off-by: Johannes Schindelin commit 3feb8f7dfb06fb497fcc8472497b05a04c1fe209 Author: Bert Belder Date: Fri Oct 26 23:42:09 2018 +0200 Win32: symlink: add test for `symlink` attribute To verify that the symlink is resolved correctly, we use the fact that `git.exe` is a native Win32 program, and that `git.exe config -f ` therefore uses the native symlink resolution. Signed-off-by: Bert Belder Signed-off-by: Johannes Schindelin commit f93344df80b4572059b2a73de44512ee4bfcba84 Author: David Lomas Date: Fri Jul 28 15:20:43 2023 +0100 mingw: work around rename() failing on a read-only file At least on _some_ APFS network shares, Git fails to rename the object files because they are marked as read-only, because that has the effect of setting the uchg flag on APFS, which then means the file can't be renamed or deleted. To work around that, when a rename failed, and the read-only flag is set, try to turn it off and on again. This fixes https://github.com/git-for-windows/git/issues/4482 Signed-off-by: David Lomas Signed-off-by: Johannes Schindelin commit 426566b1e42954be10d43b8367342cffe03f718e Author: Johannes Schindelin Date: Mon Apr 23 23:20:00 2018 +0200 mingw: Windows Docker volumes are *not* symbolic links ... even if they may look like them. As looking up the target of the "symbolic link" (just to see whether it starts with `/ContainerMappedDirectories/`) is pretty expensive, we do it when we can be *really* sure that there is a possibility that this might be the case. Signed-off-by: Johannes Schindelin Signed-off-by: JiSeop Moon commit db57f75bc17c4fad6ae9e345e321ac8ffff6ef3e Author: Johannes Schindelin Date: Thu Jul 20 22:45:01 2017 +0200 mingw: explicitly specify with which cmd to prefix the cmdline The main idea of this patch is that even if we have to look up the absolute path of the script, if only the basename was specified as argv[0], then we should use that basename on the command line, too, not the absolute path. This patch will also help with the upcoming patch where we automatically substitute "sh ..." by "busybox sh ..." if "sh" is not in the PATH but "busybox" is: we will do that by substituting the actual executable, but still keep prepending "sh" to the command line. Signed-off-by: Johannes Schindelin commit 2251b4fce26326ae9c0f163a7f32b00d226b801e Author: Bert Belder Date: Fri Oct 26 11:51:51 2018 +0200 mingw: allow to specify the symlink type in .gitattributes On Windows, symbolic links have a type: a "file symlink" must point at a file, and a "directory symlink" must point at a directory. If the type of symlink does not match its target, it doesn't work. Git does not record the type of symlink in the index or in a tree. On checkout it'll guess the type, which only works if the target exists at the time the symlink is created. This may often not be the case, for example when the link points at a directory inside a submodule. By specifying `symlink=file` or `symlink=dir` the user can specify what type of symlink Git should create, so Git doesn't have to rely on unreliable heuristics. Signed-off-by: Bert Belder Signed-off-by: Johannes Schindelin commit 3dbd00ff85659be1f0cc300fda9b1caae4b39e1f Author: JiSeop Moon Date: Mon Apr 23 22:35:26 2018 +0200 mingw: move the file_attr_to_st_mode() function definition In preparation for making this function a bit more complicated (to allow for special-casing the `ContainerMappedDirectories` in Windows containers, which look like a symbolic link, but are not), let's move it out of the header. Signed-off-by: JiSeop Moon Signed-off-by: Johannes Schindelin commit 2a564d1c88b7bcf970b41afe94997a3ecb11b225 Author: Johannes Schindelin Date: Mon Feb 11 14:19:18 2019 +0100 Introduce helper to create symlinks that knows about index_state On Windows, symbolic links actually have a type depending on the target: it can be a file or a directory. In certain circumstances, this poses problems, e.g. when a symbolic link is supposed to point into a submodule that is not checked out, so there is no way for Git to auto-detect the type. To help with that, we will add support over the course of the next commits to specify that symlink type via the Git attributes. This requires an index_state, though, something that Git for Windows' `symlink()` replacement cannot know about because the function signature is defined by the POSIX standard and not ours to change. So let's introduce a helper function to create symbolic links that *does* know about the index_state. Signed-off-by: Johannes Schindelin commit 7c48a30c9c1a82f25137192c1a3ad23ce2df45f4 Author: JiSeop Moon Date: Mon Apr 23 22:31:42 2018 +0200 mingw: when running in a Windows container, try to rename() harder It is a known issue that a rename() can fail with an "Access denied" error at times, when copying followed by deleting the original file works. Let's just fall back to that behavior. Signed-off-by: JiSeop Moon Signed-off-by: Johannes Schindelin commit 954733566ebe8baf18f0f64fc56b27094e8d8273 Author: Bert Belder Date: Fri Oct 26 11:13:45 2018 +0200 Win32: symlink: move phantom symlink creation to a separate function Signed-off-by: Bert Belder commit bd2c03e21439ab4399fabd3dd0735bed03393e0a Author: Johannes Schindelin Date: Thu Jun 4 23:16:07 2020 +0200 mingw: special-case index entries for symlinks with buggy size In https://github.com/git-for-windows/git/pull/2637, we fixed a bug where symbolic links' target path sizes were recorded incorrectly in the index. The downside of this fix was that every user with tracked symbolic links in their checkouts would see them as modified in `git status`, but not in `git diff`, and only a `git add ` (or `git add -u`) would "fix" this. Let's do better than that: we can detect that situation and simply pretend that a symbolic link with a known bad size (or a size that just happens to be that bad size, a _very_ unlikely scenario because it would overflow our buffers due to the trailing NUL byte) means that it needs to be re-checked as if we had just checked it out. Signed-off-by: Johannes Schindelin commit 783d2ff71119dc92a4c3c079f3cbe8cca6eb16cf Author: JiSeop Moon Date: Mon Apr 23 22:30:18 2018 +0900 mingw: introduce code to detect whether we're inside a Windows container This will come in handy in the next commit. Signed-off-by: JiSeop Moon Signed-off-by: Johannes Schindelin commit 048c98b04bb5a1bb785d40ba702f9847fa9d0740 Author: Johannes Schindelin Date: Mon Mar 2 21:54:29 2020 +0100 mingw: emulate stat() a little more faithfully When creating directories via `safe_create_leading_directories()`, we might encounter an already-existing directory which is not readable by the current user. To handle that situation, Git's code calls `stat()` to determine whether we're looking at a directory. In such a case, `CreateFile()` will fail, though, no matter what, and consequently `mingw_stat()` will fail, too. But POSIX semantics seem to still allow `stat()` to go forward. So let's call `mingw_lstat()` for the rescue if we fail to get a file handle due to denied permission in `mingw_stat()`, and fill the stat info that way. We need to be careful to not allow this to go forward in case that we're looking at a symbolic link: to resolve the link, we would still have to create a file handle, and we just found out that we cannot. Therefore, `stat()` still needs to fail with `EACCES` in that case. This fixes https://github.com/git-for-windows/git/issues/2531. Signed-off-by: Johannes Schindelin commit cb8f3ed7022fa1ddf2097da3db9f69ddfa968540 Author: Johannes Schindelin Date: Tue May 30 21:50:57 2017 +0200 mingw: try to create symlinks without elevated permissions With Windows 10 Build 14972 in Developer Mode, a new flag is supported by CreateSymbolicLink() to create symbolic links even when running outside of an elevated session (which was previously required). This new flag is called SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE and has the numeric value 0x02. Previous Windows 10 versions will not understand that flag and return an ERROR_INVALID_PARAMETER, therefore we have to be careful to try passing that flag only when the build number indicates that it is supported. For more information about the new flag, see this blog post: https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/ This patch is loosely based on the patch submitted by Samuel D. Leslie as https://github.com/git-for-windows/git/pull/1184. Signed-off-by: Johannes Schindelin commit cdebdc1d15e108a7b2d2671e38e60499954ec3c3 Author: Karsten Blees Date: Sun May 24 01:48:35 2015 +0200 Win32: symlink: add support for symlinks to directories Symlinks on Windows have a flag that indicates whether the target is a file or a directory. Symlinks of wrong type simply don't work. This even affects core Win32 APIs (e.g. DeleteFile() refuses to delete directory symlinks). However, CreateFile() with FILE_FLAG_BACKUP_SEMANTICS doesn't seem to care. Check the target type by first creating a tentative file symlink, opening it, and checking the type of the resulting handle. If it is a directory, recreate the symlink with the directory flag set. It is possible to create symlinks before the target exists (or in case of symlinks to symlinks: before the target type is known). If this happens, create a tentative file symlink and postpone the directory decision: keep a list of phantom symlinks to be processed whenever a new directory is created in mingw_mkdir(). Limitations: This algorithm may fail if a link target changes from file to directory or vice versa, or if the target directory is created in another process. Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit 7eb9bc36a16cbbfb90cf07d8e274de36198ed42b Author: Karsten Blees Date: Sun May 24 01:32:03 2015 +0200 Win32: implement basic symlink() functionality (file symlinks only) Implement symlink() that always creates file symlinks. Fails with ENOSYS if symlinks are disabled or unsupported. Note: CreateSymbolicLinkW() was introduced with symlink support in Windows Vista. For compatibility with Windows XP, we need to load it dynamically and fail gracefully if it isnt's available. Signed-off-by: Karsten Blees commit d039b811d62069194c5625117311e37edf825b50 Author: Bill Zissimopoulos Date: Thu May 28 16:35:57 2020 -0700 mingw: lstat: compute correct size for symlinks This commit fixes mingw_lstat by computing the proper size for symlinks according to POSIX. POSIX specifies that upon successful return from lstat: "the value of the st_size member shall be set to the length of the pathname contained in the symbolic link not including any terminating null byte". Prior to this commit the mingw_lstat function returned a fixed size of 4096. This caused problems in git repositories that were accessed by git for Cygwin or git for WSL. For example, doing `git reset --hard` using git for Windows would update the size of symlinks in the index to be 4096; at a later time git for Cygwin or git for WSL would find that symlinks have changed size during `git status`. Vice versa doing `git reset --hard` in git for Cygwin or git for WSL would update the size of symlinks in the index with the correct value, only for git for Windows to find incorrectly at a later time that the size had changed. Signed-off-by: Bill Zissimopoulos Signed-off-by: Johannes Schindelin commit ff6e3753038838be224f2f0b38670ba9acb39df6 Author: Karsten Blees Date: Sun May 24 01:24:41 2015 +0200 Win32: implement readlink() Implement readlink() by reading NTFS reparse points. Works for symlinks and directory junctions. If symlinks are disabled, fail with ENOSYS. Signed-off-by: Karsten Blees commit 9b5c73a65f3670c13c1eabe119c0c89363ce6a0c Author: Karsten Blees Date: Sun May 24 01:17:31 2015 +0200 Win32: mingw_chdir: change to symlink-resolved directory If symlinks are enabled, resolve all symlinks when changing directories, as required by POSIX. Note: Git's real_path() function bases its link resolution algorithm on this property of chdir(). Unfortunately, the current directory on Windows is limited to only MAX_PATH (260) characters. Therefore using symlinks and long paths in combination may be problematic. Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit 1c3cf1465db872d62cb260ee9a099aa8222578b1 Author: Karsten Blees Date: Tue May 19 22:42:48 2015 +0200 Win32: mingw_rename: support renaming symlinks MSVCRT's _wrename() cannot rename symlinks over existing files: it returns success without doing anything. Newer MSVCR*.dll versions probably do not have this problem: according to CRT sources, they just call MoveFileEx() with the MOVEFILE_COPY_ALLOWED flag. Get rid of _wrename() and call MoveFileEx() with proper error handling. Signed-off-by: Karsten Blees commit d241df32732a3e4c21b3465ac7f8905c9de12830 Author: Karsten Blees Date: Sun May 24 01:06:10 2015 +0200 Win32: mingw_unlink: support symlinks to directories _wunlink() / DeleteFileW() refuses to delete symlinks to directories. If _wunlink() fails with ERROR_ACCESS_DENIED, try _wrmdir() as well. Signed-off-by: Karsten Blees commit a6f791abafd69161cbb065ce8deeae68a0728223 Author: Karsten Blees Date: Sat May 16 00:32:03 2015 +0200 Win32: add symlink-specific error codes Signed-off-by: Karsten Blees commit b9a96819930b28bbba13267887871cbb843d4025 Author: Johannes Schindelin Date: Fri Feb 23 02:50:03 2018 +0100 mingw (git_terminal_prompt): do fall back to CONIN$/CONOUT$ method To support Git Bash running in a MinTTY, we use a dirty trick to access the MSYS2 pseudo terminal: we execute a Bash snippet that accesses /dev/tty. The idea was to fall back to writing to/reading from CONOUT$/CONIN$ if that Bash call failed because Bash was not found. However, we should fall back even in other error conditions, because we have not successfully read the user input. Let's make it so. Signed-off-by: Johannes Schindelin commit d8e8d4805d6469ed3e3c9e7d4404cb42414f0caf Author: Karsten Blees Date: Sun May 24 01:55:05 2015 +0200 Win32: change default of 'core.symlinks' to false Symlinks on Windows don't work the same way as on Unix systems. E.g. there are different types of symlinks for directories and files, creating symlinks requires administrative privileges etc. By default, disable symlink support on Windows. I.e. users explicitly have to enable it with 'git config [--system|--global] core.symlinks true'. The test suite ignores system / global config files. Allow testing *with* symlink support by checking if native symlinks are enabled in MSys2 (via 'MSYS=winsymlinks:nativestrict'). Reminder: This would need to be changed if / when we find a way to run the test suite in a non-MSys-based shell (e.g. dash). Signed-off-by: Karsten Blees commit 88e607cab871f5740cee61723ceadcaeadf17c4f Author: Karsten Blees Date: Tue May 19 21:48:55 2015 +0200 Win32: factor out retry logic The retry pattern is duplicated in three places. It also seems to be too hard to use: mingw_unlink() and mingw_rmdir() duplicate the code to retry, and both of them do so incompletely. They also do not restore errno if the user answers 'no'. Introduce a retry_ask_yes_no() helper function that handles retry with small delay, asking the user, and restoring errno. mingw_unlink: include _wchmod in the retry loop (which may fail if the file is locked exclusively). mingw_rmdir: include special error handling in the retry loop. Signed-off-by: Karsten Blees commit 8c63e2136907ad6472b3cbd98456735f0973d217 Author: Karsten Blees Date: Sat May 16 01:11:37 2015 +0200 Win32: lstat(): return adequate stat.st_size for symlinks Git typically doesn't trust the stat.st_size member of symlinks (e.g. see strbuf_readlink()). However, some functions take shortcuts if st_size is 0 (e.g. diff_populate_filespec()). In mingw_lstat() and fscache_lstat(), make sure to return an adequate size. The extra overhead of opening and reading the reparse point to calculate the exact size is not necessary, as git doesn't rely on the value anyway. Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit fce1ceb95c9267bec3ec05eb376ba9e792d60202 Author: Karsten Blees Date: Tue Jan 10 23:21:56 2017 +0100 mingw: teach fscache and dirent about symlinks Move S_IFLNK detection to file_attr_to_st_mode() and reuse it in fscache. Implement DT_LNK detection in dirent.c and the fscache readdir version. Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit d90d4afa7220a16ef2628b1449f5d91d43c7329c Author: Karsten Blees Date: Sun May 24 00:17:56 2015 +0200 Win32: let mingw_lstat() error early upon problems with reparse points When obtaining lstat information for reparse points, we need to call FindFirstFile() in addition to GetFileInformationEx() to obtain the type of the reparse point (symlink, mount point etc.). However, currently there is no error handling whatsoever if FindFirstFile() fails. Call FindFirstFile() before modifying the stat *buf output parameter and error out if the call fails. Note: The FindFirstFile() return value includes all the data that we get from GetFileAttributesEx(), so we could replace GetFileAttributesEx() with FindFirstFile(). We don't do that because GetFileAttributesEx() is about twice as fast for single files. I.e. we only pay the extra cost of calling FindFirstFile() in the rare case that we encounter a reparse point. Note: The indentation of the remaining reparse point code will be fixed in the next patch. Signed-off-by: Karsten Blees commit 730279b83dea7c4fd91a6a905ca48966c5679bab Author: Karsten Blees Date: Tue May 12 00:58:39 2015 +0200 Win32: remove separate do_lstat() function With the new mingw_stat() implementation, do_lstat() is only called from mingw_lstat() (with follow == 0). Remove the extra function and the old mingw_stat()-specific (follow == 1) logic. Signed-off-by: Karsten Blees commit e404d24d78940c9682f0e66ef7d2e68a26c8b90f Author: Karsten Blees Date: Sat May 16 01:18:14 2015 +0200 Win32: implement stat() with symlink support With respect to symlinks, the current stat() implementation is almost the same as lstat(): except for the file type (st_mode & S_IFMT), it returns information about the link rather than the target. Implement stat by opening the file with as little permissions as possible and calling GetFileInformationByHandle on it. This way, all link resoltion is handled by the Windows file system layer. If symlinks are disabled, use lstat() as before, but fail with ELOOP if a symlink would have to be resolved. Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit bf011e9eec04507605c77f7a826164bfe1ec2d98 Author: Karsten Blees Date: Tue May 12 11:09:01 2015 +0200 Win32: don't call GetFileAttributes twice in mingw_lstat() GetFileAttributes cannot handle paths with trailing dir separator. The current [l]stat implementation calls GetFileAttributes twice if the path has trailing slashes (first with the original path passed to [l]stat, and and a second time with a path copy with trailing '/' removed). With Unicode conversion, we get the length of the path for free and also have a (wide char) buffer that can be modified. Remove trailing directory separators before calling the Win32 API. Signed-off-by: Karsten Blees commit 743fbb109b9f674eee17e4743b54ec068dff9bd9 Author: Karsten Blees Date: Mon May 11 19:58:14 2015 +0200 lockfile.c: use is_dir_sep() instead of hardcoded '/' checks Signed-off-by: Karsten Blees commit cae0ed2b69fc4cc92fbc0a209d69b76d593be9a9 Author: Karsten Blees Date: Mon May 11 22:15:40 2015 +0200 strbuf_readlink: support link targets that exceed PATH_MAX strbuf_readlink() refuses to read link targets that exceed PATH_MAX (even if a sufficient size was specified by the caller). As some platforms support longer paths, remove this restriction (similar to strbuf_getcwd()). Signed-off-by: Karsten Blees commit ab6cf4a85e9eb7feb01e9835b12a63ee88c7b502 Author: Karsten Blees Date: Mon May 11 19:54:23 2015 +0200 strbuf_readlink: don't call readlink twice if hint is the exact link size strbuf_readlink() calls readlink() twice if the hint argument specifies the exact size of the link target (e.g. by passing stat.st_size as returned by lstat()). This is necessary because 'readlink(..., hint) == hint' could mean that the buffer was too small. Use hint + 1 as buffer size to prevent this. Signed-off-by: Karsten Blees commit 0987f685d16c047ccb9e317857339812ab1255c7 Author: Ben Boeckel Date: Fri Apr 22 09:06:23 2022 -0400 clean: suggest using `core.longPaths` if paths are too long to remove On Windows, git repositories may have extra files which need cleaned (e.g., a build directory) that may be arbitrarily deep. Suggest using `core.longPaths` if such situations are encountered. Fixes: #2715 Signed-off-by: Ben Boeckel commit 29f7afc8f88a589928379298c56bf527ad10185a Author: Jeff Hostetler Date: Fri Mar 25 16:56:04 2022 -0400 compat/fsmonitor/fsm-*-win32: support long paths Update wchar_t buffers to use MAX_LONG_PATH instead of MAX_PATH and call xutftowcs_long_path() in the Win32 backend source files. Signed-off-by: Jeff Hostetler commit b8923c8fba0c566a57bd7192f41df8f6f666c1dc Author: Johannes Schindelin Date: Wed Sep 6 09:14:47 2023 +0200 win32(long path support): leave drive-less absolute paths intact When trying to ensure that long paths are handled correctly, we first normalize absolute paths as we encounter them. However, if the path is a so-called "drive-less" absolute path, i.e. if it is relative to the current drive but _does_ start with a directory separator, we would want the normalized path to be such a drive-less absolute path, too. Let's do that, being careful to still include the drive prefix when we need to go through the `\\?\` dance (because there, the drive prefix is absolutely required). This fixes https://github.com/git-for-windows/git/issues/4586. Signed-off-by: Johannes Schindelin commit 9d5705a01a5f4a243bc26f9361cefb3092cae153 Author: Karsten Blees Date: Sat May 9 02:11:48 2015 +0200 compat/terminal.c: only use the Windows console if bash 'read -r' fails Accessing the Windows console through the special CONIN$ / CONOUT$ devices doesn't work properly for non-ASCII usernames an passwords. It also doesn't work for terminal emulators that hide the native console window (such as mintty), and 'TERM=xterm*' is not necessarily a reliable indicator for such terminals. The new shell_prompt() function, on the other hand, works fine for both MSys1 and MSys2, in native console windows as well as mintty, and properly supports Unicode. It just needs bash on the path (for 'read -s', which is bash-specific). On Windows, try to use the shell to read from the terminal. If that fails with ENOENT (i.e. bash was not found), use CONIN/OUT as fallback. Note: To test this, create a UTF-8 credential file with non-ASCII chars, e.g. in git-bash: 'echo url=http://täst.com > cred.txt'. Then in git-cmd, 'git credential fill commit 2efa530e357932080f2843537e62eb98c43cdbc7 Author: Karsten Blees Date: Sat Jul 5 00:00:36 2014 +0200 Win32: fix 'lstat("dir/")' with long paths Use a suffciently large buffer to strip the trailing slash. Signed-off-by: Karsten Blees commit 770c278c62db56b217fadeddcd2bd0d9eeef7f4b Author: Karsten Blees Date: Thu Mar 19 16:33:44 2015 +0100 mingw: Support `git_terminal_prompt` with more terminals The `git_terminal_prompt()` function expects the terminal window to be attached to a Win32 Console. However, this is not the case with terminal windows other than `cmd.exe`'s, e.g. with MSys2's own `mintty`. Non-cmd terminals such as `mintty` still have to have a Win32 Console to be proper console programs, but have to hide the Win32 Console to be able to provide more flexibility (such as being resizeable not only vertically but also horizontally). By writing to that Win32 Console, `git_terminal_prompt()` manages only to send the prompt to nowhere and to wait for input from a Console to which the user has no access. This commit introduces a function specifically to support `mintty` -- or other terminals that are compatible with MSys2's `/dev/tty` emulation. We use the `TERM` environment variable as an indicator for that: if the value starts with "xterm" (such as `mintty`'s "xterm_256color"), we prefer to let `xterm_prompt()` handle the user interaction. The most prominent user of `git_terminal_prompt()` is certainly `git-remote-https.exe`. It is an interesting use case because both `stdin` and `stdout` are redirected when Git calls said executable, yet it still wants to access the terminal. When running inside a `mintty`, the terminal is not accessible to the `git-remote-https.exe` program, though, because it is a MinGW program and the `mintty` terminal is not backed by a Win32 console. To solve that problem, we simply call out to the shell -- which is an *MSys2* program and can therefore access `/dev/tty`. Helped-by: nalla Signed-off-by: Karsten Blees Signed-off-by: Johannes Schindelin commit a260ff7b26a47034ce53d9f4076acddcb1bd4717 Author: Karsten Blees Date: Tue Jul 28 21:07:41 2015 +0200 mingw: support long paths Windows paths are typically limited to MAX_PATH = 260 characters, even though the underlying NTFS file system supports paths up to 32,767 chars. This limitation is also evident in Windows Explorer, cmd.exe and many other applications (including IDEs). Particularly annoying is that most Windows APIs return bogus error codes if a relative path only barely exceeds MAX_PATH in conjunction with the current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG. Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path. Notable exceptions include functions dealing with executables and the current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...). Introduce a handle_long_path function to check the length of a specified path properly (and fail with ENAMETOOLONG), and to optionally expand long paths using the '\\?\' file namespace prefix. Short paths will not be modified, so we don't need to worry about device names (NUL, CON, AUX). Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH (at least not on Win7), so we can use it to do the heavy lifting of the conversion (translate '/' to '\', eliminate '.' and '..', and make an absolute path). Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH limit. Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs that support long paths. While improved error checking is always active, long paths support must be explicitly enabled via 'core.longpaths' option. This is to prevent end users to shoot themselves in the foot by checking out files that Windows Explorer, cmd/bash or their favorite IDE cannot handle. Test suite: Test the case is when the full pathname length of a dir is close to 260 (MAX_PATH). Bug report and an original reproducer by Andrey Rogozhnikov: https://github.com/msysgit/git/pull/122#issuecomment-43604199 [jes: adjusted test number to avoid conflicts, added support for chdir(), etc] Thanks-to: Martin W. Kirst Thanks-to: Doug Kelly Original-test-by: Andrey Rogozhnikov Signed-off-by: Karsten Blees Signed-off-by: Stepan Kasal Signed-off-by: Johannes Schindelin commit 7ec14970001b5794b8bba4f78a5f9d845c626d1e Author: Doug Kelly Date: Wed Jan 8 20:28:15 2014 -0600 pack-objects (mingw): demonstrate a segmentation fault with large deltas There is a problem in the way 9ac3f0e5b3e4 (pack-objects: fix performance issues on packing large deltas, 2018-07-22) initializes that mutex in the `packing_data` struct. The problem manifests in a segmentation fault on Windows, when a mutex (AKA critical section) is accessed without being initialized. (With pthreads, you apparently do not really have to initialize them?) This was reported in https://github.com/git-for-windows/git/issues/1839. Signed-off-by: Doug Kelly Signed-off-by: Johannes Schindelin commit b885fdedffd72bffe11f54db2fb76118690bca05 Merge: 9e4350829d 6a9aca82e0 Author: Johannes Schindelin Date: Tue Jul 23 21:57:12 2019 +0200 Merge branch 'dont-clean-junctions-fscache' We already avoid traversing NTFS junction points in `git clean -dfx`. With this topic branch, we do that when the FSCache is enabled, too. Signed-off-by: Johannes Schindelin commit 9e4350829d0290ac5670abd6823c431a7d35d454 Merge: 52a0b9a28d e1ea5c4fc0 Author: Derrick Stolee <570044+derrickstolee@users.noreply.github.com> Date: Tue Jun 18 10:05:16 2019 -0400 Merge branch 'fscache-and-sparse-checkout' When updating the skip-worktree bits in the index to align with new values in a sparse-checkout file, Git scans the entire working directory with lstat() calls. In a sparse-checkout, many of these lstat() calls are for paths that do not exist. Enable the fscache feature during this scan. In a local test of a repo with ~2.2 million paths, updating the index with `git read-tree -m -u HEAD` with a sparse-checkout file containing only `/.gitattributes` improved from 2-3 minutes to 15-20 seconds. More work could be done to stop running lstat() calls when recursing into directories that are known to not exist. commit 52a0b9a28d0ec159f78f4fe27fa8468d77ec83a6 Merge: 95f9ec98d6 93605fd193 Author: Johannes Schindelin Date: Tue Nov 27 13:38:41 2018 +0100 Merge pull request #1937 from benpeart/fscache-NtQueryDirectoryFile-gfw fscache: teach fscache to use NtQueryDirectoryFile commit 95f9ec98d6069b4934e53b9f08305728bd3917ef Merge: 468ff65681 c21d0b4c45 Author: Johannes Schindelin Date: Mon Nov 26 16:57:06 2018 +0100 Merge pull request #1934 from benpeart/fscache-thread-safe-enable-gfw fscache: make fscache_enable() thread safe commit 468ff65681e7a806ea9e2eba46d718fa5e931db4 Merge: 7e1ff23689 963cab91b5 Author: Johannes Schindelin Date: Fri Nov 16 14:38:19 2018 +0100 Merge remote-tracking branch 'benpeart/fscache-per-thread-gfw' This brings substantial wins in performance because the FSCache is now per-thread, being merged to the primary thread only at the end, so we do not have to lock (except while merging). Signed-off-by: Johannes Schindelin commit 7e1ff23689fc3eda70692f46c4b3324b3c7f80e1 Merge: a188255022 ed68c4aa42 Author: Johannes Schindelin Date: Fri Nov 16 12:07:19 2018 +0100 Merge pull request #1910 from benpeart/fscache_statistics-gfw fscache: add fscache hit statistics commit a188255022581ad0d0924031c4f3355412c8d383 Merge: d71f705e35 61f05d9d51 Author: Johannes Schindelin Date: Mon Nov 5 14:24:12 2018 +0100 Merge pull request #1911 from benpeart/git_test_fscache-gfw fscache: add GIT_TEST_FSCACHE support commit d71f705e3568536ee12b0c36997115fb257cf8ad Merge: 020e958fb4 212eb82f18 Author: Johannes Schindelin Date: Mon Nov 5 14:01:30 2018 +0100 Merge pull request #1909 from benpeart/free-fscache-after-status-gfw status: disable and free fscache at the end of the status command commit 020e958fb4eb193219dee1e17fdfc84034927268 Merge: 3cbfc60afa 8db9e8dc6c Author: Johannes Schindelin Date: Mon Nov 5 13:53:07 2018 +0100 Merge pull request #1908 from benpeart/FindFirstFileEx-gfw fscache: use FindFirstFileExW to avoid retrieving the short name commit 3cbfc60afa93dbe520f32cc09d3860dada11b329 Merge: f7dac75e75 06abbd1d65 Author: Johannes Schindelin Date: Fri Sep 28 00:39:59 2018 +0200 Merge pull request #1827 from benpeart/fscache_refresh_index Enable the filesystem cache (fscache) in refresh_index(). commit f7dac75e7570e9a81e172dc6a6bbfa1d239da7a1 Merge: 7a43738087 04b33e8963 Author: Johannes Schindelin Date: Fri Oct 12 23:03:48 2018 +0200 Merge pull request #1468 from atetubou/fscache_checkout_flush checkout.c: enable fscache for checkout again Signed-off-by: Johannes Schindelin commit 7a43738087bf4514fd988942e987f36df2f1f461 Merge: 6502981d31 5c2f8e6390 Author: Johannes Schindelin Date: Fri Jun 8 18:38:55 2018 +0200 Merge pull request #1426 from atetubou/fetch_pack fetch-pack.c: enable fscache for stats under .git/objects commit 6502981d31f08ddd442e506d4f9b2a832991b356 Merge: eabe28d3d7 3912afc7cc Author: Johannes Schindelin Date: Fri Jun 8 18:37:18 2018 +0200 Merge pull request #1344 from jeffhostetler/perf_add_excludes_with_fscache dir.c: make add_excludes aware of fscache during status commit eabe28d3d7c2c3d3bd4c4fc5fbef1571693ced1a Merge: 5399d41fee 9623611551 Author: Johannes Schindelin Date: Fri Jun 8 18:34:57 2018 +0200 Merge pull request #971 from jeffhostetler/jeffhostetler/add_preload_fscache add: use preload-index and fscache for performance commit 5399d41fee0e341166a5e77ae3734a4383f1eb95 Merge: ac6d450932 524662c9b7 Author: Johannes Schindelin Date: Fri Jun 8 18:35:19 2018 +0200 Merge pull request #994 from jeffhostetler/jeffhostetler/fscache_nfd fscache: add not-found directory cache to fscache commit f210ba75b14a02592a673777748e5f8746eee648 Merge: 3f3e9c79f0 d4406ebbd0 Author: Johannes Schindelin Date: Thu Jun 7 08:45:09 2018 +0200 Merge 'gitk' into HEAD commit ac6d450932180eb0794be6e94b396db8fc9ccca9 Merge: b270a47d0a 227e32ecde Author: Johannes Schindelin Date: Thu Nov 15 12:23:43 2018 +0100 Merge branch 'fscache' commit 3f3e9c79f0828dfcb1499bcec4c170dd974c55d2 Merge: 1a8f24b0c7 d63ca8c1eb Author: Johannes Schindelin Date: Fri Jun 8 18:37:01 2018 +0200 Merge branch 'git-gui-askyesno' These changes are necessary to support better Git for Windows' new auto-update feature. Signed-off-by: Johannes Schindelin commit d4406ebbd0a4a187851c3172d2a1d420038f1972 Author: James J. Raden Date: Thu Jan 21 12:07:47 2016 -0500 gitk: make the "list references" default window width wider When using remotes (with git-flow especially), the remote reference names are almost always wordwrapped in the "list references" window because it's somewhat narrow by default. It's possible to resize it with a mouse, but it's annoying to have to do this every time, especially on Windows 10, where the window border seems to be only one (1) pixel wide, thus making the grabbing of the window border tricky. Signed-off-by: James J. Raden commit 1a8f24b0c79265136d094cd61f2099f24dab5e17 Merge: b270a47d0a 90d03a0d49 Author: Johannes Schindelin Date: Thu Jun 7 08:45:04 2018 +0200 Merge 'git-gui' into HEAD commit d63ca8c1ebfcd66433538238e1a695269a3fe220 Author: Johannes Schindelin Date: Wed Sep 20 21:55:45 2017 +0200 git-gui--askyesno (mingw): use Git for Windows' icon, if available For additional GUI goodness. Signed-off-by: Johannes Schindelin commit 56f1ea8635f04fe91deb422611a6eabe258a3bfe Author: Johannes Schindelin Date: Tue Feb 16 16:42:06 2016 +0100 gitk: fix arrow keys in input fields with Tcl/Tk >= 8.6 Tcl/Tk 8.6 introduced new events for the cursor left/right keys and apparently changed the behavior of the previous event. Let's work around that by using the new events when we are running with Tcl/Tk 8.6 or later. This fixes https://github.com/git-for-windows/git/issues/495 Signed-off-by: Johannes Schindelin commit 672f0bd58f305f5341a9a6226c714fe8934a5426 Author: Johannes Schindelin Date: Wed Sep 20 21:53:45 2017 +0200 git-gui--askyesno: allow overriding the window title "Question?" is maybe not the most informative thing to ask. In the absence of better information, it is the best we can do, of course. However, Git for Windows' auto updater just learned the trick to use git-gui--askyesno to ask the user whether to update now or not. And in this scripted scenario, we can easily pass a command-line option to change the window title. So let's support that with the new `--title ` option. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d73ab2ece6983905769de6db58a62f2634bd8d71 Author: Sebastian Schuberth <sschuberth@gmail.com> Date: Sun Jul 22 23:19:24 2012 +0200 gitk: Use an external icon file on Windows Git for Windows now ships with the new Git icon from git-scm.com. Use that icon file if it exists instead of the old procedurally drawn one. This patch was sent upstream but so far no decision on its inclusion was made, so commit it to our fork. Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com> commit 134569b9b0ee5e423a591b846049280a374edc2c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Sep 20 21:52:28 2017 +0200 git-gui--askyesno: fix funny text wrapping The text wrapping seems to be aligned to the right side of the Yes button, leaving an awful lot of empty space. Let's try to counter this by using pixel units. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 32dc7abe0797cbe80b0f634d261a5594b8e33e95 Author: Karsten Blees <blees@dcon.de> Date: Sat Feb 4 21:54:36 2012 +0100 gitk: Unicode file name support Assumes file names in git tree objects are UTF-8 encoded. On most unix systems, the system encoding (and thus the TCL system encoding) will be UTF-8, so file names will be displayed correctly. On Windows, it is impossible to set the system encoding to UTF-8. Changing the TCL system encoding (via 'encoding system ...', e.g. in the startup code) is explicitly discouraged by the TCL docs. Change gitk functions dealing with file names to always convert from and to UTF-8. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 6a9aca82e0a8ea7e2602cdc7d9ca2b211589aed0 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Dec 11 12:17:49 2018 +0100 clean: make use of FSCache The `git clean` command needs to enumerate plenty of files and directories, and can therefore benefit from the FSCache. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit e1ea5c4fc064da6ee0e45137d51fbe940a2c1ecb Author: Derrick Stolee <dstolee@microsoft.com> Date: Wed Jun 12 00:58:49 2019 +0000 unpack-trees: enable fscache for sparse-checkout When updating the skip-worktree bits in the index to align with new values in a sparse-checkout file, Git scans the entire working directory with lstat() calls. In a sparse-checkout, many of these lstat() calls are for paths that do not exist. Enable the fscache feature during this scan. Since enable_fscache() calls nest, the disable_fscache() method decrements a counter and would only clear the cache if that counter reaches zero. In a local test of a repo with ~2.2 million paths, updating the index with git read-tree -m -u HEAD with a sparse-checkout file containing only /.gitattributes improved from 2-3 minutes to ~6 seconds. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 93605fd19376e9f0ee12e89e623ed514a488d83b Author: Ben Peart <benpeart@microsoft.com> Date: Thu Nov 15 14:15:40 2018 -0500 fscache: teach fscache to use NtQueryDirectoryFile Using FindFirstFileExW() requires the OS to allocate a 64K buffer for each directory and then free it when we call FindClose(). Update fscache to call the underlying kernel API NtQueryDirectoryFile so that we can do the buffer management ourselves. That allows us to allocate a single buffer for the lifetime of the cache and reuse it for each directory. This change improves performance of 'git status' by 18% in a repo with ~200K files and 30k folders. Documentation for NtQueryDirectoryFile can be found at: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntquerydirectoryfile https://docs.microsoft.com/en-us/windows/desktop/FileIO/file-attribute-constants https://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-point-tags To determine if the specified directory is a symbolic link, inspect the FileAttributes member to see if the FILE_ATTRIBUTE_REPARSE_POINT flag is set. If so, EaSize will contain the reparse tag (this is a so far undocumented feature, but confirmed by the NTFS developers). To determine if the reparse point is a symbolic link (and not some other form of reparse point), test whether the tag value equals the value IO_REPARSE_TAG_SYMLINK. The NtQueryDirectoryFile() call works best (and on Windows 8.1 and earlier, it works *only*) with buffer sizes up to 64kB. Which is 32k wide characters, so let's use that as our buffer size. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 90d03a0d497ffe3b504fa3c083531e8bb2015af9 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 23 18:06:05 2010 +0200 git gui: set GIT_ASKPASS=git-gui--askpass if not set yet Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 4a565299838651801955749134872a3c2f630a80 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Dec 11 12:17:49 2018 +0100 fscache: implement an FSCache-aware is_mount_point() When FSCache is active, we can cache the reparse tag and use it directly to determine whether a path refers to an NTFS junction, without any additional, costly I/O. Note: this change only makes a difference with the next commit, which will make use of the FSCache in `git clean` (contingent on `core.fscache` set, of course). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d3ccdc54b59e965b15be902a97b195b2cddb4be0 Author: Heiko Voigt <hvoigt@hvoigt.net> Date: Sun Feb 21 21:05:04 2010 +0100 git-gui: provide question helper for retry fallback on Windows Make use of the new environment variable GIT_ASK_YESNO to support the recently implemented fallback in case unlink, rename or rmdir fail for files in use on Windows. The added dialog will present a yes/no question to the the user which will currently be used by the windows compat layer to let the user retry a failed file operation. Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net> commit d7f0539704b37d5d9deabcc653211a2fb2a9d62f Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Dec 11 12:59:29 2018 +0100 fscache: remember the reparse tag for each entry We will use this in the next commit to implement an FSCache-aware version of is_mount_point(). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c21d0b4c458e6e2f3cf1ff2096c28044cb712f34 Author: Ben Peart <benpeart@microsoft.com> Date: Fri Nov 16 10:59:18 2018 -0500 fscache: make fscache_enable() thread safe The recent change to make fscache thread specific relied on fscache_enable() being called first from the primary thread before being called in parallel from worker threads. Make that more robust and protect it with a critical section to avoid any issues. Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 963cab91b5410ddf26c2b3f6a7f9e33d58be9641 Author: Ben Peart <benpeart@microsoft.com> Date: Fri Nov 2 11:19:10 2018 -0400 fscache: teach fscache to use mempool Now that the fscache is single threaded, take advantage of the mem_pool as the allocator to significantly reduce the cost of allocations and frees. With the reduced cost of free, in future patches, we can start freeing the fscache at the end of commands instead of just leaking it. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ed68c4aa422f7e711b4a63ee84a72ba1b1952586 Author: Ben Peart <benpeart@microsoft.com> Date: Tue Sep 25 16:28:16 2018 -0400 fscache: add fscache hit statistics Track fscache hits and misses for lstat and opendir requests. Reporting of statistics is done when the cache is disabled for the last time and freed and is only reported if GIT_TRACE_FSCACHE is set. Sample output is: 11:33:11.836428 compat/win32/fscache.c:433 fscache: lstat 3775, opendir 263, total requests/misses 4052/269 Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 61f05d9d510184511107742c9d90f6fafa27392e Author: Ben Peart <benpeart@microsoft.com> Date: Thu Oct 4 18:10:21 2018 -0400 fscache: add GIT_TEST_FSCACHE support Add support to fscache to enable running the entire test suite with the fscache enabled. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 212eb82f186075493c2db46a84eff2e5a792c9c0 Author: Ben Peart <benpeart@microsoft.com> Date: Thu Nov 1 11:40:51 2018 -0400 status: disable and free fscache at the end of the status command At the end of the status command, disable and free the fscache so that we don't leak the memory and so that we can dump the fscache statistics. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit b9c94442db84bbe4655056bce53b51e6a3f892b0 Author: Ben Peart <benpeart@microsoft.com> Date: Thu Oct 4 15:38:08 2018 -0400 fscache: update fscache to be thread specific instead of global The threading model for fscache has been to have a single, global cache. This puts requirements on it to be thread safe so that callers like preload-index can call it from multiple threads. This was implemented with a single mutex and completion events which introduces contention between the calling threads. Simplify the threading model by making fscache thread specific. This allows us to remove the global mutex and synchronization events entirely and instead associate a fscache with every thread that requests one. This works well with the current multi-threading which divides the cache entries into blocks with a separate thread processing each block. At the end of each worker thread, if there is a fscache on the primary thread, merge the cached results from the worker into the primary thread cache. This enables us to reuse the cache later especially when scanning for untracked files. In testing, this reduced the time spent in preload_index() by about 25% and also reduced the CPU utilization significantly. On a repo with ~200K files, it reduced overall status times by ~12%. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 88edfe6229c498460d1e8087ddcb8da6515566fd Author: Ben Peart <benpeart@microsoft.com> Date: Fri Nov 2 11:19:10 2018 -0400 fscache: fscache takes an initial size Update enable_fscache() to take an optional initial size parameter which is used to initialize the hashmap so that it can avoid having to rehash as additional entries are added. Add a separate disable_fscache() macro to make the code clearer and easier to read. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 84a48ba955fdab013d8ae7597f711785ba20f056 Author: Ben Peart <benpeart@microsoft.com> Date: Thu Oct 4 18:10:21 2018 -0400 mem_pool: add GIT_TRACE_MEMPOOL support Add tracing around initializing and discarding mempools. In discard report on the amount of memory unused in the current block to help tune setting the initial_size. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 8db9e8dc6c0d51ecab3757727bfec48ccbf54894 Author: Ben Peart <benpeart@microsoft.com> Date: Tue Oct 23 11:42:06 2018 -0400 fscache: use FindFirstFileExW to avoid retrieving the short name Use FindFirstFileExW with FindExInfoBasic to avoid forcing NTFS to look up the short name. Also switch to a larger (64K vs 4K) buffer using FIND_FIRST_EX_LARGE_FETCH to minimize round trips to the kernel. In a repo with ~200K files, this drops warm cache status times from 3.19 seconds to 2.67 seconds for a 16% savings. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 06abbd1d6589819efb4b0b6a8b67fe7a3af7a491 Author: Ben Peart <benpeart@microsoft.com> Date: Fri Sep 7 11:39:57 2018 -0400 Enable the filesystem cache (fscache) in refresh_index(). On file systems that support it, this can dramatically speed up operations like add, commit, describe, rebase, reset, rm that would otherwise have to lstat() every file to "re-match" the stat information in the index to that of the file system. On a synthetic repo with 1M files, "git reset" dropped from 52.02 seconds to 14.42 seconds for a savings of 72%. Signed-off-by: Ben Peart <benpeart@microsoft.com> commit 04b33e89633c9a345d4127c619069b55e0a3ea66 Author: Takuto Ikuta <tikuta@chromium.org> Date: Tue Jan 30 22:42:58 2018 +0900 checkout.c: enable fscache for checkout again This is retry of #1419. I added flush_fscache macro to flush cached stats after disk writing with tests for regression reported in #1438 and #1442. git checkout checks each file path in sorted order, so cache flushing does not make performance worse unless we have large number of modified files in a directory containing many files. Using chromium repository, I tested `git checkout .` performance when I delete 10 files in different directories. With this patch: TotalSeconds: 4.307272 TotalSeconds: 4.4863595 TotalSeconds: 4.2975562 Avg: 4.36372923333333 Without this patch: TotalSeconds: 20.9705431 TotalSeconds: 22.4867685 TotalSeconds: 18.8968292 Avg: 20.7847136 I confirmed this patch passed all tests in t/ with core_fscache=1. Signed-off-by: Takuto Ikuta <tikuta@chromium.org> commit 5c2f8e6390c0f489c20511f182fdb80518842f69 Author: Takuto Ikuta <tikuta@chromium.org> Date: Wed Nov 22 20:39:38 2017 +0900 fetch-pack.c: enable fscache for stats under .git/objects When I do git fetch, git call file stats under .git/objects for each refs. This takes time when there are many refs. By enabling fscache, git takes file stats by directory traversing and that improved the speed of fetch-pack for repository having large number of refs. In my windows workstation, this improves the time of `git fetch` for chromium repository like below. I took stats 3 times. * With this patch TotalSeconds: 9.9825165 TotalSeconds: 9.1862075 TotalSeconds: 10.1956256 Avg: 9.78811653333333 * Without this patch TotalSeconds: 15.8406702 TotalSeconds: 15.6248053 TotalSeconds: 15.2085938 Avg: 15.5580231 Signed-off-by: Takuto Ikuta <tikuta@chromium.org> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 3912afc7ccb727938b8f8c9b19bcc1311ad16d77 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Wed Dec 20 11:19:27 2017 -0500 dir.c: regression fix for add_excludes with fscache Fix regression described in: https://github.com/git-for-windows/git/issues/1392 which was introduced in: https://github.com/git-for-windows/git/commit/b2353379bba414e6c00dde913497cc9c827366f2 Problem Symptoms ================ When the user has a .gitignore file that is a symlink, the fscache optimization introduced above caused the stat-data from the symlink, rather that of the target file, to be returned. Later when the ignore file was read, the buffer length did not match the stat.st_size field and we called die("cannot use <path> as an exclude file") Optimization Rationale ====================== The above optimization calls lstat() before open() primarily to ask fscache if the file exists. It gets the current stat-data as a side effect essentially for free (since we already have it in memory). If the file does not exist, it does not need to call open(). And since very few directories have .gitignore files, we can greatly reduce time spent in the filesystem. Discussion of Fix ================= The above optimization calls lstat() rather than stat() because the fscache only intercepts lstat() calls. Calls to stat() stay directed to the mingw_stat() completly bypassing fscache. Furthermore, calls to mingw_stat() always call {open, fstat, close} so that symlinks are properly dereferenced, which adds *additional* open/close calls on top of what the original code in dir.c is doing. Since the problem only manifests for symlinks, we add code to overwrite the stat-data when the path is a symlink. This preserves the effect of the performance gains provided by the fscache in the normal case. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 962361155199a52a015dcd0deaf943f6e547b261 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Tue Nov 22 11:26:38 2016 -0500 add: use preload-index and fscache for performance Teach "add" to use preload-index and fscache features to improve performance on very large repositories. During an "add", a call is made to run_diff_files() which calls check_remove() for each index-entry. This calls lstat(). On Windows, the fscache code intercepts the lstat() calls and builds a private cache using the FindFirst/FindNext routines, which are much faster. Somewhat independent of this, is the preload-index code which distributes some of the start-up costs across multiple threads. We need to keep the call to read_cache() before parsing the pathspecs (and hence cannot use the pathspecs to limit any preload) because parse_pathspec() is using the index to determine whether a pathspec is, in fact, in a submodule. If we would not read the index first, parse_pathspec() would not error out on a path that is inside a submodule, and t7400-submodule-basic.sh would fail with not ok 47 - do not add files from a submodule We still want the nice preload performance boost, though, so we simply call read_cache_preload(&pathspecs) after parsing the pathspecs. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 2e427b9e5d14c451ff1b85f79eb23d38948152fa Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Wed Dec 20 10:43:41 2017 -0500 fscache: make fscache_enabled() public Make fscache_enabled() function public rather than static. Remove unneeded fscache_is_enabled() function. Change is_fscache_enabled() macro to call fscache_enabled(). is_fscache_enabled() now takes a pathname so that the answer is more precise and mean "is fscache enabled for this pathname", since fscache only stores repo-relative paths and not absolute paths, we can avoid attempting lookups for absolute paths. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 1d08e72ae1bbd8ba24c22b8636163e0b4306bb16 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Wed Nov 1 15:05:44 2017 -0400 dir.c: make add_excludes aware of fscache during status Teach read_directory_recursive() and add_excludes() to be aware of optional fscache and avoid trying to open() and fstat() non-existant ".gitignore" files in every directory in the worktree. The current code in add_excludes() calls open() and then fstat() for a ".gitignore" file in each directory present in the worktree. Change that when fscache is enabled to call lstat() first and if present, call open(). This seems backwards because both lstat needs to do more work than fstat. But when fscache is enabled, fscache will already know if the .gitignore file exists and can completely avoid the IO calls. This works because of the lstat diversion to mingw_lstat when fscache is enabled. This reduced status times on a 350K file enlistment of the Windows repo on a NVMe SSD by 0.25 seconds. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 524662c9b7d507bf512ec279bd8bdab15e0c502b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Jan 25 18:39:16 2017 +0100 fscache: add a test for the dir-not-found optimization Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 227e32ecde23f282b84f579837987edc7079b8a5 Author: Karsten Blees <blees@dcon.de> Date: Tue Jun 24 13:22:35 2014 +0200 fscache: load directories only once If multiple threads access a directory that is not yet in the cache, the directory will be loaded by each thread. Only one of the results is added to the cache, all others are leaked. This wastes performance and memory. On cache miss, add a future object to the cache to indicate that the directory is currently being loaded. Subsequent threads register themselves with the future object and wait. When the first thread has loaded the directory, it replaces the future object with the result and notifies waiting threads. Signed-off-by: Karsten Blees <blees@dcon.de> commit b2fd5549835af6e7ae098a39259effd4c357e678 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Tue Dec 13 14:05:32 2016 -0500 fscache: remember not-found directories Teach FSCACHE to remember "not found" directories. This is a performance optimization. FSCACHE is a performance optimization available for Windows. It intercepts Posix-style lstat() calls into an in-memory directory using FindFirst/FindNext. It improves performance on Windows by catching the first lstat() call in a directory, using FindFirst/ FindNext to read the list of files (and attribute data) for the entire directory into the cache, and short-cut subsequent lstat() calls in the same directory. This gives a major performance boost on Windows. However, it does not remember "not found" directories. When STATUS runs and there are missing directories, the lstat() interception fails to find the parent directory and simply return ENOENT for the file -- it does not remember that the FindFirst on the directory failed. Thus subsequent lstat() calls in the same directory, each re-attempt the FindFirst. This completely defeats any performance gains. This can be seen by doing a sparse-checkout on a large repo and then doing a read-tree to reset the skip-worktree bits and then running status. This change reduced status times for my very large repo by 60%. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 80584b9684f4ba99c7617a71aba65bed81224c53 Author: Karsten Blees <blees@dcon.de> Date: Tue Oct 1 12:51:54 2013 +0200 mingw: add a cache below mingw's lstat and dirent implementations Checking the work tree status is quite slow on Windows, due to slow `lstat()` emulation (git calls `lstat()` once for each file in the index). Windows operating system APIs seem to be much better at scanning the status of entire directories than checking single files. Add an `lstat()` implementation that uses a cache for lstat data. Cache misses read the entire parent directory and add it to the cache. Subsequent `lstat()` calls for the same directory are served directly from the cache. Also implement `opendir()`/`readdir()`/`closedir()` so that they create and use directory listings in the cache. The cache doesn't track file system changes and doesn't plug into any modifying file APIs, so it has to be explicitly enabled for git functions that don't modify the working copy. Note: in an earlier version of this patch, the cache was always active and tracked file system changes via ReadDirectoryChangesW. However, this was much more complex and had negative impact on the performance of modifying git commands such as 'git checkout'. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 09d5fa02f532082c1fba328935ca4ea2e6c24d36 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Tue Jan 24 15:12:13 2017 -0500 fscache: add key for GIT_TRACE_FSCACHE Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 16c1baeea925a0c7081d1293d01c301a0c489a88 Author: Karsten Blees <blees@dcon.de> Date: Sun Sep 8 14:23:27 2013 +0200 mingw: add infrastructure for read-only file system level caches Add a macro to mark code sections that only read from the file system, along with a config option and documentation. This facilitates implementation of relatively simple file system level caches without the need to synchronize with the file system. Enable read-only sections for 'git status' and preload_index. Signed-off-by: Karsten Blees <blees@dcon.de> commit 6f73b6e35aaa9220633b102efbb9cc9c744a4463 Author: Karsten Blees <blees@dcon.de> Date: Sun Sep 8 14:21:30 2013 +0200 Win32: make the lstat implementation pluggable Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite slow. Windows operating system APIs seem to be much better at scanning the status of entire directories than checking single files. A caching implementation may improve performance by bulk-reading entire directories or reusing data obtained via opendir / readdir. Make the lstat implementation pluggable so that it can be switched at runtime, e.g. based on a config option. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 6a73c24734db3b277e9e06fab83102e5d396a8cc Author: Karsten Blees <blees@dcon.de> Date: Sun Sep 8 14:18:40 2013 +0200 mingw: make the dirent implementation pluggable Emulating the POSIX `dirent` API on Windows via `FindFirstFile()`/`FindNextFile()` is pretty staightforward, however, most of the information provided in the `WIN32_FIND_DATA` structure is thrown away in the process. A more sophisticated implementation may cache this data, e.g. for later reuse in calls to `lstat()`. Make the `dirent` implementation pluggable so that it can be switched at runtime, e.g. based on a config option. Define a base DIR structure with pointers to `readdir()`/`closedir()` that match the `opendir()` implementation (similar to vtable pointers in Object-Oriented Programming). Define `readdir()`/`closedir()` so that they call the function pointers in the `DIR` structure. This allows to choose the `opendir()` implementation on a call-by-call basis. Make the fixed-size `dirent.d_name` buffer a flex array, as `d_name` may be implementation specific (e.g. a caching implementation may allocate a `struct dirent` with _just_ the size needed to hold the `d_name` in question). Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit eb39de7d7121e2a26df410ef2bde03f5e08613b8 Author: Karsten Blees <blees@dcon.de> Date: Sun Sep 8 14:17:31 2013 +0200 Win32: dirent.c: Move opendir down Move opendir down in preparation for the next patch. Signed-off-by: Karsten Blees <blees@dcon.de> commit 32ec5e93931195c23055e3d4c41a0e8074978145 Author: Karsten Blees <blees@dcon.de> Date: Sat Jul 6 02:09:35 2013 +0200 Win32: make FILETIME conversion functions public We will use them in the upcoming "FSCache" patches (to accelerate sequential lstat() calls). Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b270a47d0aaca792cbd329ecc8842a3e65410562 Merge: ef4c884834 1f56fc9566 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Oct 11 13:38:58 2018 +0200 Merge branch 'ready-for-upstream' This is the branch thicket of patches in Git for Windows that are considered ready for upstream. To keep them in a ready-to-submit shape, they are kept as close to the beginning of the branch thicket as possible. commit 1f56fc9566b279f4f56ce5eff7d925c86a3e521a Merge: aa90e8b90b b4de03a3b4 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Oct 7 18:42:24 2024 +0200 Fix `git log --graph -u` hangs (#5193) This fixes https://github.com/git-for-windows/git/issues/5185 by backporting https://github.com/gitgitgadget/git/pull/1806 (which, sadly, seems not to have made it into Git v2.47.0). commit aa90e8b90b1ec25dcdfa0bf4376aadcb041cf951 Merge: 80566f93e4 cba1e83da6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Sep 26 20:32:11 2024 +0200 Add experimental 'git survey' builtin (#5174) This introduces `git survey` to Git for Windows ahead of upstream for the express purpose of getting the path-based analysis in the hands of more folks. The inspiration of this builtin is [`git-sizer`](https://github.com/github/git-sizer), but since that command relies on `git cat-file --batch` to get the contents of objects, it has limits to how much information it can provide. This is mostly a rewrite of the `git survey` builtin that was introduced into the `microsoft/git` fork in microsoft/git#667. That version had a lot more bells and whistles, including an analysis much closer to what `git-sizer` provides. The biggest difference in this version is that this one is focused on using the path-walk API in order to visit batches of objects based on a common path. This allows identifying, for instance, the path that is contributing the most to the on-disk size across all versions at that path. For example, here are the top ten paths contributing to my local Git repository (which includes `microsoft/git` and `gitster/git`): ``` TOP FILES BY DISK SIZE ============================================================================ Path | Count | Disk Size | Inflated Size -----------------------------------------+-------+-----------+-------------- whats-cooking.txt | 1373 | 11637459 | 37226854 t/helper/test-gvfs-protocol | 2 | 6847105 | 17233072 git-rebase--helper | 1 | 6027849 | 15269664 compat/mingw.c | 6111 | 5194453 | 463466970 t/helper/test-parse-options | 1 | 3420385 | 8807968 t/helper/test-pkt-line | 1 | 3408661 | 8778960 t/helper/test-dump-untracked-cache | 1 | 3408645 | 8780816 t/helper/test-dump-fsmonitor | 1 | 3406639 | 8776656 po/vi.po | 104 | 1376337 | 51441603 po/de.po | 210 | 1360112 | 71198603 ``` This kind of analysis has been helpful in identifying the reasons for growth in a few internal monorepos. Those findings motivated the changes in #5157 and #5171. With this early version in Git for Windows, we can expand the reach of the experimental tool in advance of it being contributed to the upstream project. Unfortunately, this will mean that in the next `microsoft/git` rebase, @jeffhostetler's version will need to be pulled out since there are enough conflicts. These conflicts include how tables are stored and generated, as the version in this PR is slightly more general to allow for different kinds of data. commit 80566f93e4133d63b039ddc1d9200d3ec0205fc3 Merge: 94fa321507 8a5625a8a1 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Sep 26 13:10:40 2024 +0200 Introduce 'git backfill' to get missing blobs in a partial clone (#5172) This change introduces the `git backfill` command which uses the path walk API to download missing blobs in a blobless partial clone. By downloading blobs that correspond to the same file path at the same time, we hope to maximize the potential benefits of delta compression against multiple versions. These downloads occur in a configurable batch size, presenting a mechanism to perform "resumable" clones: `git clone --filter=blob:none` gets the commits and trees, then `git backfill` will download all missing blobs. If `git backfill` is interrupted partway through, it can be restarted and will redownload only the missing objects. When combining blobless partial clones with sparse-checkout, `git backfill` will assume its `--sparse` option and download only the blobs within the sparse-checkout. Users may want to do this as the repo size will still be smaller than the full repo size, but commands like `git blame` or `git log -L` will not suffer from many one-by-one blob downloads. Future directions should consider adding a pathspec or file prefix to further focus which paths are being downloaded in a batch. commit 94fa32150794e2041b33127b43eada55134ce8c9 Merge: a674ac910e 6b6b2d86e0 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Oct 8 08:15:03 2024 +0200 path-walk API: avoid adding a root tree more than once (#5195) When adding tree objects, we are very careful to avoid adding the same tree object more than once. There was one small gap in that logic, though: when adding a root tree object. Two refs can easily share the same root tree object, and we should still not add it more than once. commit a674ac910e16f6ba8514be175bf5018c81be1c36 Merge: 965f3d41cc a529a93360 Author: Derrick Stolee <derrickstolee@github.com> Date: Wed Sep 25 16:48:41 2024 -0400 Add path walk API and its use in 'git pack-objects' (#5171) This is a follow up to #5157 as well as motivated by the RFC in gitgitgadget/git#1786. We have ways of walking all objects, but it is focused on visiting a single commit and then expanding the new trees and blobs reachable from that commit that have not been visited yet. This means that objects arrive without any locality based on their path. Add a new "path walk API" that focuses on walking objects in batches according to their type and path. This will walk all annotated tags, all commits, all root trees, and then start a depth-first search among all paths in the repo to collect trees and blobs in batches. The most important application for this is being fast-tracked to Git for Windows: `git pack-objects --path-walk`. This application of the path walk API discovers the objects to pack via this batched walk, and automatically groups objects that appear at a common path so they can be checked for delta comparisons. This use completely avoids any name-hash collisions (even the collisions that sometimes occur with the new `--full-name-hash` option) and can be much faster to compute since the first pass of delta calculations does not waste time on objects that are unlikely to be diffable. Some statistics are available in the commit messages. commit 965f3d41ccb8d3c53042f6eec4a4a190ec7e0247 Merge: 392754f7b9 13ec385303 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Sep 24 13:26:06 2024 +0200 pack-objects: create new name-hash algorithm (#5157) This is an updated version of gitgitgadget/git#1785, intended for early consumption into Git for Windows. The idea here is to add a new `--full-name-hash` option to `git pack-objects` and `git repack`. This adjusts the name-hash value used for finding delta bases in such a way that uses the full path name with a lower likelihood of collisions than the default name-hash algorithm. In many repositories with name-hash collisions and many versions of those paths, this can significantly reduce the size of a full repack. It can also help in certain cases of `git push`, but only if the pack is already artificially inflated by name-hash collisions; cases that find "sibling" deltas as better choices become worse with `--full-name-hash`. Thus, this option is currently recommended for full repacks of large repos, and on client machines without reachability bitmaps. Some care is taken to ignore this option when using bitmaps, either writing bitmaps or using a bitmap walk during reads. The bitmap file format contains name-hash values, but no way to indicate which function is used, so compatibility is a concern for bitmaps. Future work could explore this idea. After this PR is merged, then the more-involved `--path-walk` option may be considered. commit 392754f7b9222d6565be54c95bdb7f9aec6d8d70 Merge: ec653cfa7e 32b471e7d7 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 12 22:23:15 2024 +0200 Merge branch 'run-command-be-helpful-when-Git-LFS-fails-on-Windows-7' Since Git LFS v3.5.x implicitly dropped Windows 7 support, we now want users to be advised _what_ is going wrong on that Windows version. This topic branch goes out of its way to provide users with such guidance. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ec653cfa7eadd3a0588879e6e2b05dcc09073b4a Merge: f0a07ea57e 6ac9202ed3 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 12 22:23:15 2024 +0200 Merge branch 'Fallback-to-AppData-if-XDG-CONFIG-HOME-is-unset' This topic branch adds support for a more Windows-native user-wide config file than `XDG_CONFIG_HOME` (or `~/.config/`) will ever be. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit f0a07ea57e358fdb5daab1de08accfeeb3cec07f Merge: 7af4c153e2 c6bada9d70 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 12 22:23:15 2024 +0200 Merge branch 'Fix-i686-build-with-GCC-v14' This fixes a long-time compile warning turned error by GCC v14. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 7af4c153e266214b9157b32848ad5a5f12947b48 Merge: d3d236b92c 14717ba164 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 12 22:23:15 2024 +0200 Merge branch 'run-t5601-and-t7406-with-symlinks-on-windows-10' This topic branch contains a patch that made it into Git for Windows v2.45.1 but not into Git v2.45.1 (because the latter does not come with symlink support on Windows). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d3d236b92c8bedfa3c8afaba06d990cc2cfc68a2 Merge: 75fffd96ee 43a6bf7c0c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Apr 12 14:22:30 2024 +0200 common-main.c: fflush stdout buffer when exit (#4901) commit 75fffd96ee0f3faa61120b4501cc997921d6a3a0 Merge: c1f56a5f65 3a78230e81 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Dec 7 09:00:15 2023 +0100 ARM64: Embed manifest properly (#4718) Teach our ARM64 based builds to embed the manifest file correctly. This fixes #4707 commit c1f56a5f65f88700f123223dd2b9ef32684e4424 Merge: db90385eb0 26db42c646 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Dec 2 22:06:18 2023 +0100 win32: use native ANSI sequence processing, if possible (#4700) Windows 10 version 1511 (also known as Anniversary Update), according to https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences introduced native support for ANSI sequence processing. This allows using colors from the entire 24-bit color range. All we need to do is test whether the console's "virtual processing support" can be enabled. If it can, we do not even need to start the `console_thread` to handle ANSI sequences. Incidentally, this addresses https://github.com/git-for-windows/git/issues/3712. commit db90385eb0ad397a517d781bc46c0bdca1e374e5 Merge: 45c5dabd55 ba9010395e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Aug 7 21:07:40 2023 +0200 Additional error checks for issuing the windows.appendAtomically warning (#4528) Another (hopefully clean) PR for showing the error warning about atomic append on windows after failure on APFS, which returns EBADF not EINVAL. Signed-off-by: David Lomas <dl3@pale-eds.co.uk> Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> commit 45c5dabd55aaa06ad60f6b7756c876b1989cbf3f Merge: 74515a17eb 9b0e692c56 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Aug 7 22:21:04 2023 +0200 Merge branch 'nano-server' This patch adds a GitHub workflow (to be triggered manually) to allow for conveniently verifying that Git and Scalar still work as intended in Windows Nano Server (a relatively small container base image that is frequently used where a "small Windows" is needed, e.g. in automation ;-)) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 74515a17eb86f4f6091dd708dfe4e8121b027a83 Merge: a337bb3e3d dea9b1c136 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon May 15 12:50:37 2023 +0200 Lazy load libcurl, allowing for an SSL/TLS backend-specific libcurl (#4410) As per https://github.com/git-for-windows/git/issues/4350#issuecomment-1485041503, the major block for upgrading Git for Windows' OpenSSL from v1.1 to v3 is the tricky part where such an upgrade would break `git fetch`/`git clone` and `git push` because the libcurl depends on the OpenSSL DLL, and the major version bump will _change_ the file name of said DLL. To overcome that, the plan is to build libcurl flavors for each supported SSL/TLS backend, aligning with the way MSYS2 builds libcurl, then switch Git for Windows' SDK to the Secure Channel-flavored libcurl, and teach Git to look for the specific flavor of libcurl corresponding to the `http.sslBackend` setting (if that was configured). Here is the PR to teach Git that trick. commit a337bb3e3d7543cdd7ba4feddadb78590d0362a6 Merge: 72dcbf3980 e7a3b2dd48 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Mar 21 22:37:30 2023 +0100 Git GUI: fix Repository>Explore Working Copy (#4357) This is a companion PR of https://github.com/prati0100/git-gui/pull/95 Since Git v2.39.1, we are a bit more stringent in searching the PATH. In particular, we specifically require the `.exe` suffix. However, the `Repository>Explore Working Copy` command asks for `explorer.exe` to be found on the `PATH`, which _already_ has that suffix. Let's unstartle the PATH-finding logic about this scenario. This fixes https://github.com/git-for-windows/git/issues/4356 commit 72dcbf39806523b9f275a8418e29ed397d1ba787 Merge: bc3a48528a 7ba0501fdf Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jan 31 17:10:28 2023 +0000 Skip linking the "dashed" `git-<command>`s for built-ins (#4252) It is merely a historical wart that, say, `git-commit` exists in the `libexec/git-core/` directory, a tribute to the original idea to let Git be essentially a bunch of Unix shell scripts revolving around very few "plumbing" (AKA low-level) commands. Git has evolved a lot from there. These days, most of Git's functionality is contained within the `git` executable, in the form of "built-in" commands. To accommodate for scripts that use the "dashed" form of Git commands, even today, Git provides hard-links that make the `git` executable available as, say, `git-commit`, just in case that an old script has not been updated to invoke `git commit`. Those hard-links do not come cheap: they take about half a minute for every build of Git on Windows, they are mistaken for taking up huge amounts of space by some Windows Explorer versions that do not understand hard-links, and therefore many a "bug" report had to be addressed. The "dashed form" has been officially deprecated in Git version 1.5.4, which was released on February 2nd, 2008, i.e. a very long time ago. This deprecation was never finalized by skipping these hard-links, but we can start the process now, in Git for Windows. This addresses the concern raised in https://github.com/git-for-windows/git/pull/4185#discussion_r1051661894 commit bc3a48528aba3f62506d8724c7736f263635f5a6 Merge: 679959bfd1 01c83bdaa0 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Dec 20 15:13:55 2022 +0100 Fix global repository field not being cleared (#4083) It is checked for w.r.t. global repository struct down in the callstack in compatibility layer for MinGW before being assigned in the function that `free()`'d it. commit 679959bfd11b082f2f9e502523ba5e68260cd76b Merge: a937f4b267 f1a6eef669 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Oct 28 17:48:47 2022 +0200 Add support for CLANGARM64 target (#3916) **This requires an ARM64-machine with Windows 11 installed (which supports x64 emulation for MSYS2)** ### The main idea - Use the main MSYS2/git-sdk-64 setup, which works on Windows 11 on ARM thanks to x64-emulation - Configure the official `clangarm64` MSYS2 repo - Install `mingw-w64-clang-aarch64-toolchain` which contains the ARM64-native Clang compiler commit a937f4b267c1c6184acb49f685f210ac7164ecc8 Merge: 0fbe943941 a25a8315d5 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Nov 23 10:58:47 2022 +0100 Merge branch 'builtin-swap-functions' Do prefer GCC's built-in bit-swap functions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0fbe9439415a157e1bff24f1f872582be5d15544 Merge: 490b8d7a27 b9d207ad82 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Nov 3 13:48:51 2022 +0100 Fix Windows version resources (#4092) Add `FileVersion`, which is a required string ([Microsoft documentation](https://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resource)) in the `StringFileInfo` block. As not all required strings were present in the block, none were being included. Fixes #4090 After including the `FileVersion` string, all other defined strings are now being included on executables. File version information for `git.exe` has changed from: ``` PS C:\Program Files\Git\bin> [System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Data\git-sdk-64\usr\src\git\git.exe") | Select-Object * FileVersionRaw : 2.38.1.1 ProductVersionRaw : 2.38.1.1 Comments : CompanyName : FileBuildPart : 1 FileDescription : FileMajorPart : 2 FileMinorPart : 38 FileName : C:\Data\git-sdk-64\usr\src\git\git.exe FilePrivatePart : 1 FileVersion : InternalName : IsDebug : False IsPatched : False IsPrivateBuild : False IsPreRelease : False IsSpecialBuild : False Language : English (United States) LegalCopyright : LegalTrademarks : OriginalFilename : PrivateBuild : ProductBuildPart : 1 ProductMajorPart : 2 ProductMinorPart : 38 ProductName : ProductPrivatePart : 1 ProductVersion : SpecialBuild : ``` To the following: ``` PS C:\Program Files\Git\bin> [System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Data\git-sdk-64\usr\src\git\git.exe") | Select-Object * FileVersionRaw : 2.38.1.1 ProductVersionRaw : 2.38.1.1 Comments : CompanyName : The Git Development Community FileBuildPart : 1 FileDescription : Git for Windows FileMajorPart : 2 FileMinorPart : 38 FileName : C:\Data\git-sdk-64\usr\src\git\git.exe FilePrivatePart : 1 FileVersion : 2.38.1.windows.1.10.g6ed65a6fab InternalName : git IsDebug : False IsPatched : False IsPrivateBuild : False IsPreRelease : False IsSpecialBuild : False Language : English (United States) LegalCopyright : LegalTrademarks : OriginalFilename : git.exe PrivateBuild : ProductBuildPart : 1 ProductMajorPart : 2 ProductMinorPart : 38 ProductName : Git ProductPrivatePart : 1 ProductVersion : 2.38.1.windows.1.10.g6ed65a6fab SpecialBuild : ``` I wasn't really expecting `GIT_VERSION` to contain the Git commit, I was hoping for just `2.38.1` or `2.38.1.1`, at least for the `FileVersion` string. Anybody know if it's possible to concatenate the `MAJOR`, `MINOR`, `MICRO`, and `PATCHLEVEL` fields with dots, or if there's another variable that can be used (with or without `PATCHLEVEL`)? Alternatively, use the complete `GIT_VERSION` for both `FileVersion` and `ProductVersion`. commit 490b8d7a27f86542319fbd80a80d8f85393b0611 Merge: a411e16161 f106061c25 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Jul 27 18:18:58 2022 +0200 Merge pull request #3942 from rimrul/mingw-tsaware MinGW: link as terminal server aware commit a411e1616137154758ed5463f4d7e8440e1743b2 Merge: 4ab55c239e 5228689643 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Feb 11 15:24:30 2021 +0100 Merge branch 'ci-fixes' Backport a couple fixes to make the CI build run again (so much for reproducible builds...). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 4ab55c239efd2e0b6624b16cad53ed1460194cc2 Merge: 926db16df1 98fc4e8579 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Oct 12 00:26:27 2018 +0200 Merge branch 'fsync-object-files-always' Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 926db16df133102d188561697bc7e8bb92b88995 Merge: 5d81184ea0 98cd92e96f Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jun 23 13:04:03 2022 +0200 Merge branch 'optionally-dont-append-atomically-on-windows' Fix append failure issue under remote directories #2753 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 5d81184ea05f15d85f2bec0f2537d30849e3fd70 Merge: 97952334ed d3aa359168 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon May 30 20:18:05 2022 +0200 Merge pull request #3875 from 1480c1/wine/detect_msys_tty winansi: check result before using Name for pty commit 97952334edeb29088ac435735f1a52c6aa450be1 Merge: 0f1c55a6ab 7312915dd2 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Apr 21 09:31:41 2022 +0200 Merge pull request #3751 from rkitover/native-term mingw: set $env:TERM=xterm-256color for newer OSes commit 0f1c55a6ab75ae3a01abda785a38dcb8f9dded8a Merge: 897632fcdf 763dbdd60e Author: Derrick Stolee <derrickstolee@github.com> Date: Wed Apr 13 17:41:49 2022 -0400 Merge pull request #3791: Various fixes around `safe.directory` The first three commits are rebased versions of those in gitgitgadget/git#1215. These allow the following: 1. Fix `git config --global foo.bar <path>` from allowing the `<path>`. As a bonus, users with a config value starting with `/` will not get a warning about "old-style" paths needing a "`%(prefix)/`". 2. When in WSL, the path starts with `/` so it needs to be interpolated properly. Update the warning to include `%(prefix)/` to get the right value for WSL users. (This is specifically for using Git for Windows from Git Bash, but in a WSL directory.) 3. When using WSL, the ownership check fails and reports an error message. This is noisy, and happens even if the user has marked the path with `safe.directory`. Remove that error message. commit 897632fcdfb0716b5e1e78fce5794d78bf8707c7 Merge: e005f84aa6 26a985de91 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Dec 15 13:50:02 2021 +0100 Merge pull request #3533 from PhilipOakley/hashliteral_t Begin `unsigned long`->`size_t` conversion to support large files on Windows commit e005f84aa68603b303fadbce4d4f3992c99a4d2f Merge: 589f86b3d0 046d09e116 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Wed Sep 29 17:33:49 2021 -0400 Merge branch 'mark-v4-fsmonitor-experimental' into try-v4-fsmonitor commit 589f86b3d0a95de1a3bf67ce51076db4af634411 Merge: 8866b0b640 24ccbac24d Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Oct 13 16:47:05 2021 +0200 Merge pull request #3417 from dscho/initialize-core.symlinks-earlier init: respect core.symlinks before copying the templates commit 8866b0b640ac76ea8b693d5c777d5140579a9d25 Merge: fdd0065a01 21d32cf033 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Oct 12 12:06:10 2021 +0200 Merge pull request #3306 from PhilipOakley/vs-sln Make Git for Windows start builds in modern Visual Studio commit fdd0065a013c716a916e906684b0f4d9fbbc4782 Merge: 601cb175e3 7547439b8a Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Aug 10 23:40:22 2021 +0200 Merge pull request #3349 from vdye/feature/ci-subtree-tests Add `contrib/subtree` test execution to CI builds commit 601cb175e3305425d7bd2dfb3111c6b33386b9df Merge: b2274e560d 7697e7a6e4 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jun 24 20:32:51 2021 +0200 Merge pull request #3293 from pascalmuller/http-support-automatically-sending-client-certificate http: Add support for enabling automatic sending of SSL client certificate commit b2274e560ddc78add26d5dec0033b25177177289 Merge: 7bd681faaf 66cd21d536 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon May 17 18:07:10 2021 +0200 Merge pull request #3220 from dscho/there-is-no-vs/master-anymore Let the documentation reflect that there is no vs/master anymore commit 7bd681faafaf3abb75d88bb839fadcaa4349050d Merge: 6f2e235b25 da60c899b1 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Apr 3 07:58:12 2021 +0200 Merge pull request #3165 from dscho/increase-allowed-length-of-interpreter-path mingw: allow for longer paths in `parse_interpreter()` commit 6f2e235b25098482fa34ca5f498985c7cfe98191 Merge: 5757a2ce99 b2fbec6b91 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jul 19 22:09:55 2021 +0200 Merge pull request #3327 from dennisameling/fix-host-cpu cmake(): allow setting HOST_CPU for cross-compilation commit 5757a2ce99d60150517f9cd9c305fdc555d1fd95 Merge: 7847ec17a2 dff6c6ba8b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Dec 9 14:34:45 2020 +0100 Merge pull request #2915 from dennisameling/windows-arm64-support Windows arm64 support commit 7847ec17a27d91df9625731d911ce9dfdfa319c6 Merge: 71d4134dd8 51dd430371 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Oct 7 12:06:14 2019 +0200 Merge pull request #2351 from PhilipOakley/vcpkg-tip Vcpkg Install: detect lack of working Git, and note possible vcpkg time outs commit 71d4134dd86dfbc89ca6118237c1f878479047f8 Merge: 4b52d4d312 d7f33388c2 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Mar 13 14:01:59 2023 +0100 cmake: install headless-git. (#4338) Even if CMake is not the canonical way to build Git for Windows, but CMake support merely exists in Git to support building Git for Windows using Visual Studio, we should include `headless-git` in such a scenario when installing the binaries to a given location. commit 4b52d4d3121b7ce4e2d022049933b0f78cd2403a Merge: 304782a0ea 28fdf667fa Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jan 11 14:10:26 2021 +0100 Merge pull request #2974 from derrickstolee/maintenance-and-headless Include Windows-specific maintenance and headless-git commit 304782a0ea45d16e8b81121c9b5bb61565433576 Merge: 6c62818b3a 58c4696d18 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Oct 18 23:44:55 2020 +0200 Merge 'git-gui/js/intent-to-add' This merges the current version of the patch that tries to address Git GUI's problems with intent-to-add files. This patch will likely be improved substantially before it is merged into Git GUI's main branch, but we want to have _something_ resembling a fix already in Git for Windows v2.29.0. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 6c62818b3a05ddfcec1bad0c617ccc95bc88ba7d Merge: 9dc12c9a11 1296250960 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Oct 9 08:04:59 2020 +0200 Merge pull request #2655 from jglathe/jg/t0014_trace_extra_info t/t0014: fix: eliminate additional lines from trace commit 9dc12c9a114c4002ed69cc9ca5b568bb1453aad7 Merge: 3390f16b9c 2c422a0c5e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jul 3 10:33:24 2020 +0200 Merge pull request #2714 from lbonanomi/crlf-scissors Rationalize line endings for scissors-cleanup commit 3390f16b9cad4a4577a96f397c8d014424f0cb88 Merge: 31ec91bfad c1c698ad2e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jul 2 22:46:27 2020 +0200 Merge pull request #2730 from dscho/crlf-aware-git-add-i git add -i: handle CR/LF line endings in the interactive input commit 31ec91bfad573c7e0389050d0403598e1b1a376b Merge: d0472b9ced cf11bd7f59 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jun 8 13:42:36 2018 +0200 Merge 'add-p-many-files' This topic branch allows `add -p` and `add -i` with a large number of files. It is kind of a hack that was never really meant to be upstreamed. Let's see if we can do better in the built-in `add -p`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d0472b9ced2776dc67ccffd9485787fe19c9520a Merge: e7be2a923f c7562c85f6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat May 9 21:24:56 2020 +0200 Merge pull request #2618 from dscho/avoid-d/f-conflict-in-vs/master ci: avoid d/f conflict in vs/master commit e7be2a923fab78cac48aa4211712a68b63d51474 Merge: 199de55771 fd1da52208 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Mar 5 00:37:36 2020 +0100 Merge pull request #2535 from dscho/schannel-revoke-best-effort Introduce and use the new "best effort" strategy for Secure Channel revoke checking commit 199de55771e72a830b55b18415e86de33c6c332f Merge: 33fc4864c3 cbee48b6a8 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Feb 11 19:57:50 2020 +0100 Merge pull request #2506 from dscho/issue-2283 Allow running Git directly from `C:\Program Files\Git\mingw64\bin\git.exe` commit 33fc4864c31e66cf7854209ed7c2f4f01438da1b Merge: 2889f9d318 0c87ece87c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Feb 1 12:13:45 2020 +0100 Merge pull request #2504 from dscho/access-repo-via-junction Handle `git add <file>` where <file> traverses an NTFS junction commit 2889f9d318ed4a7cfafe6aaec7e85f8e62b26b6c Merge: b227650917 ddf04b993d Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jan 30 23:18:53 2020 +0100 Merge pull request #2501 from jeffhostetler/clink-debug-curl clink.pl: fix MSVC compile script to handle libcurl-d.lib commit b2276509171d8f94a8862fdafd47a99e099de274 Merge: df116b026b 57d0bfaeac Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Jan 22 19:42:25 2020 +0100 Merge pull request #2488 from bmueller84/master mingw: fix fatal error working on mapped network drives on Windows commit df116b026b9ad5b85199f4b5b51c35593e5de039 Merge: d257798beb 754e01fd54 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jan 3 20:34:06 2020 +0100 Merge pull request #2449 from dscho/mingw-getcwd-and-symlinks Do resolve symlinks in `getcwd()` commit d257798beb4d190efc7dbb4af5678ab1d268589e Merge: 559972559f 94299818f5 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Nov 25 23:21:29 2019 +0100 Merge pull request #2405 from dscho/mingw-setsockopt Make sure `errno` is set when socket operations fail commit 559972559f10ca6babdfcc125e8144327b7308f1 Merge: bac4800358 da3b994484 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Oct 30 09:18:45 2019 +0100 Merge pull request #2375 from assarbad/reintroduce-sideband-config Config option to disable side-band-64k for transport commit bac4800358f0adfb585eabb48269f7bfd2c502e5 Merge: a5525daf2d 16d55b5692 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Sep 8 20:16:45 2022 +0200 Merge branch 'mimalloc-v2.0.9' This topic vendors in mimalloc v2.0.9, a fast allocator that allows Git for Windows to perform efficiently. Switch Git for Windows to using mimalloc instead of nedmalloc commit a5525daf2d1b6300bcf7cba714360245bd7cea75 Merge: 0cdcff8ec8 32dd72d596 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri May 31 10:12:45 2019 +0200 Merge branch 'msys2-python' In MSYS2, we have two Python interpreters at our disposal, so we can include the Python stuff in the build. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0cdcff8ec8dfd502e77bde68996e97160aa08f21 Merge: e439f44fb3 ddace7f61d Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jul 30 10:19:14 2019 +0200 Merge branch 'dont-clean-junctions' This topic branch teaches `git clean` to respect NTFS junctions and Unix bind mounts: it will now stop at those boundaries. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit e439f44fb3ee3a249a72ce333fdf6da5f5b1bcb8 Merge: c8f33da33f 844fb31212 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jun 8 18:35:28 2018 +0200 Merge branch 'drive-prefix' This topic branch allows us to specify absolute paths without the drive prefix e.g. when cloning. Example: C:\Users\me> git clone https://github.com/git/git \upstream-git This will clone into a new directory C:\upstream-git, in line with how Windows interprets absolute paths. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c8f33da33f3493c97a17992b6b48a75caa4957f9 Merge: 2baa91f3e9 ff4998860b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jun 8 13:42:16 2018 +0200 Merge 'remote-hg-prerequisites' into HEAD These fixes were necessary for Sverre Rabbelier's remote-hg to work, but for some magic reason they are not necessary for the current remote-hg. Makes you wonder how that one gets away with it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 2baa91f3e90b56479b99a7ecd26156c457fa624f Merge: ef4c884834 93e1632007 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Jan 21 21:53:24 2023 +0100 Merge branch 'safe-PATH-lookup-in-gitk-on-Windows' This topic branch extends the protections introduced for Git GUI's CVE-2022-41953 to cover `gitk`, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b4de03a3b4ff103a8cf03bf5fddc4efb88e7902a Author: Jeff King <peff@peff.net> Date: Wed Oct 2 22:27:13 2024 -0400 diff: modify output_prefix function pointer The uses of the output_prefix function pointer in the diff_options struct is currently difficult to work with by returning a pointer to a strbuf. There is only one use that cares about the length of the string, which appears to be the only justification of the return type. We already noticed confusing memory issues around this return type, so use a const char * return type to make it clear that the caller does not own this string buffer. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit cba1e83da60bb433407b0fafbfe663a1de04e388 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jul 1 23:28:45 2024 +0200 survey: clearly note the experimental nature in the output While this command is definitely something we _want_, chances are that upstreaming this will require substantial changes. We still want to be able to experiment with this before that, to focus on what we need out of this command: To assist with diagnosing issues with large repositories, as well as to help monitoring the growth and the associated painpoints of such repositories. To that end, we are about to integrate this command into `microsoft/git`, to get the tool into the hands of users who need it most, with the idea to iterate in close collaboration between these users and the developers familar with Git's internals. However, we will definitely want to avoid letting anybody have the impression that this command, its exact inner workings, as well as its output format, are anywhere close to stable. To make that fact utterly clear (and thereby protect the freedom to iterate and innovate freely before upstreaming the command), let's mark its output as experimental in all-caps, as the first thing we do. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0f4160b8d04262453a3f6d4f7728eba5a877b143 Author: Jeff King <peff@peff.net> Date: Wed Oct 2 22:24:40 2024 -0400 line-log: remove output_prefix() Now that output_prefix() returns a const char * type, it matches the behavior of diff_line_prefix() and no longer needs to exist on its own. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit f5dc425f73eb23bfa13a35a1ee4af6b6e3e75ae7 Author: Derrick Stolee <stolee@gmail.com> Date: Mon Sep 23 15:38:25 2024 -0400 survey: add --top=<N> option and config The 'git survey' builtin provides several detail tables, such as "top files by on-disk size". The size of these tables defaults to 10, currently. Allow the user to specify this number via a new --top=<N> option or the new survey.top config key. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit df45c62c7153afc4cb64d8aa79ea3e15eddb186c Author: Derrick Stolee <stolee@gmail.com> Date: Wed Oct 2 11:21:42 2024 -0400 line-log: protect inner strbuf from free The output_prefix() method in line-log.c may call a function pointer via the diff_options struct. This function pointer returns a strbuf struct and then its buffer is passed back. However, that implies that the consumer is responsible to free the string. This is especially true because the default behavior is to duplicate the empty string. The existing functions used in the output_prefix pointer include: 1. idiff_prefix_cb() in diff-lib.c. This returns the data pointer, so the value exists across multiple calls. 2. diff_output_prefix_callback() in graph.c. This uses a static strbuf struct, so it reuses buffers across calls. These should not be freed. 3. output_prefix_cb() in range-diff.c. This is similar to the diff-lib.c case. In each case, we should not be freeing this buffer. We can convert the output_prefix() function to return a const char pointer and stop freeing the result. This choice is essentially the opposite of what was done in 394affd46d (line-log: always allocate the output prefix, 2024-06-07). This was discovered via 'valgrind' while investigating a public report of a bug in 'git log --graph -L' [1]. [1] https://github.com/git-for-windows/git/issues/5185 This issue would have been caught by the new test, when Git is compiled with ASan to catch these double frees. Co-authored-by: Jeff King <peff@peff.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 14a4284ad22c92fc3c8d63e6ca4d7c8dc51d2bc0 Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 22:35:40 2024 -0400 survey: add report of "largest" paths Since we are already walking our reachable objects using the path-walk API, let's now collect lists of the paths that contribute most to different metrics. Specifically, we care about * Number of versions. * Total size on disk. * Total inflated size (no delta or zlib compression). This information can be critical to discovering which parts of the repository are causing the most growth, especially on-disk size. Different packing strategies might help compress data more efficiently, but the toal inflated size is a representation of the raw size of all snapshots of those paths. Even when stored efficiently on disk, that size represents how much information must be processed to complete a command such as 'git blame'. Since the on-disk size is likely to be fragile, stop testing the exact output of 'git survey' and check that the correct set of headers is output. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 2e4f27f5dbe340f0387dd302f30272683e09f67e Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 22:35:06 2024 -0400 survey: add ability to track prioritized lists In future changes, we will make use of these methods. The intention is to keep track of the top contributors according to some metric. We don't want to store all of the entries and do a sort at the end, so track a constant-size table and remove rows that get pushed out depending on the chosen sorting algorithm. Co-authored-by: Jeff Hostetler <git@jeffhostetler.com> Signed-off-by; Jeff Hostetler <git@jeffhostetler.com> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit a1a8141c878ad3f6f85701492f9febe1e5d7ee73 Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 21:21:54 2024 -0400 survey: show progress during object walk Signed-off-by: Derrick Stolee <stolee@gmail.com> commit d13e5359d93936d44eea8c901f178d2e93973ff9 Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 20:58:35 2024 -0400 survey: summarize total sizes by object type Now that we have explored objects by count, we can expand that a bit more to summarize the data for the on-disk and inflated size of those objects. This information is helpful for diagnosing both why disk space (and perhaps clone or fetch times) is growing but also why certain operations are slow because the inflated size of the abstract objects that must be processed is so large. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 41b652d363ad8144f75fef98c14d30714fb54368 Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 20:33:47 2024 -0400 survey: add object count summary At the moment, nothing is obvious about the reason for the use of the path-walk API, but this will become more prevelant in future iterations. For now, use the path-walk API to sum up the counts of each kind of object. For example, this is the reachable object summary output for my local repo: REACHABLE OBJECT SUMMARY ======================== Object Type | Count ------------+------- Tags | 1343 Commits | 179344 Trees | 314350 Blobs | 184030 Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 6aa4b9eb5833a8344c1daa5bfbb62136b479431b Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 1 15:58:32 2024 -0400 survey: start pretty printing data in table form When 'git survey' provides information to the user, this will be presented in one of two formats: plaintext and JSON. The JSON implementation will be delayed until the functionality is complete for the plaintext format. The most important parts of the plaintext format are headers specifying the different sections of the report and tables providing concreted data. Create a custom table data structure that allows specifying a list of strings for the row values. When printing the table, check each column for the maximum width so we can create a table of the correct size from the start. The table structure is designed to be flexible to the different kinds of output that will be implemented in future changes. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 8a5625a8a104c0bb08b176da8f0004eca665d2e2 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Sep 26 02:51:06 2024 +0200 backfill: mark it as experimental This is a highly useful command, and we want it to get some testing "in the wild". However, the patches have not yet been reviewed on the Git mailing list, and are therefore subject to change. By marking the command as experimental, users will be warned to pay attention to those changes. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 671360ff0d17e3818db89e8573f3304dbb77c84d Author: Jeff Hostetler <jeffhostetler@github.com> Date: Mon Apr 29 09:51:34 2024 -0400 survey: add command line opts to select references By default we will scan all references in "refs/heads/", "refs/tags/" and "refs/remotes/". Add command line opts let the use ask for all refs or a subset of them and to include a detached HEAD. Signed-off-by: Jeff Hostetler <git@jeffhostetler.com> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 2b8f980809a2c6261ba6f86b53b29eceade3ddcf Author: Derrick Stolee <derrickstolee@github.com> Date: Sun Sep 1 14:06:11 2024 -0400 backfill: assume --sparse when sparse-checkout is enabled The previous change introduced the '--[no-]sparse' option for the 'git backfill' command, but did not assume it as enabled by default. However, this is likely the behavior that users will most often want to happen. Without this default, users with a small sparse-checkout may be confused when 'git backfill' downloads every version of every object in the full history. However, this is left as a separate change so this decision can be reviewed independently of the value of the '--[no-]sparse' option. Add a test of adding the '--sparse' option to a repo without sparse-checkout to make it clear that supplying it without a sparse-checkout is an error. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 76a01940dbde4612c08ba3de358c03f8f78fbb1e Author: Jeff Hostetler <jeffhostetler@github.com> Date: Mon Apr 29 08:55:03 2024 -0400 survey: stub in new experimental 'git-survey' command Start work on a new 'git survey' command to scan the repository for monorepo performance and scaling problems. The goal is to measure the various known "dimensions of scale" and serve as a foundation for adding additional measurements as we learn more about Git monorepo scaling problems. The initial goal is to complement the scanning and analysis performed by the GO-based 'git-sizer' (https://github.com/github/git-sizer) tool. It is hoped that by creating a builtin command, we may be able to take advantage of internal Git data structures and code that is not accessible from GO to gain further insight into potential scaling problems. Co-authored-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Jeff Hostetler <git@jeffhostetler.com> Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 89726809356605ecacc6b158bc75aa17d8a293c8 Author: Derrick Stolee <derrickstolee@github.com> Date: Sun Sep 1 13:39:18 2024 -0400 backfill: add --sparse option One way to significantly reduce the cost of a Git clone and later fetches is to use a blobless partial clone and combine that with a sparse-checkout that reduces the paths that need to be populated in the working directory. Not only does this reduce the cost of clones and fetches, the sparse-checkout reduces the number of objects needed to download from a promisor remote. However, history investigations can be expensie as computing blob diffs will trigger promisor remote requests for one object at a time. This can be avoided by downloading the blobs needed for the given sparse-checkout using 'git backfill' and its new '--sparse' mode, at a time that the user is willing to pay that extra cost. Note that this is distinctly different from the '--filter=sparse:<oid>' option, as this assumes that the partial clone has all reachable trees and we are using client-side logic to avoid downloading blobs outside of the sparse-checkout cone. This avoids the server-side cost of walking trees while also achieving a similar goal. It also downloads in batches based on similar path names, presenting a resumable download if things are interrupted. This augments the path-walk API to have a possibly-NULL 'pl' member that may point to a 'struct pattern_list'. This could be more general than the sparse-checkout definition at HEAD, but 'git backfill --sparse' is currently the only consumer. Be sure to test this in both cone mode and not cone mode. Cone mode has the benefit that the path-walk can skip certain paths once they would expand beyond the sparse-checkout. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 346c6c3f0697e77d810416ad99753987b0463fb1 Author: Derrick Stolee <derrickstolee@github.com> Date: Sun Sep 1 12:22:10 2024 -0400 backfill: add --batch-size=<n> option Users may want to specify a minimum batch size for their needs. This is only a minimum: the path-walk API provides a list of OIDs that correspond to the same path, and thus it is optimal to allow delta compression across those objects in a single server request. We could consider limiting the request to have a maximum batch size in the future. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 8baee8dacba0cc94a98840b3ed2ac092fe935656 Author: Derrick Stolee <derrickstolee@github.com> Date: Sun Sep 1 12:07:10 2024 -0400 backfill: basic functionality and tests The default behavior of 'git backfill' is to fetch all missing blobs that are reachable from HEAD. Document and test this behavior. The implementation is a very simple use of the path-walk API, initializing the revision walk at HEAD to start the path-walk from all commits reachable from HEAD. Ignore the object arrays that correspond to tree entries, assuming that they are all present already. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit b96788423c8a24a89b989cb49f3ab69933316a94 Author: Derrick Stolee <derrickstolee@github.com> Date: Fri Jun 7 12:18:50 2024 -0400 backfill: add builtin boilerplate In anticipation of implementing 'git backfill', populate the necessary files with the boilerplate of a new builtin. RFC TODO: When preparing this for a full implementation, make sure it is based on the newest standards introduced by [1]. [1] https://lore.kernel.org/git/xmqqjzfq2f0f.fsf@gitster.g/T/#m606036ea2e75a6d6819d6b5c90e729643b0ff7f7 [PATCH 1/3] builtin: add a repository parameter for builtin functions Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 6b6b2d86e0075a7c143bde2feae3d09d47544b22 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Oct 7 18:33:51 2024 +0200 path-walk API: avoid adding a root tree more than once When adding tree objects, we are very careful to avoid adding the same tree object more than once. There was one small gap in that logic, though: when adding a root tree object. Two refs can easily share the same root tree object, and we should still not add it more than once. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit a529a9336042d333513f25cb65acd65ca362a491 Author: Derrick Stolee <stolee@gmail.com> Date: Wed Sep 18 14:38:05 2024 -0400 pack-objects: thread the path-based compression Adapting the implementation of ll_find_deltas(), create a threaded version of the --path-walk compression step in 'git pack-objects'. This involves adding a 'regions' member to the thread_params struct, allowing each thread to own a section of paths. We can simplify the way jobs are split because there is no value in extending the batch based on name-hash the way sections of the object entry array are attempted to be grouped. We re-use the 'list_size' and 'remaining' items for the purpose of borrowing work in progress from other "victim" threads when a thread has finished its batch of work more quickly. Using the Git repository as a test repo, the p5313 performance test shows that the resulting size of the repo is the same, but the threaded implementation gives gains of varying degrees depending on the number of objects being packed. (This was tested on a 16-core machine.) Test HEAD~1 HEAD ------------------------------------------------------------- 5313.6: thin pack with --path-walk 0.01 0.01 +0.0% 5313.7: thin pack size with --path-walk 475 475 +0.0% 5313.12: big pack with --path-walk 1.99 1.87 -6.0% 5313.13: big pack size with --path-walk 14.4M 14.3M -0.4% 5313.18: repack with --path-walk 98.14 41.46 -57.8% 5313.19: repack size with --path-walk 197.2M 197.3M +0.0% Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 876f349a7b199f21bcd904b574c887e0f39548a5 Author: Derrick Stolee <stolee@gmail.com> Date: Wed Sep 18 13:38:08 2024 -0400 pack-objects: refactor path-walk delta phase Previously, the --path-walk option to 'git pack-objects' would compute deltas inline with the path-walk logic. This would make the progress indicator look like it is taking a long time to enumerate objects, and then very quickly computed deltas. Instead of computing deltas on each region of objects organized by tree, store a list of regions corresponding to these groups. These can later be pulled from the list for delta compression before doing the "global" delta search. This presents a new progress indicator that can be used in tests to verify that this stage is happening. The current implementation is not integrated with threads, but could be done in a future update. Since we do not attempt to sort objects by size until after exploring all trees, we can remove the previous change to t5530 due to a different error message appearing first. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 301365ce71bf2d2e3a45af26db7f853cbb08e207 Author: Derrick Stolee <stolee@gmail.com> Date: Thu Sep 5 09:51:33 2024 -0400 scalar: enable path-walk during push via config Repositories registered with Scalar are expected to be client-only repositories that are rather large. This means that they are more likely to be good candidates for using the --path-walk option when running 'git pack-objects', especially under the hood of 'git push'. Enable this config in Scalar repositories. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit e73cb1aeee4dd648a9619830c7d16af9ed9bb4c2 Author: Derrick Stolee <stolee@gmail.com> Date: Thu Sep 5 09:50:06 2024 -0400 pack-objects: enable --path-walk via config Users may want to enable the --path-walk option for 'git pack-objects' by default, especially underneath commands like 'git push' or 'git repack'. This should be limited to client repositories, since the --path-walk option disables bitmap walks, so would be bad to include in Git servers when serving fetches and clones. There is potential that it may be helpful to consider when repacking the repository, to take advantage of improved deltas across historical versions of the same files. Much like how "pack.useSparse" was introduced and included in "feature.experimental" before being enabled by default, use the repository settings infrastructure to make the new "pack.usePathWalk" config enabled by "feature.experimental" and "feature.manyFiles". Signed-off-by: Derrick Stolee <stolee@gmail.com> commit ed514d1b5b0cb4242884802d391cc784d993759a Author: Derrick Stolee <stolee@gmail.com> Date: Thu Sep 5 09:49:23 2024 -0400 repack: add --path-walk option Since 'git pack-objects' supports a --path-walk option, allow passing it through in 'git repack'. This presents interesting testing opportunities for comparing the different repacking strategies against each other. Add the --path-walk option to the performance tests in p5313. For the microsoft/fluentui repo [1] checked out at a specific commit [2], the results are very interesting: Test this tree ------------------------------------------------------------------ 5313.2: thin pack 0.40(0.47+0.04) 5313.3: thin pack size 1.2M 5313.4: thin pack with --full-name-hash 0.09(0.10+0.04) 5313.5: thin pack size with --full-name-hash 22.8K 5313.6: thin pack with --path-walk 0.08(0.06+0.02) 5313.7: thin pack size with --path-walk 20.8K 5313.8: big pack 2.16(8.43+0.23) 5313.9: big pack size 17.7M 5313.10: big pack with --full-name-hash 1.42(3.06+0.21) 5313.11: big pack size with --full-name-hash 18.0M 5313.12: big pack with --path-walk 2.21(8.39+0.24) 5313.13: big pack size with --path-walk 17.8M 5313.14: repack 98.05(662.37+2.64) 5313.15: repack size 449.1K 5313.16: repack with --full-name-hash 33.95(129.44+2.63) 5313.17: repack size with --full-name-hash 182.9K 5313.18: repack with --path-walk 106.21(121.58+0.82) 5313.19: repack size with --path-walk 159.6K [1] https://github.com/microsoft/fluentui [2] e70848ebac1cd720875bccaa3026f4a9ed700e08 This repo suffers from having a lot of paths that collide in the name hash, so examining them in groups by path leads to better deltas. Also, in this case, the single-threaded implementation is competitive with the full repack. This is saving time diffing files that have significant differences from each other. A similar, but private, repo has even more extremes in the thin packs: Test this tree -------------------------------------------------------------- 5313.2: thin pack 2.39(2.91+0.10) 5313.3: thin pack size 4.5M 5313.4: thin pack with --full-name-hash 0.29(0.47+0.12) 5313.5: thin pack size with --full-name-hash 15.5K 5313.6: thin pack with --path-walk 0.35(0.31+0.04) 5313.7: thin pack size with --path-walk 14.2K Notice, however, that while the --full-name-hash version is working quite well in these cases for the thin pack, it does poorly for some other standard cases, such as this test on the Linux kernel repository: Test this tree -------------------------------------------------------------- 5313.2: thin pack 0.01(0.00+0.00) 5313.3: thin pack size 310 5313.4: thin pack with --full-name-hash 0.00(0.00+0.00) 5313.5: thin pack size with --full-name-hash 1.4K 5313.6: thin pack with --path-walk 0.00(0.00+0.00) 5313.7: thin pack size with --path-walk 310 Here, the --full-name-hash option does much worse than the default name hash, but the path-walk option does exactly as well. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 1a1d999fb1912b887432692c417b935ff6d9c5f7 Author: Derrick Stolee <stolee@gmail.com> Date: Fri Sep 6 09:18:43 2024 -0400 pack-objects: introduce GIT_TEST_PACK_PATH_WALK There are many tests that validate whether 'git pack-objects' works as expected. Instead of duplicating these tests, add a new test environment variable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default when specified. This was useful in testing the implementation of the --path-walk implementation, especially in conjunction with test such as: - t0411-clone-from-partial.sh : One test fetches from a repo that does not have the boundary objects. This causes the path-based walk to fail. Disable the variable for this test. - t5306-pack-nobase.sh : Similar to t0411, one test fetches from a repo without a boundary object. - t5310-pack-bitmaps.sh : One test compares the case when packing with bitmaps to the case when packing without them. Since we disable the test variable when writing bitmaps, this causes a difference in the object list (the --path-walk option adds an extra object). Specify --no-path-walk in both processes for the comparison. Another test checks for a specific delta base, but when computing dynamically without using bitmaps, the base object it too small to be considered in the delta calculations so no base is used. - t5316-pack-delta-depth.sh : This script cares about certain delta choices and their chain lengths. The --path-walk option changes how these chains are selected, and thus changes the results of this test. - t5322-pack-objects-sparse.sh : This demonstrates the effectiveness of the --sparse option and how it combines with --path-walk. - t5332-multi-pack-reuse.sh : This test verifies that the preferred pack is used for delta reuse when possible. The --path-walk option is not currently aware of the preferred pack at all, so finds a different delta base. - t7406-submodule-update.sh : When using the variable, the --depth option collides with the --path-walk feature, resulting in a warning message. Disable the variable so this warning does not appear. I want to call out one specific test change that is only temporary: - t5530-upload-pack-error.sh : One test cares specifically about an "unable to read" error message. Since the current implementation performs delta calculations within the path-walk API callback, a different "unable to get size" error message appears. When this is changed in a future refactoring, this test change can be reverted. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit e6dca282dc29e799df081db418310b2bcc4f5aec Author: Derrick Stolee <stolee@gmail.com> Date: Thu Sep 5 10:04:51 2024 -0400 pack-objects: add --path-walk option In order to more easily compute delta bases among objects that appear at the exact same path, add a --path-walk option to 'git pack-objects'. This option will use the path-walk API instead of the object walk given by the revision machinery. Since objects will be provided in batches representing a common path, those objects can be tested for delta bases immediately instead of waiting for a sort of the full object list by name-hash. This has multiple benefits, including avoiding collisions by name-hash. The objects marked as UNINTERESTING are included in these batches, so we are guaranteeing some locality to find good delta bases. After the individual passes are done on a per-path basis, the default name-hash is used to find other opportunistic delta bases that did not match exactly by the full path name. RFC TODO: It is important to note that this option is inherently incompatible with using a bitmap index. This walk probably also does not work with other advanced features, such as delta islands. Getting ahead of myself, this option compares well with --full-name-hash when the packfile is large enough, but also performs at least as well as the default in all cases that I've seen. RFC TODO: this should probably be recording the batch locations to another list so they could be processed in a second phase using threads. RFC TODO: list some examples of how this outperforms previous pack-objects strategies. (This is coming in later commits that include performance test changes.) Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 9248fef0e95dc874e3996b065038359f272ff90a Author: Derrick Stolee <stolee@gmail.com> Date: Tue Sep 24 13:45:19 2024 -0400 pack-objects: extract should_attempt_deltas() This will be helpful in a future change. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 681453a9fca8cad9761d01fd95a964da7fe54b7b Author: Derrick Stolee <stolee@gmail.com> Date: Tue Sep 3 21:55:47 2024 -0400 path-walk: add prune_all_uninteresting option This option causes the path-walk API to act like the sparse tree-walk algorithm implemented by mark_trees_uninteresting_sparse() in list-objects.c. Starting from the commits marked as UNINTERESTING, their root trees and all objects reachable from those trees are UNINTERSTING, at least as we walk path-by-path. When we reach a path where all objects associated with that path are marked UNINTERESTING, then do no continue walking the children of that path. We need to be careful to pass the UNINTERESTING flag in a deep way on the UNINTERESTING objects before we start the path-walk, or else the depth-first search for the path-walk API may accidentally report some objects as interesting. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 13ec385303811150d48b78244a2812de516f8f11 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Sep 24 10:06:25 2024 +0200 repack/pack-objects: mark `--full-name-hash` as experimental This option is still under discussion on the Git mailing list. We still would like to have some real-world data, and the best way to get it is to get a Git for Windows release into users' hands so that they can test it. Nevertheless, without the official blessing of the Git maintainer, this optionis experimental, and we need to be clear about that. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 80c506cc4934ccd03d9bf9c81ef5208195f7b3c3 Author: Derrick Stolee <stolee@gmail.com> Date: Fri Sep 6 14:16:13 2024 -0400 revision: create mark_trees_uninteresting_dense() The sparse tree walk algorithm was created in d5d2e93577e (revision: implement sparse algorithm, 2019-01-16) and involves using the mark_trees_uninteresting_sparse() method. This method takes a repository and an oidset of tree IDs, some of which have the UNINTERESTING flag and some of which do not. Create a method that has an equivalent set of preconditions but uses a "dense" walk (recursively visits all reachable trees, as long as they have not previously been marked UNINTERESTING). This is an important difference from mark_tree_uninteresting(), which short-circuits if the given tree has the UNINTERESTING flag. A use of this method will be added in a later change, with a condition set whether the sparse or dense approach should be used. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 85d3ca34d7ad5652b2302fc19260da2c9a675973 Author: Derrick Stolee <stolee@gmail.com> Date: Sun Sep 8 21:04:52 2024 -0400 test-tool: add helper for name-hash values Add a new test-tool helper, name-hash, to output the value of the name-hash algorithms for the input list of strings, one per line. Since the name-hash values can be stored in the .bitmap files, it is important that these hash functions do not change across Git versions. Add a simple test to t5310-pack-bitmaps.sh to provide some testing of the current values. Due to how these functions are implemented, it would be difficult to change them without disturbing these values. Create a performance test that uses test_size to demonstrate how collisions occur for these hash algorithms. This test helps inform someone as to the behavior of the name-hash algorithms for their repo based on the paths at HEAD. My copy of the Git repository shows modest statistics around the collisions of the default name-hash algorithm: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 4.5K 5314.2: number of distinct name-hashes 4.1K 5314.3: number of distinct full-name-hashes 4.5K 5314.4: maximum multiplicity of name-hashes 13 5314.5: maximum multiplicity of fullname-hashes 1 Here, the maximum collision multiplicity is 13, but around 10% of paths have a collision with another path. In a more interesting example, the microsoft/fluentui [1] repo had these statistics at time of committing: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 19.6K 5314.2: number of distinct name-hashes 8.2K 5314.3: number of distinct full-name-hashes 19.6K 5314.4: maximum multiplicity of name-hashes 279 5314.5: maximum multiplicity of fullname-hashes 1 [1] https://github.com/microsoft/fluentui That demonstrates that of the nearly twenty thousand path names, they are assigned around eight thousand distinct values. 279 paths are assigned to a single value, leading the packing algorithm to sort objects from those paths together, by size. In this repository, no collisions occur for the full-name-hash algorithm. In a more extreme example, an internal monorepo had a much worse collision rate: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 221.6K 5314.2: number of distinct name-hashes 72.0K 5314.3: number of distinct full-name-hashes 221.6K 5314.4: maximum multiplicity of name-hashes 14.4K 5314.5: maximum multiplicity of fullname-hashes 2 Even in this repository with many more paths at HEAD, the collision rate was low and the maximum number of paths being grouped into a single bucket by the full-path-name algorithm was two. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit f2473fa1a7d1726218b3fb48cabbf5615818cb70 Author: Derrick Stolee <stolee@gmail.com> Date: Mon Sep 9 12:31:21 2024 -0400 path-walk: allow visiting tags In anticipation of using the path-walk API to analyze tags or include them in a pack-file, add the ability to walk the tags that were included in the revision walk. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit aa3b11ebcd374e2e6f57040dc603c8ada542ac2f Author: Derrick Stolee <stolee@gmail.com> Date: Wed Aug 28 12:07:42 2024 -0400 p5313: add size comparison test As custom options are added to 'git pack-objects' and 'git repack' to adjust how compression is done, use this new performance test script to demonstrate their effectiveness in performance and size. The recently-added --full-name-hash option swaps the default name-hash algorithm with one that attempts to uniformly distribute the hashes based on the full path name instead of the last 16 characters. This has a dramatic effect on full repacks for repositories with many versions of most paths. It can have a negative impact on cases such as pushing a single change. This can be seen by running pt5313 on the open source fluentui repository [1]. Most commits will have this kind of output for the thin and big pack cases, though certain commits (such as [2]) will have problematic thin pack size for other reasons. [1] https://github.com/microsoft/fluentui [2] a637a06df05360ce5ff21420803f64608226a875 Checked out at the parent of [2], I see the following statistics: Test this tree ------------------------------------------------------------------ 5313.2: thin pack 0.02(0.01+0.01) 5313.3: thin pack size 1.1K 5313.4: thin pack with --full-name-hash 0.02(0.01+0.00) 5313.5: thin pack size with --full-name-hash 3.0K 5313.6: big pack 1.65(3.35+0.24) 5313.7: big pack size 58.0M 5313.8: big pack with --full-name-hash 1.53(2.52+0.18) 5313.9: big pack size with --full-name-hash 57.6M 5313.10: repack 176.52(706.60+3.53) 5313.11: repack size 446.7K 5313.12: repack with --full-name-hash 37.47(134.18+3.06) 5313.13: repack size with --full-name-hash 183.1K Note that this demonstrates a 3x size _increase_ in the case that simulates a small "git push". The size change is neutral on the case of pushing the difference between HEAD and HEAD~1000. However, the full repack case is both faster and more efficient. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit d5b9900bbbde8694a1438b56d454d57b67f820ac Author: Derrick Stolee <derrickstolee@github.com> Date: Sun Sep 1 14:25:33 2024 -0400 path-walk: allow consumer to specify object types We add the ability to filter the object types in the path-walk API so the callback function is called fewer times. This adds the ability to ask for the commits in a list, as well. Future changes will add the ability to visit annotated tags. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit eacf5a85b7d5931ec4f4331588b4308f2f69f651 Author: Derrick Stolee <stolee@gmail.com> Date: Mon Sep 9 07:16:37 2024 -0400 git-repack: update usage to match docs This also adds the '--full-name-hash' option introduced in the previous change and adds newlines to the synopsis. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 570c27cbd6c2305f910ee92d011ccc208f8cbe49 Author: Derrick Stolee <stolee@gmail.com> Date: Wed Sep 18 10:03:03 2024 -0400 t6601: add helper for testing path-walk API Add some tests based on the current behavior, doing interesting checks for different sets of branches, ranges, and the --boundary option. This sets a baseline for the behavior and we can extend it as new options are introduced. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit a0933545a17e2c78d9a384df89a09f30af5f3680 Author: Derrick Stolee <stolee@gmail.com> Date: Tue Sep 17 15:24:02 2024 -0400 pack-objects: add GIT_TEST_FULL_NAME_HASH Add a new environment variable to opt-in to the --full-name-hash option in 'git pack-objects'. This allows for extra testing of the feature without repeating all of the test scenarios. But this option isn't free. There are a few tests that change behavior with the variable enabled. First, there are a few tests that are very sensitive to certain delta bases being picked. These are both involving the generation of thin bundles and then counting their objects via 'git index-pack --fix-thin' which pulls the delta base into the new packfile. For these tests, disable the option as a decent long-term option. Second, there are two tests in t5616-partial-clone.sh that I believe are actually broken scenarios. While the client is set up to clone the 'promisor-server' repo via a treeless partial clone filter (tree:0), that filter does not translate to the 'server' repo. Thus, fetching from these repos causes the server to think that the client has all reachable trees and blobs from the commits advertised as 'haves'. This leads the server to providing a thin pack assuming those objects as delta bases. Changing the name-hash algorithm presents new delta bases and thus breaks the expectations of these tests. An alternative could be to set up 'server' as a promisor server with the correct filter enabled. This may also point out more issues with partial clone being set up as a remote-based filtering mechanism and not a repository-wide setting. For now, do the minimal change to make the test work by disabling the test variable. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit c257ea0fb858c737d5508750f27ed89bca9dfb27 Author: Derrick Stolee <stolee@gmail.com> Date: Thu Aug 29 12:27:28 2024 -0400 path-walk: introduce an object walk by path In anticipation of a few planned applications, introduce the most basic form of a path-walk API. It currently assumes that there are no UNINTERESTING objects, and does not include any complicated filters. It calls a function pointer on groups of tree and blob objects as grouped by path. This only includes objects the first time they are discovered, so an object that appears at multiple paths will not be included in two batches. There are many future adaptations that could be made, but they are left for future updates when consumers are ready to take advantage of those features. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit ef4c884834a980d360862afba7200b5740e8ea8b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Sep 24 08:47:39 2024 +0200 ci: work around a problem with HTTP/2 vs libcurl v8.10.0 As reported in https://lore.kernel.org/git/ZuPKvYP9ZZ2mhb4m@pks.im/, libcurl v8.10.0 had a regression that was picked up by Git's t5559.30 "large fetch-pack requests can be sent using chunked encoding". This bug was fixed in libcurl v8.10.1. Sadly, the macos-13 runner image was updated in the brief window between these two libcurl versions, breaking each and every CI build, as reported at https://github.com/git-for-windows/git/issues/5159. This would usually not matter, we would just ignore the failing CI builds until the macos-13 runner image is rebuilt in a couple of days, and then the CI builds would succeed again. However. As has become the custom, a surprise Git version was released, and now that Git for Windows wants to follow suit, since Git for Windows has this custom of trying to never release a version with a failing CI build, we _must_ work around it. This patch implements this work-around, basically for the sake of Git for Windows v2.46.2's CI build. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 32b471e7d7cd1404292c989b87d8694cc5b56fd8 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jul 4 22:41:56 2024 +0200 run-command: be helpful with Git LFS fails on Windows 7 Git LFS is now built with Go 1.21 which no longer supports Windows 7. However, Git for Windows still wants to support Windows 7. Ideally, Git LFS would re-introduce Windows 7 support until Git for Windows drops support for Windows 7, but that's not going to happen: https://github.com/git-for-windows/git/issues/4996#issuecomment-2176152565 The next best thing we can do is to let the users know what is happening, and how to get out of their fix, at least. This is not quite as easy as it would first seem because programs compiled with Go 1.21 or newer will simply throw an exception and fail with an Access Violation on Windows 7. The only way I found to address this is to replicate the logic from Go's very own `version` command (which can determine the Go version with which a given executable was built) to detect the situation, and in that case offer a helpful error message. This addresses https://github.com/git-for-windows/git/issues/4996. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 6ac9202ed3a8b08d797a544ff0c5dfa7347f268f Author: Ariel Lourenco <ariellourenco@users.noreply.github.com> Date: Tue Jul 2 18:09:43 2024 -0300 Fallback to AppData if XDG_CONFIG_HOME is unset In order to be a better Windows citizenship, Git should save its configuration files on AppData folder. This can enables git configuration files be replicated between machines using the same Microsoft account logon which would reduce the friction of setting up Git on new systems. Therefore, if %APPDATA%\Git\config exists, we use it; otherwise $HOME/.config/git/config is used. Signed-off-by: Ariel Lourenco <ariellourenco@users.noreply.github.com> commit c6bada9d707556917d44b68b770dc211a18da56c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue May 21 13:55:26 2024 +0200 win32: ensure that `localtime_r()` is declared even in i686 builds The `__MINGW64__` constant is defined, surprise, surprise, only when building for a 64-bit CPU architecture. Therefore using it as a guard to define `_POSIX_C_SOURCE` (so that `localtime_r()` is declared, among other functions) is not enough, we also need to check `__MINGW32__`. Technically, the latter constant is defined even for 64-bit builds. But let's make things a bit easier to understand by testing for both constants. Making it so fixes this compile warning (turned error in GCC v14.1): archive-zip.c: In function 'dos_time': archive-zip.c:612:9: error: implicit declaration of function 'localtime_r'; did you mean 'localtime_s'? [-Wimplicit-function-declaration] 612 | localtime_r(&time, &tm); | ^~~~~~~~~~~ | localtime_s Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 14717ba1645d79ae4d82d01c616173655f1946e5 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Apr 9 16:50:56 2024 +0200 t5601/t7406(mingw): do run tests with symlink support A long time ago, we decided to run tests in Git for Windows' SDK with the default `winsymlinks` mode: copying instead of linking. This is still the default mode of MSYS2 to this day. However, this is not how most users run Git for Windows: As the majority of Git for Windows' users seem to be on Windows 10 and newer, likely having enabled Developer Mode (which allows creating symbolic links without administrator privileges), they will run with symlink support enabled. This is the reason why it is crucial to get the fixes for CVE-2024-? to the users, and also why it is crucial to ensure that the test suite exercises the related test cases. This commit ensures the latter. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 5325f5ac50cca50c2cc82679d96967a7d1caba51 Author: Derrick Stolee <stolee@gmail.com> Date: Tue Sep 17 14:53:52 2024 -0400 repack: test --full-name-hash option The new '--full-name-hash' option for 'git repack' is a simple pass-through to the underlying 'git pack-objects' subcommand. However, this subcommand may have other options and a temporary filename as part of the subcommand execution that may not be predictable or could change over time. The existing test_subcommand method requires an exact list of arguments for the subcommand. This is too rigid for our needs here, so create a new method, test_subcommand_flex. Use it to check that the --full-name-hash option is passing through. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit de082045d2e316e3262b4e0ee7f246c188c40568 Author: Derrick Stolee <stolee@gmail.com> Date: Sat Sep 7 19:43:45 2024 -0400 pack-objects: add --full-name-hash option The pack_name_hash() method has not been materially changed since it was introduced in ce0bd64299a (pack-objects: improve path grouping heuristics., 2006-06-05). The intention here is to group objects by path name, but also attempt to group similar file types together by making the most-significant digits of the hash be focused on the final characters. Here's the crux of the implementation: /* * This effectively just creates a sortable number from the * last sixteen non-whitespace characters. Last characters * count "most", so things that end in ".c" sort together. */ while ((c = *name++) != 0) { if (isspace(c)) continue; hash = (hash >> 2) + (c << 24); } As the comment mentions, this only cares about the last sixteen non-whitespace characters. This cause some filenames to collide more than others. Here are some examples that I've seen while investigating repositories that are growing more than they should be: * "/CHANGELOG.json" is 15 characters, and is created by the beachball [1] tool. Only the final character of the parent directory can differntiate different versions of this file, but also only the two most-significant digits. If that character is a letter, then this is always a collision. Similar issues occur with the similar "/CHANGELOG.md" path, though there is more opportunity for differences in the parent directory. * Localization files frequently have common filenames but differentiate via parent directories. In C#, the name "/strings.resx.lcl" is used for these localization files and they will all collide in name-hash. [1] https://github.com/microsoft/beachball I've come across many other examples where some internal tool uses a common name across multiple directories and is causing Git to repack poorly due to name-hash collisions. It is clear that the existing name-hash algorithm is optimized for repositories with short path names, but also is optimized for packing a single snapshot of a repository, not a repository with many versions of the same file. In my testing, this has proven out where the name-hash algorithm does a good job of finding peer files as delta bases when unable to use a historical version of that exact file. However, for repositories that have many versions of most files and directories, it is more important that the objects that appear at the same path are grouped together. Create a new pack_full_name_hash() method and a new --full-name-hash option for 'git pack-objects' to call that method instead. Add a simple pass-through for 'git repack --full-name-hash' for additional testing in the context of a full repack, where I expect this will be most effective. The hash algorithm is as simple as possible to be reasonably effective: for each character of the path string, add a multiple of that character and a large prime number (chosen arbitrarily, but intended to be large relative to the size of a uint32_t). Then, shift the current hash value to the right by 5, with overlap. The addition and shift parameters are standard mechanisms for creating hard-to-predict behaviors in the bits of the resulting hash. This is not meant to be cryptographic at all, but uniformly distributed across the possible hash values. This creates a hash that appears pseudorandom. There is no ability to consider similar file types as being close to each other. In a later change, a test-tool will be added so the effectiveness of this hash can be demonstrated directly. For now, let's consider how effective this mechanism is when repacking a repository with and without the --full-name-hash option. Specifically, let's use 'git repack -adf [--full-name-hash]' as our test. On the Git repository, we do not expect much difference. All path names are short. This is backed by our results: | Stage | Pack Size | Repack Time | |-----------------------|-----------|-------------| | After clone | 260 MB | N/A | | Standard Repack | 127MB | 106s | | With --full-name-hash | 126 MB | 99s | This example demonstrates how there is some natural overhead coming from the cloned copy because the server is hosting many forks and has not optimized for exactly this set of reachable objects. But the full repack has similar characteristics with and without --full-name-hash. However, we can test this in a repository that uses one of the problematic naming conventions above. The fluentui [2] repo uses beachball to generate CHANGELOG.json and CHANGELOG.md files, and these files have very poor delta characteristics when comparing against versions across parent directories. | Stage | Pack Size | Repack Time | |-----------------------|-----------|-------------| | After clone | 694 MB | N/A | | Standard Repack | 438 MB | 728s | | With --full-name-hash | 168 MB | 142s | [2] https://github.com/microsoft/fluentui In this example, we see significant gains in the compressed packfile size as well as the time taken to compute the packfile. Using a collection of repositories that use the beachball tool, I was able to make similar comparisions with dramatic results. While the fluentui repo is public, the others are private so cannot be shared for reproduction. The results are so significant that I find it important to share here: | Repo | Standard Repack | With --full-name-hash | |----------|-----------------|-----------------------| | fluentui | 438 MB | 168 MB | | Repo B | 6,255 MB | 829 MB | | Repo C | 37,737 MB | 7,125 MB | | Repo D | 130,049 MB | 6,190 MB | Future changes could include making --full-name-hash implied by a config value or even implied by default during a full repack. Signed-off-by: Derrick Stolee <stolee@gmail.com> commit 43a6bf7c0c563d491edfe0a23eb340fd862d5787 Author: MinarKotonoha <chengzhuo5@qq.com> Date: Mon Apr 8 16:41:10 2024 +0800 common-main.c: fflush stdout buffer upon exit By default, the buffer type of Windows' `stdout` is unbuffered (_IONBF), and there is no need to manually fflush `stdout`. But some programs, such as the Windows Filtering Platform driver provided by the security software, may change the buffer type of `stdout` to full buffering. This nees `fflush(stdout)` to be called manually, otherwise there will be no output to `stdout`. Signed-off-by: MinarKotonoha <chengzhuo5@qq.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 3a78230e8166fd212299b7662a59cdfd9dfc76d6 Author: Matthias Aßhauer <mha1993@live.de> Date: Sat Dec 2 12:10:00 2023 +0100 git.rc: include winuser.h winuser.h contains the definition of RT_MANIFEST that our LLVM based toolchain needs to understand that we want to embed compat/win32/git.manifest as an application manifest. It currently just embeds it as additional data that Windows doesn't understand. This also helps our GCC based toolchain understand that we only want one copy embedded. It currently embeds one working assembly manifest and one nearly identical, but useless copy as additional data. This also teaches our Visual Studio based buildsystems to pick up the manifest file from git.rc. This means we don't have to explicitly specify it in contrib/buildsystems/Generators/Vcxproj.pm anymore. Slightly counter-intuitively this also means we have to explicitly tell Cmake not to embed a default manifest. This fixes https://github.com/git-for-windows/git/issues/4707 Signed-off-by: Matthias Aßhauer <mha1993@live.de> Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> commit 26db42c6469d0f47a993ba999075e1c66c81cf8c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Nov 22 22:57:38 2023 +0100 win32: use native ANSI sequence processing, if possible Windows 10 version 1511 (also known as Anniversary Update), according to https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences introduced native support for ANSI sequence processing. This allows using colors from the entire 24-bit color range. All we need to do is test whether the console's "virtual processing support" can be enabled. If it can, we do not even need to start the `console_thread` to handle ANSI sequences. Or, almost all we need to do: When `console_thread()` does its work, it uses the Unicode-aware `write_console()` function to write to the Win32 Console, which supports Git for Windows' implicit convention that all text that is written is encoded in UTF-8. The same is not necessarily true if native ANSI sequence processing is used, as the output is then subject to the current code page. Let's ensure that the code page is set to `CP_UTF8` as long as Git writes to it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ba9010395e073980e3d54cec8e781e75c67f82c8 Author: David Lomas <dl3@pale-eds.co.uk> Date: Fri Jul 28 15:31:25 2023 +0100 mingw: suggest `windows.appendAtomically` in more cases When running Git for Windows on a remote APFS filesystem, it would appear that the `mingw_open_append()`/`write()` combination would fail almost exactly like on some CIFS-mounted shares as had been reported in https://github.com/git-for-windows/git/issues/2753, albeit with a different `errno` value. Let's handle that `errno` value just the same, by suggesting to set `windows.appendAtomically=false`. Signed-off-by: David Lomas <dl3@pale-eds.co.uk> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 9b0e692c569e62f50f34f5591a8586c6f9b37bc6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Nov 2 16:23:58 2022 +0100 Add a GitHub workflow to verify that Git/Scalar work in Nano Server In Git for Windows v2.39.0, we fixed a regression where `git.exe` would no longer work in Windows Nano Server (frequently used in Docker containers). This GitHub workflow can be used to verify manually that the Git/Scalar executables work in Nano Server. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit dea9b1c136044c4e1e23655a4ba5a28fc41c5cc1 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun May 7 22:43:37 2023 +0200 mingw: do load libcurl dynamically by default This will help with Git for Windows' maintenance going forward: It allows Git for Windows to switch its primary libcurl to a variant without the OpenSSL backend, while still loading an alternate when setting `http.sslBackend = openssl`. This is necessary to avoid maintenance headaches with upgrading OpenSSL: its major version name is encoded in the shared library's file name and hence major version updates (temporarily) break libraries that are linked against the OpenSSL library. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit e7a3b2dd48bcc00bb1d261d6c3e45ad685aa7698 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Mar 21 16:14:44 2023 +0100 windows: fix Repository>Explore Working Copy Since Git v2.39.1, we are a bit more stringent in searching the PATH. In particular, we specifically require the `.exe` suffix. However, the `Repository>Explore Working Copy` command asks for `explorer.exe` to be found on the `PATH`, which _already_ has that suffix. Let's unstartle the PATH-finding logic about this scenario. This fixes https://github.com/git-for-windows/git/issues/4356 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 7ba0501fdfdef2307cc48c7f8d635f4f95e70dd0 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jan 27 08:55:21 2023 +0100 windows: skip linking `git-<command>` for built-ins It is merely a historical wart that, say, `git-commit` exists in the `libexec/git-core/` directory, a tribute to the original idea to let Git be essentially a bunch of Unix shell scripts revolving around very few "plumbing" (AKA low-level) commands. Git has evolved a lot from there. These days, most of Git's functionality is contained within the `git` executable, in the form of "built-in" commands. To accommodate for scripts that use the "dashed" form of Git commands, even today, Git provides hard-links that make the `git` executable available as, say, `git-commit`, just in case that an old script has not been updated to invoke `git commit`. Those hard-links do not come cheap: they take about half a minute for every build of Git on Windows, they are mistaken for taking up huge amounts of space by some Windows Explorer versions that do not understand hard-links, and therefore many a "bug" report had to be addressed. The "dashed form" has been officially deprecated in Git version 1.5.4, which was released on February 2nd, 2008, i.e. a very long time ago. This deprecation was never finalized by skipping these hard-links, but we can start the process now, in Git for Windows. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 01c83bdaa00d02bd6cdd51994edda68d8142af0b Author: Andrey Zabavnikov <zabavnikov@gmail.com> Date: Fri Oct 28 17:12:06 2022 +0300 status: fix for old-style submodules with commondir In f9b7573f6b00 (repository: free fields before overwriting them, 2017-09-05), Git was taught to release memory before overwriting it, but 357a03ebe9e0 (repository.c: move env-related setup code back to environment.c, 2018-03-03) changed the code so that it would not _always_ be overwritten. As a consequence, the `commondir` attribute would point to already-free()d memory. This seems not to cause problems in core Git, but there are add-on patches in Git for Windows where the `commondir` attribute is subsequently used and causing invalid memory accesses e.g. in setups containing old-style submodules (i.e. the ones with a `.git` directory within theirs worktrees) that have `commondir` configured. This fixes https://github.com/git-for-windows/git/pull/4083. Signed-off-by: Andrey Zabavnikov <zabavnikov@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit f1a6eef66995539269dac6c1652f96cddd451fbf Author: Dennis Ameling <dennis@dennisameling.com> Date: Tue Oct 4 10:09:10 2022 +0200 ci: create clangarm64-build.yml No GitHub-hosted ARM64 runners are available at the moment of writing, but folks can leverage self-hosted runners of this architecture. This CI pipeline comes in handy for forks of the git-for-windows/git project that have such runners available. The pipeline can be kicked off manually through a workflow_dispatch. Signed-off-by: Dennis Ameling <dennis@dennisameling.com> commit a1059884ad6cdb782013e12ea9477308d82d32e4 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun May 7 22:05:33 2023 +0200 http: when loading libcurl lazily, allow for multiple SSL backends The previous commits introduced a compile-time option to load libcurl lazily, but it uses the hard-coded name "libcurl-4.dll" (or equivalent on platforms other than Windows). To allow for installing multiple libcurl flavors side by side, where each supports one specific SSL/TLS backend, let's first look whether `libcurl-<backend>-4.dll` exists, and only use `libcurl-4.dll` as a fall back. That will allow us to ship with a libcurl by default that only supports the Secure Channel backend for the `https://` protocol. This libcurl won't suffer from any dependency problem when upgrading OpenSSL to a new major version (which will change the DLL name, and hence break every program and library that depends on it). This is crucial because Git for Windows relies on libcurl to keep working when building and deploying a new OpenSSL package because that library is used by `git fetch` and `git clone`. Note that this feature is by no means specific to Windows. On Ubuntu, for example, a `git` built using `LAZY_LOAD_LIBCURL` will use `libcurl.so.4` for `http.sslbackend=openssl` and `libcurl-gnutls.so.4` for `http.sslbackend=gnutls`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b28e1a38f6e00b64b8c48952b148955f38f7a63d Author: Dennis Ameling <dennis@dennisameling.com> Date: Tue Oct 4 09:59:32 2022 +0200 config.mak.uname: add support for clangarm64 CLANGARM64 is a relatively new MSYSTEM added by the MSYS2 team. In order to have Git build correctly for this platform, let's add some configuration for it to config.mak.uname. Signed-off-by: Dennis Ameling <dennis@dennisameling.com> commit e26cdacdd4df29ebbad6d9f77c897900e0aea67c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun May 7 22:51:52 2023 +0200 http: support lazy-loading libcurl also on Windows This implements the Windows-specific support code, because everything is slightly different on Windows, even loading shared libraries. Note: I specifically do _not_ use the code from `compat/win32/lazyload.h` here because that code is optimized for loading individual functions from various system DLLs, while we specifically want to load _many_ functions from _one_ DLL here, and distinctly not a system DLL (we expect libcurl to be located outside `C:\Windows\system32`, something `INIT_PROC_ADDR` refuses to work with). Also, the `curl_easy_getinfo()`/`curl_easy_setopt()` functions are declared as vararg functions, which `lazyload.h` cannot handle. Finally, we are about to optionally override the exact file name that is to be loaded, which is a goal contrary to `lazyload.h`'s design. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b8d1b1adc9a70b3e048853dfb44df6929333f2e3 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat May 6 22:26:15 2023 +0200 http: optionally load libcurl lazily This compile-time option allows to ask Git to load libcurl dynamically at runtime. Together with a follow-up patch that optionally overrides the file name depending on the `http.sslBackend` setting, this kicks open the door for installing multiple libcurl flavors side by side, and load the one corresponding to the (runtime-)configured SSL/TLS backend. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit a25a8315d52fec126f2d7c5f3353ddeedede770d Author: Dennis Ameling <dennis@dennisameling.com> Date: Tue Oct 4 09:58:10 2022 +0200 bswap.h: add support for built-in bswap functions Newer compiler versions, like GCC 10 and Clang 12, have built-in functions for bswap32 and bswap64. This comes in handy, for example, when targeting CLANGARM64 on Windows, which would not be supported without this logic. Signed-off-by: Dennis Ameling <dennis@dennisameling.com> commit b9d207ad8243bfb30dcee89a88c351e8c32f4e64 Author: Kiel Hurley <kielhurley@gmail.com> Date: Wed Nov 2 22:56:16 2022 +1300 Fix Windows version resources Add FileVersion, which is a required field As not all required fields were present, none were being included Fixes #4090 Signed-off-by: Kiel Hurley <kielhurley@gmail.com> commit f106061c252fcb2d7799b91aaf6194ed28cb7b9e Author: Matthias Aßhauer <mha1993@live.de> Date: Sun Jul 10 11:27:25 2022 +0200 MinGW: link as terminal server aware Whith Windows 2000, Microsoft introduced a flag to the PE header to mark executables as "terminal server aware". Windows terminal servers provide a redirected Windows directory and redirected registry hives when launching legacy applications without this flag set. Since we do not use any INI files in the Windows directory and don't write to the registry, we don't need this additional preparation. Telling the OS that we don't need this should provide slightly improved startup times in terminal server environments. When building for supported Windows Versions with MSVC the /TSAWARE linker flag is automatically set, but MinGW requires us to set the --tsaware flag manually. This partially addresses https://github.com/git-for-windows/git/issues/3935. Signed-off-by: Matthias Aßhauer <mha1993@live.de> commit 5228689643f1d27095eb43a5d3fd6759d2b03f4e Author: Taylor Blau <me@ttaylorr.com> Date: Mon Feb 8 16:22:34 2021 -0500 azure-pipeline: run static-analysis on jammy This is inspired by d051ed77ee6 (.github/workflows/main.yml: run static-analysis on bionic, 2021-02-08) and by ef46584831 (ci: update 'static-analysis' to Ubuntu 22.04, 2022-08-23), adapted to the Azure Pipeline. When Azure Pipelines' build agents transitioned 'ubuntu-latest' from 18.04 to 20.04, it broke our `static-analysis` job, since Coccinelle was not madeavailable on Ubuntu focal (it is only available in the universe suite). This is not an issue with Ubuntu 22.04, but we will only know whether it is an issue with 24.04 when _that_ comes out. So let's play it safe and pin the `static_analysis` job to the latest Ubuntu version that we know to offer a working Coccinelle package. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b804b150768fadfc9538c49cdce7d3b11fe6466e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Jul 10 00:52:40 2022 +0200 azure-pipeline: downcase the job name of the `Linux32` job These many refactorings in Git sure are gifts that keep on giving. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 34c6616abe7802f0b4470a8b74a5820173dc59da Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Jul 10 00:14:53 2022 +0200 azure-pipeline: use partial clone/parallel checkout to initialize minimal-sdk The Azure Pipeline `git-sdk-64-minimal` was retired... Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ad91d62c1371c36044fbb891dd0c4ad5c1e6f7b5 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Jul 10 00:02:30 2022 +0200 azure-pipeline: drop the code to write to/read from a file share We haven't used this feature in ages, we don't actually need to. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 030da9d3de84bfa763781b418b09cd80c99c2f50 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Mar 29 12:28:12 2022 +0200 azure-pipeline: stop hard-coding `apt-get` calls We have `ci/install-dependencies.sh` for that. Incidentally, this avoids the following error in the linux-* jobs: The following packages have unmet dependencies: git-email : Depends: git (< 1:2.25.1-.) but 1:2.35.1-0ppa1~ubuntu20.04.1 is to be installed Recommends: libemail-valid-perl but it is not going to be installed Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 98fc4e8579230f6456893b19876825f228248f8e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Sep 4 11:59:45 2017 +0200 mingw: change core.fsyncObjectFiles = 1 by default From the documentation of said setting: This boolean will enable fsync() when writing object files. This is a total waste of time and effort on a filesystem that orders data writes properly, but can be useful for filesystems that do not use journalling (traditional UNIX filesystems) or that only journal metadata and not file contents (OS X’s HFS+, or Linux ext3 with "data=writeback"). The most common file system on Windows (NTFS) does not guarantee that order, therefore a sudden loss of power (or any other event causing an unclean shutdown) would cause corrupt files (i.e. files filled with NULs). Therefore we need to change the default. Note that the documentation makes it sound as if this causes really bad performance. In reality, writing loose objects is something that is done only rarely, and only a handful of files at a time. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit cca236844acee57cfbc2a9220c16ee201e5ce21a Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Mar 29 13:42:19 2022 +0200 azure-pipeline: drop the `GETTEXT_POISON` job This is a follow-up to 6c280b4142 (ci: remove GETTEXT_POISON jobs, 2021-01-20) after reinstating the Azure Pipeline. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b5b72e6884f0b8aebc9e74deae21ee1f521ae6ca Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Feb 11 15:09:57 2021 +0100 ci: reinstate Azure Pipelines support ... so that we can test a MinGit backport in a private repository (with GitHub Actions, minutes and parallel jobs are limited way more than with Azure Pipelines in private repositories). In this commit, we reinstate the exact version of `azure-pipelines.yml` as 6081d3898fe (ci: retire the Azure Pipelines definition, 2020-04-11) deleted. Naturally, many adjustments are required to make it work again. Some of the changes are actually outside of that file (such as the `runs_on_pool` changes that are needed in the Azure Pipelines part of `ci/lib.sh`) and they were made in the commits leading up to this here commit. However, other adjustments are required in the `azure-pipelines.yml` file itself, and for ease of review (read: to build confidence in those changes) they will be made in subsequent, individual commits that explain the intent, context, implementation and justification like every good commit message should do. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 241df0c63c2df73f2ca22ed0e745763b9c120563 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Jul 10 01:15:08 2022 +0200 ci: stop linking the `prove` cache It is not useful because we do not have any persisted directory anymore, not since dropping our Travis CI support. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ac081eaa5d6ec58c86e30b2efad5b9f1c848ad68 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Jul 10 00:39:32 2022 +0200 ci: adjust Azure Pipeline for `runs_on_pool` These refactorings are really gifts that keep on giving. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit bcdc2c9c1e49b83e65d6609d4f27bdac1b1f6aa6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jun 28 17:36:21 2022 +0200 vcxproj: avoid escaping double quotes in the defines Visual Studio 2022 does not like that at all. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 44ff60d59e9e956beb847a5c42fd16f6b1172a44 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jun 28 18:04:01 2022 +0200 vcxproj: handle libreftable_test, too Since ef8a6c6268 (reftable: utility functions, 2021-10-07) we not only have a libreftable, but also a libreftable_test. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 82961e3e565cf265c8817d0d53a9ec592dc513f5 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jun 28 17:00:59 2022 +0200 vcxproj: include reftable when committing `.vcxproj` files Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c8d8bcc0f06f81dca19c46aace571246a1844918 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jun 28 16:38:12 2022 +0200 vcxproj: ignore the `-pedantic` option This is now passed by default, ever since 6a8cbc41ba (developer: enable pedantic by default, 2021-09-03). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 52286c1986c0f2e4a3de609b69d5eac5209a5c9c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Jun 28 16:35:04 2022 +0200 vcxproj: require C11 This fixes the build after 7bc341e21b (git-compat-util: add a test balloon for C99 support, 2021-12-01). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 5ee69245f27afb72c93db02f58ef6bd0dfa7c6f1 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Mar 29 12:05:18 2022 +0200 vcxproj: allow building with `NO_PERL` again This is another fall-out of the recent refactoring flurry. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 98cd92e96f58873c278c332e92a0ac108c3f6de5 Author: 孙卓识 <sunzhuoshi@gmail.com> Date: Sun Jan 16 03:38:33 2022 +0800 Add config option `windows.appendAtomically` Atomic append on windows is only supported on local disk files, and it may cause errors in other situations, e.g. network file system. If that is the case, this config option should be used to turn atomic append off. Co-Authored-By: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: 孙卓识 <sunzhuoshi@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d3aa35916851e8fdb75f269f3a3f4ec4b1d7e0ce Author: Christopher Degawa <ccom@randomderp.com> Date: Sat May 28 14:53:54 2022 -0500 winansi: check result and Buffer before using Name NtQueryObject under Wine can return a success but fill out no name. In those situations, Wine will set Buffer to NULL, and set result to the sizeof(OBJECT_NAME_INFORMATION). Running a command such as echo "$(git.exe --version 2>/dev/null)" will crash due to a NULL pointer dereference when the code attempts to null terminate the buffer, although, weirdly, removing the subshell or redirecting stdout to a file will not trigger the crash. Code has been added to also check Buffer and Length to ensure the check is as robust as possible due to the current behavior being fragile at best, and could potentially change in the future This code is based on the behavior of NtQueryObject under wine and reactos. Signed-off-by: Christopher Degawa <ccom@randomderp.com> commit 7312915dd2163f02faacb849116734fb1db870c0 Author: Rafael Kitover <rkitover@gmail.com> Date: Tue Apr 12 19:53:33 2022 +0000 mingw: $env:TERM="xterm-256color" for newer OSes For Windows builds >= 15063 set $env:TERM to "xterm-256color" instead of "cygwin" because they have a more capable console system that supports this. Also set $env:COLORTERM="truecolor" if unset. $env:TERM is initialized so that ANSI colors in color.c work, see 29a3963484 (Win32: patch Windows environment on startup, 2012-01-15). See git-for-windows/git#3629 regarding problems caused by always setting $env:TERM="cygwin". This is the same heuristic used by the Cygwin runtime. Signed-off-by: Rafael Kitover <rkitover@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 763dbdd60e42c5b77a5139e22163ed3dd8fbb365 Author: Derrick Stolee <derrickstolee@github.com> Date: Wed Apr 13 14:54:43 2022 -0400 compat/mingw.c: do not warn when failing to get owner In the case of Git for Windows (say, in a Git Bash window) running in a Windows Subsystem for Linux (WSL) directory, the GetNamedSecurityInfoW() call in is_path_owned_By_current_side() returns an error code other than ERROR_SUCCESS. This is consistent behavior across this boundary. In these cases, the owner would always be different because the WSL owner is a different entity than the Windows user. The change here is to suppress the error message that looks like this: error: failed to get owner for '//wsl.localhost/...' (1) Before this change, this warning happens for every Git command, regardless of whether the directory is marked with safe.directory. Signed-off-by: Derrick Stolee <derrickstolee@github.com> commit 26a985de9163cddf8b83f7658d50029528085488 Author: Philip Oakley <philipoakley@iee.email> Date: Tue Dec 7 09:53:41 2021 +0000 hash-object: add a >4GB/LLP64 test case using filtered input To verify that the `clean` side of the `clean`/`smudge` filter code is correct with regards to LLP64 (read: to ensure that `size_t` is used instead of `unsigned long`), here is a test case using a trivial filter, specifically _not_ writing anything to the object store to limit the scope of the test case. As in previous commits, the `big` file from previous test cases is reused if available, to save setup time, otherwise re-generated. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0c556646745c0feb42eaa1c3bdcd91e8ee26d6be Author: Derrick Stolee <derrickstolee@github.com> Date: Wed Apr 13 14:49:17 2022 -0400 setup: properly use "%(prefix)/" when in WSL Signed-off-by: Derrick Stolee <derrickstolee@github.com> commit 4bf6a23451e2a040cdb071fca2568394d4184401 Author: Philip Oakley <philipoakley@iee.email> Date: Mon Dec 6 22:42:46 2021 +0000 hash-object: add another >4GB/LLP64 test case To complement the `--stdin` and `--literally` test cases that verify that we can hash files larger than 4GB on 64-bit platforms using the LLP64 data model, here is a test case that exercises `hash-object` _without_ any options. Just as before, we use the `big` file from the previous test case if it exists to save on setup time, otherwise generate it. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 79f1c0bde5a8356c5cb289539ac1afa559c4bfe9 Author: Philip Oakley <philipoakley@iee.email> Date: Mon Dec 6 22:26:50 2021 +0000 hash-object --stdin: verify that it works with >4GB/LLP64 Just like the `hash-object --literally` code path, the `--stdin` code path also needs to use `size_t` instead of `unsigned long` to represent memory sizes, otherwise it would cause problems on platforms using the LLP64 data model (such as Windows). To limit the scope of the test case, the object is explicitly not written to the object store, nor are any filters applied. The `big` file from the previous test case is reused to save setup time; To avoid relying on that side effect, it is generated if it does not exist (e.g. when running via `sh t1007-*.sh --long --run=1,41`). Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 046d09e116c4ca032e0c37264dbbb737789942a9 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Mar 5 23:12:11 2021 +0100 Enable the built-in FSMonitor as an experimental feature If `feature.experimental` and `feature.manyFiles` are set and the user has not explicitly turned off the builtin FSMonitor, we now start the built-in FSMonitor by default. Only forcing it when UNSET matches the behavior of UPDATE_DEFAULT_BOOL() used for other repo settings. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 24ccbac24d7ccd94fc8da0bad57aef349385ba8b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Sep 8 13:05:42 2021 +0200 init: do parse _all_ core.* settings early In Git for Windows, `has_symlinks` is set to 0 by default. Therefore, we need to parse the config setting `core.symlinks` to know if it has been set to `true`. In `git init`, we must do that before copying the templates because they might contain symbolic links. Even if the support for symbolic links on Windows has not made it to upstream Git yet, we really should make sure that all the `core.*` settings are parsed before proceeding, as they might very well change the behavior of `git init` in a way the user intended. This fixes https://github.com/git-for-windows/git/issues/3414 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 21d32cf033a1f7b3d3974758de7cc07dfb1210c4 Author: Philip Oakley <philipoakley@iee.email> Date: Mon May 10 16:47:40 2021 +0100 CMake: show Win32 and Generator_platform build-option values Ensure key CMake option values are part of the CMake output to facilitate user support when tool updates impact the wider CMake actions, particularly ongoing 'improvements' in Visual Studio. These CMake displays perform the same function as the build-options.txt provided in the main Git for Windows. CMake is already chatty. The setting of CMAKE_EXPORT_COMPILE_COMMANDS is also reported. Include the environment's CMAKE_EXPORT_COMPILE_COMMANDS value which may have been propogated to CMake's internal value. Testing the CMAKE_EXPORT_COMPILE_COMMANDS processing can be difficult in the Visual Studio environment, as it may be cached in many places. The 'environment' may include the OS, the user shell, CMake's own environment, along with the Visual Studio presets and caches. See previous commit for arefacts that need removing for a clean test. Signed-off-by: Philip Oakley <philipoakley@iee.email> commit de40b92771ec43868c4ce8ed102e8173923c26c2 Author: Philip Oakley <philipoakley@iee.email> Date: Fri Nov 12 21:16:51 2021 +0000 hash algorithms: use size_t for section lengths Continue walking the code path for the >4GB `hash-object --literally` test to the hash algorithm step for LLP64 systems. This patch lets the SHA1DC code use `size_t`, making it compatible with LLP64 data models (as used e.g. by Windows). The interested reader of this patch will note that we adjust the signature of the `git_SHA1DCUpdate()` function without updating _any_ call site. This certainly puzzled at least one reviewer already, so here is an explanation: This function is never called directly, but always via the macro `platform_SHA1_Update`, which is usually called via the macro `git_SHA1_Update`. However, we never call `git_SHA1_Update()` directly in `struct git_hash_algo`. Instead, we call `git_hash_sha1_update()`, which is defined thusly: static void git_hash_sha1_update(git_hash_ctx *ctx, const void *data, size_t len) { git_SHA1_Update(&ctx->sha1, data, len); } i.e. it contains an implicit downcast from `size_t` to `unsigned long` (before this here patch). With this patch, there is no downcast anymore. With this patch, finally, the t1007-hash-object.sh "files over 4GB hash literally" test case is fixed. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 3df94e751b896b413e752266d7f99075e0c3d115 Author: Philip Oakley <philipoakley@iee.email> Date: Thu Apr 22 11:11:38 2021 +0100 CMakeLists: add default "x64-windows" arch for Visual Studio In Git-for-Windows, work on using ARM64 has progressed. The commit 2d94b77b27 (cmake: allow building for Windows/ARM64, 2020-12-04) failed to notice that /compat/vcbuild/vcpkg_install.bat will default to using the "x64-windows" architecture for the vcpkg installation if not set, but CMake is not told of this default. Commit 635b6d99b3 (vcbuild: install ARM64 dependencies when building ARM64 binaries, 2020-01-31) later updated vcpkg_install.bat to accept an arch (%1) parameter, but retained the default. This default is neccessary for the use case where the project directory is opened directly in Visual Studio, which will find and build a CMakeLists.txt file without any parameters, thus expecting use of the default setting. Also Visual studio will generate internal .sln solution and .vcxproj project files needed for some extension tools. Inform users of the additional .sln/.vcxproj generation. ** How to test: rm -rf '.vs' # remove old visual studio settings rm -rf 'compat/vcbuild/vcpkg' # remove any vcpkg downloads rm -rf 'contrib/buildsystems/out' # remove builds & CMake artifacts with a fresh Visual Studio Community Edition, File>>Open>>(git *folder*) to load the project (which will take some time!). check for successful compilation. The implicit .sln (etc.) are in the hidden .vs directory created by Visual Studio. Signed-off-by: Philip Oakley <philipoakley@iee.email> commit e87b48279a8ed945868242fecaf0ec7bd65d2b7b Author: Philip Oakley <philipoakley@iee.email> Date: Fri Nov 12 21:14:50 2021 +0000 object-file.c: use size_t for header lengths Continue walking the code path for the >4GB `hash-object --literally` test. The `hash_object_file_literally()` function internally uses both `hash_object_file()` and `write_object_file_prepare()`. Both function signatures use `unsigned long` rather than `size_t` for the mem buffer sizes. Use `size_t` instead, for LLP64 compatibility. While at it, convert those function's object's header buffer length to `size_t` for consistency. The value is already upcast to `uintmax_t` for print format compatibility. Note: The hash-object test still does not pass. A subsequent commit continues to walk the call tree's lower level hash functions to identify further fixes. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b43a862eb52d40720e174cf36fb5678c7f2dba5c Author: Philip Oakley <philipoakley@iee.email> Date: Fri Nov 12 21:07:03 2021 +0000 write_object_file_literally(): use size_t The previous commit adds a test that demonstrates a problem in the `hash-object --literally` command, manifesting in an unnecessary file size limit on systems using the LLP64 data model (which includes Windows). Walking the affected code path is `cmd_hash_object()` >> `hash_fd()` >> `hash_literally()` >> `hash_object_file_literally()`. The function `hash_object_file_literally()` is the first with a file length parameter (via a mem buffer). This commit changes the type of that parameter to the LLP64 compatible `size_t` type. There are no other uses of the function. The `strbuf` type is already `size_t` compatible. Note: The hash-object test does not yet pass. Subsequent commits will continue to walk the call tree's lower level functions to identify further fixes. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 733bb391669fce30ac5a6dc0087886561397fc50 Author: Philip Oakley <philipoakley@iee.email> Date: Sun Oct 31 23:15:13 2021 +0000 hash-object: demonstrate a >4GB/LLP64 problem On LLP64 systems, such as Windows, the size of `long`, `int`, etc. is only 32 bits (for backward compatibility). Git's use of `unsigned long` for file memory sizes in many places, rather than size_t, limits the handling of large files on LLP64 systems (commonly given as `>4GB`). Provide a minimum test for handling a >4GB file. The `hash-object` command, with the `--literally` and without `-w` option avoids writing the object, either loose or packed. This avoids the code paths hitting the `bigFileThreshold` config test code, the zlib code, and the pack code. Subsequent patches will walk the test's call chain, converting types to `size_t` (which is larger in LLP64 data models) where appropriate. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 7547439b8a1156bbc7fa034fa7afbaee52b11042 Author: Victoria Dye <vdye@github.com> Date: Thu Aug 5 19:11:59 2021 -0400 ci: run `contrib/subtree` tests in CI builds Because `git subtree` (unlike most other `contrib` modules) is included as part of the standard release of Git for Windows, its stability should be verified as consistently as it is for the rest of git. By including the `git subtree` tests in the CI workflow, these tests are as much of a gate to merging and indicator of stability as the standard test suite. Signed-off-by: Victoria Dye <vdye@github.com> commit 7697e7a6e4c823f5039ea4428325fe1c30ff3e7f Author: Pascal Muller <pascalmuller@gmail.com> Date: Wed Jun 23 21:21:10 2021 +0200 http: optionally send SSL client certificate This adds support for a new http.sslAutoClientCert config value. In cURL 7.77 or later the schannel backend does not automatically send client certificates from the Windows Certificate Store anymore. This config value is only used if http.sslBackend is set to "schannel", and can be used to opt in to the old behavior and force cURL to send client certificates. This fixes https://github.com/git-for-windows/git/issues/3292 Signed-off-by: Pascal Muller <pascalmuller@gmail.com> commit 66cd21d53635e2a3419b6fcb92f2688ccee45624 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon May 17 10:46:52 2021 +0200 compat/vcbuild: document preferred way to build in Visual Studio We used to have that `make vcxproj` hack, but a hack it is. In the meantime, we have a much cleaner solution: using CMake, either explicitly, or even more conveniently via Visual Studio's built-in CMake support (simply open Git's top-level directory via File>Open>Folder...). Let's let the `README` reflect this. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit da60c899b1c5d9a24f9051956cd38b84cafe47dc Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Apr 2 22:50:54 2021 +0200 mingw: allow for longer paths in `parse_interpreter()` As reported in https://github.com/newren/git-filter-repo/pull/225, it looks like 99 bytes is not really sufficient to represent e.g. the full path to Python when installed via Windows Store (and this path is used in the hasb bang line when installing scripts via `pip`). Let's increase it to what is probably the maximum sensible path size: MAX_PATH. This makes `parse_interpreter()` in line with what `lookup_prog()` handles. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Vilius Šumskas <vilius@sumskas.eu> commit b2fbec6b9127d1fdadf3c1193e6d96b141217203 Author: Dennis Ameling <dennis@dennisameling.com> Date: Mon Jul 19 13:02:16 2021 +0200 cmake(): allow setting HOST_CPU for cross-compilation Git's regular Makefile mentions that HOST_CPU should be defined when cross-compiling Git: https://github.com/git-for-windows/git/blob/37796bca76ef4180c39ee508ca3e42c0777ba444/Makefile#L438-L439 This is then used to set the GIT_HOST_CPU variable when compiling Git: https://github.com/git-for-windows/git/blob/37796bca76ef4180c39ee508ca3e42c0777ba444/Makefile#L1337-L1341 Then, when the user runs `git version --build-options`, it returns that value: https://github.com/git-for-windows/git/blob/37796bca76ef4180c39ee508ca3e42c0777ba444/help.c#L658 This commit adds the same functionality to the CMake configuration. Users can now set -DHOST_CPU= to set the target architecture. Signed-off-by: Dennis Ameling <dennis@dennisameling.com> commit 2fd18243468240ad9e036b465dcb8033878dbe62 Author: Philip Oakley <philipoakley@iee.email> Date: Sat Apr 24 11:09:58 2021 +0100 .gitignore: add Visual Studio CMakeSetting.json file The CMakeSettings.json file is tool generated. Developers may track it should they provide additional settings. Signed-off-by: Philip Oakley <philipoakley@iee.email> commit 8c385641928fdcf8cf5a93904af399d12b9812fc Author: Victoria Dye <vdye@github.com> Date: Thu Aug 5 19:04:13 2021 -0400 subtree: update `contrib/subtree` `test` target The intention of this change is to align with how the top-level git `Makefile` defines its own test target (which also internally calls `$(MAKE) -C t/ all`). This change also ensures the consistency of `make -C contrib/subtree test` with other testing in CI executions (which rely on `$DEFAULT_TEST_TARGET` being defined as `prove`). Signed-off-by: Victoria Dye <vdye@github.com> commit 3f721844ec1515c423662723d0f5b151814698c0 Author: Philip Oakley <philipoakley@iee.email> Date: Fri Jul 2 00:30:24 2021 +0100 CMake: default Visual Studio generator has changed Correct some wording and inform users regarding the Visual Studio changes (from V16.6) to the default generator. Subsequent commits ensure that Git for Windows can be directly opened in modern Visual Studio without needing special configuration of the CMakeLists settings. It appeares that internally Visual Studio creates it's own version of the .sln file (etc.) for extension tools that expect them. The large number of references below document the shifting of Visual Studio default and CMake setting options. refs: https://docs.microsoft.com/en-us/search/?scope=C%2B%2B&view=msvc-150&terms=Ninja 1. https://docs.microsoft.com/en-us/cpp/linux/cmake-linux-configure?view=msvc-160 (note the linux bit) "In Visual Studio 2019 version 16.6 or later ***, Ninja is the default generator for configurations targeting a remote system or WSL. For more information, see this post on the C++ Team Blog [https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/]. For more information about these settings, see CMakeSettings.json reference [https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160]." 2. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160 "CMake supports two files that allow users to specify common configure, build, and test options and share them with others: CMakePresets.json and CMakeUserPresets.json." " Both files are supported in Visual Studio 2019 version 16.10 or later. ***" 3. https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/ " Ninja has been the default generator (underlying build system) for CMake configurations targeting Windows for some time***, but in Visual Studio 2019 version 16.6 Preview 3*** we added support for Ninja on Linux." 4. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160 " `generator`: specifies CMake generator to use for this configuration. May be one of: Visual Studio 2019 only: Visual Studio 16 2019 Visual Studio 16 2019 Win64 Visual Studio 16 2019 ARM Visual Studio 2017 and later: Visual Studio 15 2017 Visual Studio 15 2017 Win64 Visual Studio 15 2017 ARM Visual Studio 14 2015 Visual Studio 14 2015 Win64 Visual Studio 14 2015 ARM Unix Makefiles Ninja Because Ninja is designed for fast build speeds instead of flexibility and function, it is set as the default. However, some CMake projects may be unable to correctly build using Ninja. If this occurs, you can instruct CMake to generate Visual Studio projects instead. To specify a Visual Studio generator in Visual Studio 2017, open the settings editor from the main menu by choosing CMake | Change CMake Settings. Delete "Ninja" and type "V". This activates IntelliSense, which enables you to choose the generator you want." "To specify a Visual Studio generator in Visual Studio 2019, right-click on the CMakeLists.txt file in Solution Explorer and choose CMake Settings for project > Show Advanced Settings > CMake Generator. When the active configuration specifies a Visual Studio generator, by default MSBuild.exe is invoked with` -m -v:minimal` arguments." 5. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019 "Enable CMakePresets.json integration in Visual Studio 2019 CMakePresets.json integration isn't enabled by default in Visual Studio 2019. You can enable it for all CMake projects in Tools > Options > CMake > General: (tick a box)" ... see more. 6. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-140 (whichever v140 is..) "CMake projects are supported in Visual Studio 2017 and later." 7. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150 "Support added for the CMake Ninja generator." 8. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150#cmake-support-via-open-folder "CMake support via Open Folder Visual Studio 2017 introduces support for using CMake projects without converting to MSBuild project files (.vcxproj). For more information, see CMake projects in Visual Studio[https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-150]. Opening CMake projects with Open Folder automatically configures the environment for C++ editing, building, and debugging." ... +more! 9. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#supported-cmake-and-cmakepresetsjson-versions "Visual Studio reads and evaluates CMakePresets.json and CMakeUserPresets.json itself and doesn't invoke CMake directly with the --preset option. So, CMake version 3.20 or later isn't strictly required when you're building with CMakePresets.json inside Visual Studio. We recommend using CMake version 3.14 or later." 10. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019 "If you don't want to enable CMakePresets.json integration for all CMake projects, you can enable CMakePresets.json integration for a single CMake project by adding a CMakePresets.json file to the root of the open folder. You must close and reopen the folder in Visual Studio to activate the integration. 11. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#default-configure-presets ***(doesn't actually say which version..) "Default Configure Presets If no CMakePresets.json or CMakeUserPresets.json file exists, or if CMakePresets.json or CMakeUserPresets.json is invalid, Visual Studio will fall back*** on the following default Configure Presets: Windows example JSON { "name": "windows-default", "displayName": "Windows x64 Debug", "description": "Sets Ninja generator, compilers, x64 architecture, build and install directory, debug build type", "generator": "Ninja", "binaryDir": "${sourceDir}/out/build/${presetName}", "architecture": { "value": "x64", "strategy": "external" }, "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug", "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}" }, "vendor": { "microsoft.com/VisualStudioSettings/CMake/1.0": { "hostOS": [ "Windows" ] } } }, " Signed-off-by: Philip Oakley <philipoakley@iee.email> commit dff6c6ba8b9b1db323cc557c733ff592a90a4d40 Author: Dennis Ameling <dennis@dennisameling.com> Date: Sun Dec 6 18:39:26 2020 +0100 Add schannel to curl installation Signed-off-by: Dennis Ameling <dennis@dennisameling.com> commit 51dd4303713c537138887606a2b3add68aa0c4f3 Author: Philip Oakley <philipoakley@iee.email> Date: Sun Oct 6 18:43:57 2019 +0100 vcpkg_install: add comment regarding slow network connections The vcpkg downloads may not succeed. Warn careful readers of the time out. A simple retry will usually resolve the issue. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit d7f33388c20d479b32501ea00e8aeaaef916e537 Author: Yuyi Wang <Strawberry_Str@hotmail.com> Date: Sat Mar 11 17:51:18 2023 +0800 cmake: install headless-git. headless-git is a git executable without opening a console window. It is useful when other GUI executables want to call git. We should install it together with git on Windows. Signed-off-by: Yuyi Wang <Strawberry_Str@hotmail.com> commit 28fdf667fa994c3ed0f978d74c606777257ad736 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Apr 7 15:48:50 2021 +0200 vcxproj: handle GUI programs, too So far, we only built Console programs, but we are about to introduce a program that targets the Windows subsystem (i.e. it is a so-called "GUI" program). Let's handle this preemptively in the script that generates the Visual Studio files. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 2bc5e7fc049f2a10fec84ef08ee9f405d174594c Author: Dennis Ameling <dennis@dennisameling.com> Date: Sun Nov 29 00:12:26 2020 +0100 ci(vs-build) also build Windows/ARM64 artifacts There are no Windows/ARM64 agents in GitHub Actions yet, therefore we just skip adjusting the `vs-test` job for now. Signed-off-by: Dennis Ameling <dennis@dennisameling.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 343825a58c1b5d208f59cf845fd23c3d0f78700c Author: Philip Oakley <philipoakley@iee.email> Date: Sun Oct 6 18:40:55 2019 +0100 vcpkg_install: detect lack of Git The vcpkg_install batch file depends on the availability of a working Git on the CMD path. This may not be present if the user has selected the 'bash only' option during Git-for-Windows install. Detect and tell the user about their lack of a working Git in the CMD window. Fixes #2348. A separate PR https://github.com/git-for-windows/build-extra/pull/258 now highlights the recommended path setting during install. Signed-off-by: Philip Oakley <philipoakley@iee.email> commit f41e9d75d0546e0eef9fd49db4c4b05b27649ce9 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Apr 7 21:57:31 2021 +0200 vcxproj: ignore -fno-stack-protector and -fno-common An upcoming commit will introduce those compile options; MSVC does not understand them, so let's suppress them when generating the Visual Studio project files. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 83debec48cd7524eb9e5bb52233464072c52a736 Author: Dennis Ameling <dennis@dennisameling.com> Date: Fri Dec 4 14:11:34 2020 +0100 cmake: allow building for Windows/ARM64 Signed-off-by: Dennis Ameling <dennis@dennisameling.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ee7186765f7eb197d27f74347b1c35ced02da68e Author: Ian Bearman <ianb@microsoft.com> Date: Tue Feb 4 10:34:40 2020 -0800 vcbuild: add an option to install individual 'features' In this context, a "feature" is a dependency combined with its own dependencies. Signed-off-by: Ian Bearman <ianb@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit cc84a125aedf365f6a939f44e872c16047a6fd30 Author: Ian Bearman <ianb@microsoft.com> Date: Fri Jan 31 16:00:25 2020 -0800 vcbuild: install ARM64 dependencies when building ARM64 binaries Co-authored-by: Dennis Ameling <dennis@dennisameling.com> Signed-off-by: Ian Bearman <ianb@microsoft.com> Signed-off-by: Dennis Ameling <dennis@dennisameling.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 83a0d9356c864f99e52cce47e8cbca74441e9826 Author: Ian Bearman <ianb@microsoft.com> Date: Fri Jan 31 15:37:27 2020 -0800 vcxproj: support building Windows/ARM64 binaries Signed-off-by: Ian Bearman <ianb@microsoft.com> Signed-off-by: Dennis Ameling <dennis@dennisameling.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 58c4696d18de9f07e9cb4239331567fd76f651bb Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Aug 12 15:06:17 2020 +0000 git-gui: accommodate for intent-to-add files As of Git v2.28.0, the diff for files staged via `git add -N` marks them as new files. Git GUI was ill-prepared for that, and this patch teaches Git GUI about them. Please note that this will not even fix things with v2.28.0, as the `rp/apply-cached-with-i-t-a` patches are required on Git's side, too. This fixes https://github.com/git-for-windows/git/issues/2779 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Pratyush Yadav <me@yadavpratyush.com> commit d9ab7f783ae798870c6881cce9343a65e205cacd Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Apr 7 15:15:08 2021 +0200 vcxproj: handle resource files, too On Windows, we also compile a "resource" file, which is similar to source code, but contains metadata (such as the program version). So far, we did not compile it in `MSVC` mode, only when compiling Git for Windows with the GNU C Compiler. In preparation for including it also when compiling with MS Visual C, let's teach our `vcxproj` generator to handle those sort of files, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c58ce203568aa0828713c78b226f78146923217c Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Apr 7 15:29:21 2021 +0200 buildsystems: remove duplicate clause This seems to have been there since 259d87c35495 (Add scripts to generate projects for other buildsystems (MSVC vcproj, QMake), 2009-09-16), i.e. since the beginning of that file. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 7dc045a5e862440ffd8214717212fdb23caa93a7 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Mon Apr 5 14:39:33 2021 -0400 clink.pl: move default linker options for MSVC=1 builds Move the default `-ENTRY` and `-SUBSYSTEM` arguments for MSVC=1 builds from `config.mak.uname` into `clink.pl`. These args are constant for console-mode executables. Add support to `clink.pl` for generating a Win32 GUI application using the `-mwindows` argument (to match how GCC does it). This changes the `-ENTRY` and `-SUBSYSTEM` arguments accordingly. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit d0887b60c08148255907861dca28cf2ea8aef4a7 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Mon Apr 5 14:24:52 2021 -0400 clink.pl: ignore no-stack-protector arg on MSVC=1 builds Ignore the `-fno-stack-protector` compiler argument when building with MSVC. This will be used in a later commit that needs to build a Win32 GUI app. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 4f7ca433925af5d93cda8ff9054c86fc046a3904 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Mon Apr 5 14:12:14 2021 -0400 config.mak.uname: add git.rc to MSVC builds Teach MSVC=1 builds to depend on the `git.rc` file so that the resulting executables have Windows-style resources and version number information within them. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 2512ea23996e88cbf649a917101d47ef564d06cd Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Mon Apr 5 14:08:22 2021 -0400 vcbuild: add support for compiling Windows resource files Create a wrapper for the Windows Resource Compiler (RC.EXE) for use by the MSVC=1 builds. This is similar to the CL.EXE and LIB.EXE wrappers used for the MSVC=1 builds. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 8511a13b18b14c8b467e54de234f3d91a51f5bb8 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Mon Apr 5 15:27:38 2021 -0400 Makefile: clean up .ilk files when MSVC=1 Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit 59b13d1f2ed618606f250353253dde5eb29fd8e0 Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Tue Mar 30 14:25:31 2021 -0400 clink.pl: fix libexpatd.lib link error when using MSVC When building with `make MSVC=1 DEBUG=1`, link to `libexpatd.lib` rather than `libexpat.lib`. It appears that the `vcpkg` package for "libexpat" has changed and now creates `libexpatd.lib` for debug mode builds. Previously, both debug and release builds created a ".lib" with the same basename. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> commit e894f80bd0aa3fd0f80e7d0b77170c0980e2a452 Merge: cbee48b6a8 c7562c85f6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Apr 13 10:00:00 2021 +0200 Merge branch 'dscho-avoid-d-f-conflict-in-vs-master' Merge this early to resolve merge conflicts early. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 12962509601bf897d9d349c1c2bf8d543dc592c0 Author: Jens Glathe <jens.glathe@oldschoolsolutions.biz> Date: Tue Jun 2 12:12:25 2020 +0200 t0014: fix indentation For some reason, this test case was indented with 4 spaces instead of 1 horizontal tab. The other test cases in the same test script are fine. Signed-off-by: Jens Glathe <jens.glathe@oldschoolsolutions.biz> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 2c422a0c5ed38bb5492c1a2b5304e8a93e7dbc72 Author: Luke Bonanomi <lbonanomi@gmail.com> Date: Wed Jun 24 07:45:52 2020 -0400 commit: accept "scissors" with CR/LF line endings This change enhances `git commit --cleanup=scissors` by detecting scissors lines ending in either LF (UNIX-style) or CR/LF (DOS-style). Regression tests are included to specifically test for trailing comments after a CR/LF-terminated scissors line. Signed-off-by: Luke Bonanomi <lbonanomi@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c1c698ad2efb6165b16918999d5c8c1df6fb0b36 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jul 2 16:35:05 2020 +0200 git add -i: handle CR/LF line endings in the interactive input As of Git for Windows v2.27.0, there is an option to use Windows' newly-introduced Pseudo Console support. When running an interactive add operation with this support enabled, Git will receive CR/LF line endings. Therefore, let's not pretend that we are expecting Unix line endings. This fixes https://github.com/git-for-windows/git/issues/2729 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit cf11bd7f59742c706ff68a5d1db3b99e2074f017 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Sep 12 12:25:47 2015 +0200 t3701: verify that we can add *lots* of files interactively Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c7562c85f62e3550ec1ec78f64201eb8f192da99 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat May 9 19:24:23 2020 +0200 t5505/t5516: fix white-space around redirectors The convention in Git project's shell scripts is to have white-space _before_, but not _after_ the `>` (or `<`). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit fd1da52208537bbb12784e3caba72259d822e76e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Mar 4 21:55:28 2020 +0100 http: use new "best effort" strategy for Secure Channel revoke checking The native Windows HTTPS backend is based on Secure Channel which lets the caller decide how to handle revocation checking problems caused by missing information in the certificate or offline CRL distribution points. Unfortunately, cURL chose to handle these problems differently than OpenSSL by default: while OpenSSL happily ignores those problems (essentially saying "¯\_(ツ)_/¯"), the Secure Channel backend will error out instead. As a remedy, the "no revoke" mode was introduced, which turns off revocation checking altogether. This is a bit heavy-handed. We support this via the `http.schannelCheckRevoke` setting. In https://github.com/curl/curl/pull/4981, we contributed an opt-in "best effort" strategy that emulates what OpenSSL seems to do. In Git for Windows, we actually want this to be the default. This patch makes it so, introducing it as a new value for the `http.schannelCheckRevoke" setting, which now becmes a tristate: it accepts the values "false", "true" or "best-effort" (defaulting to the last one). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 7d8614419c0341e84fc89e3a191f71bfd7837c25 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat May 9 16:19:06 2020 +0200 t5505/t5516: allow running without `.git/branches/` in the templates When we commit the template directory as part of `make vcxproj`, the `branches/` directory is not actually commited, as it is empty. Two tests were not prepared for that situation. This developer tried to get rid of the support for `.git/branches/` a long time ago, but that effort did not bear fruit, so the best we can do is work around in these here tests. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 3fd22ac051fa3f864a9264e130bd3739413561c0 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat May 9 14:08:36 2020 +0200 vcxproj: unclash project directories with build outputs It already caused problems with the test suite that the directory containing `git.vcxproj` is called the same as the Git executable without its file extension: `./git` is ambiguous, it could refer both to the directory `git/` as well as to `git.exe`. Now there is one more problem: when our GitHub workflow runs on the `vs/master` branch, it fails in all but the Windows builds, as they want to write the file `git` but there is already a directory in the way. Let's just go ahead and append `.proj` to all of those directories, e.g. `git.proj/` instead of `git/`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit cbee48b6a89d056c73cddc04d9e6ff7fe0343ebd Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Aug 25 12:13:26 2020 +0200 mingw: ignore HOMEDRIVE/HOMEPATH if it points to Windows' system directory Internally, Git expects the environment variable `HOME` to be set, and to point to the current user's home directory. This environment variable is not set by default on Windows, and therefore Git tries its best to construct one if it finds `HOME` unset. There are actually two different approaches Git tries: first, it looks at `HOMEDRIVE`/`HOMEPATH` because this is widely used in corporate environments with roaming profiles, and a user generally wants their global Git settings to be in a roaming profile. Only when `HOMEDRIVE`/`HOMEPATH` is either unset or does not point to a valid location, Git will fall back to using `USERPROFILE` instead. However, starting with Windows Vista, for secondary logons and services, the environment variables `HOMEDRIVE`/`HOMEPATH` point to Windows' system directory (usually `C:\Windows\system32`). That is undesirable, and that location is usually write-protected anyway. So let's verify that the `HOMEDRIVE`/`HOMEPATH` combo does not point to Windows' system directory before using it, falling back to `USERPROFILE` if it does. This fixes git-for-windows#2709 Initial-Path-by: Ivan Pozdeev <vano@mail.mipt.ru> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0c87ece87c458bef888d75c50e3a28911e162736 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jan 31 11:49:04 2020 +0100 mingw: implement a platform-specific `strbuf_realpath()` There is a Win32 API function to resolve symbolic links, and we can use that instead of resolving them manually. Even better, this function also resolves NTFS junction points (which are somewhat similar to bind mounts). This fixes https://github.com/git-for-windows/git/issues/2481. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ddf04b993d2589788c8e007f30bea8ec1c79f8ab Author: Jeff Hostetler <jeffhost@microsoft.com> Date: Thu Jan 30 14:22:27 2020 -0500 clink.pl: fix MSVC compile script to handle libcurl-d.lib Update clink.pl to link with either libcurl.lib or libcurl-d.lib depending on whether DEBUG=1 is set. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 57d0bfaeac519a74da1c34b76af27996c0e1f891 Author: Bjoern Mueller <bjoernm@gmx.de> Date: Wed Jan 22 13:49:13 2020 +0100 mingw: fix fatal error working on mapped network drives on Windows In 1e64d18 (mingw: do resolve symlinks in `getcwd()`) a problem was introduced that causes git for Windows to stop working with certain mapped network drives (in particular, drives that are mapped to locations with long path names). Error message was "fatal: Unable to read current working directory: No such file or directory". Present change fixes this issue as discussed in https://github.com/git-for-windows/git/issues/2480 Signed-off-by: Bjoern Mueller <bjoernm@gmx.de> commit 754e01fd54f415b1628298dd6aa77a2b54ebb99e Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed Jan 1 21:07:22 2020 +0100 mingw: do resolve symlinks in `getcwd()` As pointed out in https://github.com/git-for-windows/git/issues/1676, the `git rev-parse --is-inside-work-tree` command currently fails when the current directory's path contains symbolic links. The underlying reason for this bug is that `getcwd()` is supposed to resolve symbolic links, but our `mingw_getcwd()` implementation did not. We do have all the building blocks for that, though: the `GetFinalPathByHandleW()` function will resolve symbolic links. However, we only called that function if `GetLongPathNameW()` failed, for historical reasons: the latter function was supported for a long time, but the former API function was introduced only with Windows Vista, and we used to support also Windows XP. With that support having been dropped, we are free to call the symbolic link-resolving function right away. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 1b5f91a7b059915b536985fb4dc29e89b7e1d37f Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sat Feb 1 00:31:16 2020 +0100 mingw: allow `git.exe` to be used instead of the "Git wrapper" Git for Windows wants to add `git.exe` to the users' `PATH`, without cluttering the latter with unnecessary executables such as `wish.exe`. To that end, it invented the concept of its "Git wrapper", i.e. a tiny executable located in `C:\Program Files\Git\cmd\git.exe` (originally a CMD script) whose sole purpose is to set up a couple of environment variables and then spawn the _actual_ `git.exe` (which nowadays lives in `C:\Program Files\Git\mingw64\bin\git.exe` for 64-bit, and the obvious equivalent for 32-bit installations). Currently, the following environment variables are set unless already initialized: - `MSYSTEM`, to make sure that the MSYS2 Bash and the MSYS2 Perl interpreter behave as expected, and - `PLINK_PROTOCOL`, to force PuTTY's `plink.exe` to use the SSH protocol instead of Telnet, - `PATH`, to make sure that the `bin` folder in the user's home directory, as well as the `/mingw64/bin` and the `/usr/bin` directories are included. The trick here is that the `/mingw64/bin/` and `/usr/bin/` directories are relative to the top-level installation directory of Git for Windows (which the included Bash interprets as `/`, i.e. as the MSYS pseudo root directory). Using the absence of `MSYSTEM` as a tell-tale, we can detect in `git.exe` whether these environment variables have been initialized properly. Therefore we can call `C:\Program Files\Git\mingw64\bin\git` in-place after this change, without having to call Git through the Git wrapper. Obviously, above-mentioned directories must be _prepended_ to the `PATH` variable, otherwise we risk picking up executables from unrelated Git installations. We do that by constructing the new `PATH` value from scratch, appending `$HOME/bin` (if `HOME` is set), then the MSYS2 system directories, and then appending the original `PATH`. Side note: this modification of the `PATH` variable is independent of the modification necessary to reach the executables and scripts in `/mingw64/libexec/git-core/`, i.e. the `GIT_EXEC_PATH`. That modification is still performed by Git, elsewhere, long after making the changes described above. While we _still_ cannot simply hard-link `mingw64\bin\git.exe` to `cmd` (because the former depends on a couple of `.dll` files that are only in `mingw64\bin`, i.e. calling `...\cmd\git.exe` would fail to load due to missing dependencies), at least we can now avoid that extra process of running the Git wrapper (which then has to wait for the spawned `git.exe` to finish) by calling `...\mingw64\bin\git.exe` directly, via its absolute path. Testing this is in Git's test suite tricky: we set up a "new" MSYS pseudo-root and copy the `git.exe` file into the appropriate location, then verify that `MSYSTEM` is set properly, and also that the `PATH` is modified so that scripts can be found in `$HOME/bin`, `/mingw64/bin/` and `/usr/bin/`. This addresses https://github.com/git-for-windows/git/issues/2283 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 58bb3d3db46f3c239bfbcb41a947bfa8599e3a07 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jan 31 11:44:31 2020 +0100 strbuf_realpath(): use platform-dependent API if available Some platforms (e.g. Windows) provide API functions to resolve paths much quicker. Let's offer a way to short-cut `strbuf_realpath()` on those platforms. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 697daa3dd835524d03e32d3abb7cb7d50cd92d9f Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Feb 21 13:28:58 2017 +0100 mingw: ensure valid CTYPE A change between versions 2.4.1 and 2.6.0 of the MSYS2 runtime modified how Cygwin's runtime (and hence Git for Windows' MSYS2 runtime derivative) handles locales: d16a56306d (Consolidate wctomb/mbtowc calls for POSIX-1.2008, 2016-07-20). An unintended side-effect is that "cold-calling" into the POSIX emulation will start with a locale based on the current code page, something that Git for Windows is very ill-prepared for, as it expects to be able to pass a command-line containing non-ASCII characters to the shell without having those characters munged. One symptom of this behavior: when `git clone` or `git fetch` shell out to call `git-upload-pack` with a path that contains non-ASCII characters, the shell tried to interpret the entire command-line (including command-line parameters) as executable path, which obviously must fail. This fixes https://github.com/git-for-windows/git/issues/1036 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit da23f1c4816588f05ea5db8f937ee4b8e0458336 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Jan 31 12:02:47 2020 +0100 mingw: demonstrate a `git add` issue with NTFS junctions NTFS junctions are somewhat similar in spirit to Unix bind mounts: they point to a different directory and are resolved by the filesystem driver. As such, they appear to `lstat()` as if they are directories, not as if they are symbolic links. _Any_ user can create junctions, while symbolic links can only be created by non-administrators in Developer Mode on Windows 10. Hence NTFS junctions are much more common "in the wild" than NTFS symbolic links. It was reported in https://github.com/git-for-windows/git/issues/2481 that adding files via an absolute path that traverses an NTFS junction: since 1e64d18 (mingw: do resolve symlinks in `getcwd()`), we resolve not only symbolic links but also NTFS junctions when determining the absolute path of the current directory. The same is not true for `git add <file>`, where symbolic links are resolved in `<file>`, but not NTFS junctions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 94299818f5aca02594528048d9d95576458f6a73 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Nov 14 20:09:23 2019 +0100 mingw: make sure `errno` is set correctly when socket operations fail The winsock2 library provides functions that work on different data types than file descriptors, therefore we wrap them. But that is not the only difference: they also do not set `errno` but expect the callers to enquire about errors via `WSAGetLastError()`. Let's translate that into appropriate `errno` values whenever the socket operations fail so that Git's code base does not have to change its expectations. This closes https://github.com/git-for-windows/git/issues/2404 Helped-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit da3b994484335eb1da2c6b6ec4df0488aa667bfc Author: Thomas Braun <thomas.braun@byte-physics.de> Date: Thu May 8 21:43:24 2014 +0200 transport: optionally disable side-band-64k Since commit 0c499ea60fda (send-pack: demultiplex a sideband stream with status data, 2010-02-05) the send-pack builtin uses the side-band-64k capability if advertised by the server. Unfortunately this breaks pushing over the dump git protocol if used over a network connection. The detailed reasons for this breakage are (by courtesy of Jeff Preshing, quoted from https://groups.google.com/d/msg/msysgit/at8D7J-h7mw/eaLujILGUWoJ): MinGW wraps Windows sockets in CRT file descriptors in order to mimic the functionality of POSIX sockets. This causes msvcrt.dll to treat sockets as Installable File System (IFS) handles, calling ReadFile, WriteFile, DuplicateHandle and CloseHandle on them. This approach works well in simple cases on recent versions of Windows, but does not support all usage patterns. In particular, using this approach, any attempt to read & write concurrently on the same socket (from one or more processes) will deadlock in a scenario where the read waits for a response from the server which is only invoked after the write. This is what send_pack currently attempts to do in the use_sideband codepath. The new config option `sendpack.sideband` allows to override the side-band-64k capability of the server, and thus makes the dumb git protocol work. Other transportation methods like ssh and http/https still benefit from the sideband channel, therefore the default value of `sendpack.sideband` is still true. Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de> Signed-off-by: Oliver Schneider <oliver@assarbad.net> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 16d55b569283cff5239128c3d05acd22d82d6119 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jun 24 23:45:21 2019 +0200 mingw: use mimalloc Thorough benchmarking with repacking a subset of linux.git (the commit history reachable from 93a6fefe2f ([PATCH] fix the SYSCTL=n compilation, 2007-02-28), to be precise) suggest that this allocator is on par, in multi-threaded situations maybe even better than nedmalloc: `git repack -adfq` with mimalloc, 8 threads: 31.166991900 27.576763800 28.712311000 27.373859000 27.163141900 `git repack -adfq` with nedmalloc, 8 threads: 31.915032900 27.149883100 28.244933700 27.240188800 28.580849500 In a different test using GitHub Actions build agents (probably single-threaded, a core-strength of nedmalloc)): `git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with mimalloc: 943.426 978.500 939.709 959.811 954.605 `git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with nedmalloc: 995.383 952.179 943.253 963.043 980.468 While these measurements were not executed with complete scientific rigor, as no hardware was set aside specifically for these benchmarks, it shows that mimalloc and nedmalloc perform almost the same, nedmalloc with a bit higher variance and also slightly higher average (further testing suggests that nedmalloc performs worse in multi-threaded situations than in single-threaded ones). In short: mimalloc seems to be slightly better suited for our purposes than nedmalloc. Seeing that mimalloc is developed actively, while nedmalloc ceased to see any updates in eight years, let's use mimalloc on Windows instead. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 0f676159cd1216bd4caa5d848b66ba18a88ea4da Author: Jeff Hostetler <jeffhostetler@github.com> Date: Fri May 12 15:54:11 2023 -0400 mimalloc: use "weak" random seed when statically linked Always use the internal "use_weak" random seed when initializing the "mimalloc" heap when statically linked on Windows. The imported "mimalloc" routines support several random sources to seed the heap data structures, including BCrypt.dll and RtlGenRandom. Crashes have been reported when using BCrypt.dll if it initialized during an `atexit()` handler function. Granted, such DLL initialization should not happen in an atexit handler, but yet the crashes remain. It should be noted that on Windows when statically linked, the mimalloc startup code (called by the GCC CRT to initialize static data prior to calling `main()`) always uses the internal "weak" random seed. "mimalloc" does not try to load an alternate random source until after the OS initialization has completed. Heap data is stored in `__declspec(thread)` TLS data and in theory each Git thread will have its own heap data. However, testing shows that the "mimalloc" library doesn't actually call `os_random_buf()` (to load a new random source) when creating these new per-thread heap structures. However, if an atexit handler is forced to run on a non-main thread, the "mimalloc" library *WILL* try to create a new heap and seed it with `os_random_buf()`. (The reason for this is still a mystery to this author.) The `os_random_buf()` call can cause the (previously uninitialized BCrypt.dll library) to be dynamically loaded and a call made into it. Crashes have been reported in v2.40.1.vfs.0.0 while in this call. As a workaround, the fix here forces the use of the internal "use_weak" random code for the subsequent `os_random_buf()` calls. Since we have been using that random generator for the majority of the program, it seems safe to use it for the final few mallocs in the atexit handler (of which there really shouldn't be that many. Signed-off-by: Jeff Hostetler <jeffhostetler@github.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 58b4d8c370b2d053726917b8aecb9b5d1f3d6894 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jun 24 23:43:06 2019 +0200 mimalloc: offer a build-time option to enable it By defining `USE_MIMALLOC`, Git can now be compiled with that nicely-fast and small allocator. Note that we have to disable a couple `DEVELOPER` options to build mimalloc's source code, as it makes heavy use of declarations after statements, among other things that disagree with Git's conventions. We even have to silence some GCC warnings in non-DEVELOPER mode. For example, the `-Wno-array-bounds` flag is needed because in `-O2` builds, trying to call `NtCurrentTeb()` (which `_mi_thread_id()` does on Windows) causes the bogus warning about a system header, likely related to https://sourceforge.net/p/mingw-w64/mailman/message/37674519/ and to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578: C:/git-sdk-64-minimal/mingw64/include/psdk_inc/intrin-impl.h:838:1: error: array subscript 0 is outside array bounds of 'long long unsigned int[0]' [-Werror=array-bounds] 838 | __buildreadseg(__readgsqword, unsigned __int64, "gs", "q") | ^~~~~~~~~~~~~~ Also: The `mimalloc` library uses C11-style atomics, therefore we must require that standard when compiling with GCC if we want to use `mimalloc` (instead of requiring "only" C99). This is what we do in the CMake definition already, therefore this commit does not need to touch `contrib/buildsystems/`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit e243cdaf6ba99fffe187507620f863ddb84ba287 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jun 24 23:41:27 2019 +0200 mimalloc: adjust for building inside Git We want to compile mimalloc's source code as part of Git, rather than requiring the code to be built as an external library: mimalloc uses a CMake-based build, which is not necessarily easy to integrate into the flavors of Git for Windows (which will be the main benefitting port). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit b65cfe1a47233c66b63be816cd8784d442abd876 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Jun 24 21:31:30 2019 +0200 Import the source code of mimalloc v2.1.2 This commit imports mimalloc's source code as per v2.1.2, fetched from the tag at https://github.com/microsoft/mimalloc. The .c files are from the src/ subdirectory, and the .h files from the include/ and include/mimalloc/ subdirectories. We will subsequently modify the source code to accommodate building within Git's context. Since we plan on using the `mi_*()` family of functions, we skip the C++-specific source code, some POSIX compliant functions to interact with mimalloc, and the code that wants to support auto-magic overriding of the `malloc()` function (mimalloc-new-delete.h, alloc-posix.c, mimalloc-override.h, alloc-override.c, alloc-override-osx.c, alloc-override-win.c and static.c). To appease the `check-whitespace` job of Git's Continuous Integration, this commit was washed one time via `git rebase --whitespace=fix`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit eb7fa50d90847a059ebb323c7fd5122857c61c52 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Aug 12 12:44:15 2022 +0200 git-compat-util: avoid redeclaring _DEFAULT_SOURCE We are about to vendor in `mimalloc`'s source code which we will want to include `git-compat-util.h` after defining that constant. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit fcd078ed4995f752e0618c20bd63de0336131687 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun Oct 20 22:08:58 2019 +0200 win32/pthread: avoid name clashes with winpthread The mingw-w64 GCC seems to link implicitly to libwinpthread, which does implement a pthread emulation (that is more complete than Git's). Let's keep preferring Git's. To avoid linker errors where it thinks that the `pthread_self` and the `pthread_create` symbols are defined twice, let's give our version a `win32_` prefix, just like we already do for `pthread_join()`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 32dd72d596d0ad6726911419f3b33f4a49873acd Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Feb 16 14:06:59 2015 +0100 mingw: include the Python parts in the build While Git for Windows does not _ship_ Python (in order to save on bandwidth), MSYS2 provides very fine Python interpreters that users can easily take advantage of, by using Git for Windows within its SDK. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ddace7f61dda9da662fb20aa6448efd62c8b94a6 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Dec 11 12:55:26 2018 +0100 clean: remove mount points when possible Windows' equivalent to "bind mounts", NTFS junction points, can be unlinked without affecting the mount target. This is clearly what users expect to happen when they call `git clean -dfx` in a worktree that contains NTFS junction points: the junction should be removed, and the target directory of said junction should be left alone (unless it is inside the worktree). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 844fb312127207cbed645ff77b3ac29fd9abe012 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Apr 18 12:38:30 2017 +0200 mingw: allow absolute paths without drive prefix When specifying an absolute path without a drive prefix, we convert that path internally. Let's make sure that we handle that case properly, too ;-) This fixes the command git clone https://github.com/git-for-windows/git \G4W Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit ff4998860b42e522c5ab94620ac1c788f0a7cdc7 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Mon Apr 9 13:04:35 2012 -0500 Always auto-gc after calling a fast-import transport After importing anything with fast-import, we should always let the garbage collector do its job, since the objects are written to disk inefficiently. This brings down an initial import of http://selenic.com/hg from about 230 megabytes to about 14. In the future, we may want to make this configurable on a per-remote basis, or maybe teach fast-import about it in the first place. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 93e163200733b757ddae61dcb63f7c0eb546185b Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Thu Jan 19 13:40:31 2023 +0100 gitk(Windows): avoid inadvertently calling executables in the worktree Just like CVE-2022-41953 for Git GUI, there exists a vulnerability of `gitk` where it looks for `taskkill.exe` in the current directory before searching `PATH`. Note that the many `exec git` calls are unaffected, due to an obscure quirk in Tcl's `exec` function. Typically, `git.exe` lives next to `wish.exe` (i.e. the program that is run to execute `gitk` or Git GUI) in Git for Windows, and that is the saving grace for `git.exe because `exec` searches the directory where `wish.exe` lives even before the current directory, according to https://www.tcl-lang.org/man/tcl/TclCmd/exec.htm#M24: If a directory name was not specified as part of the application name, the following directories are automatically searched in order when attempting to locate the application: The directory from which the Tcl executable was loaded. The current directory. The Windows 32-bit system directory. The Windows home directory. The directories listed in the path. The same is not true, however, for `taskkill.exe`: it lives in the Windows system directory (never mind the 32-bit, Tcl's documentation is outdated on that point, it really means `C:\Windows\system32`). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 4cfb5524f55197c0e249198c5ba427642813f0ab Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Dec 7 13:39:30 2018 +0100 clean: do not traverse mount points It seems to be not exactly rare on Windows to install NTFS junction points (the equivalent of "bind mounts" on Linux/Unix) in worktrees, e.g. to map some development tools into a subdirectory. In such a scenario, it is pretty horrible if `git clean -dfx` traverses into the mapped directory and starts to "clean up". Let's just not do that. Let's make sure before we traverse into a directory that it is not a mount point (or junction). This addresses https://github.com/git-for-windows/git/issues/607 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit c9cdc6ae671b5c92991366496d45fa672b6eeb62 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Tue Apr 18 12:09:08 2017 +0200 mingw: demonstrate a problem with certain absolute paths On Windows, there are several categories of absolute paths. One such category starts with a backslash and is implicitly relative to the drive associated with the current working directory. Example: c: git clone https://github.com/git-for-windows/git \G4W should clone into C:\G4W. There is currently a problem with that, in that mingw_mktemp() does not expect the _wmktemp() function to prefix the absolute path with the drive prefix, and as a consequence, the resulting path does not fit into the originally-passed string buffer. The symptom is a "Result too large" error. Reported by Juan Carlos Arevalo Baeza. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit 8c9b0e5484d59d698795b737e65b3ab9e53eb377 Author: Sverre Rabbelier <srabbelier@gmail.com> Date: Sun Jul 24 00:06:00 2011 +0200 remote-helper: check helper status after import/export Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com> commit e81e966ab8154320b4f0efca1ba0ee76263dc0de Author: Sverre Rabbelier <srabbelier@gmail.com> Date: Sat Aug 28 20:49:01 2010 -0500 transport-helper: add trailing -- [PT: ensure we add an additional element to the argv array] Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> commit dd04cfe0a0c900d5870f5af82e330f737494b695 Author: Sverre Rabbelier <srabbelier@gmail.com> Date: Sun Jul 24 15:54:04 2011 +0200 t9350: point out that refs are not updated correctly This happens only when the corresponding commits are not exported in the current fast-export run. This can happen either when the relevant commit is already marked, or when the commit is explicitly marked as UNINTERESTING with a negative ref by another argument. This breaks fast-export basec remote helpers. Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com> commit 3968bdb9b319f0211106c3c511e20f1050f4bbf1 Merge: 777489f9e0 8597dced10 Author: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Fri Oct 11 16:41:03 2024 +0200 Start the merging-rebase to upstream/maint This commit starts the rebase of 1bdf95713e894 to 777489f9e09c8 --- builtin/backfill.c | 2 +- builtin/gc.c | 7 ++----- repo-settings.c | 26 ++++++++++++++++++++++++++ t/t7900-maintenance.sh | 16 ---------------- t/test-lib.sh | 6 ------ 5 files changed, 29 insertions(+), 28 deletions(-) diff --git a/builtin/backfill.c b/builtin/backfill.c index ea8558b5041c44..290529c5c35ef8 100644 --- a/builtin/backfill.c +++ b/builtin/backfill.c @@ -1,4 +1,4 @@ -#define USE_THE_REPOSITORY_VARIABLE /* for core_apply_sparse_checkout */ +#define USE_THE_REPOSITORY_VARIABLE #include "builtin.h" #include "git-compat-util.h" diff --git a/builtin/gc.c b/builtin/gc.c index d52735354c9f87..6a7a2da006eeee 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1832,7 +1832,7 @@ static const char *get_extra_launchctl_strings(void) { * | Input | Output | * | *cmd | return code | *out | *is_available | * +-------+-------------+-------------------+---------------+ - * | "foo" | false | "foo" (allocated) | (unchanged) | + * | "foo" | false | NULL | (unchanged) | * +-------+-------------+-------------------+---------------+ * * GIT_TEST_MAINT_SCHEDULER set to “foo:./mock_foo.sh,bar:./mock_bar.sh” @@ -1850,11 +1850,8 @@ static int get_schedule_cmd(const char *cmd, int *is_available, char **out) struct string_list_item *item; struct string_list list = STRING_LIST_INIT_NODUP; - if (!testing) { - if (out) - *out = xstrdup(cmd); + if (!testing) return 0; - } if (is_available) *is_available = 0; diff --git a/repo-settings.c b/repo-settings.c index d8123b9323d23e..3885e81b97d170 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -3,6 +3,8 @@ #include "repo-settings.h" #include "repository.h" #include "midx.h" +#include "fsmonitor-ipc.h" +#include "fsmonitor-settings.h" static void repo_cfg_bool(struct repository *r, const char *key, int *dest, int def) @@ -46,6 +48,30 @@ void prepare_repo_settings(struct repository *r) r->settings.pack_use_bitmap_boundary_traversal = 1; r->settings.pack_use_multi_pack_reuse = 1; r->settings.pack_use_path_walk = 1; + + /* + * Force enable the builtin FSMonitor (unless the repo + * is incompatible or they've already selected it or + * the hook version). But only if they haven't + * explicitly turned it off -- so only if our config + * value is UNSET. + * + * lookup_fsmonitor_settings() and check_for_ipc() do + * not distinguish between explicitly set FALSE and + * UNSET, so we re-test for an UNSET config key here. + * + * I'm not sure I want to fix fsmonitor-settings.c to + * have more than one _DISABLED state since our usage + * here is only to support this experimental period + * (and I don't want to overload the _reason field + * because it describes incompabilities). + */ + if (manyfiles && + fsmonitor_ipc__is_supported() && + fsm_settings__get_mode(r) == FSMONITOR_MODE_DISABLED && + repo_config_get_maybe_bool(r, "core.fsmonitor", &value) > 0 && + repo_config_get_bool(r, "core.useBuiltinFSMonitor", &value)) + fsm_settings__set_ipc(r); } if (manyfiles) { r->settings.index_version = 4; diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 86bc77e73ffcb2..a66d0e089d2d3a 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -646,22 +646,6 @@ test_expect_success !MINGW 'register and unregister with regex metacharacters' ' maintenance.repo "$(pwd)/$META" ' -test_expect_success !MINGW,!DARWIN 'start without GIT_TEST_MAINT_SCHEDULER' ' - test_when_finished "rm -rf crontab.log script repo" && - mkdir script && - write_script script/crontab <<-EOF && - echo "\$*" >>"$(pwd)"/crontab.log - EOF - git init repo && - ( - cd repo && - sane_unset GIT_TEST_MAINT_SCHEDULER && - PATH="$(pwd)/../script:$PATH" git maintenance start --scheduler=crontab - ) && - test_grep -- -l crontab.log && - test_grep -- git_cron_edit_tmp crontab.log -' - test_expect_success 'start --scheduler=<scheduler>' ' test_expect_code 129 git maintenance start --scheduler=foo 2>err && test_grep "unrecognized --scheduler argument" err && diff --git a/t/test-lib.sh b/t/test-lib.sh index 6f15145341b702..9ca34a2bb82887 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1738,12 +1738,6 @@ case $uname_s in test_set_prereq GREP_STRIPS_CR test_set_prereq WINDOWS ;; -*Darwin*) - test_set_prereq POSIXPERM - test_set_prereq BSLASHPSPEC - test_set_prereq EXECKEEPSPID - test_set_prereq DARWIN - ;; *) test_set_prereq POSIXPERM test_set_prereq BSLASHPSPEC