Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase to v2.46.2 #688

Merged
merged 231 commits into from
Sep 25, 2024
Merged

Rebase to v2.46.2 #688

merged 231 commits into from
Sep 25, 2024

Conversation

dscho
Copy link
Member

@dscho dscho commented Sep 24, 2024

Range-diff relative to vfs-2.46.1
  • 4: 9c19694 = 1: 1b91965 sparse-index.c: fix use of index hashes in expand_index

  • 7: 3c7548c = 2: 014e13a t: remove advice from some tests

  • 8: 0ffd36c = 3: aa31f50 t1092: add test for untracked files and directories

  • 1: 0082dc7 = 4: 0a63e54 survey: stub in new experimental git-survey command

  • 2: 5a54e19 = 5: 8b66a8b survey: add command line opts to select references

  • 3: ca6d3d8 = 6: 3e43593 survey: collect the set of requested refs

  • 6: 4c9a2e2 = 7: f984f7c survey: calculate stats on refs and print results

  • 11: 62d0c8c = 8: c9e2ad6 survey: stub in treewalk of reachable commits and objects

  • 12: b7da696 = 9: 72491d7 survey: add traverse callback for commits

  • 13: e388414 = 10: 79f0a6b survey: add vector of largest objects for various scaling dimensions

  • 5: 297189c = 11: 8b9f007 t5300: confirm failure of git index-pack when non-idx suffix requested

  • 14: 7905a6e = 12: ab094c4 survey: add pathname of blob or tree to large_item_vec

  • 9: 22f5216 = 13: 2371769 index-pack: disable rev-index if index file has non .idx suffix

  • 10: d361c6d = 14: a6007ea trace2: prefetch value of GIT_TRACE2_DST_DEBUG at startup

  • 15: dd0c573 = 15: 9ea4cce survey: add commit-oid to large_item detail

  • 16: 6c6b28c = 16: 5330029 survey: add commit name-rev lookup to each large_item

  • 17: 5a6a80b = 17: 0d9f6ae survey: add --json option and setup for pretty output

  • 18: c84296d = 18: 50d2203 survey: add pretty printing of stats

  • 19: 592b09f = 19: acf0691 t8100: create test for git-survey

  • 20: 494e9c5 = 20: 440477b survey: add --no-name-rev option

  • 21: 2332438 = 21: 31ecb35 survey: started TODO list at bottom of source file

  • 22: 4b9fd55 = 22: 630e4a6 survey: expanded TODO list at the bottom of the source file

  • 23: 95bc929 = 23: 458e9bc survey: expanded TODO with more notes

  • 24: bd44fa9 = 24: cb271c6 survey: clearly note the experimental nature in the output

  • 25: d5ff2d2 = 25: e45516e reset --stdin: trim carriage return from the paths

  • 26: 0f6a2ed ! 26: 3da93cd Identify microsoft/git via a distinct version suffix

    @@ Commit message
      ## GIT-VERSION-GEN ##
     @@
      GVF=GIT-VERSION-FILE
    - DEF_VER=v2.46.1
    + DEF_VER=v2.46.2
      
     +# Identify microsoft/git via a distinct version suffix
     +DEF_VER=$DEF_VER.vfs.0.0
  • 27: 81e5a4c = 27: 17050de gvfs: ensure that the version is based on a GVFS tag

  • 28: a441c1f = 28: 4a42a9e gvfs: add a GVFS-specific header file

  • 29: 4cfd73a = 29: 6240eb8 gvfs: add the core.gvfs config setting

  • 30: d94ca7d = 30: 3b84be3 gvfs: add the feature to skip writing the index' SHA-1

  • 31: 363f882 = 31: 7d85cde gvfs: add the feature that blobs may be missing

  • 32: 4bd9426 = 32: f17a9a5 gvfs: prevent files to be deleted outside the sparse checkout

  • 33: 77d8f4a = 33: e2892a4 gvfs: optionally skip reachability checks/upload pack during fetch

  • 34: 4c1c45a = 34: 0643eb6 gvfs: ensure all filters and EOL conversions are blocked

  • 35: 695e263 = 35: 8e96976 gvfs: allow "virtualizing" objects

  • 36: 5bcdb2c = 36: 8aee64b Hydrate missing loose objects in check_and_freshen()

  • 37: b1fd1df = 37: 95454ff sha1_file: when writing objects, skip the read_object_hook

  • 38: a168919 = 38: 6d01c9d gvfs: add global command pre and post hook procs

  • 39: 8e7be74 = 39: 44d9df5 t0400: verify that the hook is called correctly from a subdirectory

  • 40: f27a1ff = 40: 0c9bf08 Pass PID of git process to hooks.

  • 41: a64248f = 41: 517213c pre-command: always respect core.hooksPath

  • 42: 7a255d4 = 42: 3248536 sparse-checkout: update files with a modify/delete conflict

  • 43: 8417502 = 43: a7cbc6b sparse-checkout: avoid writing entries with the skip-worktree bit

  • 44: 7be6acd = 44: e227ef3 Do not remove files outside the sparse-checkout

  • 45: 783668a = 45: f074ddd send-pack: do not check for sha1 file when GVFS_MISSING_OK set

  • 46: deee997 = 46: 32c6f9d cache-tree: remove use of strbuf_addf in update_one

  • 47: 335dfe0 = 47: a5121b2 gvfs: block unsupported commands when running in a GVFS repo

  • 48: 6a34107 = 48: 83b6856 worktree: allow in Scalar repositories

  • 49: fe61f9e = 49: 0b755a3 gvfs: allow overriding core.gvfs

  • 50: 234684e = 50: 001f001 BRANCHES.md: Add explanation of branches and using forks

  • 51: a833057 = 51: 4075e54 Add virtual file system settings and hook proc

  • 52: 487969d = 52: a2db7b5 virtualfilesystem: don't run the virtual file system hook if the index has been redirected

  • 53: 4ddc91e = 53: 00f813b virtualfilesystem: check if directory is included

  • 54: 9d2dedb = 54: 200f719 backwards-compatibility: support the post-indexchanged hook

  • 55: 5f83dae = 55: 4534f6d gvfs: verify that the built-in FSMonitor is disabled

  • 56: ba33c06 = 56: f431829 wt-status: add trace2 data for sparse-checkout percentage

  • 57: 9995935 = 57: 3bb4d01 wt-status: add VFS hydration percentage to normal git status output

  • 58: 5e9b797 = 58: 5815d28 status: add status serialization mechanism

  • 59: 601706a = 59: e3f91fb Teach ahead-behind and serialized status to play nicely together

  • 60: 510ca56 = 60: 7d3f3d4 status: serialize to path

  • 61: 5ef9e22 = 61: cad41b3 status: reject deserialize in V2 and conflicts

  • 62: 1076a66 = 62: aa6ef5b serialize-status: serialize global and repo-local exclude file metadata

  • 63: 27e5fa9 = 63: 10f3e9b status: deserialization wait

  • 64: c287796 = 64: 0380fe4 merge-recursive: avoid confusing logic in was_dirty()

  • 65: 757f5d7 = 65: 711bc73 merge-recursive: add some defensive coding to was_dirty()

  • 66: c295584 = 66: 2ec8717 merge-recursive: teach was_dirty() about the virtualfilesystem

  • 67: c766116 = 67: e588615 status: deserialize with -uno does not print correct hint

  • 68: d9b2bce = 68: e1cc0a8 fsmonitor: check CE_FSMONITOR_VALID in ce_uptodate

  • 69: a92b304 = 69: 4c43cb6 fsmonitor: add script for debugging and update script for tests

  • 70: 418ce15 = 70: c3e357e status: disable deserialize when verbose output requested.

  • 71: 0c260d9 = 71: 0e76b25 t7524: add test for verbose status deserialzation

  • 72: 52dc6c6 = 72: 6e000d0 deserialize-status: silently fallback if we cannot read cache file

  • 73: 863ebb5 = 73: a4af462 gvfs:trace2:data: add trace2 tracing around read_object_process

  • 74: 865bb62 = 74: fd0a957 gvfs:trace2:data: status deserialization information

  • 75: a7f1150 = 75: e96d2ad gvfs:trace2:data: status serialization

  • 76: 237b0dd = 76: 1c1310e gvfs:trace2:data: add vfs stats

  • 77: 67070df = 77: 3fcf590 trace2: refactor setting process starting time

  • 78: 2e027f7 = 78: 5f9233c trace2:gvfs:experiment: clear_ce_flags_1

  • 79: e9d77c0 = 79: 960852b trace2:gvfs:experiment: report_tracking

  • 80: a066334 = 80: f48ec62 trace2:gvfs:experiment: read_cache: annotate thread usage in read-cache

  • 81: 3cd3022 = 81: ee669c7 trace2:gvfs:experiment: read-cache: time read/write of cache-tree extension

  • 82: a6dbfe4 = 82: 01f4d47 trace2:gvfs:experiment: add region to apply_virtualfilesystem()

  • 83: 619b945 = 83: 777285c trace2:gvfs:experiment: add region around unpack_trees()

  • 84: e9ce59e = 84: 5ef245a trace2:gvfs:experiment: add region to cache_tree_fully_valid()

  • 85: 69cf26e = 85: 2db38c6 trace2:gvfs:experiment: add unpack_entry() counter to unpack_trees() and report_tracking()

  • 86: d5d0685 = 86: 72c890b trace2:gvfs:experiment: increase default event depth for unpack-tree data

  • 87: bcbacc0 = 87: 5d02713 trace2:gvfs:experiment: add data for check_updates() in unpack_trees()

  • 88: ae9ea80 = 88: b50d3ed Trace2:gvfs:experiment: capture more 'tracking' details

  • 89: 82bf4da = 89: 5b083bc credential: set trace2_child_class for credential manager children

  • 90: 3f3b8c0 = 90: e7c4a23 sub-process: do not borrow cmd pointer from caller

  • 91: bf56ae8 = 91: 636b8e5 sub-process: add subprocess_start_argv()

  • 92: 9af167d = 92: f94912f sha1-file: add function to update existing loose object cache

  • 93: b576d35 = 93: 00670b4 packfile: add install_packed_git_and_mru()

  • 94: d004438 = 94: 21bfa56 index-pack: avoid immediate object fetch while parsing packfile

  • 95: 603ac02 = 95: f9821d3 gvfs-helper: create tool to fetch objects using the GVFS Protocol

  • 96: 9eb624f = 96: b8d3637 sha1-file: create shared-cache directory if it doesn't exist

  • 97: 446fe4c = 97: f703137 gvfs-helper: better handling of network errors

  • 98: 348147c = 98: 5024531 gvfs-helper-client: properly update loose cache with fetched OID

  • 99: bcab9a9 = 99: 32cf85c gvfs-helper: V2 robust retry and throttling

  • 100: 7b6ee22 = 100: befdc17 gvfs-helper: expose gvfs/objects GET and POST semantics

  • 101: 9f0699f = 101: 7c17f5b gvfs-helper: dramatically reduce progress noise

  • 102: deefbdc = 102: 5d6188b gvfs-helper-client.h: define struct object_id

  • 103: c19200a = 103: 01d14e7 gvfs-helper: handle pack-file after single POST request

  • 104: cd7b1b8 = 104: 7b434ec test-gvfs-prococol, t5799: tests for gvfs-helper

  • 105: 6bb3ef8 = 105: 0775a95 gvfs-helper: move result-list construction into install functions

  • 106: 68080f7 = 106: 8e04cbe t5799: add support for POST to return either a loose object or packfile

  • 107: 9cf1ba6 = 107: 6579f08 t5799: cleanup wc-l and grep-c lines

  • 108: f21f2d5 = 108: bfcfecb gvfs-helper: verify loose objects after write

  • 109: f49f07d = 109: 7bae689 t7599: create corrupt blob test

  • 110: 20657aa = 110: f662ced gvfs-helper: add prefetch support

  • 111: 73409ad = 111: 93eff03 gvfs-helper: add prefetch .keep file for last packfile

  • 112: 6f6779a = 112: 7cf41b9 gvfs-helper: do one read in my_copy_fd_len_tail()

  • 113: afe9833 = 113: b7ede6b gvfs-helper: move content-type warning for prefetch packs

  • 114: a7ba4d6 = 114: 7d5b867 fetch: use gvfs-helper prefetch under config

  • 115: 514d5d3 = 115: 79daed3 gvfs-helper: better support for concurrent packfile fetches

  • 116: a0c575e = 116: 2985ef0 remote-curl: do not call fetch-pack when using gvfs-helper

  • 117: 0a766d8 = 117: c5f51bc fetch: reprepare packs before checking connectivity

  • 118: c03ec2f = 118: acded19 gvfs-helper: retry when creating temp files

  • 119: 653d77d = 119: f83c749 sparse: avoid warnings about known cURL issues in gvfs-helper.c

  • 120: 1761a30 = 120: 1d9c1e9 gvfs-helper: add --max-retries to prefetch verb

  • 121: dd843bb = 121: 15eb9dd t5799: add tests to detect corrupt pack/idx files in prefetch

  • 122: 92da944 = 122: d50498a gvfs-helper: ignore .idx files in prefetch multi-part responses

  • 123: 3234bdc = 123: 84d2766 t5799: explicitly test gvfs-helper --fallback and --no-fallback

  • 124: de72a93 = 124: d051973 gvfs-helper: don't fallback with new config

  • 125: 839d493 = 125: de58eef maintenance: care about gvfs.sharedCache config

  • 126: 3663e7e = 126: b92f2ca test-gvfs-protocol: add cache_http_503 to mayhem

  • 127: 9a6377d = 127: a206b58 unpack-trees:virtualfilesystem: Improve efficiency of clear_ce_flags

  • 128: e771d82 = 128: 95d06e2 t5799: add unit tests for new gvfs.fallback config setting

  • 129: 101945b = 129: a5c517d homebrew: add GitHub workflow to release Cask

  • 130: 77e27ca = 130: 2eb7a61 Adding winget workflows

  • 131: bbb2ad5 = 131: 30280b5 Disable the monitor-components workflow in msft-git

  • 132: 3e25db3 = 132: 59dcbd6 .github: enable windows builds on microsoft fork

  • 133: 0b19ffc = 133: a2822c6 release: create initial Windows installer build workflow

  • 134: ee7951a = 134: 4727ea2 help: special-case HOST_CPU universal

  • 135: c668ede = 135: 856ebb5 release: add Mac OSX installer build

  • 136: ee6ac30 = 136: e813360 release: build unsigned Ubuntu .deb package

  • 137: 845e9d6 = 137: 1ac5a78 release: add signing step for .deb package

  • 138: 12b573b = 138: af7ad78 release: create draft GitHub release with packages & installers

  • 139: cb20661 = 139: fcf9c7d build-git-installers: publish gpg public key

  • 140: 400e242 = 140: 2815c83 release: continue pestering until user upgrades

  • 141: 66ca679 = 141: ee9e836 Makefile: allow specifying GIT_BUILT_FROM_COMMIT

  • 144: 22f5bc6 = 142: bb44a96 update-microsoft-git: create barebones builtin

  • 146: f759434 = 143: 3dcc799 update-microsoft-git: Windows implementation

  • 148: 4897e1f = 144: bc01181 update-microsoft-git: use brew on macOS

  • 150: a7390fa = 145: 652e163 .github: update ISSUE_TEMPLATE.md for microsoft/git

  • 142: a168637 = 146: 8bc6b5e dist: archive HEAD instead of HEAD^{tree}

  • 152: 7372c14 = 147: 67fd8b4 .github: update PULL_REQUEST_TEMPLATE.md

  • 143: 64a8621 = 148: 2f4cb35 release: include GIT_BUILT_FROM_COMMIT in MacOS build

  • 147: 6bb3782 = 149: 2f54fc6 git_config_set_multivar_in_file_gently(): add a lock timeout

  • 159: 3391dff = 150: 30562db Adjust README.md for microsoft/git

  • 145: 9de58d0 = 151: ef80584 release: add installer validation

  • 149: 6816304 = 152: e6eb3f7 scalar: set the config write-lock timeout to 150ms

  • 151: 9aecedb = 153: 4d7e22a scalar: add docs from microsoft/scalar

  • 153: 4d7b52c = 154: 7c7c1ad scalar (Windows): use forward slashes as directory separators

  • 154: 8f56aa9 = 155: 781d148 scalar: add retry logic to run_git()

  • 155: d38da58 = 156: 5fea241 scalar: support the config command for backwards compatibility

  • 160: 67da18b = 157: ad6eab0 scalar: implement a minimal JSON parser

  • 161: 30bcb31 = 158: 11bb71f scalar clone: support GVFS-enabled remote repositories

  • 162: ef7cd18 = 159: 3ff0e00 test-gvfs-protocol: also serve smart protocol

  • 163: 22ff170 = 160: 93bb955 gvfs-helper: add the endpoint command

  • 164: 8d87b2f = 161: f2bbd02 dir_inside_of(): handle directory separators correctly

  • 165: 769c39d = 162: f022633 scalar: disable authentication in unattended mode

  • 166: d06dfeb = 163: 66d10f4 scalar: do initialize gvfs.sharedCache

  • 167: 05ca72c = 164: cbfe604 scalar diagnose: include shared cache info

  • 168: 14ce147 = 165: 3084606 scalar: only try GVFS protocol on https:// URLs

  • 169: ebece2f = 166: c236b82 scalar: verify that we can use a GVFS-enabled repository

  • 170: c068d34 = 167: 996498c scalar: add the cache-server command

  • 171: 195b11a = 168: f5bedc6 scalar: add a test toggle to skip accessing the vsts/info endpoint

  • 172: e6d6640 = 169: d075285 scalar: adjust documentation to the microsoft/git fork

  • 173: 6dd1010 = 170: 4f0a5e1 scalar: enable untracked cache unconditionally

  • 174: 4563e4d = 171: 1d0996e scalar: parse clone --no-fetch-commits-and-trees for backwards compatibility

  • 175: 7151fbc = 172: fc91983 scalar: make GVFS Protocol a forced choice

  • 176: 73ced48 = 173: 24947c7 scalar diagnose: accommodate Scalar's Functional Tests

  • 177: fa2be91 = 174: 42c88dc ci: run Scalar's Functional Tests

  • 178: 8dcfc11 = 175: 90d4293 scalar: upgrade to newest FSMonitor config setting

  • 156: 942d84a = 176: 37b15e9 credential: add new interactive config option

  • 157: 40ac109 = 177: 937190d maintenance: add custom config to background jobs

  • 179: 8c19a36 = 178: 3c6cd1f scalar: configure maintenance during 'reconfigure'

  • 180: 0a1116e = 179: c952c7b abspath: make strip_last_path_component() global

  • 181: 27a86ef = 180: 69204a8 scalar: .scalarCache should live above enlistment

  • 182: 3aa70c7 = 181: 0c3fe0a add/rm: allow adding sparse entries when virtual

  • 183: 0f85e97 = 182: 0019730 sparse-checkout: add config to disable deleting dirs

  • 184: 47d20a2 = 183: 801bc28 diff: ignore sparse paths in diffstat

  • 185: 32cd3a0 = 184: 660d08e repo-settings: enable sparse index by default

  • 186: 6ae1b53 = 185: 6317787 diff(sparse-index): verify with partially-sparse

  • 187: 1ebca13 = 186: 20363ab stash: expand testing for git stash -u

  • 158: 0d6a051 = 187: ded2fed sequencer: avoid progress when stderr is redirected

  • 188: 259f138 = 188: dd9c2a7 sparse: add vfs-specific precautions

  • 189: 1828ff0 = 189: dc24488 reset: fix mixed reset when using virtual filesystem

vdye and others added 30 commits September 24, 2024 14:46
In ac8acb4 (sparse-index: complete partial expansion, 2022-05-23),
'expand_index()' was updated to expand the index to a given pathspec.
However, the 'path_matches_pattern_list()' method used to facilitate this
has the side effect of initializing or updating the index hash variables
('name_hash', 'dir_hash', and 'name_hash_initialized'). This operation is
performed on 'istate', though, not 'full'; as a result, the initialized
hashes are later overwritten when copied from 'full'. To ensure the correct
hashes are in 'istate' after the index expansion, change the arg used in
'path_matches_pattern_list()' from 'istate' to 'full'.

Note that this does not fully solve the problem. If 'istate' does not have
an initialized 'name_hash' when its contents are copied to 'full',
initialized hashes will be copied back into 'istate' but
'name_hash_initialized' will be 0. Therefore, we also need to copy
'full->name_hash_initialized' back to 'istate' after the index expansion is
complete.

Signed-off-by: Victoria Dye <[email protected]>
These seem to be custom tests to microsoft/git as they break without
these changes, but these changes are not needed upstream.

Signed-off-by: Derrick Stolee <[email protected]>
Add a test verifying that sparse-checkout (with and without sparse index
enabled) treat untracked files & directories correctly when changing sparse
patterns. Specifically, it ensures that 'git sparse-checkout set'

* deletes empty directories outside the sparse cone
* does _not_ delete untracked files outside the sparse cone

Signed-off-by: Victoria Dye <[email protected]>
Start work on a new `git survey` command to scan the repository
for monorepo performance and scaling problems.  The goal is to
measure the various known "dimensions of scale" and serve as a
foundation for adding additional measurements as we learn more
about Git monorepo scaling problems.

Results will be logged to the console and to Trace2.

The initial goal is to complement the scanning and analysis performed
by the GO-based `git-sizer` (https://github.com/github/git-sizer) tool.
It is hoped that by creating a builtin command, we may be able to take
advantage of internal Git data structures and code that is not
accessible from GO to gain further insight into potential scaling
problems.

Signed-off-by: Jeff Hostetler <[email protected]>
By default we will scan all references in "refs/heads/", "refs/tags/"
and "refs/remotes/".

Add command line opts let the use ask for all refs or a subset of them
and to include a detached HEAD.

Signed-off-by: Jeff Hostetler <[email protected]>
Collect the set of requested branches, tags, and etc into a
ref_array and collect the set of requested patterns into a
strvec.

Signed-off-by: Jeff Hostetler <[email protected]>
Calculate stats on the set of refs. This includes the number of
branches, plain and annotated tags, remotes, and etc.  Calculate
the number of packed vs loose refs.  Calculate the size of the
set of refnames.

Print results on the console in JSON format.

Add Trace2 logging of the results as a data_json event.

Signed-off-by: Jeff Hostetler <[email protected]>
Add treewalk on the commits and objects reachable from the set
of refs.

This commit sets up the treewalk, but only stubs in the traverse
callbacks.  We'll actually look at the commit and object data
in the next commit.

Signed-off-by: Jeff Hostetler <[email protected]>
Add callback to handle commit objects during the treewalk. Count the
number of commits and group them by the number of parents.

Signed-off-by: Jeff Hostetler <[email protected]>
Create `struct large_item` and `struct large_item_vec` to capture the
n largest commits, trees, and blobs under various scaling dimensions,
such as size in bytes, number of commit parents, or number of entries
in a tree.

Each of these have a command line option to set them independently.

Signed-off-by: Jeff Hostetler <[email protected]>
Add test case to demonstrate that `git index-pack -o <idx-path> pack-path`
fails if <idx-path> does not end in ".idx" when `--rev-index` is
enabled.

In e37d0b8 (builtin/index-pack.c: write reverse indexes, 2021-01-25)
we learned to create `.rev` reverse indexes in addition to `.idx` index
files.  The `.rev` file pathname is constructed by replacing the suffix
on the `.idx` file.  The code assumes a hard-coded "idx" suffix.

In a8dd7e0 (config: enable `pack.writeReverseIndex` by default, 2023-04-12)
reverse indexes were enabled by default.

If the `-o <idx-path>` argument is used, the index file may have a
different suffix.  This causes an error when it tries to create the
reverse index pathname.

The test here demonstrates the failure.  (The test forces `--rev-index`
to avoid interaction with `GIT_TEST_NO_WRITE_REV_INDEX` during CI runs.)

Signed-off-by: Jeff Hostetler <[email protected]>
Include the pathname of each blob or tree in the large_item_vec
to help identify the file or directory associated with the OID
and size information.

This pathname is computed during the treewalk, so it reflects the
first observed pathname seen for that OID during the traversal over
all of the refs.  Since the file or directory could have moved
(without being modified), there may be multiple "correct" pathnames
for a particular OID.  Since we do not control the ref traversal
order, we should consider it to be a "suggested pathname" for the OID.

Signed-off-by: Jeff Hostetler <[email protected]>
Teach index-pack to silently omit the reverse index if the
index file does not have the standard ".idx" suffix.

In e37d0b8 (builtin/index-pack.c: write reverse indexes, 2021-01-25)
we learned to create `.rev` reverse indexes in addition to `.idx` index
files.  The `.rev` file pathname is constructed by replacing the suffix
on the `.idx` file.  The code assumes a hard-coded "idx" suffix.

In a8dd7e0 (config: enable `pack.writeReverseIndex` by default, 2023-04-12)
reverse indexes were enabled by default.

If the `-o <idx-path>` argument is used, the index file may have a
different suffix.  This causes an error when it tries to create the
reverse index pathname.

Since we do not know why the user requested a non-standard suffix for
the index, we cannot guess what the proper corresponding suffix should
be for the reverse index.  So we disable it.

The t5300 test has been updated to verify that we no longer error
out and that the .rev file is not created.

TODO We could warn the user that we skipped it (perhaps only if they
TODO explicitly requested `--rev-index` on the command line).
TODO
TODO Ideally, we should add an `--rev-index-path=<path>` argument
TODO or change `--rev-index` to take a pathname.
TODO
TODO I'll leave these questions for a future series.

Signed-off-by: Jeff Hostetler <[email protected]>
Prefetch the value of GIT_TRACE2_DST_DEBUG during startup and before
we try to open any Trace2 destination pathnames.

Normally, Trace2 always silently fails if a destination target
cannot be opened so that it doesn't affect the execution of a
Git command.  The command should run normally, but just not
generate any trace data.  This can make it difficult to debug
a telemetry setup, since the user doesn't know why telemetry
isn't being generated.  If the environment variable
GIT_TRACE2_DST_DEBUG is true, the Trace2 startup will print
a warning message with the `errno` to make debugging easier.

However, on Windows, looking up the env variable resets `errno`
so the warning message always ends with `...tracing: No error`
which is not very helpful.

Prefetch the env variable at startup.  This avoids the need
to update each call-site to capture `errno` in the usual
`saved-errno` variable.

Signed-off-by: Jeff Hostetler <[email protected]>
Computing `git name-rev` on each commit, tree, and blob in each
of the various large_item_vec can be very expensive if there are
too many refs, especially if the user doesn't need the result.
Lets make it optional.

The `--no-name-rev` option can save 50 calls to `git name-rev`
since we have 5 large_item_vec's and each defaults to 10 items.

Signed-off-by: Jeff Hostetler <[email protected]>
This topic branch brings in a new, experimental built-in command to
assess the dimensions of a local repository.

It is experimental and subject to change! It might grow new options,
change its output, or even be moved into `git diagnose --analyze` or
something like that.

The hope is that this command, which was inspired by `git sizer`
(https://github.com/github/git-sizer), will be helpful not only in
diagnosing issues with large repositories, but also in modeling what
shapes and sizes of repositories can be handled by Git (and as a
corollary: where Git needs to improve to be able to accommodate the
natural growth of repositories).

Signed-off-by: Johannes Schindelin <[email protected]>
While this command is definitely something we _want_, chances are that
upstreaming this will require substantial changes.

We still want to be able to experiment with this before that, to focus
on what we need out of this command: To assist with diagnosing issues
with large repositories, as well as to help monitoring the growth and
the associated painpoints of such repositories.

To that end, we are about to integrate this command into
`microsoft/git`, to get the tool into the hands of users who need it
most, with the idea to iterate in close collaboration between these
users and the developers familar with Git's internals.

However, we will definitely want to avoid letting anybody have the
impression that this command, its exact inner workings, as well as its
output format, are anywhere close to stable. To make that fact utterly
clear (and thereby protect the freedom to iterate and innovate freely
before upstreaming the command), let's mark its output as experimental
in all-caps, as the first thing we do.

Signed-off-by: Johannes Schindelin <[email protected]>
This backports the `ds/advice-sparse-index-expansion` patches into
`microsoft/git` which _just_ missed the v2.46.0 window.

Signed-off-by: Johannes Schindelin <[email protected]>
Cherry-pick rev-index fixes from v2.41.0.vfs.0.5 into v2.42.0.*
Prefetch the value of GIT_TRACE2_DST_DEBUG during startup and before we
try to open any Trace2 destination pathnames.

Normally, Trace2 always silently fails if a destination target cannot be
opened so that it doesn't affect the execution of a Git command. The
command should run normally, but just not generate any trace data. This
can make it difficult to debug a telemetry setup, since the user doesn't
know why telemetry isn't being generated. If the environment variable
GIT_TRACE2_DST_DEBUG is true, the Trace2 startup will print a warning
message with the `errno` to make debugging easier.

However, on Windows, looking up the env variable resets `errno` so the
warning message always ends with `...tracing: No error` which is not
very helpful.

Prefetch the env variable at startup. This avoids the need to update
each call-site to capture `errno` in the usual `saved-errno` variable.
…sitories (#667)

This command is inspired by [`git
sizer`](https://github.com/github/git-sizer), having the advantage of
being much closer to the internals of Git.

The intention is to provide a built-in command that can be used to
analyze large repositories for performance and scaling problems, for
growth over time, and to correlate with other measurements (in
particular with Trace2 data collected e.g. via
https://github.com/git-ecosystem/trace2receiver/).
dscho and others added 18 commits September 24, 2024 14:47
Most of these were done in private before microsoft/git. However,
the following pull requests modified the core feature:

	#85
	#89
	#91
	#98
	#243
	#263

Signed-off-by: Derrick Stolee <[email protected]>
Add virtual file system settings and hook proc.  On index load,
clear/set the skip worktree bits based on the virtual file system data.
Use virtual file system data to update skip-worktree bit in
unpack-trees. Use virtual file system data to exclude files and folders
not explicitly requested.

The hook was first contributed in private, but was extended via the
following pull requests:

	#15
	#27
	#33
	#70

Signed-off-by: Ben Peart <[email protected]>
Signed-off-by: Derrick Stolee <[email protected]>
Includes these pull requests:

	#1
	#6
	#10
	#11
	#157
	#212
	#260
	#270

Signed-off-by: Derrick Stolee <[email protected]>
Includes gvfs-specific commits from these pull requests:

	#158
	#159
	#160
	#164

Signed-off-by: Derrick Stolee <[email protected]>
Includes commits from these pull requests:

	#191
	#205
	#206
	#207
	#208
	#215
	#220
	#221

Signed-off-by: Derrick Stolee <[email protected]>
Includes these pull requests:

 #227
 #228
 #229
 #231
 #240

Signed-off-by: Derrick Stolee <[email protected]>
…response (#571)

Teach `gvfs-helper` to ignore the optional `.idx` files that may be
included in a `prefetch` response and always use `git index-pack` to
create them from the `.pack` files received in the data stream.

This is a little wasteful in terms of client-side compute and of the
network bandwidth, but allows us to use the full packfile verification
code contained within `git index-pack` to ensure that the received
packfiles are valid.
By default, GVFS Protocol-enabled Scalar clones will fall back to the
origin server if there is a network issue with the cache servers.
However (and especially for the prefetch endpoint) this may be a very
expensive operation for the origin server, leading to the user being
throttled. This shows up later in cases such as 'git push' or other web
operations.

To avoid this, create a new config option, 'gvfs.fallback', which
defaults to true. When set to 'false', pass '--no-fallback' from the
gvfs-helper client to the child gvfs-helper server process.

This will allow users who have hit this problem to avoid it in the
future. In case this becomes a more widespread problem, engineering
systems can enable the config option more broadly.

Enabling the config will of course lead to immediate failures for users,
but at least that will help diagnose the problem when it occurs instead
of later when the throttling shows up and the server load has already
passed, damage done.

 This change only applies to interactions with Azure DevOps and the
GVFS Protocol.

---

* [x] This change only applies to interactions with Azure DevOps and the
      GVFS Protocol.
This PR updates our `vfs-2.29.0` branch's version of `git maintenance` to match the latest in upstream. Unfortunately, not all of these commits made it to the `2.30` release candidate, but there are more commits from the series making it in. They will cause a conflict in the `vfs-2.30.0` rebase, so merge them in here. This also includes the `fixup!` reverts of the earlier versions.

Finally, I also noticed that we started depending on `git maintenance start` in Scalar for macOS, but we never checked that this worked with the shared object cache. It doesn't! 😨  The very tip commit of this PR includes logic to make `git maintenance run` care about `gvfs.sharedCache`. Functional test updates in Scalar will follow.
…ciency of clear_ce_flags

When the virtualfilesystem is enabled the previous implementation of
clear_ce_flags would iterate all of the cache entries and query whether
each one is in the virtual filesystem to determine whether to clear one
of the SKIP_WORKTREE bits. For each cache entry, we would do a hash
lookup for each parent directory in the is_included_in_virtualfilesystem
function.

The former approach is slow for a typical Windows OS enlistment with
3 million files where only a small percentage is in the virtual
filesystem. The cost is
O(n_index_entries * n_chars_per_path * n_parent_directories_per_path).

In this change, we use the same approach as apply_virtualfilesystem,
which iterates the set of entries in the virtualfilesystem and searches
in the cache for the corresponding entries in order to clear their
flags. This approach has a cost of
O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)).

The apply_virtualfilesystem code was refactored a bit and modified to
clear flags for all names that 'alias' a given virtual filesystem name
when ignore_case is set.

n_virtual_filesystem_entries is typically much less than
n_index_entries, in which case the new approach is much faster. We wind
up building the name hash for the index, but this occurs quickly thanks
to the multi-threading.
Adding a few workflows to publish releases! 🥳
This disables the `monitor-components` workflow in msft-git, and
re-enables the `win+VS` tests in this repository, too.
Implement workflow to create GitHub release with attached `git` installers
Fixes for MacOS release build & build options
This adds a new builtin, `git update-microsoft-git`, that executes the platform-specific upgrade steps to get the latest version of `microsoft-git`.

On Windows, this means running `git update-git-for-windows` which was updated to use the `microsoft/git` releases page, when appropriate. See #321 for details.

On macOS, this means running a sequence of `brew` commands. These are adapted from the `UpgradeVerb` in `microsoft/scalar`, with an important simplification: we don't need to differentiate between the `scalar` and `scalar-azrepos` cask.
…x-built-in-fsmonitor

Fix the built-in FSMonitor, and run Scalar's Functional Tests as part of the automated builds
This is random stuff that probably all got upstream in the meantime.
@dscho dscho self-assigned this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.