ReposVerb: allow registering vanilla Git repos#258
Closed
derrickstolee wants to merge 23 commits into
Closed
Conversation
derrickstolee
changed the title
derrickstolee
changed the title
jrbriggs
approved these changes
Dec 17, 2019
Member
jrbriggs
left a comment
jrbriggs
left a comment
There was a problem hiding this comment.
Looking good, some minor suggestions.
derrickstolee
changed the title
Contributor
Author
|
@jrbriggs: Thanks for the feedback. I'll respond to that soon. I'm marking this PR as WIP for the remainder of the holidays. I'm going to go a bit nuts adding stuff to this PR until I feel I have a "complete" picture of the feature. Then, we can split it into smaller pieces for review when people have bandwidth. |
Signed-off-by: Derrick Stolee <stolee@gmail.com>
This also allows re-registering Scalar repos if they somehow become unregistered. (We will follow-up with a way to un-watch later.) The only required update to the maintenance steps right now is to make the fetch-commits-and-trees step be a no-op for non-Scalar repos. Eventually, we will want that to run a "git fetch" (in a way that does not update the refs/remotes space). Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
When we run functional tests inside the Scalar source code, we need our vanilla Git repos to be recognized inside that repo. The old logic would look for "src/.git" first, finding the Scalar repo before looking just for ".git". This makes the code simpler, too! Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
derrickstolee
changed the title
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
derrickstolee
added a commit
that referenced
this pull request
Jan 2, 2020
These commits are cherry-picked from #258 with slight modifications. The functionality here makes sense for repos cloned by Scalar, allowing us to focus later on the interactions with vanilla Git repos. 1. Add `ConfigStep` for running the Git config commands at clone time and as the service starts up. This already has some split for vanilla Git repos, but that code does not get exercised. Resolves #252. 2. The `PackfileMaintenanceStep` attempts to repack all but the largest pack when those packs sum to less than 2gb. 3. Some cleanup things here and there. 4. `scalar repos (add|list)` allows tracking the registered repos. Resolves some of #214.
derrickstolee
added a commit
that referenced
this pull request
Jan 8, 2020
Replaces #258. Resolves #254. If you have a "vanilla" Git repo, then run `scalar repos add` in that repo. Scalar will detect the repo root, register that root with the service, and the maintenance steps will run. The way we identify a vanilla Git repo (now) is if the object cache dir is not given by config and hence we do all operations on the local `.git` folder. This will not work with Git worktrees, but we will also not discover a `.git` _folder_ (worktrees use a `.git` _file_) so the enlistment will be invalid. * The `commit-graph` task will generate the incremental commit-graph files using the `--reachable` option. This will be a no-op if the user has not changed their refs. * The `packfile` task will write a `multi-pack-index` and do the expire/repack logic. To actually do something, the batch-size is decreased if the total pack size is smaller than 2gb. (This was done in #272.) * The `loose-objects` task will delete loose objects already in pack-files and put them into a new `from-loose` pack. This required no change. * The `fetch-commits-and-trees` task is renamed the `fetch` task. On a vanilla Git repo, it will simply call `git fetch <remote> --no-update-remote-refs +refs/head/*:refs/hidden/<remote>/*` for each `<remote>` given by `git remote`. This downloads a pack-file with the new objects not reachable form the current refs, but also does not update `refs/remotes/`. The user will see their refs update as normal, but the pack download is much smaller. * The `config` task (added by #272) runs the necessary Git config. This step is run in `scalar repos add`, but also in the background with the service. All config options become optional for vanilla Git repos, so a user can opt-out of the settings we recommend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #254 and half of #214. Resolves #252.
The
scalar reposverb allows two modes:scalar repos add: add the current enlistment to the list being watched by the Scalar service.scalar repos list: list the current set of enlistments being watched by the Scalar service.scalar repos addinside a Scalar or Git repository will register the repo with the Scalar service. The service will start running maintenance on that repo.The way we identify a vanilla Git repo (now) is if the object cache dir is not given by config and hence we do all operations on the local
.gitfolder. This will not work with Git worktrees, but we will also not discover a.gitfolder (worktrees use a.gitfile) so the enlistment will be invalid.The
commit-graphtask will generate the incremental commit-graph files using the--reachableoption. This will be a no-op if the user has not changed their refs.The
packfiletask will write amulti-pack-indexand do the expire/repack logic. To actually do something, the batch-size is decreased if the total pack size is smaller than 2gb.The
loose-objectstask will delete loose objects already in pack-files and put them into a newfrom-loosepack.The
fetch-commits-and-treestask is renamed thefetchtask. On a vanilla Git repo, it will simply callgit fetch origin --no-update-remote-refs +refs/head/*:refs/hidden/*. This downloads a pack-file with the new objects not reachable form the current refs, but also does not updaterefs/remotes/. The user will see their refs update as normal, but the pack download is much smaller.The new
configtask runs the necessary Git config. This step is run atcloneandaddtime, but also in the background with the service. Some config options are removed as required for vanilla Git repos.