Lines Matching +full:github +full:- +full:upload +full:- +full:release
2 Moving LLVM Projects to GitHub
8 We are planning to complete the transition to GitHub by Oct 21, 2019. See
9 the GitHub migration `status page <https://llvm.org/GitHubMigrationStatus.html>`_
20 hosted Subversion to GitHub. Below are the financial and technical arguments as
22 infrastructure) will continue to work with a Git-based LLVM.
29 This proposal relates only to moving the hosting of our source-code repository
30 from SVN hosted on our own servers to Git hosted on GitHub. We are not proposing
31 using GitHub's issue tracker, pull-requests, or code-review.
34 Policy, except that that a GitHub account will be required instead of SVN
35 username/password-hash.
37 Why Git, and Why GitHub?
41 ----------------
52 We should take advantage of one of the services out there (GitHub, GitLab,
57 --------
60 SVN, CVS, or anything else. Websites like GitHub have changed the landscape
66 through the Git-SVN integration.
72 * Collaborate on these branches (e.g. through your own fork of llvm on GitHub).
81 Why GitHub?
82 -----------
84 GitHub, like GitLab and BitBucket, provides free code hosting for open source
85 projects. Any of these could replace the code-hosting infrastructure that we
91 GitHub has one important advantage over GitLab and
92 BitBucket: it offers read-write **SVN** access to the repository
93 (https://github.com/blog/626-announcing-svn-support).
94 This would enable people to continue working post-migration as though our code
97 In addition, there are already multiple LLVM mirrors on GitHub, indicating that
101 -------------------------------------
103 The current SVN repository hosts all the LLVM sub-projects alongside each other.
105 all LLVM sub-projects.
113 - "The 'branch' I most care about is mainline, and losing the ability to say
116 - "I like those results sorted by time and the chronology should be obvious, but
119 - "There is still the major regression with unreadable version numbers.
121 non-trivial issue." [JSonnRevNum]_
122 - "Sequential IDs are important for LNT and llvmlab bisection tool." [MatthewsRevNum]_.
125 ``git rev-list --count <commit-hash>``. This identifier is unique only
126 within a single branch, but this means the tuple `(num, branch-name)` uniquely
129 We can thus use this revision number to ensure that e.g. `clang -v` reports a
130 user-friendly revision number (e.g. `main-12345` or `4.0-5321`), addressing
134 -------------------------------
140 Unfortunately, GitHub does not support server side hooks to enforce such a
143 GitHub offers a feature called `Status Checks`: a branch protected by
145 We could supply a pre-push hook on the client side that would run and check the
152 -------------------------
161 -------------------------
164 2. Set up a read-only version of the GitHub project, mirroring our current SVN
167 umbrella repository update (if the multirepo is selected) or the read-only
168 Git views for the sub-projects (if the monorepo is selected).
171 ------------------
173 4. Update the buildbots to pick up updates and commits from the GitHub
176 5. Update Phabricator to pick up commits from the GitHub repository.
179 7. Instruct downstream integrators to pick up commits from the GitHub
191 --------------------------
193 9. Collect developers' GitHub account information, and add them to the project.
194 10. Switch the SVN repository to read-only and allow pushes to the GitHub repository.
199 -------------------
203 point to GitHub instead.
205 GitHub Repository Description
209 ----------------
211 The LLVM git repository hosted at https://github.com/llvm/llvm-project contains all
212 sub-projects in a single source tree. It is often referred to as a monorepo and
213 mimics an export of the current SVN repository, with each sub-project having its
214 own top-level directory. Not all sub-projects are used for building toolchains.
215 For example, www/ and test-suite/ are not part of the monorepo.
217 Putting all sub-projects in a single checkout makes cross-project refactoring
220 * New sub-projects can be trivially split out for better reuse and/or layering
223 * Changing an API in LLVM and upgrading the sub-projects will always be done in
225 * Moving code across sub-project (during refactoring for instance) in a single
227 * Tooling based on `git grep` works natively across sub-projects, allowing to
230 * Having all the sources present encourages maintaining the other sub-projects
234 the sub-projects move synchronously, and a single revision number (or commit
239 Building a single sub-project
243 all sub-projects together. It is trivial to configure builds for a single
244 sub-project.
252 cmake path/to/monorepo -DLLVM_ENABLE_PROJECTS=lld
254 cmake path/to/monorepo -DLLVM_ENABLE_PROJECTS=clang
256 .. _git-svn-mirror:
259 ---------------------
261 Read-only sub-project mirrors
264 With the Monorepo, it is undecided whether the existing single-subproject
265 mirrors (e.g. https://git.llvm.org/git/compiler-rt.git) will continue to
271 GitHub supports a read/write SVN bridge for its repositories. However,
276 ------------------
279 standalone sub-project, particularly on runtimes like libcxx and compiler-rt
283 use the SVN bridge or the single-subproject Git mirrors. However, it's
286 standalone sub-project, even if they aren't contributing to it, due to the
288 sub-project Git mirrors would addresses this.
289 * Preservation of the existing read/write SVN-based workflows relies on the
290 GitHub SVN bridge, which is an extra dependency. Maintaining this locks us
291 into GitHub and could restrict future workflow changes.
296 * :ref:`Checkout/Clone a Single Project, without Commit Access <workflow-checkout-commit>`.
297 * :ref:`Checkout/Clone Multiple Projects, with Commit Access <workflow-monocheckout-multicommit>`.
298 * :ref:`Commit an API Change in LLVM and Update the Sub-projects <workflow-cross-repo-commit>`.
299 …* :ref:`Branching/Stashing/Updating for Local Development or Experiments <workflow-mono-branching>…
300 * :ref:`Bisecting <workflow-mono-bisecting>`.
306 how end-users or developers would interact with the repository for
307 various use-cases.
309 .. _workflow-checkout-commit:
312 ---------------------------------------------------
320 svn co https://user@llvm.org/svn/llvm-project/llvm/trunk llvm
321 # or using the read-only Git view, with git-svn
324 git svn init https://llvm.org/svn/llvm-project/llvm/trunk --username=<username>
325 git config svn-remote.svn.fetch :refs/remotes/origin/main
326 git svn rebase -l # -l avoids fetching ahead of the git mirror.
331 .. _workflow-multicheckout-nocommit:
339 git clone https://github.com/llvm/llvm-project.git
341 At this point you have every sub-project (llvm, clang, lld, lldb, ...), which
343 can still build only compiler-rt for instance. In this way it's not different
350 echo /compiler-rt > .git/info/sparse-checkout
351 git read-tree -mu HEAD
353 The data for all sub-projects is still in your `.git` directory, but in your
354 checkout, you only see `compiler-rt`.
355 Before you push, you'll need to fetch and rebase (`git pull --rebase`) as
358 Note that when you fetch you'll likely pull in changes to sub-projects you don't
365 git log origin/main@{1}..origin/main -- libcxx
371 Note that today with SVN or git-svn, this step is not possible since the
375 ----------------------------------------------------
384 svn co https://llvm.org/svn/llvm-project/llvm/trunk llvm -r $REVISION
386 svn co https://llvm.org/svn/llvm-project/clang/trunk clang -r $REVISION
388 svn co https://llvm.org/svn/llvm-project/libcxx/trunk libcxx -r $REVISION
390 Or using git-svn::
394 git svn init https://llvm.org/svn/llvm-project/llvm/trunk --username=<username>
395 git config svn-remote.svn.fetch :refs/remotes/origin/main
396 git svn rebase -l
397 git checkout `git svn find-rev -B r258109`
401 git svn init https://llvm.org/svn/llvm-project/clang/trunk --username=<username>
402 git config svn-remote.svn.fetch :refs/remotes/origin/main
403 git svn rebase -l
404 git checkout `git svn find-rev -B r258109`
408 git svn init https://llvm.org/svn/llvm-project/libcxx/trunk --username=<username>
409 git config svn-remote.svn.fetch :refs/remotes/origin/main
410 git svn rebase -l
411 git checkout `git svn find-rev -B r258109`
413 Note that the list would be longer with more sub-projects.
415 .. _workflow-monocheckout-multicommit:
420 The repository contains natively the source for every sub-projects at the right
423 git clone https://github.com/llvm/llvm-project.git
424 cd llvm-projects
430 .. _workflow-cross-repo-commit:
432 Commit an API Change in LLVM and Update the Sub-projects
433 --------------------------------------------------------
436 subversion users and for git-svn users. For example, few Git users try to update
447 ----------------------------------------------------------------
453 git-svn can do it. Let's look in practice what it means when dealing with
454 multiple sub-projects.
466 git checkout -b MyBranch
468 git checkout -b MyBranch
470 git checkout -b MyBranch
480 .. _workflow-mono-branching:
494 git checkout -b MyBranch
501 ---------
509 sub-projects makes it possible to script around.
511 Using the existing Git read-only view of the repositories, it is possible to use
515 .. _workflow-mono-bisecting:
524 The same example, finding which commit introduces a regression where clang-3.9
525 crashes but not clang-3.8 passes, will look like::
552 ``migrate-downstream-fork.py`` tool at
553 https://github.com/jyknight/llvm-git-migration.
556 ---------------
558 Basic instructions for ``migrate-downstream-fork.py`` are in the
563 mkdir my-monorepo
564 git -C my-monorepo init
567 git -C my-monorepo remote add upstream/monorepo https://github.com/llvm/llvm-project.git
573 clang-tools-extra
574 compiler-rt
575 debuginfo-tests
585 git -C my-monorepo remote add upstream/split/${p} https://github.com/llvm-mirror/${p}.git
586 git -C my-monorepo remote add local/split/${p} https://my.local.mirror.org/${p}.git
590 git -C my-monorepo fetch --all
592 # Run migrate-downstream-fork to rewrite local branches on top of
595 cd my-monorepo
596 migrate-downstream-fork.py \
599 --new-repo-prefix=refs/remotes/upstream/monorepo \
600 --old-repo-prefix=refs/remotes/upstream/split \
601 --source-kind=split \
602 --revmap-out=monorepo-map.txt
605 # Octopus-merge the resulting local split histories to unify them.
612 git -C my-monorepo branch --no-track local/octopus/main \
613 $(git -C my-monorepo merge-base refs/remotes/upstream/monorepo/main \
615 git -C my-monorepo checkout local/octopus/${my_local_branch}
620 git -C my-monorepo branch ${subproject_branch} \
627 git -C my-monorepo merge ${subproject_branches[@]}
631 git -C my-monorepo branch -d ${subproject_branch}
635 for ref in $(git -C my-monorepo for-each-ref --format="%(refname)" \
638 git -C my-monorepo branch upstream/${upstream_branch} ${ref}
643 U1 - U2 - U3 <- upstream/main
645 \ \ - Llld1 - Llld2 -
647 \ - Lclang1 - Lclang2-- Lmerge <- local/octopus/main
649 - Lllvm1 - Lllvm2-----
659 ``local/octopus/<local branch>`` need not use ``git-merge-base`` to
664 ---------------------
675 UM1 ---- UM2 -- UM3 -- UM4 ---- UM5 ---- UM6 ---- UM7 ---- UM8 <- main
685 The tool ``zip-downstream-fork.py`` at
686 https://github.com/greened/llvm-git-migration/tree/zip can be used to
687 convert the umbrella history into a monorepo-based history with
690 U1 - U2 - U3 <- upstream/main
692 \ -----\--------------- local/zip--.
694 - Lllvm1 - Llld1 - UM3 - Lclang1 - Lclang2 - Lllvm2 - Llld2 - Lmyproj1 <-'
709 history and so the edge ``U2 -> Lclang1`` is a visual reminder of what
712 Even so, the edge ``U3 -> Llld1`` could be problematic for future
719 running ``zip-downstream-fork.py``. If downstream merged each project
739 top-level directory in the umbrella, just as they do in the monorepo .
743 Given the above run of ``migrate-downstream-fork.py``, a recipe to
746 # Import any non-LLVM repositories the umbrella references.
747 git -C my-monorepo remote add localrepo \
751 subprojects=( clang clang-tools-extra compiler-rt debuginfo-tests libclc
753 parallel-libs polly pstl )
756 # already done for the ``migrate-downstream-fork.py`` run).
759 https://github.com/llvm-mirror/${subproject}.git
764 # already done for the ``migrate-downstream-fork.py`` run).
772 git -C my-monorepo remote add umbrella \
777 echo "myproj local/myproj" > my-monorepo/submodule-map.txt
781 cd my-monorepo
782 zip-downstream-fork.py \
784 --new-repo-prefix=refs/remotes/upstream/monorepo \
785 --old-repo-prefix=refs/remotes/upstream/split \
786 --revmap-in=monorepo-map.txt \
787 --revmap-out=zip-map.txt \
788 --subdir=local \
789 --submodule-map=submodule-map.txt \
790 --update-tags
794 git -C my-monorepo branch --no-track local/zip/main refs/remotes/umbrella/main
796 Note that if the umbrella has submodules to non-LLVM repositories,
797 ``zip-downstream-fork.py`` needs to know about them to be able to
801 With ``--update-tags`` the tool will migrate annotated tags pointing
806 commit after rewriting, or the ``--update-tags`` option may be
813 ``submodule-map.txt`` is a list of pairs, one per line. The first
826 tools/clang/tools/extra clang-tools-extra
827 projects/compiler-rt compiler-rt
828 projects/debuginfo-tests debuginfo-tests
846 Because llvm is itself the umbrella, we use --subdir to write its
849 # Import any non-LLVM repositories the umbrella references.
850 git -C my-monorepo remote add localrepo \
854 subprojects=( clang clang-tools-extra compiler-rt debuginfo-tests libclc
856 parallel-libs polly pstl )
859 # already done for the ``migrate-downstream-fork.py`` run).
862 https://github.com/llvm-mirror/${subproject}.git
867 # already done for the ``migrate-downstream-fork.py`` run).
875 # so zip-downstream-fork.py knows what it is.
876 git -C my-monorepo remote add umbrella \
881 echo "tools/clang clang" > my-monorepo/submodule-map.txt
882 echo "tools/clang/tools/extra clang-tools-extra" >> my-monorepo/submodule-map.txt
883 echo "projects/compiler-rt compiler-rt" >> my-monorepo/submodule-map.txt
884 echo "projects/debuginfo-tests debuginfo-tests" >> my-monorepo/submodule-map.txt
885 echo "projects/libclc libclc" >> my-monorepo/submodule-map.txt
886 echo "projects/libcxx libcxx" >> my-monorepo/submodule-map.txt
887 echo "projects/libcxxabi libcxxabi" >> my-monorepo/submodule-map.txt
888 echo "projects/libunwind libunwind" >> my-monorepo/submodule-map.txt
889 echo "tools/lld lld" >> my-monorepo/submodule-map.txt
890 echo "tools/lldb lldb" >> my-monorepo/submodule-map.txt
891 echo "projects/openmp openmp" >> my-monorepo/submodule-map.txt
892 echo "tools/polly polly" >> my-monorepo/submodule-map.txt
893 echo "projects/myproj local/myproj" >> my-monorepo/submodule-map.txt
897 cd my-monorepo
898 zip-downstream-fork.py \
900 --new-repo-prefix=refs/remotes/upstream/monorepo \
901 --old-repo-prefix=refs/remotes/upstream/split \
902 --revmap-in=monorepo-map.txt \
903 --revmap-out=zip-map.txt \
904 --subdir=llvm \
905 --submodule-map=submodule-map.txt \
906 --update-tags
910 git -C my-monorepo branch --no-track local/zip/main refs/remotes/umbrella/main
913 Comments at the top of ``zip-downstream-fork.py`` describe in more
917 ----------------------------
927 an umbrella setup, the ``import-downstream-repo.py`` tool at
928 https://github.com/greened/llvm-git-migration/tree/import can help with
932 git -C my-monorepo remote add myrepo https://my.local.mirror.org/myrepo.git
935 my_local_tags=( refs/tags/release
939 cd my-monorepo
940 import-downstream-repo.py \
943 --new-repo-prefix=refs/remotes/upstream/monorepo \
944 --subdir=myrepo \
945 --tag-prefix="myrepo-"
948 # Preserve release branches.
949 for ref in $(git -C my-monorepo for-each-ref --format="%(refname)" \
950 refs/remotes/myrepo/release); do
952 git -C my-monorepo branch --no-track myrepo/${branch} ${ref}
956 git -C my-monorepo branch --no-track myrepo/main refs/remotes/myrepo/main
959 git -C my-monorepo checkout local/zip/main # Or local/octopus/main
960 git -C my-monorepo merge myrepo/main
963 ``myrepo`` release branches if they were in lockstep with LLVM project
966 ``--tag-prefix`` tells ``import-downstream-repo.py`` to rename
970 the upstream monorepo had its tags rewritten with an "llvmorg-"
971 prefix, name conflicts should not be an issue. ``--tag-prefix`` can
977 R1 - R2 - R3 <- main
980 release/1
984 U1 - U2 - U3 <- upstream/main
986 \ -----\--------------- local/zip--.
988 - Lllvm1 - Llld1 - UM3 - Lclang1 - Lclang2 - Lllvm2 - Llld2 - Lmyproj1 - M1 <-'
990 R1 - R2 - R3 <-.
993 myrepo-release/1 |
995 myrepo/main--'
1008 commits based on some criteria (``zip-downstream-fork.py`` uses the
1018 ----------------------------
1021 clean up. The python tools use ``git-fast-import`` which leaves a lot
1025 git -C my-monorepo checkout main
1029 git -C my-monorepo branch -D local/zip/main || true
1030 git -C my-monorepo branch -D local/octopus/main || true
1033 git -C my-monorepo remote remove upstream/monorepo
1036 git -C my-monorepo remote remove upstream/split/${p}
1037 git -C my-monorepo remote remove local/split/${p}
1040 git -C my-monorepo remote remove localrepo
1041 git -C my-monorepo remote remove umbrella
1042 git -C my-monorepo remote remove myrepo
1044 # Add anything else here you don't need. refs/tags/release is
1051 refs/tags/release
1054 git -C my-monorepo for-each-ref --format="%(refname)" ${refs_to_clean[@]} |
1055 xargs -n1 --no-run-if-empty git -C my-monorepo update-ref -d
1057 git -C my-monorepo reflog expire --all --expire=now
1060 while ! git -C my-monorepo \
1061 -c gc.reflogExpire=0 \
1062 -c gc.reflogExpireUnreachable=0 \
1063 -c gc.rerereresolved=0 \
1064 -c gc.rerereunresolved=0 \
1065 -c gc.pruneExpire=now \
1066 gc --prune=now; do
1071 git -C my-monorepo repack -A -d -f --depth=250 --window=250
1073 git -C my-monorepo prune-packed
1074 git -C my-monorepo prune
1076 You should now have a trim monorepo. Upload it to your git server and
1082 .. [LattnerRevNum] Chris Lattner, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041739.html
1083 .. [TrickRevNum] Andrew Trick, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041721.html
1084 .. [JSonnRevNum] Joerg Sonnenberger, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041688.html
1085 .. [MatthewsRevNum] Chris Matthews, http://lists.llvm.org/pipermail/cfe-dev/2016-July/049886.html
1086 .. [statuschecks] GitHub status-checks, https://help.github.com/articles/about-required-status-chec…