Lines Matching +full:llvm +full:- +full:project
2 Moving LLVM Projects to GitHub
9 the GitHub migration `status page <https://llvm.org/GitHubMigrationStatus.html>`_
22 infrastructure) will continue to work with a Git-based LLVM.
29 This proposal relates only to moving the hosting of our source-code repository
31 using GitHub's issue tracker, pull-requests, or code-review.
35 username/password-hash.
41 ----------------
44 and Git mirror on a voluntary basis. The LLVM Foundation sponsors the server and
57 --------
64 Git is also the version control many LLVM developers use. Despite the
66 through the Git-SVN integration.
72 * Collaborate on these branches (e.g. through your own fork of llvm on GitHub).
82 -----------
85 projects. Any of these could replace the code-hosting infrastructure that we
92 BitBucket: it offers read-write **SVN** access to the repository
93 (https://github.com/blog/626-announcing-svn-support).
94 This would enable people to continue working post-migration as though our code
97 In addition, there are already multiple LLVM mirrors on GitHub, indicating that
101 -------------------------------------
103 The current SVN repository hosts all the LLVM sub-projects alongside each other.
105 all LLVM sub-projects.
113 - "The 'branch' I most care about is mainline, and losing the ability to say
116 - "I like those results sorted by time and the chronology should be obvious, but
119 - "There is still the major regression with unreadable version numbers.
121 non-trivial issue." [JSonnRevNum]_
122 - "Sequential IDs are important for LNT and llvmlab bisection tool." [MatthewsRevNum]_.
125 ``git rev-list --count <commit-hash>``. This identifier is unique only
126 within a single branch, but this means the tuple `(num, branch-name)` uniquely
129 We can thus use this revision number to ensure that e.g. `clang -v` reports a
130 user-friendly revision number (e.g. `main-12345` or `4.0-5321`), addressing
134 -------------------------------
145 We could supply a pre-push hook on the client side that would run and check the
152 -------------------------
161 -------------------------
164 2. Set up a read-only version of the GitHub project, mirroring our current SVN
167 umbrella repository update (if the multirepo is selected) or the read-only
168 Git views for the sub-projects (if the monorepo is selected).
171 ------------------
181 8. Review and prepare an update for the LLVM documentation.
191 --------------------------
193 9. Collect developers' GitHub account information, and add them to the project.
194 10. Switch the SVN repository to read-only and allow pushes to the GitHub repository.
199 -------------------
202 14. Update links on the LLVM website pointing to viewvc/klaus/phab etc. to
209 ----------------
211 The LLVM git repository hosted at https://github.com/llvm/llvm-project contains all
212 sub-projects in a single source tree. It is often referred to as a monorepo and
213 mimics an export of the current SVN repository, with each sub-project having its
214 own top-level directory. Not all sub-projects are used for building toolchains.
215 For example, www/ and test-suite/ are not part of the monorepo.
217 Putting all sub-projects in a single checkout makes cross-project refactoring
220 * New sub-projects can be trivially split out for better reuse and/or layering
222 dependency on LLVM).
223 * Changing an API in LLVM and upgrading the sub-projects will always be done in
225 * Moving code across sub-project (during refactoring for instance) in a single
227 * Tooling based on `git grep` works natively across sub-projects, allowing to
230 * Having all the sources present encourages maintaining the other sub-projects
234 the sub-projects move synchronously, and a single revision number (or commit
239 Building a single sub-project
243 all sub-projects together. It is trivial to configure builds for a single
244 sub-project.
249 # Configure only LLVM (default)
251 # Configure LLVM and lld
252 cmake path/to/monorepo -DLLVM_ENABLE_PROJECTS=lld
253 # Configure LLVM and clang
254 cmake path/to/monorepo -DLLVM_ENABLE_PROJECTS=clang
256 .. _git-svn-mirror:
259 ---------------------
261 Read-only sub-project mirrors
264 With the Monorepo, it is undecided whether the existing single-subproject
265 mirrors (e.g. https://git.llvm.org/git/compiler-rt.git) will continue to
276 ------------------
279 standalone sub-project, particularly on runtimes like libcxx and compiler-rt
280 that don't rely on LLVM; currently, a fresh clone of libcxx is only 15MB (vs.
281 1GB for the monorepo), and the commit rate of LLVM may cause more frequent
283 use the SVN bridge or the single-subproject Git mirrors. However, it's
286 standalone sub-project, even if they aren't contributing to it, due to the
288 sub-project Git mirrors would addresses this.
289 * Preservation of the existing read/write SVN-based workflows relies on the
296 * :ref:`Checkout/Clone a Single Project, without Commit Access <workflow-checkout-commit>`.
297 * :ref:`Checkout/Clone Multiple Projects, with Commit Access <workflow-monocheckout-multicommit>`.
298 * :ref:`Commit an API Change in LLVM and Update the Sub-projects <workflow-cross-repo-commit>`.
299 …* :ref:`Branching/Stashing/Updating for Local Development or Experiments <workflow-mono-branching>…
300 * :ref:`Bisecting <workflow-mono-bisecting>`.
306 how end-users or developers would interact with the repository for
307 various use-cases.
309 .. _workflow-checkout-commit:
311 Checkout/Clone a Single Project, with Commit Access
312 ---------------------------------------------------
320 svn co https://user@llvm.org/svn/llvm-project/llvm/trunk llvm
321 # or using the read-only Git view, with git-svn
322 git clone https://llvm.org/git/llvm.git
323 cd llvm
324 git svn init https://llvm.org/svn/llvm-project/llvm/trunk --username=<username>
325 git config svn-remote.svn.fetch :refs/remotes/origin/main
326 git svn rebase -l # -l avoids fetching ahead of the git mirror.
331 .. _workflow-multicheckout-nocommit:
339 git clone https://github.com/llvm/llvm-project.git
341 At this point you have every sub-project (llvm, clang, lld, lldb, ...), which
343 can still build only compiler-rt for instance. In this way it's not different
350 echo /compiler-rt > .git/info/sparse-checkout
351 git read-tree -mu HEAD
353 The data for all sub-projects is still in your `.git` directory, but in your
354 checkout, you only see `compiler-rt`.
355 Before you push, you'll need to fetch and rebase (`git pull --rebase`) as
358 Note that when you fetch you'll likely pull in changes to sub-projects you don't
365 git log origin/main@{1}..origin/main -- libcxx
371 Note that today with SVN or git-svn, this step is not possible since the
375 ----------------------------------------------------
377 Let's look how to assemble llvm+clang+libcxx at a given revision.
384 svn co https://llvm.org/svn/llvm-project/llvm/trunk llvm -r $REVISION
385 cd llvm/tools
386 svn co https://llvm.org/svn/llvm-project/clang/trunk clang -r $REVISION
388 svn co https://llvm.org/svn/llvm-project/libcxx/trunk libcxx -r $REVISION
390 Or using git-svn::
392 git clone https://llvm.org/git/llvm.git
393 cd llvm/
394 git svn init https://llvm.org/svn/llvm-project/llvm/trunk --username=<username>
395 git config svn-remote.svn.fetch :refs/remotes/origin/main
396 git svn rebase -l
397 git checkout `git svn find-rev -B r258109`
399 git clone https://llvm.org/git/clang.git
401 git svn init https://llvm.org/svn/llvm-project/clang/trunk --username=<username>
402 git config svn-remote.svn.fetch :refs/remotes/origin/main
403 git svn rebase -l
404 git checkout `git svn find-rev -B r258109`
406 git clone https://llvm.org/git/libcxx.git
408 git svn init https://llvm.org/svn/llvm-project/libcxx/trunk --username=<username>
409 git config svn-remote.svn.fetch :refs/remotes/origin/main
410 git svn rebase -l
411 git checkout `git svn find-rev -B r258109`
413 Note that the list would be longer with more sub-projects.
415 .. _workflow-monocheckout-multicommit:
420 The repository contains natively the source for every sub-projects at the right
423 git clone https://github.com/llvm/llvm-project.git
424 cd llvm-projects
427 As before, at this point clang, llvm, and libcxx are stored in directories
430 .. _workflow-cross-repo-commit:
432 Commit an API Change in LLVM and Update the Sub-projects
433 --------------------------------------------------------
436 subversion users and for git-svn users. For example, few Git users try to update
437 LLD or Clang in the same commit as they change an LLVM API.
447 ----------------------------------------------------------------
453 git-svn can do it. Let's look in practice what it means when dealing with
454 multiple sub-projects.
466 git checkout -b MyBranch
468 git checkout -b MyBranch
470 git checkout -b MyBranch
480 .. _workflow-mono-branching:
494 git checkout -b MyBranch
501 ---------
509 sub-projects makes it possible to script around.
511 Using the existing Git read-only view of the repositories, it is possible to use
512 the native Git bisection script over the llvm repository, and use some scripting
513 to synchronize the clang repository to match the llvm revision.
515 .. _workflow-mono-bisecting:
524 The same example, finding which commit introduces a regression where clang-3.9
525 crashes but not clang-3.8 passes, will look like::
541 less like to encounter a build failure where a commit change an API in LLVM and
547 Suppose you have been developing against the existing LLVM git
552 ``migrate-downstream-fork.py`` tool at
553 https://github.com/jyknight/llvm-git-migration.
556 ---------------
558 Basic instructions for ``migrate-downstream-fork.py`` are in the
563 mkdir my-monorepo
564 git -C my-monorepo init
567 git -C my-monorepo remote add upstream/monorepo https://github.com/llvm/llvm-project.git
573 clang-tools-extra
574 compiler-rt
575 debuginfo-tests
581 llvm
585 git -C my-monorepo remote add upstream/split/${p} https://github.com/llvm-mirror/${p}.git
586 git -C my-monorepo remote add local/split/${p} https://my.local.mirror.org/${p}.git
590 git -C my-monorepo fetch --all
592 # Run migrate-downstream-fork to rewrite local branches on top of
595 cd my-monorepo
596 migrate-downstream-fork.py \
599 --new-repo-prefix=refs/remotes/upstream/monorepo \
600 --old-repo-prefix=refs/remotes/upstream/split \
601 --source-kind=split \
602 --revmap-out=monorepo-map.txt
605 # Octopus-merge the resulting local split histories to unify them.
612 git -C my-monorepo branch --no-track local/octopus/main \
613 $(git -C my-monorepo merge-base refs/remotes/upstream/monorepo/main \
614 refs/remotes/local/split/llvm/${my_local_branch})
615 git -C my-monorepo checkout local/octopus/${my_local_branch}
620 git -C my-monorepo branch ${subproject_branch} \
622 if [[ "${p}" != "llvm" ]]; then
627 git -C my-monorepo merge ${subproject_branches[@]}
631 git -C my-monorepo branch -d ${subproject_branch}
635 for ref in $(git -C my-monorepo for-each-ref --format="%(refname)" \
638 git -C my-monorepo branch upstream/${upstream_branch} ${ref}
643 U1 - U2 - U3 <- upstream/main
645 \ \ - Llld1 - Llld2 -
647 \ - Lclang1 - Lclang2-- Lmerge <- local/octopus/main
649 - Lllvm1 - Lllvm2-----
659 ``local/octopus/<local branch>`` need not use ``git-merge-base`` to
661 appropriate component branch (say, ``llvm/local_release_X``).
664 ---------------------
671 with some kind of "umbrella" project that imports the project git
675 UM1 ---- UM2 -- UM3 -- UM4 ---- UM5 ---- UM6 ---- UM7 ---- UM8 <- main
680 commit in the project mirror. ``UM3`` in this case is a commit of
682 perhaps a ``README`` or project build script update. Commit ``UM8``
683 updates a submodule of local project ``myproj``.
685 The tool ``zip-downstream-fork.py`` at
686 https://github.com/greened/llvm-git-migration/tree/zip can be used to
687 convert the umbrella history into a monorepo-based history with
690 U1 - U2 - U3 <- upstream/main
692 \ -----\--------------- local/zip--.
694 - Lllvm1 - Llld1 - UM3 - Lclang1 - Lclang2 - Lllvm2 - Llld2 - Lmyproj1 <-'
709 history and so the edge ``U2 -> Lclang1`` is a visual reminder of what
712 Even so, the edge ``U3 -> Llld1`` could be problematic for future
719 running ``zip-downstream-fork.py``. If downstream merged each project
727 ``U3``. The trees for llvm and lld should correctly represent commits
739 top-level directory in the umbrella, just as they do in the monorepo .
743 Given the above run of ``migrate-downstream-fork.py``, a recipe to
746 # Import any non-LLVM repositories the umbrella references.
747 git -C my-monorepo remote add localrepo \
751 subprojects=( clang clang-tools-extra compiler-rt debuginfo-tests libclc
752 libcxx libcxxabi libunwind lld lldb llgo llvm openmp
753 parallel-libs polly pstl )
756 # already done for the ``migrate-downstream-fork.py`` run).
757 for project in ${subprojects[@]}; do
758 git remote add upstream/split/${project} \
759 https://github.com/llvm-mirror/${subproject}.git
760 git fetch umbrella/split/${project}
764 # already done for the ``migrate-downstream-fork.py`` run).
765 for project in ${subprojects[@]}; do
766 git remote add local/split/${project} \
768 git fetch local/split/${project}
772 git -C my-monorepo remote add umbrella \
777 echo "myproj local/myproj" > my-monorepo/submodule-map.txt
781 cd my-monorepo
782 zip-downstream-fork.py \
784 --new-repo-prefix=refs/remotes/upstream/monorepo \
785 --old-repo-prefix=refs/remotes/upstream/split \
786 --revmap-in=monorepo-map.txt \
787 --revmap-out=zip-map.txt \
788 --subdir=local \
789 --submodule-map=submodule-map.txt \
790 --update-tags
794 git -C my-monorepo branch --no-track local/zip/main refs/remotes/umbrella/main
796 Note that if the umbrella has submodules to non-LLVM repositories,
797 ``zip-downstream-fork.py`` needs to know about them to be able to
801 With ``--update-tags`` the tool will migrate annotated tags pointing
806 commit after rewriting, or the ``--update-tags`` option may be
811 The tool handles nested submodules (e.g. llvm is a submodule in
812 umbrella and clang is a submodule in llvm). The file
813 ``submodule-map.txt`` is a list of pairs, one per line. The first
818 Let's say your umbrella repository is actually the llvm repository and
826 tools/clang/tools/extra clang-tools-extra
827 projects/compiler-rt compiler-rt
828 projects/debuginfo-tests debuginfo-tests
842 map entries for all of the projects in your umbrella (except llvm).
843 Otherwise trees from submodule updates will appear underneath llvm in
846 Because llvm is itself the umbrella, we use --subdir to write its
847 content into ``llvm`` in the zippped history::
849 # Import any non-LLVM repositories the umbrella references.
850 git -C my-monorepo remote add localrepo \
854 subprojects=( clang clang-tools-extra compiler-rt debuginfo-tests libclc
855 libcxx libcxxabi libunwind lld lldb llgo llvm openmp
856 parallel-libs polly pstl )
859 # already done for the ``migrate-downstream-fork.py`` run).
860 for project in ${subprojects[@]}; do
861 git remote add upstream/split/${project} \
862 https://github.com/llvm-mirror/${subproject}.git
863 git fetch umbrella/split/${project}
867 # already done for the ``migrate-downstream-fork.py`` run).
868 for project in ${subprojects[@]}; do
869 git remote add local/split/${project} \
871 git fetch local/split/${project}
875 # so zip-downstream-fork.py knows what it is.
876 git -C my-monorepo remote add umbrella \
877 https://my.local.mirror.org/llvm.git
881 echo "tools/clang clang" > my-monorepo/submodule-map.txt
882 echo "tools/clang/tools/extra clang-tools-extra" >> my-monorepo/submodule-map.txt
883 echo "projects/compiler-rt compiler-rt" >> my-monorepo/submodule-map.txt
884 echo "projects/debuginfo-tests debuginfo-tests" >> my-monorepo/submodule-map.txt
885 echo "projects/libclc libclc" >> my-monorepo/submodule-map.txt
886 echo "projects/libcxx libcxx" >> my-monorepo/submodule-map.txt
887 echo "projects/libcxxabi libcxxabi" >> my-monorepo/submodule-map.txt
888 echo "projects/libunwind libunwind" >> my-monorepo/submodule-map.txt
889 echo "tools/lld lld" >> my-monorepo/submodule-map.txt
890 echo "tools/lldb lldb" >> my-monorepo/submodule-map.txt
891 echo "projects/openmp openmp" >> my-monorepo/submodule-map.txt
892 echo "tools/polly polly" >> my-monorepo/submodule-map.txt
893 echo "projects/myproj local/myproj" >> my-monorepo/submodule-map.txt
897 cd my-monorepo
898 zip-downstream-fork.py \
900 --new-repo-prefix=refs/remotes/upstream/monorepo \
901 --old-repo-prefix=refs/remotes/upstream/split \
902 --revmap-in=monorepo-map.txt \
903 --revmap-out=zip-map.txt \
904 --subdir=llvm \
905 --submodule-map=submodule-map.txt \
906 --update-tags
910 git -C my-monorepo branch --no-track local/zip/main refs/remotes/umbrella/main
913 Comments at the top of ``zip-downstream-fork.py`` describe in more
917 ----------------------------
919 You may have additional repositories that integrate with the LLVM
921 repositories are tightly coupled with LLVM, it may make sense to
927 an umbrella setup, the ``import-downstream-repo.py`` tool at
928 https://github.com/greened/llvm-git-migration/tree/import can help with
932 git -C my-monorepo remote add myrepo https://my.local.mirror.org/myrepo.git
939 cd my-monorepo
940 import-downstream-repo.py \
943 --new-repo-prefix=refs/remotes/upstream/monorepo \
944 --subdir=myrepo \
945 --tag-prefix="myrepo-"
949 for ref in $(git -C my-monorepo for-each-ref --format="%(refname)" \
952 git -C my-monorepo branch --no-track myrepo/${branch} ${ref}
956 git -C my-monorepo branch --no-track myrepo/main refs/remotes/myrepo/main
959 git -C my-monorepo checkout local/zip/main # Or local/octopus/main
960 git -C my-monorepo merge myrepo/main
963 ``myrepo`` release branches if they were in lockstep with LLVM project
966 ``--tag-prefix`` tells ``import-downstream-repo.py`` to rename
970 the upstream monorepo had its tags rewritten with an "llvmorg-"
971 prefix, name conflicts should not be an issue. ``--tag-prefix`` can
977 R1 - R2 - R3 <- main
984 U1 - U2 - U3 <- upstream/main
986 \ -----\--------------- local/zip--.
988 - Lllvm1 - Llld1 - UM3 - Lclang1 - Lclang2 - Lllvm2 - Llld2 - Lmyproj1 - M1 <-'
990 R1 - R2 - R3 <-.
993 myrepo-release/1 |
995 myrepo/main--'
999 interleaved with commits on local project branches (for example,
1008 commits based on some criteria (``zip-downstream-fork.py`` uses the
1018 ----------------------------
1021 clean up. The python tools use ``git-fast-import`` which leaves a lot
1025 git -C my-monorepo checkout main
1029 git -C my-monorepo branch -D local/zip/main || true
1030 git -C my-monorepo branch -D local/octopus/main || true
1033 git -C my-monorepo remote remove upstream/monorepo
1036 git -C my-monorepo remote remove upstream/split/${p}
1037 git -C my-monorepo remote remove local/split/${p}
1040 git -C my-monorepo remote remove localrepo
1041 git -C my-monorepo remote remove umbrella
1042 git -C my-monorepo remote remove myrepo
1054 git -C my-monorepo for-each-ref --format="%(refname)" ${refs_to_clean[@]} |
1055 xargs -n1 --no-run-if-empty git -C my-monorepo update-ref -d
1057 git -C my-monorepo reflog expire --all --expire=now
1060 while ! git -C my-monorepo \
1061 -c gc.reflogExpire=0 \
1062 -c gc.reflogExpireUnreachable=0 \
1063 -c gc.rerereresolved=0 \
1064 -c gc.rerereunresolved=0 \
1065 -c gc.pruneExpire=now \
1066 gc --prune=now; do
1071 git -C my-monorepo repack -A -d -f --depth=250 --window=250
1073 git -C my-monorepo prune-packed
1074 git -C my-monorepo prune
1082 .. [LattnerRevNum] Chris Lattner, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041739.html
1083 .. [TrickRevNum] Andrew Trick, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041721.html
1084 .. [JSonnRevNum] Joerg Sonnenberger, http://lists.llvm.org/pipermail/llvm-dev/2011-July/041688.html
1085 .. [MatthewsRevNum] Chris Matthews, http://lists.llvm.org/pipermail/cfe-dev/2016-July/049886.html
1086 .. [statuschecks] GitHub status-checks, https://help.github.com/articles/about-required-status-chec…