llvm-snapshots

Everything to build LLVM snapshots for Fedora/RHEL/CentOS Stream

https://github.com/fedora-llvm-team/llvm-snapshots

Keywords

Repository

Everything to build LLVM snapshots for Fedora/RHEL/CentOS Stream

README.adoc

:toc:
:toc-placement: preamble
:sectnums:
:experimental:
:showtitle:
:homepage: https://github.com/fedora-llvm-team/llvm-snapshots

image:https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/fedora-copr-build.yml/badge.svg[link="https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/fedora-copr-build.yml"]
image:https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/check-snapshots.yml/badge.svg[link="https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/check-snapshots.yml"]
image:https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/python-format-and-tests.yml/badge.svg[link="https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/python-format-and-tests.yml"]
image:https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/update-build-time-diagrams.yml/badge.svg[link="https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/update-build-time-diagrams.yml"]
image:https://img.shields.io/badge/code%20style-black-000000.svg[link="https://github.com/psf/black"]
link:https://fedora-llvm-team.github.io/llvm-snapshots/index.html[Build time Diagrams]
image:https://coveralls.io/repos/github/fedora-llvm-team/llvm-snapshots/badge.svg[link="https://coveralls.io/github/fedora-llvm-team/llvm-snapshots"]

== Maintaining the LLVM Snapshots for Fedora
Konrad Kleine 

This document is supposed to give you an idea of what goes into maintaining the
LLVM snapshots for Fedora. Don't feel like you have read this document from top
to bottom but instead consider it a reference. There is an <> section at the botton that probably contains many
answers. Feel free to add your own there.

This is home for the generation of daily

* link:https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/monitor/[Fedora LLVM snapshot RPMs]
** See link:https://github.com/fedora-llvm-team/llvm-snapshots/actions/workflows/fedora-copr-build.yml[fedora-copr-build] workflow

=== Getting the latest snapshot

We need a bit of post-configuration after enabling the copr repository for this project.

----
$ dnf -y install jq envsubst
$ dnf -y install --skip-broken 'dnf-command(copr)' 'dnf5-command(copr)'
$ dnf -y copr enable @fedora-llvm-team/llvm-snapshots
$ repo_file=$(dnf repoinfo --json *llvm-snapshots* | jq -r ".[0].repo_file_path")
$ distname=$(rpm --eval "%{?fedora:fedora}%{?rhel:rhel}") envsubst '$distname' < $repo_file > /tmp/new_repo_file
$ cat /tmp/new_repo_file > $repo_file
----

Then install `clang` or some of the other packages.

== Tips on contributing

This project uses `pre-commit` to validate that certain files (e.g. fmf or python) are not broken. Please install `pre-commit` using `pip install pre-commit` as described link:https://pre-commit.com/#install[here]. Then navigate to your clone of this project and install the git hook scripts using `cd ~/llvm-snapshots && pre-commit install`. This will run pre-commit on every `git commit`.

== Process overview [[overview]]

We build LLVM snapshots on a daily basis on Copr and create a github issue in this very repository for every day to keep track of the build and testing status. The link:https://github.com/fedora-llvm-team/llvm-snapshots/blob/816d4e061e748c82f15ca3d1c772d93814a91afe/.github/workflows/check-snapshots.yml#L40[maintainer] changes on a monthly basis and will be assigned to the issue.

For a while now we've switched from standalone builds to the building everything from a single link:https://src.fedoraproject.org/rpms/llvm/tree/rawhide[dist git repo] with the working title "big-merge" on the **Fedora Package Source**. Before we used to have one spec file repository for all the llvm subprojects (clang, lld, lldb, etc.). Now we've taken it one step further and build the snapshots directly out of the `rawhide` branch.

Our snapshot builds are done under our link:https://copr.fedorainfracloud.org/groups/g/fedora-llvm-team/coprs/[group account] on Fedora Copr. There build for all supported Fedora version and some RHEL versions.

We carefully create a new copr project for each day. These projects are called incubator projects. Only if all packages for all operating systems and architectures in an incubator project were successfully built without errors, we will promote it to be the next "official" snapshot link:https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/[here].

For each chroot (except i386) we will also run testing-farm tests and report them in the daily issue. Some of these tests run on Red Hat internal hardware and are only reachable within the VPN. We try our best to utilize github labels to reflect the current state of builds and tests for each chroot.

NOTE: For now we "promote" a daily snapshot to the next "official" snapshot **without** having run testing-farm tests. This might change in the future.

That is the reason why sometimes it can take days until a new version of LLVM will be published here. If you're interested in the version for a particular day, feel free to open https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots-big-merge-YYYYMMDD/ (replace YYYYMMDD with the date you desire). Notice, that we cannot keep the invdividual incubator projects around forever.

== Find out about the health of snapshots

Go to the Fedora Copr project and look for anything red in the monitor for today (see <>).

== How to fix an error in a package build for a specific OS/arch combination?

Suppose you want to work on the `llvm` package with nothing checkout locally, yet.

1. Export your Fedora Account System username so you can copy and paste the commands below:
+
--
----
$ export FAS_USER=
----

NOTE: This is optional and only for copy pasting the below commands. Use the username to login in here: https://id.fedoraproject.org/login.
--

2. Clone the original source code's `rawhide` branch and navigate to that directory.
+
--
----
$ fedpkg clone llvm -b rawhide ~/dev/llvm-rpms/llvm

$ cd ~/dev/llvm-rpms/llvm
----
--

3. Go to https://src.fedoraproject.org/rpms/llvm and click the "fork" button in the top-right corner to create fork just for you. Then add it as a remote:
+
----
$ git remote add $FAS_USER \
    ssh://$FAS_USER@pkgs.fedoraproject.org/forks/$FAS_USER/rpms/llvm.git
----
+
You should have two git remote now: `origin` and one that is named after your FAS login.

4. Make your changes to `llvm` and commit them locally.
+
----
$ git add -p
$ git commit -m "My changes"
----

5. Push your local changes to a branch of your liking (e.g. `mybranch`) in your fork
+
----
$ git push $FAS_USER HEAD:mybranch
----

6. Create a pull-request for your changes so they can go into the `rawhide` branch by executing this and heading over to your browser:
+
----
$ xdg-open https://src.fedoraproject.org/fork/$FAS_USER/rpms/compiler-rt/diff/rawhide..mybranch
----

7. Wait for your changes to be approved and merged. Maybe ping someone from the team.
+
Then your changes will automatically be taken into account for the next daily build.

8. If you want to kick-off a build yourself, you can run:
+
--
----
$ copr build-package \
    --name llvm \
    -r fedora-rawhide-x86_64 \
    --timeout $((30*3600)) \
    @fedora-llvm-team/llvm-snapshots-big-merge-`date +%Y%m%d`
----

This will build the `llvm` package in today's Copr project with your changes as soon as they landed in the `rawhide` branch.
--


== Frequently Asked Questions [[faq]]

=== How can I clone `llvm`?

I cloned `llvm` with the appropriate tool (e.g. `fedpkg`, `centpkg` and `rhpkg`).

----
$ fedpkg clone llvm -b rawhide ~/src/llvm-rpms/llvm-rawhide #<1>
$ centpkg clone llvm -b c9s ~/src/llvm-rpms/llvm-centos #<2>
$ rhpkg clone llvm -b rhel-9-main ~/src/llvm-rpms/llvm-rhel #<3>
----
<1> This is for the regular fedora work on rawhide.
<2> This is for the work on CentOS stream.
<3> This is for the internal work on RHEL9.

=== How to sync with rawhide?

We do the snapshot work in the `rawhide` branch of fedora.

These are the benefits that we see:
* No special handling for contributors. They can just use rawhide.
* When making changes to the spec files for snapshots we can be sure that we’re not breaking rawhide because it is tested by Fedora CI.
* No complicated git merging with `big-merge` or `upstream-snapshot` branches that we've had in the past.
* Reverts are possible
* Smoother release process?
  * Just increment the `Version:` tag used in the non-snapshot part and consume the previously tested changes from the snapshots.

Of course there are some downsides as well:
* Possibly a lot of commit activity can happen in the `rawhide` branch that doesn’t change anything related to `rawhide` but only snapshots.
* A bit more if/else branches in the spec files.
* Duplicated patch files with slightly different content as they apply to each supported LLVM version.

We need `%if %{maj_ver} >= 20` conditions or alike for different LLVM versions (see link:https://src.fedoraproject.org/rpms/llvm/pull-request/286#_4__13[this PR] for example).

=== How do I find the monitor? [[monitor]]

You can find the snapshot monitor for LLVM Fedora builds on Copr here:

https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/monitor/

The above link brings you to the latest "forked" build. It will only contain successful builds.

To find out where this build came from, take a look at the title of the project. There it should say something like:

>  ( forked from @fedora-llvm-team/llvm-snapshots-big-merge-20230221 )

Go to the project from which `@fedora-llvm-team/llvm-snapshots` was forked to find failing builds.

=== How do I run a local snapshot build?

----
$ git clone https://src.fedoraproject.org/rpms/llvm.git
$ cd llvm
$ make snapshot-rpm
----

= README
:icons: font

You might need to install missing dependencies. The build process itself probably takes quite some time.

You're going to find `results/YYYYMMDD/snapshot-rpm.log` with logging everything from this makefile target.

== Mass Rebuilds ==

This repository uses GitHub Actions to periodically perform rebuilds of selected
Fedora packages.  Once a mass rebuild is complete there is also automation
that will create a new issue with the results of the rebuild.

The rebuild process will attempt to automatically bisect the failures to a specific upstream
LLVM commit.

The rebuild can be started manually using the rebuilder.py script in
`.github/workflows/`

[source,console]
---
$ python3 rebuilder.py rebuild
---

You can also view the regression report once the rebuild is complete using
the same script.

[source,console]
---
$ python3 rebuilder.py get-regressions --start-date=
---

The start date should be the day the rebuild was started (In reality
it can be any date between when the last rebuild ended and the
new rebuild began).

Owner

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Kleine"
  given-names: "Konrad"
- family-names: "Stellard"
  given-names: "Tom"
- family-names: "Popov"
  given-names: "Nikita"
- family-names: "Quites Machado Filho"
  given-names: "Tulio Magno"
- family-names: "Bäder"
  given-names: "Timm"
title: "LLVM Snapshots"
version: 2.0.0
date-released: 2024-04-01
url: "https://github.com/fedora-llvm-team/llvm-snapshots"

Issues and Pull Requests

Last synced: 6 months ago