Skip to content

Latest commit

 

History

History
68 lines (47 loc) · 3.02 KB

ci.md

File metadata and controls

68 lines (47 loc) · 3.02 KB
Raw

Guide on CI infrastructure

Context

Apache XTable (incubating) uses Azure DevOps and Azure infrastructure for its CI, providing more flexibility by not being constrained to the Apache org's shared quota for CI. This approach is similar to those adopted by Apache Flink and Apache Hudi communities.

Overview

We have created apachextable-ci to manage all CI-related repositories:

  • xtable-mirror: This repository mirrors the official repository (apache/incubator-xtable) to build commits merged into master and release-* branches.
  • xtable-branch-ci: This repository is dedicated to building branches created PRs. Its main branch remains at the commit of the branch's creation and does not require synchronization with the official repository.

The diagram below gives an overview of the infrastructure.

!TODO

Azure VM

We are running a B1s VM instance in this Azure XTable account.

Bootstrap Steps

sudo apt update -y
sudo apt install -y git cron openjdk-8-jdk maven
git clone git@github.com:apachextable-ci/git-repo-sync.git
git clone git@github.com:apachextable-ci/ci-bot.git
cd ci-bot
mvn clean install

Mirroring Master & Release Commits

Use crontab -e to schedule the mirroring job, currently configured to mirror main and release commits to the apachextable-ci repository.

*/10 * * * * $HOME/git-repo-sync/sync_repo.sh > /dev/null 2>&1

Scanning PRs and Triggering Branch Builds

Run $HOME/git-repo-sync/run_cibot.sh to start the CI branch builds in the background.

Maintenance

Manage the background process with htop. Typical maintenance steps include killing the process from htop, cleaning up ~/ci-bot.log, and re-running the script.

Check the expiry date of Azure and GitHub tokens and update them accordingly. They are used in $HOME/git-repo-sync/run_cibot.sh.

Potential Issues

  • Mirrored repositories (e.g., master, release-*) not being pushed, leading to failure in mirrored CI execution.
    • Issues with git fetch or push in $HOME/git-repo-sync/sync_repo.sh. Manually execute the git commands and troubleshoot as needed.

Azure Pipelines

There are two pipelines defined in this Azure DevOps project.

  • xtable-mirror: for master/release version builds
  • xtable-branch-ci: for PR builds

For each xtable-branch-ci build, xtable-bot will post and update comments on its corresponding PR as follows.

!TODO

PR reviewers should consider the results of this CI report as one of the merging criteria.

Get Help