Contents
Mirror Layout
The Ubuntu repository mirror is maintained primarily on the old instsrv (= nr1) in /var/Ubuntu/mirror . From there it is rsynced to the new instsrv (=picard) using the sync-picard-from-nr1 script with um as the target. On both hosts, the subdirectory staged contains time-versioned snapshots of the updates, e.g. 2017.08.22-232222. It also contains symbolic links relating to the update aging. Clients use both hosts with a load sharing and failover configuration, thus it's important to keep them in sync.
Snapshots
Updates are staged using a patched version of repository-timeline as developped by the colleagues in Hamburg. Note the version on github is out of date, and our local package contains patches to address changes in the repository metadata in the 18.04 and 20.04 releases of Ubuntu.
Note: If it's required to extend the list of files or directories mrepo-* shouldn't hardlink but rather copy, it's not sufficient to change it in timeline.py, since the values in there are only copied to an actual repo config when a new repository is instantiated. In addition to patching the script, it's mandatory to modify /var/Ubuntu/mirror/staged/timeline.cfg on any system where mrepo-create-snapshot.py is to be run.
The aging of the updates happens in this order:
Z => A => B => 3 => 4 => C/D => 6
Snapshot |
Description |
Z |
recently mirrored packages not yet part of a snapshot |
A |
updates for test machines with vars.DZPM_AGING=A in Vamos |
B |
updates for test machines with vars.DZPM_AGING=B in Vamos |
3 |
packages age 3 |
4 |
packages age 4 |
C/D |
updates for production machines with vars.DZPM_AGING=C or vars.DZPM_AGING=D |
6 |
packages age 6 |
Procedures
Updating the mirror
Currently on the old instsrv (= nr1) only.
Simply use the apt-mirror command. Note there's no dryrun mode, so make sure the current state is saved in a snapshot, that is check-state does not display any packages in the "Z" (pseudo-)snapshot.
% apt-mirror Downloading 504 index files using 16 threads... Begin time: Thu Jul 27 09:48:21 2017 [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Thu Jul 27 09:48:25 2017 Processing tranlation indexes: [TTTTTTTTTTTT] Downloading 765 translation files using 16 threads... Begin time: Thu Jul 27 09:48:25 2017 [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Thu Jul 27 09:48:29 2017 Processing DEP-11 indexes: [DDDDDDDDDDDD] Downloading 42 dep11 files using 16 threads... Begin time: Thu Jul 27 09:48:29 2017 [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Thu Jul 27 09:48:29 2017 Processing indexes: [PPPPPPPPPPPP] 1.4 GiB will be downloaded into archive. Downloading 470 archive files using 16 threads... Begin time: Thu Jul 27 09:48:54 2017 [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Thu Jul 27 09:49:02 2017 611.0 MiB in 264 files and 0 directories can be freed. Run /var/Ubuntu/mirror/var/clean.sh for this purpose. Running the Post Mirror script ... (/var/Ubuntu/mirror/var/postmirror.sh) /bin/sh: /var/Ubuntu/mirror/var/postmirror.sh: No such file or directory Post Mirror script has completed. See above output for any possible errors.
(that postmirror error is normal and harmless). If there are no new packages, it will say "0 bytes will be downloaded into archive".
Checking the mirrored repository's integrity
It does happen occasionally that the upstream repository is in an inconsistent state, and hence after apt-mirror so is the local mirror. This would result in apt on the clients rejecting the repository and not doing anything. Run check-hashes to verify the checksums in the repository metadata:
[nr1] ~ % /project/linux/Ubuntu/scripts/check-hashes.pl /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic...............................ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic-security......ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic-updates......ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal...............................ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal-security......ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal-updates......ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial..............................ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security......ok /var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates......ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/bionic...ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/focal...ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/xenial...ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/bionic...ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/focal...ok /var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/xenial...ok
Do not proceed unless all results are ok. Re-run apt-mirror until they are. This may take several days - if you're getting nervous about rolling out updates to production systems, you'll have to revert by rsyncing back from the not yet changed repo on the new instsrv:
rsync -e ssh --delete -avHO --exclude=status --exclude=staged picard:/var/Ubuntu/mirror/ /var/Ubuntu/mirror
Obviously, try a dry-run with -n first.
Cleaning out the mirror
It's not a bad idea to free the space used by obsoleted packages:
% /var/Ubuntu/mirror/var/clean.sh Removing 264 unnecessary files [611.0 MiB]... [0%]...........................done. Removing 0 unnecessary directories... done.
To play it safe, maybe run check-hashes again. It only takes a few seconds.
Checking the Snapshot Status
This step is kind of optional, but do it if you'd like to understand better what is going on.
The script /var/Ubuntu/mirror/status/check-state.sh shows the changes in the debian-installer and main (including restricted, multiverse, universe and the assorted security and updates) repositories for the supported Ubuntu releases (currently 16.04 xenial, 18.04 bionic and 20.04 focal).
It lists changes between snapshots grouped by source package listing first the snapshot containing the old package followed by the updated snapshot:
* ghostscript * ghostscript-dbg * ghostscript-doc * ghostscript-x * libgs-dev * libgs9 * libgs9-common 6: 9.10~dfsg-0ubuntu10.9 B: 9.10~dfsg-0ubuntu10.10
In this example, 6 and the newer snapshots C/D,4,3 contain ghostscript 9.10~dfsg-0ubuntu10.9 while B and A contain 9.10~dfsg-0ubuntu10.10.
Syncing the mirror to the new instsrv
If things look ok so far, replicate the mirror to the new instsrv (=picard):
[picard] ~ % /project/linux/SL/scripts/sync-picard-from-nr1.pl um -n -x
No need to bother colleagues with a mail, thus the -n.
Creating a Snapshot
The following has to be carried out simultaneously on the old and new instsrv.
Creating a new snapshot will promote updates from upstream (Z) to early test systems (A) to more test systems (B) to 3 to 4 to production systems (C/D) to 6 to later points in snapshot history.
A snapshot is created by the command
mrepo-create-snapshot.py /var/Ubuntu/mirror/staged
After creating a snapshot, you can optionally view a diff using
/var/Ubuntu/mirror/status/check-state.sh
and should save the state using the command
/var/Ubuntu/mirror/status/save-repostate.pl -v -x
If the latter is executed every time after a new snapshot was created, one can at any time quickly get an overview of changes in the pipeline with the command
/var/Ubuntu/mirror/status/diff-lastest.pl
This will not change anything, but just display the difference between the latest snapshot and the previous one.
Summary
To update the mirror, check it, sync it, and promote updates:
[nr1] ~ % apt-mirror
[nr1] ~ % /var/Ubuntu/mirror/status/check-hashes.pl
[nr1] ~ % /var/Ubuntu/mirror/var/clean.sh
[nr1] ~ % /var/Ubuntu/mirror/status/check-hashes.pl
[picard] ~ % /project/linux/SL/scripts/sync-picard-from-nr1.pl um -n -x
- create snapshot:
[nr1] ~ % mrepo-create-snapshot.py /var/Ubuntu/mirror/staged
[picard] ~ % mrepo-create-snapshot.py /var/Ubuntu/mirror/staged
- save repo states:
[nr1] ~ % /var/Ubuntu/mirror/status/save-repostate.pl -v -x
[picard] ~ % /var/Ubuntu/mirror/status/save-repostate.pl -v -x
- view changes:
[nr1] ~ % /var/Ubuntu/mirror/status/diff-latest.pl
[picard] ~ % /var/Ubuntu/mirror/status/diff-latest.pl
Make sure the output is the same on both hosts. If it isn't, you probably forgot to rsync from the old to the new instsrv (and you have to delete the snapshots and the saved repo states on both servers).
How to download packages from a ppa
- install apt-mirror
create a file apt-mirror.list (anywhere) with content like this:
set base_path /tmp/Ubuntu/mirror deb-amd64 http://ppa.launchpad.net/openafs/stable/ubuntu xenial main deb-src http://ppa.launchpad.net/openafs/stable/ubuntu xenial main deb-amd64 http://ppa.launchpad.net/openafs/stable/ubuntu trusty main deb-src http://ppa.launchpad.net/openafs/stable/ubuntu trusty main
mkdir -p /tmp/Ubuntu/mirror
apt-mirror /path/to/apt-mirror.list
This will download the amd64 debs + sources/dscs. No need to be root, except for step 1. Obviously, make sure /tmp/Ubuntu or whatever is owned by you and not a symlink.