Mirror Layout

The Ubuntu repository mirror is maintained primarily on the old instsrv (= nr1) in /var/Ubuntu/mirror . From there it is rsynced to the new instsrv (=picard) using the sync-picard-from-nr1 script with um as the target. On both hosts, the subdirectory staged contains time-versioned snapshots of the updates, e.g. 2017.08.22-232222. It also contains symbolic links relating to the update aging. Clients use both hosts with a load sharing and failover configuration, thus it's important to keep them in sync.

Snapshots

Updates are staged using a patched version of repository-timeline as developped by the colleagues in Hamburg. Note the version on github is out of date, and our local package contains patches to address changes in the repository metadata in the 18.04 and 20.04 releases of Ubuntu.

Note: If it's required to extend the list of files or directories mrepo-* shouldn't hardlink but rather copy, it's not sufficient to change it in timeline.py, since the values in there are only copied to an actual repo config when a new repository is instantiated. In addition to patching the script, it's mandatory to modify /var/Ubuntu/mirror/staged/timeline.cfg on any system where mrepo-create-snapshot.py is to be run.

The aging of the updates happens in this order:

Z => A => B => 3 => 4 => C/D => 6

Procedures

Updating the mirror

Currently on the old instsrv (= nr1) only.

Simply use the apt-mirror command. Note there's no dryrun mode, so make sure the current state is saved in a snapshot, that is check-state does not display any packages in the "Z" (pseudo-)snapshot.

 % apt-mirror 
Downloading 504 index files using 16 threads...
Begin time: Thu Jul 27 09:48:21 2017
[16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Thu Jul 27 09:48:25 2017

Processing tranlation indexes: [TTTTTTTTTTTT]

Downloading 765 translation files using 16 threads...
Begin time: Thu Jul 27 09:48:25 2017
[16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Thu Jul 27 09:48:29 2017

Processing DEP-11 indexes: [DDDDDDDDDDDD]

Downloading 42 dep11 files using 16 threads...
Begin time: Thu Jul 27 09:48:29 2017
[16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Thu Jul 27 09:48:29 2017

Processing indexes: [PPPPPPPPPPPP]

1.4 GiB will be downloaded into archive.
Downloading 470 archive files using 16 threads...
Begin time: Thu Jul 27 09:48:54 2017
[16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Thu Jul 27 09:49:02 2017

611.0 MiB in 264 files and 0 directories can be freed.
Run /var/Ubuntu/mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/var/Ubuntu/mirror/var/postmirror.sh)

/bin/sh: /var/Ubuntu/mirror/var/postmirror.sh: No such file or directory

Post Mirror script has completed. See above output for any possible errors.

(that postmirror error is normal and harmless). If there are no new packages, it will say "0 bytes will be downloaded into archive".

Checking the mirrored repository's integrity

It does happen occasionally that the upstream repository is in an inconsistent state, and hence after apt-mirror so is the local mirror. This would result in apt on the clients rejecting the repository and not doing anything. Run check-hashes to verify the checksums in the repository metadata:

[nr1] ~ % /project/linux/Ubuntu/scripts/check-hashes.pl
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic...............................ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic-security......ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/bionic-updates......ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal...............................ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal-security......ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/focal-updates......ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial..............................ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security......ok
/var/Ubuntu/mirror/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates......ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/bionic...ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/focal...ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/nextcloud-devs/client/ubuntu/dists/xenial...ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/bionic...ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/focal...ok
/var/Ubuntu/mirror/mirror/ppa.launchpad.net/openafs/stable/ubuntu/dists/xenial...ok

Do not proceed unless all results are ok. Re-run apt-mirror until they are. This may take several days - if you're getting nervous about rolling out updates to production systems, you'll have to revert by rsyncing back from the not yet changed repo on the new instsrv:

rsync  -e ssh --delete -avHO --exclude=status --exclude=staged picard:/var/Ubuntu/mirror/ /var/Ubuntu/mirror

Obviously, try a dry-run with -n first.

Cleaning out the mirror

It's not a bad idea to free the space used by obsoleted packages:

% /var/Ubuntu/mirror/var/clean.sh
Removing 264 unnecessary files [611.0 MiB]...
[0%]...........................done.

Removing 0 unnecessary directories...
done.

To play it safe, maybe run check-hashes again. It only takes a few seconds.

Checking the Snapshot Status

This step is kind of optional, but do it if you'd like to understand better what is going on.

The script /var/Ubuntu/mirror/status/check-state.sh shows the changes in the debian-installer and main (including restricted, multiverse, universe and the assorted security and updates) repositories for the supported Ubuntu releases (currently 16.04 xenial, 18.04 bionic and 20.04 focal).

It lists changes between snapshots grouped by source package listing first the snapshot containing the old package followed by the updated snapshot:

* ghostscript
* ghostscript-dbg
* ghostscript-doc
* ghostscript-x
* libgs-dev
* libgs9
* libgs9-common
    6: 9.10~dfsg-0ubuntu10.9
    B: 9.10~dfsg-0ubuntu10.10

In this example, 6 and the newer snapshots C/D,4,3 contain ghostscript 9.10~dfsg-0ubuntu10.9 while B and A contain 9.10~dfsg-0ubuntu10.10.

Syncing the mirror to the new instsrv

If things look ok so far, replicate the mirror to the new instsrv (=picard):

[picard] ~ % /project/linux/SL/scripts/sync-picard-from-nr1.pl um -n -x

No need to bother colleagues with a mail, thus the -n.

Creating a Snapshot

The following has to be carried out simultaneously on the old and new instsrv.

Creating a new snapshot will promote updates from upstream (Z) to early test systems (A) to more test systems (B) to 3 to 4 to production systems (C/D) to 6 to later points in snapshot history.

A snapshot is created by the command

mrepo-create-snapshot.py /var/Ubuntu/mirror/staged

After creating a snapshot, you can optionally view a diff using

/var/Ubuntu/mirror/status/check-state.sh

and should save the state using the command

/var/Ubuntu/mirror/status/save-repostate.pl -v -x

If the latter is executed every time after a new snapshot was created, one can at any time quickly get an overview of changes in the pipeline with the command

/var/Ubuntu/mirror/status/diff-lastest.pl

This will not change anything, but just display the difference between the latest snapshot and the previous one.

Summary

To update the mirror, check it, sync it, and promote updates:

1. [nr1] ~ % apt-mirror

2. [nr1] ~ % /var/Ubuntu/mirror/status/check-hashes.pl

3. [nr1] ~ % /var/Ubuntu/mirror/var/clean.sh

4. [nr1] ~ % /var/Ubuntu/mirror/status/check-hashes.pl

5. [picard] ~ % /project/linux/SL/scripts/sync-picard-from-nr1.pl um -n -x

6. create snapshot:

   1. [nr1] ~ % mrepo-create-snapshot.py /var/Ubuntu/mirror/staged

   2. [picard] ~ % mrepo-create-snapshot.py /var/Ubuntu/mirror/staged

7. save repo states:

   1. [nr1] ~ % /var/Ubuntu/mirror/status/save-repostate.pl -v -x

   2. [picard] ~ % /var/Ubuntu/mirror/status/save-repostate.pl -v -x

8. view changes:

   1. [nr1] ~ % /var/Ubuntu/mirror/status/diff-latest.pl

   2. [picard] ~ % /var/Ubuntu/mirror/status/diff-latest.pl

Make sure the output is the same on both hosts. If it isn't, you probably forgot to rsync from the old to the new instsrv (and you have to delete the snapshots and the saved repo states on both servers).

How to download packages from a ppa

1. install apt-mirror

2. create a file apt-mirror.list (anywhere) with content like this:

   set base_path      /tmp/Ubuntu/mirror
   
   deb-amd64 http://ppa.launchpad.net/openafs/stable/ubuntu xenial main
   deb-src   http://ppa.launchpad.net/openafs/stable/ubuntu xenial main
   
   deb-amd64 http://ppa.launchpad.net/openafs/stable/ubuntu trusty main
   deb-src   http://ppa.launchpad.net/openafs/stable/ubuntu trusty main

3. mkdir -p /tmp/Ubuntu/mirror

4. apt-mirror /path/to/apt-mirror.list

This will download the amd64 debs + sources/dscs. No need to be root, except for step 1. Obviously, make sure /tmp/Ubuntu or whatever is owned by you and not a symlink.