path: root/english/mirror/ftpmirror.wml

#use wml::debian::template title="Setting up a Debian archive mirror"
#use wml::debian::toc
#include "$(ENGLISHDIR)/releases/sid/archive.data"
#include "$(ENGLISHDIR)/releases/arches.data"

<toc-display />

<toc-add-entry name="whether">Whether to mirror</toc-add-entry>

<p>While we appreciate new mirrors, every prospective mirror maintainer
should make sure that they can answer these questions before trying to
start their own mirror:</p>

<ul>
  <li>Is a mirror necessary at my location? Maybe there are already
      mirrors nearby.</li>
  <li>Do I have the resources to host a mirror? Mirrors take up considerable
      <a href="size">disk space</a> and bandwidth, one has to be able to commit to the cost.</li>
  <li>Is a mirror the right choice?  If you primarily want to support
      users at your ISP/facility, then maybe a caching proxy such as
      apt-cacher-ng, squid, or varnish might be the better choice.</li>
</ul>

<toc-add-entry name="what">What to mirror</toc-add-entry>

<p>The <a href="./">main mirror page</a> lists the archives available for
mirroring.</p>

<ul>
<li>
Users will look for the debian/ archive to install Debian over
the network, to build CDs from (with jigdo), and to upgrade already
installed systems.  <em>We recommend you mirror this repository.</em></li>

<li>
debian-cd/ is an archive that isn't identical across all the different
mirror servers. On some sites it contains jigdo templates to build CD images
from (used in combination with files from debian/), on some it contains
already built CD images, and on some sites both.
<br />
Please see the page for <a href="$(HOME)/CD/mirroring/">mirroring the CD
images</a> for further information on this.</li>

<li>
debian-archive/ contains the true <em>archive</em>, the old and obsolete
versions of Debian. It will generally be interesting only to a small segment
of users. (If you don't know whether you want to mirror this, you probably don't.)</li>

</ul>

<p>Please see <a href="size">the mirror size</a> page for
more precise information about mirror sizes.</p>

<p>The debian-security/ archives contain the security updates released by
the Debian security team. While it sounds interesting to everyone, we do
not recommend to our users to use mirrors to obtain security updates and
instead ask them to directly download them from our distributed
security.debian.org service.  <em>We recommend debian-security
<strong>not</strong> be mirrored.</em></p>

<toc-add-entry name="wherefrom">Where to mirror from</toc-add-entry>

<p>Note that <code>ftp.debian.org</code> is not canonical location of Debian
packages, instead it is merely one of several servers that get updated from an
internal Debian server.

There are many <a href="list-full">public mirrors</a> which support rsync
and that are good to mirror from. Please use one that is close to you
network-wise.</p>

<p>You should avoid mirroring from any service name that resolves to more
than one address (such as <code>ftp.us.debian.org</code>) as this might cause
you to sync between different states across your own mirror-runs should your
upstream mirrors get out of sync.
#
Also note that HTTP is the only service that we guarantee to exist on
<code>ftp.CC.debian.org</code>.  If you want to mirror using rsync (using
ftpsync as recommended), we suggest you pick the proper site name for the
machine that currently provides the <code>ftp.CC.debian.org</code>.  (Consult
the <code>/debian/project/trace</code> directory of that server to learn it).

<toc-add-entry name="how">How to mirror</toc-add-entry>

<p>The recommended method of mirroring is with the ftpsync scriptset, available
in those forms:</p>
<ul>
    <li>as a tarball from <url "https://ftp-master.debian.org/ftpsync.tar.gz"></li>
    <li>as a git repository: <kbd>git clone https://salsa.debian.org/mirror-team/archvsync.git</kbd> (see <url "https://salsa.debian.org/mirror-team/archvsync/">)</li>
    <li>as a Debian package: <a href="https://packages.debian.org/stable/ftpsync">ftpsync</a></li>
</ul>

<p>Do not use your own scripts, and do not just use single-pass rsyncs.  Using
ftpsync ensures updates are done in a way so that apt does not get confused.
In particular, ftpsync processes translations, contents, and other metadata
files in an order such that apt is not running into validation errors should a
user update the package list while a mirror-run is in progress.  Furthermore,
it also produces trace files that contain more information that is useful for
determining whether a mirror works, what architectures it carries, and where it
syncs from.</p>

<toc-add-entry name="partial">Partial mirroring</toc-add-entry>

<p>Considering the already <a href="size">large size of Debian archive</a>,
it might be advisable to only mirror parts of the archive.  Public mirrors
should carry all suites (testing, unstable, etc.), but they may restrict the
architecture set that they carry.  The configuration file for ftpsync has
ARCH_EXCLUDE and ARCH_INCLUDE settings for this purpose.</p>

<toc-add-entry name="when">When to mirror</toc-add-entry>

<p>The main archive gets updated four times a day.
The mirrors commonly start updating around 3:00, 9:00, 15:00 and 21:00 (all times UTC),
but these are never fixed times and you should not fixate your mirror on them.</p>

<p>Your mirror should update a few hours after one of the main archive mirror
pulses.  You should check if the site you're mirroring from leaves a time stamp
file in their <kbd>project/trace/</kbd> subdirectory. The time stamp file
will be named like that site, and it will contain the completion time of
their last mirror update. Add a couple of hours to that time (to be safe)
and mirror then.</p>

<p><b>It is essential that your mirror is in sync with the main archive</b>. A
minimum of 4 updates per 24 hours will ensure that your mirror is a true
reflection of the archive. Please understand that mirrors that are not in sync
with the main archive will not be listed in the official mirrors listing.</p>

<p>The easiest way to automatically have the mirror run every day is to use
cron. See <kbd>man crontab</kbd> for details.</p>

<p>Note that if your site is being triggered with a push mechanism, then you
don't need to worry about any of this.</p>

<h3>Push-triggered mirroring</h3>

<p><q>Push</q> mirroring is a form of mirroring that we have developed
to minimize the time it takes for changes to the archive to reach mirrors.
An upstream mirror uses an SSH trigger to tell the downstream mirror to
update itself.
For a more detailed description of how this works, why it is secure, and
how to set a push mirror, please see <a href="push_mirroring">the complete
explanation</a>.</p>

<toc-add-entry name="settings">Recommended additional settings</toc-add-entry>

<p>Public mirrors should make the Debian archive available over HTTP at <code>/debian</code>.</p>

<p>Furthermore, please ensure directory listings are enabled (with full length
filenames) and symlinks followed.

If you use Apache, something like this might work:
<pre>
&lt;Directory <var>/path/to/your/debian/mirror</var>&gt;
   Options +Indexes +SymlinksIfOwnerMatch
   IndexOptions NameWidth=* +SuppressDescription
&lt;/Directory&gt;
</pre>

<toc-add-entry name="submit">How to add a mirror to the mirror list</toc-add-entry>

<p>
If you would like to have your mirror listed on the official mirrors list please:
</p>

<ul>
<li>Ensure that your mirror synchronizes 4 times per 24 hours with the archive</li>
<li>Ensure that your mirror includes the source files for the architectures the
mirror carries</li>
</ul>

<p>Once the mirror is set up, it should be <a href="submit">registered with Debian</a>
in order to get included in the <a href="list">official mirror list</a>.
Submissions can be done using our <a href="submit">simple web form</a>.</p>


<p>Any problems or enquiries can be sent to <email mirrors@debian.org>.</p>

<toc-add-entry name="mailinglists">Mailinglists</toc-add-entry>

<p>There are two public <a href="../MailingLists/">mailing lists</a> about
Debian mirrors,
<a href="https://lists.debian.org/debian-mirrors-announce/">debian-mirrors-announce</a>
and
<a href="https://lists.debian.org/debian-mirrors/">debian-mirrors</a>.
We encourage all mirror maintainers to subscribe to the announcement list as
it will be used for any important announcements. This list is moderated and will
receive only a low amount of traffic. The second mailing list is meant for general
discussion and is open to all.</p>

<p>If you have any questions that aren't answered on these web pages, you
can contact us at <email mirrors@debian.org> or using IRC, #debian-mirrors on
<tt>irc.debian.org</tt>.</p>


<toc-add-entry name="private-mirror">Notes for private (partial) mirrors</toc-add-entry>

<p>
If you want to operate a mirror for your own site only, and you need to carry only a
subset of suites (such as stable), <a href="https://packages.debian.org/stable/debmirror">debmirror</a>
might be a good fit for you also.
</p>