Optimizing Group Opening

The Need for Speed

MT-NewsWatcher is fast. A number of changes over the original NewsWatcher have been made to speed up various operations, especially opening groups. However, a number of different factors can impact on group opening speed. This chapter describes them.

The original NewsWatcher fetches only Subject and Author headers from the news server, and does so using the XHDR command which must be given once for each header type. MT-NewsWatcher can obtain much more information about articles, such as their posting date and line count, so making decisions about which articles to read easier for the user. However, this incurs an overhead in terms of the time taken to download such information.

You can control whether MT-NewsWatcher needs to download header in addition to the Subject header (which is always downloaded) using the settings in the 'Subject window' panel of the preferences. MT-NewsWatcher implements a new way of getting header information from the server, using the XOVER command. When this is activated, fetching headers is usually much faster.

However, as explained below, downloading time is not a simple function of these settings. This document explains how the various settings impact downloading time.

Server Options and Speed

In the 'Server options' panel of the News Servers dialog, there are a number of setting that determine how MT-NewsWatcher communicates with your news server. Most of them have speed implications.

Use XPAT command for searches

Searching article headers for a certain string can be performed either by the news server, or by MT-NewsWatcher itself. If this option is checked, searches are performed by sending an XPAT request to the server, which then responds with a list of matching articles. Although this may take some time, the job of searching is done by the server. If this option is unchecked, MT-NewsWatcher must download all the headers for that group and search them itself, which is much slower and requires more memory.

Nearly all servers support XPAT, so you should turn this option on, and only uncheck it if the server sends back an error message when you try to search.

Use XOVER command to get headers

This is the main option that affects speed. When active, article header information is downloaded in larger chunks, which is much faster. It is described in more detail below.

Again, most servers support XOVER, so you should turn this on. Beware that some servers may cause a warning message to be displayed the first time you use it; the reason for this is described below.

Don't open new connection when checking for new articles

Some servers return bogus article counts if the news client reuses an existing connection to request group article information. To work around this, the client has to open a new connection to check for new articles.

This is not a problem with most servers, so this option should be checked. If the 'Check for new articles' command never returns new articles, you could try turning this option off.

Don't send MODE READER command

Some servers require that news clients identify themselves as such, by sending the 'mode reader' command (to distinguish them from other servers with which your server exchanges news). Most don't, so you should turn this option on.

Allow up to N simultaneous connections

This controls the maximum number of concurrent connections to the news server that can be opened. 4 is a sensible value. More detail is given in the threading document.

The Options panel in the News servers dialog also contains a setting that affects speed.

Use accurate (but slow) article counts

Checking this option causes MT-NW to fetch the full list of articles in a group from the server when checking the group for new articles (for example, when you first open your User Groups file). This can be quite slow for large groups, depending on the speed of your connection. However, it does allow MT-NW to build a precise list of what articles are available for that group, and thus give accurate counts for the number of unread articles, and the number of articles to fetch.

Reading Habits and Speed

The way you read news can also affect how long it takes to open a group. The basic problem is fragmentation; when you open a group with lots of articles, particularly if you've just marked that group unread, then the news client has to send just a single command to get all the articles in the unread range, like:

get headers for 2350-4567

Now, when you read an article here and there, next time you open the group the news client requests those articles that you have not read before. So if you read articles 2360, 3420 and 4560 and close the group, next time you open it the clients sends

get headers for 2350-2359
get headers for 2361-3419
get headers for 3421-4559
get headers for 4561-4567

After each request, the client waits for the server to send back headers for those articles before requesting the next batch (at least, that's how the NewsWatchers are implemented). So there are a lot more round-trips to the server, and opening the group takes longer.

This is why the style of reading news where you mark a group read, after just reading those articles you are interested in, is faster. In the MT-NewsWatcher, the 'Next Group' command does just this.

What determines which headers are downloaded?

There are a number of settings that determine which headers need to be obtained from the server. These are:

'Subject windows' Preferences

There are three checkboxes in this preferences panel control whether or not authors, dates and line counts are shown in subject windows. If checked, this information has to be obtained from the server.

'Server options' settings

The Use XOVER command to get article headers is the important setting here; its workings are described below. If checked, download time will not be affected by the choice of options in the 'Subject window' preferences, because the fetching headers with XOVER always includes author, subject, date and line headers.

'Thread articles by references' option

This option, in the 'Subject windows' preferences panel, specifies that MT-NW should attempt to build discussion threads based on the contents of the References header. When active, Message-ID and References headers are required from the server, because they are needed to build the threads. Both these headers are included in XOVER responses.

Article Management options

There are two options that specify whether MT-NewsWatcher removes duplicate articles that you have read in another group, or have been killed by a filter in another group. These are the checkboxes in the 'Article Management' preferences panel labelled Read in another group and Filter out (killed) in another group. If you have either of these turned on, MT-NewsWatcher must fetch information from the server about which other groups an article has been posted to, which is contained in the Xref header.

Filters

The set of filters that apply to the group being downloaded may also require that additional information be obtained from the server. If you create a filter that acts on a header that is not being downloaded for other reasons, then that header will have to specifically fetched from the server. Examples of such headers include Newsgroups, NNTP-Posting-Host, Path, User-Agent.

In summary, all headers which are either to be displayed, or filtered, or used to build threads are required from the server. If XOVER is active, then most information will be contained in the response to the XOVER command, and additional headers will only be required if they are not contained in the XOVER data, which is described below.

Customizing header retrieval for certain groups

If you find that certain groups always take a long time to open, and you want to speed them up, you can override various default settings to make opening these groups faster. For example, you may find that opening binaries groups takes a long time, and seems to spend most of the time fetching the Xref header. In such cases, you can use Group Settings to modify the settings for one or a set of groups.

The Group Settings that are relevant here are in the Article Lists panel:

Thread articles by:

Threading by references will require that Message-ID and References headers are downloaded. Choose a different threading type to avoid this overhead (only useful if XOVER is turned off for this server).

Don't fetch crosspost data

Check this box to not fetch the Xref header for this group, which can make a big difference on some groups on some servers. However, this will prevent MT-NewsWatcher from being able to mark crossposts read for this group.

The XOVER Command

The XOVER command is a command in the NNTP protocol that a new client can use to get batches of header data from the server. When the news client (e.g. MT-NewsWatcher) requests XOVER information for a set of articles, the server replies with a block of data for each article that must contain the following headers:

Header	Notes
Subject
Author
Date	MT-NW converts this into a Macintosh date format
Message-ID	Needed for reference-based threading
References	Needed for reference-based threading
Byte count	Unused
Line count
Xref	Contains cross-posting information (some servers may be configured to not include this in XOVER data)

Because this information is sent in big chunks, this is much faster than fetching each of these headers independently. When receiving these data, MT-NW processes them, discards information that is not required either for display or filtering, and keeps the rest.

With XOVER active, almost all the information that you will normally need for display, threading, and a basic set of filters, will be downloaded in one go, which is fast. The exceptions are some older news servers which don't include Xref information (which is useful for filters designed to catch excessive cross-posting), and filters using the less common headers like X-Mailer, and X-Newsreader.

Why do I a get a warning with XOVER?

When you activate XOVER for the first time, you may see a warning which says:

This news server fails to respond to a command which provides information about the format of XOVER request results.

and gives you the options of either continuing at your peril, or cancelling.

This occurs because your news server does not conform to some recent specifications about the format of XOVER responses. News clients are supposed to be able to get information from the server on the content of XOVER responses by using the LIST OVERVIEW.FMT command. MT-NewsWatcher uses this to determine what to expect from an XOVER command. However, some servers (notably those by Microsoft and the DNEWS server) don't understand that LIST command. In these cases, MT-NewsWatcher assumes that such responses must contain the headers listed in the table above, and if it finds Xref headers, it will use these as well. But to warn you that it is making these assumptions (which might, for some future server, fail), you will see this warning dialog. Hit the 'Continue' button to continue to use XOVER with the assumed format.

Fetching Extra Headers on Demand

MT-NewsWatcher stores those headers that it has downloaded for a group until you close the subject window. Some actions may require MT-NewsWatcher to fetch more headers from the server. For example, if you edit a filter to use a new header, and then re-filter the articles, new headers will have to be fetched from the server.