Sending and Viewing binaries

Introduction to Attached Files

Sometimes people attach files directly to their postings instead of referring to an FTP site. As with referenced files, attached files can be programs, documents, pictures, sounds, or any other kind of file. MT-NewsWatcher makes it easy to view or extract these attached files and copy them to your hard drive.

Several popular newsgroups are used almost exclusively for distributing files in this way. These are commonly called "binary groups", and attached files are called "binaries". For example, the moderated group comp.binaries.mac is used to distribute Mac freeware and shareware and demo programs and other Mac files. As another example, the hierarchy of groups alt.binaries.pictures.* is used to distribute pictures.

How MT-NewsWatcher handles binaries

MT-NewsWatcher can decode and display images which are included in news articles right in the article window. When you open such an article, MT-NewsWatcher automatically identifies which parts of the article consist of encoded binary text, separates them out, and decodes them. If a NewsWatcher plugin is found that can present the data, then this is used to display the image, play the sound etc. This decoding takes place automatically even if there is more than one binary in the article, or if the binary is split into multiple parts.

When it encounters encoded binary data, MT-NewsWatcher looks for a file name, either in the begin line, or in MIME section headers. If a filename is found, it then uses the file extension to choose which plugin to use to decode and display the image. Each plugin specifies a list of file extensions that it knows about.

As article data is received from the server, MT-NewsWatcher does the first level of decoding (extracting binary data from the uuencoded, Base64 or BinHex text), then passes the binary data off to the plugin for display. Because data is streamed to the plugin while being received from the server, image display is progressive, so on a slow link you'll see the image arrive in bands.

Note: MT-NewsWatcher uses custom plugins to decode and display various binary types, such as images and sounds. Plugins are provided for the display of GIF (including animated GIF), JPEG, BMP and PCX images, and for showing MPEG and QuickTime movies, and for playing MP3 sound files.
To see the NewsWatcher Plugins folder on Mac OS X, you have to look inside the application package. To do this, context-click on the application icon in the Finder, and choose 'Show Package Contents'. In the resulting window, open up the 'Contents' folder, then the 'MacOS' folder, to see the application and the 'NewsWatcher Plugins' folder.

Inline Images

As example, here's an image posted to alt.binaries.pictures.fractals.

Note that MT-NewsWatcher has replaced all the encoded text between the begin and end of the image, and replaced it with an image decoded by the JPEG plugin.

Images are shrunk down to fit within the width of the article window, and the window is resized if necessary to show as much of the image as possible. If you find that the article window continually resizes when viewing images, you can click the padlock icon in the lower left corner to lock the article window size.

Inline Movies and Sounds

As well as images, MT-NewsWatcher comes with plugins that can display movies and sounds using QuickTime. Most of these will be displayed progressively, allowing you to start playing the movie, or MP3, while it is being downloaded.

Bandwidth permitting, you can download multiple movies and MP3s at the same time. Since most movies and sounds are large, they are posted in multiple parts. MT-NW does the work of putting the parts together, though you will only get inline display when you open a binary for which all parts are present.

Note: Because of issues in QuickTime, it is not recommended that you try to display more than one inline MPEG movie at a time. The movies may have display problems, or MT-NewsWatcher might crash. This problem does not affect MP3s or QuickTime movies (.mov files).

Inline binary files

If MT-NewsWatcher cannot find a plugin to decode a binary, it will use its built-in 'embedded file' view for that binary. In this case, the binary appears as a file icon in the body of the article:

MT-NewsWatcher obtains an icon for this file using the File extension mappings specified in Internet Config (or the Internet Control Panel).

You can drag this file to the Finder to get a copy of the binary, or double-click it to open it in its owning application.

Saving files to disk

Whether or not the binary can be display with a plugin, you can save the binary file to disk in the same ways. Any of the following ways are available:

• Click on the image or file icon, and drag it to the Finder. Dragging from an image will make a picture clipping in the finder. Dragging a file icon will make a copy of the file in the Finder.

  Tip: If you want to get a copy of the image file, rather than a clipping, hold down the option key when dragging the image.

• Double click the image or file icon. It will be copied to the folder you specify in the preferences for saving binaries (in the Viewing Binaries preferences panel), and opened in the owning application. This application is found using Internet Config mappings and the file extension.

• Control-click on the image or file icon. In the context menu that comes up, choose one of the Save or Load options.

• With the article window foremost, use the Save Binaries... item on the File menu. MT-NewsWatcher will prompt you to save each binary in the article.

Turning off inline images

If you are on a slow connection and don't wish to download data for binary attachments, then you can turn this feature off in the 'Viewing binaries' Preferences panel. Uncheck the box labelled "View images inline".

If you have viewing inline binaries turned off, and you fetch an article with a binary, then you'll see an icon in the article window:

You can click this icon to extract the binary for this article; it will get passed to the decode helper application, and saved in your downloaded binaries folder.

Mass Extracting Binaries

If you don't want to have to open individual articles to view or save binaries, then you can do this en masse from the subject window, by selecting the articles that you want to extract, and choosing the Extract Binaries item from the News menu.

Extracting binaries in this way bypasses MT-Newswatcher's internal decoding and viewing routines, and saves the raw article text out to a file which is then decoded by one of the helper applications that you specify. You can then use the resulting images, sounds etc all at one time.

This image show a subject window for a binaries group (in this case, a group containing MP3 binaries). In this group, most of the binaries are posted in multiple parts, because they are too large to be posted in one big article.

Note that MT-NewsWatcher uses various marks on each thread to show you the status of that thread.

Multiple Part Binaries

The Usenet rules say that individual articles should be reasonably small. For this reason, articles with large attached files are usually posted in several parts. There are no formal rules for identifying parts, but there is an informal convention which most people follow most of the time. Under this convention, each part of a multiple part posting contains a "part identifier" somewhere in the subject of the article. The part identifier is usually in one of the following formats:

For example, you might see a four part posting in comp.binaries.mac with the following subjects:

• My awesome new Wild Tetris game (part 1 of 4)
• My awesome new Wild Tetris game (part 2 of 4)
• My awesome new Wild Tetris game (part 3 of 4)
• My awesome new Wild Tetris game (part 4 of 4)

The first part of a multiple part posting usually contains a short description of the attached file, followed immediately by the first part of the unreadable BinHex or uuencode text. The remaining parts of the posting usually contain only encoded text, nothing readable.

Sometimes you see a part number 0 which contains just the description, with the encoded text starting in part number 1.

When you combine all these conventions, the result is complicated. If you were to try to extract the attached files from a multiple part binary posting by hand, you would have to do all of the following:

• Locate the parts.
• Get all the parts from the news server.
• Put the parts into the correct order.
• Combine the parts together into a single file.
• Separate the descriptive readable text sections and intermixed news article header sections from the BinHex or uuencode sections.
• Decode the BinHex or uuencode text.
• Expand and decompress the archive file.

Fortunately, MT-NewsWatcher takes care of all of these details for you, and more.

Multiple Part Postings and Subject Window Threads

When MT-NewsWatcher opens a subject window for a group, it sorts all the parts of each multiple part posting together and puts them all into their own thread. With collapsed threads, you see only a single line for each multiple part posting. If you expand such a thread, you see all the parts in proper order as individual articles within the thread.

When you are reading articles in a binary group, use the Next Thread command when you finish reading a description in an article and want to read the next one. This command skips over all the other parts in the multiple part posting.

Extracting Binaries in Subject Windows

You can use the Extract Binaries command in subject windows to extract binaries for many threads at once. Select the article or articles from which you want to extract binaries, then use the command.

When you tell MT-NewsWatcher to extract binaries like this, it locates all the other parts of the posting (if any), fetches all of them from your news server, extracts the encoded text from each article, and saves the resulting data to a file.

By default, MT-NewsWatcher presents a file save file dialog whenever you extract binaries. If you wish, you can set a default folder for extracting binaries. To do this, use the Viewing Binaries preferences topic.

Tip: To force a standard file dialog to appear even if you have a default folder, hold down the Option key while you select the Extract Binaries command or while you click the attached file icon.

To extract all the binaries in a group, open the subject window for the group. Use the Select All command in the Edit menu to select all the articles in the window. Then use the Extract Binaries command to extract the binaries in a batch. In a large group, this operation can take a long time. If you wish, you can let the operation run in the background and work in some other program in the foreground, or keep reading news in another group.

Note: Older versions of MT-NewsWatcher used "helper applications" to convert the encoded data (like Uuencoded or Base64 text) into the resulting file. MT-NewsWatcher now handles this internally, which is faster. However, if you find that internal decoding fails for some reason, you can go back to using the external helper apps. Click the "Advanced..." button in the "Viewing binaries" preferences panel to set this option.

Extracting Binaries Manually

Sometimes MT-NewsWatcher's heuristics for locating and sorting binary parts don't work, especially when the poster doesn't follow the conventions for naming the parts, or when replies get mixed in with a binary thread. In this case, you must use the subject window and extract binaries "manually". With this method, you must do a bit more work, but you have more control over the process.

First, select all the parts of the article in the subject window. Command-click the parts to select them. You must do this by hand. NewsWatcher doesn't find the parts for you in this case. If NewsWatcher's heuristics don't work, the parts are probably in different threads in the subject window, and you may need to expand threads to find them. After selecting the parts, use the Extract Binaries Manually command in the Special menu. NewsWatcher shows you the list of subjects you selected in a dialog. Use your mouse to reorder the parts if they aren't in the right order. Note that you can also resize the dialog if the subject lines are very long, and click on the column headers to resort.

The numbers on the left-hand side are article numbers, which show the order in which the parts arrived on your news server. These are useful as an ordering hint.

When you are done rearranging, click the Extract button. NewsWatcher fetches the parts in the order you specified, writes them to your hard drive, and runs the helper program to expand the file.

Extracting binaries manually using a subject window is the only way to bypass NewsWatcher's heuristics to get total control over part selection and ordering. It is inevitable that NewsWatcher's heuristics will fail sometimes, given the lack of any formal standards for identifying parts. We have tried, however, to tune the heuristics so that they work most of the time with the major binary groups.

Missing Parts and Incomplete Threads

When you try to extract binaries, NewsWatcher may tell you that some parts are missing. This is a common problem, because the parts don't arrive at your news server all at the same time. For example, your server might have only the first five parts of a ten part posting. In your subject window, if you expand the thread, you only see the first five parts.

To solve this problem, you must wait until all the parts arrive, then extract the binaries.

Threads which are missing one or more parts are called "incomplete threads". To make it easier to tell if all the parts are available, NewsWatcher marks incomplete threads with a "<" character in the subject window, in the same column where the check mark appears for read articles. When the last part arrives, the thread is marked with a bullet character "•".

Another problem is that news servers remove articles when they get old. In this case, for an old multiple part posting, you might see later parts in your subject window, but not earlier parts, because the earlier parts have already been deleted. There's not much you can do about this problem.

In extreme situations, it is possible that a posting with a very large number of parts might have its earlier parts deleted on your news server before the last part arrives! This is very unfortunate. NewsWatcher doesn't have any special features to help with this problem. You should complain to your news administrator and ask him or her to increase the expiration time for that group to prevent the problem from happening in the future.

With multiple part articles, when the final part arrives, NewsWatcher automatically includes all of the earlier parts in the displayed thread. Thus, when the final part arrives, and you want to manipulate the multiple part article as a whole (e.g., extract binaries, or save all the parts to a single file), all of the parts are available in the thread so you can see and use all of them and so NewsWatcher can easily find all of them.

NewsWatcher uses a special "article cache" to remember previous parts of incomplete threads. This cache is stored on the NewsWatcher Prefs file in between runs of the program.

Tip: Don't worry about marking parts of incomplete threads unread because you are afraid they will "disappear". NewsWatcher automatically restores these old parts for you when the last part arrives. This is an important feature of the program. Let it do the work for you!

Note: In these days of binary groups containing over 100,000 articles, the article cache is no longer as useful as it once was. In fact, when reading large binary groups, this cache can become counter- productive: it bloats the size of the prefs file, and can make opening these groups even slower. For this reason, there is now a preference to allow you turn off article part caching: uncheck the checkbox in the Viewing Binaries Preferences panel called Cache headers for incomplete parts.

How binaries are encoded

The Usenet news system was designed to transmit and distribute plain text messages, not files. To get around this problem, various encoding techniques are used to translate files to plain text before attaching them to postings. The most common is called "uuencode" (pronounced "U-U-encode"). This technique is a standard in the UNIX community, and is often used to encode pictures and sounds and other kinds of data files. Also commonly used for images and sounds (like mp3 files) is the Base64 encoding, which is a used when articles are send in MIME format. You may also see articles, particularly binary posts, using a new encoding called yEnc. This encoding format doesn't actually produce ASCII text, but munges binary data just enough that it can be safely sent over NNTP. Finally, a encoding called "BinHex" is commonly used for Macintosh files.

When encoded in these formats, the text that is sent looks like this:

begin 644 Kittens_on_fence.jpg
M_]C_X``02D9)1@`!`0```0`!``#_VP!#``@&!@<&!0@'!P<)"0@*#!0-#`L+
M#!D2$P\4'1H?'AT:'!P@)"XG("(L(QP<*#<I+#`Q-#0T'R<Y/3@R/"XS-#+_

Content-Type: image/jpeg; name="pic8315.jpg"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="pic8315.jpg"

/9j/4AAQSkZJRgABAQEBLAEsAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a
HBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy

(This file must be converted with BinHex 4.0)
:#daTCQ9)CAJZFfPd!&0*9%46593K!*!$!3aK!*!%q`a6593K!!%!!3aKFNaKG3)
!N!3@!!!J)!G-D@CP5'9i!*!Bmid!N!0f!%X"-J(I!*!$"!#3$iB!!!%Jrj!%!3#

=ybegin line=128 size=111401 name=al_larsonbw030_ball.jpg 
)_)=J*:tpsp*++++V+V**)_*m*0./0/.00/011024:44334>896:A>....

The text that results when files are converted to BinHex, uuencode or Base64 format is completely unreadable, and is not intended to be viewed or understood by humans. You should not see any of this confusing and incomprehensible encoded text in MT-NewsWatcher.

In the Mac community, in addition to using BinHex, people often compress and combine files into archives before posting them, using one of the popular utilities like StuffIt or Compact Pro. Stuffit Expander will be able to decompress most such files.

Posting Binaries

You can use MT-NewsWatcher to post binaries to newsgroups too. Before you do this, please check that binaries are acceptable in the groups to which you are posting (they are not in most non-binaries groups), and that you are not breaking any copyright or other laws by posting such material. Also be sensitive to the preferred format for binaries in the target groups.

To attach a file to an outgoing message, choose Attach... on the File menu, or Show Details in the message window, and drag a file from the Finder into the list of attached files.

When you attach a file to a message, you will be presented with a dialog that allows you to choose encoding options for that file. This is the dialog that you will see if you have 'Send message with MIME' turned on in the Message Options preferences. If MIME is turned off, you will see a different list of encoding options.

First, in this dialog, you should specify a file name for this attachment, which will be the name used when sending the message. By default, this is the name of the file in the Finder. You almost always want to ensure that the filename has an appropriate file extension (e.g. ".gif" for GIF files, ".jpg" for JPEGs and so on). This is because non-Macintosh platforms use this file extension to identify the type of the file.

Second, you need to choose an encoding for this file. MT-NewsWatcher will have made a best guess for the encoding to use (making use of information from Internet Config). You will need to know a little about the format of Macintosh files to make an informed decision here.

Note: Macintosh files have two parts, or "forks" to them. The data fork is used to store the bulk of the file's data, for example the contents of a word processor document, the image data for GIF or JPEG files, or the contents of a text file. The resource fork is often used to store data used by applications when viewing files. When Macintosh files are transferred to other platforms, like Windows, it's often only the data fork which is important.

Sending with MIME

"MIME" (Multipurpose Internet Mail Extensions) is a way to add extra information to internet messages (in mail or news) which allows different types of data to be encoded in the message, and makes it easier for software to determine what is in the message.

In MT-NewsWatcher, you can ensure that messages that you post contain MIME information by checking the Send with MIME checkbox in the Message Options Preferences panel. It's highly recommended that you do this.

If you are sending with MIME turned on, then you get these options:

Use of BinHex in MIME-formatted messages is discouraged. You should use AppleSingle or AppleDouble, depending on whether the data fork of the file you are sending is in a format that people on other platforms can handle.

Note: When sending binary attachments in a MIME-formatted message, MT-NewsWatcher creates a message with a Content-Type of multipart/mixed, with the message text in a text/plain section, and the binaries in one or more sections of the appropriate MIME type (e.g. image/jpeg). If the message has to be split into multiple parts, then this message is enclosed in a message/partial envelope, and subsequent parts receive the appropriate message/partial headers.

Sending without MIME

If sending with MIME is turned off, these are your options:

The choice here is easier; if you are posting files that are only useful to Macintosh users, use BinHex, otherwise use Uuencoding.

At the bottom of the attachment dialog is an entry field in which you enter the maximum number of lines to post in a single part. If the article is larger than this number of lines after all the attachments have been encoded, then MT-NewsWatcher will post multiple parts. MT-NewsWatcher estimates how many parts will be required given the current line threshold.

Note: This estimate of the number of parts is accurate for most types of encoding. However, because BinHex encoding involves some extra data compression, the estimate with BinHex attachments should be considered an approximation.

When you are done choosing encoding options, click the OK button, and that file will be attached to your outgoing message. To see the list of attached files, Show Details on the message window, and click on the Attachments tab if necessary. To remove an attachment from this list, select it an hit the delete key. You can also double-click on an attachment to change its encoding options, or to remove it.

Tip: Sometimes you might need to repost just a few parts of a multipart binary post. This can be done by holding down the Option key when posting the message; you'll see a dialog that allows you to post one or more contiguous parts. It's important when doing a repost that the message body has the same number of lines as the original (so the reposted parts match up); if you anticipate that you'll need to do a repost, save such messages before posting.

Binary-related Preferences

There are a number of preference options that affect binary viewing, and binary posting. An overview is presented here; these options are discussed in more detail in the Preferences chapter.

The "Viewing Binaries" preferences panel contains settings that enable inline binary viewing, and allow you to specify a default folder for saved binaries. This folder is used both when you use the Extract binaries command, and when double-clicking on an inline binary to save it and load into the viewer program.

There are also options to automatically open any binaries that you download (so long as they aren't applications, which might be harmful to your system), and to decompress archives, like .sit or .gz files.

The options in the Advanced dialog in this panel specify whether to use external decoders, and if so, which helper applications are used to decode BinHex encoded, uuencoded, and Base64 encoded binaries. For any of these, click on the Set button to choose a new helper application.

For a list of suitable helper applications, see the MacOrchard helpers list.