How to setup a feed with aioe.org

When an user posts an article, it is initially only available on that user’s news server. Each news server, however, talks to one or more other servers - which are called "newsfeeds" - and exchanges articles with them using permantent connections and a special set of NNTP commands. In this fashion, every article is copied from server to server and eventually reaches every server in the network. These links among NNTP sites are called "feeds" and they assure the propagation of articles across the USENET sites. This implies that every NNTP server needs some newsfeed which sends the remote articles to that host and receives the ones posted by local users. Those who need more informations should read the Wikipedia page about USENET before proceeding.
Those who need some USENET site which is willing to exchange an USENET feed should read news.admin.peering which is a group with this specific topic. Many USENET sites have a peering page that includes all details needed to setup a feed with them, including requisites and condictions.

For exchanging an USENET feed, every NNTP host must fulfil some general prerequisites :

1. A static IP.
   USENET feeds require hosts which don’t change often their IP addresses.
    
2. A reasonably fast internet connection.
   A full text only feed weighs just under 500 MB/day so a modern DSL connection should be enough if it’s dedicated only to this service.
    
3. A news server able to hadle USENET feeds.
   INND > 2.4.0, DNEWS or DIABLO are able to handle incoming and outgoing feeds in a safe manner, programs like leafnode or Papercut NNTP server can’t. We strongly recommend those who still run ancient versions of INND ( < 2.4.0 ) to upgrade to a more modern release as soon as possible.

Before asking us for a feed, newsmasters must also configure their hosts in order to satisfy the following rules :

1. An antispam script must be installed in order to filter the incoming and outgoing feeds. For those who use INND a good option is Cleanfeed which doesn’t require extensive configurations.
2. The outgoing feed (the flow from you to us) has to be filtered against misplaced binaries (binary articles sent to text only groups).
3. A simple set of statistical pages must be available. Those who run INND should make public at least the innfeed and inn_status reports which show how many articles were accepted and sent by the local server in the last days.
4. Every peer must be configured to send statistical reports about incoming feeds to The Official Usenet TOP1000 Servers page. INND includes a program called ninpaths which automatically performs this task but it has to be activated by hand modifying some INND configuration files (notably /etc/news/newsfeeds and /etc/cron.daily/inn2)

 

Those who wish a feed from Aioe.org must send a request to usenet@aioe.org including the technical data needed to configure it and some information about the site. We favour non commercial projects but this isn’t a mandatory rule. All Aioe.org feeds are provided for free.

In order to configure INND to handle an USENET feed with another host, three files must be modified :

1. /etc/news/newsfeeds determines which articles have to be sent to remote hosts
2. /etc/news/innfeed.conf fixes which remote hosts are authorized to receive the articles from the local site
3. /etc/news/incoming.conf sets the remote hosts allowed to send articles to the local site

newsfeeds is the most complex innd configuration file and its syntax is quite obscure for beginners. Every newsfeed is represented by a structure which is divided into 4 elements separated by a colon. Usually, every structure is written using four lines, one per element, with a reverse slash ("\") at the end of the first three lines like a bash command split among several rows. Even if this format isn’t a mandatory convention, it makes easier to detect the most common syntax errors and we recommend to follow it. Those who need to establish a feed with Aioe.org must insert these lines at the end of /etc/news/newsfeeds :

aioe/aioe.org\
       :*,!*.bina*,!*.bain*,!*.dateien*,!*.pictures*,!junk/!local\
       :Af,Ap,C20,G12,U5,<32768,Tm\
       :innfeed!

The first one defines a conventional site name ("aioe") and a path entry ("aioe.org") : only those messages which don’t include in the path the string "aioe.org" will be sent to "aioe".
The second line sets which groups and distributions will be sent to the remote host : all the articles ("*") except the ones posted to binary groups will be transferred to the remote host. Binary groups are defined here as those newsgroups which contain in the name the strings ".bina", ".bain", ".dateien", ".pictures" or finally "junk" that is an useless group for trash (" !*.bina*, !*.bain*, !*.dateien*, !*.pictures*, !junk"). Everybody should note that the initial character " !" excludes from transfer those articles sent to groups which match the following string. The articles which have got a Distribution header set to "local" aren’t transferred ("/ !local\") : this is an USENET convention that allows the end users to avoid the propagation of an article to the other NNTP sites simply adding in the headers :

Distribution: local

In this case, that article will be only available on the same server used to post it.
The third line defines a set of rules which determines whether each article has to be transferred to that peer. If a post doesn’t include in the Path header that string set by the second field in the first line ("/aioe.org\") and if that message is sent to a group which is included in the list of newsgroups to transfer, INND vierifies whether that article matches those rules which are defined in the third line. This line is divided into 7 fields separated by a comma and each one defines a check which is performed by INND before sending every article to the remote site. The first capital letter in each field (ie. "Ap") determines the family of checks and the following minuscule character or number (ie. "Ap") sets the exact tests performed by INND. When many tests are selected in the same family, they can be also written all together : "Afp" is equivalent to "Af,Ap" but the latter form is considerably easier to read and we recommend to choose this one.
The item "Af" configures INND to exclude from transfer those articles which are rejected by the internal antispam filter (probably cleanfeed). Without this field, every article is sent to the remote host even if it was locally rejected due spam and this is definately an useless waste of bandwidth.
Without "Ap", INND checks whether an entry in the Path matches the site name (first field in the first line) or the path string (second field, first line) in order to detemine whether an article has to be transferred. This is dangerous. Large sites usually assign to every peer a mnemonic name which may conflict with a path entry of another host. Those who need to establish a feed with two hosts that use the same domain name with a different top level domain (ie example.org and example.com) must always insert this line in order to allow the transfer. With "Ap", INND checks only whether the path string matches the Path.
"C20" configures INND to transfer an article only if the number of groups it is posted to, plus the square of the number of groups followups would appear in, is no more than 20. This value is optimized for a full text only site. The Wikipedia page about USENET Spam offers some introductory knowledge for beginners.
"G12" excludes from transfer those articles sent to more than 12 groups and "U5" those which include more than 5 groups in the Followup-To header. Nobody needs to send an article to so many groups at the same time and a message with a lot of followups is almost always a flood attempt. Even if these items are less effective than the flag C (see above), they are able to reduce the spam ratio saving bandwidth.
"<32768" has the effect to block those posts which are larger than 32 Kb. The articles sent to text groups must be smaller than 32 KB; larger posts are always misplaced binary articles or attempts to post binary spam in non binary groups.
"Tm" defines how INND has to internally hadle this feed. With "Tm", all the articles are given to an external program (defined by the fourth line) which is attended to deliver them. The following line configures INND to transfer the articles to the remote host through innfeed ("innfeed !"). innfeed.conf is the configuration file used by innfeed to determine which host has to receive the articles. Those who need to establish a feed with us must insert at the end of /etc/news/innfeed.conf the following lines :

peer aioe
{
        ip-name: news-in.aioe.org
	initial-connections: 2
	max-connections: 5
	dynamic-method: 1	
}

This structure configures innfeed to transfer the articles to news-in.aioe.org opening maximum 5 connections and trying to maximize the number of articles per second. When INND determines that an article has to be sent to an other newsfeed through innfeed, it transfers that post to innfeed using an unix pipe. innfeed reads the site name (which must be the same set in the first element in the first line of newsfeeds) and sends that message to the corresponding host. incoming.conf defines which remote hosts are allowed to send articles through an USENET feed to the local site. When a site listed in this file tries to open a connection with the local server, instead of delegating nnrpd to serve that client, INND treats it as a newsfeed and allows that host to transfer articles through the NNTP commands designed for servers. Those who need to setup a feed with Aioe.org should add at the end of /etc/news/incoming.conf an entry like this :

peer aioe
{
	hostname: "news-out.aioe.org"
}