Aggregator Market Share

In reply to one of my Browser Market Share postings, Ian Brown wrote to point out that with an increasing portion of the traffic going through newsreaders, it might be interesting to do some breakdown on that. So I did. [Updated to say the results should be taken with a large grain of salt.]

Here is a list of the top 100 different agents who fetched the ongoing RSS feed in the week ending March 20th. Note that the number for Bloglines is the product of the number of fetches and the number of subscribers. After the numbers, I’ve included the quickie Perl script that generated them.

Comments on whether the right thing is being measured, whether it’s being measured accurately, whether a regular report would be welcome, and ideas on graphics when there are dozens as opposed to just a handful of interesting players, would all be welcome. Later: I got lots. Thus...

The Problem · The problem is, as Scott Henry and Dave Sifry both pointed out, Bloglines is running 24/7 and fetching the feed whether or not anyone is looking; whereas the desktop aggregators mostly run only when the user is sitting in front of them. Scott also observed that if there any inactive Bloglines subscribers, they’ll still show up in the subscription count.

Dave had some good ideas how you might correct for this, and I’m kicking some others around too. Having said that, I do think that in the case of ongoing, NetNewsWire and Bloglines are the big dogs.

Thanks to Joseph Lindsay who pointed to four similar pieces from Richard McManus based on Feedburner stats: 1, 2, 3, 4. Elliott Hughes suggested computing percentages and doing a graph cut off at 5%. Wow, good idea; but for the moment I think 1% would be the place to cut.

448728 55.0% Bloglines (2671 subscribers)
187000 22.9% NetNewsWire
32197 3.9% BlogMatrix
23849 2.9% Firefox
17004 2.1% SharpReader
10959 1.3% NewsGator
10156 1.2% (unrecognized)
8586 1.1% Thunderbird
7626 0.9% RssBandit
5439 0.7% NewsFire
3461 0.4% Shrook
3330 0.4% Java
3067 0.4% FeedDemon
3035 0.4% Sage
2837 0.3% FeedOnFeeds
2713 0.3% Liferea
2702 0.3% BitTorrent
2378 0.3% Feedreader
2029 0.2% Opera
1907 0.2% RssReader
1833 0.2% Radio UserLand
1732 0.2% UniversalFeedParser
1592 0.2% MagpieRSS
1371 0.2% Straw
1365 0.2% RssBar
1311 0.2% NewzCrawler
1215 0.1% Onfolio
1006 0.1% nntprss
975 0.1% BlogBridge
914 0.1% rss2email
781 0.1% PulpFiction
715 0.1% NewsPipe
710 0.1% libwww-perl
707 0.1% Python-urllib
699 0.1% AppleSyndication
669 0.1% Roller
650 0.1% reFeed 1.0 (+http:
613 0.1% rawdog
600 0.1% vw-http
547 0.1% JetBrains Omea Reader 1.0.3 (http:
508 0.1% intraVnews
504 0.1% Meerkat-Collector
495 0.1% Pluck Soap Client
484 0.1% NewsGatorOnline
427 0.1% KNewsTicker
419 0.1% Drupal (+http:
394 0.0% Snoopy
394 0.0% ActiveRefresh
385 0.0% RSSOwl
354 0.0% Frontier
337 0.0% Interesting People +http:
336 0.0% Waggr_Fetcher
336 0.0% SimpleAggregator
335 0.0% Planet ClearAirTurbulence +Unconfigured Planet Planet
334 0.0% feedtree.Proxy
331 0.0% PlanetJava.org http:
330 0.0% PlanetJava.org M http:
329 0.0% SearchBlog Crawler. http:
316 0.0% VPSBot1
295 0.0% RssFu
292 0.0% MimirAggregator
267 0.0% LWP::Simple
266 0.0% Awasu
262 0.0% MarineToolbar
253 0.0% Jakarta Commons-HttpClient
248 0.0% FeedBurner
243 0.0% PulpFictionLite
212 0.0% MT::Telegraph::Agent
203 0.0% freenews
186 0.0% LiveJournal.com (webmaster@livejournal.com; for http:
181 0.0% iPodderX
168 0.0% inkdroid
168 0.0% Planet Sun Planet
168 0.0% Planet Sun +http:
168 0.0% Planet Search +http:
168 0.0% PHP
167 0.0% Planet XML Aggregator 0.2; http:
167 0.0% Abilon
166 0.0% HyperDig_Portal
166 0.0% Desktop
162 0.0% JetBrains Omea Pro 1.0.3 (http:
162 0.0% Feedster Crawler
161 0.0% JetBrains Omea Reader (Tokaj) build 500 (http:
160 0.0% MSIE
150 0.0% YahooFeedSeeker
150 0.0% ERA
147 0.0% Temboz (http:
141 0.0% Oddbot
131 0.0% Gush
125 0.0% plagg
122 0.0% PluckExplorerBar
121 0.0% Gregarius
119 0.0% Urchin RSS Aggregator (via libwww-perl
116 0.0% NIF
111 0.0% Sauce Reader
111 0.0% Rojo
108 0.0% TrillianPRO
100 0.0% Planet Fawad +http:
96 0.0% Wget
91 0.0% Journster.com RSS

Here’s the script:

use strict;
my %readers;
my $bloglines = 1;
my $f= '\S+\s+';

while (<STDIN>)
{
    # bloglines multiplier
    if (/^crawler\d*.bloglines.com.* (\d+) subscribers/)
    {
	if ($1 > $bloglines) { $bloglines = $1; }
    }

    s/^$f$f$f$f$f$f$f$f$f$f$f//o;
    my $reader;

    # special cases
    if (/^\"Mozilla.*Firefox/)        { $reader = 'Firefox'; }
    elsif (/^\"Mozilla.*Thunderbird/) { $reader = 'Thunderbird'; }
    elsif (/^\"Mozilla.*\(Sage\)/)    { $reader = 'Sage'; }
    elsif (/^\"Mozilla.* Opera /)     { $reader = 'Opera'; }
    elsif (/^\"Mozilla.*Rojo/)        { $reader = 'Rojo'; }
    elsif (/^\"Mozilla.*Habari/)      { $reader = 'Habari'; }
    elsif (/^\"Mozilla/)              { $reader = '(unrecognized)'; }
    elsif (/^\"BlogMatrix Jaeger/)    { $reader = 'BlogMatrix'; }
    
    elsif (m@^\"([^/]*)/@)            { $reader = $1; }
    elsif (/"-"/)                     { $reader = '(unrecognized)'; }
    elsif (/^\"([a-zA-Z0-9_-]*)/)     { $reader = $1; }
    else
    {
	print STDERR "************ $_";
    }
    $readers{$reader}++;
}

$readers{"Bloglines ($bloglines subscribers)"} =
    $readers{Bloglines} * $bloglines;
undef $readers{Bloglines};

my $total = 0;
my $reader;
foreach $reader (keys %readers)  { $total += $readers{$reader}; }

foreach $reader (keys %readers)
{
    printf "$readers{$reader} %.1f%% $reader\n",
           100.0 * $readers{$reader} / $total;
}

---