A few days ago, our CEO Jonathan Schwartz sent a letter to SEC Chairman Christopher Cox calling for SEC financial-disclosure regulations to allow for publishing material financials on the Web. It’s obviously a good idea, but there are some implementation issues. (Hey, I’m an engineer, I can’t help it.)
It seems self-evident that the Web ought to be at least as good as the traditional vehicles (press releases, telecons) at getting important business news out to everyone at the same time. But for any really material information, we’ll need a commitment to a little more than “posting it on the Web site”. Where on the web site? And does the world get any help in noticing that it’s arrived?
Need For Feed · It’s no accident that Jonathan wrote specifically of posting to a blog. Because a blog has an RSS feed, and people who want to follow the news can subscribe to it, and they’ll find out if anything shows up there. I’d say that if a company plans to release information on the Web, at a minimum you’d want to require that it appear at a well-known location, and that that location have a feed. I’d further urge that an Atom 1.0 feed be required; it’s a stable Internet Standard and you really don’t want to have to worry about the RSS propensity for silent data loss.
Need For Speed · Even a nice clean well-known feed doesn’t quite solve the whole problem. Your typical feed-reader is set up to poll every half-hour or even less often, and there are those in the financial community who are not going to be happy with a potential half-hour’s latency in getting the news.
I can think of one simple brute-force way to approach the problem, and another that’s a little more sophisticated. The simple solution is, assume that everyone who really cares will want to poll that material-news feed every few seconds. So, you stage Jonathan’s feed, not on the ordinary blogging infrastructure, but on a hyper-fast cache that can take that kind of transaction load; there might even be a business opportunity here for some infrastructure player to offer this kind of special-purpose staging.
If you want to get fancy, you could use something like the proposed new Atom-over-XMPP trick. The idea is that people who want ultra-low-latency feeds don’t poll, but set up a persistent connection to the provider’s server, which pushes entries down the wire the moment they become available. This is elegant in theory; in practice I’d bet on the brute-force polling approach, at least off the top.
Comment feed for ongoing:
From: Domas Mituzas (Oct 05 2006, at 01:27)
The web already has pinging services, which provide quite reasonable approach to pushing data.
There's no need to poll every blog you read - simply checking status on ping aggregators would do. Now what one needs - proper services, that would provide instant notifications about blogs changed. Probably someone somewhere is doing that already. You don't need XMPP, web is already all there to help you.
[link]
From: wx (Oct 05 2006, at 04:41)
On the ultra-low-latency idea: I think LiveJournal provides some service like this, where you connect to the "latest posts" feed and try to capture them all. They even implement a mechanism to notify the reading side that it's lagging behind and is going to lose few messages.
[link]
From: katre (Oct 05 2006, at 06:20)
Hmm, interesting idea, but I worry about leaving the data with the company. Not that many people would do this, but surely if Enron could have gone back and quietly adjusted some numbers on previous financial statements and press releases, they'd have considered it. The benefit of the current system is that all interested parties have a copy, and can compare later if shenannigans happen.
[link]
From: seav (Oct 05 2006, at 07:24)
Hmmm, "publishing material financials"? Maybe you meant "publishing financial materials"?
Anyway, what about if the company updates the financial feed at regular intervals? Say every half-hour on the dot? That would probably stem the tide of people who will poll the feed every few seconds. The new problem then is that at those regular times, the traffic to the feed would peak like crazy during those times and be quite quiet the rest of the time. I wonder if web servers can handle such regular spikes.
[link]
From: Stephen De Gabrielle (Oct 05 2006, at 08:08)
we’ll need a commitment to a little more than “posting it on the Web site”.
Have you heard of an Institutional Repository? They are all the rage in Universities, but may fit your organisations needs as to showing a bit more committment. href="http://www.dspace.org/ Dspace is a good example to take a first look at, as is http://fedora.info/ Fedora .
Wikipedia has a reasonable http://en.wikipedia.org/w/index.php?title=Institutional_repository&oldid=77198025 definition of 'Institutional Repository' , but the best bit about it's definition is noting the part that 'the the Open Archives Initiative (OAI) and its http://www.openarchives.org/ Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) ' ; which would work well in assosciation with ATOM in addressing the Where on the web site? part of the question in a more long term sense.
How do you let the world know about the contents of your website? Your feed only goes back a couple of weeks.
Have you considered http://www.modoai.org/ MOD_OAI as a way of improving how google gets people to your site? MOD_OAI lets regular sites make use of OAI-PMH to help google (and others) improve their searches of your site. (see http://www.google.com/support/webmasters/bin/answer.py?answer=34655 heres how to tell google about your OAI-PMH metadata )
Stephen
[link]
From: Tim Bray (Oct 05 2006, at 08:10)
Thanks to all who wrote in to point out the broken link.
[link]
From: Dominic Jones (Oct 05 2006, at 11:36)
Tim,
Seems we have to separate scheduled releases of information from unscheduled ones.
When you know you will report information on a specific date at a specific place on the Web, you can forewarn people via a web posting, email alert and regular RSS feed notice etc. That way everyone knows the information is coming and no one is disadvantaged relative to the current process of using a wire service to make these announcements.
For unscheduled or surprise material annoucements, you have push the information out, and that's where the newswires have played a vital role.
Essentially, what every company needs now is its own "open newswire" that aggregators or end users can subscribe to. Cut out the wire services, which really have had a wonderfully free ride at the pleasure of the regulators. (Hey, Buffett bought Business Wire, so it's got to be a great business).
Your technical solutions to this particular problem sound like a terrific business opportunity for someone.
[link]
From: Adam Kalsey (Oct 05 2006, at 14:19)
Atom over XMPP sounds interesting, but is obviously some time off and needs client support to boot.
But since you're thinking along those lines, you might be interested in Feed Crier (http://feedcrier.com/), a service I run that watches feeds and sends alerts via IM. Feed Crier uses several methods to try and ensure that feed alerts are sent as they're published. Everything from brute-force polling to heuristic-determination of optimum polling intervals is used. Alerts are sent within minutes of publication, and I'm working to get "minutes" down to "seconds."
AIM is the only IM platform supported right now, but others are coming.
[link]
From: Benjamin Carlyle (Oct 06 2006, at 15:31)
I have been thinking about subscription for a while. I recently posted a <a href="http://soundadvice.id.au/blog/2006/09/30#jep-0060">blog entry</a> referring to some older attempts to achieve this, and generally dissing XMPP's JEP-0060 and its lack of discussion about service guarantees.
I listed the following principles that I think need to be incorporated into an internet-scale subscription mechanism:
Summarisation. This is the organised discarding of information within an update stream to ensure that slow clients recieve as much information as their connection characteristics permit. This avoids overall system overload due to infinite buffering.
Differential flow control. A slow client should not prevent fast ones from getting updates, nor cause them to recieve the slow client's summarised stream.
Localised resynchronisation. A client need not reach back to the origin server for the current resource status if its immediate server is already handling the subscription.
Patch updates. For large resources (especially lists), the ability to deliver a message that indicates the change from last time, only. Not the whole state.
Security Measures. Pub/Sub can be a source of denial of service attacks. The subscrpition mechanism must be able to detect when its notifications are being treated as spam and end the subscription
Benjamin.
[link]
From: M. David Peterson (Oct 07 2006, at 17:50)
@Adam,
>> Atom over XMPP sounds interesting, but is obviously some time off and needs client support to boot.
What's the difference between the required client support and,
and
>> AIM is the only IM platform supported right now, but others are coming.
The claim that "needs client support" is justified when you are speaking in terms of a web-browser -- e.g. it's fair to make such a statement in regards to XForms or SVG where a browser is the primary client interface for an application. Browser's are a ubiquitous expectation, so whether or not each and every requesting browser of significance -- i.e. basically IE, Moz/Fx, Safari, and Opera -- supports a given technology is important. AIM isn't exactly ubiquitous.
In fact, one could easily argue that Jabber/XMPP is a more realistic solution in regards to Instant Messaging as there are TONS of clients that support Jabber/XMPP out-of-the-box, a list in which is growing every day due to the fact that its an open standard in which anyone can freely provide support for. The fact that GoogleTalk is built on Jabber/XMPP coupled with the fact that they have invested time and money into developing the wonderful Jingle protocol suggests that if you want to be a player in the IM-business in the next 5-7years you'd better provide direct support for at very least Jabber/XMPP (though Jingle would be a good idea as well, the other option being SIP) otherwise you will be incompatible with the other systems, and therefore insignificant.
I'm not attempting to dis on your application, but I am attempting to help bring to light the fact that closed, proprietary IM systems that do not provide open support of Jabber/XMPP will quickly become dinosaurs in a VERY short space of time, so supporting Jabber/XMPP first makes a ton more sense than does something more proprietary like AIM, Yahoo!, or MSN/Windows Live Messenger.
Add to this the fact that you can freely access a vast array of Jabber/XMPP libraries that provide support for inter/intra-application communication and the possibilities that Jabber/XMPP provides is something that you can not ignore and/or blow off as "requires client support" -- well yeah, everything requires client support in one form or another, and of all of the messaging protocols out there, only one of them already has support for EVERY major platform while providing royalty-free usage of the protocol/XML-format itself.
[link]
From: Anthony B. Coates (Oct 11 2006, at 03:22)
Finance is really big business, and it's an increasingly fast business. With algorithmic trading now the rage, 20ms can be the difference between gaining or losing millions on a transaction.
For this reason, although polling the Web site might be OK for individual investors who aren't into day trading, I can't see it satisfying the big investors. Polling is fine where it doesn't matter if you see the data now or in an hour, but I don't think it's the right solution here.
The only way I can see Web publication working is if the publishing company pre-announces the publication, and guaranteed the publication time to the nearest millisecond. That might work, although I myself would be inclined to set up a private server for large investors, to ensure they can get a timely response to their requests for the information.
Cheers, Tony.
[link]
From: John Cowan (Oct 11 2006, at 09:30)
Tony, a computer can react to an XBRL report in milliseconds -- maybe -- but a human being can't react within milliseconds, probably not within seconds, to the kind of things said on a conference call. I don't think that's a realistic timeframe for this particular application.
[link]