I was up late on IM with a much-younger computer programmer and he asked “Damn, there’s a lot going on. Is it always like this?” Well, no, it hasn’t been. But in the future, it may be.
Near as I can tell, we’re simultaneously at inflection points in programming languages and databases and network programming and processor architectures and Web development and IT business models and desktop environments. Did I miss anything? What’s bigger news is that we might be inflection-point mode pretty steadily for the next few years.
Programming Languages · Up till not too damn long ago, for a big serious software project you could pick Java or .NET or, if you really liked pain, C++. Today you’d be nuts not to look seriously at PHP, Python, and Ruby. What’s the future mind-share of all these things? I have no idea, but that decision is being made collectively by the community right now.
By the way, on this subject, check out the Interview with the TIOBE guy; interesting stuff, even though I disagree with some.
Databases · CouchDB. SimpleDB. BigTable. Need I say more?
No, I don’t think relational databases are going away anytime soon. But I think that SQL’s brain-lock on the development community for the past couple of decades has been actively harmful, and I’m glad that it’s now OK to look at alternatives.
Will the non-relational alternatives carve out a piece of the market? I suspect so, but that decision is being made by the community, right now.
Network Programming · CORBA is dead. DCOM is dead. WS-* is coughing its way down the slope to dusty death. REST, they say, is the way to go. Which I believe, actually. Still, there’s not much yet in tooling or best practices or received wisdom or blue-suit consultants or the other apparatus of a mainstream technology.
So what are they going to be teaching the kids, a few years hence, the right way is to build an application across a network full of heterogeneous technology? That’s being worked out by the community, right now.
Processors · Moore’s law is still holding, but the processors get wider not faster. Now that the best and the brightest have spent a decade building and debugging threading frameworks in Java and .NET, it’s increasingly starting to look like threading is a bad idea; don’t go there. I’ve personally changed my formerly-pro-threading position on this 180º since joining Sun four years ago.
We still haven’t figured out the right way for ordinary people to program many-core processors; check out the inconclusive results of my Wide Finder project last year. (By the way, I’ve now got an Internet-facing T2000 all of my own and will be re-launching Wide Finder as soon as I get some data staged on it; come one, come all).
And I can’t even repeat my crack about the right answer being worked out right now, because I’m not actually sure that anyone has a grip on it just yet. But we’re sure enough at an inflection point.
Web Development · Used to be, it was Java EE or Perl or ASP.NET. Now all of a sudden it’s PHP and then Rails and a bunch of other frameworks bubbling up over the horizon; not a month goes buy that I don’t see a bit of buzz over something that includes the term “Rails-like”.
It seems obvious to me that pretty soon there’s going to be a Rails++ that combines the good ideas from RoR with some others that will be obvious once we see them.
Also, that some of those “Rails-like” frameworks, even if they’re not a huge step forward, will get some real market share because they’ll have some combination of of minor advantages.
Once again, I can’t say it’s being worked out right now, because for right now I see a pretty uniform picture of Rails’ market share advancing steadily. It won’t last.
Business Models · Servers, they’re easy to understand. Blue-suited salesmen sell them to CIOs a few hundred thousand dollars’ worth at a time, they get loaded into data centers where they suck up too much power and HVAC.
Well, unless you’re gonna do your storage and compute and load-balancing and so on out in the cloud. Are you? The CIOs and data-center guys are wrestling this problem to the ground right now.
And as for software, used to be you shipped binaries on magnetic media and charged ’em a right-to-use license. Nope, nowadays it’s open-source and they download it for free and you charge them a support contract. Nope, that was last century; maybe the software’s all going to be out there in the cloud and you never download anything, just pay to use what’s there.
Personally, I don’t think any of those models are actually going to go away. But which works best where? The market’s working that out, right now.
Desktops · As I wrote a couple of months ago: how long can the public and private sector IT management continue to go on ignoring the fact that in OS X and Ubuntu, there are not one but two alternatives to the Windows desktop that are more reliable, more secure, more efficient, and cheaper? More or less everybody now has a friend or relative that’s on Mac or Linux and is going to be wondering why their desktop can’t be that slick.
What’s going to happen? I don’t know, but it’s going to be dramatic once we get to the tipping point, and I think we’re approaching it right now.
Will It Always Be Like This? · You know, just maybe. Our mastery of the IT technologies is still in its brawny youth, with lots of low-hanging fruit to be snatched and big advances to be made. And these days, with the advent of blogs and unconferences and all those new communication channels, our thought leaders are busy chattering at each other about all these problems all the time, 24/7/365. The gap between the leading edge and technology that’s actually deployed in the enterprise is as wide as it’s ever been and to me, that feels like a recipe for permanent disruption. Cowabunga!
Comment feed for ongoing:
From: Andrew Walkingshaw (Apr 24 2008, at 17:10)
On the non-relational data store front, I wonder if there's a tie-in with what the Linked Data guys are doing - and that means SPARQL.
[link]
From: Michael Neale (Apr 24 2008, at 18:49)
"Programming Languages · Up till not too damn long ago, for a big serious software project you could pick Java or .NET or, if you really liked pain, C++. Today you’d be nuts not to look seriously at PHP, Python, and Ruby. "
You know - there are some people in the world that don't write database backed web apps. Just a thought...
[link]
From: John (Apr 24 2008, at 19:43)
Tim, I never see you write much about virtualization (ala VMware or containers). That's a technology that's solving big problems. Thoughts?
[link]
From: robert (Apr 24 2008, at 19:46)
Relational databases are not harmful. You really should justify such poppycock, if you can.
They just haven't been used by folks who really know what they're doing. So, there is another inflection point: the RDBMS and the solid state disk multi-core multi-processor machine. In the hands of folks who know what 5NF means, a fully normalized, zero latency, zero redundant, full integrity data system. Nothing from the "non relational" world of data will be able to compete. Call it the minimal cover of data systems.
The other inflection point: the days of cheap power are over. The solid state relational database machine thrives here, too. xml bloatware, not so much.
Take that to the bank.
[link]
From: Alex (Apr 24 2008, at 21:02)
How am I supposed to use REST for a system that uses both HTTP- and MOM-based interactions in every transaction?
How should I do signing or encryption when there might be three or more hops, some non-HTTP, between the ultimate originator of an invocation and the ultimate recipients?
What if one of the intermediaries through which my messages travel is not fully trusted?
WS-* isn't going away, it's just overkill for the easy stuff.
[link]
From: Pedant (Apr 24 2008, at 21:08)
I think you mean critical point. An inflection point is where the change in slope changes from negative to positive. A critical point, on the other hand, is where the SLOPE change from negative to positive. At the top of hills and the bottom of valleys are critical points. About half way up and down are where the inflection points are usually found.
[link]
From: kbob (Apr 24 2008, at 23:18)
Alvin Toffler described it as future shock in the 1960s. Ray Kurzweil has given it a mathematical model: complexity doesn't grow linearly, it grows exponentially. That model applies to organisms, ecosystems, cultures, and technologies.
So yes, I expect us to be seeing even greater range and rate of technological changes next year and next decade.
[link]
From: JeanHuguesRobert (Apr 25 2008, at 01:57)
That was brilliant! Congratulations.
Some additional points:
- Is the desktop metaphor becoming irrelevant? Will some "Browser as a Service" displace it? Like Mozilla's announced initiative to store Firefox's bookmarks in some "in the cloud" service, versus locally on your PC.
- Is the software crisis, i.e. bugs, going to be the solution to privacy? Like: "That's not my IP address, it must be a bug". Does reliability really matters after all?
- Is mail dead or are we going to have "mail II, the empire strikes back"?
- Is text dead and voice/video the way future, or is it the opposite?
- When will IP v6?
- Are western software developers doomed or will demand exceed supply even with indian, russian and chinese guys factored in?
[link]
From: Marcus L (Apr 25 2008, at 06:25)
Hello, in regards to work on making it easier to take advantage of multi-core CPUs, check out the Parallel Extensions to the .NET Framework:
http://msdn2.microsoft.com/en-us/magazine/cc163340.aspx
Manually threading things or even using helpers is pretty painful, but it looks like this project will make many aspects of utilizing multi-core processing much easier.
[link]
From: Karl Waclawek (Apr 25 2008, at 07:45)
REST/SOAP/POX+HTTP are only usable for coarse grained interfaces. For anything else you will have to use something like ICE or AMQP.
[link]
From: Jeremy Bowers (Apr 25 2008, at 07:57)
Tim didn't say relational databases are harmful, he said (and I quote) "But I think that SQL’s brain-lock on the development community for the past couple of decades has been actively harmful".
If you are a relational "true believer", you should be in total agreement with this statement. We are 10 years overdue for a new query language that fixes the numerous and longstanding problems with SQL, not least of which is its brain-dead half-cocked idea of what "relational" is!
The problem is that SQL has had a monopoly, and I hope that one of the side effects of people starting to look around at non-relational technologies will be that people will also start to consider a relational database query language that's *actually* relational, instead of just posing as such and sucking up all the oxygen.
[link]
From: Jay Carlson (Apr 25 2008, at 08:13)
REST is the way to go now because it shoves your face right in the full terror of network programming. As a species, we're remarkably good at creating comforting false abstractions to make daily life seem more sane. Eventually REST will get papered over by enough random cargo cultism, Java libraries, and CIO magazines to make it pleasant and at that point it will have lost something.
Because I've been using Python/Lua/Tcl/etc for "real problems" for ~18 years, I'm amused at their new-found acceptability. I wonder if a lot of the flavor of Rails is anti-Java flavor---things like "choice of names affects behavior"---and once we recover from our Java hangover, Ruby will look like one of those embarrassing yearbook photos.
[link]
From: Mike Norman (Apr 25 2008, at 08:14)
re: Processors and Threading
Dr. Cliff Click over at Azul Systems is doing a lot of the 'heavy lifting' for better threading for Java and multi/mega-CPU boxes with his work on lock-free Data Structures and Software Transactional Memory
[link]
From: Keith Gaughan (Apr 25 2008, at 09:15)
@robert: Tim never said relational databases were harmful, he said that *SQL* was harmful, something Codd would've agreed with.
@Karl Waclawek: Erm, mind expanding on that?
[link]
From: Paul Tiseo (Apr 25 2008, at 13:53)
@Keith Gaughan and Jeremy Bowers: I agree with robert. My interpretation of Tim's comments are that his "poo-pooing" include relational databases broadly, and not just SQL. Like many developers, I think he's conflating RDBMSes, SQL and the Relational Model. Tim does ask: "Will the non-relational alternatives carve out a piece of the market?" That seems to indicate that the above paragraph is not about SQL alone. Most people don't appreciate the reasons-d'etre of the Relational Model and can't differentiate it from the flawed-to-various-degrees RDMBSes.
Yet another common trap: arguing OLAP vs. OLTP. Put the work in the writes or in the reads. You can't avoid the work, though! :)
Also, another common developer conflation: comparing conceptual models like the Relational Model to physical implementations like BigTable. Don't get me wrong. BigTable may be fine for some uses, especially "data-simple" apps like a search engine, but not others. I'd love to see a fully-denormalized, one-table ERP built on BigTable! Goodbye data integrity!
I know, it's all been said before, but there are new impressionable developers everyday that recycle the misunderstandings...
- alphadog
[link]
From: max stirner (Apr 25 2008, at 14:35)
i enjoyed your post; i am also excited about the current state of computing.
the desktop computing tipping point is certainly nigh, thanks to major mistakes by the monopolist, and enormous enthusiasm and hard work on the part of the community.
if you hold MS shares, sell now! the ship is s(t)inking.
if you havent tried linux yet, try linux! be free
[link]
From: Karl Waclawek (Apr 25 2008, at 15:32)
@Keith Gaughan: Well, to expand a little:
If call overhead (latency) is of the same order of magnitude or larger than the actual work done for the operation, then you spend a significant portion of your time and CPU cycles communicating rather than doing work.
A coarse grained interface means that the work done for each operation is substantial (as opposed to just calling a getter or setter, for instance) and outweighs your communication overhead.
[link]
From: len (Apr 26 2008, at 08:48)
"Alvin Toffler described it as future shock in the 1960s. Ray Kurzweil has given it a mathematical model: complexity doesn't grow linearly, it grows exponentially. That model applies to organisms, ecosystems, cultures, and technologies."
That's not quite right. In a complex system, there are multiple layers of feedback, some accelerating and some retarding or even destroying emergence and sustainability of other layers. It isn't exponential or even linear complexity except in the model. It is lumpy.
Wiener came closer to the mark.
[link]
From: Carl Meyer (Apr 26 2008, at 17:14)
@Michael Neale: That might be a fair retort to PHP alone, but Python and Ruby (especially Python) are everywhere, not just on the web in front of databases...
[link]
From: robert (Apr 26 2008, at 18:57)
@Paul: we thank you for your support. If you remember where that quote comes from...
@et alia: Tim made specific reference to flat file systems, none of which move closer to what Dr. Codd had in mind; back to what Dr. Codd was condemning in fact. xml as datastore is such as well.
What none have demonstrated is why the reactionary retreat to what COBOL programmers did in 1970 is *progress* in 2008. My take: coders never learned relational systems, or never understood them, in university or since; and they intuitively see that intelligent datastores make coders more or less redundant. No intellectual argument, just pique.
[link]
From: Tom Smith (Apr 27 2008, at 02:34)
I agree completely with the problem being the "R" in RDMS... It's just that object databases, such as the Newton's soup or the DB lurking in Zope haven't quite got the ease of use/speed of the more traditional alternatives.
[link]
From: Aristotle Pagaltzis (Apr 27 2008, at 09:17)
Tom Smith:
You are joking, right? The widely used “R”DBMSs are relational only in title. I prefer to call them SQL DBMSs.
[link]
From: paddy3118 (Apr 30 2008, at 22:58)
Hi Tim,
Just a thought, that what you say about programming in your processors section is not addressed by the languages you mention in your programming languages section. We have a problem in in being able to harness parallel programming effectively, if I were to buy a four-core desktop, most of the power would sit there being wasted.
- Paddy.
[link]
From: Nick Johnson (May 01 2008, at 03:07)
For what it's worth, I think the scope of your "wide finder" project was too small. If you instead looked at processing, say, 10 billion log entries, distributed over a whole cluster of web servers, there's a whole lot of interesting and fast solutions (MapReduce is the most obvious) that can do this very efficiently and scale well with the data. They're just not practical when you're sorting such a small amount of data.
[link]