[This fragment is available in an audio version.]
This week saw the cancellation of the Australian Stock Exchange’s long-running effort to build a blockchain-based trading system. Which, oddly, has me thinking of 2016, when AWS decided not to make a strategic investment in blockchain, with my input a contributing factor. It felt like a good story while it was happening.
Since I left AWS in 2020, I’ve been super-careful not to share things from behind the scenes. I can’t actually remember the details of the nondisclosure agreement, but I have strong feelings about the ethics. This story, though, doesn’t reflect poorly on anyone and I’m pretty sure nothing in it is material to any business plans at AWS or elsewhere.
Andy meeting · At some point in mid-2016 I got hauled into a conversation with Andy Jassy. I can’t remember if it was video or f2f, can’t remember how many of his staff were there. There were four of us present who were senior techs, not Jassy staff.
On Bleeker Street, in lower Manhattan.
There’s a reason for the picture; read on.
Andy is an outstanding communicator and was eloquent on this occasion. You have to understand that one of the most important parts of his job was listening to the CIOs and CTOs of huge enterprises explain their problems and concerns.
He said something like this: “All these leaders are asking me what our blockchain strategy is. They tell me that everyone’s saying it’s the future, the platform that’s going to obsolete everything else. I need to have a good answer for them. I’ll be honest, when they explain why it’s wonderful I just don’t get it. You guys got to go figure it out for us.”
Well, OK then. I can’t remember whether it was right there in the room or by email after a short caucus, we got back to Andy along the lines of “We mostly think it’s mostly bullshit and probably not strategic for AWS, but we’ll look harder.”
Before I move along, Dear Reader: There was a dead give-away in Andy’s presentation of the problem. I’ll get back to it later but do you see it?
The hunt · We went looking around the industry from behind our screens and discovered a few things:
Actual working business applications of blockchain were really, really hard to find.
Plenty of blockchain products were on offer, characterized as “polished”, “robust”, “production-ready”, and “regulator-approved”. But if you looked hard at their customer stories, it got pretty vaporous pretty fast.
The throughput of proof-of-waste blockchains was just as bad as we thought.
In practice, the technology is a database. Everyone who did anything had some sort of a database structure mapped over the actual blockchain, with the usual B-trees and so on. It wasn’t obvious how anything would be different if there were something other than a blockchain behind the B-trees.
The Australian Stock Exchange was betting the farm on blockchain, which seemed to prove this was no joke.
A huge, almost incomprehensible, volume of venture capital was flowing into the sector, and that money was localized in the Finance sector, specifically in Manhattan.
AWS was already making a lot of money off blockchain. All these venture-financed companies had to build out infrastructure, and most of them were all-in on cloud, either AWS or GCP (don’t think Azure got much of that biz). So there was a serious flow of cash from VC firms into AWS.
We found ourselves in conversation with the Finance-sector sales organization, naturally headquartered in Manhattan. We got their entire attention right away, because they’d noticed the crazy venture-money flow, and they were getting questions, the same kind that Andy had described, from their institutional clients.
The right thing to do was obvious.
New York, New York! · In August of 2016, four of us went down there for a few days. We talked to big established Wall-Street firms and a few startups.
There’s this amazing thing about working for AWS. You can get the attention of the IT leadership of pretty well any organization in the world and say “Hey, could we drop by and talk about…” and most times they’re happy to do it.
(That weeping you hear is the former me, the one who had two startups and spent months bashing at the doors of enterprises in the usually-vain hope of getting anyone meaningful to look at our technology.)
So, no doors were closed to us. Which was pretty nice, except for it was stinking hot that August and we were on multiple steamy subway trips every day; got back to our hotel rooms pretty well emptied out.
(The pictures decorating this entry are from that trip.)
In a lower-Manhattan bar.
The questions · We really only had two questions, both for the big-finance players and for the startups. “What is it you want to do?” and “How does blockchain help?”
The answers, to our disappointment, failed to shatter any preconceptions. The things they wanted to do were perfectly reasonable. Some of them were damn exciting. They all needed databases. They could all make use of ledger-like data structures, also cryptographic hashing and signing. But, um, why did they need blockchain? Severe lack of clarity on that.
The key moment was when we got in a room with the CTO of this one startup, in Tribeca I think. When I heard their VC funding number I thought it was the valuation, not the investment dollars. The customer list was blue fucking ribbon and don’t you forget it. These guys were razor-sharp.
They presented some of the systems they’d built and yep, we were impressed. Then, with the startup CTO in the room, one of my fellow engineers asked the key question: “All these systems, are there any that wouldn’t work without blockchain?” The guy didn’t even hesitate: “No, not really.”
And that was about that.
Obviously we also talked to a few leading lights from the crypto scene. That wasn’t very helpful, because they seemed mostly concerned with the aspects that got you out from under troublesome government regulation and contract law; it all had an unsubtle aroma of libertarianism.
Lots of them mentioned the Australian Stock Exchange and said you must be a peasant if you questioned technology that was about to be adopted in such a Serious Enterprise full of Serious People. After all, it was being built by Accenture, no less! (Cue eye-roll.)
And then there was the other faction, all about number-go-up-we’ll-get-rich; these were the days of ICOs, each sketchier than the last.
There was still a gaping hole when we asked “What useful thing does it do?”
What Andy said · It looked like the conclusion was going to be “no there there”. We shouldn’t have been surprised. Remember Andy Jassy saying “when they explain why it’s wonderful I just don’t get it”?
At this point in history Andy was possibly the world’s single most accomplished person at listening to people talk about Enterprise IT problems and the tools needed to fix them. If he couldn’t see the value in blockchain, that was really a dead give-away right there, at the business level. Maybe looking at the technology side had been a waste of time.
An AWS service? · We didn’t just try to ascertain market realities. We asked ourselves “OK, if we had a distributed ledger technology, what would we build with it?” then came up with a few proposed AWS services.
Mine was a database with special features optimized for representing fungible units of value (you could call them “tokens” but I didn’t) that could be exchanged between parties, with each such exchange accompanied by a description of whatever was being exchanged for the tokens. I thought the schema setup for describing the “whatever” was pretty clever at the time, but I forget it now.
[Tim, so you designed NFTs? -Ed.] [No comment. -T.]
It included a shared public ledger with crypto-provable integrity that recorded all this stuff. But no blockchain because we just couldn’t convince ourselves that the real world wanted zero-trust; so there was a transaction manager you had to trust.
Dear Reader: I think that at some point, in a civilization, there has to be trust. I think that’s maybe the main reason we have civilizations. Call me crazy.
Do I think AWS should have built this thing? Eh, I can’t get excited either way.
The end · I can’t remember the details of how our findings got back to Andy. They were of the form “Ledgers are useful, cryptography tech is useful, blockchains aren’t, the field is full of grifters, but we could build distributed-ledger infrastructure and then these cool services on top of it.”
I seem to recall that about a year later we got another outreach from Andy’s office saying they were still getting blockchain heat, oh and Australian Stock Exchange. Could we have another look and get back to them? We did and I think the only thing that changed was we said “Keep an eye on Ethereum.”
Subsequently another group decided that if people were spending their VC money building crypto infrastructure on AWS, we should help them do that more efficiently, and thus was born Amazon Managed Blockchain, offering Hyperledger and Ethereum platforms as a managed service, in the typical AWS style.
I wasn’t involved and have no idea how that product did; wouldn’t surprise me if it made a pleasing but nonstrategic amount of money, because the VC crypto spasm didn’t stop till this year.
Now in November 2022 I can’t imagine that service has a bright future, but I’ve been wrong before.
Heading home; EWR sunset.
Coda: Property marks · At some point toward the end of this story, I got a call asking me to come down to Seattle. An important customer was coming in for a few days at the Executive Briefing Center; they wanted to hear about Blockchain and someone pointed them at me.
The customer was a very major international agency that you’ve heard of; not sure if they’re big AWS users or were just kicking the tires. In any case, they were being taken very seriously; presentations from a dozen or more product teams stretched over two or three days.
I huddled with the sales team lead and disclosed that we weren’t planning to ship anything big and that the engineering leadership was skeptical of blockchain in general. I wondered if it would be OK to share that. She looked troubled and said “That’s a judgment call and I can’t make it, you have to. But listen first.”
So I made a presentation about the virtues of ledgers and of cryptographic signatures, mentioned Byzantine generals, and noted that the area was extremely interesting but somewhat unproven in practice.
Silence fell.
The customer CIO, an extremely smart person, spoke up, in beautifully-rounded European vowels: “Here’s a use case I’ve been told about that’s on my mind.” He named a region in Asia and explained that the small farmers there mark their landholdings carefully, but then the annual floods sometimes wash the markers away. Then unscrupulous larger landowners use the absence of markers to cut away at the smallholdings of the poorest. “But if the boundary markers were on the blockchain,” he said, “they wouldn’t be able to do that, would they?”
Dear Reader: What do you think?
I thought. Then said “As a lifelong technologist, I’ve always been dubious about technology as a solution to a political problem. It seems a good idea to have a land-registry database but, blockchain or no, I wonder if the large landowners might be able to find another way to fiddle the records and still steal the land? Perhaps this is more about power than boundary markers?”
Later in the ensuing discussion I cautiously offered something like the following, locking eyes on the CIO: “There are many among Amazon’s senior engineers who think blockchain is a solution looking for a problem.” He went entirely expressionless and the discussion moved on.
P.S.: What now? · I still think about those boundary markers.
Up until this year, most of my conversations with crypto promoters have had problems. For example “This person thinks proof-of-waste is OK and thus can be safely ignored.” And “This person is pretending that the proportion of crypto biz that is actually Ponzi doesn’t round to 100%.”
Bitcoin itself can rot in hell and I hope it implodes tomorrow. But the Eth folk managed to get proof-of-stake to work at scale; good on ’em. Still, the explosions of non-Btc crypto Ponzi schemes continue, in a boring and discouraging rhythm.
I’m not prepared to say that no blockchain-based system will ever be useful for anything. But I’m gonna stay negative until I see one actually at work doing something useful, without number-go-up greedheads clamped on its teats.
And I’m glad AWS didn’t make a big bet on it back then.
And I’m sorry for the folk at the Australian Stock Exchange who tried so hard for so long.
Comment feed for ongoing:
From: Geoff Arnold (Nov 20 2022, at 14:33)
Blockchain: a "cure for which there is no known disease". Very nice piece, Tim. Forwarded.....
[link]
From: Tiny (Nov 20 2022, at 18:30)
https://www.radicalxchange.org/
[link]
From: Nathaniel Gaertner (Nov 20 2022, at 19:42)
For the foreseeable future, I think you can get many orders of magnitude more trusted compute capacity by filling a room with computers you can personally see than from any zero trust network. It doesn't even have to be a big room and they don't even have to be very fancy computers.
[link]
From: David Chase (Nov 20 2022, at 20:28)
I have heard, from someone working in a country south of the equator, that reliable, detailed record-keeping is difficult to obtain in their country, and that their solution was Hedera, which I think is a trust-an-authority blockchain, as opposed to proof of drowning polar bears or trusting some crypto oligarchs (who else has "stake"?).
It's hard to tell from this distance if this is really necessary to solve their problem, but the problems are in a different place with very different constraints, so, maybe?
But in this country, it's scams all the way down. I feel sorry for anyone who gets involved in it out of ignorance, but the rest are purveyors of a perhaps-legal con, and should be shunned.
[link]
From: Nathanaël Havez (Nov 20 2022, at 22:02)
Hell of a nice story. The ending part about a "solution looking for a problem" is so full of sense.
Most of cryptos/blockchains (as of today) don't really create any real values. NFTs are just a big pile of nothingness. And most crypto exchanges are bound to fail at some point (FTX, anyone ?).
But yeah, based on what your colleague said, you're definitely the precursor of the NFT era. Which is kinda funny.
[link]
From: Rob Sayre (Nov 20 2022, at 22:30)
The only use case I know of that works well is concert tickets. These must be underpriced so there are no empty shows, but ETH contracts let artists get a transparent share of the final price.
You could of course build this with a regular database, but the transparency is the important part here, since the middlemen often just lie.
[link]
From: Neil S. (Nov 21 2022, at 00:16)
How do you value a product like Overledger from Quant networks? Their product offers interoperability between DLT's in which the token to access and license this product is based on an Ethereum / ERC-20 token (QNT). The problem is interoperability between different (permission and permission less) DLT's. Their solution offers this interop. They are partnered with AWS as well as Oracle. Although not strictly a blockchain product, they seem to be solving a problem. Interested in your thoughts on this.
[link]
From: tab (Nov 21 2022, at 01:39)
To my thinking the value of bitcoin/ block chain is that it demonstrates the possibilities of today's crypto. Bitcoin is about storing real money out in the open. It is like seeing a stack of 1000 dollar or Euro bills on a table. It is right there but you can't get to it. (modulo human errors). Now that is pretty cool. If we can do that, what other things can we do with crypto?
[link]
From: Paul Morriss (Nov 21 2022, at 01:50)
I'm glad AWS came to the same conclusion as me :-)
If I could get some coherent thoughts on it I would write something about "you've got to trust somebody", like the person who created that executable that's your on-ramp to the blockchain.
Every so often, to amuse myself, I go looking at Salesforce's blockchain pages, though I haven't gone as far as asking our account rep if we can have a blockchain because everyone else has one.
[link]
From: Sri (Nov 21 2022, at 03:33)
Quote: “After all, it was being built by Accenture, no less!”
It appears Accenture merely reviewed the project and it was being built by a company named Digital Asset. Source: https://www.forbes.com/sites/michaeldelcastillo/2022/11/16/seminal-blockchain-project--goes-down-the-drain-chairman-apologizes/
[link]
From: John Siegrist (Nov 21 2022, at 04:04)
Have you had a look at the iota tangle? As for use cases, real-time contract settlement is a pretty good one. The cryptographic ledger replaces the clearinghouse from the financial transactions. It also means a lot fewer funds have to be tied up waiting for net-30 day or net-90 day settlement.
[link]
From: c1ue (Nov 21 2022, at 05:51)
The one application I have seen in all these years, for which blockchain makes some sense, is positive credit reporting. In countries like Australia, they have credit systems based on positive credit actions as opposed to US negative credit actions.
But positive credit reporting requires effectively a sharing of all positive credit activity across a range of different financial institutions and other entities. The danger is that sharing such information is tantamount to sharing your customer base.
A system was supposedly set up - basically a smart contract - that the positive credit data set that was contributed is not accessible until your own entity has contributed its data set.
Of course, in reality this smart contract is equally likely to be gamed.
The problem I have, as a cyber security professional who furthermore has specialized in forensic accounting and civil/criminal litigation support, is that the law profession has yet to come up with a "bulletproof" contract in literally centuries of practice.
Combining this lack of contractual certitude with the even less reliable software engineering field does not look to me like a negatives canceling out result...
[link]
From: Troy McConaghy (Nov 21 2022, at 08:02)
AWS eventually came out with a service that's now called Amazon Quantum Ledger Database (QLDB), which seems a lot like the thing you designed. The headline is, "Maintain an immutable, cryptographically verifiable log of data changes."
[link]
From: SS (Nov 21 2022, at 10:16)
"number-go-up greedheads clamped on its teats"
and they say engineers can't write...
[link]
From: Koobee (Nov 21 2022, at 10:51)
It’s truly amazing how many folks can get swept up in a kind of collective hysteria. I guess since it intersected with a get-rich vibe. Was it all just a speculative bubble? Crowds are bizarre!
[link]
From: Doug K (Nov 21 2022, at 12:09)
thanks Tim, I'm glad to hear smart people like you and Andy confirm my prejudice ;-)
and an interesting bit of history.
Paul said,
"If I could get some coherent thoughts on it I would write something about "you've got to trust somebody", like the person who created that executable that's your on-ramp to the blockchain."
Bruce Schneier wrote that coherent essay, look on his website for the post Blockchain and Trust.
TLDR summary,
"What blockchain does is shift some of the trust in people and institutions to trust in technology. You need to trust the cryptography, the protocols, the software, the computers and the network. And you need to trust them absolutely, because they're often single points of failure."
As a programmer I certainly do not trust the software, mine own not exempted.
[link]
From: Ole Eichhorn (Nov 21 2022, at 14:22)
Agree entirely. I like the tech behind blockchain but have never yet seen a useful application that uses it which couldn't have been built another way.
[link]
From: Marco Crotta (Nov 21 2022, at 23:39)
Well, this mostly shows one thing: it makes no sense at all to have a "blokchain as a service by AWS", because a blockchain already is a service and it must be public. Private blockchains are just a database, useless. Public blockchain give you extra features that a private one do not have: immutability, authonomous intedependent verification, involvment of third parties and so on.
I work in the blockchain space and I'm very glad the "giants" like AWS, GCP etc are out of it.
[link]
From: Rhiannon Monks (Nov 22 2022, at 07:56)
This is such an insightful piece.
This is incredibly accurate IMO:
The answers, to our disappointment, failed to shatter any preconceptions. The things they wanted to do were perfectly reasonable. Some of them were damn exciting. They all needed databases. They could all make use of ledger-like data structures, also cryptographic hashing and signing. But, um, why did they need blockchain? Severe lack of clarity on that.
I thought this back in early 2018 when I first scrutinised blockchain (because I liked the appeal of its so called immutability at first) along with crypto currencies and couldn't quite get why they trumped Fiat considering you still have to have a reserve and peg it somehow to the dollar. And briefly looked at raising equity for my startup via and ICO before realising how stupid that would be.
I like the AWS senior team took the subway and not taxis (despite the steaming heat).
Things I relate to:
There’s this amazing thing about working for AWS. You can get the attention of the IT leadership of pretty well any organization in the world and say “Hey, could we drop by and talk about…” and most times they’re happy to do it.
(That weeping you hear is the former me, the one who had two startups and spent months bashing at the doors of enterprises in the usually-vain hope of getting anyone meaningful to look at our technology.)
[link]
From: Mateo (Nov 22 2022, at 14:27)
Rob commented above about concert tickets being a useful application for blockchains:
> You could of course build this with a regular database, but the transparency is the important part here, since the middlemen often just lie.
Hate to break it to you, Rob but concert tickets aren't the only place middlemen are trying to swindle you. If there is an incentive to lie *and get away with it*, people will lie *and get away with it*.
Ergo blockchains, Tim.
Human nature 101. You can punish after the fact using the heavy, bribe-able hand of government, or... you can just do away the incentive!
[link]
From: Rupert (Nov 24 2022, at 04:19)
Great post, Tim.
My additional observations (having worked with small, medium and large enterprises to explore the potential value they can get from the technology for ~4 years of my life!):
1. Good use cases - there is a bit of an obsession and expectation that you must use EVERY SINGLE FACET of the technology for it to be a credible use case (eg cryptography, a shared ledger, a distributed ledger, the immutability and provenance provided by the blockchain, etc etc). The fact that it’s been called “distributed ledger technology” and “blockchain” of course contributes to that expectation, I get that - people understandably expect a whoppingly good use of the core facets
2. Combinatorial evolution is a thing - they aren’t new and differentiated facets in isolation, but the combination might be.
3. The opportunity - there certainly are pockets of opportunity across the world of business through some of the facets; smart contracts conceptually are quite unique - embedding shared business rules and logic INSIDE a shared database
4. Lowering our threshold - we don’t have that same expectation of most other great tools and technologies that are more established; this is true of a home power drill, a fridge, a TV, AWS itself, ‘AI’
5. The role of AWS - whether or not AWS should have a commercial play for blockchain is of course separate to my points 1, 2, 3 and 4 I do tend to agree with your recommendations to Andy!
[link]
From: Rob Sayre (Nov 25 2022, at 16:34)
To Mateo: Oh, yes, there are certainly many ways for middlemen to swindle you. Blockchains, in the form of ERC-721 or equivalent, allow the original issuer to recoup a percentage of any resale price on-chain. That's why they match concert tickets so well, because of the secondary market.
[link]
From: Moshe (Nov 26 2022, at 23:29)
While I agree that the current crop of crypto & Blockchain companies and solutions do appear to be gobblygook, I do see a (far) future for Blockchain, or more correctly a distributed, 0-trust, processing/data solution for extra-wide networks.
I know it's currently mostly SciFi, but I feel like it's coming, sooner rather than later.
I know you read it, and would love to get your thoughts on that, since with other such technologies (e.g. federated like mastodon, you definitely see the value)
I'm pretty sure the results will not include Bitcoin or Ethereum - maybe as some legacy code, but there will be something.
[link]
From: Paul Vixie (Nov 27 2022, at 01:32)
Tim I wish we had overlapped at Amazon. This is a great story and I'd have backed your play.
[link]
From: Mike (Nov 28 2022, at 12:19)
Maybe you could go into more detail on why you hate Bitcoin? Agree with everything else you said about blockchains, but Bitcoin is the one notable exception.
[link]
From: Austin Crispens (Nov 28 2022, at 21:10)
>After all, it was being built by Accenture, no less! (Cue eye-roll.)
I see other comment that pinned it being built by Digital Asset. Haven't heard of them.
Accenture US took away employee paid time off and 401(k) for computer programmers but NOT the business consultants they report to. Retention must be of no great concern.
That being said, what consulting company isn't eye-roll inducing? Accenture at least does phone/video call technical interviews for experienced hires.
[link]
From: JD Svo (Dec 01 2022, at 21:34)
Great article. I teach Econ to engineers. The men, and only the men, were so Crypto-chain stoked. Some taking side jobs just to be able to buy more coin. They thought I was grandpa, dissing their dreams of fortune. Not looking so much the doddering crank now I guess.
I will make an odd analogy: if the world were built by an all- knowing God, would you expect it to look like ours? And if it were instead of randomness and evolutionary forces?
If a small group of very bright coders were to perpetrate the most phenomenal novel currency fraud, what would it look like?
[link]
From: Dan Ciruli (Dec 02 2022, at 11:33)
The one place I've thought that I figured blockchain might be useful is in proving property ownership.
At least in the United States, when you buy real estate you need to pay for a "title search" to determine if the person you're buying the property from actually owns it, and you have to pay title insurance in case they don't. Both of THOSE industries always feel like a scam.
A publicly visible, validated trail of "who actually owns parcel XXXXX in ZIP code XXXXX" might actually let us get rid of that scam and finally provide a use case for blockchain.
[link]
From: Roy Brander (Dec 04 2022, at 14:08)
I can't find the David Roberts interview (at his "Volts.WTF") where a clean-energy developer made a case for blockchain as a way to post up "energy or storage needed" and "energy or storage available" to an open market in electricity.
It was compelling precisely because Roberts went "oh, no, not blockchain" and the guy replied "I hate it, too, but this seems to be one situation where everybody immediately hates anybody trying to be a trusted market overseer, nobody wants the job, blockchain was the least-worst suggestion".
I wouldn't have posted, since I can't find the one podcast, but when I googled on "blockchain energy grid", there have been many stories about it for years.
It would be nice if some real use could be found, after all the fanaticism that's gone into it.
[link]
From: BMD (Dec 19 2022, at 10:24)
No real use for trustless trust? Poppycock
[link]
From: Preston L. Bannister (Jan 04 2023, at 15:54)
As an aside, read the Bitcoin paper early, then read about Money. Wrote an answer on Quora back in 2017:
https://www.quora.com/Is-Bitcoin-a-pyramid-scheme/answer/Preston-L-Bannister
First, the answer took rank from "line-goes-up" folk. Then it was reported as spam.
Of note, the number of views has increased by quite a lot from 2021 and onwards. Also the tone of the comments is more in agreement. Something going on...
Just possible that healthy skepticism is taking root. :)
More to your topic, from the Bitcoin paper read the "blockchain" as a distributed ledger backed by strong cryptography. Seems a perfectly sound general notion, though not something for which I had use. Also suspected the Bitcoin blockchain implementation would not scale well.
I firmly believe that a distributed ledger backed by strong cryptography is a great solution. (Not the specific Bitcoin blockchain implementation, likely.)
But yeh ... not sure what problem fits.
[link]