Eventing Facets

What happened was, at re:Invent 2019 I gave a talk entitled Moving to Event-Driven Architecture, discussing a list of characteristics that distinguish eventing and messaging services. It was a lot of work pulling the material together and I’ve learned a couple of things since then; thus, welcome to the Eventing Facets blog series, of which this is the hub. It’s going to take a while to fill this out.

Eventing or messaging? · They’re just labels. If it’s publish-and-subscribe people usually say “event”; if there’s a designated receiver, “message”. But there’s a lot of messy ground in between; and in my experience, the software does more or less the same stuff whichever label’s on the cover. So I may simultaneously talk about eventing and use the word “message” to describe the byte packages being shuffled around.

Disclosure · I work for AWS and am a member of the “Serverless” group that includes Lambda, SQS, SNS, MQ, and EventBridge. I also support services that make large-scale use of Kinesis. You can expect this to have obvious effects on the perceptions and opinions offered herein. I’m also reasonably well-acquainted with Rabbit/MQ and Kafka, but without any hands-on.

Process · For each facet, I’ll introduce it here and write a blog piece or link to an existing one that covers the territory. In the process, I’ll build up a table relating how a list of well-known eventing/messaging services support it, or don’t; I premiered a version of that table in the talk (see the picture above) and more than one person has asked for a copy.

OK, here we go.

Facet: FIFO · When you inject events into the cloud, can you rely on them coming out in the same order? For details, see Facet: FIFO.

Facet: Deduplication · When you fire an event into the cloud, can you be sure it’ll only come out again once? See Facet: Deduping.

Facet: Subscribe or Receive · When there’s an event in the cloud, how many different receivers can receive it? Just one? See Facet: Point-to-Point vs. Pub/Sub.

Facet: Serverless or Broker · Does your eventing infrastructure look like a cloud service, just an endpoint in the cloud? Or is it an actual machine, or cluster of machines, that you own and operate and connect to? For details, see Facet: Broker vs Serverless.

Facet: Push or Pull · Do you have to poll the cloud for events, or can you get them pushed to you? For details, see Facet: Push vs Pull.

Facet: Filtering vs firehose · When you reach out into the cloud and use a queue or broker or bus to receive events, the simplest model is “I subscribe and you send me everything.” Often this is perfectly appropriate. But not always; I’ve seen software subscribed to high-volume sources that starts with really dumb code to look at the messages and decide whether they’re interesting, and ends up throwing most of them on the ground.

In recent years I’ve increasingly noticed the infrastructure software offering some sort of built-in filtering capability, so you can provide a query or pattern-match expression that says which events you want to see.

We built this into CloudWatch Events (now known as EventBridge) back in 2014, and people seemed to like it. Then a couple of years ago, SNS added filters to subscriptions and, since then, I’ve noticed that a steadily-increasing proportion of all subscriptions use them.

So I’m pretty convinced that some sort of filtering capability is a basic feature that eventing and messaging software really need. There’s no standard for how to do this. Some of the filtering expressions look like SQL and some look like regular expressions.

{
  "vendor": [ "vitamix", "instapot" ],
  "product_detail": {
    "price_usd": [
      { "numeric": ["<=", 150] }
    ]
  }
}

I don’t think this subject deserves its own article, especially since back in December I wrote Content-based Filtering, which describes on particular flavor of this technology that looks like the sample above, i.e. not like either SQL or regexes. I think the piece gives a pretty good feeling for the power and utility of event filtering generally.

Still to come · Here are the ones I know I’m going to write about, and maybe there’ll be more: Payload Flavors, Records vs. Documents, Uniform vs Heterogeneous Events, Replaying and Sidelining. Like I said, this will take a while.

Summarizing · Since this table includes references to software that I’ve never actually used, it’s perfectly possible that my research has been imperfect and there are errors. Please let me know if so and I’ll dig in deeper and correct if necessary.

	ActiveMQ	Artemis	Event Bridge	Kafka	Kinesis	RabbitMQ	SNS	SQS	SQS FIFO
FIFO?	Yes	Yes	No	Yes	Yes	Yes	No	No	Yes
Dedupe?	*	Yes	No	Yes	No	No	No	No	Yes
P2P or Pub/Sub	Both	Both	P/S	P/S	P/S	Both	P/S	P2P	P2P
Serverless or Broker	B	B	S	B	†	B	S	S	S
Push or Pull	Both	Both	Push	Pull	‡	Both	Push	Pull	Pull
Filtering?	Yes	Yes	Yes	*	No	No	Yes	No	No

Notes ·

* It looks like the software can do this but either it isn’t straightforward or doesn’t work out of the box.

† Kinesis is a cloud service but since you have to provision shards, it kind of acts like a broker too. On the other hand, adding and deleting shards is possible if a bit awkward.

‡ Kinesis is pull-based, but there’s the Kinesis Client Library which gives a very push-like feeling.

---

---

Contributions

---