[This fragment is available in an audio version.]
Hey, look what’s been open-sourced: AWS Event Ruler! Check out the announcement blog. I built v1.0 of this Java library while I was at AWS, and wrote about it in Filtering Lessons. Tl;dr: It offers APIs for declaring pattern-matching Rules, as many as you like, then for presenting data records called Events and finding out which Rules each Event matches, very quickly. It’s in production in multiple Amazon (not just AWS) services, notably EventBridge. Also see: Content-based Filtering.
I wrote the Quamina library because I liked the Ruler concept so much that I wanted a do-over in another language. The Quamina Diary blog series has lots more on algorithms and data structures; the most compact how-it-works intro is in Meet Quamina.
Before I go on, I want to thank David and Usman and Rishi and Shawn and the others who succumbed to my low-key but persistent lobbying and blessed this release.
Deets · Ruler’s only runtime dependency is Jackson, which brings along basically zero sub-dependencies. Ruler has a Maven POM, which I pointed at when I opened it in a new IntelliJ project, and was running the unit tests within minutes. Java 8 is good enough and there’s not that much code, 15K or so lines between implementation and test. Unit-test coverage is over 90%.
“Event Ruler”? ·
Inside AWS it was just “Ruler”. That name was obvious back when the software was written, for V1.0 of what was originally called
“CloudWatch Events” and is now
EventBridge. The central function of that service is to post rules against
event streams and route matching events to useful destinations. The APIs have names like addRule and
rulesForEvent so “Ruler” wasn’t exactly a creative triumph. But it’s short and memorable.
The processes inside AWS that led to this announcement yielded the name “Event Ruler” and there’s nothing wrong with that. But it’ll always be just Ruler to me.
Why it matters · The software is widely used inside AWS. Will it be useful outside the cloud-infrastructure world? My bet is yes, because more and more apps use loosely-coupled event-driven interconnections.
For example, I think there are probably a lot of Kafka applications where the consumers could be made more efficient by the application of this sort of high-performance declarative filtering. Yes, I know there are already predicates, but still.
Second, the Rule syntax used for pattern-matching is unconventional and the teams who use it seem to like it. Yes, there’s the occasional Couldn’t it be SQL? but I do think there’s a place for this flavor of structured query.
Finally, AWS has benefited hugely from the use of open-source. So it’s nice to see them giving something back, something built from scratch, something that doesn’t particularly advantage AWS.
It’s worth noting that the GitHub project has four committers (I’m one), only two of whom are Amazon employees.
Battle-tested · No nontrivial software is bug-free. But by my arithmetic, in its AWS life Ruler has been invoked on the order of 1016 times, including lots of millions of times while you’ve been reading this. On extremely heterogeneous data. With an extremely miscellaneous selection of Rules. Which is to say, you’re fairly unlikely to get a nasty surprise from it.
Shredding automata · [There’s a point to this anecdote, stay with me.]
It’s a rare EventBridge customer that has more than a few dozen different rules compiled together to match Events. But there was this service that really needed a fast event matcher and, for complicated reasons, needed to run huge amounts of traffic through a single server. One of them asked me “Suppose we needed to post a million rules?” I said “You’re kidding, right?” They weren’t, and they did. To my joy and astonishment, it worked fine.
But then they said “We can’t afford to rebuild that automaton at run-time, so we need to be able to delete rules.” My initial implementation of Ruler never considered the idea of doing this because just thinking about it made my head hurt (still does). So I said “Can’t be done, what’s your Plan B?”
At which point this dude Long Zhang, then in the requesting organization, just went ahead and built it. It works great.
Long became a primary contributor; he and I worked together on lots of features and he added some on his own. He knows more about Java concurrency and its memory model than anyone, and is a fun guy to work with. At this point, I bet he’s got more lines of code in the project than I do.
I remember one time we were having trouble agreeing on how to build anything-but matching and I couldn’t
understand his explanation. We ended up emailing set-arithmetic expressions back and forth until we converged. To this day,
Long and I have never actually been in the same room. Isn’t the future weird?
After all that was working, the big-single-machine service came back to us and said “Ruler is using too much memory.” Yes, Dear Reader, in the guts of cloud infrastructure, memory really matters. So Long and I dug into that problem. I’m telling this story to help explain…
The hard bits ·
Conceptually, Ruler is simple. In practice, the code starting at
ByteMachine is,
uh, non-obvious in places. One reason is supporting the deleteRule API. Which means reaching into a compiled state
machine and ripping out all the bits that match some Rule, without breaking anything. While the state machine is being used to
process a huge number of queries per second.
If you think that sounds hard, you’re
correct.
Then, there’s that RAM issue. There is fairly deep weirdness, notably in the inheritance and implementation hierarchy
around ByteMachine, aimed at shrinking the memory footprint.
Another hairy bit that I mention mostly for amusement value is building numeric-range-match state machines. If you
want a giggle, scan the comments in addRangePattern starting
here.
Just JSON? · Out of the box, Ruler can filter data presented as JSON texts, as parsed JSON nodes, and as “flattened” lists of field name/value pairs. It’d be nice if there were “flattener” plug-ins for Avro and CBOR and Protobufs and so on that turned data blobs into field lists. Jackson, which Ruler uses for tokenization, already knows some of these formats, which should make things easier. Hint, hint.
Ruler and Quamina · Obviously, since they’re library code and in different programming languages, neither of Ruler nor Quamina can be a direct replacement for the other. But I’m going to try really hard to make the the syntax for what Quamina calls “Patterns” be 100% compatible with what Ruler calls “Rules”, and to make them as feature-compatible as possible.
Interested? · Anyone out there think they might want to use this? Feel free to ping me, happy to offer moderate amounts of free advice.
