Event-driven architecture in the era of 'AI' - Part I

As an engineer, I’ve come to understand that a lot of what we build mimics things we’ve already seen in nature. It may not be a conscious realization where our mind develops the analogy right away, but regardless, many of us excel at pattern recognition and application. Personally, I believe that’s where it starts.

One of the reasons humans are so blown away by LLMs (large-language models) and mistake it for ‘intelligence’ is because of the dense amount of patterns stored within and the model’s ability to quickly surface that information. These models are trained on massive amounts of known information, all produced or derived by human beings.

All the good things we’ve learned and produced. All the bad things we’ve learned and acted on. All the things we’ve learned correctly. All the things we still haven’t gotten just right…

So now, we have these systems capable of rapid knowledge retrieval and we apply them everywhere. They are not deterministic, although many argue they are “deterministic enough”. They have no understanding of causality and they are not world-aware. As engineers we often find ourselves building to offset some of the downsides of these newer technologies.

Driven by necessity to event-driven architecture

One way I’ve been able to support downstream ‘AI’-enabled applications is with event-driven architectures (EDA). By the end of this post it should become clear organically why they can help.

An Event-driven architecture is a way of communicating and responding to events. As I mentioned earlier, a lot of what we build as engineers can be seen as mimics of things in nature. I like to think a lot of engineers in tech are in tune with the world around them and so these patterns tend to remain front of mind.

Engineer or no, you may have heard these three entities below:

• event producer: produce events (publisher)
• event consumer: consume events (subscriber)
• event broker: ingest, filter, and route events

Note: For event brokerage, there are additional components I won’t go into just yet, such as event bus which is like an event broker, but less robust and typically with less responsibility

Event-driven architecture of the brain

Not surprisingly, the brain’s neural architecture maps well to event-driven systems.

• Neurons (the electrically excitable cells in your nervous system) produce events and consume them.

• Action potentials are the events fired (produced) and received (consumed) by neurons.

• Synapses are the physical junctions (event channel) between two neurons where the event transmission happens.

• Neurons are event producers AND consumers.

And now, an illustrating GIF generated lo-fi purposefully by

.

A neuron labeled 'publisher: action_potential' fires a small pulse traveling down its axon. The pulse hits three branching dendrite subscribers, each labeled with what they do on receipt: flee(), eat(), duck(). After the pulse arrives, each subscriber lights up briefly in sequence. Created by my GIFslop project

In my opinion, there’s not a great analogy for ‘event broker’ in the architecture of the nervous system. I’ll use one that’s maybe a bit of a stretch: the thalamus.

The thalamus is the relay station (event broker) in your brain that receives incoming sensory signals from the peripheral nervous system (PNS) and routes them to the appropriate ‘cortical regions’ for processing. Visual signals (or events) to the visual cortex. Sounds to the auditory cortex, etc. It’s much more complicated than just message passing, which is why I say it’s a stretch of an analogy, but regardless there is filtering, amplification, attenuation, etc prior to forwarding, which is exactly what a robust event broker does for events it ingests.

But why though?

What if you have thousands of producers? What if you have thousands of subscribers?

This is the main value prop for EDA and why technologies like

exist. In fact, Kafka’s intro docs literally use the human body’s central nervous system as an example in the opening sentence.

As software/platforms (hopefully) gain traction and adoption from internal teams or externally as they scoop up that sweet market share, usage explodes from individual users or downstream API systems. At this point mature teams begin to think in terms of ‘distributed systems’ and not the sprawling monolithic services common in early-stage companies. I won’t go into the reasons for ‘why distributed’, but the ‘why’ and ‘when is it appropriate’ should be clear to mature teams.

At some point every service in the lifecycle of ‘what your company does’ will have some ‘event’ that ‘something’ needs to react to. At scale your largest consumer of your software may actually be more software. If you’ve made the choice to decouple things within your stack or overall architecture, how can you ensure that System A is properly receiving and responding to events emitted from System 7-alpha-9? What if there are actually 4 systems that need to know about events from 7-alpha-9? What if 2 of those systems only care about part of the payload? What if they’re not allowed to see the full payload?

Up Next

In Part II (coming soon), we’ll actually do a high-level design & build of an event driven architecture and talk more about event brokers and other ways of doing ‘event-driven’. There are a lot of components and patterns you can reach for, as long as you can analyze the tradeoffs and have that “right tool, right job” awareness. We’ll dive into some of those then!