Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production

Transcript

Webhooks might be the simplest distributed system

most teams underestimate. You give a service

a URL. It sends an HTTP POST when something happens.

Done. Until your endpoint is down during a deploy.

Or the sender retries and you process the same

event twice. Or events arrive out of order. Or

a valid request triggers something it should

not. Or you discover the provider stopped retrying

hours ago and nobody noticed. Suddenly, that

simple HTTP POST needs queues, idempotency,

signature verification, replay, observability,

retention, and someone willing to own it when

it breaks. That is the strange thing about webhooks.

The interface is simple. The operational contract

is not. I'm Brian Teller from Teller's Tech.

And this is Ship It Weekly. Welcome back to Ship

It Weekly, where I filter the noise and focus

on what matters when you are the one running

infrastructure and owning reliability. Most weeks,

it's a quick DevOps, SRE, platform, cloud, and

security news recap. In between those, I do conversation

episodes with people building and operating the

systems we all depend on. Today I'm joined by

Jay Lark, Principal DevOps Engineer and the founder

of Hookbridge. Jay has spent years dealing with

the operational problems hiding behind webhooks.

Failed deliveries, duplicate events, retry behavior,

signature verification, replay, payload visibility,

and the awkward process of testing webhook integrations

locally. We talk about what at-least-once delivery

actually means, why the receiver usually carries

most of the reliability burden, what teams need

to log without accidentally storing sensitive

data, and why replay feels optional right up

until the first incident. Jay also explains why

polling may be more useful than engineers give

it credit for, and why every team probably does

not need to build its own webhook delivery platform.

All right, let's jump in. Today I'm joined by Jay

Lark, principal DevOps engineer and founder of

Hookbridge. We're talking about webhooks, why

they became the default way services talk to

each other, why they're harder than they look,

and what teams need to understand before relying

on them in production. Jay, thank you for joining

me. Thanks for having me. Glad to be here. So

I guess let's start at the top. What are webhooks?

Yeah, we should define it so we all know what

we're talking about here. They're deceptively

simple. At least they seem like they're simple

until they're not. But basically, it's an HTTP

POST request from one website to another website,

one app to another. Just a real simple example.

If you want to, say, get a notification when

something happens on GitHub, like your PR merges

in or something, you can give GitHub a URL. And

when your PR merges in, GitHub will send an HTTP

POST to the URL you gave it with a little JSON

body saying PR merge or whatever. And that's

it. It's real simple. HTTP POST. Why are they

harder than they look? There's a whole bunch

of things that can happen. If the receiver is

down, what happens then? Many webhooks don't

retry. They just kind of sit there. And some

webhook providers give you an option to go manually

hit the retry button, but many don't. A lot of

the big webhook providers will send webhooks

kind of out of order. So you might get an order

updated before order created, and your system

receiving it might not be able to update an order

that it doesn't have yet. There are latency issues.

If your servers are slow to say, yes, I received

this webhook, the sending party may decide, hey,

they didn't get that. There's an error. Let me

send it again. And now all of a sudden you have

duplicates. There's a whole host of things that

can go wrong in the whole transaction. So on

paper, a webhook is just an HTTP POST, as you

said. Why is that mental model the source of so

much pain? Probably because it seems so simple.

So people write kind of the happy path in their

code, right? It's one or two lines of code

to send an HTTP POST. But all of the things that

you would have to do when it fails, and, you know,

you're wishing you did a little bit more than

the one or two lines of code. All of the things

that you end up having to do to kind of catch

all those edge cases, it's pretty significant.

Separating events by different IDs, deduplicating

events if you get the same one, and verifying

that the webhook came in from whoever you expect

it to. It's a whole set of things that you really

need to account for and to build for. A lot of

people don't. And then there's at-least-once delivery.

What does that actually mean for the person receiving

the webhook? Yeah, that's good wording, isn't

it? The big webhook senders, like

Stripe or PayPal, say, at-least-once

delivery: We will send you the event at least

once. We might send it 10 times. We might send

it more, yeah. So, you know, if the webhook is

charging your credit card for an order, you certainly

don't want to process that 10 times. So you're

going to want to make sure you only process it

that once. Okay, so there's an asymmetry with webhooks.

The sender fires an event, but the receiver has

to expose an endpoint, keep it available, verify

it, process it, and not lose anything. Is that

a fair burden to put on every consumer? Fair?

That's a tough call. That's the way it is now.

You know, before webhooks became the main thing,

we used to poll for events. So if you go back

to our example of a GitHub PR merge, you could

change your app to check every five minutes or

every 10 minutes, or maybe every minute: Has

that PR merged in? Has that PR merged

in? And most of the time you're going to get nothing

back. The thing didn't happen, so it's pretty

wasteful. And also then, on GitHub's end, they're

getting hammered by you every minute and then

like a million other people. And I'm not blaming

GitHub; literally everybody does this. But

they've switched the contract now, right?

We'll just send it to you as it

happens. So it's good for you. You get the event

as soon as it happens. But yes, there's

a huge burden on you now. You have to be up all

the time. You can't go down, or like we said,

they may not retry. You might never get it. You

have to do all these deduplications. The

events may not come in in the order you

expect, so you have to maybe queue them somewhere

and wait for event number one in a chain to arrive

before you process events two, three,

and four. Then you have to do all the verification

and all those types of things. So is it fair?

Maybe. I mean, it's your data you're looking

for, really. Maybe. But it is just a fact. That's

how the modern web works now. If you're optimizing

for like six nines with like 31 seconds per year

or whatever, if it happens to go down during

that 31 seconds, you lose out on that webhook,

right? I mean... For a lot of providers,

yeah. Somebody like Stripe has a web console

you can go in and you can hit retry and it'll

resend it to you. But you have to know that you

missed it. You don't know that somebody submitted

an order and you didn't get it. So you're building

it. Then you're building some checker to go on

to Stripe's API and check for a missed order.

Like, wow, you could have built the redundancy

into your system in the first place. So I guess

that leads to debugging the failed webhooks.

That's been miserable for so long. Is there

a way to combat that? How do we improve it?

Yeah, it's very, very difficult. A lot

of the time, if you've ever debugged a failed webhook

that GitHub sends you, it's this big red

failed message in the GitHub console, but

there's not a lot of detail about what happened.

You might get the HTTP error code back that your

system sent, but there's really not that

much there. So really, if you're building

this yourself, if you're receiving

webhooks, you have to build in a whole bunch

of logging to find out what happened. Did the

signature verification go wrong? Is there just

too much of a latency? A lot of webhook senders

will just fail pretty fast if you don't respond

and acknowledge and process the data quickly enough.

A whole host of things can go wrong. So you need

that detailed logging and information to be able

to debug it. And most senders don't give you

what you need. How do we get around the latency

requirements or the availability requirements

without just throwing a ton of infrastructure

at it, right? Yeah, it's tough. It's a tough

call. We have a lot of customers that use n8n

to receive webhooks. It's a great product, really

easy to use. And a lot of the workflows get kicked

off by webhooks. I think a lot of people use

n8n, and it's hard for me to say, because it's

cheap, and they end up putting it on pretty

under-specced servers, which just take a little bit

longer to respond. And a lot of providers time

out really fast on that. So the answer to your

question is that you have to give it enough horsepower.

That's just the fact of the matter. Going down

for deployments and things like that, you have

to build in that redundancy. So however many

listeners you have up, they have to stay up while

the new ones come take their place. It's not all

that different from kind of rotating on a web

server underneath a load balancer, but it's just

something you do have to account for when you're

receiving webhooks because you have no idea when

they might come in. Whose problem is it when

there is a failure? Is it on the sender? Is it

on the receiver? Is it, I guess, case by case?

Yeah, I mean, it can be both, right? Depending

on why. Ultimately, it's kind of the receiver's

problem to deal with, right? They want the

data. They have to kind of figure it out, and

whether that's creating a whole bunch of support

tickets with the upstream provider and, you know,

maybe someday getting a response, or beefing up your

infrastructure and, you know, solving whatever

problem is causing the delivery problem.

Ultimately, I think it comes down on the person

receiving the webhook. That's fair. And I guess

before we even know, right, we don't know if

there was an issue if the webhook didn't go through

if it was a latency drop-off, if our servers just

didn't receive the webhook for whatever reason

during that time that it was sent. How much visibility

should a provider have or service have to look

into webhook calls? Like what's a standard as

far as visibility? Yeah, I think you need a lot

there. You need good logging, like I said, between

the whole TCP handshake to see whether

things are getting cut off because of latency.

You need visibility into where that might be

dying. You need the request and the response

headers. Maybe the signature verification is

just not working. A lot of people, at first,

if they're actually trying to do signature verification,

actually verify on the wrong part of the message.

And that's going to be pretty clear if you're

looking at the headers of the response, but sometimes

you just don't get those. So you're going to

need the full request and response headers.

You really need everything except for probably

the body of the message, right? There's some

privacy and security issues with seeing

whatever might be sent in that body. But other

than that, you really need a pretty comprehensive

observability view into that transaction. So

that was actually going to be my next question

on the privacy compliance angle. Is it just the

body? Is that the only thing that we're really

concerned with, I guess, with PII? I mean, well,

that's a huge one. People use webhooks for anything.

So there's no telling what could be in the body.

Obviously, listeners out there, they'll know

what their app does. Maybe there's nothing super

sensitive in a PR merged message, but typically

there's going to be something more sensitive

in there. But it's not only the body. Tokens

and signature verification data

can be in the headers. Those can also be intercepted

and used for nefarious purposes if somebody has

access to things they shouldn't. What about replayability?

Is that a core requirement, or does that feel

optional until the first incident? It probably

feels optional to a lot of people at first. Because

going back to the idea of sending an HTTP POST,

it's usually one or two lines of code, right?

You just do it. The first time you have an incident,

you're going to want to be able to replay things.

And the fact is a lot of providers just don't

allow that. But yeah, you can, especially if

you are receiving a webhook and your system is

sending a 200 response back, yeah, I got it.

But something internal to your system broke and

you didn't process the message correctly. It

might be gone now from the sender. So, you know,

maybe you'll fix your back end and say, oh, things

work now, but you need to be able to go and replay

those messages. Okay, so we have observability.

We have webhooks being received. We maybe have

the ability to replay them. Now from the security

side of things, the trust boundary: Once you expose

that public endpoint for incoming events,

what are the security basics that people should

get right from the get-go? Yeah, this isn't

everywhere, but most providers now are going

to give you a cryptographic signature that you

can verify that the sender is who you expect.

It's them sending it. So if the provider

you're receiving webhooks from offers that,

please do that. Verify it. But there's a second

part that I definitely see people ignoring is

just because the webhook came from

a trusted source doesn't mean whatever the payload

is asking you to do in your system is okay. Like

you need to verify the business logic and say,

okay, yeah, this order can go to this or this

thing is owned by this project ID. So it's not

just blind trust. It's a trusted message, but still

do all the things you would normally do

on an HTML form or something, you know,

in your app. What does HMAC signing actually

prove? Like more than just like tokens or? Yeah,

so they'll give you a key and then they'll sign

their message and your system can compare, make

sure that the message is signed by them with

the key that they give you. So if you're doing

this signature verification with Stripe, the

messages that are coming in, you can verify those

are coming from Stripe. Okay. Like I said, you

still want to make sure that, you know, the

orders, or probably more importantly, refunds, are

going to the right accounts and you know you're

somebody's not spamming you with refunds

to one credit card. But that's all it does. It

verifies that the message is coming from the

person you expect it to come from. And in regard

to security models, how do replay attacks work

in this context? Yeah, so this is a big

problem. I don't know if it's a big problem.

I don't know if people are intercepting a lot

of webhook messages, but it's a security precaution

that you need to take. With signature verification,

you need to be able to kind of dedupe messages.

So if a refund is coming in, hopefully your provider

has an event ID attached to it. Some providers

don't do event IDs or anything like that. So,

you use an event ID so you only process that

message once, and probably attach timestamps

to it. So a timestamp will say, okay, this, you

know, maybe you have some rules that say only

process this message within a certain window

of the timestamp. So you can do these things

to protect yourself from somebody who has

access to those keys or whatever, so they can't

not spamming you with webhooks you shouldn't

follow. Okay. And so moving from that, I guess,

into like local development. So we have security

in place. I want to do local testing of my webhooks.

Why is that so painful? It is a very painful

thing. Because in our earlier example, if you

want to monitor, if you want a message from GitHub

when a PR merges, you have to go into GitHub

and you have to put in, like, a URL of where

you want GitHub to send that message. GitHub's

not going to be able to send a message directly

to your laptop, your localhost environment.

Maybe you have some sort of tunneling program.

There's all sorts of them out there. Set up some

secure connection that pops your laptop out to

the web with a unique URL. But 99% of those

programs, the URL changes like literally every

time. So every time you boot up your laptop,

you've got to go into GitHub and change something.

I mean, the other option is deploying to your

staging environment or your testing environment

somewhere up in the cloud where you can receive

webhooks, but every single time you change,

even if it's a simple spelling mistake, you've got to

do this whole CI deployment pipeline, and it

just takes time. So the time spent deploying up

to the cloud or every single time going up and

changing this URL, it's annoying. It's a lot

of friction. I'm assuming you thought about this

too while building Hookbridge. So how did you

build that out for local development? Yeah, so

even developing a product like Hookbridge, because

we ran into this firsthand. So we built a little

program, a little binary you can run on your

computer. You go into our Hookbridge app and

you create a one-time URL to hand to GitHub

or whoever you want. And you put that in the

app and that URL never changes. And then you

run this little binary on your computer, and

it'll create a secure tunnel from your laptop

up to our system. And our system will just kind

of forward those webhooks through to your laptop.

So GitHub can send it to us, and then we'll send

it down to your laptop. You set it up once, just

like any other webhook, and then that URL is

live forever. Interesting. So when you started

building webhook infrastructure. What kind

of things surprised you the most? Just how many

ways it can break and how, you know, not our

system, just webhooks in general. Just our whole

conversation, how many things you have to account

for on something that seems so simple. And it

really turns into this huge distributed thing

that you have to build to account for all the

ways things can go wrong. And that was kind of

the inspiration for building Hookbridge.

I've been in the industry for quite

a while now, and I've seen this play

out over and over everywhere I've

been that has accepted webhooks. You

have to do it right. You do it right at the

beginning or eventually you learn, you have to

do it, but you have to eventually build this

huge thing to accept something that seems so

simple. Yeah, it's just a cURL command,

right? It's simple. Yeah, it's deceptively simple

for sure. Okay, so if a platform team is building

or buying a webhook infrastructure, what would

you say are the most non-negotiables that

they should look for or should be table stakes?

Yeah, I mean, table stakes, obviously, a lot

of the security stuff that we talked about, I

mean, I'm obsessed with security. So you have

to be able to do the signature verification and

things like that. But you need the tools to be

able to troubleshoot when there are problems. All the

observability and logging that we talked about, and

inspecting all the request and response

headers. That's

what I'm trying to say. You just need to be able

to see the whole chain to see where it's going

wrong because it doesn't matter if you're sending

webhooks or if you're receiving them. The other

party is usually not very helpful in troubleshooting

the problem. And then obviously all the other

problems that we talked about, like you got to

be up all the time, that kind of thing. So what

prompted you to build Hookbridge to solve this?

Yeah, because literally every place that I've

been that uses webhooks runs into these exact

same problems. And finally, I'm like,

just going to build it. Build it once and we

can use it forever. And what kind of users are

you seeing use Hookbridge and what kind of problems

are they solving? Yeah, a lot of it is the

n8n users, like I talked about. We'll just act

as an intermediary in front of their n8n box.

We'll keep retrying for as long as they want,

and we'll try a little bit longer, and we won't

time out as fast as maybe some

providers that are having issues with

n8n. But an interesting newish thing is, I

mean, I'm sure we've all heard about OpenClaw

and all these other claws, all the home AI automation

things that are out there. A lot of people are

setting up systems, tools at home that are great,

do a lot of different things. A lot of them are

requiring to receive data through webhooks. Don't

let OpenClaw open holes in your router

to send them to your laptop. So one thing that

one thing we've added to Hookbridge

that's really caught on is we call it a pull

endpoint, where you go into Hookbridge, you add

an endpoint, a webhook-receiving endpoint, just

like you would normally. You hand that to Stripe

or whoever is sending you webhooks and they send

webhooks normally. But we don't forward them

onto you. We just hold them and then OpenClaw,

whatever system you have that wants to receive

webhooks, they can just hit our API and they

can pull down the events from us when you want

them. So you don't have to open up any firewall

holes or anything like that. It also solves some

issues with ordering and things. So you can

pull down by event ID if you want all the

orders first and then all the updates and things

like that. So the out-of-order processing

and process-once issues kind of go away when

you're pulling things with an

API instead of just receiving them whenever

they might happen. Yeah, that's cool. Okay, so if

I wasn't ready to start with a

platform yet, but I want to shore up or fix

my webhook issues so I don't run into these

problems, what are the top three things

that we should do right off the bat?

We kind of talked a little bit about this, but

is it HMAC? What is it? My

number one thing is always going to be the HMAC

signature. Like everyone should do that. Just

don't accept random webhooks from whoever's pretending

to be somebody else. Beef up your observability

platform. Whatever you have now, just make sure

that you can diagnose any problems, whether they're

on your end receiving, or as much as you can

kind of gain insight from whoever's sending you,

or if you're sending webhooks, you know, the

responses you're getting back from the person

you're trying to send to. Just make sure that

your observability platform is as good as it

could possibly be. Have you noticed, we talked

a little bit, you talked a little bit about OpenClaw.

Fable is now back online after being taken away

for a couple weeks. Have you noticed any trends

on the security side? There's been a lot of RCEs,

I know, outside of just webhooks, but there's

been a lot of CVEs out there lately. Have you

noticed anything with the platform, like from

your customers or from other friends in the industry

around webhooks, trends related to AI or related

to just infiltration in general, like trends

that are happening? Maybe AI is making that easier,

but maybe it's not even AI related. Yeah, a couple.

Like I said, the people running OpenClaw at

home that need to receive webhooks. That was

kind of an unexpected surprise. It's nice.

It's more people than you might think, too, that

are all of a sudden like, I need to receive webhooks

to my iMac. That's weird. Okay, I get it. I get

a lot of, and this has always happened, even

when I didn't run the business. Just being

on the DevOps team, you do occasionally get these

emails, but there's a lot of people now that

have access to OpenClaw or AnyClaw, any AI, whatever,

who are security researchers. And they send you

all sorts of emails: "I found whatever vulnerability.

I'll tell you about it for whatever fee," you

know, so you get a lot of that. And I think that

has probably increased since the dawn of AI.

I don't think they're finding legitimate things.

The couple that I've asked, oh, hey, can you

show me some proof? And maybe I'll give you the

bounty. They're not finding real things, at least

from what I've seen. Locally, personally, I'm

using it. Yeah, it gives me good confidence when

I can run the code through multiple AI models

and say, give me a thorough security review.

And we can definitely find things that way. Yeah,

out on the open web, I haven't seen a whole lot

coming our way yet. Fingers crossed, because

I know some communities are really under fire

from it. And what kind of customers primarily

do you have or users do you have in the platform?

And what are they using Hookbridge for? Like,

what problems are they solving? Yeah, big one

is the n8n users. Someday I'll be able to say

that well. But a lot of them are the opposite

of what we've been talking about. We've mostly

been talking about how to build a system if

you are going to receive webhooks. And it's a

big thing, like we discussed. You've got to have

a queue to hold all these messages and make sure

you process them in the right way and do all

this signature stuff. A lot of people don't

do it. So if you are sending webhooks and the

person you are sending to doesn't have a system

that is reliable and can receive webhooks and

things, you need to kind of build the reverse

of it, be able to retry and stuff. So instead

of having to do that, people just use us because

we will queue the message, keep retrying

it, sign it for you, and do all that kind

of stuff. So yeah, everything

we talked about receiving, you kind of have to

do it on the reverse too. It's interesting. So

okay, wrapping up, is there any webhook opinion

that you have that maybe other engineers might

disagree with? Well, as the owner of Hookbridge, I'll

probably say, you know, most teams shouldn't

build their own webhook infrastructure. I would

say a controversial webhook take is to not use

webhooks. I would say polling is an underrated

thing. Polling still has a place. Like, obviously,

be a good citizen. Don't hammer someone. Or,

you know, this wasn't my intention, but again,

use the Hookbridge pull endpoint. Poll us.

We don't care. Let's say someone was looking

into growing their webhook journey and they

don't want to do all of this themselves. What

can Hookbridge offer them and what does it cost

to get started? What's the barrier to entry?

Yeah, I don't blame you if you don't want to

build this huge thing because it's a lot. We

have a free plan. You can sign up. All the features

are there. There's no big thing that you can't

do with the free plan. I think you can send like

a thousand webhooks over a month. That should hopefully

plenty to test something out. And for a lot of

our users, that's enough for their home OpenClaw

installs, like we talked about. A thousand

a month is fine, and they can just send and

receive that many for free. But then obviously

we have paid plans if you need to send more messages

per month or you need a whole bunch of endpoints

and things like that. The paid plans have long

retention periods. So if you're receiving webhooks

and you're down or actually you're probably going

to want to get your own site up really fast too.

But you have no control over your customers,

right? If the people you're sending to are down

for a lengthy amount of time, we'll hold the

message and we'll keep retrying it for a long

time. And our paid plans have kind of lengthy

retry periods, retention periods there. Very

cool. So where can people find out more about

you and Hookbridge? Hookbridge is at www.hookbridge.io.

Send me an email, jay@hookbridge.io,

J-A-Y. Happy to talk about webhooks. Happy

to talk about anything. Yeah, let us know what

you think. Let us know your pain points with

webhooks, because I know there are tons of

them out there, and we are happy to discuss new

use cases and how you might use

webhooks or how Hookbridge could work for you.

Very cool. Thanks so much, Jay, for coming on.

Really appreciate your time. Thank you. That

was my conversation with Jay Lark, Principal

DevOps Engineer and founder of Hookbridge. The

biggest takeaway for me is that webhooks are

not unreliable because HTTP is unreliable or

because the basic pattern is bad. They become

unreliable when teams build only the happy path

and treat delivery like a single request instead

of a distributed workflow. Once a webhook matters

to your business, you need to assume duplicates

will happen. Events may arrive out of order.

Endpoints will occasionally be unavailable. And

something inside your own system may fail after

you already acknowledged the request. That changes

what success means. A 200 response does not necessarily

mean the event was processed correctly. A valid

signature does not mean the requested business

action is safe. And a retry does not help much

if nobody can see what failed or replay the event

later. I also liked Jay's point about the burden

placed on the receiver. The sender gets to fire

the request and move on, while the receiving

team has to expose a public endpoint, keep it

available, validate the sender, protect the business

logic, deduplicate events, and build enough observability

to figure out what happened when something goes

wrong. And sometimes the answer really might

be polling, not hammering someone's API every

second, but using polling intentionally when

it gives you better control over timing, ordering,

or infrastructure exposure. So if you are building

a webhook integration, start with the failure

path. Verify signatures. Use event IDs for idempotency.

Acknowledge requests quickly. Retain

enough information to debug failures. And make

sure you have a replay story before you need

one. You can learn more about Jay and Hookbridge

at hookbridge.io. I'll have that and anything

else we mentioned linked in the show notes. If

you enjoyed this conversation, follow or subscribe

to Ship It Weekly wherever you listen to podcasts.

You can find the show, previous episodes, and

the weekly DevOps and SRE news recaps at shipitweekly.fm.

Thanks for listening, and I'll see you later

this week.

Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production

Watch this episode here

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

👤 Guest

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

Ship It Conversations: Guardsquare’s Joel DeStefano on Mobile App Security, Runtime Protection, App Hardening, and Why Scanning Isn’t Enough

PeopleSoft Zero-Day Exploited, npm v12 Install Script Changes, GitHub Agentic Tokens, Anthropic Model Risk, and Default Trust Breaking

Get the next episode in your inbox