Phil Leggetter
Why Hookdeck Should Be Your Webhooks Backup Plan for Black Friday / Cyber Monday (BFCM)
You’ve spent the past few months ensuring you’re ready for Black Friday / Cyber Monday (BFCM) because you know that missing a webhook from Shopify, Big Commerce, Stripe, PayPal, etc., means you miss an event representing an important transaction, such as a purchase, inventory change, shipping status change, or payment. These events all equate to revenue or customer experience, which are fundamental to running a successful business.
So, you’re ready for a significant increase in traffic and traffic spikes during Black Friday / Cyber Monday: You’ve minimized your webhook ingestion time by pushing events into a well-resourced queuing service, ensured you have load-balancing in place for horizontal scaling, you’re logging all events to ensure you can recover from failure scenarios and perform audits, and your infrastructure code freeze is on the horizon.
You’re ready! Right?
Black Friday and Cyber Monday Webhook Traffic Spikes
Since campaigns tend to run leading up to and through Black Friday / Cyber Monday, you should expect a consistent increase in traffic throughout the period and slightly beyond. During this period last year, we saw the number of webhooks that our customers had to ingest and process double.
There may also be considerable traffic spikes if a specific deal or offer goes viral; this is great for the business but a challenge for your infrastructure.
No matter how well you plan, there’s always a chance that something unexpected can happen. A DNS outage, a 3rd party component in your stack going down, or an edge case you haven’t foreseen. Make Hookdeck your Black Friday / Cyber Monday backup plan!
Use Hookdeck as your Black Friday and Cyber Monday Webhooks Backup Plan
The first thing to point out is that Hookdeck is 100% interoperable with receiving the webhook events directly from any 3rd-party provider. You do not need to make any changes within your codebase.
Although we believe you should choose Hookdeck as your inbound webhook infrastructure, if you have already deployed your infrastructure and written custom logic, you may feel you don’t need Hookdeck. But Hookdeck can still help!
Here’s how:
- In addition to having your own webhook ingestion endpoints configured with your 3rd-party providers, also have the 3rd-party providers setup to deliver webhooks to Hookdeck. We’ll ingest and store the event payloads… just in case.
- If something unexpected does happen and your own webhook ingestion infrastructure is struggling with the load or has some other unforeseen problem, you can disable the delivery of webhooks directly to your own infrastructure but continue to have the webhooks delivered to Hookdeck.
- Once you have resolved the problems, deliver the missed webhooks from Hookdeck.
- Revert to direct webhook delivery and begin receiving events from the 3rd-party provider directly to your infrastructure again, or Stick with Hookdeck and deliver the rest of the events to your webhook endpoints from Hookdeck at a rate-limit that you know your infrastructure can handle.
Once you’ve followed these steps, your system is now in sync, and you’ve ensured you haven’t missed any webhooks.
A Step-by-Step Guide to Using Hookdeck as Your Black Friday / Cyber Monday Webhooks Backup
The following steps walk you through creating a Hookdeck Connection for webhook routing between a single 3rd-party provider and your own webhook endpoint. You should repeat this process for each 3rd-party provider you have.
Before you begin, you’ll need to signup for a Hookdeck account.
Note: The following contains instructions and screen captures from our upcoming new dashboard. However, the steps are all but identical in the current dashboard.
1. Backup Your WebHooks with Hookdeck
From the Connection section of your Hookdeck project, create a new connection. A Connection consists of a Source (the inbound request), optional Rules, and the Destination (where the webhook is routed to).
Start with the Source:
Give the Source a meaningful name. Assuming that you are handling webhook verification in your own code, you don’t need to use Hookdeck’s webhook signature verification and can leave ****************Verification**************** disabled.
Next, define a mock destination for the events (you’ll update this later):
The reason you’ll use a mock destination is that we’re assuming you are just using Hookdeck for backup purposes.
Provide a meaningful name for the mock endpoint. Leave the rate-limit disabled and skip over the retry, transform, filter, and delay rules. However, if you’re interested in finding out more about these features, you can read the docs.
Click Create, and you will be presented with a dialog with the Hookdeck URL to be used with your 3rd-party provider:
Use the Hookdeck URL as a new webhook subscription with your 3rd-party provider. This should be configured alongside your existing webhook subscription.
Your 3rd-party webhooks are now being delivered to both your webhook endpoint and to Hookdeck. Hookdeck is storing the events, ready to be delivered if required.
2. Managing Your Webhook Infrastructure Downtime during Black Friday/Cyber Monday
If the unexpected happens and you have a problem handling the webhooks directly from the 3rd-party provider during BFCM, you should start by disabling the delivery of those webhooks directly to your infrastructure. This drops the load on your webhook infrastructure to zero and gives you breathing room to remedy the problem. How you disable webhook delivery to your own endpoint will depend on the provider.
After you have done this, the webhooks will still be delivered to and stored within Hookdeck.
Before you proceed, remember you have two options once you have fixed your infrastructure:
- Revert to Direct Delivery: Begin receiving webhooks directly again. In this case, you’ll only use Hookdeck to retry any missed webhooks.
- Stick with Hookdeck: You'll use Hookdeck to retry any missed webhooks and begin using Hookdeck to improve the reliability of your inbound webhooks.
Whatever you choose, you now have time to rectify the problem with your infrastructure. Good luck!
You can move on to the next step when that's resolved.
3. Bulk Retry the Missed Webhooks
Whether you go with option 1 or 2, you should pause your Hookdeck Connection so that no more delivery attempts are made for the time being.
Update the Destination to point to your live webhook-consuming endpoint:
From the Connections section of your Hookdeck project, open up the Destination and change the Type to HTTP. Enter your webhook endpoint and enable and set the Rate Limit at which the webhooks will be delivered to your endpoint. Finally, click Save.
Now, deliver the missed events.
Begin by identifying the timestamp of the last webhook you successfully received and processed. Then, go to the Events section within your Hookdeck project to view all the events. Filter the events further based on your Connection (e.g., shopify-prod) and the time period you wish to Bulk Retry events for. This time period will depend on how you plan to move forward:
- Revert to Direct Delivery: From the last successful event you received directly from the 3rd-party provider to the first successful direct event you received following the fix.
- Stick with Hookdeck: From the last successful event you received directly from the 3rd-party provider to the time that you paused the Connection.
First, filter by the Connection that you want to retry events for. You can then filter the events by clicking and dragging the UI to drill into the required time period or selecting specific dates and times from the Date filter.
Once you have identified the events, you should deliver them to your webhook endpoint. Click the Bulk Retry button followed by the Retry All button to remove the unwanted events.
Your webhook endpoint will receive the missed webhooks at the rate-limit you’ve specified for the Connection. Once they have all been delivered, you are back up to date.
4. Revert to Direct Delivery or Stick with Hookdeck
At this point, you need to decide how you want to continue to receive webhooks.
If you choose to Revert to Direct Delivery and thus continue to get webhooks directly from the 3rd-party provider:
Edit the Destination to use the Mock API, remove the rate-limit, and unpause the connection. This way, Hookdeck is still backing up your webhooks, and you’re not building up a queue of webhooks to deliver in a paused connection.
If you have decided to Stick with Hookdeck:
Unpause the connection to begin receiving webhooks from your 3rd-party provider via Hookdeck at the specified rate-limit.
Conclusion
We hope you don’t need this guide! But if you’ve read this far, you are at least now equipped with a webhooks backup plan if anything unexpected does happen with your webhooks ingestion during Black Friday / Cyber Monday.
If you’re reading this during BFCM, and you’re scrambling for a solution to your webhooks problems, we’re here to help! Follow the guide, but also get in touch with any questions you may have.
Here are some additional useful resources to help: