Deform webhooks - choosing messaging service

Choosing messaging service can be a very annoying problem.

What is messaging service?

This is a service where you can send a message to some mechanism which will do something with it.

It has producers ( create a message ) and consumers ( receive a message ).

For example:

  • plate -> Producer - Provides food
  • cat -> Consumer - Consumes food
  • food -> Message - What consumer needs
  • house -> Queue Service ( humans included )

Ok, how can I choose a service?

First, we must determine your project's needs.

General questions are:

  • the language was used to develop project. Php, Python, Golang, Ruby and etc
  • the amount of events happen
  • how often will it happen
  • how much time do we have to setup a queue. In case of deadlines, urgency and etc

Deform example

Let's begin with our example. How did we choose a service?

  • the language was used to develop project. We used Golang
  • the amount of events happen/how often it happen. We have ~10 RPS ( requests per second ). 30% of these requests are to modify data. This means we have 3 message per second to be produced.
  • how much time do we have to setup a queue. No time, better to use some docker based service. This will give us a full control of a queue.

We'll also need some features provided by different services.

Features and Guarantees

Every messaging service provides some basic guarantees and features. For example:

  • messages are delivered at least once
  • discovery service for consumers
  • horizontally scalable

We'll need these 3.

Setup

Initial setup can be difficult. We have no need to spend a couple days installing environment, configuring brokers, tuning configs and etc.

Usage costs

Usage costs. Pay each month for some numbers of backends ( own installation ) or pay for number of messages ( cloud service ).

Obviously this is 80% priority requirement :D

Candidates

Here is a list of messaging services I chose:

Amazon SQS

Golang package: aws-sdk-go
Features and Guarantees:

  • messages are delivered at least once

Setup: No need to setup - it's cloud based.

Usage costs: First 1 million Amazon SQS Requests per month are free. Details.

IronMQ

Golang package: iron_go3

Features and Guarantees:

  • messages are delivered at least once

Setup: No need to setup - it's cloud based.

Usage costs: Has a free plan.

Usage costs:

NSQ

Golang package: go-nsq

Features and Guarantees:

  • messages are delivered at least once
  • horizontally scalable

Setup: Docker containers

  • nsqd - daemon itself
  • nsqlookupd - discovery service
  • nsqadmin - administration interface

Usage cost: Included to backend pricing.

What did Deform choose?

We got the NSQ. The reasons are:

  • Docker containers. Full control: from network to data
  • Usage costs. Included to backend pricing
  • Easy to maintain

We've used amazon SQS for a while. The reason we left:

  • Got an invoice after some millions messages :D