Story of retry
Day 1. The beginning.
👨 John
We need to make an http service for getting user name by its id.
🤓 Carl
Oh, but we already have one, it is http://internal.com. Look:
$ curl -XPOST http://internal.com/1
✅ HTTP/1.1 200 OK
John
$ curl -XPOST http://internal.com/2
✅ HTTP/1.1 200 OK
Carl
👨 John
Yeh, but it’s internal api. We want to make requests from the internet.
🤓 Carl
Then we need some authorization for it.
👨 John
Let’s make it dead simple for now and send the password as a request parameter.
🤓 Carl
Ok, wait a minute, I’ll write some code.
import requests
from flask import Flask, Response
app = Flask('service')
@app.route("/")
def get(request):
if request.args['p'] != '123':
return Response('Wrong password', status=401)
else:
user_id = request.args['u']
response = requests.post('http://internal.com/' + user_id)
response.raise_for_status()
return Response(response.json()['name'])
🤓 Carl
Save...Deploy...Done. With bad password:
$ curl http://service.com/?p=111&u=1
❌ HTTP/1.1 401 Unauthorized
Wrong password
🤓 Carl
With good password:
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
👨 John
Cool!
Day 2. Five hundred problems and a girl ain't one.
👨 John
Houston, we have a problem! Our new service is not working!
🤓 Carl
What? Wait I minute, I’ll check:
$ curl http://service.com/?p=123&u=1
❌ HTTP/1.1 500 Internal Error
🤓 Carl
Oh shit! I’ll look at logs:
❌ Response from http://internal.com/1 is 500 Internal server error
🤓 Carl
The internal service responds with an error. I’ll talk to them.
...Carl switches to the chat with the internal service developer...
🤓 Carl
Hey man. What’s wrong with your service? It responds with an error and crushes our service?
👷 Rob
Hi. We have database issues. Sometimes we respond with the error but sometimes without the error. Look:
$ curl -XPOST http://internal.com/1
❌ HTTP/1.1 500 Internal Error
$ curl -XPOST http://internal.com/1
❌ HTTP/1.1 500 Internal Error
$ curl -XPOST http://internal.com/1
✅ HTTP/1.1 200 OK
John
🤓 Carl
And what should I do?
👷 Rob
Man, networks are unreliable. Make some retry requests and get the successful one.
🤓 Carl
Ok, wait a minute.
-import requests
+from requests.adapters import HTTPAdapter
+from requests import Session
+from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
app = Flask('service')
+session = Session()
+retry = Retry(
+ total=5,
+ method_whitelist=['POST'],
+ status_forcelist=[500]
+)
+session.mount(
+ 'http://',
+ HTTPAdapter(max_retries=retry)
+)
@app.route("/")
def get(request):
password = request.args['p']
if password != '123':
return Response('Wrong password', status=401)
else:
user_id = request.args['u']
- response = requests.post('http://internal.com/' + user_id)
+ response = session.post('http://internal/' + user_id)
response.raise_for_status()
return Response(response.json()['name'])
🤓 Carl
Save...Deploy...Done. Let's try.
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
🤓 Carl
3 of 3! And in the logs I see the retry requests. Thank you!
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
✅ Response from http://internal.com/1 is 200 OK
...Carl switches to the chat with John...
🤓 Carl
I’ve fixed the problem. The internal service has some database issues. I've added 5 retries for every request and we are working normally now.
👨 John
Nice!
Day 3. Know your limits.
👨 John
Houston, we have a problem! Our new service is not working!
🤓 Carl
What, again? Arrr, let’s look at logs.
❌ Response from http://internal.com/1 is 429 Too Many Requests
...Carl switches to the chat with the internal service developer...
🤓 Carl
Hey man, we have a problem with your service again. It responds with 429 error. What is it?
👷 Rob
Hi. You're making too many requests. Do you still retrying the requests?
🤓 Carl
Yep.
👷 Rob
What is your backoff factor?
🤓 Carl
Ehhhh, what is my what?
👷 Rob
I mean, you can do only 100 requests per second to our service. You make a request, get the error with status code 500 (yes, we still have the database issues) and then make another request immediately. Just sleep after the 500 error for a while.
🤓 Carl
Let's see.
from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
app = Flask('service')
session = Session()
retry = Retry(
total=5,
method_whitelist=['POST'],
- status_forcelist=[500],
+ status_forcelist=[500, 429],
+ backoff_factor=0.1
)
session.mount(
'http://',
HTTPAdapter(max_retries=retry)
)
@app.route("/")
def get(request):
password = request.args['password']
if password != 'password123':
return Response('Wrong password', status=401)
else:
user_id = request.args['user_id']
response = session.post('http://internal/' + user_id)
response.raise_for_status()
return Response(response.json()['name'])
🤓 Carl
I've added backoff factor equals to 0.1. As I understand it will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. Also I've added retry for 429 error. Save...Deploy...Done. Let's try.
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John
🤓 Carl
3 of 3! Thanks!
Day 4. Your time is out.
👨 John
Houston, we have a problem. Our service is working but response is too slow!
🤓 Carl
What? Let's look...Hmm, yeah. It’s approx 5 seconds per request response. What do you want?
👨 John
We need minimum 100ms.
🤓 ➜ 👷
🤓 Carl
Hey man, we have a problem with your service again. It responds too slow and that’s why our service responds too slow.
👷 Rob
We have some balancer issue. Usually it routes all the requests to the closest data center. But now it started to route half of the requests to the furthest.
🤓 Carl
What should I do?
👷 Rob
Networks are unreliable, dude. You can set the timeout for the request. I mean, if the request didn’t finished in 30ms try to make another request. It should help.
from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
app = Flask('service')
session = Session()
retry = Retry(
total=5,
method_whitelist=['POST'],
status_forcelist=[500, 429],
backoff_factor=0.1
)
session.mount(
'http://',
HTTPAdapter(max_retries=retry)
)
@app.route("/")
def get(request):
password = request.args['password']
if password != 'password123':
return Response('Wrong password', status=401)
else:
user_id = request.args['user_id']
- response = session.post('http://internal/' + user_id)
+ response = session.post(
+ 'http://internal/' + user_id,
+ timeout=0.3
+ )
response.raise_for_status()
return Response(response.json()['name'])
Yes, it helped. Thank you.
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
✅ Response from http://internal.com/1 is 200 OK
Day 5. Timeouts strike back.
👨 John
Houston, we have a problem. Your service is not working!
🤓 Carl
I know, I know...
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
❌ Max retries exceeded.
🤓 ➜ 👷
🤓 Carl
Dude, we have a problem. Your service is not responding in 30ms. We make 5 retry requests and then give up.
👷 Rob
We are upgrading the service and some instances have connectivity issues. In some cases it can not even start any connections. But you can detect that your request came to the bad instance. Our networks are quite fast and should establish connections in 5ms. Just split the timeout to the connection and response read.
🤓 Carl
But I'll still have 5 retry requests.
👷 Rob
As I remember your service must return the response in 100ms. It's 25ms*2 + 5ms*10.
🤓 Carl
Yeh, I got it. I'll make 2 retries for 25ms read timeouts and 10 retries for 5ms connect timeouts.
from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
app = Flask('service')
session = Session()
retry = Retry(
total=5,
method_whitelist=['POST'],
status_forcelist=[500, 429],
backoff_factor=0.1,
+ connect=10,
+ read=2,
)
session.mount(
'http://',
HTTPAdapter(max_retries=retry)
)
@app.route("/")
def get(request):
password = request.args['password']
if password != 'password123':
return Response('Wrong password', status=401)
else:
user_id = request.args['user_id']
response = session.post(
'http://internal/' + user_id,
- timeout=0.3
+ timeout=(
+ 0.05, # connect timeout
+ 0.25, # read timeout
+ )
)
response.raise_for_status()
return Response(response.json()['name'])
🤓 Carl
It works! Thanks!
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.25).
✅ Response from http://internal.com/1 is 200 OK
Day 6. Stop this madness!
👨 John
Houston, we have not such a big problem by the problem it is.
🤓 Carl
???
👨 John
When we make a request with not existing user id we get 500 error. It would be awesome to have 404 error. Can you help us?
$ curl http://service.com/?p=123&u=999
❌ HTTP/1.1 500 Internal server error
🤓 Carl
I see. That's because the retries are exceeded.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
❌ Max retries exceeded.
🤓 ➜ 👷
🤓 Carl
Hey man, your service constantly returns 500 error for not existing user. We make 5 retry requests and then give up.
👷 Rob
It’s not a bug, it’s a feature. It’s part of the protocol and can not be fixed. You always can check the response. You can stop doing retries based on it. Just stop the retry cycle if the response is User not exists.
$ curl http://internal.com/999
❌ HTTP/1.1 500 Internal server error
User not exists.
🤓 Carl
Let's see... We're always make the retry request when you return 500 error. It's impossible to check the response via the default Retry class. I'll have to extend it.
from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
+from requests.packages.urllib3 import (
+ exceptions as urllib3_exceptions
+)
+class MyRetry(Retry):
+ def increment(self, *args, **kwargs):
+ if (kwargs.get('response') and
+ kwargs['response'].status == 500 and
+ 'User not exists' in kwargs['response'].data):
+ raise urllib3_exceptions.MaxRetryError(
+ pool=kwargs.get('_pool'),
+ url=args[1],
+ reason=urllib3_exceptions.ResponseError(
+ 'User not exists.'
+ )
+ )
+ return super(MyRetry, self).increment(*args, **kwargs)
app = Flask('service')
session = Session()
-retry = Retry(
+retry = MyRetry(
total=5,
method_whitelist=['POST'],
status_forcelist=[500, 429],
backoff_factor=0.1,
connect=10,
read=2,
+ raise_on_status=False,
)
session.mount(
'http://',
HTTPAdapter(max_retries=retry)
)
@app.route("/")
def get(request):
password = request.args['password']
if password != 'password123':
return Response('Wrong password', status=401)
else:
user_id = request.args['user_id']
response = session.post(
'http://internal/' + user_id,
timeout=(
0.05, # socket timeout
0.25, # read timeout
)
)
- response.raise_for_status()
- return Response(response.json()['name'])
+ if response.ok:
+ return Response(response.json()['name'])
+ elif response.content == 'User not exists':
+ return Response('User not exists', status=404)
🤓 Carl
It works!
$ curl http://service.com/?p=123&u=999
✅ HTTP/1.1 404 Not found
User not exists
🤓 Carl
We're not retrying on User not exists response and return 404 right away.
Three months later.
👨 John
Hey dude. It’s been 3 months as your service works without any error. You rock, man!
🤓 Carl
Thank you, it was a hard work to make it happen.
👨 John
Actually we have some problem. Someone hacked our system and downloaded all the names of our customers. What do you think? How did they do so?
🤓 Carl
Mmm, didn’t you told anyone the password to the service?
👨 John
No, I didn’t!
🤓 Carl
It’s better to change the password I believe. Let’s try. Change… Save… Deploy… Done! It’s 123456 now!
👨 John
You’re the cyber security ninja, man!
🤓 Carl
💪
The end.