Day 1. The beginning.

 
👨 John
We need to make an http service for getting user name by its id.
🤓 Carl
Oh, but we already have one, it is http://internal.com. Look:

$ curl -XPOST http://internal.com/1
✅ HTTP/1.1 200 OK
John

$ curl -XPOST http://internal.com/2
✅ HTTP/1.1 200 OK
Carl

👨 John
Yeh, but it’s internal api. We want to make requests from the internet.
🤓 Carl
Then we need some authorization for it.
👨 John
Let’s make it dead simple for now and send the password as a request parameter.
🤓 Carl
Ok, wait a minute, I’ll write some code.

import requests
from flask import Flask, Response

app = Flask('service')

@app.route("/")
def get(request):
    if request.args['p'] != '123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['u']
        response = requests.post('http://internal.com/' + user_id)
        response.raise_for_status()
        return Response(response.json()['name'])

🤓 Carl
Save...Deploy...Done. With bad password:

$ curl http://service.com/?p=111&u=1
❌ HTTP/1.1 401 Unauthorized
Wrong password

🤓 Carl
With good password:

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

👨 John
Cool!

Day 2. Five hundred problems and a girl ain't one.

 
👨 John
Houston, we have a problem! Our new service is not working!
🤓 Carl
What? Wait I minute, I’ll check:

$ curl http://service.com/?p=123&u=1
❌ HTTP/1.1 500 Internal Error

🤓 Carl
Oh shit! I’ll look at logs:

❌ Response from http://internal.com/1 is 500 Internal server error

🤓 Carl
The internal service responds with an error. I’ll talk to them.

...Carl switches to the chat with the internal service developer...

🤓 Carl
Hey man. What’s wrong with your service? It responds with an error and crushes our service?
👷 Rob
Hi. We have database issues. Sometimes we respond with the error but sometimes without the error. Look:

$ curl -XPOST http://internal.com/1
❌ HTTP/1.1 500 Internal Error

$ curl -XPOST http://internal.com/1
❌ HTTP/1.1 500 Internal Error

$ curl -XPOST http://internal.com/1
✅ HTTP/1.1 200 OK
John

🤓 Carl
And what should I do?
👷 Rob
Man, networks are unreliable. Make some retry requests and get the successful one.
🤓 Carl
Ok, wait a minute.

-import requests
+from requests.adapters import HTTPAdapter
+from requests import Session
+from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response

app = Flask('service')

+session = Session()
+retry = Retry(
+    total=5,
+    method_whitelist=['POST'],
+    status_forcelist=[500]
+)
+session.mount(
+    'http://',
+    HTTPAdapter(max_retries=retry)
+)

@app.route("/")
def get(request):
    password = request.args['p']
    if password != '123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['u']
-       response = requests.post('http://internal.com/' + user_id)
+       response = session.post('http://internal/' + user_id)
        response.raise_for_status()
        return Response(response.json()['name'])

🤓 Carl
Save...Deploy...Done. Let's try.

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

🤓 Carl
3 of 3! And in the logs I see the retry requests. Thank you!

⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
✅ Response from http://internal.com/1 is 200 OK

...Carl switches to the chat with John...

🤓 Carl
I’ve fixed the problem. The internal service has some database issues. I've added 5 retries for every request and we are working normally now.

👨 John
Nice!

Day 3. Know your limits.

 
👨 John
Houston, we have a problem! Our new service is not working!
🤓 Carl
What, again? Arrr, let’s look at logs.

❌ Response from http://internal.com/1 is 429 Too Many Requests

...Carl switches to the chat with the internal service developer...

🤓 Carl
Hey man, we have a problem with your service again. It responds with 429 error. What is it?
👷 Rob
Hi. You're making too many requests. Do you still retrying the requests?
🤓 Carl
Yep.
👷 Rob
What is your backoff factor?
🤓 Carl
Ehhhh, what is my what?
👷 Rob
I mean, you can do only 100 requests per second to our service. You make a request, get the error with status code 500 (yes, we still have the database issues) and then make another request immediately. Just sleep after the 500 error for a while.
🤓 Carl
Let's see.

from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response

app = Flask('service')

session = Session()
retry = Retry(
    total=5,
    method_whitelist=['POST'],
-   status_forcelist=[500],
+   status_forcelist=[500, 429],
+   backoff_factor=0.1
)
session.mount(
    'http://',
    HTTPAdapter(max_retries=retry)
)

@app.route("/")
def get(request):
    password = request.args['password']
    if password != 'password123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['user_id']
        response = session.post('http://internal/' + user_id)
        response.raise_for_status()
        return Response(response.json()['name'])

🤓 Carl
I've added backoff factor equals to 0.1. As I understand it will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. Also I've added retry for 429 error. Save...Deploy...Done. Let's try.

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

$ curl http://service.com/?p=123&u=1
✅ HTTP/1.1 200 OK
John

🤓 Carl
3 of 3! Thanks!

Day 4. Your time is out.

 
👨 John
Houston, we have a problem. Our service is working but response is too slow!
🤓 Carl
What? Let's look...Hmm, yeah. It’s approx 5 seconds per request response. What do you want?
👨 John
We need minimum 100ms.

🤓 ➜ 👷

🤓 Carl
Hey man, we have a problem with your service again. It responds too slow and that’s why our service responds too slow.
👷 Rob
We have some balancer issue. Usually it routes all the requests to the closest data center. But now it started to route half of the requests to the furthest.
🤓 Carl
What should I do?
👷 Rob
Networks are unreliable, dude. You can set the timeout for the request. I mean, if the request didn’t finished in 30ms try to make another request. It should help.

from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response

app = Flask('service')

session = Session()
retry = Retry(
    total=5,
    method_whitelist=['POST'],
    status_forcelist=[500, 429],
    backoff_factor=0.1
)
session.mount(
    'http://',
    HTTPAdapter(max_retries=retry)
)

@app.route("/")
def get(request):
    password = request.args['password']
    if password != 'password123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['user_id']
-       response = session.post('http://internal/' + user_id)
+       response = session.post(
+           'http://internal/' + user_id,
+           timeout=0.3
+       )
        response.raise_for_status()
        return Response(response.json()['name'])

Yes, it helped. Thank you.

⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
✅ Response from http://internal.com/1 is 200 OK

Day 5. Timeouts strike back.

 
👨 John
Houston, we have a problem. Your service is not working!
🤓 Carl
I know, I know...

⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
⚠️ Retrying. Connection timed out (0.3).
❌ Max retries exceeded.

🤓 ➜ 👷

🤓 Carl
Dude, we have a problem. Your service is not responding in 30ms. We make 5 retry requests and then give up.
👷 Rob
We are upgrading the service and some instances have connectivity issues. In some cases it can not even start any connections. But you can detect that your request came to the bad instance. Our networks are quite fast and should establish connections in 5ms. Just split the timeout to the connection and response read.
🤓 Carl
But I'll still have 5 retry requests.
👷 Rob
As I remember your service must return the response in 100ms. It's 25ms*2 + 5ms*10.
🤓 Carl
Yeh, I got it. I'll make 2 retries for 25ms read timeouts and 10 retries for 5ms connect timeouts.

from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response

app = Flask('service')

session = Session()
retry = Retry(
    total=5,
    method_whitelist=['POST'],
    status_forcelist=[500, 429],
    backoff_factor=0.1,
+   connect=10,
+   read=2,
)
session.mount(
    'http://',
    HTTPAdapter(max_retries=retry)
)

@app.route("/")
def get(request):
    password = request.args['password']
    if password != 'password123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['user_id']
        response = session.post(
            'http://internal/' + user_id,
-           timeout=0.3
+           timeout=(
+               0.05,  # connect timeout
+               0.25,  # read timeout
+           )
        )
        response.raise_for_status()
        return Response(response.json()['name'])

🤓 Carl
It works! Thanks!

⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.05).
⚠️ Retrying. Connection timed out (0.25).
✅ Response from http://internal.com/1 is 200 OK

Day 6. Stop this madness!

 
👨 John
Houston, we have not such a big problem by the problem it is.
🤓 Carl
???
👨 John
When we make a request with not existing user id we get 500 error. It would be awesome to have 404 error. Can you help us?

$ curl http://service.com/?p=123&u=999
❌ HTTP/1.1 500 Internal server error

🤓 Carl
I see. That's because the retries are exceeded.

⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
⚠️ Retrying because response is 500.
❌ Max retries exceeded.

🤓 ➜ 👷

🤓 Carl
Hey man, your service constantly returns 500 error for not existing user. We make 5 retry requests and then give up.
👷 Rob
It’s not a bug, it’s a feature. It’s part of the protocol and can not be fixed. You always can check the response. You can stop doing retries based on it. Just stop the retry cycle if the response is User not exists.

$ curl http://internal.com/999
❌ HTTP/1.1 500 Internal server error
User not exists.

🤓 Carl
Let's see... We're always make the retry request when you return 500 error. It's impossible to check the response via the default Retry class. I'll have to extend it.

from requests.adapters import HTTPAdapter
from requests import Session
from requests.packages.urllib3.util.retry import Retry
from flask import Flask, Response
+from requests.packages.urllib3 import (
+    exceptions as urllib3_exceptions
+)

+class MyRetry(Retry):
+    def increment(self, *args, **kwargs):
+        if (kwargs.get('response') and 
+            kwargs['response'].status == 500 and 
+            'User not exists' in kwargs['response'].data):
+            raise urllib3_exceptions.MaxRetryError(
+                pool=kwargs.get('_pool'),
+                url=args[1],
+                reason=urllib3_exceptions.ResponseError(
+                    'User not exists.'
+                )
+            )
+        return super(MyRetry, self).increment(*args, **kwargs)
    

app = Flask('service')

session = Session()
-retry = Retry(
+retry = MyRetry(
    total=5,
    method_whitelist=['POST'],
    status_forcelist=[500, 429],
    backoff_factor=0.1,
    connect=10,
    read=2,
+   raise_on_status=False,
)
session.mount(
    'http://',
    HTTPAdapter(max_retries=retry)
)

@app.route("/")
def get(request):
    password = request.args['password']
    if password != 'password123':
        return Response('Wrong password', status=401)
    else:
        user_id = request.args['user_id']
        response = session.post(
            'http://internal/' + user_id,
            timeout=(
                0.05,  # socket timeout
                0.25,  # read timeout
            )
        )
-       response.raise_for_status()
-       return Response(response.json()['name'])
+       if response.ok:
+           return Response(response.json()['name'])
+       elif response.content == 'User not exists':
+           return Response('User not exists', status=404)

🤓 Carl
It works!

$ curl http://service.com/?p=123&u=999
✅ HTTP/1.1 404 Not found
User not exists

🤓 Carl
We're not retrying on User not exists response and return 404 right away.

Three months later.

 
👨 John
Hey dude. It’s been 3 months as your service works without any error. You rock, man!
🤓 Carl
Thank you, it was a hard work to make it happen.
👨 John
Actually we have some problem. Someone hacked our system and downloaded all the names of our customers. What do you think? How did they do so?
🤓 Carl
Mmm, didn’t you told anyone the password to the service?
👨 John
No, I didn’t!
🤓 Carl
It’s better to change the password I believe. Let’s try. Change… Save… Deploy… Done! It’s 123456 now!
👨 John
You’re the cyber security ninja, man!
🤓 Carl
💪

The end.

Have you ever wanted to convert mysql database to sqlite? Or postgres to mysql? Or mysql to postgres?

Recently I've been migrating my django project from MySQL to SQLite. I tried to use dumpdata and loaddata commands.

• dumpdata can save your database to json file
• loaddata can populate your database with entities from the json file

But it is not so simple as it sounds:

• You must migrate your SQLite database first, because json file will not contain any schema altering instructions
• You must remove any conflicting content from contrib.auth application. For example, permissions or initialized superusers
• Even if you did all these steps you can have problems with indexes or duplicates

I decided to move on and try to convert my databases directly (without the django facilities). I checked stackoverflow and found many scripts for converting your MySQL dump to SQLite dump. For example mysql2sqlite.sh (don't use it!). But it failed too.

When I was already despaired I found an awesome project called sequel. It's ORM written in Ruby. Sequel has a command line interface and an option for copying one database to another even between different database types. Using -C option, you provide two connection strings:

sequel -C mysql://host1/database postgres://host2/database2

This copies the table structure, table data, indexes, and foreign keys from the MySQL database to the PostgreSQL database.

As I understand sequel performs 4 major steps for copying databases:

• It connects to the source database and extracts the metadata (table structure, table data, indexes, foreign keys)
• Then it builds an ORM models based on the extracted metadata
• Then it migrates a target database based on the ORM models
• Finally it populates the target database with the data from the source database

Sequel is awesome but for using it you have to install Ruby, gem package manager, sequel itself and all drivers you need for connecting to your databases. I found it inconvenient and have built a docker image called chibisov/sequel. It packed with sequel + drivers for MySQL, Postgres and SQLite.

With just one line of code I can migrate MySQL database to SQLite file:

$ docker run --name mysql2sqlite chibisov/sequel:4.31.0 sequel -C mysql2://user:password@host/database sqlite://db.sqlite

Be careful with the mysql connection string. You should use mysql2:// adapter because the default mysql:// adapter has problems with utf-8 encoding.

That's it! Now I can copy the database file to my host machine and remove the intermediate container:

$ docker cp mysql2sqlite:/db.sqlite ./
$ docker rm -v mysql2sqlite

You can watch how I convert databases in my terminal session:

In Docast we love to watch Youtube. Sometimes it's hard to find time to watch videos but thankfully Youtube has "Watch Later" button. Video will be added to the special playlist after you clicked it.

Why is it so convenient? Because it's one or two clicks action button. On any mobile app it's the fastest way to save video for watching it later.

Houston we have a problem

As I've described "Watch Later" button is the fastest way to save video to the special playlist. But it isn't the regular playlist. Look at the difference.

With "Watch Later" playlist you are not able to:

• Share it
• Change settings. E.g. change video ordering

With normal playlist you can set ordering by date added (newest first).

And here we come to the biggest problem of the default "Watch Later" playlist - you cannot change the default ordering of the videos which is Date added (oldest). When you add a new video to the playlist it will be added to the end of the list. You'll have to scroll down to see the latest items.

The solution

We've built a service which solves the problem. The website is https://watchlater.chib.me. I'll explain how it works:

• Ensure that you have a youtube channel. To create a youtube channel use this link.
• Click "Start using"
• Allow to our service to work with your youtube account
• After that we will create a normal playlist named "Watch Later Service"
• Every five minutes we will move all your videos from default "Watch Later" list to the normal playlist.
• That's it 😊

Why is it cool? Because you don't have to move videos manually and you will be able to use convenient "Watch Later" button. Enjoy!

I love football but I don't have enough time to watch all matches. With Docast.me I can watch the latest football highlights anytime from many special websites. For example timesoccer.com.

Such sites have all recent football matches but they are inconvenient in many ways:

• Videos with highlights could have ads. I don't want to waste time on it.
• There are no notifications about new games.
• There are no convenient controles (skipping 30 forward, increasing speed, etc...)
• There is no way I could watch videos in offline mode (e.g. in the underground).

You know what - there is an app with the all described features. It’s already on your device. It’s called the podcasting app!

RSS + Docast = ♥

We've added RSS integration to Docast recently. It opens possibilities for listening text articles and watch videos from any webpage. Actually it works the same way as the Pocket integration:

The difference is:

• With the Pocket you subscribe to your personal pocket bookmarks.
• With the RSS you subscribe to opened RSS feed.

There are just two steps you should perform for watching football highlights in any podcasting app:

• Create a podcast from your favorite website's RSS.
• Add the podcast link to your app as it described in the video above.

I'll give you a short link for the timesoccer.com. Just follow it and click "Create podcast" button:

Now you can enjoy football highlights right in your favorite podcasting app! On any device. Without ads. With notifications. With convenient speed controles. In offline mode!

P.S.

I've been using timesoccer.com feed for a while and found it inconvenient. It's slow updating and sometimes they mix text articles into their feed.
I found another great website footyroom.com but they don't have RSS feed I could use.

Thankfully as a developer I can build my own RSS feed. I wrote little script with GO which parses footyroom.com. Resulting feed you can find here. You can start watching highlights from the feed using Docast. Just follow a short link and click "Create podcast" button.

I love to listen and I’m not alone. There are so many people who love to listen music, radio, podcasts, audiobooks. Recently we’ve built a small service for integrating different data sources into your favorite podcasting app, which is the most convenient app for listening audios and watching video content. The service could be found here https://docast.me.

In general, Docast builds a podcasting feed from your Dropbox, Youtube, Pocket and other services accounts. For futher details keep reading.

Audiobooks

Recently I’ve started listening historic and fiction audiobooks. But I was surprised that iTunes is not capable to play them in a simple mp3 format. I have to use third party apps to import them to the iPhone. But what if I could just place my folder with mp3 files to the Dropbox and start listening audiobooks?

Special apps for audiobooks is another problem. They all are out of date or wants you to use their cloud (e.g. audible). Try to find these apps for Windows Phone.

I want more! I want to change speed, remove silence, boost voice (especially when I’m in the subway).

To sum up. App for reading audiobooks should:

• Be capable to play simple mp3 files from simple folders
• Have audio filters
• Track my listen history
• Be integrated with Dropbox

There is an app with the all (except latest) described features. It’s already on your device and it’s called the podcasting app!

Watch short video how you can listen audiobooks from the Dropbox in Overcast:

Youtube

I like to watch interesting videos from Youtube. Most of them I want to listen. I don’t need to see an image from Antony C. Sutton's interview. I just want to listen, that’s it! But Youtube app on my iPhone has a big problem - it stops playing audio when I lock my phone!

Let’s talk about an offline mode. I use a public transport to commute to my work. And it’s horrible to watch youtube with 3G, LTE or even with WI-FI in an underground. Yeah, I know about Youtube Red, but:

• It wants 10$ a month
• It still has problems with stopping audio when phone is locked
• It isn’t available in my country =(

Another feature I need is a synchronization. I want to start listening or watching process on iPhone, continue with iPad and finish with my Mac. Moreover I use an Android device! And I want to work it there to. Thankfully there is an app for that. It is called Pocket Casts. To watch videos in offline you just have to enable auto download. It keeps in sync your watch progress across the all devices. It’s cross-platform. Did you know that Pocket Casts is available in your car or chromecast?

Look at a short video how you can watch videos from your Youtube’s "Watch later" playlist:

By the way, if you use only Apple devices then it keeps your episodes in sync out of the box.

Pocket

I like to read articles from the techcrunch. If want to read some publication I’ll add it to my Pocket. But after a day in front of my computer I want to listen it, not to read. Yeah, I know about text to speech integrated into Pocket, but it’s inconvenient in many ways:

• When you lock your phone it will hide most of the controle buttons
• Sometimes it flushes all your listening history
• There are no audio effects mentioned above
• It works only on iOS and Android devices

With Docast you won’t have these problems. All you need is just to connect it with your Pocket account and subscribe to podcast.

Look at a short video how you can listen text articles from any page in the Internet. Moreover, Docast can extract audio and video content from any page. It’s compatible with more than 500 services (e.g. Youtube, Vimeo, Soundcloud, etc…):

To sum up

I’ve been consuming all my media content in a podcasting app for more than 3 months by now. I’ve been listening articles from Pocket. I’ve been listening audiobooks from Dropbox. I’ve been watching videos from Youtube. On any device. It’s all in sync. In offline mode. Magic!

One day I came to a conclusion that podcasting app is the most convenient media center app with opened capabilities for integration. Till now there was no simple way to integrate your personal content into this media center. That’s why we’ve built Docast.