Clash Royale CLAN TAG#URR8PPP

Fixing Flask timeout due to instantiation of large Keras model on Heroku

So I have a simple demo Flask application meant to serve a Keras model that I trained. The application works fine running locally on my computer but often times out on Heroku. In the application the model is loaded when Flask is started. This makes the most sense to as then we don't have to reload the model on each request. Normally this would be fine since Flask would run continuously. However, given this is a free Heroku app, Heroku deactivates my instance after 30 minutes. This means each time the app has to reload the model from scratch which causes a timeout error due to the model being instantiated (which often takes 20+ seconds) (occasionally it also has an out of memory error). I would like to either reduce the model load time if possible. Barring that find another way to avoid timeout on Heroku without upgrading (i.e multi-threading perhaps). I realize I could host my model on a GPU on AWS or something then call it, but I would like to keep it as simple and cost effective as possible. It's not meant to serve 1000s of requests but merely work when 1 or 2 people click the link from my paper.

Code is below.

import os
from flask import Flask, redirect, url_for, request, render_template, send_from_directory
from werkzeug import secure_filename
from examples2.example_keras import SimpleResNet50, ResNet2

# folder to upload pictures
UPLOAD_FOLDER = 'uploads/'
# what files can upload
ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg'])

# start + config
app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
app.config['ALLOWED_EXTENSIONS']=ALLOWED_EXTENSIONS
model = ResNet2("long_path/model_weights.h5")
# main route
@app.route('/')
def index():
return render_template('upload.html')


Example error code

at=error code=H12 desc="Request timeout" method=GET path="/" host=lung-r.herokuapp.com request_id=79bfba10-810e-444d-a73a-d32bcd68d603 fwd="24.198.104.217" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
2018-06-25T05:22:18.893433+00:00 app[web.1]: load model weights_path:


4 Answers
4

You only expect it to work with 1-2 people, who click the link from your paper?

Have you considered making a Jupyter Notebook, including the compiled output in the ipynb, then linking to it on your github?

Github has an inbuilt preview render, and then you can just include a setup tutorial for them to work from locally.

The only other thing I can think of, is to use Heroku workers, a subprocess, or:

model = None
# main route
@app.route('/')
def index():
global model
if model is None: model = ResNet2("long_path/model_weights.h5")
# to inform the user, load this^ in another thread/process,
# and return 'loading' message
return render_template('upload.html')


Create a class that loads the model on initialization and offers a predict function.

Inject an instance of that class using flask injector (https://github.com/alecthomas/flask_injector) to the api function.

The timeout error is often due to the fact that Keras has to load the TensorFlow backend which causes a dyno timeout after 30 seconds if it hasn't finished loading. The model loading times should not usually be the problem.

You should check what the heroku logs are before the timeout occurs. If importing the Keras library leads to "Loading Tensorflow backend" and then the dyno times out, you should try to use the TensorFlow Keras extension instead.

I would look at the documentation here: https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model

You could save the model as an h5 and then import it via this function. It should work fine on Heroku - not sure if there are pre-trained models like ResNet in the TF Keras package.

Hope that helps.

This is a bit of a hack, but it should work.

Instead of importing the behemoth keras library, just import the stuff you need from tensorflow.contrib.

keras

tensorflow.contrib

Example:

from tensorflow.python.keras.layers import LSTM, TimeDistributed, Dense, ...


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.