slopfront
Reddit, read-only. No JavaScript. No cookies. No tracking.
preferences

SQLite on Lambda with Python3.6? (read-only)

2 points · submitted Aug 2017 by u/flitsmasterfred

We'd like to use a small SQLite3 database as a read-only data source in our Lambda project. We tried using Python 3.6 but the Lambda Linux doesn't have the native drivers (known problem):

Unable to import module 'my_program': No module named '_sqlite3'

I see some people on stackoverflow messing with SO files and such, is that viable?

This project is a webapp, so Flask (or a stripped down Django) so we'd prefer not to have to include Anaconda/SciPy stuff.

7 comments · 100% upvoted · other discussions

comments

lost_send_berries 2 points · Aug 2017

You could try writing an AWS Lambda function that runs pip wheel sqlite and uploads it. Then that wheel will include the _sqlite3 module which you can include in your deployment .zip.

moduspwnens14 2 points · Aug 2017

Have you had to add platform-specific libraries for a different module before?

Zappa has a github project here where they package various popular Python libraries with the native drivers.

If I'm not misunderstanding something, sqlite3 shouldn't be much different. Essentially you just need to run Amazon Linux (or the Docker image), build in everything you need there, and then package from that.

It can just be tricky because everything you need might not be conveniently in the "site-packages" directory. But it'll be on your Amazon Linux instance somewhere, and it needs to be in the Lambda ZIP.

flitsmasterfred [OP] 1 points · Aug 2017

I think what you describe is what Anaconda does, looks like the most viable way.

jallits 1 points · Aug 2017

It's been a while since I played with Lambda, but as I recall there is no local file store, which means you would have to write to SQLite on an EC2 instance.

flitsmasterfred [OP] 2 points · Aug 2017

We can distribute a file with the code, and lambda has a 512 mb temporary directory we can use to pull a larger file down from S3 and cache on the hot instance.

I've done this readonly data approach with a some fat JSON files before and it works fine, but we need queries and would like to use a standard API like sqlite.

Chokidar1 1 points · Aug 2017

Perhaps you can look into DynamoDB or RDS.

flitsmasterfred [OP] 1 points · Aug 2017

We did, but this use case would be perfect for a read only database and we'd like sqlite for compatibility.