slopfront
Reddit, read-only. No JavaScript. No cookies. No tracking.
preferences

Tracking API usage

5 points · submitted Dec 2012 by u/xangelo

Hey all,

I'm working on a ReSTful API that's getting ready to be released, running in Node.js. One of the things that we want to before we start opening it up publically is to track API usage. Each user has their own API key that they'll be utilizing to access our endpoints.

I have a few ideas for tracking API usage, but I was hoping someone could suggest a few things that I missed.

Architecture overview:

  • MySQL as our storage
  • Redis as a caching layer
  • Heroku

Ideas

  1. Google Analytics - our app could toss a payload on a proper queue system (gearman, rabbit) or Redis. A secondary app could grab the payload and make the request to the .gif that GA uses.
  2. Write payloads to redis, nightly push to MySQL
  3. Write straight to MySQL
  4. Parse logs

I'm personally leaning towards idea 1, probably utilizing Redis as our queue storage since it's in place already, but I'd love to hear what other ideas people have for tracking API usage.

6 comments · 78% upvoted · other discussions

comments

rooosta 1 points · Dec 2012

Unless you're expecting huge volume from day 1, I'd go with "the simplest thing that could possibly work". For me that's usually writing directly to MySQL because:

  • I already have mysql set up, monitored, etc.
  • I know I'll be able to write arbitrary queries against it to answer whatever questions I come up with
  • When it becomes a performance bottleneck or queries get slow, etc., I can use my usual bag of tricks to make it scale (batch via log files + load data infile, etls for complex queries, etc.).

This is we're doing for ratchet.io and what I've done previously on sites that scaled to 10k+ requests/sec.

xangelo [OP] 1 points · Dec 2012

Thanks, this is actually the solution we ended up going with. We're toying with the idea of utilizing our cache (redis atm) to do storage of api requests and then having a node instance just popping the data off redis and writing to mysql.

This should scale well, but I'm hoping to do some tests with it before we actually open up to the public.

rooosta 1 points · Dec 2012

Cool. That sounds like it could work well, especially if you batch the writes to mysql.

Another related option is to do an atomic collection swap within redis (redis may call this something else -- atomic queue rename?). That amounts to a long running process that does the following periodically:

  1. your app is writing to a queue named "logs"
  2. long running process creates a new queue named "logs-new"
  3. atomically rename "logs" to "logs-{timestamp}", and "logs-new" to "logs"
  4. write everything in "logs-{timestsamp}" to a file, and load it into mysql using LOAD DATA INFILE

I keep mentioning LOAD DATA INFILE because it is really, really fast, significantly faster than regular bulk INSERTs and I've seen it be 1000s of times faster than individual inserts.

xangelo [OP] 1 points · Dec 2012

I've never actually used LOAD DATA INFILE, but it looks like a way to do exactly what we need a lot easier than doing it ourselves.

I'm not too sure about atomic operations in the fashion we require, but it is something I am investigating.