binpress

Building a Robust JSON API Client with Ruby

If you’re building a Ruby on Rails site and consuming a popular API, odds are there’s a gem for it and it’s as simple as a few lines of code to query and return what you need. But on the other hand, you could be introducing low-quality gem code into your application, a much bigger library than your use case requires, or code you just don’t understand well. Instead of pulling in 60k of Ruby, you might be able to build your own in 60 lines. If it’s a smaller service or a private API that was built just for you, you probably need to roll your own API client anyway. When I needed to integrate a service for showing sample photographs on my camera rental web site, I built a very simple JSON API client using HTTParty in Ruby.

Testing the API

Nothing is more frustrating than wasting time debugging an API only to discover that your API key was invalid or the failure was otherwise on the side of the API server. In my case, the API maintainer gave me the following test API call:

  1. http://www.pixel-peeper.com/rest/?method=list_photos&api_key=[REDACTED]&camera=[CAMERAID]

Before doing anything else, you should test the API in your browser (if it’s a simple GET based API) or using curl from the command line if you need to POST data. You should see the JSON response:

  1. {
  2.   data: {
  3.     results: [
  4.       {
  5.         camname: "EOS 5D Mark III",
  6.         cammake: "Canon",
  7.         camexifid: "CANON EOS 5D MARK III",
  8.         lensname: "Canon EF 24-105mm f/4 L IS USM",
  9.         author_name: null,
  10.         author_url: null,
  11.         id: "8175491230",
  12.         iso: "1600",
  13.         aperture: "4",
  14.         exposure: "0.00625",
  15.         focal_length: "105",
  16.         small_url: "http://farm9.staticflickr.com/8478/8175491230_c94258586b_m.jpg",
  17.         pixels: "22118400",
  18.         lens_id: "920",
  19.         flickr_user_id: "23722023@N06",
  20.         camera_id: "1659",
  21.         big_url: "https://www.flickr.com/photos/23722023@N06/8175491230/sizes/o/"
  22.         },
  23.       ...

Minimum viable integration

Ok, the API works manually — time to code the bare minimum to replicate in Ruby. One of the decisions to make is which library to use to interact with the API. Ruby’s standard library has open-uri built in which provides you with open(url). However, there doesn’t seem to be a good way to set timeouts using open(). The best “solution” I found to this was to require 'timeout' and use a Timeout::timeout(seconds) do block — which seemed like a hack to me. Net::HTTP is a great library for interacting over HTTP, but I’m going one step further and using HTTParty, which uses Net::HTTP under the hood. HTTParty provides much of the boilerplate around interacting with an API, is well tested, etc. Other good alternatives include Faraday, which provides low-level controls over the HTTP requests. If you have a finicky or complicated API, I’d be more inclined to use Faraday. For our purposes, HTTParty will be just fine.

Following the sample code for HTTParty, we end up with a relatively short library:

  1. require 'httparty'
  2.  
  3. class PixelPeeper
  4.   include HTTParty
  5.   base_uri 'www.pixel-peeper.com'
  6.  
  7.   def api_key
  8.     ENV['PIXELPEEPER_API_KEY']
  9.   end
  10.  
  11.   def base_path
  12.     "/rest/?method=list_photos&api_key=#{ api_key }"
  13.   end
  14.  
  15.   def examples_for_camera(camera_id, options = {})
  16.     url = "#{ base_path }&camera=#{ camera_id }"
  17.     self.class.get(url, options)['data']['results']
  18.   end
  19.  
  20.   def examples_for_lens(lens_id, options = {})
  21.     url = "#{ base_path }&lens=#{ lens_id }"
  22.     self.class.get(url, options)['data']['results']
  23.   end
  24. end

It’s a best practice to not include secrets like API keys in source code/source control, so we’re reading the key from an environment variable. If you’re running Heroku, you’ll have to set this via heroku config:set PIXELPEEPER_API_KEY=$key. For your local server/consoles, you’ll need to export the variable or set it inline when you invoke your command.

By including HTTParty in your class, all of your API GET requests are invoked through self.class.get, it will use all the built-in HTTParty tricks, like using the base_uri and automatically decoding the JSON response into a native hash. As a result of leveraging HTTParty, you don’t need much code for a fully-functional API client.

Timeouts

A fully-functional, but not a robust API client. If your Rails app calls the API inline with a request (which is common), a 30-second API response will mean your site also takes 30 seconds to respond. If many requests hit the same slow API, it could tie up all your web servers and bring your site down. This used to be common for Facebook apps before they solidified their graph API. The best practice for consuming an API inline with requests is to hard timeout and gracefully degrade. In our case, we want to hard timeout after just one second and return a “fake” empty request, so the template that consumes it can gracefully degrade.

HTTParty will raise Net::OpenTimeout if it can’t connect to the server and Net::ReadTimeout if reading the response from the server times out (both in the case that it stalls sending data or is still sending data). So we simply need to handle both exceptions to return empty hashes, and set the timeout to 1 second. Instead of implementing this timeout-handling logic in both methods, I’m opting for a handle_timeouts function that takes a block. If the block raises an exception, handle_timeouts will catch the exception and return an empty hash.

  1. require 'httparty'
  2.  
  3. class PixelPeeper
  4.   include HTTParty
  5.   base_uri 'www.pixel-peeper.com'
  6.   default_timeout 1 # hard timeout after 1 second
  7.  
  8.   def api_key
  9.     ENV['PIXELPEEPER_API_KEY']
  10.   end
  11.  
  12.   def base_path
  13.     "/rest/?method=list_photos&api_key=#{ api_key }"
  14.   end
  15.  
  16.   def handle_timeouts
  17.     begin
  18.       yield
  19.     rescue Net::OpenTimeout, Net::ReadTimeout
  20.       {}
  21.     end
  22.   end
  23.  
  24.   def examples_for_camera(camera_id, options = {})
  25.     handle_timeouts do
  26.       url = "#{ base_path }&camera=#{ camera_id }"
  27.       self.class.get(url, options)['data']['results']
  28.     end
  29.   end
  30.  
  31.   def examples_for_lens(lens_id, options = {})
  32.     handle_timeouts do
  33.       url = "#{ base_path }&lens=#{ lens_id }"
  34.       self.class.get(url, options)['data']['results']
  35.     end
  36.   end
  37. end

This class now meets my requirements for “production-ready,” since the worst case is adding one second to the request and not showing content, in the case of a timeout. The API server going down or taking a long time to respond can never adversely affect our site more than that.

Caching API responses locally

We’ve protected our site, but one second is still a lot to add to each request — especially if it’s a response we see quite a lot. By implementing a local cache in Redis, we can obviate the need for an external API request and return in mere milliseconds with the cached data. Additionally, your cached responses will still be there if the API server goes down. And in general, it’s being a nice Internet neighbor to not overwhelm the API server with what are essentially duplicate requests. Be careful though; some API servers disallow local caching of responses in their Terms of Service. But unless its sensitive user data, most API servers would prefer that you cache.

Following the style of handle_timeouts, we’re handling caching via the handle_cachingmethod, which again takes a block (and also an options parameter). Both camera and lense methods have been folded into one examples(options) method, where options is a hash with either a camera_id or lends_id key set.

  1. require 'httparty'
  2.  
  3. class PixelPeeper
  4.   include HTTParty
  5.   base_uri 'www.pixel-peeper.com'
  6.   default_timeout 1 # hard timeout after 1 second
  7.  
  8.   def api_key
  9.     ENV['PIXELPEEPER_API_KEY']
  10.   end
  11.  
  12.   def base_path
  13.     "/rest/?method=list_photos&api_key=#{ api_key }"
  14.   end
  15.  
  16.   def handle_timeouts
  17.     begin
  18.       yield
  19.     rescue Net::OpenTimeout, Net::ReadTimeout
  20.       {}
  21.     end
  22.   end
  23.  
  24.   def cache_key(options)
  25.     if options[:camera_id]
  26.       "pixelpeeper:camera:#{ options[:camera_id] }"
  27.     elsif options[:lens_id]
  28.       "pixelpeeper:lens:#{ options[:lens_id] }"
  29.     end
  30.   end
  31.  
  32.   def handle_caching(options)
  33.     if cached = REDIS.get(cache_key(options))
  34.       JSON[cached]
  35.     else
  36.       yield.tap do |results|
  37.         REDIS.set(cache_key(options), results.to_json)
  38.       end
  39.     end
  40.   end
  41.  
  42.   def build_url_from_options(options)
  43.     if options[:camera_id]
  44.       "#{ base_path }&camera=#{ options[:camera_id] }"
  45.     elsif options[:lens_id]
  46.       "#{ base_path }&lens=#{ options[:lens_id] }"
  47.     else
  48.       raise ArgumentError, "options must specify camera_id or lens_id"
  49.     end
  50.   end
  51.  
  52.   def examples(options)
  53.     handle_timeouts do
  54.       handle_caching(options) do
  55.         self.class.get(build_url_from_options(options))['data']['results']
  56.       end
  57.     end
  58.   end
  59. end

All of the caching logic is in handle_caching(options) and cache_key(options), the latter of which builds a unique key to store the cached response based on which type of request and for which ID. Redis encourages human-readable key names, but if your key name starts getting unwieldy or unpredictabel in length, it’s perfectly reasonable to compute a hash of your unique identifiers, like SHA1(options.to_json) (pseudocode).

handle_caching(options) checks for the existence of a key and returns the payload if available. Otherwise, it yields to the block passed in and stores the result in Redis. Object#tap is a neat method that always returns the Object, but gives you a block with the object as the first named parameter. It’s a nice pattern for when you finish computing your return value but still need to reference it (to store in a cache, to log, to send an email with, etc).

Consuming the API

Using the API client is very straightforward: one line to create an API client instance, and one line to query by camera_id or lens_id.

  1. def example_pictures_for(gear)
  2.   pp = PixelPeeper.new
  3.   if gear.pp_lens_id.present?
  4.     pp.examples(lens_id: gear.pp_lens_id)
  5.   elsif gear.pp_camera_id.present?
  6.     pp.examples(camera_id: gear.pp_camera_id)
  7.   else
  8.     []
  9.   end.take(8)
  10. end

Summary and final thoughts

As we’ve shown, HTTParty can be used to quickly create a robust and performant JSON API client. Using helper methods that take blocks allow us to create composable helpers and keep concerns separated into their own functions, instead of intermingled.

Should you build your own API client or use an existing gem? There’s not a single answer for every context; some API client gems are very high quality and maintained quite well. But if the quality of the client is doubtable or you only need to use a small subset of an API’s functionality, you may want to consider writing your own simple API client.

Author: Adam Derewecki

Contact
Contact
Scroll to Top