Augmenting APIs with Nginx and Lua.

by Raimon Grau

Nginx as an API proxy

At 3scale we are big fans nginx. Few weeks ago we launched an open-source API proxy which is basically nginx with a set of custom lua scripts that allow for simple configuration of your API proxy. How easy? Check the how-to. It can’t get easier than that.

There are many reasons why you should use nginx as your API proxy. One is because it is open-source. Nginx has a massive install base, it has a very strong community behind it and performance-wise is second to none. For us it is a no-brainer that if open-source solutions exist it is [insert your favorite word here] to rely on proprietary software that claims to do the same.

Another big win of nginx is the lua support. Nginx + lua is a terrific combination that allows to extend the nginx functionalities with an extremely high-performance scripting language. Nginx does a lot of things out of the box, but with lua, the possibilities are unlimited.

The rationale is simple. Is there something that you would like your nginx-based API proxy (or web-server) to do that does not do it out-of-the-box? Add it yourself. It is super simple.

The Victim: Sentiment API (could be any API)

To show how powerful nginx and lua are we will use a simple REST API called Sentiment and augment the heck out of it without touching a single line of the API source code (which by the way is available at github).

The Sentiment API is a very basic API that returns the emotional value of a word or a sentence. For instance, the following request (you can try it yourself)

curl "http://api-sentiment.3scale.net/v1/word/fantastic.json"

will return a JSON with the emotional value of the word fantastic

{"sentiment":4,"word":"fantastic"}

So, we have our victim. Moving along

The What-If part: extending the Sentiment API

There are many ways that you can extend the Sentiment API (or your own API). For the sake of the blog post, we will limit to three different scenarios that will showcase how powerful and extensible nginx+lua can be.

1) Want to do data transformation?

Convert the output from JSON to (god-forbid) XML? Or better yet, transform XML to JSON.

2) Want to change the signature of your API methods?

So instead of nice REST path /v1/word/WORD.json to have a more “classy” signature abusing query_string parameters /sentiment?action=word&word=WORD&version=v1.

We do not condone to go that route :-) the example should be the reverse but since the Sentiment API is already RESTful the example will uglify it instead of the other way around.

3) Want to aggregate API methods to create new ones?

No problem, you can create new API methods fitting your needs. Or more likely, you can extend your API methods without having to touch the source code of your API.

We will show you how we create a totally new method to the Sentiment API that finds the the word with the highest emotional value in a sentence. This method is not provided by the vanilla Sentiment API, but with nginx and lua we can create it right away.

This use case has a lot of potential both for consumers and for producers of an API. It basically allows you to customize an API without changing the source code, or, and that’s the sweet part, allows you to customize an API that you do no control. Want to create a new method that aggregates a bunch of methods of the Twitter API? Well you can, as a result, you application code might be much cleaner.

These are just three examples of extensions that are possible with nginx+lua. There are many others but we just want to highlight how easy it is, and how powerful your API can become by using nginx + lua.

Let’s get started with the real stuff

Extending Nginx with Lua

We will assume you already know your way around basic nginx concepts (servers, locations, etc…)

To extend nginx we will need to add lua support since it is not part of vanilla nginx. That should not scare anyone because there are bundles that have lua already built-in for your convenience. For instance:

If you want to go hard-core :-) You can install the module yourself:

In fact, if you do not want to use lua and prefer perl, you are lucky. Take a look at the CPAN page, which provides full documentation.

The Basics

The overall procedure is to proxy the request to the real API by: 1) doing a sub-request (capture) to the API, 2) grabbing the response, and then 3) mangling it.

This is how the relevant part of the nginx config file should look:

upstream backend {
  # service name: API ;
  server api-sentiment.3scale.net:80 max_fails=5 fail_timeout=30;
}

server {
  listen 8181;

  location ~ /v1/word/(.*)\.json$ {
    proxy_pass http://backend/v1/word/$1.json ;
  }
}

Here we just set up one route in our nginx: /v1/word/your-word-goes-here.json. This route will just return a result from the Sentiment API. Nginx is doing a simple pass-through.

You can start your nginx (listening in localhost port 8181) and do a request like

curl "http://localhost:8181/v1/word/fantastic.json"

and it will return the same JSON

{"sentiment":4,"word":"fantastic"}

We just have done a pass-through to the real Sentiment API. Let us start with the fun

1) Data Transformation

JSON to XML

We will have to add a new route to our nginx conf, so it will look like this

upstream backend {
  # service name: API ;
  server api-sentiment.3scale.net:80 max_fails=5 fail_timeout=30;
}

server {
  listen 8181;

  location ~ /v1/word/(.*)\.json$ {
    proxy_pass http://backend/v1/word/$1.json ;
  }

  location ~ /v1/word/(.*)\.xml$ {
    content_by_lua_file /PATH_TO/json_to_xml.lua;
  }
}

We only added a new route: /v1/word/your-word-goes-here.xml. This route will do the transformation from the JSON output of the vanilla Sentiment API to XML. We are not doing a pass-through, what we are doing here is to call a lua file that implements this logic (do not worry, it is easy).

Now you are able to do the following,

curl "http://localhost:8181/v1/word/fantastic.xml"

And get,

<response>
  <sentiment>4</sentiment>
  <word>fantastic</word>
</response>

What happened here? Well, we basically converted the Sentiment API that outputs JSON to output both JSON and XML!

The “Magic” of Lua

To parse JSON and generate an XML we will require a couple of lua libs:

  • cjson : Install it through luarocks or manually installing it from it’s home page.

  • luaXml : We will use a patched version to make it work with nginx which you can download our patched version from here

If you have problems installing luaxml, a workaround is installing it with luarocks, and put this file under lualib directory inside your openresty. It is a default path were openresty looks for lua libs first.

When we hit the XML route nginx will call the lua file

The lua file does a request to the JSON location, with

 local res = ngx.location.capture("/v1/word/".. m[1] .. ".json" )

which does a pass-through to the real Sentiment API. Once you have the JSON object we convert it to XML with the format that we want, to go from

{"sentiment":4,"word":"fantastic"}

to

<response>
  <sentiment>4</sentiment>
  <word>fantastic</word>
</response>

Note that the split function does not exist in lua but you can use this one.

Right now, the transformation is kind of manual since we know the fields of the JSON, but it can be done automatically, assigning the fields name of the JSON object (the keys of the associative array) to tag names in XML.

Now that we have the conversion to XML done, we can try to additional fields to the XML output. What about timestamp?

Adding a timestamp

Inside lua blocks of code you have the whole lua environment at your disposal, so we’ll use the well known os module to give us the current timestamp.

We just have to add the following lines just before the ngx.say line.

require("os")
response.timestamp = xml.new("timestamp")
table.insert(bar.timestamp, os.date())

And the output of the api when we call /xml will be

<response>
  <sentiment>3</sentiment>
  <word>hello</word>
  <timestamp>Wed Jan  9 15:34:56 2013</timestamp>
</response>

Cool, right? and it was not that hard :)

XML to JSON

Just for the sake of the example we might want to convert the XML output back to JSON. Let us add a new location on our nginx config file:

location ~ ^/round-trip/v1/word/(.*).json$ {
  content_by_lua_file /PATH_TO/xml_to_json.lua;
}

and our xml_to_json.lua will look like this:

As you see, we are hitting our XML endpoint that we just created. Then, we parse the XML using LuaXml to build the proper JSON with cjson.

Note that we are not following any convention in here. Converting XML to JSON is a bit problematic for the general case because XML is more expressive than JSON. To have a general conversion you must decide for a convention, for instance BadgerFish or Parker, or create your own.

2) Rewriting API methods

With nginx is trivial to rewrite your API methods so that they are easier for the developers using your API. The typical example is the old and contorted API that we want to beautify to make it more REST friendly.

One way to solve this issue it to modify the routes of the API in the source code. However, many times you do not want to mess with the code that is working but it is old, undocumented and own by no-one. To overcome the do not touch what is not broken fear we can add a layer in nginx so that we do not have to touch and redeploy that odd code :-)

For the sake of the example, we will convert the REST-like API method

/v1/word/WORD.json

to something more “classy” that abuses the query_string parameters

/sentiment?action=word&version=v1&word=WORD

This “upgrade” can be achieve in multiple ways. For people well-versed in nginx (or other web-servers configuration) this problem is solvable with a simple Rewrite rule. However, if you prefer a less “sysadmin” way of doing things you can just do this:

location ~ /sentiment$ {
    content_by_lua '
      local params = ngx.req.get_query_args()
      if (params.action == "word" and params.version ~= nil) then
        local res= ngx.location.capture("/".. params.version ..
          "/word/" .. params.word .. ".json")
        ngx.say(res.body)
      end
    ';
}

And this is it. Now the sentiment API also accepts old-style API methods like this one:

curl "http://localhost:8181/sentiment?action=word&word=fantastic&version=v1"

which returns the expected JSON object.

3) Data Aggregation

Nginx and lua can also help us to do more complex stuff like creating totally new API methods that are a composition of different methods.

In the example, we will extend the original Sentiment API by creating a new methods that given a sentence it returns the work with the highest emotional value.

Perhaps the use of such a method is nothing to write home about :-) but think of all the times that you have wished that those 4 methods of an API were just one, or the times that you had to call 3 different APIs to do a single task. You can aggregate whatever you want in a single API method that is called by your application!

Let us continue with the example, first we need to create the new end-point,

location ~ ^/v1/max/(.*).json$ {
  content_by_lua_file /PATH_TO/max.lua;
}

and then, we only have to write the aggregation in lua:

As you can see it can not be simpler. First, we get the sentence, strip it words, and for each word we do an API request to /v1/word. We store the object whose sentiment value is higher.

The end result is simple, for a request like this:

curl -g "http://localhost:8181/v1/max/nginx+and+lua+are+amazing.json"

We get the word with highest positive emotion,

{"sentiment":4,"word":"amazing"}

The logic in the aggregator function max.lua can be as complex as you want, and the captures can be used to fetch data from any method API method regardless if you have control over the API or not.

Thinking about mashups? Spot on. You can create arbitrarily large and complex mashups and keep them “hidden” from your application.

Conclusions

The three examples that we covered are only toy experiments of what you can do with nginx and lua.

At 3scale we are using such construction in production, heavy-load production, and we could not be happier with the result.

We keep seeing more and more places where this feature could be exploited, for example, the recent netflix post reminded us that reducing chattiness on APIs can be a big win for some high traffic endpoints or handicapped devices.

Nginx + lua is one of these game-changer technologies that are not yet commonly used, but take our word for it, once you try it out you will get hooked by how powerful, flexible and simple it is.

Augmenting an API has never been so easy.

Published: January 09 2013

  • category:
blog comments powered by Disqus