Making custom APIs

The Worker Gateway service not only exposes the low-level REST API for invocations but also provides a way to define fully customizable HTTP APIs. Future versions will support other protocols as well such as gRPC or GraphQL.

Example

Take the shopping cart example (available via golem-cli new --example rust-example-shopping-cart):

$ golem-cli component get --component-name cart

Component with URN urn:component:c57de1ee-fbe5-4068-a67d-908addc8aa44. Version: 0. Component size is 2900607 bytes.
Component name: cart.
Exports:
	golem:component/api.{initialize-cart}(user-id: string)
	golem:component/api.{add-item}(item: record { product-id: string, name: string, price: float32, quantity: u32 })
	golem:component/api.{remove-item}(product-id: string)
	golem:component/api.{update-item-quantity}(product-id: string, quantity: u32)
	golem:component/api.{checkout}() -> variant { error(string), success(record { order-id: string }) }
	golem:component/api.{get-cart-contents}() -> list<record { product-id: string, name: string, price: float32, quantity: u32 }>

Here is an example of an API definition, which we will use to expose an HTTP API over the function golem:component/api.{get-cart-contents}.

{
  "id": "my-shopping-cart-v1",
  "draft": true,
  "version": "0.0.4",
  "routes": [
    {
      "method": "Get",
      "path": "/v4/{user-id}/get-cart-contents",
      "binding": {
        "type": "wit-worker",
        "componentId": {
          "componentId": "dba38841-013a-49fa-a1dc-064949832f0c",
          "version": 0
        },
        "workerName": "let user: u64 = request.path.user-id; \"my-worker-${user}\"",
        "response": "let result = golem:it/api.{get-cart-contents}(); {status: 200u64, body: result}"
      }
    }
  ]
}

Please use this as a reference instead of using it with zero changes. For example, you might need to give a different version, componentId etc (explained below). You can get the component details including componentId by running golem-cli component list command.

The API definition's fields have the following meaning:

Golem Worker Binding

Let's break down the binding object.

Note that workerName and response are actually rib expressions. It gives you the flexibility to make necessary changes to the input and output to a worker function, as well as dynamically create a worker name.

If your worker name is a constant, the you can simply make it a constant string such as "foo". Please note that it has to be quoted for it a valid Rib string. See the Rib reference for the full reference of the binding language.

In this example, the following Rib expression defines the request:

let result = golem:it/api.{get-cart-contents}();
{status: 200u64, body: result}

The first line is about calling the worker-function and assigning its result to the variable result. The last line is a WASM record where you are mapping to the response. Here body is result itself. That means, we are not manipulating the result returned by the function, and simply forward it as response body. Status is 200u64. u64 is the type of the number as Rib is typed and it needs to know the full type of the WASM record. 200u64 is equivalent to let x: u64 = 200; x; in Rib.

As mentioned above, see the Rib reference to know more about Rib language.

There are various possibilities here to manipulate the data using Rib language, such as selecting only a particular field from the result of the function invocation. Say for example, the result is a WASM result which can be ok or err. In this case you can have a complex Rib expression as below

let result = golem:it/api.{get-cart-contents}();
 
let response_status = match result {
  ok(_) => 200u32,
  err(_) => 400u32
};
 
let response_body = match result {
  ok(result) => result,
  err(msg) => msg
}
 
{status: response_status, body: response_body}
 

The result of the Rib expression is whatever the result of the last line/expression in the above Rib program, similar to many programming languages. Note, the last line of Rib is not ending with a ;.

Upload API definition

The API definitions can be uploaded to Golem using one of the following methods:

• Use the Golem CLI tool
• Use the Rest APIs
• Use Golem Console (opens in a new tab) (Only available in Golem Cloud)

The following example uses golem-cli to add a new API definition to the system.

golem-cli api-definition add api-definition.json
 

This returns:

{
  "id": "my-shopping-cart-v1",
  "version": "0.0.4",
  "routes": [
    {
      "method": "Get",
      "path": "/v4/{user-id}/get-cart-contents",
      "binding": {
        "componentId": {
          "componentId": "dba38841-013a-49fa-a1dc-064949832f0c",
          "version": 0
        },
        "workerName": "let user: u64 = request.path.user-id;\n\"my-worker-${user}\"",
        "idempotencyKey": null,
        "response": "let result = golem:it/api.{get-cart-contents}();\n{status: 200u64, body: result}",
        "responseMappingInput": {
          "types": {}
        },
        "workerNameInput": {
          "types": {
            "request": {
              "type": "Record",
              "fields": [
                {
                  "name": "path",
                  "typ": {
                    "type": "Record",
                    "fields": [
                      {
                        "name": "user-id",
                        "typ": {
                          "type": "U64"
                        }
                      }
                    ]
                  }
                }
              ]
            }
          }
        },
        "idempotencyKeyInput": null
      }
    }
  ],
  "draft": true,
  "createdAt": "2024-10-31T07:40:16.239763054Z"
}

This API definition is still a draft since it is still not deployed yet. Once it is deployed you cannot further update this definition unless you change the version, to avoid breaking your public API.

The API definition can also be uploaded directly using REST APIs.

curl -X POST http://localhost:9881/v1/api/definitions -H "Content-Type: application/json" -d @api-definition.json

Here localhost:9881 is where worker-service (that implements the Worker Gateway) is running, ready to accept these API management requests. The docker-compose example (explained in quick-start) currently exposes the port 9881 for worker-service.

Deploy the API Definition

Using golem-cli, deploying the API definition is straight forward.

golem-cli api-deployment deploy --definition my-shopping-cart-v1/0.0.4 --host localhost:9006
 

The above command returns the following:

 
{
  "apiDefinitions": [
    {
      "id": "my-shopping-cart-v1",
      "version": "0.0.4"
    }
  ],
  "site": {
    "host": "localhost:9006",
    "subdomain": null
  },
  "createdAt": "2024-10-31T07:36:47.497267721Z"
}
 

Here we deploy the definition and now the endpoints should be available at host localhost:9006. If you were following docker examples to spin up Golem, then the port has to be 9006. Refer to .env in docker-examples folder in Golem OSS.

You can test the deployment now, by trying to call the endpoint defined in the API definition.

Watch out for errors during deployment

Please note the following: If you already deployed an API definition before with the same path /{user-id}/get-cart-contents, then you will receive a conflict error as given below, instead of the response above.

API deployment definitions conflict error: /{user-id}/get-cart-contents

This implies, when you update the version, you have to update the path as well, such as /v5/{user-id}/get-cart-contents under the path field in route.

Use the deployed API

Given, you have successfully deployed, now you can try to invoke the following:

curl -H "Accept: application/json" -X GET http://localhost:9006/v4/100/get-cart-contents
[]

Here the worker-gateway identifies the user to be 100 and evaluates the worker-name expr to be my-worker-100 by evaluating workerName Rib expression.

Once it identifies the worker-name, it goes on evaluating the response Rib expression which internally invokes the worker function function golem:it/api.{get-cart-contents} with empty parameters, and simply get the worker-response (which is a WASM value) and converts it to Json and sends it back to the client.

Watch out for errors when using API

Given the Rib script is strongly typed, tere are chances of failures if you break the types. Here is an example:

Based on the workerName script that is deployed above (specified below for quick reference), the user-id in the path should be u64

let user: u64 = request.path.user-id; \"my-worker-${user}\""

This implies Rib expects the input user-id to be u64. Otherwise, it will fail with type-error. If you are using golem-OSS, you can see the following in docker logs

Failed to resolve the API definition; error: Worker binding resolution error: Failed to resolve rib input value from http request details Rib input type mismatch:
Input request details don't match the requirements for rib expression to execute:
Invalid value for the key path.
Error: Invalid value for the key user-id. Error: Expected function parameter type is u64.
But found String.
Requirements. Record(TypeRecord { fields: [NameTypePair { name: "path", typ: Record(TypeRecord { fields: [NameTypePair { name: "user-id", typ: U64(TypeU64) }] }) }] })

This implies the request input should have a field called path in it, which in turn has a field called user-id, and its type should be u64.

This could have been a BadRequest error. But sometimes, the route may not be resolved due to wrong types. and this time client may not get back the reason for error. We will try to improve this as we go. In any case, during development phase, you will need to investigate errors like this by inspecting the logs in worker-service docker container.

REST API to deploy API

As of now, we have deployed the API using golem-cli. We can do the same using REST APIs as an alternative.

For this, let's create a deployment.json as given below. Based on the example above, the apiDefinitionId is shopping-cart-v1 and version is 0.0.4.

If you were having a domain registered such as my-site.com, you would replace localhost:9006 with my-site.com, that in turn redirects to localhost:9006, or wherever the worker-service is running (perhaps AWS).

{
  "apiDefinitions": [
    {
      "id": "my-shopping-cart-v1",
      "version": "0.0.6"
    }
  ],
  "site": {
    "host": "locahost:9006"
  }
}

subdomain field under site is optional. But you can provide this if its backed by a valid host. For localhost subdomain doesn't make much sense.

 
curl -X POST http://localhost:9881/v1/api/deployments/deploy -H "Content-Type: application/json"  -d @api-deployment.json
 
{"apiDefinitions":[{"id":"my-shopping-cart-v1","version":"0.0.4"}],"createdAt":"2024-10-31T07:59:00.390103588+00:00","site":{"host":"locahost:9006","subdomain":null}}⏎

More details on Response Mapping

Response Mapping is under the response field under each binding. This is written using Rib to call the worker function and manipulate it's results and return a value that's understandable for each protocol we use.

Here is a more complex example of an API definition that gives you more idea on what can be achieved with Rib expression under response field.

Please use this as a reference instead of using it with zero changes.

{
  "id": "my-shopping-cart-v2",
  "draft": true,
  "version": "2.0.11",
  "routes": [
    {
      "method": "Get",
      "path": "/{user-id}/get-cart-contents",
      "binding": {
        ..
        "workerName": "\"my-worker\"",
        "response": "let result = golem:it/api.{get-cart-contents}(); {status: 200u64, body: result}"
      }
    },
    {
      "method": "Post",
      "path": "/{user-id}/initialize-cart",
      "binding": {
        ..
        "workerName": "\"my-worker\"",
        "response": "let empty: list<u16> = []; ${golem:it/api.{initialize-cart}(request.path.user-id); {status: 200u64, body: empty}}"
      }
    },
    {
      "method": "Post",
      "path": "/{user-id}/add-item",
      "binding": {
        ..
        "workerName": "\"my-worker\"",
        "response": "let quantity: f64 = request.body.quantity; let product_id: u64 = request.body.product-id; let name: str = request.body.name; let price: f64 = request.body.price; let input = {product-id: product_id, name: name, price: price, quantity: quantity}; golem:it/api.{add-item}(input); {status: 200u64, body: \"success\"}"
      }
    },
    {
      "method": "Post",
      "path": "/{user-id}/checkout",
      "binding": {
        ..
        "workerName": "\"my-worker\"",
        "response": "let result = golem:it/api.{checkout}(); {status: 200u64, body: match result { success(resp) => resp.order-id, error(msg) => msg }}"
      }
    }
  ]
}

This example exposes multiple functions of the shopping-cart component through a custom HTTP API. Some of these examples include longer Rib scripts under response field. Also since you are not using golem console (from golem cloud), and relying on directly encoding Rib in a Http request, everything had to be in 1 single line, since multi-line doesn't work in JSON body in http request. Improvements in this space are in progress.

Note that each line in Rib code block is separated by ; and the last line in the Rib script is the return value which shouldn't have ;.

Most of the above examples are self explanatory, but worth pointing out a few details.

Some of the response fields above have Rib script where we pass parameters to the function. This is more or less same as many programming languages - a comma separated list of arguments. Arguments are valid Rib code.

 
golem:it/api.{initialize-cart}(request.path.user-id)
 

Here Rib compiler knows the requirement of initialize-cart function and infers that request.path.user-id should be u64. This is a hint, that Rib is aware of the types in the code. In places where it find hard to infer these types, you may face some compile time errors (when uploading API definition), and the best solution is to be explicit about types. For example

let user_id: u64 = request.path.user-id;
golem:it/api.{initialize-cart}(user_id)
 

The most interesting example out of all the endpoints is the checkout endpoint.

Here the response mapping in Rib is this:

let result = golem:it/api.{checkout}();
 
{status: 200u64, body: match result { success(resp) => resp.order-id, error(msg) => msg }}
 

This can also be written as the following for more readability (with golem console (opens in a new tab):

let result = golem:it/api.{checkout}();
let status: u64 = 200;
let body =  match result { success(resp) => resp.order-id, error(msg) => msg };
{status: status, body: body}

Here we manipulate the result from worker function using a match expression. More details on match expression is explained under Rib documentation. It matches on success and error variants because the result type of golem:it/api.{checkout} is a variant which can be either success or error.

Looking up data from the request body and path

In the above example, we have an endpoint to add products into the cart.

{
  "method": "Post",
  "path": "/{user-id}/add-item",
  "binding": {
    ....
    "response": "${let product_id: u64 = request.body.product-id; golem:it/api.{add-item}({product-id: product_id}); let empty_body: list<u16> = []; {status: 200u64, body: empty_body}}"
  }
}

This implies, the http request should be method POST with request body having the following fields: product-id, name, price and quantity.

Input request

Worker gateway allows you to look up the values of request path or query variables using request.path.* syntax. For request body, you can use request.body.* syntax. Similarly for headers it is request.headers.*

Constraints on Components after deployment

Please note that, once you have deployed an API using a specific component-version, then you may not be able to update the component-version (ex: golem-cli component update) unless the arguments to the functions used in the Rib script in the deployed API definition is backward compatible.

If there are conflicts (i.e, backward incompatible) then you will get a conflict report message, which details especially the missing functions and conflicting functions, where conflicting functions detail about existing parameter types, new parameter types, existing return type, and new return type.

Support to import OpenAPI spec

Until now, we used the native API definition format of Golem. Golem allows you to import API definition in OpenAPI spec format too. This is because, users may have already written an OpenAPI spec for their endpoints for various purposes. By adding extra details of into the same spec, we can use it as an API definition. Internally it gets converted to the Worker Gateway's native format of API definition, discussed in the beginning of this documentation.

The main advantage of this feature is the re-usability of the same endpoint definitions across your system. For example, you can use the same file now to register with Golem's Worker Gateway and register with another external API gateway. More on this below. However once deployed, it returns native format then onwards, and updates needs to be done in the native format.

{
  "openapi": "3.0.0",
  "info": {
    "title": "MyOpenAPISpec",
    "version": "1.0.2"
  },
  "x-golem-api-definition-id": "my-shopping-cart-v1",
  "x-golem-api-definition-version": "0.0.7",
  "paths": {
    "/{user-id}/get-cart-contents": {
      "x-golem-worker-bridge": {
        "worker-name": "\"foo\"",
        "component-id": "dba38841-013a-49fa-a1dc-064949832f0c",
        "component-version": 0,
        "response": "let x = golem:it/api.{checkout}();\nlet status: u64 = 200; {headers : {ContentType: \"json\", userid: \"foo\"}, body: \"foo\", status: status}"
      },
      "get": {
        "summary": "Get Cart Contents",
        "description": "Get the contents of a user's cart",
        "parameters": [
          {
            "name": "user-id",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/CartItem"
                }
              }
            }
          },
          "404": {
            "description": "Contents not found"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "CartItem": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "price": {
            "type": "number"
          }
        }
      }
    }
  }
}

To import OpenAPI spec using Golem CLI,

 
golem-cli api-definition import open_api.json
 

This will return

 
{
  "id": "my-shopping-cart-v1",
  "version": "0.0.7",
  "routes": [
    {
      "method": "Get",
      "path": "/{user-id}/get-cart-contents",
      "binding": {
        "componentId": {
          "componentId": "dba38841-013a-49fa-a1dc-064949832f0c",
          "version": 0
        },
        "workerName": "\"foo\"",
        "idempotencyKey": null,
        "response": "let x = golem:it/api.{checkout}();\nlet status: u64 = 200;\n{headers: {ContentType: \"json\", userid: \"foo\"}, body: \"foo\", status: status}",
        "responseMappingInput": {
          "types": {}
        },
        "workerNameInput": {
          "types": {}
        },
        "idempotencyKeyInput": null
      }
    }
  ],
  "draft": true,
  "createdAt": "2024-10-31T08:56:02.412860043Z"
}
 

Integrating worker-gateway with existing API Gateways

Note that, while the Golem's Worker Gateway is a powerful tool for defining and managing API endpoints, it is not a full-fledged API Gateway. However, it can be used conjunction with existing API Gateways, allowing you to leverage the capabilities of both systems. For example, you can use Golem's Worker Gateway to define and manage the worker bindings for your API endpoints, while using an API Gateway to handle other aspects of API management, such as authentication, rate limiting, and monitoring. In this scenario, the third party API Gateway would route incoming requests to Golem based on the defined endpoints, allowing Golem's Worker Gateway to handle the request processing and response generation.

Tyk

This section shows how to integrate with Tyk API gateway (opens in a new tab).

Note that Tyk allows users to upload OpenAPI spec similar to Golem. You can upload the same OpenAPI spec with worker-gateway info to Tyk, with 1 more extra information which is servers block with the value of the URL of the Golem Worker Gateway (which is the URL of the worker-service service in docker), that tells the Tyk API gateway of Tyk to route the request to worker-gateway.

Please note, the above set up will work depending on how you installed Tyk. If you installed Tyk in the same network as worker-gateway, you can use the localhost as servers.

If you are using a separate docker network with Tyk, you will need to give the machine IP address to reach the Worker Gateway URL.

{
  "openapi": "3.0.0",
  "x-golem-api-definition-version": "0.0.3",
  "x-golem-api-definition-id": "shopping-cart-v1",
  "info": {
    "title": "Sample API",
    "version": "1.0.2"
  },
  "servers": [
    {
      "url": "http://ip-address-of-your-local-machine:9881"
    }
  ],
  "paths": {
    "/{user-id}/get-cart-contents": {
      "x-golem-worker-bridge": {
        "worker-name": "\"foo\"",
        "component-id": "dba38841-013a-49fa-a1dc-064949832f0c",
        "component-version": 0,
        "response": "let x = golem:it/api.{checkout}();\nlet status: u64 = 200; {headers : {ContentType: \"json\", userid: \"foo\"}, body: \"foo\", status: status}"
      },
      "get": {
        "summary": "Get Cart Contents",
        "description": "Get the contents of a user's cart",
        "parameters": [
          {
            "name": "user-id",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/CartItem"
                }
              }
            }
          },
          "404": {
            "description": "Contents not found"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "CartItem": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "price": {
            "type": "number"
          }
        }
      }
    }
  }
}

The difference here is we added the server's block. You can include this servers block from the beginning so that it's exactly the same OpenAPI spec which you used to upload to Golem as well as Tyk.

Install Tyk

git clone https://github.com/TykTechnologies/tyk-gateway-docker
cd tyk-gateway-docker
docker-compose up

Registration with Tyk

Let's say we saved the above json as open_api.json

curl -X POST http://localhost:8080/tyk/apis/oas/import --header 'x-tyk-authorization: foo' --header 'Content-Type: text/plain' -d @open_api.json
 

Reload the Tyk API Gateway, otherwise the API is not deployed with Tyk yet, so this is an important step. Note that, if you are encountering issues following these steps, please refer to Tyk documentations.

curl -H "x-tyk-authorization: foo" -s http://localhost:8080/tyk/reload/group
 

Note that Tyk is now running at 8080, and now requests has to go into 8080 and not the Golem Worker Gateway.

curl -X GET http://localhost:9006/adam/get-cart-contents

FAQ

How does worker service know which API definition to pick for a given endpoint?

When a request comes in, worker-service looks at the host in the request and matches it with the site in the deployment. If there is a deployment corresponding to the site, it picks the API definition ID and version from the deployment and gets the API definition, to further process the request