http.q

HTTP Query Library

This library provides HTTP GET, POST, PUT and DELETE interfaces with additional features from the default GET (.Q.hg) and POST (.Q.hp):

• Additional headers can be sent with each request
• The "User-Agent" header will be sent (if configured), to identify the source process
• HTTP repsonses are parsed into a dictionary for easier processing of HTTP result
• Automatically follow redirects
• Automatic URL encoding of query string fields and values
• Automatic conversion of responses based on Content-Type

The following features present in .Q.hg and .Q.hp are also supported:

• Routing via proxy if HTTP_PROXY or HTTPS_PROXY set
• Bypassing proxy if target URL present in NO_PROXY
• Basic authentication supported with http://username:password@host syntax
• GZIP encoding request and decoding (only with kdb+ 4.0 or greater)

The WebSocket client library wsc.q is built on top of the internal functions within this library.

There are 5 standard HTTP method functions:

• .http.get[url; headers]
• .http.post[url; body; contentType; headers]
• .http.put[url; body; contentType; headers]
• .http.delete[url; body; contentType; headers]
• .http.patch[url; body; contentType; headers]

NOTE: All of these functions send the header Connection: close.

These standard HTTP method functions all defer to .http.send for sending and parsing the response.

Configuration

The following configuration is available:

• .http.cfg.cacheProxy
  • If true, current proxy settings will be loaded on library initialisation only. If false, the proxy settings will be queried on every HTTP invocation
• .http.cfg.sendUserAgent
  • If true, the user agent header will be sent with each request. A default is built on library initialisation, but can be manually specified by overriding .http.userAgent
• .http.cfg.errorOnInvaildContentEncoding
  • If true, if a HTTP response contains a content encoding that is not supported, throw an exception. If false, an error will be logged but the body will be returned as received
• .http.cfg.followRedirects
  • If true, if the HTTP response is a redirect type, another request to the specified target location will be made. If false, the redirect response will be returned
• .http.cfg.jsonContentTypes
  • The list of Content-Types to indicate a JSON repsonse and automatically run the JSON parser (.j.k) on it
  • By default this includes application/json only
• .http.cfg.autoContentTypes
  • Dictionary of Content-Type values and the function to automatically convert the response
  • By default this includes application/json and application/kdb-ipc
• .http.cfg.gzipContentEncodings
  • List of Content-Encoding values that should be treated as GZIP compressed and decompressed if available

.http.send[method; url; body; contentType; headers]

This function parses the URL, builds the HTTP request based on the specified parameters, sends the request and parses the response.

The redirect logic is also present in this function, if enabled by .http.cfg.followRedirects

The following headers are set:

• Content-Type
  • If body is not empty, it will be set to the supplied contentType or text/plain if none is specified
• Authorization
  • If the URL format is http://username:password@host
• User-Agent
  • If configured with .http.cfg.sendUserAgent
• Accept-Encoding
  • If running kdb+ 4.0 or later and GZIP decoding is available
• Accept
  • If not specified by the caller and .http.acceptHeader is non-empty (default is */*)

.http.status

This dictionary provides English descriptions of all of the HTTP status codes as defined by https://httpstatuses.com/.

q) .http.status 418
"I'm a teapot"

HTTP Response Format

Using the main HTTP functions, .http.send or using .http.i.parseResponse will return a parsed version of the HTTP response in dictionary format.

The returned keys are:

• statusCode: The HTTP status code as an integer (e.g. 200i)
• statusType: A symbol representing the status code type (e.g. success for status code 200)
• statusDetail: String of the status text from the HTTP response (e.g. OK for status code 200)
• headers: Dictionary of headers, the dictionary key being the header name and the dictionary value being the header value
• body: A string of all remaining content in the response after the headers, joined with the new line \n character
  • If the response is JSON (Content-Type matching one of .http.cfg.jsonContentTypes), it will be converted with .j.k

Status Types

The following status types are returned in the parsed response:

• informational: 100 -> 199
• success: 200 -> 299
• redirect: 300 -> 399
• clientError: 400 -> 499
• serverError: 500 -> 599

Examples

HTTP GET

• Equivalent of .Q.hg:
  • .http.get["http://httpbin.org/get"; ()!()]
• Equivalent of .Q.hg with GZIP decoding:
  • .http.get["http://httpbin.org/gzip"; ()!()]
• Additional headers:
  • .http.get["http://httpbin.org/get"; enlist[`$"X-Test-Header"]!enlist "test-value"]
• Query string URL encoding:
  • .http.get["http://httpbin.org/get?key 1=value 1"; ()!()]

kdb+ IPC over HTTP

In order to take advantage of kdb+ IPC over HTTP, a webserver should return the IPC bytes representation of the response along with the content type application/kdb-ipc.

The benefits of this are reduced response size, particularly when GZIP compression is not available:

Response Type	No GZIP Response	GZIP Response
JSON	848,990 bytes	125,871 bytes
kdb+ IPC	560,235 bytes	122,574 bytes

/ Webserver

q) .z.ph:{ get 1_ x 0 };
q) .h.ty[`kdbipc]:"application/kdb-ipc";

q) tbl:update time:.z.p + 100 from flip `sym`price`volume!100?/:(`3; 10f; 100000j);
q) .http.asIpc: { :.h.hnz["200 OK"; 0b; `kdbipc; raze string -18!tbl] };
q) .http.asJson:{ :.h.hnz["200 OK"; 0b; `json; .j.j tbl] };

/ Client

q).http.get["http://localhost:12345/#.http.asIpc[]"; ()!()]`body
...
sym price     volume time
--------------------------------------------------
mil 0.6165008 12378  2021.03.21D10:19:20.121358100
igf 2.85799   63682  2021.03.21D10:19:20.121358100
...