HTTP Module

The http module provides secure HTTP request functionality with comprehensive features for API integration, web scraping, and external service communication. It includes built-in SSRF protection, automatic retries, and response parsing.

Functions

http.request(request_table)

Makes an HTTP request with extensive configuration options and security features.

Parameters:

• request_table (table): Request configuration with the following keys:
  • method (string, optional): HTTP method ("get", "post", "put", "patch", "delete", "head", "options"). Default: "get"
  • url (string, required): Target URL (must be HTTPS for production)
  • headers (table, optional): Map of HTTP headers
  • body (string, optional): Request body (for POST, PUT, PATCH)
  • format (string, optional): Response format ("json", "xml", "text"). Default: "json"
  • params (table, optional): Query parameters (will be URL-encoded)
  • auth (table, optional): Authentication configuration:
    • For Basic auth: {type = "basic", username = "user", password = "pass"}
    • For Bearer auth: {type = "bearer", token = "your-token"}
  • timeout (number, optional): Request timeout in seconds. Default: 30
  • follow_redirects (boolean, optional): Follow HTTP redirects. Default: true
  • max_redirects (number, optional): Maximum redirects to follow. Default: 5

Returns:

• Table with the following structure:
  • success (boolean): true if request succeeded, false otherwise
  • data (table): Response data when successful:
    • status (number): HTTP status code
    • headers (table): Response headers
    • body (mixed): Response body (parsed based on format)
  • error (string): Error message when request fails

Basic Examples:

-- Simple GET request
local response = http.request({
    url = "https://api.example.com/health"
})

if response.success then
    print("API is healthy: " .. response.data.body.status)
else
    print("API check failed: " .. response.error)
end

-- GET with query parameters
local response = http.request({
    method = "get",
    url = "https://api.example.com/search",
    params = {
        q = "print jobs",
        limit = 10,
        offset = 0
    },
    headers = {
        ["Accept"] = "application/json"
    }
})

-- POST with JSON body and Bearer authentication
local user_data = {
    name = "John Doe",
    email = "john@example.com",
    role = "operator"
}

local response = http.request({
    method = "post",
    url = "https://api.example.com/users",
    headers = {
        ["Content-Type"] = "application/json"
    },
    auth = {type = "bearer", token = api_token},
    body = encoding.json(user_data),
    format = "json"
})

if response.success then
    print("Created user ID: " .. response.data.body.id)
end

-- PUT request with form data
local response = http.request({
    method = "put",
    url = "https://api.example.com/profile",
    headers = {
        ["Content-Type"] = "application/x-www-form-urlencoded"
    },
    body = "name=Jane%20Smith&email=jane%40example.com",
    format = "json"
})

-- DELETE request
local response = http.request({
    method = "delete",
    url = "https://api.example.com/sessions/" .. session_id,
    headers = {
        ["Authorization"] = "Bearer " .. api_token
    }
})

Advanced Request Patterns

API Authentication Methods

The http module now has built-in support for common authentication methods:

-- Bearer Token Authentication (built-in)
local response = http.request({
    method = "get",
    url = "https://api.example.com/protected",
    auth = {type = "bearer", token = "your-api-token"},
    format = "json"
})

-- Basic Authentication (built-in)
local response = http.request({
    method = "get",
    url = "https://api.example.com/secure",
    auth = {type = "basic", username = "user", password = "pass"},
    format = "json"
})

-- Helper function for Bearer Token requests
function api_request_with_bearer(endpoint, token, method, body)
    return http.request({
        method = method or "get",
        url = "https://api.example.com" .. endpoint,
        auth = {type = "bearer", token = token},
        headers = {
            ["Content-Type"] = "application/json"
        },
        body = body and encoding.json(body),
        format = "json"
    })
end

-- Helper function for Basic Auth requests
function api_request_with_basic_auth(url, username, password, method)
    return http.request({
        method = method or "get",
        url = url,
        auth = {type = "basic", username = username, password = password},
        format = "json"
    })
end

-- API Key Authentication (via headers)
function api_request_with_key(endpoint, api_key)
    return http.request({
        url = "https://api.example.com" .. endpoint,
        headers = {
            ["X-API-Key"] = api_key
        },
        format = "json"
    })
end

-- OAuth Bearer Token
function api_request_with_oauth(endpoint, oauth_token)
    return http.request({
        url = "https://api.example.com" .. endpoint,
        auth = {type = "bearer", token = oauth_token},
        format = "json"
    })
end

Pagination Handling

function fetch_paginated_data(base_url, api_token, page_size)
    local all_data = {}
    local page = 1
    local has_more = true

    while has_more do
        local response = http.request({
            url = base_url,
            params = {
                page = page,
                per_page = page_size or 100
            },
            headers = {
                ["Authorization"] = "Bearer " .. api_token
            },
            format = "json"
        })

        if not response.success then
            print("Failed to fetch page " .. page .. ": " .. response.error)
            break
        end

        local data = response.data.body

        -- Append results
        for _, item in ipairs(data.items or data.results or data) do
            table.insert(all_data, item)
        end

        -- Check for more pages (API-specific logic)
        has_more = data.has_next or
                   data.next_page or
                   (#data.items == page_size)
        page = page + 1

        -- Respect rate limits
        if has_more and page % 10 == 0 then
            -- Brief pause every 10 requests
            print("Fetched " .. #all_data .. " items so far...")
        end
    end

    return all_data
end

File Upload

function upload_file(file_content, filename, content_type)
    -- Multipart form data boundary
    local boundary = "----FormBoundary" .. os.time()

    -- Build multipart body
    local body_parts = {
        "--" .. boundary,
        'Content-Disposition: form-data; name="file"; filename="' .. filename .. '"',
        "Content-Type: " .. content_type,
        "",
        file_content,
        "--" .. boundary .. "--"
    }

    local body = table.concat(body_parts, "\r\n")

    return http.request({
        method = "post",
        url = "https://api.example.com/upload",
        headers = {
            ["Content-Type"] = "multipart/form-data; boundary=" .. boundary,
            ["Content-Length"] = tostring(#body)
        },
        body = body,
        format = "json"
    })
end

Webhook Handling

function send_webhook(webhook_url, event_type, payload)
    local webhook_data = {
        event = event_type,
        timestamp = os.time(),
        data = payload
    }

    -- Add signature for security
    local signature = generate_webhook_signature(webhook_data)

    local response = http.request({
        method = "post",
        url = webhook_url,
        headers = {
            ["Content-Type"] = "application/json",
            ["X-Webhook-Signature"] = signature,
            ["X-Event-Type"] = event_type
        },
        body = encoding.json(webhook_data),
        timeout = 10,  -- Short timeout for webhooks
        format = "json"
    })

    -- Log webhook attempt
    database.query(
        "main-db",
        "INSERT INTO webhook_log (url, event_type, success, response_code) VALUES (?, ?, ?, ?)",
        {webhook_url, event_type, response.success, response.data and response.data.status or 0}
    )

    return response.success
end

Error Handling and Retries

function http_request_with_retry(config, max_retries, backoff_base)
    max_retries = max_retries or 3
    backoff_base = backoff_base or 2

    local last_error = nil

    for attempt = 1, max_retries do
        local response = http.request(config)

        if response.success then
            return response
        end

        last_error = response.error

        -- Check if error is retryable
        if response.data and response.data.status then
            local status = response.data.status

            -- Don't retry client errors (4xx) except 429
            if status >= 400 and status < 500 and status ~= 429 then
                return response
            end

            -- Handle rate limiting
            if status == 429 then
                local retry_after = response.data.headers["Retry-After"]
                if retry_after then
                    print("Rate limited, waiting " .. retry_after .. " seconds")
                    -- Would need actual sleep here
                end
            end
        end

        -- Exponential backoff
        if attempt < max_retries then
            local wait_time = math.pow(backoff_base, attempt - 1)
            print("Retry " .. attempt .. " after " .. wait_time .. " seconds")
        end
    end

    return {
        success = false,
        error = "Max retries exceeded. Last error: " .. last_error
    }
end

Response Processing

-- Parse different response formats
function process_api_response(response)
    if not response.success then
        return nil, response.error
    end

    local status = response.data.status
    local body = response.data.body
    local headers = response.data.headers

    -- Check status codes
    if status >= 200 and status < 300 then
        -- Success
        return body, nil
    elseif status == 304 then
        -- Not modified
        return nil, "not_modified"
    elseif status >= 400 and status < 500 then
        -- Client error
        local error_msg = body.error or body.message or "Client error"
        return nil, error_msg
    elseif status >= 500 then
        -- Server error
        return nil, "Server error: " .. status
    else
        -- Unexpected status
        return nil, "Unexpected status: " .. status
    end
end

-- Extract pagination info from headers
function parse_link_header(link_header)
    local links = {}
    if not link_header then
        return links
    end

    -- Parse Link header: <url>; rel="next", <url>; rel="prev"
    for url, rel in string.gmatch(link_header, '<([^>]+)>; rel="([^"]+)"') do
        links[rel] = url
    end

    return links
end

Integration Examples

REST API Client

function create_api_client(base_url, api_key)
    local client = {
        base_url = base_url,
        api_key = api_key
    }

    function client:request(endpoint, method, data)
        local config = {
            method = method or "get",
            url = self.base_url .. endpoint,
            headers = {
                ["Authorization"] = "Bearer " .. self.api_key,
                ["Content-Type"] = "application/json",
                ["Accept"] = "application/json"
            },
            format = "json"
        }

        if data then
            if method == "get" then
                config.params = data
            else
                config.body = encoding.json(data)
            end
        end

        return http.request(config)
    end

    function client:get(endpoint, params)
        return self:request(endpoint, "get", params)
    end

    function client:post(endpoint, data)
        return self:request(endpoint, "post", data)
    end

    function client:put(endpoint, data)
        return self:request(endpoint, "put", data)
    end

    function client:delete(endpoint)
        return self:request(endpoint, "delete")
    end

    return client
end

-- Usage
local api = create_api_client("https://api.printshop.com", api_key)

local response = api:get("/jobs", {status = "pending"})
if response.success then
    for _, job in ipairs(response.data.body.jobs) do
        print("Job: " .. job.id)
    end
end

OAuth 2.0 Flow

function oauth_token_exchange(client_id, client_secret, auth_code)
    local response = http.request({
        method = "post",
        url = "https://oauth.example.com/token",
        headers = {
            ["Content-Type"] = "application/x-www-form-urlencoded"
        },
        body = string.format(
            "grant_type=authorization_code&code=%s&client_id=%s&client_secret=%s",
            auth_code, client_id, client_secret
        ),
        format = "json"
    })

    if response.success then
        local tokens = response.data.body
        -- Store tokens securely
        cache.put_ttl("oauth:access_token", tokens.access_token, tokens.expires_in)
        cache.put_ttl("oauth:refresh_token", tokens.refresh_token, 86400 * 30)
        return tokens
    end

    return nil
end

Best Practices

1. Always validate URLs: Check URL format and protocol

   function validate_url(url)
       -- Must be HTTPS in production
       if not string.match(url, "^https://") then
           return false, "Only HTTPS URLs are allowed"
       end
   
       -- Check against whitelist if needed
       local allowed_domains = {"api.example.com", "webhook.partner.com"}
       local domain = string.match(url, "^https://([^/]+)")
   
       for _, allowed in ipairs(allowed_domains) do
           if domain == allowed then
               return true
           end
       end
   
       return false, "Domain not whitelisted"
   end

2. Handle timeouts gracefully: Set appropriate timeouts

   local response = http.request({
       url = external_api_url,
       timeout = 5,  -- Short timeout for non-critical requests
       format = "json"
   })
   
   if not response.success and string.match(response.error, "timeout") then
       -- Use cached data or default
       return get_cached_or_default()
   end

3. Respect rate limits: Implement rate limiting

   function rate_limited_request(url, requests_per_minute)
       local rate_key = "http_rate:" .. url
       local count = cache.get(rate_key, 0)
   
       if count >= requests_per_minute then
           return {success = false, error = "Rate limit exceeded"}
       end
   
       local response = http.request({url = url})
       cache.put_ttl(rate_key, count + 1, 60)
   
       return response
   end

4. Log external requests: Track API usage

   function logged_request(config)
       local start_time = os.time()
       local response = http.request(config)
       local duration = os.time() - start_time
   
       database.query("logs-db",
           "INSERT INTO api_log (url, method, status, duration, success) VALUES (?, ?, ?, ?, ?)",
           {config.url, config.method or "get",
            response.data and response.data.status or 0,
            duration, response.success}
       )
   
       return response
   end

Security Features

• SSRF Protection: Blocks requests to private IPs and internal networks
• DNS Resolution Caching: Prevents DNS rebinding attacks
• URL Validation: Enforces HTTPS in production environments
• Blocked Networks:
  • Loopback (127.x.x.x)
  • Private Class A (10.x.x.x)
  • Private Class B (172.16-31.x.x)
  • Private Class C (192.168.x.x)
  • Link-local (169.254.x.x)
  • Multicast and reserved ranges

Performance

• Connection pooling for improved performance
• Automatic gzip decompression
• Configurable timeouts and retry logic
• Response size limits to prevent memory issues

Supported HTTP Methods

• GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS