Management API
Francis can serve an optional management REST API. You can use it to look at the cluster (hosts, placements, live activations, actor state, jobs, alarms, and workflow instances) and to run administrative actions: deactivating an actor, draining a host, and cancelling, suspending, or resuming a workflow instance.
The API is disabled by default in both topologies. Until you enable it, no TCP port is opened.
Enabling the API#
Remote topology#
In the remote topology, the runtime serves the API. Enable it in the runtime’s configuration file:
management:
enabled: true
# TCP address the API listens on, default "127.0.0.1:7401"
bind: "0.0.0.0:7401"
# Tokens that can read everything but perform no actions
readOnlyTokens:
- "<a random string of at least 32 characters>"
# Tokens that can also perform actions
managementTokens:
- "<another random string of at least 32 characters>"
# Optional: serve HTTPS directly (set both or neither)
tls:
certFile: "/etc/francis/management.crt"
keyFile: "/etc/francis/management.key"
# Optional: browser origins allowed to call the API, such as a standalone dashboard
allowedOrigins:
- "http://localhost:7402"Every replica serves the whole cluster. When a request concerns a host that’s connected to a different replica, the replica forwards it to that one over the runtime’s existing UDP port. This requires PostgreSQL (the only provider that supports multiple replicas), an address on each replica that the others can dial, and UDP connectivity between replicas: see Running multiple runtime replicas.
With the Helm chart, set management.enabled and the token lists in your values. The chart binds the API on all interfaces, adds a TCP container port and a separate <release>-francis-management Service, and can serve HTTPS from a kubernetes.io/tls Secret with management.tls.existingSecret. If you supply the configuration with existingConfigSecret, you must add the management block to that Secret yourself. The chart’s README has the details.
Local topology#
In the local topology, any host can serve the API with the local.WithManagementAPI option:
h, err := local.NewHost(
local.WithManagementAPI(local.ManagementOptions{
Bind: "0.0.0.0:7401",
ReadOnlyTokens: []string{readOnlyToken},
ManagementTokens: []string{managementToken},
// Optional: serve HTTPS directly
TLSConfig: tlsConfig,
// Optional: browser origins allowed to call the API, such as a standalone dashboard
AllowedOrigins: []string{"http://localhost:7402"},
}),
// ...
)Without this option the host starts no listener. The options follow the same rules as the runtime’s configuration, except that Bind defaults to 127.0.0.1:7401, which only accepts connections from the same machine. Your application is responsible for sourcing the tokens.
While you can enable the API on every host, you only need it on one host: that host can serve the whole cluster. It reads durable data from the provider and reaches the other hosts through their existing peer ports, which local hosts already need to be able to reach each other on. Every host answers these internal requests whether or not it enables its own listener. The management port is an additional TCP port, only on the hosts that enable it.
Dashboard#
The runtime also serves a web dashboard for the API, at the root of the management listener, and the francis dashboard command serves it on its own. See Dashboard.
Security#
The API exposes actor state and workflow inputs and outputs, so treat access to it like access to the database.
- Tokens. Callers authenticate with a bearer token in the
Authorizationheader. When the API is enabled, at least one token is required. Every token must be at least 32 characters long. - Rotation. Each list accepts multiple tokens. To rotate one, add the new token, update your clients, then remove the old one.
- Audit logs. Actions, and reads of actor state and workflow input and output, are logged with the request ID, the target, and the last 5 characters of the token that was used. The full token is never logged, and neither are state or workflow payloads (different tokens can end with the same 5 characters, so the suffix helps you tell tokens apart but isn’t a unique key).
- TLS. TLS is optional. Both the runtime and a local host listen on
127.0.0.1:7401by default, so the API only accepts connections from the same machine unless you set the bind address. If the API listens on an address other machines can reach without TLS configured, the server logs a warning at startup: in that case, terminate TLS in a proxy in front of it, or configure a certificate. - Separate credentials. Management tokens are independent of the runtime PSKs and the host bootstrap credentials. Traffic between runtime replicas, and between local hosts, is authenticated with their certificates and never with management tokens.
Scopes#
Every route requires a scope (except GET /api/v1/token). Tokens in readOnlyTokens receive every scope except those ending in :manage, while tokens in managementTokens receive every scope.
| Scope | Grants | Management token required |
|---|---|---|
cluster:read | Cluster summary, runtimes, and hosts | - |
actors:read | Live activations, placements, actor types, and the list of actors with stored state | - |
actors:state:read | The durable state of an actor, including built-in actor types (a workflow’s actor types also need workflows:data:read) | - |
workflows:read | Workflows, instance metadata, and instance event history | - |
workflows:data:read | Workflow input and output in instance details, and the stored state of a workflow’s actor types | - |
jobs:read | Jobs and alarms | - |
actors:manage | Deactivating an actor | Yes |
hosts:manage | Draining a host | Yes |
workflows:manage | Cancelling, suspending, and resuming a workflow instance | Yes |
Endpoints#
Every API route is under /api/v1.
In addition, there are two public routes which don’t require a token:
GET /healthzfor healthchecksGET /api/v1/openapi.yamlreturning the OpenAPI description of the API
GET /api/v1/token accepts any valid token, without requiring a scope, and returns the scopes the token grants. Clients can use it to tell a read-only token from a management token before offering actions.
Paths outside /api/ and /healthz belong to the dashboard.
Browsers can call the API from other origins only when management.allowedOrigins lists them, as the standalone and hosted dashboards do. Third-party origins not in the allowlist fail with a CORS error.
Path segments must be percent-encoded. This matters for workflow child instance IDs, which contain
|, and for actor IDs, which may contain any character other than/.
Cluster and hosts#
| Route | Scope | Description |
|---|---|---|
GET /api/v1/cluster/summary | cluster:read | Counts of runtimes, hosts by state, placements, live activations, jobs, and workflow instances by status, plus the holder of the cluster’s exclusive-access lease when one is held. Counts stop at 10,000 and are then flagged as truncated. |
GET /api/v1/runtimes | cluster:read | Runtime replicas with a live membership (those with the management API enabled), their advertised addresses, and last heartbeat. Remote topology only. |
GET /api/v1/hosts | cluster:read | Paginated hosts, with their state (connected, draining, or, in the remote topology, unreachable), last health check, owner runtime, and actor counts. Filter with state. |
GET /api/v1/hosts/{hostId} | cluster:read | One host’s registration, actor types with their placement limits and usage, and the capacity, drain state, and workflow definitions the host reports. |
GET /api/v1/hosts/{hostId}/activations | actors:read | Paginated actors that are active in memory on one host right now, read from the host itself. Filter with type, which can be repeated. |
POST /api/v1/hosts/{hostId}/drain | hosts:manage | Drain a host so it deactivates its actors and shuts down. See Draining a host. |
Only hosts with a live registration are listed. A host whose health check is older than the deadline is removed by the provider, so there’s no “expired” state.
Actors#
| Route | Scope | Description |
|---|---|---|
GET /api/v1/activations | actors:read | Paginated, cluster-wide list of active actors, collected from every reachable host. Filter with host and type, which can be repeated. |
GET /api/v1/placements | actors:read | Paginated placements recorded in the provider (actor type, ID, host, and idle timeout). Filter with host and type, which can be repeated. |
GET /api/v1/actor-types | actors:read | Registered actor types, the hosts serving each, placement usage and limits, job retention, and execution capacity groups. |
GET /api/v1/actor-states?type={type} | actors:read | Paginated IDs of the actors of a type that have stored state, whether or not they’re active. type is required. |
GET /api/v1/actor-states/{type}/{id} | actors:state:read | An actor’s stored state. A workflow’s actor types also need workflows:data:read. See Actor state. |
POST /api/v1/actors/{type}/{id}/deactivate | actors:manage | Ask the host an actor is active on to deactivate it. See Deactivating an actor. |
Jobs and alarms#
| Route | Scope | Description |
|---|---|---|
GET /api/v1/jobs | jobs:read | Paginated jobs: pending, active, and, while their actor type retains them, completed and dead-lettered. Filter with type, id, and status (pending, active, completed, or dead), which can all be repeated. |
GET /api/v1/jobs/{jobId} | jobs:read | One job. Terminal jobs include their attempts, last error, and end time. |
GET /api/v1/alarms | jobs:read | Paginated alarms, excluding jobs, with due time, repeat interval, and lease state. Filter with type and id, which can be repeated. |
Workflows#
| Route | Scope | Description |
|---|---|---|
GET /api/v1/workflows | workflows:read | Known workflows, their registered versions and definition fingerprints, the hosts serving them, and any definition conflicts. |
GET /api/v1/workflows/{name}/instances | workflows:read | Paginated instances of a workflow, with status, version, parent, and creation time. Filter with status, which can be repeated, version, parent, createdFrom, and createdTo (RFC 3339). |
GET /api/v1/workflows/{name}/instances/{instanceId} | workflows:read | An instance’s status, steps with their timings, child instances, compensation outcome, and dead-lettered jobs. With workflows:data:read it also includes the input and output. |
GET /api/v1/workflows/{name}/instances/{instanceId}/events | workflows:read | Paginated event history of an instance, in order. |
POST /api/v1/workflows/{name}/instances/{instanceId}/cancel | workflows:manage | Cancel an instance. A reason is required. |
POST /api/v1/workflows/{name}/instances/{instanceId}/suspend | workflows:manage | Suspend an instance, with an optional reason. |
POST /api/v1/workflows/{name}/instances/{instanceId}/resume | workflows:manage | Resume a suspended instance. Takes no reason. |
Instance details include hasOutput, which tells apart an instance with no output from one whose output is JSON null. Callers without workflows:data:read get dataRedacted: true instead of the input and output. Reading input and output is recorded in the audit log.
A pending instance (one that hasn’t started yet) is reported as long as its start job is live. The instance list reads only labels, so a pending instance whose start job was dead-lettered can stay in the list as pending for a while after its detail route has started returning 404.
Event history is recorded for every workflow by default. A workflow can opt out with the workflow.WithoutEventHistory() option, because the history adds a write per transition and wide fan-outs produce many events. For a workflow that opted out, the events route returns 404 with the code eventHistoryDisabled.
The API reads stored data only, so completed jobs and finished workflow instances are only visible while they’re retained.
Examples#
List the hosts:
curl -H "Authorization: Bearer $FRANCIS_TOKEN" \
http://127.0.0.1:7401/api/v1/hostsRead an actor’s state as the exact stored MessagePack bytes:
curl -H "Authorization: Bearer $FRANCIS_TOKEN" \
-o state.msgpack \
http://127.0.0.1:7401/api/v1/actor-states/counter/my-counterList the failed instances of a workflow:
curl -H "Authorization: Bearer $FRANCIS_TOKEN" \
"http://127.0.0.1:7401/api/v1/workflows/order/instances?status=failed&limit=50"Cancel a workflow instance:
curl -X POST \
-H "Authorization: Bearer $FRANCIS_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"reason": "Duplicate order"}' \
http://127.0.0.1:7401/api/v1/workflows/order/instances/order-1234/cancelDrain a host, allowing up to two minutes for its actors to deactivate:
curl -X POST \
-H "Authorization: Bearer $FRANCIS_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"timeout": "2m", "reason": "Node maintenance"}' \
http://127.0.0.1:7401/api/v1/hosts/<host-id>/drainActor state#
Actor state is stored as MessagePack. GET /api/v1/actor-states/{type}/{id} reads it from the provider without activating the actor, and returns the last committed state: work in progress inside a running actor call isn’t visible. A missing or expired state returns 404.
The response body contains the exact stored bytes, with Content-Type: application/msgpack. Error responses use JSON.
Built-in actor types (such as workflow orchestrators and workers) store their internals in the same way, and actors:state:read can read them. That’s useful for debugging, but their format is internal and can change between releases. Use the workflow endpoints to inspect workflows.
The state of a workflow’s actor types holds the instances’ input and output, so reading it also requires workflows:data:read, the same scope that the instance details need to include them.
Reads of actor state are recorded in the audit log.
Actions#
Actions are synchronous. Each request returns once its target has accepted it, or fails with the reason. Nothing is stored about an action (apart from the audit log). Anything that continues after the response, such as a host shutting down or a workflow compensating, shows up on the host or the workflow instance.
Repeating an action is safe. Deactivating an actor that isn’t active, or draining a host that’s already draining, succeeds without doing anything. A request that failed because its target was unreachable can be safely retried.
While a clusteradmin exclusive-access lease is held on the cluster, for example during a restore, drains and workflow controls return 409 with the code exclusiveLeaseHeld, and GET /api/v1/cluster/summary reports who holds the lease and until when.
Action request bodies are JSON objects. An empty body is the same as {}, and unknown fields are rejected. A reason can be at most 1024 bytes, and is recorded in the audit log.
Deactivating an actor#
POST /api/v1/actors/{type}/{id}/deactivate asks the host the actor is active on to deactivate it, and returns once the host has done so. The actor’s durable state is kept, and it’s activated again by the next call. The response has notActive: true when the actor wasn’t active.
Deactivating is just hybernating the actor’s state. It is not pausing the actor: a pending alarm or job for the actor can activate it again immediately.
Draining a host#
POST /api/v1/hosts/{hostId}/drain marks the host as draining, so no new actors are placed on it, then asks it to finish its work, deactivate its actors, and unregister. After that the host’s Run method returns an error that matches host.ErrAdministrativeDrain, so your application can tell a requested drain apart and exit. A drained host can’t be run again, and an external orchestrator (like Kubernetes or Docker) is responsible for starting a replacement process, which registers as a new host.
The request body accepts:
timeout: a Go duration string, up to5m, controlling how long the host lets its actors’ in-flight calls run before it cancels them. When omitted, the host waits for every call to finish, with no time limit. The host keeps sending health checks until its actors have halted, so its registration doesn’t expire while they finish.force: drain the host even if it’s the last live host serving one or more actor types. Without it, the request fails with409and the codelastServer, listing the affected types indetails.actorTypes. The check is atomic with other drain requests, so concurrent drains of the last hosts serving a type can’t both succeed withoutforce.reason: an optional reason, logged by the host and in the audit log.
The request returns 202 once the host has accepted the drain, without waiting for it to shut down. The host is listed as draining until it unregisters.
There’s no way to undo a drain once the host has accepted it. If the host reconnects before it receives the request, including to a different runtime replica, the request is sent again to its new connection. Only if the host keeps reconnecting does the request fail with the code hostReattached: send it again. A host that reconnects on its own after an unrelated disconnection, without a drain request, is back in service.
The host is marked as draining before it’s asked to drain, so a request that fails part of the way could leave a host that never accepted the drain out of placement. When the request fails, the API asks the host whether it’s draining:
- If the host is draining, it accepted the drain and only its acknowledgement was lost, so the request succeeds.
- If the host isn’t draining, its mark is removed only while this request still owns it. A competing drain request or the host accepting a drain invalidates that ownership atomically. When the mark is removed, the host goes back into service and the error has
details.hostDraining: false, otherwise it stays marked and the error hasdetails.hostDraining: true. - If the host can’t be asked, it may have accepted the drain, so its mark stays, and the request fails with
details.hostDraining: true. Send the drain again once the host is reachable. The mark also clears when the host registers again, which in the remote topology includes reconnecting to a runtime.
Cancelling, suspending, and resuming a workflow instance#
These requests send the instance the same durable control job that WorkflowService.Cancel, Suspend, and Resume send, and return 202 once the job is stored. The instance applies it on its next turn, and the result shows up in its status and event history.
coalesced: truemeans a job of the same kind was already pending, so this request’s reason was dropped.noEffect: true(with status200) means the instance is already finished, or is compensating and the request is a cancel, so no job was sent.
An instance that’s waiting for a host serving its exact workflow version and definition applies the job only once such a host is available.
Pagination#
List routes accept limit (default 100, maximum 1_000) and cursor. Responses have an items array and, when there are more results, a nextCursor: pass it as cursor to get the next page. Cursors are opaque.
Some filters are applied to each page after it’s read, so a page can have fewer items than limit while more pages follow. Keep going until nextCursor is absent.
Partial results#
Routes that collect data from hosts (/activations, /actor-types, /workflows, and the activation counts in /cluster/summary) query every reachable host with a limited number of concurrent requests. When some hosts can’t be queried, the response has partial: true and an errors list with the host ID, code, and message of each failure. /activations returns 503 with the code noHostsReachable when none of the hosts could be queried.
Each host is read independently, and the data can change while the request runs, so results are not necessarily a consistent snapshot of the cluster. An actor that moves between hosts during a request can show up twice or not at all, and the same is true across pages. Responses include the time they were observed (observedAt, or startedAt and finishedAt). For the freshest view of a single host, use /hosts/{hostId}/activations.
Errors#
Errors return a JSON body:
{
"code": "lastServer",
"message": "the host is the last live server of one or more actor types, set force to drain it anyway",
"requestId": "0192f0c4-6f5e-7b7e-9d1a-3c2b1a0f9e8d",
"details": {
"actorTypes": ["counter"]
}
}codeis a stable, machine-readable error code, such asbadRequest,unauthorized,forbidden,notFound,notApplicable(a route that doesn’t exist in this topology, like/runtimesin the local topology),exclusiveLeaseHeld,lastServer,hostUnavailable,hostReattached,noHostsReachable,eventHistoryDisabled,payloadTooLarge,timeout, orinternal.requestIdis also returned in theX-Request-Idheader of every response, and matches the audit log.retryableis present, andtrue, when sending the same request again may succeed.detailscarries extra information for some errors.