laya on Tenstorrent p150 and p150x4

Serves convaiinnovations/laya by Convai Innovations at revision 7b928d82 (the English root checkpoint), built on answerdotai/ModernBERT-large. See the original model card for training and evaluation details.

Laya is a 421M-parameter non-LLM calibrated decision model: a ModernBERT-large encoder (28 layers, 1024 hidden, GeGLU, dual-theta RoPE, one global attention layer per three) with a 2-layer decision head that has an option-marker scorer and an act head, trained with reinforcement learning against proper scoring rules. One forward pass answers typed noul, choice and score questions about a text or JSON state with calibrated probabilities; it never generates text. This package serves the Jev-compatible /v1/systemone API of the authors' laya-serve, with a demo page at /demo/.

Runs on Tenstorrent p150 or p150x4 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Calibrated typed decisions over English text states: routing, triage, guardrails, relevance and scoring.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150 or p150x4

Quickstart

tt serve tt-hous/laya

tt serve downloads the Docker image and the convaiinnovations/laya at 7b928d828b7b0e022f929d9bd2e44165aa270148 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/laya.

The demo page is at /demo/ on the port your serve command reported. Existing laya and Jev clients work by changing their base URL.

Serving profiles

Select a different serving profile using --profile.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /v1/systemone takes {"state": ..., "questions": {id: {"type": "choice"|"score"|"noul", "instructions": ..., "criteria": ...}}, "max_len", "head_max_len", "min_confidence"} and returns {"model", "answers": {id: {...}}, "usage": {"input_tokens", "output_tokens", "state_tokens", "state_tokens_dropped", "truncated", "truncated_questions"}}. Use max_len up to 512 and head_max_len up to 192 (this checkpoint's limits). POST /v1/systemone/batch takes {"states": [...], "questions": {...}} (1 to 64 states sharing one question set) and returns {"results": [...], "total_usage": {...}}. GET /v1/health reports backend, precision, mesh, buckets and the startup sanity check. Every /v1 response carries X-Inference-Time-Ms, X-Laya-Device-Ms and X-Laya-Batch. shim/laya_tt_backend.py under code/ installs a TtBackend on a pip laya Agent that routes agent.model.forward to this server's /v1/forward.

Expected performance

Evaluations

Zero-shot, from this package on one p150; the reference figures are the authors' on the convaiinnovations/laya model card.

Parity against the CPU fp32 reference of the same checkpoint:

Latency

Client-side medians per call, warm, with the authors' bench_latency protocol; device forward time in parentheses. The reference column is the authors' Tesla T4 figure on the model card for the same number of questions.

Batched /v1/systemone/batch: 203 to 250 questions per second on p150 (the authors' T4: 103 to 332) and 349 to 940 on p150x4. A single question is slower on p150x4 than on one chip (13.2 against 10.9 ms).

Measured from this package served on one Blackhole p150 chip (p150) and on the four-chip p150x4 mesh, client on the same host.

Limitations

• English root checkpoint only (max_len 512, head_max_len 192); non-English input is not supported (the authors' multilingual checkpoint is not packaged). Tested on one p150 chip and on the four-chip p150x4 mesh; no two-chip profile is offered.
• Not for text generation. Not for zero-shot typed decisions: this checkpoint scores 0.359 on typed-decisions, near chance; the authors' 0.766 belongs to the fine-tuned checkpoint (see Related packages). Not for choice questions with more than about 20 options at the default head budget.
• Raw calibration as shipped: ECE 0.172 on typed-decisions. The authors' ECE of 0.081 is after domain temperature fitting (their raw value is 0.466). The shipped temperatures are over-confident per the authors; refit on your own data before trusting the probabilities.
• action.act_probability carries no usable signal (per the authors); gate on answer_confidence.
• Temperatures are clamped to [0.5, 5.0] as pip laya 0.3.27 does, so choice questions with more than 10 options use 0.5 instead of the shipped 0.1006.
• Requests are padded to 128, 256 or 512 tokens and 1 to 64 rows (exact buckets at 5, 10 and 50 rows); longer sequences are refused with 422. Long option texts are truncated to 48 tokens silently, as in the reference.
• Agreement with the CPU fp32 reference depends on the input. On the 488-decision parity corpus (5-row calls) 12 decisions flip, all with a reference margin under 0.10, and none of the 403 confident decisions. On 400 single-row DAIR Emotion decisions 6 flip, 2 of them confident (reference margins 0.125 and 0.369), with probability moves up to 0.27 (7 of 400 above 0.12); on 400 single-row AG News decisions 2 flip, none confident, with moves up to 0.056. Short single-row inputs with several near-tied options are the most sensitive. LAYA_PRECISION=bf16_hifi4 keeps every decision within 0.11 of the reference with no flip on the seven most-affected Emotion texts and on a 200-decision subset, at about 11 percent more latency; it was not measured over all 800 single-row decisions.
• The same question answered alone or inside a batch: max probability spread 0.009 on the parity corpus and 0.019 on 16 short single-row questions at the 512-token bucket, with the same answer in every placement.
• One device process serialises requests. p150x4 is data parallel over rows and bit-identical to one chip at equal per-chip batch; it reports healthy in 20 s.
• Not reproduced from the authors' card: MASSIVE, XNLI, the 45 of 51 languages, and post-temperature ECE.

Licensing

Laya weights and the vendored rl_common.py, rl_agent_api.py and email_utils.py: Apache-2.0 (Convai Innovations); ModernBERT-large: Apache-2.0 (Answer.AI); TTNN port and serving code: Apache-2.0 (Tenstorrent AI ULC); demo feed cases: LocalLLaMA/typed-decisions, Apache-2.0.

Related packages

• tt-hous/laya-typed-decisions: the authors' fine-tune for the four typed-decisions workflows (0.764 accuracy on that benchmark), same server and API.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/laya/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: