Event logs

HappyView keeps an event log of system activity: lexicon changes, record operations, script runs and errors, operator actions, API key events, backfills and Jetstream connectivity. Events are stored in the database and read from System > Event Logs or the admin API.

Event types

Events follow a category.action naming convention. Each event has a severity level (info, warn, or error), an optional actor_did (who triggered it), an optional subject (what was affected), and a detail JSON object with event-specific data.

Lexicon events

Event Type	Severity	Subject	Detail
lexicon.created	info	Lexicon NSID	revision, source
lexicon.updated	info	Lexicon NSID	revision, source
lexicon.deleted	info	Lexicon NSID	job_id and collection when its records are deleted too
lexicon.published	info	Lexicon NSID	did, uri, cid
lexicon.publish_failed	warn	Lexicon NSID	error
lexicon.unpublished	info	Lexicon NSID	did, uri
lexicon_publishing.rule_created, .rule_updated, .rule_deleted	info	The rule’s pattern	rule_id, target_kind, linked_repo_id, auto_publish

The first three are logged when lexicons are uploaded, updated, or deleted via the admin API. The actor_did is the operator who performed the action. The rest come from publishing lexicons. A failure repeating the last error isn’t logged again.

Project events

Event Type	Severity	Subject	Detail
project.deployed	info	The bundle’s sha256	changes, backfills

Logged when a project is deployed through PUT /admin/project. A dry run logs nothing. changes and backfills are the ones the deploy answered with.

Record events

Event Type	Severity	Subject	Detail
record.created	info	Record AT URI	collection, did, rkey
record.deleted	info	Record AT URI	collection, did, rkey
record.skipped	info	Record AT URI	collection, did, rkey, reason
record.skip_ignored	warn	Record AT URI	collection, did, rkey, ignored, reason
record.cid_mismatch	warn	Record AT URI	collection, did, rkey, claimed_cid, reason
record.index_failed	warn	Record AT URI	error
record.delete_hook_failed	error	Record AT URI	origin, job_id, cid, error

Logged when records are received from Jetstream and stored or removed from the local database, or skipped because a record script returned nil. These are system-triggered events (actor_did is null). They write one row per record, so successful operations are only logged when verbose_event_logging is on. If a database error occurs, the same event type is logged with error severity regardless of that setting, and the error message is included in the detail.

Script events

Event Type	Severity	Subject	Detail
script.executed	info	Method NSID	method, caller_did, duration_ms, response_size, input, response
script.error	error	Method NSID	error, script_source, input, caller_did, method, duration_ms
script.executed	info	Record or label AT URI	host_kind, host_id, trigger, attempts
script.dead_lettered	error	Record or label AT URI	host_kind, host_id, trigger, error
script.log	as logged	The script’s trigger	trigger, level, message, fields
script.created, script.updated	info	Script id	script_type
script.deleted	info	Script id	none
script.codemod_applied	info	Script id	script_type, markers

For XRPC query and procedure scripts, script.executed is written on every call and stores the full request and response, so it is only logged when verbose_event_logging is on. script.error is always logged, with what you need to reproduce the failure: the error message, the complete script source, the input that triggered it, and the caller’s DID.

For record and label scripts, script.executed fires once per record or label that reaches a script, so it too is only logged when verbose_event_logging is on. script.dead_lettered means a script failed all retry attempts; see Dead Letters below.

script.log is a line from require("internal.logging"), with the severity the script named.

Note

For query scripts, input and caller_did are left out of the detail, and script.executed carries params instead.

User events

Event Type	Severity	Subject	Detail
user.created	info	New operator’s DID	template, permissions
user.deleted	info	Removed operator’s id	none
user.bootstrapped	info	Bootstrapped operator’s DID	source when SUPERADMIN_DID seeded them
user.permissions_updated	info	Operator’s id	granted, revoked
user.super_transferred	warn	New superadmin’s id	from_user_id, to_user_id
user.super_set_by_platform	info	New superadmin’s DID	user_id

These describe operators, the users of the admin API. The user.bootstrapped event is logged when an operator is given superadmin automatically (see Auth: Auto-bootstrap).

Auth events

Event Type	Severity	Subject	Detail
auth.permission_denied	warn	The permission name	required_permission, user_id

Logged when an operator attempts to access an endpoint they don’t have permission for.

Space events

Event Type	Severity	Subject	Detail
space.moderator_read	info	Space AT URI	action, space_id, user_id, and what was read (see below)

Logged when an operator reads a space’s records or blobs through the admin spaces API. action is list_records or get_blob. A list_records event includes the repo and collection filters and the uris returned. A get_blob event includes the blob cid and the repo that references it. Listing spaces and viewing their metadata or members is not logged.

API Key events

Event Type	Severity	Subject	Detail
api_key.created	info	Key name	key_prefix, permissions
api_key.revoked	info	Key id	none

Script variable events

Event Type	Severity	Subject	Detail
script_variable.upserted	info	Variable key	none
script_variable.deleted	info	Variable key	none

Backfill events

Event Type	Severity	Subject	Detail
backfill.started	info	Collection NSID	job_id, scope, account_count, source
backfill.completed	info	Collection NSID	job_id, total_repos, total_records, or source for a replay
backfill.cancelling, .cancelled, .pausing, .resumed	info	none	job_id
backfill.retry_started	info	Collection NSID	job_id, source_job_id, and retried_repos or blocks
backfill.discovery_failed	warn	Collection NSID	job_id, collection, error
backfill.units_left	warn	Collection NSID	job_id, queued_units, reason
backfill.write_failed	error	The job’s id	job_id, write, error

A backfill that fails writes no event of its own: its error is on the backfill, in the admin API and the dashboard. See Backfill for background.

Job events

Event Type	Severity	Subject	Detail
job.started	info	Job type	job_id, job_type
job.paused	info	Job type	job_id
job.cancelled	info	Job type	job_id
job.completed	info	Job type	job_id, result
job.failed	error	Job type	job_id, error

Jetstream events

Event Type	Severity	Subject	Detail
jetstream.connected	info	none	url, protocol, compression, dictionary_id
jetstream.disconnected	warn	none	reason
jetstream.gap	warn	none	reason, and more by reason (below)
jetstream.invalid_request	error	none	message, retry_in_secs
jetstream.account_deleted	info	Account DID	purged, seq
jetstream.account_takendown	info	Account DID	purged, seq, status (takendown or suspended)
jetstream.account_inactive	info	Account DID	status (deactivated, desynchronized or throttled)
jetstream.account_status_mismatch	warn	Account DID	status, pds, seq
jetstream.sync_purged	info	Account DID	purged, rev
jetstream.record_invalid	warn	Account DID	uri, seq, error
jetstream.apply_failed	error	Account DID	uri (or kind and, for an account event, status), seq, error

jetstream.connected and jetstream.disconnected are logged when the WebSocket connection to Jetstream is established or lost.

jetstream.gap means events may have been missed. The same gap (same reason and cursor) is logged once per run of the Jetstream consumer. Its reason is one of:

• cursor_too_old: the saved cursor was past Jetstream’s retention and no saved time could replace it, so HappyView resumed from the live tip. after_seq is the saved seq (the last one known applied), resumed_at_seq the first seq received after it, and message Jetstream’s explanation. When HappyView replays the gap from the Jetstream archive instead, resumed_at_seq is the archive’s sealed tip and replay_job_id names the backfill job that fills the gap.
• outdated_cursor: Jetstream moved a time cursor older than its retention up to the oldest event it has. message names where it resumed.
• future_cursor: the saved seq was past Jetstream’s newest, because its seq numbering restarted. HappyView reconnects by the saved witnessed time, which Jetstream translates to the new numbering, or, with none saved, follows the new numbering from the live tip. from_seq is the cursor it asked for.

jetstream.invalid_request means Jetstream refused the subscription as invalid, or kept refusing it without saying why after HappyView dropped what it could from the request. HappyView retries after retry_in_secs (an hour by default).

jetstream.account_deleted is logged when an account this instance holds records for is deleted: its records (other than local writes not yet echoed back) are removed. jetstream.account_takendown is logged the same way for an account that is taken down or suspended. A tombstone is written for every deleted, taken-down or suspended account, but the event is logged only when this instance held, and purged, records for it. The tombstone means a late or replayed copy of the account’s records is refused. jetstream.account_status_mismatch is logged instead when the relay reports an account deleted (status) but the account’s current PDS (pds) says it is active: HappyView keeps its records and writes no tombstone. This happens when the relay is still relaying a PDS the account moved away from. Takedowns and suspensions are never checked this way, because a relay may impose them without the account’s PDS knowing. jetstream.account_inactive is logged when such an account is deactivated, desynchronized or throttled; its records are kept. When an account becomes active again, its tombstone is lifted, but removed records come back only through a new backfill. jetstream.sync_purged is logged when an account’s repo is resynced: its records in registered collections are removed, and the current ones that follow in the stream replace them. Events for accounts this instance holds nothing for are not logged.

jetstream.record_invalid is logged once for an event HappyView skips immediately because retrying cannot help: a create or update with no record body, an unknown operation, data the database rejects, or a network lexicon schema that cannot be applied.

jetstream.apply_failed is logged when an event keeps failing for a reason that is not a database outage. HappyView retries it for about 31 seconds, logs this event and skips it, so one bad event does not stall the stream. Database outages are never skipped: see Real-time indexing.

Other events

Event Type	Severity	Subject	Detail
api_client.created	info	Client name	client_key, client_id_url
api_client.updated, api_client.deleted	info	Client id	none
api_client.secret_created	info	Client name	client_id, secret_id, label
api_client.secret_revoked	info	Client name	client_id, secret_id, active_secrets
api_client.auth_key_rotated	info	Client id	kid, status, activates_at, orphaned_sessions
api_client.auth_key_revoked	warn	Client id	kid, sessions_destroyed
api_client.auth_keys_revoked_all	warn	Client id	kids, sessions_destroyed, revoked
oauth.instance_key_rotated	info	none	kid, status, activates_at, orphaned_sessions
oauth.instance_key_revoked	warn	none	kid, sessions_destroyed
dpop_key.provisioned	info	Provision id	client_key, thumbprint, confidential
dpop_session.created	info, or warn when not refreshable	Client key	scopes, refreshable
dpop_session.deleted	info	Client key	none
dpop_session.device_deleted	info	Client key	session_id
delegation.account_linked	info	Account DID	did
delegation.account_unlinked	info	Account DID	none
delegation.delegate_added	info	Delegate’s DID	account_did, role
delegation.delegate_removed	info	Delegate’s DID	account_did
domain.created, domain.deleted, domain.primary_changed	info	Domain URL	id
external_auth.refreshed	info	Plugin id	expires_at
external_auth.refresh_failed	warn	Plugin id	reason
plugin.added, plugin.reloaded, plugin.secrets_updated	info	Plugin id	none
plugin.allowed_hosts_updated	info	Plugin id	allowed_hosts
plugin.removed	info	Plugin id	removed
plugin.log	as logged	Plugin id	level, message
labeler.added, labeler.deleted	info	Labeler DID	none
labeler.updated	info	Labeler DID	status
labeler.connected	info	Labeler DID	did
labeler.disconnected	warn	Labeler DID	did, last_seq
service_auth.access_denied	error	Method NSID	fragment, reason
service_identity.updated	info	none	mode
service_entry.deleted	info	Entry id	none
service_entry.plc_synced	info	none	mode
verification_method.created, verification_method.deleted	info	Fragment id	fragment_id
setting.updated	info	Setting key	value, or content_type and size_bytes for the logo
setting.deleted	info	Setting key	none
setup.completed	info	none	none
event_logs.purged	warn	The purge job’s id	job_id

Querying events

Use the admin API to query event logs with filters:

TypeScript

const TOKEN = "hv_..."; // your API key
const headers = { Authorization: `Bearer ${TOKEN}` };

interface Event {
  id: string;
  event_type: string;
  severity: string;
  actor_did?: string;
  subject?: string;
  detail: Record<string, unknown>;
  created_at: string;
}

interface EventsResponse {
  events: Event[];
  cursor?: string;
}

// Get all errors
const errors: EventsResponse = await fetch(
  "http://127.0.0.1:3000/admin/events?severity=error",
  { headers },
).then((r) => r.json());

// Get script errors for a specific lexicon
const scriptErrors: EventsResponse = await fetch(
  "http://127.0.0.1:3000/admin/events?event_type=script.error&subject=com.example.feed.like",
  { headers },
).then((r) => r.json());

// Get all lexicon-related events
const lexiconEvents: EventsResponse = await fetch(
  "http://127.0.0.1:3000/admin/events?category=lexicon",
  { headers },
).then((r) => r.json());

// Paginate through results
const page: EventsResponse = await fetch(
  "http://127.0.0.1:3000/admin/events?limit=20&cursor=2026-03-01T11:59:00Z",
  { headers },
).then((r) => r.json());

JavaScript

const TOKEN = "hv_..."; // your API key
const headers = { Authorization: `Bearer ${TOKEN}` };

// Get all errors
const errors = await fetch(
  "http://127.0.0.1:3000/admin/events?severity=error",
  { headers },
).then((r) => r.json());

// Get script errors for a specific lexicon
const scriptErrors = await fetch(
  "http://127.0.0.1:3000/admin/events?event_type=script.error&subject=com.example.feed.like",
  { headers },
).then((r) => r.json());

// Get all lexicon-related events
const lexiconEvents = await fetch(
  "http://127.0.0.1:3000/admin/events?category=lexicon",
  { headers },
).then((r) => r.json());

// Paginate through results
const page = await fetch(
  "http://127.0.0.1:3000/admin/events?limit=20&cursor=2026-03-01T11:59:00Z",
  { headers },
).then((r) => r.json());

Rust

let client = reqwest::Client::new();
let token = "hv_..."; // your API key

// Get all errors
let errors: serde_json::Value = client
    .get("http://127.0.0.1:3000/admin/events")
    .query(&[("severity", "error")])
    .bearer_auth(token)
    .send()
    .await?
    .json()
    .await?;

// Get script errors for a specific lexicon
let script_errors: serde_json::Value = client
    .get("http://127.0.0.1:3000/admin/events")
    .query(&[
        ("event_type", "script.error"),
        ("subject", "com.example.feed.like"),
    ])
    .bearer_auth(token)
    .send()
    .await?
    .json()
    .await?;

// Get all lexicon-related events
let lexicon_events: serde_json::Value = client
    .get("http://127.0.0.1:3000/admin/events")
    .query(&[("category", "lexicon")])
    .bearer_auth(token)
    .send()
    .await?
    .json()
    .await?;

// Paginate through results
let page: serde_json::Value = client
    .get("http://127.0.0.1:3000/admin/events")
    .query(&[("limit", "20"), ("cursor", "2026-03-01T11:59:00Z")])
    .bearer_auth(token)
    .send()
    .await?
    .json()
    .await?;

Go

token := "hv_..." // your API key

// Get all errors
req, _ := http.NewRequest("GET",
  "http://127.0.0.1:3000/admin/events?severity=error", nil)
req.Header.Set("Authorization", "Bearer "+token)
errors, err := http.DefaultClient.Do(req)

// Get script errors for a specific lexicon
req, _ = http.NewRequest("GET",
  "http://127.0.0.1:3000/admin/events?event_type=script.error&subject=com.example.feed.like", nil)
req.Header.Set("Authorization", "Bearer "+token)
scriptErrors, err := http.DefaultClient.Do(req)

// Get all lexicon-related events
req, _ = http.NewRequest("GET",
  "http://127.0.0.1:3000/admin/events?category=lexicon", nil)
req.Header.Set("Authorization", "Bearer "+token)
lexiconEvents, err := http.DefaultClient.Do(req)

// Paginate through results
req, _ = http.NewRequest("GET",
  "http://127.0.0.1:3000/admin/events?limit=20&cursor=2026-03-01T11:59:00Z", nil)
req.Header.Set("Authorization", "Bearer "+token)
page, err := http.DefaultClient.Do(req)

cURL

AUTH="Authorization: Bearer hv_..." # your API key

# Get all errors
curl "http://127.0.0.1:3000/admin/events?severity=error" -H "$AUTH"

# Get script errors for a specific lexicon
curl "http://127.0.0.1:3000/admin/events?event_type=script.error&subject=com.example.feed.like" -H "$AUTH"

# Get all lexicon-related events
curl "http://127.0.0.1:3000/admin/events?category=lexicon" -H "$AUTH"

# Paginate through results
curl "http://127.0.0.1:3000/admin/events?limit=20&cursor=2026-03-01T11:59:00Z" -H "$AUTH"

See the Admin API reference for full parameter documentation.

Retention

Event logs are automatically cleaned up after 30 days. Change that with Event Log Retention (days) under System > General, or the EVENT_LOG_RETENTION_DAYS environment variable; the setting wins. A background task runs hourly to delete events older than the retention period. Job logs and stored script performance metrics follow the same period.

Set the value to 0 to disable automatic cleanup and keep logs indefinitely.

See the Configuration reference for the other retention settings.

Dead Letters

When a record or label script fails after all retry attempts, the event is stored as a dead letter. You can manage dead letters from the Data > Dead Letters page in the dashboard.

From the dead letters page you can:

• Retry: replay the stored event through the script (use after fixing the script)
• Re-index: fetch the record fresh from the PDS and run it through the full indexing pipeline (use when the record may have changed)
• Dismiss: mark the dead letter as resolved without retrying

Bulk actions are available for selected rows or all entries matching the current filters.

Next steps

• Event logs: full query parameters and response format
• Permissions: control which operators can read event logs
• Troubleshooting: what an error means and how to fix it