Index of every page on this site: llms.txt
All skills
bluesky-social avatar

/lex-data

@d9cdc54
by bluesky-socialbluesky-social/atproto9.7k stars
941
Run it yourself

Represent, parse, serialize, hash, or convert AT Protocol data values with `@atproto/lex`. Use whenever code touches CIDs, raw bytes, `$link` / `$bytes` JSON, CBOR / DRISL encoding, blob references, or AT Protocol identifier and datetime strings — DIDs, handles, AT URIs, NSIDs, TIDs, record keys, `createdAt` / `indexedAt`. Applies when the ask sounds like "serialize this record", "hash these bytes", "why is this datetime rejected", "count the characters in this field", or when replacing `multiformats` CIDs, `new Date().toISOString()`, or hand-rolled base64. Read it before typing a DID, AT URI, or datetime as a bare `string`.

  • 1 file
  • 14.3 KB
  • Updated 3 weeks ago
  • GitHub
Use this Skill: https://skilld.dev/gh/bluesky-social/atproto/lex-data

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈161 tokens for metadata: the name and description. ≈3.5k when used: this file.

Description uses 8.0% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈643 characters, or 8.0%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Data model: types, CIDs, JSON, CBOR, identifier strings, blobs

The AT Protocol data model is JSON plus two primitives — CIDs (content-addressed links) and bytes (raw binary) — encodable as JSON or as CBOR.

Which package exports what

@atproto/lex re-exports lex-data, lex-json, lex-cbor, lex-schema and lex-client, but packages in this repo only list it as a devDependency (see the lex-setup skill): import from the package that defines or re-exports each symbol below.

Symbols Defined in
LexValue, Cid, blob refs, base64/utf8 helpers, LexError @atproto/lex-data
lexParse, lexStringify, jsonToLex, lexToJson @atproto/lex-json
Identifier & datetime string types + guards @atproto/syntax, re-exported by @atproto/lex-schema
encode, decode, decodeAll, cidForLex @atproto/lex-cbor

Low-level packages (repo, common, xrpc-server) import @atproto/lex-data directly to avoid pulling in the client and schema layers.

[!NOTE] This repo sets verbatimModuleSyntax: true, so type-only names need the type modifier — import { type DidString, isDidString } from '@atproto/lex-schema'. All examples below follow that form.

Lex value types

import {
  type LexValue,
  type TypedLexMap,
  isLexValue,
  isTypedLexMap,
} from '@atproto/lex-data'

// LexScalar:    number (integers only) | string | boolean | null | Cid | Uint8Array
// LexValue:     LexScalar | LexValue[] | { [key: string]?: LexValue }
// LexMap:       { [key: string]?: LexValue }
// TypedLexMap:  LexMap & { $type: string }

isLexValue, isLexMap, isLexArray and isLexScalar take unknown; isTypedLexMap takes a LexValue, so narrow with isLexValue first when the input is untyped. isLexValue walks the whole structure and rejects cycles, non-integer numbers, and non-plain objects (Date, class instances, functions) — it is not a cheap check, so avoid it on hot paths where a schema already validates the value.

CIDs

Cid is an interface, not a class. @atproto/lex-data defines it rather than re-exporting multiformats' CID, because that class breaks under node16 / bundler module resolution. Treat CID and asMultiformatsCID as deprecated escape hatches.

import {
  type Cid,
  asCid,
  decodeCid,
  ifCid,
  isCid,
  parseCid,
  parseCidSafe,
} from '@atproto/lex-data'

parseCid(str) // Cid — throws on invalid
parseCidSafe(str) // Cid | null
decodeCid(bytes) // Cid from binary form — throws on invalid
isCid(value) // type guard
ifCid(value) // Cid | null
asCid(value) // Cid — throws on invalid

Each accepts a { flavor } option that additionally constrains the CID, which is how you enforce that a blob CID really is a blob CID rather than trusting the caller:

  • 'raw' — v1 + raw multicodec (blob content)
  • 'cbor' — v1 + dag-cbor + sha256 (records, MST nodes)
  • 'dasl' — either codec, v1 + sha256

For strings you already hold, validateCidString(str, options) returns a boolean and also checks the string is the canonical encoding; ensureValidCidString throws instead.

Creating CIDs — hashing is async because it goes through WebCrypto:

import {
  cidForCbor,
  cidForRawBytes,
  cidForRawHash,
  isCidForBytes,
} from '@atproto/lex-data'
import { cidForLex } from '@atproto/lex-cbor'

await cidForLex(record) // CBOR-encode then hash — the usual entry point
await cidForCbor(cborBytes) // hash bytes that are already CBOR
await cidForRawBytes(blobBytes) // raw CID for blob content
cidForRawHash(sha256Digest) // sync, when you already streamed the hash
await isCidForBytes(cid, bytes) // verify a CID matches its content

In Lex JSON, CIDs are encoded as { "$link": "bafyrei..." }.

Bytes

Binary data is Uint8Array; JSON-encoded as { "$bytes": "base64..." }. @atproto/lex-data ships the byte utilities so packages don't add their own base64 dependency: toBase64 / fromBase64 (both take an optional 'base64' | 'base64url' alphabet), utf8ToBase64 / utf8FromBase64, utf8FromBytes, ui8Concat, ui8Equals, asUint8Array, ifUint8Array.

JSON ↔ Lex

Function Direction Input Output
lexParse JSON string → Lex string LexValue (with Cid, Uint8Array)
lexStringify Lex → JSON string LexValue string
jsonToLex parsed JSON → Lex plain JS object LexValue
lexToJson Lex → plain JS LexValue plain JS object

lexParse<T>() takes a type parameter, so no as cast is needed:

import { lexParse, lexStringify } from '@atproto/lex-json'

const sub =
  lexParse<app.bsky.notification.defs.SubjectActivitySubscription>(json)
const out = lexStringify({ ref: someCid, data: someBytes })

lexParse and jsonToLex accept { strict }, defaulting to false: malformed $link / $bytes objects and non-integer numbers pass through unchanged rather than throwing. Pass { strict: true } when the input is untrusted and a silently-wrong value would be worse than an exception. lexParseJsonBytes(bytes, options) is the same parse starting from UTF-8 bytes.

Field-level encode/decode is rarely needed, but note the return contract — these signal failure by returning undefined, they do not throw:

import {
  encodeLexBytes,
  encodeLexLink,
  parseLexBytes,
  parseLexLink,
} from '@atproto/lex-json'

parseLexLink({ $link: 'bafy...' }) // Cid | undefined
parseLexBytes({ $bytes: 'SGVsbG8=' }) // Uint8Array | undefined
encodeLexLink(someCid) // { $link: '...' }
encodeLexBytes(new Uint8Array([1, 2, 3])) // { $bytes: '...' }

CBOR (DRISL)

Use CBOR for repo storage, signed records, CAR files, and event frames — anywhere DRISL deterministic encoding is required. Add @atproto/lex-cbor as a dependency.

import { type LexValue } from '@atproto/lex-data'
import { cidForLex, decode, decodeAll, encode } from '@atproto/lex-cbor'

const bytes: Uint8Array = encode(someLexValue)
const value = decode<{ foo: LexValue }>(bytes)

// Concatenated values (CAR blocks, subscription frames)
for (const frame of decodeAll(buffer)) {
  /* … */
}

encode throws on anything the AT data model forbids: non-string map keys, undefined, and non-integer numbers. That strictness is the point — it is what makes the resulting CID stable.

Datetime strings

DatetimeString is the branded type for AT Protocol datetimes. Date.prototype.toISOString() is not guaranteed to conform (years outside 0–9999 serialize with a ±YYYYYY prefix), so route Date → string through the helpers rather than calling toISOString() directly:

import {
  type DatetimeString,
  asDatetimeString,
  currentDatetimeString,
  ifDatetimeString,
  isDatetimeString,
  toDatetimeString,
} from '@atproto/lex-schema'

currentDatetimeString() // now
toDatetimeString(date) // throws InvalidDatetimeError on an out-of-range date
asDatetimeString(str) // validates + brands a string, throws on invalid
isDatetimeString(value) // type guard
ifDatetimeString(value) // DatetimeString | undefined

isDatetimeStringLenient accepts ISO-ish strings that the spec rejects (e.g. missing a timezone) — use it when reading legacy records, not when writing. For repairing bad historical createdAt values, normalizeDatetime / normalizeDatetimeAlways live in @atproto/syntax only; they are not re-exported by @atproto/lex-schema.

Use DatetimeString on record and DB row types so a raw string can't drift in:

import type { DatetimeString, DidString } from '@atproto/lex-schema'

interface Row {
  did: DidString
  indexedAt: DatetimeString
}

Identifier strings

Nominal string types keep unvalidated strings from being passed where a validated identifier is expected.

Type Format Guard
DidString did:method:specific-id isDidString
HandleString DNS-style handle isHandleString
AtIdentifierString DID or handle isAtIdentifierString
AtUriString at://… isAtUriString
UriString any scheme:… URI isUriString
NsidString app.bsky.feed.post isNsidString
TidString timestamp identifier isTidString
RecordKeyString rkey (TID or literal) isRecordKeyString
CidString CID in string form isCidString
LanguageString BCP-47 tag (en, pt-BR) isLanguageString
DatetimeString AT Proto datetime isDatetimeString

Only AtUriString, AtIdentifierString and DatetimeString also ship as… / assert… / if… variants. For the rest, the generic format helpers cover the same ground:

import {
  asStringFormat,
  ifStringFormat,
  isStringFormat,
} from '@atproto/lex-schema'

isStringFormat(value, 'did') // narrows to DidString
asStringFormat(value, 'nsid') // throws TypeError on invalid
ifStringFormat(value, 'handle') // HandleString | undefined

isAtUriString, isLanguageString and isStringFormat accept { strict: false } for a lenient pass that tolerates non-conforming real-world data.

[!NOTE] These types are defined in @atproto/syntax and re-exported through @atproto/lex-schema. Import from @atproto/lex-schema where the package already depends on it — that keeps identifier types and generated schemas coming from one place and avoids adding a @atproto/syntax dependency.

Boundaries

Protobuf messages, data-plane responses and Kysely rows all arrive as bare string. Brand them once at the boundary so downstream code is typed, instead of asserting at each use:

dids: dids as DidString[],
post: { uri: item.uri as AtUriString, cid: item.cid || undefined },

An as cast is appropriate where the value provably came from a validated source (your own DB, a schema-validated response). Where it did not — env vars, CLI args, user input — run the guard: assert(isDidString(serverDid)).

Blob references

Two shapes coexist on the network:

type TypedBlobRef = { $type: 'blob'; ref: Cid; mimeType: string; size: number }
type LegacyBlobRef = { cid: string; mimeType: string } // no $type, no size
type BlobRef = TypedBlobRef | LegacyBlobRef

New uploads are always TypedBlobRef — the PDS uploadBlob endpoint returns one. Reading code has to tolerate both, which is what the format-agnostic accessors are for:

import {
  type BlobRef,
  enumBlobRefs,
  getBlobCid,
  getBlobCidString,
  getBlobMime,
  getBlobSize,
  isLegacyBlobRef,
  isTypedBlobRef,
} from '@atproto/lex-data'

getBlobCid(blob) // Cid — parses the string for legacy refs, so it can throw
getBlobCidString(blob) // string — skips the parse, prefer it when you only need the string
getBlobMime(blob) // string
getBlobSize(blob) // number | undefined — legacy refs carry no size

// Deep-walk a record for its blobs (what the PDS does on record write/import)
for (const ref of enumBlobRefs(record, { allowLegacy: true, strict: false })) {
  /* … */
}

enumBlobRefs skips legacy refs unless allowLegacy: true.

BlobRef is an interface, so instanceof does not apply — use isBlobRef / isTypedBlobRef / isLegacyBlobRef.

What strict means for blobs

The word means different things at two layers, and conflating them causes confusing validation failures:

  • On the guards (isBlobRef, isTypedBlobRef, isLegacyBlobRef, enumBlobRefs), strict constrains the CID flavor. Default true requires a raw v1 sha256 CID; strict: false accepts any CID (and tolerates size: -1). It does not reject the legacy shape — isBlobRef accepts legacy refs either way.
  • On schema validation (l.blob()), strict: true (the default) rejects legacy refs outright and enforces accept / maxSize; strict: false admits them and skips those checks.

A Client constructed with strictResponseProcessing: false threads strict: false through both response parsing and schema validation, which is how you read a server that still emits legacy refs.

String lengths

Lexicon string constraints are measured in graphemes (user-perceived characters) or UTF-8 bytes, never in JS string length:

import { graphemeLen, utf8Len } from '@atproto/lex-data'

graphemeLen('👨‍👩‍👧‍👦') // 1
utf8Len('👨‍👩‍👧‍👦') // 25

Comparison

lexEquals(a, b) deep-compares two LexValues, handling Cid and Uint8Array where === and JSON.stringify would not.

Related skills

  • lex-schema — validating these values against Lexicon schemas.
  • lex-setup — adding @atproto/lex-cbor and the rest of the package family to a package.
  • lexification-client — migrating off multiformats CID, the BlobRef class from @atproto/lexicon, and jsonStringToLex / stringifyLex.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at d9cdc54. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 weeks ago
disable-model-invocation
false

README badge

README badge for bluesky-social/atproto/lex-data