Writing records from Rust

The substrio_wasm::db module in the substrio_wasm crate gives you updates by id and by filter, upserts, batch inserts, replaces, deletes, the JSON body of a query, SQL through a collection, and the declarations that shape a collection. Each call uses the platform write path directly. There is no copy of the document on the way in and no second parse.

An update is built once with Update and handed to the platform as a fixed list of cells. A filter is the query you already build for reads, so a filtered write is the query's last step.

One call for a whole chain

A chain is written as a short program in your module's memory. Each step adds a few bytes without crossing into the platform. The terminal adds an operation and sends the program in one call. The platform decodes it using the same code that builds a JavaScript chain plan, then runs the plan. A ten-step chain takes one call; making a call for each step would take thirteen.

A chain you run again and again with different values can be kept with prepare(). Its steps are decoded once. Each later run passes only its values and its terminal, so no step is decoded twice. Mark a value with db::arg(n) and pass the values with bind.

use substrio_wasm::db::{self, Order, Param};

let orders = db::collection("orders")?;
let recent = orders.gte("total", db::arg(0)).sort("placed", Order::Desc).limit(20).prepare()?;
let big = recent.bind(&[Param::I64(500)]).all(|row| row.to_json())?;
let small = recent.bind(&[Param::I64(10)]).count()?;

bind works on any chain, prepared or not. A later bind replaces the values of an earlier one. Prepared chain steps are fixed at preparation time. Adding a step causes an error on the next run.

A worked example

#![no_std]
extern crate alloc;

use substrio_wasm::db::{self, Update};
use substrio_wasm::{Ctx, Env, Request, Response};

fn fetch(_req: &Request, _env: &Env, _ctx: &Ctx) -> Response {
    let orders = match db::collection("orders") {
        Ok(c) => c,
        Err(e) => return Response::json(500, alloc::format!("{{\"code\":{}}}", e.code)),
    };
    // Every open order older than the cutoff is marked late, in one call.
    let late = orders
        .eq("state", "open")
        .lt("placed", 1_767_225_600_000i64)
        .update(&Update::new().set("state", "late").inc("reminders", 1));
    match late {
        Ok(n) => Response::text(200, &alloc::format!("{} orders marked late", n)),
        Err(e) => Response::json(500, alloc::format!("{{\"code\":{}}}", e.code)),
    }
}

substrio_wasm::export!(fetch);

Updates

Update::new() starts an update, and each method adds one operator. The names follow the JavaScript operators.

A failed guard leaves the document unchanged. The write returns as if no document had matched.

Collection calls

By default, insert_many stores the whole batch or none of it. A unique-field conflict prevents the batch from being stored. If skip_conflicts is set, documents with conflicts are skipped and the rest are stored.

Query calls

Walking rows

each(read, then) passes each row to then in query order, with its position in the walk. The walk ends when then returns false. chunk(n, read, then) passes rows in pages of n, with each page's index. Both return the number of rows passed to the callback. The query's ids are taken first, and each row is read when its turn comes, so then may write to the collection. A row removed during the walk is skipped. In each it keeps its position, as JavaScript's index does, and in chunk a page left with no rows is passed over and takes no index. A row added during the walk is not part of it. read makes the value of a row while the row is read, and it may only read.

cursor() walks the collection's documents in id order without a query, as JavaScript's raw collection cursor does. Each call of page(at, n, read) makes one move and answers up to n documents from there, and the cursor then stands on the last of them. The moves are Seek::First, Seek::Last, Seek::Ge(&id), Seek::Le(&id), Seek::Lt(&id), Seek::Next, and Seek::Prev. Last, Le, Lt, and Prev walk backwards. A document that has expired and has not been swept yet is passed over. The cursor is only the id it stands on, so it needs no closing, and a write between two pages is seen by the next page. id() answers the id the cursor stands on. JavaScript's seekPrefix and seekPrefixLast have no Rust form, because a document id is a number with no text for a prefix to match.

let deck = db::collection("deck")?;
let mut seen = Vec::new();
deck.sort("i", Asc).each(|card| card.get("i").and_then(|v| v.as_i64()), |i, _| { seen.push(i); seen.len() < 4 })?;

let mut cur = deck.cursor();
let mut page = cur.page(Seek::First, 100, |r| r.to_json())?;
while !page.is_empty() {
    page = cur.page(Seek::Next, 100, |r| r.to_json())?;
}

An upsert takes the seed you pass, a JSON object holding the filter's equality fields, and lays the update's values into it when no document matches. When the update holds only plain fields, the update itself is the new document.

An upsert takes the collection's upsert gate, the same gate a JavaScript upsert takes, and holds it from the find to the write. Two upserts of one key, from Rust or from JavaScript, never both insert. The second one updates the document the first one inserted. The key is the chain's equality conditions, so eq("email", e) and JavaScript's {email: e} share one gate. A chain with any other condition, or with no condition, takes the gate of the whole collection. When the call's time limit passes before the gate is free, the upsert is refused with TIMEOUT and nothing is written. A Date in the filter is part of the key as a Date, in Rust as in JavaScript, so both share its gate.

upsert_many(&keys, docs) runs an array of documents in one call. Each document is matched on its values of the key fields, which must be strings, numbers, or booleans. A match takes the document's fields, and a miss inserts the document with its $setOnInsert fields laid in. The call holds the gate of the whole collection, so no other upsert runs between a document's find and its write. A document may match one that an earlier document in the same call inserted. The call is all or nothing. When a unique field, a schema, or a quota refuses one document, the call puts every updated document back exactly as it was, removes every document it inserted, and is refused. For the key _id, use put_many.

let people = db::collection("people")?;
let ids = people.upsert_many(&["email"], r#"[{"email":"[email protected]","visits":1},{"email":"[email protected]"}]"#)?;

upsert_many_typed(key, &docs) takes the documents as Update values instead of JSON. Each document holds its fields as set cells and its insert-only fields as set_on_insert cells, and it is matched on its value of the one field key. The call is all or nothing, as upsert_many is. When every field of a document is a scalar set of a field the collection already has, the document takes the same planned path a JavaScript upsertMany takes. Its cells are read where they lie, with no JSON written or parsed. The document is found by one seek of the key's index, and it is then patched in place or inserted from that same read. A document that brings a new field, a nested value, or a set_on_insert takes the general path instead. A document that holds any other operator, or no value for its key, is refused with INVALID_ARGUMENT.

let names: Vec<String> = (0..200).map(|i| format!("user{i}")).collect();
let docs: Vec<Update> = names.iter().enumerate().map(|(i, n)| Update::new().set("email", n.as_str()).set("visits", i as i64)).collect();
let ids = people.upsert_many_typed("email", &docs)?;

Search

The search steps are chain steps like any other. Each one becomes the same search leaf a JavaScript chain adds, and it is run by the same index search. There is no query plan to encode and no JSON on the way in or out.

search embeds the text with the collection's RAG configuration and then finds the nearest documents in its vector field. A kept chain embeds its text once, when it is kept. all_scored answers each row with the score of the ranking, which is the squared distance for a nearest search and NaN when no search step ranked the run.

Search also carries the options of the HTTP $search. The mode is SearchMode::Dense, Sparse, or Hybrid, where the hybrid mode fuses the dense, sparse, and text rankings. decay lowers the score of older documents, feedback re-ranks by the collection's feedback signals, and strict_model keeps only documents embedded by the current model. These options travel as typed values into the same structure that the HTTP $search reads its JSON into, so there is no JSON on the way in or out. A search with a mode other than dense, or with any option, is the chain's only step besides limit. That chain answers all, first, all_scored, ids, count, and exists, and all_scored gives NaN scores because the ranking is the order of the rows.

use substrio_wasm::db::{self, Decay, Search, SearchMode};

let notes = db::collection("notes")?;
let hits = notes
    .query()
    .search("quarterly plan", &Search { mode: SearchMode::Hybrid, decay: Some(Decay::new(7.0 * 86_400_000.0)), ..Search::default() })
    .limit(10)
    .all(|row| row.id().to_owned())?;

substrio_wasm::services::search(collection, text, &SearchOptions) corresponds to substrio.search. With field set it runs a text search of that field, and otherwise it runs the search above with a default k of 10.

Relational steps

The relational steps are chain steps too. Each one is decoded into the same plan a JavaScript chain builds, so the same engine runs it. Field names are read straight from your module's memory, and a join names the other collection by its handle, so a run looks up no collection by name.

A grouped, aggregated, distinct, or compound chain, or one that selects a joined field, returns plain rows. End it in rows() or rows_first(), which return the same sql::Rows that sql::sql returns, or in count(). A joined chain ends in joined(|row, joined| ...), which hands you each row with one joined row per join, in join order, and None where a join found no match. Each joined row is read in place, as any row is. A plain chain keeps all, first, and the rest.

use substrio_wasm::db::{self, Aggregate, Compare, Join};

let orders = db::collection("orders")?;
let people = db::collection("people")?;
let totals = orders
    .query()
    .group_by(&["person"])
    .aggregate(Aggregate::Sum, "amount", "total")
    .having("total", Compare::Gt, 100)
    .rows()?;
let named = orders
    .query()
    .join(Join::First, people, "person", "_id", "who")
    .joined(|order, joined| (order.id().to_owned(), joined[0].as_ref().map(|p| p.id().to_owned())))?;

Inside and, or, and not, a part holds conditions only: the filters, the field compares, and nested groups. A compound's arm takes no search with options. An arm may join other collections, and its rows are the rows its joins keep, as a JavaScript arm's are. An arm may be a kept chain (prepare()). Its steps are copied into the chain once, as an unprepared arm's steps are, and a db::arg(n) in the arm reads the nth argument of the chain's run. The kept chain still runs alone afterwards. A chain that breaks one of these rules is refused with UNSUPPORTED_OPERATION, and so is a plain-rows chain that ends in all or a joined chain that ends in anything but joined or count.

Relations

A relation is declared on a collection once, when your function starts a call. has_many(name, target, key) says that the documents of target whose field key holds a document's id are that document's related documents. has_one declares the same for one document, and belongs_to declares the reverse. The platform keeps the declaration on the collection, in the same place a JavaScript hasMany keeps it, and each call answers a Relation that the chain steps take.

belongs_to_many declares a many-to-many relation. Each link is one document in the through collection, which holds this collection's id in key and the target's id in other_key. An empty field name takes the collection's name in the singular with Id, as in JavaScript. link inserts the one document that names both ids, and writes nothing when the two are linked already. unlink deletes every document that names both. Each costs one call: one lookup of the pair by its two fields, then one insert or one delete for each link found.

A chain with with or with_count ends in related(|row, related| ...) or related_first, or in count(). related.rows(k) holds the related documents of the chain's kth with, and related.count(k) holds their count. Each related row is read in place, as any row is.

use substrio_wasm::db::{self, Order::Asc};

let people = db::collection("people")?;
let posts = db::collection("posts")?;
let written = people.has_many("posts", posts, "author")?;
let shelves = people
    .query()
    .with(&written)
    .with_count(&written)
    .sort("name", Asc)
    .related(|person, related| (person.id().to_owned(), related.rows(0).len(), related.count(1)))?;
let authors = people.query().where_has(&written).count()?;

The related documents of every row come from one pass over the relation's key. The key's values are read once and the index is held once for the whole pass, and with_count reads each count from the count table without reading a document. A with_where constraint, or a key whose values are not all plain document ids, runs the related chain once for each row instead, with the row's id as its key. That is the same path every chain takes.

where_has and where_none read the ids the key field holds, straight from its index, and keep the rows whose id is among them. A kept chain (prepare()) keeps the ids it read with the write generation of each collection they came from. A run reads them again only when one of those collections was written since, so a related document added between two runs is seen by the second. A run after no such write uses the kept ids and reads no index. A related collection that expires documents is read at every run, because time hides its documents with no write.

where_has and where_none also take a belongs_to and a belongs_to_many. A belongs_to keeps the rows whose key field names a document of the related collection. A belongs_to_many keeps the rows that a link document names. Where the key field's index does not hold plain document ids, the related rows are read and each row's key is taken from the index or the record. A kept chain keeps these keys in the same way. A belongs_to_many keeps the write generation of the link collection as well, so a link made after the prepare is seen by the next run. The same code reads these keys for JavaScript and for Rust.

These forms are refused with UNSUPPORTED_OPERATION in Rust for now: with of a has_one or a belongs_to (use join), a dotted path such as posts.comments, with_count with a constraint, with beside a join or a grouped chain, and a relation step inside a union's arm or a constraint. A relation name that is not declared is refused with INVALID_ARGUMENT.

Many writes in one call

Batch holds a list of writes and runs them in one call, in order, under one write scope. Consecutive writes to one collection are recorded as one change entry with one timestamp, so replication and the change history receive them together. This holds for every kind of write, so inserts, sets, updates, deletes, and puts to one collection share one entry. Consecutive inserts into one collection go through one insert scope. A write to another collection starts the next entry. Each write answers one word, read straight from your module's memory, so there is no JSON of the list and no JSON answer.

let people = db::collection("people")?;
let bump = Update::new().inc("visits", 1);
let answers = db::Batch::new()
    .insert(people, r#"{"name":"Ada"}"#)
    .update(people, id, &bump)
    .delete(people, old_id)
    .run()?;

run returns one result for each write up to the first one that is refused, and that refused write is the last result. The writes before it stand.

Declarations

text(field), top(field, weight), keyword(field, term_bytes), unique(field), expire(field, after_ms, allow_existing), and drop_expiry() declare the same indexes and rules as their JavaScript twins. has_many, has_one, belongs_to, and belongs_to_many declare relations (see Relations).

compound(&lead, order) declares a compound index. An equality on every lead field and a range or a sort on the order field then read one key range. elements(field, &members) keys each element object of an array field by its members, so an elemMatch of equalities on the first members and one range on the next reads one key range. drop_compound and drop_elements take a declaration away and answer true when it was there. schema(definition) takes the same JSON definition as JavaScript's schema({rules, constraints, strict, existing}). Stored documents that break it make the call fail with SCHEMA_VIOLATION, and its message names the first one. Each declaration is one call, made once when your function starts.

compound_indexes(), element_indexes(), and schema_json() read the declarations back as JSON text. It is the same text JavaScript's compoundIndexes(), elementIndexes(), and schema() parse, and the same entries an index.list request answers. A collection with no declarations answers [], and one with no schema answers null. Each read is one call.

let compounds = people.compound_indexes()?;  // [{"kind":"collection","collection":"people","compound":[...],...}]
let schema = people.schema_json()?;          // null until a schema is declared

Cloudflare shapes

db::D1 has prepare(sql), exec(sql), and batch(statements), and a statement has bind, all, run, first, and raw. db::KvNamespace::open(name) has get, put, delete, and list. Both are written in Rust over the SQL and KV calls, so they cost the same as calling those directly. As in JavaScript, batch runs the statements one after another and is not a transaction.

Refusals

A failed call returns Err(db::Error) with the code JavaScript throws by name.

The message JavaScript would carry is kept with the refusal, and substrio_wasm::sql::last_error_message() reads it.

What a call costs

What is not here yet

with over a belongs_to_many and the relation forms listed under Relations are still JavaScript only.

Not in Rust, and why

Two JavaScript features have no Rust form, and none is planned.

Scopes are not lifted. In JavaScript, collection.scope(name, fn) keeps a function that adds steps to a chain, so a chain can call it by name. In Rust, a plain function from Query to Query does the same work. It is compiled into the function, so it costs nothing at run time and has nothing to declare.

Live rows are not lifted. In JavaScript, live(id), a bound Model class, and for...of over a collection hand out objects whose field assignments are written at once. They are built on the JavaScript engine's object model. In Rust, update(id, &ops) makes the same write in one call. With a guard on the update, a write over a document another writer changed is refused, as a live row's write is refused with ROW_STALE.

See also Rust functions, the entry page: when to pick Rust, every call exposed to it, deploy and testing.