Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ target/
*.wasm
.idea/
__pycache__/
apps/*/src/pins.rs
15 changes: 8 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ INPUTS=<dir> make run APP=<app> # answers from <dir>, not apps/<app>/inputs
sh tools/check.sh # what CI runs: links, builds, runs, ports
```

`make run` mounts only the paths an app's manifest declares, read-only, and
runs it with `/` as the data directory, the way an Ark does. An optional path
is mounted when the fixture root holds it, and an app's answers at `/inputs`
when it declares inputs. The three fixture roots are described in
`fixtures/README.md`. An app whose required grants a root lacks stops before
its report pass there, which is expected.
`make run` mounts each app's grants, inputs and dependency outputs read-only,
and its own declared outputs writable. It runs with `/` as the data directory.
An optional path is mounted when the fixture root holds it. The main app's
answers are mounted at `/inputs` when it declares inputs. The fixture roots
are described in `fixtures/README.md`. An app whose required grants a root
lacks stops before its processing pass there, which is expected.

## Adding an app

Expand Down Expand Up @@ -80,5 +80,6 @@ Follow `docs/06-reports.md`. In short:
- Never read absence as homozygous reference.
- Never assume the ALT allele is the risk allele. Derive it from the study.
- Never grant more than the question needs.
- Never depend on randomness, time, the network or writable storage.
- Never depend on randomness, time, the network or storage outside declared
outputs.
- Never put anything in a report the owner can't read.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,5 +45,5 @@ run-%: FORCE
@sh tools/run.sh "$(BUILD)/$*.wasm" "$(FIXTURES)" "$(INPUTS)"

clean:
@rm -rf $(BUILD) apps/*/target apps/*/*.wasm
@rm -rf $(BUILD) apps/*/target apps/*/*.wasm apps/*/src/pins.rs
@echo "cleaned build outputs"
29 changes: 20 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,31 @@ Worked examples for writing apps that run on an Ark.

An Ark holds a person's genome on a device only they control. Apps never receive
a copy of that data. They are sent to it instead, and run in a deterministic
sandbox on the Ark with no network and no writable storage. An app reads the
data it was granted as plain files and prints a report. The owner approves each
app on their phone before it runs, and decides whether to release its report.
sandbox on the Ark with no network or storage that outlives the task. An app
reads the data it was granted as plain files and may use other apps in the same
task. The owner approves the task once on their phone, with the combined grants
of all its apps. The main app prints the report, and the owner decides whether
to release it.
Because the sandbox contains the code, anyone can write an app and anyone can
run anyone else's. The [whitepaper](https://dark.bio/whitepaper.pdf) lays out
the trust model.

These examples teach that model one idea at a time, from printing a line to
scanning a whole call file. Every app is a single source file.
scanning a whole call file and composing apps. Every app has one handwritten
source file.

## What an app is

An app is a [WebAssembly](https://webassembly.org/) module built for
[WASI Preview 1](https://github.com/WebAssembly/WASI). The Ark runs it twice.
[WASI Preview 1](https://github.com/WebAssembly/WASI). The Ark runs each app
in two passes.

- **With no arguments**, the app prints a short TOML manifest naming itself and
the data it wants. The Ark checks it and shows it to the owner for approval.
the data it wants. The Ark checks every app's manifest before asking the
owner to approve the task.
- **With a data directory as its first argument**, the app reads its granted
files and prints its report, a Markdown page the owner reads first.
files and inputs, and writes its declared outputs. Dependencies run before
the main app, whose standard output is the Markdown report the owner reads.

```rust
fn main() {
Expand All @@ -39,7 +45,9 @@ fn main() {
## Quickstart

Build and run an example against the fixture data in this repository, with
[`wasmtime`](https://wasmtime.dev/) and the toolchain for its language:
[`wasmtime`](https://wasmtime.dev/), a POSIX shell and AWK for the local runner,
and the toolchain for its language. Building Python apps needs
CPython 3.13 or newer to cross compile the interpreter they carry.

```sh
make run APP=01-hello-rust # one app
Expand Down Expand Up @@ -71,7 +79,7 @@ and a full app turns the same read into a report. Each row is a folder under

| App | What it shows | Languages |
| :-- | :-- | :-- |
| `01-hello` | the manifest pass and the report pass | Rust, Go, C, Python |
| `01-hello` | the manifest pass and the processing pass | Rust, Go, C, Python |
| `02-permissions` | requesting data, optional grants, least privilege and the `develop` flag | Rust |
| `03-sundae-order` | asking the owner questions with inputs | Rust |
| `04-cilantro-mini` | one variant through `rsids/` | Rust, Go, C, Python |
Expand All @@ -88,6 +96,9 @@ and a full app turns the same read into a report. Each row is a folder under
| `09-vcf-roll-call` | parsing the whole call file as a report | Rust |
| `10-motif-finder` | computing over a gene's reference sequence | Rust |
| `11-fortune-cookie` | why the sandbox is deterministic | Rust |
| `12-pigment-variants` | gathering six genotypes into an output, with a typed ordering input | Rust |
| `12-eye-color-model` | turning an untyped variant input into IrisPlex probabilities | Rust |
| `12-eye-color` | composing both apps with digest pins and input bindings | Rust |

## Documentation

Expand Down
2 changes: 1 addition & 1 deletion apps/01-hello-c/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ To see each pass by hand, build the module and run it twice:
```sh
make build APP=01-hello-c
wasmtime build/01-hello-c.wasm # manifest pass
wasmtime build/01-hello-c.wasm / # report pass
wasmtime build/01-hello-c.wasm / # processing pass
```
5 changes: 3 additions & 2 deletions apps/01-hello-c/main.c
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,16 @@ int main(int argc, char *argv[]) {
"description = '''\n"
"An Ark app is one WebAssembly file built for WASI Preview 1, and the Ark runs it twice.\n\n"
"**The manifest pass.** No arguments. The app prints a short TOML manifest with its name, its version and the data it wants, and your phone shows it to you.\n\n"
"**The report pass.** A data directory as its first argument. The app reads what you granted and prints a Markdown report, which Ark Hub renders.\n\n"
"**The processing pass.** A data directory as its first argument. The app reads\n"
"what you granted and prints a Markdown report, which Ark Hub renders.\n\n"
"This one asks for nothing, so approving it shares nothing. It's the whole round trip, upload to approval to report, in a handful of lines.\n\n"
"Each language's folder in the examples repository builds and runs it with one `make run`.\n\n"
"Pick a language, read the source, then run it on your Ark.\n"
"'''\n");
return 0;
}

// The report pass receives a data directory. This app reads nothing.
// The processing pass receives a data directory even without grants
printf("Hello from inside an Ark. No network, no disk, no way out. C says hi.\n");
return 0;
}
2 changes: 1 addition & 1 deletion apps/01-hello-go/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,5 @@ To see each pass by hand, build the module and run it twice:
```sh
make build APP=01-hello-go
wasmtime build/01-hello-go.wasm # manifest pass
wasmtime build/01-hello-go.wasm / # report pass
wasmtime build/01-hello-go.wasm / # processing pass
```
5 changes: 3 additions & 2 deletions apps/01-hello-go/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,15 @@ func main() {
"description = '''\n" +
"An Ark app is one WebAssembly file built for WASI Preview 1, and the Ark runs it twice.\n\n" +
"**The manifest pass.** No arguments. The app prints a short TOML manifest with its name, its version and the data it wants, and your phone shows it to you.\n\n" +
"**The report pass.** A data directory as its first argument. The app reads what you granted and prints a Markdown report, which Ark Hub renders.\n\n" +
"**The processing pass.** A data directory as its first argument. The app reads\n" +
"what you granted and prints a Markdown report, which Ark Hub renders.\n\n" +
"This one asks for nothing, so approving it shares nothing. It's the whole round trip, upload to approval to report, in a handful of lines.\n\n" +
"Each language's folder in the examples repository builds and runs it with one `make run`.\n\n" +
"Pick a language, read the source, then run it on your Ark.\n" +
"'''\n")
return
}

// The report pass receives a data directory. This app reads nothing.
// The processing pass receives a data directory even without grants
fmt.Println("Hello from inside an Ark. No network, no disk, no way out. Go says hi.")
}
2 changes: 1 addition & 1 deletion apps/01-hello-python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ To see each pass by hand, build the module and run it twice:
```sh
make build APP=01-hello-python
wasmtime build/01-hello-python.wasm # manifest pass
wasmtime build/01-hello-python.wasm / # report pass
wasmtime build/01-hello-python.wasm / # processing pass
```
5 changes: 3 additions & 2 deletions apps/01-hello-python/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ def main():

**The manifest pass.** No arguments. The app prints a short TOML manifest with its name, its version and the data it wants, and your phone shows it to you.

**The report pass.** A data directory as its first argument. The app reads what you granted and prints a Markdown report, which Ark Hub renders.
**The processing pass.** A data directory as its first argument. The app reads
what you granted and prints a Markdown report, which Ark Hub renders.

This one asks for nothing, so approving it shares nothing. It's the whole round trip, upload to approval to report, in a handful of lines.

Expand All @@ -34,7 +35,7 @@ def main():
'''""")
return

# The report pass receives a data directory, even when no data is requested
# The processing pass receives a data directory even without grants
print("Hello from inside an Ark. No network, no disk, no way out. Python says hi.")


Expand Down
2 changes: 1 addition & 1 deletion apps/01-hello-rust/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ To see each pass by hand, build the module and run it twice:
```sh
make build APP=01-hello-rust
wasmtime build/01-hello-rust.wasm # manifest pass
wasmtime build/01-hello-rust.wasm / # report pass
wasmtime build/01-hello-rust.wasm / # processing pass
```
5 changes: 3 additions & 2 deletions apps/01-hello-rust/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ An Ark app is one WebAssembly file built for WASI Preview 1, and the Ark runs it

**The manifest pass.** No arguments. The app prints a short TOML manifest with its name, its version and the data it wants, and your phone shows it to you.

**The report pass.** A data directory as its first argument. The app reads what you granted and prints a Markdown report, which Ark Hub renders.
**The processing pass.** A data directory as its first argument. The app reads
what you granted and prints a Markdown report, which Ark Hub renders.

This one asks for nothing, so approving it shares nothing. It's the whole round trip, upload to approval to report, in a handful of lines.

Expand All @@ -41,6 +42,6 @@ Pick a language, read the source, then run it on your Ark.
return;
}

// The report pass receives a data directory. This app reads nothing.
// The processing pass receives a data directory even without grants
println!("Hello from inside an Ark. No network, no disk, no way out. Rust says hi.");
}
2 changes: 1 addition & 1 deletion apps/02-permissions/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
use std::path::Path;
use std::{fs, io};

/// Required grant that the Ark mounts before the report pass.
/// Required grant that the Ark mounts before the processing pass.
const GRANTED: &str = "v1/genome/rsids/rs72921001";
/// Optional grant that the Ark mounts when available and approved.
const OPTIONAL: &str = "v1/genome/rsids/rs671";
Expand Down
2 changes: 1 addition & 1 deletion apps/04-cilantro-mini-go/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func main() {
return
}

// The report pass reads everything from the one variant directory
// The processing pass reads from the one variant directory
base := filepath.Join(os.Args[1], lens)

// ENOENT means no answer, including reference sites in variants-only calls.
Expand Down
2 changes: 1 addition & 1 deletion apps/04-cilantro-mini-rust/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Written in Rust, Go, C and Python. Compare them, then meet the full report, Cila
return;
};

// The report pass reads everything from the one variant directory
// The processing pass reads from the one variant directory
let base = Path::new(&dir).join(LENS);

// ENOENT means no answer, including reference sites in variants-only calls.
Expand Down
7 changes: 7 additions & 0 deletions apps/12-eye-color-model/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions apps/12-eye-color-model/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[package]
name = "eye-color-model"
version = "1.0.0"
edition = "2024"
publish = false

# A smaller module uploads and starts faster on an Ark.
[profile.release]
opt-level = "z"
lto = "fat"
codegen-units = 1
panic = "abort"
strip = "symbols"
84 changes: 84 additions & 0 deletions apps/12-eye-color-model/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# 12 - eye-color-model

Applies the original IrisPlex multinomial logistic regression to six
genotypes from [12-pigment-variants](../12-pigment-variants). It reads no
grants. Its untyped `variants` input requires a binding on an Ark, so this
app has no listing. [12-eye-color](../12-eye-color) binds its inputs and
reports its probabilities.

## Build and run

The local runner can debug a dependency alone, including its untyped input:

```sh
make run APP=12-eye-color-model
```

The sample `inputs/variants` holds the gatherer's default output, and
`inputs/detail` holds `full` without a trailing newline. `INPUTS=<dir>`
selects another directory containing both files. The model prints its output
for local inspection. The Ark drops a dependency's printout.

## Text passed between apps

`inputs/variants` uses the gatherer's
[four-field TSV format](../12-pigment-variants#the-variants-output), in either
order. Every rsID must appear once. A missing row, duplicate or unknown rsID
is a format error. Two called alleles from the modeled pair are required at
each site. Phase and a leading phase marker do not affect the copy count.

`outputs/colors` is UTF-8 text, with no header and a final newline. Each row
holds a color and its probability, separated by one tab. Probabilities range
from 0 to 1 and have nine decimal places. `full` writes `blue`, `intermediate`
and `brown`, in that order. `summary` writes only the largest probability,
with the first color in that order winning a tie. This is a probability
estimate without a classification threshold.

For the sample input, `full` writes:

```text
blue 0.946340538
intermediate 0.035254828
brown 0.018404634
```

An absent genotype, no-call, deletion, non-diploid call or allele outside the
modeled pair gives one row instead:

```text
unavailable missing-or-unsupported-genotype
```

## Coefficients and source

The coefficients are from Walsh et al. (2011), Supplementary Table 3. Each
weight adds to a log odds score per copy of the counted forward-strand
allele. Brown is the baseline, with score zero.

| Variant | Gene | Counted allele | Other allele | Blue vs brown | Intermediate vs brown |
| :-- | :-- | :-- | :-- | --: | --: |
| intercept | | | | 3.94 | 0.65 |
| rs12913832 | *HERC2* | A | G | -4.81 | -1.79 |
| rs1800407 | *OCA2* | T | C | 1.40 | 0.87 |
| rs12896399 | *SLC24A4* | G | T | -0.58 | -0.03 |
| rs16891982 | *SLC45A2* | C | G | -1.30 | -0.50 |
| rs1393350 | *TYR* | A | G | 0.47 | 0.27 |
| rs12203592 | *IRF4* | T | C | 0.70 | 0.73 |

For copy counts `x`, the scores are `b = 3.94 + sum(blue_weight * x)` and
`i = 0.65 + sum(intermediate_weight * x)`. Divide `exp(b)`, `exp(i)` and `1`
by their sum to obtain blue, intermediate and brown probabilities. The code
subtracts the largest score before exponentiating, which preserves those
ratios.

The model was fitted with Dutch European data. It separates blue from brown
well and calls intermediate colors, such as green and hazel, poorly.

- Walsh S, Liu F, Ballantyne KN, van Oven M, Lao O, Kayser M. IrisPlex: A
sensitive DNA tool for accurate prediction of blue and brown eye colour in
the absence of ancestry information. Forensic Science International:
Genetics. 2011;5:170-180.
[Paper and supplementary data](https://doi.org/10.1016/j.fsigen.2010.02.004).
- The author reproduces Supplementary Table 3 on printed page 336 of her
[thesis](https://repub.eur.nl/pub/40312/130605_Walsh%2C%20Susan%20-%20BEWERKT.pdf),
PDF page 281. This is the table used here.
1 change: 1 addition & 0 deletions apps/12-eye-color-model/inputs/detail
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
full
6 changes: 6 additions & 0 deletions apps/12-eye-color-model/inputs/variants
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
rs1393350 chr11 89277878 G/G
rs1800407 chr15 27985172 C/C
rs12203592 chr6 396321 C/C
rs12896399 chr14 92307319 T/T
rs12913832 chr15 28120472 G/G
rs16891982 chr5 33951588 G/G
Loading
Loading