{% extends "base.html" %}
{% block title %}Code — Myrdal{% endblock %}
{% block head %}
  <script src="/static/_shared/ark-code.js" defer></script>
{% endblock %}
{% block content %}
<section class="ark-wrap ark-prose">
  <h1>Code</h1>
  <p class="ark-lede">Two bodies of code stand behind this site. One is downloadable today; the other is not yet, and this page says which is which rather than implying both.</p>

  <h2>Downloadable now — the package and site build</h2>
  <p>The <a href="{{ meta.bundles.code.href if meta.bundles and meta.bundles.code else '/data' }}">site code bundle</a>
  is the tooling that turns the Anu chopped output into what you are reading:
  <code>tools/build_web_package.py</code> (writes the Parquet and XLSX copies, regenerates the data
  dictionary and the manifest, and asserts the manifest against the disk in both directions),
  <code>tools/build_bundles.py</code> (the reproducible zips), the FastAPI application, and the
  templates. It carries no credentials and no absolute paths, and it is MIT-licensed.</p>
  <p>Reproducing the published package from the chopped CSVs is one command:</p>
  <div class="ark-code" data-arkcode data-filename="myrdal_rebuild" data-default="python">
    <div class="ark-code-head"><span class="ark-code-path">Rebuild the package · verify the counts</span></div>
    <pre data-lang="python"><code>{% filter forceescape %}{% raw %}# 1. regenerate parquet + xlsx + dictionary + manifest, and assert both ways
python tools/build_web_package.py --pkg /path/to/Myrdal_Web_v1.0.0

# 2. read the counts back out of the manifest the build just wrote
import json
m = json.load(open("/path/to/Myrdal_Web_v1.0.0/WEB_MANIFEST.json", encoding="utf-8"))
c = m["counts"]
assert c["series_registered"] == c["series_published"] == c["chopped_csv"]
assert c["chopped_parquet"] == c["chopped_xlsx"] == c["chopped_csv"]
# these three are DELIBERATELY smaller -- do not "fix" them to match
print(c["series_with_data"], c["series_chartable"], c["observations_total"]){% endraw %}{% endfilter %}</code></pre>
    <pre data-lang="r"><code>{% filter forceescape %}{% raw %}# 1. regenerate parquet + xlsx + dictionary + manifest (the builder is Python)
system("python tools/build_web_package.py --pkg /path/to/Myrdal_Web_v1.0.0")

# 2. read the counts back out of the manifest the build just wrote
library(jsonlite)
m <- fromJSON("/path/to/Myrdal_Web_v1.0.0/WEB_MANIFEST.json")
c <- m$counts
stopifnot(c$series_registered == c$series_published,
          c$series_published  == c$chopped_csv,
          c$chopped_parquet   == c$chopped_xlsx)
# these three are DELIBERATELY smaller -- do not "fix" them to match
c(c$series_with_data, c$series_chartable, c$observations_total){% endraw %}{% endfilter %}</code></pre>
  </div>

  <h2>Reading a series</h2>
  <p>Every series is one file. Parquet is the fastest way in; the CSV is identical.</p>
  <div class="ark-code" data-arkcode data-filename="myrdal_read_series" data-default="python">
    <div class="ark-code-head"><span class="ark-code-path">D001 — long-run world population</span></div>
    <pre data-lang="python"><code>{% filter forceescape %}{% raw %}import pandas as pd

# Parquet (preferred) or CSV -- identical rows.
d = pd.read_parquet("https://myrdal.heterodata.org/downloads/series/D001.parquet")

# D001 carries TWO measures: the sum over individual countries and the
# publisher's own world aggregate. They differ; both are kept.
print(d[["year", "pop_persons", "wld_official_total"]].tail())

# The pre-1960 rows come from a different publisher than the rest. The seam is
# recorded IN THE FILE, in the `source` column -- never assume one series.
print(d["source"].unique()){% endraw %}{% endfilter %}</code></pre>
    <pre data-lang="r"><code>{% filter forceescape %}{% raw %}library(arrow)   # or readr::read_csv for the CSV
library(dplyr)

# Parquet (preferred) or CSV -- identical rows.
d <- read_parquet("https://myrdal.heterodata.org/downloads/series/D001.parquet")

# D001 carries TWO measures: the sum over individual countries and the
# publisher's own world aggregate. They differ; both are kept.
d %>% select(year, pop_persons, wld_official_total) %>% tail()

# The pre-1960 rows come from a different publisher than the rest. The seam is
# recorded IN THE FILE, in the `source` column -- never assume one series.
unique(d$source){% endraw %}{% endfilter %}</code></pre>
  </div>

  <h2>Not downloadable yet — the Anu construction pipeline</h2>
  <p>The upstream pipeline that produced the chopped CSVs lives in the project rather than on this
  site: ten loaders (<code>L01</code> Census baseline, <code>L02</code> World Development Indicators,
  <code>L03</code> Clio-Infra, <code>L04</code> Maddison, <code>L05</code> World Inequality Database,
  <code>L07</code> United Nations World Population Prospects, <code>L08</code> Internal Revenue Service
  Statistics of Income migration, <code>L10</code> American Community Survey counties,
  <code>L12</code> Population Estimates Program, <code>L14</code> Census International Database), six
  processors (dependency ratios, demographic transition, migration matrix, age–sex pyramid, nativity
  and citizenship, long-run population), and two validators (schema; cross-source reconciliation).</p>
  <p class="ark-note"><strong>Why it is not offered here.</strong> Those scripts still contain absolute
  workstation paths to the local source archive. Publishing them as they stand would leak filesystem
  layout, so they are withheld until they have been through the framework's publication scrub. That is
  a queued task, not a missing capability, and it is recorded on <a href="/methodology">Methodology</a>.
  Until then the loaders' <em>inputs</em> are fully named there, so the construction is reproducible
  from the published sources even though the scripts are not yet in your hands.</p>

  <p class="ark-fineprint">No public repository exists for this site yet, so the footer shows no
  “Website source” button rather than a link that would 404. Site and tooling: MIT. Data: see
  <a href="/downloads/TERMS.md">TERMS.md</a>.</p>
</section>
{% endblock %}