Parallel b3dm Encoding with a ProcessPoolExecutor

This guide parallelises the glTF→b3dm+Draco encode across CPU cores using Python’s concurrent.futures.ProcessPoolExecutor, driving the Node-based 3d-tiles-tools CLI through subprocess while chunking leaf jobs to amortise process startup, sizing workers to physical cores because Draco compression is CPU-bound, collecting per-job failures instead of crashing the batch, and enforcing a deterministic ordering so a CI rerun produces byte-identical tiles.

You hit this the moment a batch tiling pipeline grows past a few hundred shards: the encode is embarrassingly parallel — each tile is independent — but a naive for loop pins one core while the other fifteen idle, and every 3d-tiles-tools call pays a fresh Node startup. The fix is a bounded process pool over deterministically chunked jobs.

Chunked leaf jobs distributed across per-core worker processes A sorted list of leaf jobs is split by a chunker into one contiguous chunk per physical core, each chunk is encoded by a worker process running the subprocess batch, and the futures are collected into byte-stable outputs plus a separate failure list. Sorted leaf jobs (stable) Chunker one/core Worker · core 0 subprocess batch Worker · core 1 subprocess batch Worker · core N-1 subprocess batch Collect futures Byte-stable outputs Failure list
Sorted jobs are split into one contiguous chunk per physical core; each worker runs the subprocess encode, and the collected futures separate byte-stable outputs from a failure list.

Prerequisites

  • Python 3.10+ (concurrent.futures and os.cpu_count are standard library).
  • 3d-tiles-tools 0.4+ on PATH (Node 18+), and trimesh 4.4+ for the merge step.
  • A directory of per-tile .glb files already merged and placed in the local ENU frame, produced by the batch tiling pipeline. Geometry is authored around a survey anchor whose ENU→ECEF transform (source EPSG:32618+5703 → EPSG:4979 → EPSG:4978) lives on the tileset root, so the encode itself is CRS-agnostic — it only rewrites container bytes.
  • Physical vs logical core count. os.cpu_count() reports logical CPUs (hyperthreads); Draco saturates ALU throughput, so hyperthreads add little. Prefer physical cores where you can detect them.

Step-by-Step

1. Enumerate leaf jobs in a deterministic order

Sort the inputs once. A ProcessPoolExecutor may complete jobs out of order, but the submission order and each job’s output path must be fixed so two CI runs over the same inputs write the same files.

python
from pathlib import Path

SRC_DIR = Path("shards")          # one merged .glb per leaf tile
OUT_DIR = Path("tiles")
OUT_DIR.mkdir(parents=True, exist_ok=True)

jobs = [(glb, OUT_DIR / (glb.stem + ".b3dm"))
        for glb in sorted(SRC_DIR.glob("*.glb"))]   # sorted -> stable order
print(f"{len(jobs)} leaf encode jobs")

2. Size the pool to physical cores

Draco quantization and connectivity encoding are CPU-bound, so oversubscribing logical cores past the physical count only adds context-switch overhead. Use physical cores when detectable, and cap workers at the job count for small batches.

python
import os

def physical_workers() -> int:
    """Best-effort physical-core count; fall back to logical, then 1."""
    try:
        return len(os.sched_getaffinity(0))          # respects cgroup/CI limits
    except AttributeError:
        pass
    logical = os.cpu_count() or 1
    return max(1, logical // 2)                       # assume 2 threads/core

workers = min(physical_workers(), len(jobs)) or 1
print(f"encoding with {workers} worker processes")
Encoding throughput against process-pool size Throughput rises almost linearly up to the number of physical cores, gains only a few per cent from the hyperthreads above that, and then falls as workers contend for memory bandwidth and disk. The best pool size is the count of physical cores, not of logical processors. Adding workers past the physical core count costs throughput, it does not buy it 8 physical cores 16 logical — throughput falling 1 2 4 6 8 10 12 14 16 worker processes tiles / min Measure it on the target machine — the knee moves with mesh size, because the limit is memory bandwidth, not CPU
The curve is flat, then negative. Sizing the pool from os.cpu_count() lands on the far side of the knee on every hyperthreaded machine.

3. Chunk jobs to amortise Node startup

Each 3d-tiles-tools invocation starts a Node runtime (tens to hundreds of milliseconds). For thousands of tiny leaves that startup dwarfs the encode. Split the sorted job list into one contiguous chunk per worker so a worker starts Node a handful of times, not once per tile.

python
def chunked(seq, n_chunks):
    """Split seq into n_chunks contiguous, near-equal chunks (order preserved)."""
    k, m = divmod(len(seq), n_chunks)
    out, start = [], 0
    for i in range(n_chunks):
        size = k + (1 if i < m else 0)
        out.append(seq[start:start + size])
        start += size
    return [c for c in out if c]                      # drop empty tail chunks

chunks = chunked(jobs, workers)
print(f"{len(chunks)} chunks, sizes {[len(c) for c in chunks]}")

4. Define the worker: encode a whole chunk, collect failures

The worker runs in a separate process, so it must be a top-level function (picklable) and must not raise on a single bad tile — it returns a result list the parent aggregates. Encode to a temp path and atomically rename so an interrupted run never leaves a half-written b3dm.

python
import os
import subprocess

def encode_chunk(chunk):
    """Encode a list of (glb, out) jobs. Returns (ok_paths, failures)."""
    ok, failures = [], []
    for glb, out in chunk:
        tmp = out.with_suffix(".b3dm.tmp")
        try:
            subprocess.run(["3d-tiles-tools", "glbToB3dm",
                            "-i", str(glb), "-o", str(tmp), "-f"],
                           check=True, capture_output=True, text=True)
            subprocess.run(["3d-tiles-tools", "optimizeB3dm", "-i", str(tmp),
                            "-o", str(tmp), "-f",
                            "--options", "--draco.compressMeshes"],
                           check=True, capture_output=True, text=True)
            os.replace(tmp, out)                      # atomic on POSIX
            ok.append(str(out))
        except subprocess.CalledProcessError as exc:
            failures.append((str(glb), exc.stderr.strip().splitlines()[-1]
                             if exc.stderr else f"exit {exc.returncode}"))
            tmp.unlink(missing_ok=True)
    return ok, failures
Node startup cost, per job and per chunk Dispatching one job per worker invocation pays the Node interpreter's startup cost before every tile, which for small tiles is most of the wall clock. Sending a chunk of jobs to a single invocation pays that cost once and amortises it across the whole chunk. one job per invocation — 696 ms for six tiles one chunk per invocation — 466 ms for the same six red blocks are interpreter startup: paid six times paid once, then six encodes back to back The saving grows as tiles get smaller, which is exactly when a city has the most of them
Chunking is not about parallelism at all — it is about how often you pay for a cold interpreter. Size chunks so a worker runs for seconds, not milliseconds.

The pool size and the chunk size answer different questions, and it is worth keeping them apart. Pool size is a contention question: how many encodes can run at once before they start competing for the same memory bandwidth and the same disk queue. Chunk size is an overhead question: how much work each invocation must carry before the fixed cost of starting it stops mattering. Tuning one to compensate for the other produces a configuration that stops working the moment the tile size changes.

There is also a determinism cost to get right. A pool returns results in completion order, which varies run to run, so anything that consumes the results — a manifest, a tileset index, a content hash — has to re-sort them into the job order before writing. The cheapest way to guarantee that is to have each worker return its input index alongside its output, then sort on that index before serialising. Skipping the sort produces builds that are byte-different on every run for no reason other than scheduling jitter, which in turn defeats the content-hash comparison the incremental pipeline depends on.

5. Run the pool and aggregate results deterministically

Submit one future per chunk, then merge results in a fixed order (sorted by output path) so the aggregated report and any manifest written from it are identical across runs regardless of which worker finished first.

python
from concurrent.futures import ProcessPoolExecutor

encoded, failed = [], []
with ProcessPoolExecutor(max_workers=workers) as pool:
    for ok, failures in pool.map(encode_chunk, chunks):   # map preserves input order
        encoded.extend(ok)
        failed.extend(failures)

encoded.sort()                                            # deterministic report
failed.sort()
print(f"encoded {len(encoded)}/{len(jobs)} tiles, {len(failed)} failed")
for glb, msg in failed:
    print(f"  FAIL {glb}: {msg}")

if failed:
    raise SystemExit(1)                                   # fail the CI job

6. Verify byte-stability across reruns

The determinism contract is what lets the parent batch tiling pipeline trust its hash cache. Re-encode one tile in isolation and compare bytes to the pooled output.

python
import hashlib

def digest(path) -> str:
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

sample_glb, sample_out = jobs[0]
before = digest(sample_out)
encode_chunk([(sample_glb, OUT_DIR / "_recheck.b3dm")])
after = digest(OUT_DIR / "_recheck.b3dm")
assert before == after, "encode is non-deterministic — CI reruns will churn"
print("byte-stable:", before[:12])

Expected Output & Verification

On a 16-logical-core (8 physical) machine encoding 480 leaf tiles, the pool reports physical sizing, a handful of chunks, and a byte-stable sample:

text
480 leaf encode jobs
encoding with 8 worker processes
8 chunks, sizes [60, 60, 60, 60, 60, 60, 60, 60]
encoded 480/480 tiles, 0 failed
byte-stable: 3f9a1c7e0b52

Wall-clock scales close to linearly with physical cores until the Node startup floor and disk I/O dominate. Confirm the speedup and the core sizing from the shell:

bash
nproc --all                                   # logical cores visible to the process
python -c "import os; print(len(os.sched_getaffinity(0)))"   # honored affinity
time python encode_pool.py                    # compare against a serial baseline

Expect roughly a 6–7x speedup on 8 physical cores (not the full 8x — Node startup and the final os.replace are serial per tile), and identical sha256 sums for every b3dm across two runs.

Keep the encoder’s stdout out of the parent process. A pool of eight workers each streaming progress lines through a shared pipe will spend real time serialising on that pipe, and on a slow terminal it can dominate the run. Redirect worker output to per-job log files and surface only the failures; the logs are more useful afterwards anyway, because they are attributable to a specific tile.

Common Errors

BrokenProcessPool: A process in the process pool was terminated abruptly. A worker was killed by the OS out-of-memory reaper because too many Node encoders ran at once on large meshes, or a worker segfaulted. ProcessPoolExecutor cannot recover and the whole pool dies. Fix: size workers to physical cores (step 2), cap peak resident meshes by chunking rather than submitting one future per tile, and set your CI runner’s memory limit above workers × peak-Node-RSS.

AttributeError: Can't pickle local object when submitting jobs. The worker function or an object it closes over was defined inside another function, so ProcessPoolExecutor cannot pickle it to send to the child process. Fix: define encode_chunk at module top level and pass only picklable arguments (strings and Path objects), never open file handles or lambdas.

FileNotFoundError: [Errno 2] No such file or directory: '3d-tiles-tools'. The child process inherited a PATH that does not include the Node bin directory — common when the pool is spawned (macOS/Windows default) rather than forked, or inside a minimal CI container. Fix: install 3d-tiles-tools globally on PATH, or pass an absolute path to the executable in the subprocess.run argument list so every worker resolves it identically.

Back to 3D Tiles Batch Tiling Pipelines.