BitCrafts Toolkit

Role

Single repository for the entire bc-* ecosystem: six C libraries that form the foundation, three CLI tools built on top, and the BitCrafts Vigil desktop frontend. Pure C11, zero external runtime dependencies, baseline x86-64-v3 (AVX2 + BMI2 + FMA + LZCNT + MOVBE), GPLv3 (tools) / LGPLv3 (libraries).

Numbers at a glance

On a Ryzen 7 5700G (16 SMT threads, Linux 6.x, warm cache) against the Linux 6.12 + Node.js v22 source corpus (113k files, 2.2 GB):

bcduplicate peaks at 758k files/s — 18× rmlint on the same thread count. Full methodology, datasets, and reproduction scripts live under benchmarks/ in the repo, and the deep-dive walkthrough is in Introducing bit-crafts.

Layout

bit-crafts/
├── subprojects/        # libraries (LGPLv3)
│   ├── bc-core         # CPU primitives: hash, SIMD memory, checked arithmetic
│   ├── bc-allocators   # pool, arena, slab, context allocators
│   ├── bc-containers   # vector, hashmap, set, ring buffer, bitset
│   ├── bc-concurrency  # worker pool, lock-free MPMC queue, signal-safe shutdown
│   ├── bc-io           # streams, filesystem helpers, mmap, io_uring wrappers
│   └── bc-runtime      # lifecycle, layered config, structured logging, metrics, CLI
├── tools/              # CLI binaries (GPLv3)
│   ├── bc-hash         # recursive file-tree hashing (SHA-256 / xxh3 / BLAKE2b ...)
│   ├── bc-duplicate    # duplicate detection via size → fast-hash → full-hash funnel
│   └── bc-integrity    # JSONL filesystem manifests, verify, diff
├── applications/       # end-user apps built on the toolkit (GPLv3)
│   └── vigil           # GTK4 + libadwaita desktop frontend (BitCrafts Vigil)
└── benchmarks/         # comparator runs, datasets, hyperfine outputs

Highlights

• Adaptive parallel I/O: io_uring batched reads with per-worker queues, fallback to threadpool. Adaptive dispatch decides serial vs parallel per workload — break-even at roughly 90 files or 1 MB on Zen 3.
• Lock-free MPMC queue (Vyukov sequence-based) at the heart of every parallel walk; zero shared mutable state between workers except through the queue and atomic counters.
• Hardware-accelerated hashing: SHA-NI for SHA-256, AVX2 paths selected at runtime. ISA-neutral public headers — no intrinsics leak across the API boundary.
• Reproducible benchmarks vs sha256sum, openssl, hashdeep, jdupes, rmlint, czkawka_cli, mtree. Numbers, datasets, and commands committed under benchmarks/.
• CI matrix: debug, ASan + UBSan, TSan, release. TSan is load-bearing — every parallel path is validated under it.

Releases

Current release: v0.2.1 — Debian / Ubuntu .deb packages, GPG-signed (verify with GPG-KEY-bitcrafts.asc and SHA256SUMS.asc). Pick the x86-v3 variant for any Haswell-or-later CPU, or x86-v4 if your CPU supports AVX-512.

• bitcrafts-tools — the three CLI binaries bchash, bcduplicate, bcintegrity under /usr/bin/, statically linked. x86-v3 · x86-v4
• bitcrafts-dev — public headers, .pc files, and shared objects for the six bc-* libraries. x86-v3 · x86-v4
• bitcrafts-vigil — the GTK4 desktop frontend. Depends on bitcrafts-tools. See the BitCrafts Vigil page.

Build from source

Debian / Ubuntu 24.04+ baseline. liburing >= 2.6 is required (CI builds 2.7 from source if the distro ships an older version).

git clone https://github.com/Unmanaged-Bytes/bit-crafts.git
cd bit-crafts
scripts/install-deps.sh build       # compiler, meson, cmocka
scripts/bx build release            # build all libs + tools
sudo meson install -C build/release

The scripts/bx wrapper exposes named build variants — debug, release, asan, tsan, ubsan, bench, coverage — and a matrix command iterates the default set.

Origins

The libraries and tools used to live in separate repositories — one per project, each with its own CI, its own version, its own packaging. Coordinating cross-cutting changes across that DAG became the bottleneck. In 2026-05 the ecosystem was consolidated into a single Meson workspace (meson.override_dependency lets each subproject resolve siblings without pkg-config), one CI pipeline, one source of truth.

Historical articles below were written when each bc-* project was its own repo. Where they reference github.com/Unmanaged-Bytes/bc-X, the equivalent code now lives at github.com/Unmanaged-Bytes/bit-crafts/tree/main/subprojects/bc-X (or tools/bc-X for the CLI binaries).