Open-source engineering

Find the invariant. Find the code. Prove the change.

This book is a feature-first map of the Release 1.0 implementation. It groups first-party entry points by behavior, names the functions to read first and makes the current activation and qualification boundaries explicit.

Rust source files

REST operations

338/338

Recorded query/result tests

Browse every cataloged file and function

01 · system map

Six planes, one immutable answer boundary

Control plane

Dataset namespaces, upload metadata, idempotent compilation jobs, artifact roots, qualification and atomic publication.

services/api · ngkg-catalog · operators

Compilation plane

Bounded TriG scanning, semantic projection, deterministic dictionaries, partition reduction and Arrow/Parquet generation.

distributed-worker · ngkg-distributed-build · ngkg-projection

Semantic plane

OWL 2 DL qualification, pinned imports, HermiT exact reasoning, proofs and completeness certificates.

ngkg-owl-direct · ngkg-online-reasoning · HermiT adapter

Serving plane

Authorized immutable snapshots, SPARQL Protocol, c8 query evidence, query routing, locator lookup and hydration.

online-serving · ngkg-sparql-compiler · ngkg-reference

Execution plane

Typed algebra/path DAGs, stable partition ownership, Arrow exchange, joins, frontiers, checkpoints and spill.

ngkg-query-planner · ngkg-query-executor · ngkg-grace-join

Evidence plane

Checksums, snapshot and graph-set hashes, manifests, proof support and differential equality against the scalar oracle.

ngkg-types · contracts · reference qualification

Authorized named graphs

Immutable qualified snapshot

Complete answer + evidence

A request is admitted for a tenant, restricted to authorized graph IRIs, bound to one active snapshot, planned and executed under explicit ceilings, then returned only when its result and evidence are complete. Unknown semantic coverage routes to exact reasoning; it is never reinterpreted as false.

02 · implementation lanes

Rust owns the runtime; Java owns exact OWL; C++ remains planned

95 source files

Rust · implemented

Services, CRD controllers, deterministic plans, bounded Tokio/blocking lanes, Arrow/Parquet, Grace joins, mmap locators, NVMe spill, caching and integrity validation.

HermiT adapter

Java · implemented

OWLAPI loads the pinned import closure; HermiT validates global consistency and answers exact OWL 2 Direct-Semantics entailment requests through a checksum-bound process boundary.

0 first-party files

C++ · planned

Phase 42 is the documented acceleration lane for measured sparse kernels such as CSR traversal, bitmap intersection and SIMD decode. It is not active in this candidate.

Why this division matters: Rust supplies memory safety and predictable bounded concurrency for the always-on database path. HermiT supplies standards-grounded logical completeness. A future C++ kernel must beat the Rust reference on a measured workload and pass byte-independent differential equality before it can enter the trusted path.

03 · design law

Invariants every contribution must preserve

Authorization precedes assembly

Never construct a query or reasoner dataset before tenant admission and graph authorization. Exact OWL input is authorized */semkg plus pinned imports only.

Identity is content-bound

Snapshot, graph-set, request, partition and artifact identities are deterministic and checksum-bound. Do not introduce mutable aliases into trusted execution.

Completeness is explicit

Missing or duplicate partitions, bad checksums, partial workers and exceeded ceilings fail closed. Success cannot mean best effort.

Optimization follows equality

The scalar evaluator is the correctness oracle. A distributed operator becomes normal-path eligible only after differential equality passes for its supported domain.

04 · contribution loop

Build the smallest proof-bearing change

01LocateStart with the alphabetical feature index below, then trace its types and JSON contracts before changing an implementation.
02State the invariantWrite down the authorization, integrity, completeness and resource-ceiling behavior your change must retain.
03Add the failing testPrefer a minimal scalar/reference case, a bounded adversarial case and a deterministic retry or duplicate-delivery case.
04ImplementKeep transport, planning, execution and evidence layers separate. Avoid a convenience path that bypasses catalog truth or validation.
05Differentially verifyFor query optimizations, compare canonical results with the scalar oracle. For artifacts, compare independently computed roots.
06Run native gatesFormat, compile, lint, test, package the adapter, lint/render Helm and prove the locked offline build.

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features
mvn --batch-mode --no-transfer-progress \
  -f adapters/hermit-reasoner/pom.xml clean test package
helm lint charts/ngkg-crds
helm lint charts/ngkg-platform
helm lint charts/ngkg-workloads
python3 scripts/structural_validate.py --root .

If those tools are unavailable, report the gate as unexecuted. Source inspection is useful review evidence, but it is not a substitute for native compilation or runtime qualification.

05 · next milestones

High-value work that is still open

Activate distributed algebra

Exchange Arrow partitions between pods, execute scalar-oracle stages where needed, require complete sets and gate each operator by equality.

Activate property paths

Connect NFA frontiers to snapshot adjacency indexes, fragment pods and atomic checkpoints; resume safely and feed bindings back to SPARQL.

Complete protocol surface

Implement secured SERVICE/SERVICE SILENT, dataset parameters, remaining result negotiation, SSRF controls and qualified service description.

Qualify multinode behavior

Prove scaling, skew, hot vertices, NVMe spill, worker death, duplicate delivery, drain, upgrade, rollback, backup, restore and soak behavior.

Design large-source ingestion

The direct raw upload is bounded at 100 GiB. Version-bound cloud-object registration, resumable multipart upload and sharded source manifests for hundreds of GiB are proposals, not current behavior.

Finish release evidence

Run W3C/Jena differential suites, security and tenant-isolation tests, SBOM/license/CVE scans and signed reproducible offline builds.

06 · ordered delivery

Phase 40.13.11 to c8 1.0.0

The sequence is dependency-ordered: make cloud objects deterministic compilation inputs, build and qualify the semantic snapshot, activate distributed runtimes, then prove storage, scaling, security, standards and release behavior.

Phase 40.13.11

Cloud-Source Compiler Handoff

01/16

Convert frozen cloud manifests into syntax-aware distributed RDF decode and partition plans; existing bucket TriG becomes an automatic compilation input without HTTP upload.

Phase 40.13.12

Distributed Semantic Compilation

02/16

Build GUID dictionaries, graph partitions, Parquet payloads, adjacency structures, semantic indexes and union-default metadata across nodes; large datasets become efficiently queryable artifacts.

Phase 40.13.13

OWL 2 DL Snapshot Qualification

03/16

Assemble authorized asserted */semkg graphs with pinned imports and validate profile, datatypes and consistency; each snapshot receives deterministic semantic identity and qualification evidence.

Phase 40.13.14

Distributed Offline Reasoning

04/16

Generate certified finite closure, semantic extents, equality components, property-chain consequences and proof support with distributed workers plus exact HermiT verification.

Phase 40.13.15

Atomic Publication and Query Activation

05/16

Certify every compiler and reasoner partition before atomically publishing; cloud-hosted TriG becomes visible through ordinary /sparql and /query routes without partial exposure.

Phase 40.13.16

Complete Distributed SPARQL Runtime

06/16

Enable distributed OPTIONAL, UNION, MINUS, aggregation, sorting, DISTINCT, subqueries, modifiers and all query forms while preserving exact equality with the scalar oracle.

Phase 40.13.17

Distributed Property-Path Runtime

07/16

Connect adjacency indexes to frontiers, hot-vertex splitting, spill, checkpoint recovery and termination detection for massive graph traversal across cores and nodes.

Phase 40.13.18

Secured Federation and SPARQL Protocol

08/16

Implement controlled SERVICE/SERVICE SILENT, variable endpoints, protocol datasets, result negotiation and service description for secured federated RDF environments.

Phase 40.13.19

Multinode Storage and Recovery

09/16

Qualify replication, relocation, node loss, retries, duplicate delivery, checksum failure, backup and restore so snapshots remain deterministic and recoverable.

Phase 40.13.20

Production Autoscaling Qualification

10/16

Exercise Kueue, HPA/KEDA, cluster autoscaling, scale-from-zero pools, workload isolation and cgroup-aware budgets without changing results.

Phase 40.13.21

Enterprise Security and Operations

11/16

Complete workload identity, tenant isolation, encryption, network policy, audit trails, rate limits, observability, SLOs and disaster procedures.

Phase 40.13.22

Standards and Differential Qualification

12/16

Pass applicable SPARQL, TriG, OWL Direct-Semantics, result-format, federation and failure suites against HermiT and Apache Jena.

Phase 40.13.23

Performance and Capacity Qualification

13/16

Benchmark ingestion, traversal, reasoning, concurrency, latency and cost against Apache Jena on representative enterprise datasets.

Phase 40.13.24

Kubernetes Release Qualification

14/16

Complete multinode soak, chaos, upgrade, rollback, backup, restore, Helm, image, SBOM, CVE, license and reproducible-build gates.

Phase 1.0.0-RC1

Final Release Candidate

15/16

Freeze APIs, schemas, images, Helm charts, documentation, checksums and qualification reports for final acceptance testing.

Phase 1.0.0

General Availability

16/16

Resolve RC defects and publish signed, reproducible artifacts for the production autoscaling RDF/OWL database—with no ontology-alignment or raw-data-mapping claim.

07 · distributed skew

Hot-vertex splitting across worker pods

Yes—this is a blended distributed-software and Kubernetes multi-node feature. A hot vertex is an RDF node with an unusually large adjacency list. If USA has 50 million inbound locatedIn edges, assigning the vertex to one partition can exhaust one worker while the rest of the cluster sits idle.

Deterministic split · complete merge

Hot vertex

millions of edges

Worker pod A

edges 0–999,999

Worker pod B

edges 1M–1,999,999

Worker pod C

edges 2M–end

Completion barrier

verify every split
merge + deduplicate
advance exact frontier

Execution sequence

01Detect that degree exceeds the configured hot-vertex threshold.
02Canonically sort or partition the vertex adjacency edges.
03Create deterministic edge ranges with stable work identities.
04Dispatch ranges to worker pods, potentially on different nodes.
05Expand the property-path automaton independently per range.
06Require every split result with the expected identity and checksum.
07Merge and deduplicate endpoints into the global visited set.
08Advance only after the completion barrier proves the set is whole.

Who owns what

Layer	Responsibility	Capacity signal
c8 software	Detect skew, create deterministic splits, preserve SPARQL semantics, checkpoint, merge, deduplicate and fail closed.	Split count · frontier work
Rust / HPC runtime	Process bounded edge ranges with CPU lanes, Arrow batches, partitioned adjacency indexes and NVMe spill.	CPU · memory · spill bytes
Kubernetes	Schedule split workers across nodes while enforcing anti-affinity, resource guarantees and disruption policy.	Pending pods · node fit
Kueue / HPA / KEDA / autoscaler	Add workload and node capacity as pending split work, frontier size, pressure or queue delay grows.	Queue depth · delay · utilization

Kubernetes never understands the graph vertex. c8 creates deterministic graph work units and enforces semantic completion; Kubernetes supplies and scales the physical compute that runs them.

Where the pattern matters

property pathstransitive traversalhigh-degree joinsclosure materializationfrontier expansionskewed graph partitions

08 · source index

API & protocols

Public dataset, source, ingestion, snapshot, SPARQL, reasoning and physical-retrieval surfaces.

services/api/src/main.rs

Control-plane HTTP service for datasets, raw TriG uploads, asynchronous compilation jobs and snapshot publication.

create_datasetupload_trig_sourceinspect_trigcreate_ingestionget_jobcancel_jobget_snapshotpublish_snapshot

services/online-serving/src/main.rs

Serving-plane routes for SPARQL Protocol, c8 JSON queries, Direct-Semantics validation/routing, locator lookup and hydration.

sparql_getsparql_postsparql_service_descriptionexecute_sparql_protocolquerylocatehydrate

This candidate exposes 32 explicit REST operations plus two Swagger asset handlers. Ingestion is polled through the jobs route; there are no callbacks or webhooks.

09 · source index

Artifacts & snapshots

Checksum-bound object-store materialization, immutable snapshot identity and compare-and-swap publication.

crates/ngkg-artifact-store/src/lib.rs

Fetches and publishes immutable artifacts while enforcing declared digests and resource budgets.

from_base_urlmaterialize_verifiedmaterialize_verified_with_budgetput_file_immutable

crates/ngkg-snapshot/src/lib.rs

Validates serving manifests and activates qualified snapshots without mutating prior versions.

validate_manifestpublish

crates/ngkg-distributed-artifacts/src/lib.rs

Builds and finalizes partitioned Arrow/Parquet and catalog artifacts, then compares independent roots.

materialize_artifact_partitionfinalize_artifact_partitionsfinalize_catalog_artifact_partitionscompare_artifact_roots

10 · source index

Catalog & identity

Durable control-plane truth, deterministic entity identity and collision-safe dictionary construction.

crates/ngkg-catalog/src/lib.rs

Owns dataset, compilation, distributed-plan, work-completion, certification and active-snapshot records in PostgreSQL.

create_datasetcreate_or_get_compilationregister_distributed_plancommit_distributed_workcommit_serving_rootget_active_serving_snapshotpublish_snapshot

crates/ngkg-identity/src/lib.rs

Derives stable GUIDs and fact identities from canonical IRIs and merges deterministic dictionary runs.

guid_for_canonical_iriskolem_irifact_identityverify_fact_collisionmerge_dictionary_runs

11 · source index

Dataset & authorization

Named-graph catalogs, graph roles, visibility labels and authorized active-dataset assembly.

crates/ngkg-dataset/src/lib.rs

Compiles graph catalogs, resolves service datasets and restricts them to admitted roles before planning or reasoning.

compile_catalogresolve_datasetvalidate_resolved_datasetrestrict_resolved_dataset_to_rolesvalid_authorization_label

services/online-serving/src/main.rs

Applies tenant admission and graph authorization before computing the active graph-set and dataset hashes.

authorize_requestresolve_authorized_datasetvalidate_active_snapshot

12 · source index

Distributed build

Bounded, restartable TriG scanning and partition work for compilation and physical artifact creation.

crates/ngkg-distributed-build/src/lib.rs

Scans TriG under ceilings, projects stable partitions, reduces runs and compares independently computed roots.

safe_scan_trigproject_partitionreduce_projection_runsfinalize_reducerscompare_roots

services/distributed-worker/src/main.rs

Worker binary that executes locator, artifact, scan, projection, reduce, finalize and comparison work items.

compile_locatormaterialize_artifactssafe_scanprojectreducefinalizecompare

services/distributed-worker/src/object_stage.rs

Plans and executes object-store-backed stages through artifact and serving-root preparation.

planprojectreducefinalizeprepare_artifactsmaterialize_artifact_object_storeprepare_serving_root_object_store

13 · source index

HPC, joins & spill

Whole-core budgets, bounded Rust concurrency, Grace hash joins and NVMe-backed spill accounting.

crates/ngkg-hpc-runtime/src/lib.rs

Validates runtime thread budgets and reports node capabilities against workload saturation targets.

node_saturation_targetvalidate_thread_budgetcapability_report

crates/ngkg-grace-join/src/lib.rs

Runs bounded Grace hash joins with streaming inputs and observable spill usage.

openjoinjoin_streamactive_spill_bytes

crates/ngkg-shuffle-cache/src/lib.rs

Caches checksum-addressed shuffle objects with bounded usage and invalidation.

digestopengetinsertinvalidateusage

14 · source index

Hydration & retrieval

Turns semantic GUID results into bounded physical Parquet reads without weakening graph authorization.

crates/ngkg-locator/src/lib.rs

Compiles and queries sharded GUID-to-Parquet locator indexes and coalesces safe byte ranges.

compile_sharded_locatoropenlookupvalidateowner_forcoalesce_ranges

crates/ngkg-hydration/src/lib.rs

Plans and performs graph-filtered payload hydration after verifying every referenced shard.

verify_payload_shardplan_hydrationhydrate_sharded_payloadhydrate_sharded_payload_for_graphstruncate_optional_context

crates/ngkg-index/src/lib.rs

Opens physical indexes only after integrity checks succeed.

verify_open

15 · source index

Kubernetes & operators

CRD state machines, phase barriers, dedicated worker pools and workload ownership for RKE2/Kubernetes.

crates/ngkg-kube/src/lib.rs

Defines the namespaced ngkg.io/v1alpha1 NgkgCompilation custom resource shared by the API and both controllers.

NgkgCompilationNgkgCompilationSpecNgkgCompilationStatus

crates/ngkg-operator-core/src/lib.rs

Pure reconciliation decisions and Phase 40 resource ceilings used by the controllers.

next_actionPhase40Ceilings

services/distributed-operator/src/main.rs

Schedules distributed stages, enforces completion barriers and reconciles worker-job state.

mainreconcilereconcile_artifact_barrierschedule_stageensure_jobpatch_status

services/operator/src/main.rs

Coordinates reference compilation and top-level compilation lifecycle.

mainreconcileensure_reference_jobreference_jobpatch_status

charts/ngkg-workloads/

Deploys dedicated query, fragment, locator, hydration and reasoner roles with HPAs, affinity and policy.

values.schema.jsontemplates/HPAsNetworkPolicies

16 · source index

OWL & exact reasoning

Direct-Semantics legality, authorized ontology assembly, HermiT execution, proof binding and complete partition merging.

crates/ngkg-owl-direct/src/lib.rs

Extracts candidate OWL Direct-Semantics BGP templates and classifies legal graph patterns.

extract_direct_bgp_templateclassify_direct_bgps

crates/ngkg-online-reasoning/src/lib.rs

Selects authorized semkg modules, builds reasoner partitions, retries dispatch and rejects incomplete result sets.

select_authorized_asserted_modulesvalidate_ontology_snapshot_bindingbuild_distributed_reasoner_plandispatch_exact_partitions_with_retrycomplete_distributed_exact_bgp

crates/ngkg-direct-reasoner/src/lib.rs

Builds, executes and merges exact Direct-BGP partition requests with deterministic request-set identity.

execute_exact_direct_bgpprepare_exact_direct_bgp_requestsexecute_exact_direct_partitionmerge_partition_resultsdirect_exact_request_set_sha256

crates/ngkg-reasoner-client/src/lib.rs

Validates legality, result, certificate and proof bindings before accepting reasoner output.

require_legal_direct_bgpreasondecide_coverage

crates/ngkg-reference/src/direct_exact.rs

Assembles the active exact ontology only from authorized */semkg graphs and pinned imports.

build_direct_active_ontology_bundle

crates/ngkg-reference/src/reasoner.rs

Invokes the checksum-pinned HermiT adapter under bounded execution rules.

invoke_reasoner

adapters/hermit-reasoner/src/main/java/io/ngkg/reasoner/DirectBgpExecutor.java

Executes OWLAPI/HermiT Direct-BGP requests and emits bounded exact evidence.

executerenderResultsrenderProofSupport

17 · source index

Property paths

Exact NFA traversal state, origin-preserving frontiers, partition ownership, checkpoints and termination.

crates/ngkg-query-executor/src/distributed_path.rs

Seeds, partitions and expands path frontiers; validates complete iterations and atomic resume checkpoints.

seed_path_frontierpath_partition_ownerpath_expansion_work_itemsexpand_path_work_itemcomplete_path_iterationbuild_path_checkpointvalidate_path_checkpoint

The kernel and contracts are present. Live adjacency retrieval and endpoint activation still require Phase 40.13.9 integration and differential qualification.

crates/ngkg-sparql-compiler/src/lib.rs

Compiles SPARQL path expressions into exact distributed property-path plans.

distributed_property_path_plans

18 · source index

Reference oracle & qualification

Scalar correctness path, semantic compilation, expected-result verification and differential gates.

crates/ngkg-reference/src/query.rs

Builds the standards-correct scalar store, executes compiled queries and canonicalizes results for comparison.

build_storebuild_store_with_dataset_policyexecute_compiled_querycanonical_result_hashverify_bindingsload_rdf_fixture

crates/ngkg-reference/src/compiler.rs

Compiles source manifests into semantic catalogs and certifies representative query behavior.

compile_from_manifestvalidate_graph_catalogwrite_semantic_capabilitiescertify_queries

crates/ngkg-reference/src/rdf.rs

Parses TriG/N-Quads, normalizes facts and enforces the RDF/datatype policy.

parse_trigparse_nquadsnormalized_factsnquad_linentriple_line

services/reference-worker/src/object_compile.rs

Runs object-store-backed reference attempts, uploads artifacts and certifies sharded hydration.

compile_object_storerun_attemptcertify_sharded_hydrationupload_snapshot

19 · source index

SPARQL & distributed algebra

Parsing, canonical algebra, graph fragments, bounded partition operators and scalar differential comparison.

crates/ngkg-sparql-compiler/src/lib.rs

Parses SPARQL, applies base IRIs, requires certifiable constructs and emits canonical or distributed plans.

parseparse_with_base_irirequire_certifiablecanonical_ssedistributed_graph_fragmentsdistributed_algebra_plan

crates/ngkg-query-planner/src/lib.rs

Validates typed distributed plans, computes execution waves, routes relevant graphs and chooses exchanges.

validate_distributed_algebra_planalgebra_execution_wavesroute_relevant_graphschoose_exchange

crates/ngkg-query-executor/src/lib.rs

Reads/writes Arrow streams and implements stable partition ownership, exact semijoins, joins, projection and frontier utilities.

read_arrow_streamwrite_arrow_streampartition_rowsexact_semijoininner_joinproject_sparql_jsonnext_frontier

crates/ngkg-query-executor/src/distributed_algebra.rs

Implements native UNION, OPTIONAL/left join, MINUS, DISTINCT, grouping, ordered merge and global slicing under complete-partition barriers.

execute_native_algebra_taskunion_sparql_jsonleft_join_sparql_jsonminus_sparql_jsondistinct_sparql_jsongroup_owned_partitionsmerge_ordered_partitions_byglobal_slice_sparql_json

The normal endpoint retains the scalar correctness path until every optimized operator is transported to workers and passes its differential gate.

services/online-serving/src/main.rs

Coordinates protocol parsing, scalar execution, exact reasoning and the distributed evidence lane.

execute_sparql_protocolexecute_distributed_queryexecute_fragmentexecute_partitioned_shuffleexecute_uncertified_exact_query

crates/ngkg-query-cache/src/lib.rs

Stores deterministic query results under snapshot-, graph-set- and plan-bound cache keys.

digestopengetinsertinvalidateusage

20 · source index

Types, contracts & integrity

Typed requests, plans, certificates, proofs and manifests that bind every result to inputs and completeness evidence.

crates/ngkg-types/src/direct_bgp.rs

Defines and validates normalized Direct-BGP result envelopes.

validate_result

crates/ngkg-types/src/direct_certificate.rs

Validates exact certificates, result hashes and snapshot/graph bindings.

validate_certificatevalidate_result_hash

crates/ngkg-types/src/direct_exact.rs

Defines exact requests and partition results with integrity validation.

validate_exact_requestvalidate_partition_result

crates/ngkg-types/src/direct_proof.rs

Builds support identities and validates proof manifests and bundles.

support_idsupport_hashvalidate_manifestvalidate_bundle

contracts/

Versioned JSON Schemas for graph catalogs, distributed plans, snapshots, Direct-BGP evidence and serving roots.

graph-catalogdirect-bgpdirect-proofdistributed-*serving-root