Data RESTorage technical overview

^ Architectural Overview & Feature Whitepaper

Data RESTorage (DRS) is a lightweight, zero-overhead hybrid document and relational backend engine designed to eliminate the complexity, resource overhead, and unpredictable query performance of traditional NoSQL document databases and heavy ORM stacks.

By replacing abstract query languages and dynamic query optimizers with deterministic, sculptured asynchronous modules (TypeScript/JavaScript) and introducing hybrid Deep Indexing, DRS provides predictable sub-millisecond execution, complete data integrity, and zero-friction development.

^ Core Architectural Breakthroughs

                      ┌────────────────────────────────────────────────────────┐
                      │    REST Client (GET / POST - JSON / XML Payloads)      │
                      └───────────────────────────┬────────────────────────────┘
                                                  │
                                                  ▼
                      ┌────────────────────────────────────────────────────────┐
                      │   Cascading Autoexec Middleware (Dataset/Schema/Entity)│
                      └───────────────────────────┬────────────────────────────┘
                                                  │
                                                  ▼
                      ┌────────────────────────────────────────────────────────┐
                      │   Target Operation Module: op-*.ts / op-*.mjs           │
                      │   (Deterministic Business Logic & Execution Flow)      │
                      └───────────────────────────┬────────────────────────────┘
                                                  │
                        ┌─────────────────────────┴──────────────────────────┐
                        ▼                                                    ▼
          [ 1. Complete Tree Document ]                         [ 2. Targeted Deep Indexing ]
          Complex JSON document storage                          Relational SQL Index Table
          (Leverages OS Kernel Page Cache)                     (MariaDB/PostgreSQL Projections)
                        │                                                    │
                        ▼                                                    ▼
              `drs.set(docId, json)`                               `drs.sql(query, values)`
             Fast O(1) Key/File Access                          Flat, strongly-typed indices
                        │                                                    │
                        └─────────────────────────┬──────────────────────────┘
                                                  │
                                                  ▼
                                           `drs.commit()`
                       Unified Atomic Transaction (Filesystem + Relational DB)

^ 1. Eliminating both the Query Language overhead and the Query Optimizer task

Traditional database engines (both SQL and NoSQL engines like MongoDB) rely on complex Query Optimizers that translate declarative queries (SQL, MQL, GraphQL) into physical execution plans.

^ The Problem with Query Optimizers

• Plan Regression & Imprevedibility: Optimizers rely on probabilistic statistics and data sampling. When datasets scale or cardinality shifts, the optimizer may abruptly choose a suboptimal execution plan, turning a 2 ms query into a multi-second bottleneck without any code changes.
• Continuous Computational Overhead: Every incoming query incurs parsing, Abstract Syntax Tree (AST) compilation, algebraic rewriting, and plan evaluation before a single byte of data is read.
• The ORM Problem: To bridge application code and database queries, developers introduce Object-Relational Mappers (ORMs), which add latency, memory bloat, and notorious N+1 query problems.

^ The DRS Solution: Explicit Procedural Determinism

• No Query Language Overhead: Endpoints are mapped directly to async JavaScript/TypeScript functions (op-*.ts / op-*.mjs).
• Deterministic O(1) and O(log N) Access: The developer defines the exact data access path. Execution complexity is explicit, transparent, and completely predictable.
• Modern Stored Procedures: Combines the performance, encapsulation, and colocation of classic Stored Procedures with standard, modern TypeScript/JavaScript, complete with Git versioning, zero-downtime hot reloading, and native debugging.

^ 2. Deep Indexing & Hybrid Relational Projection

Querying deeply nested, tree-structured JSON data is one of the most resource-intensive tasks in document-oriented databases.

^ The Pitfalls of MongoDB & Document Stores

• Index Bloat & RAM Saturation: Indexing deeply nested arrays or paths in MongoDB requires Multikey or Wildcard indexes. These indexes rapidly explode in size, consuming disproportionate RAM in the storage engine (WiredTiger) and causing severe write amplification.
• Rigid Indexing Constraints: Document stores can only index raw document attributes. Creating indexes on derived, transformed, or business-calculated values requires complex, asynchronous Aggregation Pipelines or Change Streams.
• Costly Traversals: Filtering on nested structures requires expensive operators ($unwind, $elemMatch, $lookup), leading to high CPU usage and in-memory aggregation limits.

^ The DRS Solution: Targeted Relational Projection

• Separation of Storage and Search:
  • Complete Document: The rich, nested JSON tree is stored cleanly on disk, taking full advantage of the OS kernel's native page cache with zero memory overhead.
  • Compact SQL Projections: Dedicated operations extract specific XPath/JSONPath expressions, nested leaves, or computed keys into lightweight, strongly-typed relational tables (MariaDB or PostgreSQL).
• Power of SQL JOINs: Searches are executed via high-speed relational queries (with standard JOIN, WHERE, ORDER BY, and pagination) returning only matching document IDs (doc_id).
• Instant Retrieval: Matching documents are retrieved via direct O(1) key lookups
  (drs.get(id)).

^ 3. Unified Atomic Transactions & Rollback

Coordinating writes across disk and relational databases usually requires heavy distributed transaction managers (2PC/XA).

• Single Transaction Pipeline: Operations stage changes via drs.set() (filesystem) and drs.sql() / drs.pgInsert() (database).
• All-or-Nothing Commit: Invocations of drs.commit() atomically apply all pending file mutations and SQL transactions.
• Automatic Rollback: If any database constraint fails or an unhandled exception is thrown, all pending file writes and SQL operations are immediately rolled back, preventing state corruption.
• Built-in Dry Run Mode: Endpoints natively support non-destructive simulation (dry_run: true), rolling back all changes automatically while returning the full execution outcome.

^ 4. Zero-Friction Developer Experience

• Lightest learning curve: there is no proprietary Query language. JavaScript and/or TypeScript are used, together with SQL.
• Zero-Downtime Hot Reloading: Operations and dataset configurations are managed by HotPluggableDir. Edits to .ts or .mjs files take effect immediately on the next request without server restarts.
• Native TypeScript Support (Zero-Config): Leverages Node.js native type stripping (Node 22/23+). No compilation pipelines (tsc), bundlers, or third-party loaders (ts-node, tsx) required.
• Cascading autoexec Middleware: Middleware files (autoexec.ts / autoexec.mjs) execute in cascade at Dataset, Schema, and Entity levels, enabling DRY shared logic, context mutation, and early security guard exits.
• Dual Transport Formats (JSON & XML): Seamless automatic deserialization and serialization for both JSON and XML payloads in request bodies and responses.

^ Feature Comparison Matrix

^ Summary

Data RESTorage bridges the gap between unstructured document flexibility and high-performance relational search.
By eliminating the query planner, isolating document storage from index projection, and running native TypeScript/JavaScript directly on the incoming REST path, DRS provides unmatched execution predictability, minimal infrastructure footprint, and maximum development speed.