Collegium:Imperium System - Revision history

AdminIsidore: /* Mission Plan */

2025-10-09T16:08:21Z

Mission Plan

← Older revision		Revision as of 16:08, 9 October 2025
Line 20:		Line 20:
	\| NFS (''nfs-kernel-server'', ''nfs-common''); configure ''/etc/exports'' on ''Torta'', mount on ''Roma''/''Horreum''		\| NFS (''nfs-kernel-server'', ''nfs-common''); configure ''/etc/exports'' on ''Torta'', mount on ''Roma''/''Horreum''
	\| Read/write access across nodes with static IPs, tested via file creation/listing on ''pomerium_via'' paths		\| Read/write access across nodes with static IPs, tested via file creation/listing on ''pomerium_via'' paths
	\| Completed ~~(October 09, 2025)~~		\| Completed
	\|-		\|-
	\| '''NFS-Plus GPU Dispatching'''		\| '''NFS-Plus GPU Dispatching'''
Line 26:		Line 26:
	\| NFS mounts, CUDA toolkit on ''Horreum'', SSH-based job dispatching (e.g., ''ssh'' or SLURM)		\| NFS mounts, CUDA toolkit on ''Horreum'', SSH-based job dispatching (e.g., ''ssh'' or SLURM)
	\| Run a sample GPU task (e.g., Python/CUDA script) from ''Roma'' using ''Horreum''’s GPU		\| Run a sample GPU task (e.g., Python/CUDA script) from ''Roma'' using ''Horreum''’s GPU
			\| Pending
	\|-		\|-
	\| '''Preparing Dockers and Directories on Latium and Torta'''		\| '''Preparing Dockers and Directories on Latium and Torta'''
Line 31:		Line 32:
	\| Docker, ''.bashrc'' modifications, directory scripts		\| Docker, ''.bashrc'' modifications, directory scripts
	\| Functional containers and directories, tested by mock commands in each context		\| Functional containers and directories, tested by mock commands in each context
			\| Pending
	\|-		\|-
	\| '''NFS-Plus Setup on Torta Hard Drives and Pomerium on Latium'''		\| '''NFS-Plus Setup on Torta Hard Drives and Pomerium on Latium'''
Line 36:		Line 38:
	\| NFS, WireGuard, ''ufw''		\| NFS, WireGuard, ''ufw''
	\| Read-only NFS access from ''Latium'' to ''Torta''’s smaller HDD, tested via mount and file read		\| Read-only NFS access from ''Latium'' to ''Torta''’s smaller HDD, tested via mount and file read
			\| Pending
	\|-		\|-
	\| '''Flamen Martialis and Salii Separation'''		\| '''Flamen Martialis and Salii Separation'''
Line 41:		Line 44:
	\| Python, NFS, SSH		\| Python, NFS, SSH
	\| ''Flamen Martialis'' collecting ''aqua_datum'' and ''Salii'' processing to ''grana_datum'', tested with a mock dataset		\| ''Flamen Martialis'' collecting ''aqua_datum'' and ''Salii'' processing to ''grana_datum'', tested with a mock dataset
			\| Pending
	\|-		\|-
	\| '''Simple Data Diodes'''		\| '''Simple Data Diodes'''
Line 46:		Line 50:
	\| RSYNC, ''ufw'', WireGuard		\| RSYNC, ''ufw'', WireGuard
	\| One-way ''aqua_datum'' push to ''/mnt/lacus'', tested by verifying no reverse access		\| One-way ''aqua_datum'' push to ''/mnt/lacus'', tested by verifying no reverse access
			\| Pending
	\|-		\|-
	\| '''RSYNC Optimization'''		\| '''RSYNC Optimization'''
Line 51:		Line 56:
	\| RSYNC, WireGuard, cron		\| RSYNC, WireGuard, cron
	\| Transfer mock JSON files in <1s, tested by comparing transfer times		\| Transfer mock JSON files in <1s, tested by comparing transfer times
			\| Pending
	\|-		\|-
	\| '''Tar + Netcat (nc) Implementation'''		\| '''Tar + Netcat (nc) Implementation'''
Line 56:		Line 62:
	\| Tar, Netcat, WireGuard		\| Tar, Netcat, WireGuard
	\| Functional burst transfer with a decision matrix, tested with mock data		\| Functional burst transfer with a decision matrix, tested with mock data
			\| Pending
	\|-		\|-
	\| '''Firejail/Bubblewrap Sandboxing'''		\| '''Firejail/Bubblewrap Sandboxing'''
Line 61:		Line 68:
	\| Firejail, Python		\| Firejail, Python
	\| Sandboxed mock script with restricted access, tested via confinement checks		\| Sandboxed mock script with restricted access, tested via confinement checks
			\| Pending
	\|-		\|-
	\| '''Supabase Integration'''		\| '''Supabase Integration'''
Line 66:		Line 74:
	\| Supabase client libraries, REST API, WireGuard		\| Supabase client libraries, REST API, WireGuard
	\| Validated data push/pull, tested with a mock schema		\| Validated data push/pull, tested with a mock schema
			\| Pending
	\|-		\|-
	\| '''JSONPlaceholder Data Pipeline Test'''		\| '''JSONPlaceholder Data Pipeline Test'''
Line 71:		Line 80:
	\| Python, RSYNC/nc, NFS, pywikibots		\| Python, RSYNC/nc, NFS, pywikibots
	\| Complete data cycle, tested by verifying output on ''OodaWiki''		\| Complete data cycle, tested by verifying output on ''OodaWiki''
			\| Pending
	\|-		\|-
	\| '''NOTAM Data Pipeline Test'''		\| '''NOTAM Data Pipeline Test'''
Line 76:		Line 86:
	\| Python, Supabase (optional), RSYNC/nc, NFS		\| Python, Supabase (optional), RSYNC/nc, NFS
	\| NOTAM ingestion to ''Roma'' SQL or ''OodaWiki'', tested by data accuracy		\| NOTAM ingestion to ''Roma'' SQL or ''OodaWiki'', tested by data accuracy
			\| Pending
	\|-		\|-
	\| '''RapidAPI via Supabase Test'''		\| '''RapidAPI via Supabase Test'''
Line 81:		Line 92:
	\| Supabase, Python, RSYNC/nc, pywikibots		\| Supabase, Python, RSYNC/nc, pywikibots
	\| API-to-Wiki flow, tested by published data on ''OodaWiki''		\| API-to-Wiki flow, tested by published data on ''OodaWiki''
			\| Pending
	\|-		\|-
	\| '''Automation/Standardized Deployment Script'''		\| '''Automation/Standardized Deployment Script'''
Line 86:		Line 98:
	\| Bash/Python, Docker, NFS, Supabase		\| Bash/Python, Docker, NFS, Supabase
	\| Script for project setup with one command, tested by deploying a mock project		\| Script for project setup with one command, tested by deploying a mock project
			\| Pending
	\|}		\|}

AdminIsidore: /* Mission Plan */

2025-10-09T16:06:49Z

Mission Plan

← Older revision		Revision as of 16:06, 9 October 2025
Line 14:		Line 14:
	! Tools/Dependencies		! Tools/Dependencies
	! Objectives		! Objectives
			! Status
	\|-		\|-
	\| '''NFS Setup on Roma, Horreum, and Torta'''		\| '''NFS Setup on Roma, Horreum, and Torta'''

AdminIsidore at 16:06, 9 October 2025

2025-10-09T16:06:20Z

← Older revision		Revision as of 16:06, 9 October 2025
Line 19:		Line 19:
	\| NFS (''nfs-kernel-server'', ''nfs-common''); configure ''/etc/exports'' on ''Torta'', mount on ''Roma''/''Horreum''		\| NFS (''nfs-kernel-server'', ''nfs-common''); configure ''/etc/exports'' on ''Torta'', mount on ''Roma''/''Horreum''
	\| Read/write access across nodes with static IPs, tested via file creation/listing on ''pomerium_via'' paths		\| Read/write access across nodes with static IPs, tested via file creation/listing on ''pomerium_via'' paths
			\| Completed (October 09, 2025)
	\|-		\|-
	\| '''NFS-Plus GPU Dispatching'''		\| '''NFS-Plus GPU Dispatching'''
Line 97:		Line 98:
	* Missions adhere to the Lingua standard, using Latin terms (e.g., ''pomerium_via'' for NFS paths, ''frumentarii_transfer'' for RSYNC jobs) to support script interoperability and quarterly audits for AI training.		* Missions adhere to the Lingua standard, using Latin terms (e.g., ''pomerium_via'' for NFS paths, ''frumentarii_transfer'' for RSYNC jobs) to support script interoperability and quarterly audits for AI training.
	* The plan prioritizes foundational infrastructure (NFS, GPU dispatching) before security mechanisms (data diodes, sandboxing) and pipeline tests, culminating in automation for scalability.		* The plan prioritizes foundational infrastructure (NFS, GPU dispatching) before security mechanisms (data diodes, sandboxing) and pipeline tests, culminating in automation for scalability.
			* Mission 1 (NFS Setup) was completed with fixed port configurations for NFS services and verified through multi-node file creation, concurrent writes, and cleanup tests, establishing unified access in Pomerium.

AdminIsidore at 22:24, 25 September 2025

2025-09-25T22:24:46Z

Show changes

AdminIsidore: Created page with "{{DISPLAYTITLE:Imperium System Mission Plan}} {{italic title}} == Overview == This document outlines the mission plan for building and testing the Imperium system, a distributed data processing pipeline. Each mission is designed to be executed in a separate thread, with phased OODA (Observe, Orient, Decide, Act) loops, and validated via independent tests upon completion. == Mission List == {| class="wikitable" |+ ! Mission Name ! Description ! Key Tools & Objectives |-..."

2025-09-25T22:14:58Z

Created page with "{{DISPLAYTITLE:Imperium System Mission Plan}} {{italic title}} == Overview == This document outlines the mission plan for building and testing the Imperium system, a distributed data processing pipeline. Each mission is designed to be executed in a separate thread, with phased OODA (Observe, Orient, Decide, Act) loops, and validated via independent tests upon completion. == Mission List == {| class="wikitable" |+ ! Mission Name ! Description ! Key Tools & Objectives |-..."

New page

{{DISPLAYTITLE:Imperium System Mission Plan}}
{{italic title}}
== Overview ==
This document outlines the mission plan for building and testing the Imperium system, a distributed data processing pipeline. Each mission is designed to be executed in a separate thread, with phased OODA (Observe, Orient, Decide, Act) loops, and validated via independent tests upon completion.

== Mission List ==
{| class="wikitable"
|+
! Mission Name
! Description
! Key Tools & Objectives
|-
! colspan="3" | Phase 1: Pomerium & Internal Network Foundation
|-
| '''NFS Setup on Roma, Horreum, and Torta'''
| Configure NFS mounts to unify Roma, Horreum, and Torta's "grana" drive (~698 GB) as a single logical system within the Pomerium. This enables seamless file sharing and supports the "single machine" goal.
|
* '''Tools:''' <code>nfs-kernel-server</code>, <code>nfs-common</code>
* '''Objective:''' Full read/write access to the "grana" share on Torta from both Roma and Horreum, verified via file creation and listing.
|-
| '''NFS-Plus GPU Dispatching'''
| Extend the NFS setup to enable scripts on Roma or Torta to dispatch GPU-intensive tasks to Horreum’s NVIDIA RTX 5060 Ti, using the shared filesystem for data access.
|
* '''Tools:''' NFS, CUDA Toolkit, SSH
* '''Objective:''' Successfully run a sample GPU task (e.g., a small AI inference) initiated from Roma that reads/writes data on the NFS share and executes on Horreum's GPU.
|-
! colspan="3" | Phase 2: Aquaeductus & External Pipeline Foundation
|-
| '''Preparing Dockers and Directories on Latium and Torta'''
| Set up the Docker containers (Pomerium, Campus Martius, Flamen Martialis) on Latium and the required directory structure on Torta's "aqua" drive (`/mnt/aqua/aqua_datum_raw`) for pipeline operations.
|
* '''Tools:''' Docker, <code>.bashrc</code>, Shell scripts
* '''Objective:''' Functional containers and directories, tested by mock commands in each context.
|-
| '''Simple Data Diodes & RSYNC Optimization'''
| Establish a fast, secure, one-way data flow (the Aquaeductus) from Latium to Torta using RSYNC over the WireGuard tunnel. This replaces SCP to avoid bottlenecks.
|
* '''Tools:''' RSYNC, WireGuard, <code>cron</code>
* '''Objective:''' A one-way push of `aqua_datum` to Torta's "aqua" drive, tested by verifying transfer speed and the inability to initiate a connection from Torta back to the originating service on Latium.
|-
| '''Firejail/Bubblewrap Sandboxing'''
| Deploy Firejail on Latium to sandbox the `Flamen Martialis` script, ensuring secure processing of external `aqua_datum` by restricting its filesystem and network access.
|
* '''Tools:''' Firejail, Python
* '''Objective:''' A sandboxed mock script with verifiably restricted access to the host system.
|-
! colspan="3" | Phase 3: End-to-End Pipeline Integration & Testing
|-
| '''Flamen Martialis and Salii Separation'''
| Implement the full Flamen/Salii workflow: Flamen on Latium collects and sanitizes `aqua_datum`, which is then transferred to Torta. Salii on Roma detects the new data and orchestrates internal processing.
|
* '''Tools:''' Python, RSYNC, NFS, SSH
* '''Objective:''' Flamen Martialis successfully collects `aqua_datum`, and Salii on Roma successfully processes it to `grana_datum` on the NFS share.
|-
| '''JSONPlaceholder Data Pipeline Test'''
| Test the full, simple pipeline using JSONPlaceholder’s mock API, simulating the data flow from Latium -> Torta -> Roma -> OodaWiki. This validates the entire end-to-end architecture.
|
* '''Tools:''' Python, RSYNC, NFS, pywikibots
* '''Objective:''' A complete data cycle, verified by seeing the mock data correctly published on an OodaWiki page.
|-
| '''NOTAM Data Pipeline Test'''
| Test the pipeline with a more complex, authenticated source using the FAA's NOTAM API. This focuses on scheduled pulls and performance with real-world data.
|
* '''Tools:''' Python, RSYNC, NFS
* '''Objective:''' Successful and reliable ingestion of NOTAM data, stored as `grana_datum` on the NFS share.
|-
! colspan="3" | Phase 4: Optimization & Future Development
|-
| '''Tar + Netcat (nc) Implementation'''
| Implement and benchmark `tar` + `nc` for large, one-time "burst" transfers and compare its performance to RSYNC to establish a decision matrix for future pipeline tool selection.
|
* '''Tools:''' Tar, Netcat, WireGuard
* '''Objective:''' A functional burst transfer and a documented decision matrix for when to use RSYNC vs. `tar` + `nc`.
|-
| '''Supabase Integration'''
| Integrate Supabase as an optional, advanced filtering buffer for `aqua_datum`, using its edge functions and Row-Level Security (RLS) to validate data before it enters the Aquaeductus.
|
* '''Tools:''' Supabase client libraries, REST APIs
* '''Objective:''' A validated data push/pull through Supabase, tested with a mock schema and RLS policy.
|-
| '''Automation/Standardized Deployment Script'''
| Develop a `creo_castellum.sh` CLI script to automate the setup of new data pipelines across Imperium, based on the lessons learned from the manual builds.
|
* '''Tools:''' Bash/Python, Docker, NFS
* '''Objective:''' A script that can provision a new data pipeline with a single command, tested by deploying a new mock project.
|}