[bss-openc5isr] 05 Operations & Administration Operations, monitoring, backup, recovery, identity, security and troubleshooting. Edition: BSS — OpenC5ISR (openc5isr.bss.dev). Start Here Start Here Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Documentation structure and release scan Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Purpose This book provides a safe operational baseline for the named distribution. Replace placeholders with tested unit names, URLs, owners, thresholds and escalation paths. Contents Section What it contains Service Control Start, stop, restart and inspect the deployed services. Monitoring & Assurance Health checks, logs, audit and daily operational controls. Continuity & Recovery Backups, recovery and structured troubleshooting. How to maintain this book Generated sections are replaced from release evidence on each run. Put reviewed corrections, deployment-specific facts, links and decisions in the Maintainer Notes area below the generated block. Mark pages as reviewed only after testing them against the named distribution and release. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Service Control Managed documentation for Service Control. Starting, Stopping and Restarting Starting, Stopping and Restarting Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Observed service units plus operational control pattern Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Observed service candidates Unit Status command Log command opencybersec-agent.service sudo systemctl status opencybersec-agent.service --no-pager sudo journalctl -u opencybersec-agent.service -n 200 --no-pager openlvc-ais-bridge.service sudo systemctl status openlvc-ais-bridge.service --no-pager sudo journalctl -u openlvc-ais-bridge.service -n 200 --no-pager openlvc-dis-bridge.service sudo systemctl status openlvc-dis-bridge.service --no-pager sudo journalctl -u openlvc-dis-bridge.service -n 200 --no-pager openlvc-flightgear-bridge.service sudo systemctl status openlvc-flightgear-bridge.service --no-pager sudo journalctl -u openlvc-flightgear-bridge.service -n 200 --no-pager openlvc-rti-gateway.service sudo systemctl status openlvc-rti-gateway.service --no-pager sudo journalctl -u openlvc-rti-gateway.service -n 200 --no-pager openpnt-phc2sys@.service sudo systemctl status openpnt-phc2sys@.service --no-pager sudo journalctl -u openpnt-phc2sys@.service -n 200 --no-pager openpnt-ptp4l@.service sudo systemctl status openpnt-ptp4l@.service --no-pager sudo journalctl -u openpnt-ptp4l@.service -n 200 --no-pager openpnt-ts2phc@.service sudo systemctl status openpnt-ts2phc@.service --no-pager sudo journalctl -u openpnt-ts2phc@.service -n 200 --no-pager openrf-agent.service sudo systemctl status openrf-agent.service --no-pager sudo journalctl -u openrf-agent.service -n 200 --no-pager openc5isr-bss.service sudo systemctl status openc5isr-bss.service --no-pager sudo journalctl -u openc5isr-bss.service -n 200 --no-pager Safe control sequence # Inspect before changing anything systemctl status .service --no-pager journalctl -u .service -n 100 --no-pager # Start/stop/restart only the intended unit sudo systemctl start .service sudo systemctl stop .service sudo systemctl restart .service # Confirm process, port and local response systemctl status .service --no-pager ss -lntup curl -fsS http://127.0.0.1:/ Control rules Inspect current state and recent logs before restart; a restart can erase evidence or amplify a dependency failure. Restart the smallest affected scope instead of all platform services. Record who initiated the change, why, exact commands, result and rollback decision. Verify local service first, then reverse proxy, then end-to-end workflow. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Monitoring & Assurance Managed documentation for Monitoring & Assurance. Health Checks Health Checks Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Route scan and operational health model Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Candidate health endpoints No route name strongly associated with health/readiness was detected. Layered health model Layer Check Good result Process systemctl is-active / container state Active without restart loop. Socket ss -lntup Expected address/port owned by expected process. Local HTTP/API curl loopback endpoint Expected status/body within threshold. Proxy/TLS curl public hostname Valid certificate, correct routing and headers. Dependency Database/bus/files/knowledge checks Authenticated dependency response and acceptable latency. Workflow Primary user transaction End-to-end result is visible, correlated and retained. Health endpoint requirements A liveness endpoint should prove the process can respond; readiness should prove required dependencies and configuration are usable. Do not expose secrets, stack traces or sensitive topology in public health output. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Logs and Audit Logs and Audit Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Operational logging standard Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Log sources Source Typical command/location Use systemd/journald journalctl -u --since ... Process lifecycle, stdout/stderr and service errors. Nginx /var/log/nginx/access.log, /var/log/nginx/error.log Routing, status, latency, upstream and TLS symptoms. Application Structured logs or configured log directory Domain events, validation, dependency calls and workflow state. Audit Protected append/central audit store Actor, action, target, policy decision, result and correlation. Sensor/edge Device/agent logs and telemetry Connection, clock, calibration, sample loss and product generation. Required structured fields UTC timestamp plus source clock/host. Severity, service/component, release version and instance. Request/event/task/track/case correlation identifier. Authenticated actor or service identity when applicable. Outcome/error code and safe diagnostic context. Do not log Passwords, API token secrets, private keys, complete authorization headers or session cookies. Sensitive content unless the approved audit/evidence policy explicitly requires and protects it. Unbounded raw sensor/media streams in ordinary application logs. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Operational Checklist Operational Checklist Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Operational standard Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Daily/shift-start checks System time synchronized and no large clock offset alerts. Required services active without restart loops. Disk, memory, CPU and network within operating limits. Public hostnames and local health endpoints respond correctly. OpenBus/data/knowledge/files dependencies reachable. Representative sensor/track/task workflow completes. Backups completed and no unresolved critical security/audit alerts. After any change Configuration syntax validated. Only intended service/process changed. Local, proxy and end-to-end tests passed. Logs reviewed for new warnings/errors. Version/change record and rollback state updated. Handover record Record active release, known degradations, temporary workarounds, open incidents, pending changes, sensor availability, external dependency state and the next required action. Operational continuity should not depend on oral memory. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Continuity & Recovery Managed documentation for Continuity & Recovery. Backup and Recovery Backup and Recovery Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Continuity standard Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Backup scope Asset Backup requirement Restore verification Release artifacts Retain exact immutable packages/checksums; normally do not back up regenerated dependencies. Checksum and extraction test. Configuration Versioned protected copy excluding or separately encrypting secrets. Syntax test and controlled comparison. Databases Consistent database-native dump/snapshot with schema/version metadata. Restore into isolated environment and run integrity/query checks. Files/evidence Content plus metadata, ownership, access policy, checksums and provenance. Sample restore, checksum and access-control test. Knowledge/ontology Graph data, ontology versions, shapes, indexes and configuration. SPARQL/constraint tests against restored graph. Audit/history Protected retention according to policy; preserve ordering and integrity evidence. Query selected events and verify integrity chain. Recovery order Infrastructure, DNS/TLS and time. Protected configuration and secrets. Databases, knowledge and artifact stores. OpenBus/shared backends. Edition-specific applications and reverse proxy. Sensors/edge agents and external integrations. End-to-end acceptance tests and user access. Recovery objective Define and test RPO/RTO by data class. A backup that has never been restored is an optimistic file collection, not a recovery capability. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs. Troubleshooting Troubleshooting Field Value Distribution BSS — OpenC5ISR Product OpenC5ISR Release 0.15.1 Deployment openc5isr.bss.dev Source /srv/bss/releases/OpenC5ISR-v0.15.0-20app Evidence Operational diagnostic standard Source fingerprint 93926e8c834f1ed79c9017a4096f70c76d73ce4f870f104fbd0cf54b2bed50ee Status Generated baseline — human review required Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment. Diagnostic sequence State the exact user-visible symptom, start time, scope and last known good state. Check DNS/routing/time before assuming an application defect. Check process/container state and recent logs. Test the backend locally on its bound address/port. Test proxy/TLS path and WebSocket upgrade separately. Test dependencies: database, OpenBus, OpenKnowledge, OpenFiles and external services. Reproduce one primary workflow with correlation identifiers and capture evidence. Apply the smallest reversible change, then retest all layers. Symptom matrix Symptom Likely layer First checks Connection refused Process/socket systemctl status, ss -lntup, local curl Empty response/reset App/upstream crash or protocol mismatch Application logs, internal HTTP response, proxy protocol. 502/504 Reverse proxy/upstream nginx -t, error log, proxy_pass, backend latency Page loads but live data fails WebSocket/CORS/auth Browser network console, upgrade headers, token/role, route base. Wrong distribution/branding Nginx/service instance Hostname routing, service unit, current release symlink, environment. Tracks/tasks/products missing Dependency/data flow Producer logs, OpenBus/API, authorization, semantic type, time window. Intermittent correlation error Time/PNT/data quality Clock offset, source timestamps, coordinate frame and calibration. Evidence rule Before changing configuration, preserve the relevant logs, status output, request/response, release version and timestamps. Troubleshooting without evidence tends to become configuration roulette—with worse odds and fewer free drinks. Maintainer Notes Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.