[wct-en] 05 Operations & Administration
Operations, monitoring, backup, recovery, identity, security and troubleshooting. Edition: WCT English — OpenC5ISR (openc5isr-en.wallacecorptech.com).
Start Here
Start Here
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Documentation structure and release scan |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Purpose
This book provides a safe operational baseline for the named distribution. Replace placeholders with tested unit names, URLs, owners, thresholds and escalation paths.
Contents
| Section | What it contains |
|---|---|
| Service Control | Start, stop, restart and inspect the deployed services. |
| Monitoring & Assurance | Health checks, logs, audit and daily operational controls. |
| Continuity & Recovery | Backups, recovery and structured troubleshooting. |
How to maintain this book
Generated sections are replaced from release evidence on each run. Put reviewed corrections, deployment-specific facts, links and decisions in the Maintainer Notes area below the generated block. Mark pages as reviewed only after testing them against the named distribution and release.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Service Control
Managed documentation for Service Control.
Starting, Stopping and Restarting
Starting, Stopping and Restarting
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Observed service units plus operational control pattern |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Observed service candidates
| Unit | Status command | Log command |
|---|---|---|
| opencybersec-agent.service | sudo systemctl status opencybersec-agent.service --no-pager |
sudo journalctl -u opencybersec-agent.service -n 200 --no-pager |
| openlvc-ais-bridge.service | sudo systemctl status openlvc-ais-bridge.service --no-pager |
sudo journalctl -u openlvc-ais-bridge.service -n 200 --no-pager |
| openlvc-dis-bridge.service | sudo systemctl status openlvc-dis-bridge.service --no-pager |
sudo journalctl -u openlvc-dis-bridge.service -n 200 --no-pager |
| openlvc-flightgear-bridge.service | sudo systemctl status openlvc-flightgear-bridge.service --no-pager |
sudo journalctl -u openlvc-flightgear-bridge.service -n 200 --no-pager |
| openlvc-rti-gateway.service | sudo systemctl status openlvc-rti-gateway.service --no-pager |
sudo journalctl -u openlvc-rti-gateway.service -n 200 --no-pager |
| openpnt-phc2sys@.service | sudo systemctl status openpnt-phc2sys@.service --no-pager |
sudo journalctl -u openpnt-phc2sys@.service -n 200 --no-pager |
| openpnt-ptp4l@.service | sudo systemctl status openpnt-ptp4l@.service --no-pager |
sudo journalctl -u openpnt-ptp4l@.service -n 200 --no-pager |
| openpnt-ts2phc@.service | sudo systemctl status openpnt-ts2phc@.service --no-pager |
sudo journalctl -u openpnt-ts2phc@.service -n 200 --no-pager |
| openrf-agent.service | sudo systemctl status openrf-agent.service --no-pager |
sudo journalctl -u openrf-agent.service -n 200 --no-pager |
| openc5isr-wct-en.service | sudo systemctl status openc5isr-wct-en.service --no-pager |
sudo journalctl -u openc5isr-wct-en.service -n 200 --no-pager |
Safe control sequence
# Inspect before changing anything
systemctl status <unit>.service --no-pager
journalctl -u <unit>.service -n 100 --no-pager
# Start/stop/restart only the intended unit
sudo systemctl start <unit>.service
sudo systemctl stop <unit>.service
sudo systemctl restart <unit>.service
# Confirm process, port and local response
systemctl status <unit>.service --no-pager
ss -lntup
curl -fsS http://127.0.0.1:<port>/<health-path>
Control rules
- Inspect current state and recent logs before restart; a restart can erase evidence or amplify a dependency failure.
- Restart the smallest affected scope instead of all platform services.
- Record who initiated the change, why, exact commands, result and rollback decision.
- Verify local service first, then reverse proxy, then end-to-end workflow.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Monitoring & Assurance
Managed documentation for Monitoring & Assurance.
Health Checks
Health Checks
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Route scan and operational health model |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Candidate health endpoints
No route name strongly associated with health/readiness was detected.
Layered health model
| Layer | Check | Good result |
|---|---|---|
| Process | systemctl is-active / container state |
Active without restart loop. |
| Socket | ss -lntup |
Expected address/port owned by expected process. |
| Local HTTP/API | curl loopback endpoint |
Expected status/body within threshold. |
| Proxy/TLS | curl public hostname |
Valid certificate, correct routing and headers. |
| Dependency | Database/bus/files/knowledge checks | Authenticated dependency response and acceptable latency. |
| Workflow | Primary user transaction | End-to-end result is visible, correlated and retained. |
Health endpoint requirements
A liveness endpoint should prove the process can respond; readiness should prove required dependencies and configuration are usable. Do not expose secrets, stack traces or sensitive topology in public health output.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Logs and Audit
Logs and Audit
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Operational logging standard |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Log sources
| Source | Typical command/location | Use |
|---|---|---|
| systemd/journald | journalctl -u <unit> --since ... |
Process lifecycle, stdout/stderr and service errors. |
| Nginx | /var/log/nginx/access.log, /var/log/nginx/error.log |
Routing, status, latency, upstream and TLS symptoms. |
| Application | Structured logs or configured log directory | Domain events, validation, dependency calls and workflow state. |
| Audit | Protected append/central audit store | Actor, action, target, policy decision, result and correlation. |
| Sensor/edge | Device/agent logs and telemetry | Connection, clock, calibration, sample loss and product generation. |
Required structured fields
- UTC timestamp plus source clock/host.
- Severity, service/component, release version and instance.
- Request/event/task/track/case correlation identifier.
- Authenticated actor or service identity when applicable.
- Outcome/error code and safe diagnostic context.
Do not log
- Passwords, API token secrets, private keys, complete authorization headers or session cookies.
- Sensitive content unless the approved audit/evidence policy explicitly requires and protects it.
- Unbounded raw sensor/media streams in ordinary application logs.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Operational Checklist
Operational Checklist
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Operational standard |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Daily/shift-start checks
- System time synchronized and no large clock offset alerts.
- Required services active without restart loops.
- Disk, memory, CPU and network within operating limits.
- Public hostnames and local health endpoints respond correctly.
- OpenBus/data/knowledge/files dependencies reachable.
- Representative sensor/track/task workflow completes.
- Backups completed and no unresolved critical security/audit alerts.
After any change
- Configuration syntax validated.
- Only intended service/process changed.
- Local, proxy and end-to-end tests passed.
- Logs reviewed for new warnings/errors.
- Version/change record and rollback state updated.
Handover record
Record active release, known degradations, temporary workarounds, open incidents, pending changes, sensor availability, external dependency state and the next required action. Operational continuity should not depend on oral memory.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Continuity & Recovery
Managed documentation for Continuity & Recovery.
Backup and Recovery
Backup and Recovery
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Continuity standard |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Backup scope
| Asset | Backup requirement | Restore verification |
|---|---|---|
| Release artifacts | Retain exact immutable packages/checksums; normally do not back up regenerated dependencies. | Checksum and extraction test. |
| Configuration | Versioned protected copy excluding or separately encrypting secrets. | Syntax test and controlled comparison. |
| Databases | Consistent database-native dump/snapshot with schema/version metadata. | Restore into isolated environment and run integrity/query checks. |
| Files/evidence | Content plus metadata, ownership, access policy, checksums and provenance. | Sample restore, checksum and access-control test. |
| Knowledge/ontology | Graph data, ontology versions, shapes, indexes and configuration. | SPARQL/constraint tests against restored graph. |
| Audit/history | Protected retention according to policy; preserve ordering and integrity evidence. | Query selected events and verify integrity chain. |
Recovery order
- Infrastructure, DNS/TLS and time.
- Protected configuration and secrets.
- Databases, knowledge and artifact stores.
- OpenBus/shared backends.
- Edition-specific applications and reverse proxy.
- Sensors/edge agents and external integrations.
- End-to-end acceptance tests and user access.
Recovery objective
Define and test RPO/RTO by data class. A backup that has never been restored is an optimistic file collection, not a recovery capability.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.
Troubleshooting
Troubleshooting
| Field | Value |
|---|---|
| Distribution | WCT English — OpenC5ISR |
| Product | OpenC5ISR |
| Release | 0.15.1 |
| Deployment | openc5isr-en.wallacecorptech.com |
| Source | /srv/wct/releases/OpenC5ISR-v0.15.0 |
| Evidence | Operational diagnostic standard |
| Source fingerprint | 63d2b1097707e25d666401cdc6362e395968600a40a08690210cd8c0b75fbf50 |
| Status | Generated baseline — human review required |
Verification boundary: This page combines platform design guidance with static evidence from the release. It does not prove that every detected interface is enabled, reachable, secure, or operational in the deployed environment.
Diagnostic sequence
- State the exact user-visible symptom, start time, scope and last known good state.
- Check DNS/routing/time before assuming an application defect.
- Check process/container state and recent logs.
- Test the backend locally on its bound address/port.
- Test proxy/TLS path and WebSocket upgrade separately.
- Test dependencies: database, OpenBus, OpenKnowledge, OpenFiles and external services.
- Reproduce one primary workflow with correlation identifiers and capture evidence.
- Apply the smallest reversible change, then retest all layers.
Symptom matrix
| Symptom | Likely layer | First checks |
|---|---|---|
| Connection refused | Process/socket | systemctl status, ss -lntup, local curl |
| Empty response/reset | App/upstream crash or protocol mismatch | Application logs, internal HTTP response, proxy protocol. |
| 502/504 | Reverse proxy/upstream | nginx -t, error log, proxy_pass, backend latency |
| Page loads but live data fails | WebSocket/CORS/auth | Browser network console, upgrade headers, token/role, route base. |
| Wrong distribution/branding | Nginx/service instance | Hostname routing, service unit, current release symlink, environment. |
| Tracks/tasks/products missing | Dependency/data flow | Producer logs, OpenBus/API, authorization, semantic type, time window. |
| Intermittent correlation error | Time/PNT/data quality | Clock offset, source timestamps, coordinate frame and calibration. |
Evidence rule
Before changing configuration, preserve the relevant logs, status output, request/response, release version and timestamps. Troubleshooting without evidence tends to become configuration roulette—with worse odds and fewer free drinks.
Maintainer Notes
Add human-reviewed deployment notes, corrections, decisions, screenshots, and links here. Content outside the generated block is preserved on future runs.