Storage architecture // phase one
Keep every packet. Make it searchable.
The Data Vault is the backend plan for permanent OKI Mesh / MESH BUDDY MQTT history. It separates raw capture from decoded intelligence so we can rebuild future analytics without losing the original payload stream.
Recommended stack
Built for your homelab and long-term retention.
A
Collector servicePython asyncio MQTT subscriber running in Docker or K3s. One config file defines brokers, credentials, topics, and decode rules.
B
Hot analytics DBClickHouse for high-volume packet/time-series queries, or TimescaleDB/Postgres if we want simpler ops first.
C
Permanent archiveDaily compressed raw files to Synology/NFS or MinIO object storage:
meshbuddy/year=2026/month=07/day=07/*.ndjson.zst.D
API layerFastAPI service serving the analytics pages with rollups, topology edges, route history, and replay slices.
Ingest sources
Everything is source-tagged so we know where each event came from.
OKI public MQTT broker
public mesh traffic / community topics
public mesh traffic / community topics
MESH BUDDY bridge broker
local MQTT, MeshCore, scoreboard, future private feeds
local MQTT, MeshCore, scoreboard, future private feeds
Static cache files
/data/nodes.json / nodes-ao.json / scoreboard.json
/data/nodes.json / nodes-ao.json / scoreboard.json
Derived jobs
hourly rollups, route edges, node lifecycle snapshots
hourly rollups, route edges, node lifecycle snapshots
First database tables
Start raw and minimal. Add smarter decode tables after capture is reliable.
mqtt_raw
id · rx_ts · broker · topic · qos · retain · payload_text · payload_sha256 · bytes
id · rx_ts · broker · topic · qos · retain · payload_text · payload_sha256 · bytes
mqtt_decode
raw_id · packet_type · node_key · origin · rssi · snr · hops · channel · decoded_json
raw_id · packet_type · node_key · origin · rssi · snr · hops · channel · decoded_json
mesh_node_seen
node_key · name · role · lat · lon · first_seen · last_seen · source
node_key · name · role · lat · lon · first_seen · last_seen · source
mesh_route_edge
src_key · dst_key · repeater_key · packet_hash · ts · score
src_key · dst_key · repeater_key · packet_hash · ts · score
topic_rollup_hour
hour · broker · topic_prefix · message_count · byte_count · unique_nodes
hour · broker · topic_prefix · message_count · byte_count · unique_nodes
archive_manifest
day · path · rows · sha256 · compressed_bytes · storage_uri
day · path · rows · sha256 · compressed_bytes · storage_uri
Rules for indefinite storage
This keeps us from painting ourselves into a corner.
1
Never throw away raw MQTT.Decoded tables can change. Raw payload history is the truth.
2
Dedupe, but do not overwrite.Mark duplicates by hash; keep enough metadata to study bridge loops and re-transmissions.
3
Partition everything by day.Makes backup, restore, and long-term archive easy.
4
Protect sensitive/private channels.Public OKI traffic can be broadly visualized. Private MESH BUDDY topics should support redaction and access control before public display.