ENELIX/Enelix-EMS
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(data): add passive versioned raw capture independent of control

Browse Source
This commit is contained in:
ENELIX Agent
2026-10-02 14:55:43 +00:00
parent e0e52d2454
commit 76f8b067cc
7 changed files with 603 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/netplan-v4-data-capture.md
+67
View File
@@ -0,0 +1,67 @@
# Passive V4 data capture - user inventory and raw acquisition
Status 2026-10-02: source implementation, CLI/mocked tests and staged user installation.
Not part of the live manager, not a control or training release.
## Lihrenmoos user-reported inventory (not applied to running configuration)
- GoodWe 1/2: each 50 kW AC, each 156 kWh reported physical battery capacity.
- SolarEdge: 10 kW AC, 10 kWh reported physical battery capacity.
- EV allocation: 30 kW / 160 kWh total, understood as 10 kW EV per inverter.
- PV roof: approximately 20 kWp east and 20 kWp west. Tilt/individual strings unknown.
- Nominal battery sum: 322 kWh. 162 kWh nominal difference to EV allocation is NOT a
verified usable SDL capacity. AC ratings are not verified battery charge/discharge ratings.
- Exact types, firmware, common AC meter boundary and device watchdog remain unverified.
- Current saved manager topology still reports 161.44 kWh / 39 kW. Gateway currently
lists 130/130/8 kWh and 57/57/5 kW. These may use a different usable/nominal basis.
Do not rebase virtual SOC or change current controls based on these notes alone.
## Why collect before boundary approval
Record the actual original numerical channels now, with source metadata and timestamps.
The algebraic base-load candidate is useful for diagnosis, but NEVER a certified base-load
training sample until the physical AC/DC boundary and historical recipe are verified.
No dependence on hardware-watchdog approval for passive recording.
21 channels: PCC power, legacy displayed PCC power, PV for three inverters, three physical
battery powers, EV/SDL virtual powers and SOCs, three physical SOCs, dynamic EV limits,
T1/T2 import counters, EV/SDL requested setpoints (READ ONLY).
A cyclic script reads values already present in Symcon every 30 seconds. It does not issue
additional Modbus polls. Original VariableUpdated/VariableChanged are retained; these are
Symcon timestamps, not independent device-measurement timestamps. Two-pass reads detect
concurrent changes; missing/non-numeric/re-used channels remain explicit quality failures.
## Storage and interpretation
Daily append-only raw-YYYYMMDD.jsonl in the separate capture stage data directory, UTC
capture times plus original source timestamps, immutable mapping/inventory snapshots by
SHA256. Raw errors and quality_hold are recorded too; nothing is invented as zero.
The raw observations are a source for later documented 5-minute aggregation, not themselves
aligned 5-minute means. No interpolation, training publication or future SDL assumption.
A 512 MiB journal quota, 64 MiB daily limit, 256 KiB record cap and 100 MiB free-space
reserve bound recording. Limits fail visibly without deleting old data. File permissions
0640; no upload credentials, request bodies, complete instance configurations or secrets.
A separate status variable and STATUS.json show latest capture and failures. Process restart
retains files; an incomplete final JSONL row is preserved for manual review, not extended.
## Install / stop
Staged user entry point: /srv/agent/netplan-v4-data-capture-stage/install.php
Execute in the Symcon PHP editor (not a Linux shell):
require '/srv/agent/netplan-v4-data-capture-stage/install.php';
Installer checks exact source hashes and numeric source identities before creating a NEW
root category ENELIX_V4_PASSIVE_CAPTURE, a status string, collector script and stop script.
IPS_SetScriptTimer enables only this new script at 30 seconds. Repeating installation is
idempotent. Unrelated/edited scripts or object identity collisions are not overwritten.
Stop using the newly created script 'Nur diese Messaufnahme stoppen'; it disables only
this collection timer, retains collected files, and does not stop the manager or battery.
No module reload, existing archive change, existing timer/poller change, service restart,
forecast selection, battery reservation or actuator command is included.
Current tests: 36 core/file tests + 19 mocked native lifecycle tests with PHP CLI on the
test host. First real capture and future timer execution require the user activation.
These tests do not certify hardware mappings or a Symcon-kernel run.
examples/V4DataCapture/install.php
+89
View File
@@ -0,0 +1,89 @@
<?php
declare(strict_types=1);
require_once __DIR__ . '/runtime.php';
/** Creates ONLY dedicated capture objects. No module reload, device settings or archives. */
function v4captureEnsure(int $parent, string $ident, int $type, string $name, string $marker, array &$created): int
{
$existing = v4captureChild($parent, $ident);
if ($existing !== null) {
v4captureAssertObject($existing, $parent, $ident, $type, $marker);
return $existing;
}
$id = match ($type) { 0 => IPS_CreateCategory(), 2 => IPS_CreateVariable(3), 3 => IPS_CreateScript(0) };
if (!is_int($id) || $id <= 0) throw new RuntimeException('Cannot create capture object');
$created[] = $id;
if (!IPS_SetParent($id, $parent) || !IPS_SetIdent($id, $ident) || !IPS_SetName($id, $name) || !IPS_SetInfo($id, $marker)) {
throw new RuntimeException('Cannot label capture object');
}
return $id;
}
function v4captureScript(int $id, string $content, array $created): void
{
$current = IPS_GetScriptContent($id);
if ($current === $content) return;
if (!in_array($id, $created, true)) throw new RuntimeException('Existing capture script changed; not overwritten');
if (!IPS_SetScriptContent($id, $content) || IPS_GetScriptContent($id) !== $content) throw new RuntimeException('Cannot store capture script');
}
function v4captureInstall(string $stage): array
{
if (!function_exists('IPS_GetKernelVersion') || version_compare(IPS_GetKernelVersion(),'8.0','<')) {
throw new RuntimeException('Nur im IP-Symcon-Skripteditor der Testanlage ausfuehren.');
}
$c = v4captureConfig($stage); $marker = v4captureMarker($c);
if (!IPS_InstanceExists($c['managerId'])
|| IPS_GetInstance($c['managerId'])['ModuleInfo']['ModuleID'] !== '{6F771B18-59D4-4C8A-B951-3B2FE9F6A2C4}') {
throw new RuntimeException('Expected manager not found');
}
// Verify layout only; stale/missing sensor updates remain diagnostic raw data, not fake zeros.
foreach (array_merge($c['accounting']['sources'], $c['extraSources']) as $s) {
if (!IPS_VariableExists($s['variableId'])) throw new RuntimeException('Expected source missing: ' . $s['key']);
$o = IPS_GetObject($s['variableId']); $v = IPS_GetVariable($s['variableId']);
if ($o['ParentID'] !== $s['parentId'] || $o['ObjectIdent'] !== $s['ident'] || !in_array($v['VariableType'],[1,2],true)) {
throw new RuntimeException('Source identity changed: ' . $s['key']);
}
}
$data = $stage . '/data';
if (is_link($data) || !is_dir($data) || !is_writable($data)) throw new RuntimeException('Dedicated capture directory not writable');
$key = 'ENELIX.V4.CaptureInstall.' . $c['installationId'];
if (!IPS_SemaphoreEnter($key, 3000)) throw new RuntimeException('Capture installation busy');
$created = []; $script = null; $timerSet = false;
$report = ['startedAt' => gmdate('c'), 'scope' => 'passive_raw_capture_only', 'controlEnabled' => false,
'existingModulesChanged' => false, 'archivesChanged' => false, 'existingHistoriesChanged' => false];
try {
$category = v4captureEnsure(0,'ENELIX_V4_PASSIVE_CAPTURE',0,'V4 - separate Messaufnahme (keine Steuerung)',$marker,$created);
v4captureEnsure($category,'CaptureStatus',2,'Aufzeichnungsstatus (nur lesen)',$marker,$created);
$script = v4captureEnsure($category,'RawCapture',3,'Messwerte alle 30 Sekunden erfassen',$marker,$created);
$stop = v4captureEnsure($category,'StopCapture',3,'Nur diese Messaufnahme stoppen',$marker,$created);
$include = '<?php' . "\n" . 'require_once ' . var_export($stage . '/source/examples/V4DataCapture/runtime.php',true) . ';' . "\n";
v4captureScript($script, $include . 'v4captureRun(' . var_export($stage,true) . ', ' . $category . ');' . "\n", $created);
v4captureScript($stop, $include . 'v4captureStop(' . var_export($stage,true) . ', ' . $category . ');' . "\n", $created);
$first = v4captureRun($stage, $category);
if ($first['status'] !== 'recorded') throw new RuntimeException('Capture not stored: ' . ($first['reason'] ?? $first['status']));
// This new timer never invokes a manager, gateway, battery or Modbus function.
if (!IPS_SetScriptTimer($script, 30)) throw new RuntimeException('Cannot start dedicated capture timer');
$timerSet = true;
$report += ['status' => 'capture_installed', 'categoryId' => $category, 'scriptId' => $script,
'stopScriptId' => $stop, 'intervalSeconds' => 30, 'firstCapture' => $first,
'futureTimerExecutionVerified' => false];
echo "OK: Separate Rohdatenaufnahme eingerichtet, alle 30 Sekunden.\n";
echo 'Erste Aufnahme: ' . $first['capturedAt'] . '; Quellen: ' . $first['sourceCount'] . ".\n";
echo 'Bilanzqualitaet: ' . $first['accountingQuality'] . " (noch keine Trainings-/Stellfreigabe).\n";
echo "Bestehende Manager-, Batterie-, Archiv- und Pollingkonfiguration unveraendert.\n";
} catch (Throwable $e) {
$report['status'] = 'needs_review'; $report['reason'] = substr($e->getMessage(),0,300);
if ($timerSet && $script !== null) IPS_SetScriptTimer($script,0);
// Never delete unrelated objects or stored data during recovery.
echo 'FEHLER: ' . $report['reason'] . "\n";
} finally {
$report['createdObjects'] = $created; $report['finishedAt'] = gmdate('c');
try { v4captureAtomicJson($stage . '/INSTALL_RESULT.json', $report); }
finally { IPS_SemaphoreLeave($key); }
echo 'CAPTURE REPORT: ' . $stage . "/INSTALL_RESULT.json\n";
}
return $report;
}
examples/V4DataCapture/runtime.php
+143
View File
@@ -0,0 +1,143 @@
<?php
declare(strict_types=1);
require_once dirname(__DIR__, 2) . '/libs/NetzfahrplanV4Messaufnahme.php';
use Belevo\EnelixEMS\NetzfahrplanV4Messaufnahme;
/** Native bindings for a dedicated passive capture category. No network or actuator calls. */
function v4captureConfig(string $stage): array
{
if (!is_dir($stage) || is_link($stage) || realpath($stage) !== $stage) throw new RuntimeException('Invalid capture stage');
$manifest = json_decode((string)file_get_contents($stage . '/MANIFEST.json'), true, 64, JSON_THROW_ON_ERROR);
$expected = ['source/libs/NetzfahrplanV4Bilanzierung.php','source/libs/NetzfahrplanV4Messaufnahme.php',
'source/examples/V4DataCapture/runtime.php','source/examples/V4DataCapture/install.php',
'config.json','USER_REPORTED_INVENTORY.json'];
if (($manifest['scope'] ?? null) !== 'passive_raw_capture_only' || array_keys($manifest['hashes'] ?? []) !== $expected) {
throw new RuntimeException('Unexpected capture manifest');
}
foreach ($expected as $name) {
$p = $stage . '/' . $name;
if (is_link($p) || !is_file($p) || realpath($p) !== $p || hash_file('sha256', $p) !== $manifest['hashes'][$name]) {
throw new RuntimeException('Reviewed capture source changed: ' . $name);
}
}
$c = json_decode((string)file_get_contents($stage . '/config.json'), true, 64, JSON_THROW_ON_ERROR);
if (($c['managerId'] ?? null) !== 17004 || ($c['installationId'] ?? null) !== 'e3a08f9e-af12-4695-99bd-8b51c0520021'
|| ($c['reportedInventorySha256'] ?? null) !== $manifest['hashes']['USER_REPORTED_INVENTORY.json']) {
throw new RuntimeException('Wrong commissioning installation or inventory');
}
return NetzfahrplanV4Messaufnahme::configuration($c);
}
function v4captureMarker(array $c): string
{
return 'ENELIX passive raw capture v1|' . $c['installationId'];
}
function v4captureChild(int $parent, string $ident): ?int
{
$found = null;
foreach (IPS_GetChildrenIDs($parent) as $id) {
if (IPS_GetObject($id)['ObjectIdent'] !== $ident) continue;
if ($found !== null) throw new RuntimeException('Duplicate capture object');
$found = $id;
}
return $found;
}
function v4captureAssertObject(int $id, int $parent, string $ident, int $type, string $marker): void
{
$o = IPS_GetObject($id);
if ($o['ParentID'] !== $parent || $o['ObjectIdent'] !== $ident || $o['ObjectType'] !== $type || $o['ObjectInfo'] !== $marker) {
throw new RuntimeException('Unexpected capture object ownership');
}
if ($type === 2 && IPS_GetVariable($id)['VariableType'] !== 3) throw new RuntimeException('Capture status is not a string');
}
function v4captureAtomicJson(string $file, array $value): void
{
if (is_link($file)) throw new RuntimeException('Capture report symlink refused');
$tmp = tempnam(dirname($file), '.capture-');
if ($tmp === false) throw new RuntimeException('Cannot create report');
try {
$data = json_encode($value, JSON_THROW_ON_ERROR | JSON_PRETTY_PRINT) . "\n";
if (file_put_contents($tmp, $data, LOCK_EX) !== strlen($data) || !chmod($tmp, 0640) || !rename($tmp, $file)) {
throw new RuntimeException('Cannot store capture report');
}
} finally { if (is_file($tmp)) unlink($tmp); }
}
function v4captureKeepSnapshot(string $path, string $bytes): void
{
if (is_link($path)) throw new RuntimeException('Metadata symlink refused');
if (is_file($path)) {
if (hash_file('sha256', $path) !== hash('sha256', $bytes)) throw new RuntimeException('Immutable capture metadata differs');
return;
}
$h = fopen($path, 'xb');
if ($h === false) throw new RuntimeException('Cannot store capture metadata');
try {
if (fwrite($h, $bytes) !== strlen($bytes) || !fflush($h) || !chmod($path, 0640)) throw new RuntimeException('Capture metadata write failed');
} finally { fclose($h); }
}
function v4captureRead(int $id): array
{
if (!IPS_VariableExists($id)) throw new RuntimeException('Missing source');
$v = IPS_GetVariable($id); $o = IPS_GetObject($id);
if (!in_array($v['VariableType'], [1,2], true)) throw new RuntimeException('Non-numeric source');
// Capture layer reads all sources twice; source timestamps are NOT replaced with now.
return ['value' => GetValue($id), 'updated' => (int)$v['VariableUpdated'], 'changed' => (int)$v['VariableChanged'],
'parentID' => $o['ParentID'], 'ident' => $o['ObjectIdent']];
}
function v4captureRun(string $stage, int $category): array
{
if (!function_exists('IPS_GetKernelVersion')) throw new RuntimeException('Run inside IP-Symcon only');
$c = v4captureConfig($stage); $marker = v4captureMarker($c);
v4captureAssertObject($category, 0, 'ENELIX_V4_PASSIVE_CAPTURE', 0, $marker);
$statusId = v4captureChild($category, 'CaptureStatus');
if ($statusId === null) throw new RuntimeException('Capture status missing');
v4captureAssertObject($statusId, $category, 'CaptureStatus', 2, $marker);
$lock = 'ENELIX.V4.RawCapture.' . $c['installationId'];
if (!IPS_SemaphoreEnter($lock, 0)) return ['status' => 'busy', 'controlEnabled' => false];
try {
try {
$record = NetzfahrplanV4Messaufnahme::capture($c, 'v4captureRead', static fn(): int => time());
v4captureKeepSnapshot($stage . '/data/mapping-' . $record['mappingSha256'] . '.json', json_encode($c, JSON_THROW_ON_ERROR | JSON_PRESERVE_ZERO_FRACTION));
v4captureKeepSnapshot($stage . '/data/inventory-' . $record['reportedInventorySha256'] . '.json', (string)file_get_contents($stage . '/USER_REPORTED_INVENTORY.json'));
$path = NetzfahrplanV4Messaufnahme::append($stage . '/data', $record);
$status = ['status' => 'recorded', 'capturedAt' => $record['capturedAt'], 'journal' => $path,
'sourceCount' => count($record['raw']), 'rawIssueCount' => count($record['issues']),
'accountingQuality' => $record['assessment']['status'] ?? 'unavailable',
'qualityIssues' => $record['assessment']['issues'] ?? $record['issues'],
'baseLoadCandidateW' => $record['assessment']['baseLoadCandidateW'] ?? null,
'sourceSkewSeconds' => $record['assessment']['sourceSkewSeconds'] ?? null,
'trainingEligible' => false, 'controlEnabled' => false, 'measurementBoundaryVerified' => false];
v4captureAtomicJson($stage . '/data/STATUS.json', $status);
} catch (Throwable $e) {
$status = ['status' => 'capture_error', 'at' => gmdate('c'), 'reason' => substr($e->getMessage(),0,250),
'trainingEligible' => false, 'controlEnabled' => false];
try { v4captureAtomicJson($stage . '/data/STATUS.json', $status); } catch (Throwable $ignored) { }
}
// Only this newly created diagnosis variable, never a meter or controller variable.
SetValue($statusId, json_encode($status, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES));
return $status;
} finally { IPS_SemaphoreLeave($lock); }
}
function v4captureStop(string $stage, int $category): void
{
$c = v4captureConfig($stage); $marker = v4captureMarker($c);
v4captureAssertObject($category, 0, 'ENELIX_V4_PASSIVE_CAPTURE', 0, $marker);
$script = v4captureChild($category, 'RawCapture'); $status = v4captureChild($category, 'CaptureStatus');
if ($script === null || $status === null) throw new RuntimeException('Capture objects missing');
v4captureAssertObject($script, $category, 'RawCapture', 3, $marker);
v4captureAssertObject($status, $category, 'CaptureStatus', 2, $marker);
if (!IPS_SetScriptTimer($script, 0)) throw new RuntimeException('Cannot stop capture timer');
$r = ['status' => 'stopped', 'at' => gmdate('c'), 'oldDataRetained' => true, 'controlEnabled' => false];
SetValue($status, json_encode($r, JSON_THROW_ON_ERROR));
v4captureAtomicJson($stage . '/data/STATUS.json', $r);
}
libs/NetzfahrplanV4Messaufnahme.php
+165
View File
@@ -0,0 +1,165 @@
<?php
declare(strict_types=1);
namespace Belevo\EnelixEMS;
use InvalidArgumentException;
use RuntimeException;
use Throwable;
require_once __DIR__ . '/NetzfahrplanV4Bilanzierung.php';
/** Passive, versioned RAW capture. Never grants training/control eligibility.
* Reader supplies numeric values only; no configuration, credentials or network.
*/
final class NetzfahrplanV4Messaufnahme
{
public static function configuration(array $c): array
{
if (($c['version'] ?? null) !== 1 || !is_string($c['installationId'] ?? null)
|| !preg_match('/^[0-9a-f-]{36}$/D', $c['installationId'])
|| !is_array($c['extraSources'] ?? null) || count($c['extraSources']) > 60
|| !is_string($c['reportedInventorySha256'] ?? null)
|| !preg_match('/^[0-9a-f]{64}$/D', $c['reportedInventorySha256'])) {
throw new InvalidArgumentException('Invalid capture configuration');
}
$c['accounting'] = NetzfahrplanV4Bilanzierung::konfigurieren($c['accounting'] ?? []);
$keys = []; $ids = [];
foreach ($c['accounting']['sources'] as $s) { $keys[$s['key']] = true; $ids[$s['variableId']] = true; }
foreach ($c['extraSources'] as $s) {
if (count($s) !== 5 || !is_string($s['key'] ?? null)
|| !preg_match('/^[A-Za-z][A-Za-z0-9_-]{0,63}$/D', $s['key']) || isset($keys[$s['key']])
|| !is_int($s['variableId'] ?? null) || $s['variableId'] < 1 || isset($ids[$s['variableId']])
|| !is_int($s['parentId'] ?? null) || $s['parentId'] < 1
|| !is_string($s['ident'] ?? null) || strlen($s['ident']) > 100
|| !in_array($s['unit'] ?? null, ['W','kWh','percent'], true)) {
throw new InvalidArgumentException('Invalid or duplicate raw source');
}
$keys[$s['key']] = true; $ids[$s['variableId']] = true;
}
usort($c['extraSources'], static fn(array $a, array $b): int => strcmp($a['key'], $b['key']));
return $c;
}
private static function readSource(array $s, callable $read): array
{
try {
$r = $read($s['variableId']);
if (!is_array($r)) throw new RuntimeException('Missing reading');
} catch (Throwable $e) {
return ['value' => null, 'sourceUpdatedAt' => null, 'sourceChangedAt' => null, 'issues' => ['read_failed']];
}
$issues = [];
$v = $r['value'] ?? null;
if ((!is_int($v) && !is_float($v)) || !is_finite((float) $v) || abs((float)$v) > 1.0e12) {
$v = null; $issues[] = 'invalid_numeric_value';
}
$at = $r['updated'] ?? null;
if (!is_int($at) || $at < 1) { $at = null; $issues[] = 'missing_original_timestamp'; }
$changed = $r['changed'] ?? null;
if (!is_int($changed) || $changed < 0) $changed = null;
if (($r['parentID'] ?? null) !== $s['parentId'] || ($r['ident'] ?? null) !== $s['ident']) {
// Do not record a numeric value from an object re-used for a different purpose.
$v = null; $issues[] = 'source_identity_mismatch';
}
return ['value' => $v, 'sourceUpdatedAt' => $at, 'sourceChangedAt' => $changed, 'issues' => $issues];
}
public static function capture(array $config, callable $read, callable $clock): array
{
$c = self::configuration($config);
$start = $clock();
if (!is_int($start) || $start <= 0) throw new InvalidArgumentException('Integer UTC epoch required');
$sources = array_merge($c['accounting']['sources'], $c['extraSources']);
$first = []; $second = []; $raw = []; $issues = [];
foreach ($sources as $s) $first[$s['key']] = self::readSource($s, $read);
foreach ($sources as $s) $second[$s['key']] = self::readSource($s, $read);
$end = $clock();
if (!is_int($end) || $end < $start) throw new RuntimeException('Acquisition clock moved backwards');
foreach ($sources as $s) {
$key = $s['key']; $r = $first[$key];
if ($r !== $second[$key]) {
$r['issues'][] = 'changed_during_capture';
$r['secondValue'] = $second[$key]['value'];
$r['secondUpdatedAt'] = $second[$key]['sourceUpdatedAt'];
}
if ($r['sourceUpdatedAt'] !== null && $r['sourceUpdatedAt'] > $end) $r['issues'][] = 'future_source';
$r['sourceAgeSeconds'] = $r['sourceUpdatedAt'] === null ? null : $end - $r['sourceUpdatedAt'];
$raw[$key] = ['variableId' => $s['variableId']] + $r;
foreach ($r['issues'] as $issue) $issues[] = $key . ':' . $issue;
}
$assessment = null; $accountingUsable = true; $samples = [];
foreach ($c['accounting']['sources'] as $s) {
$r = $raw[$s['key']];
if ($r['issues'] !== []) $accountingUsable = false;
$samples[$s['variableId']] = ['value' => $r['value'], 'updated' => $r['sourceUpdatedAt'], 'parentID' => $s['parentId'], 'ident' => $s['ident']];
}
if ($accountingUsable) {
try { $assessment = NetzfahrplanV4Bilanzierung::aufnehmen($c['accounting'], static fn(int $id): array => $samples[$id], $end); }
catch (Throwable $e) { $issues[] = 'accounting_input_invalid'; }
} else { $issues[] = 'accounting_snapshot_unusable'; }
if ($end - $start > 5) $issues[] = 'long_acquisition';
return ['schemaVersion' => 1, 'kind' => 'raw_accounting_capture', 'installationId' => $c['installationId'],
'captureStartedAt' => gmdate('c', $start), 'capturedAt' => gmdate('c', $end),
'mappingSha256' => hash('sha256', json_encode($c, JSON_THROW_ON_ERROR | JSON_PRESERVE_ZERO_FRACTION)),
'reportedInventorySha256' => $c['reportedInventorySha256'],
'timestampMeaning' => 'Symcon VariableUpdated; not proof of an independent device timestamp',
'raw' => $raw, 'issues' => $issues, 'assessment' => $assessment,
'trainingEligible' => false, 'controlEligible' => false, 'measurementBoundaryVerified' => false,
'futureSdl' => 'unknown_no_assumption', 'historicalDataRewritten' => false];
}
/** Append-only daily JSONL with exclusive lock, bounds and crash-tail detection.
* No retention deletion; quota stops capture visibly, never deletes old histories.
*/
public static function append(string $directory, array $record, int $quota = 536870912): string
{
if (!is_dir($directory) || is_link($directory) || realpath($directory) !== $directory || $quota < 1) {
throw new RuntimeException('Invalid dedicated data directory');
}
$json = json_encode($record, JSON_THROW_ON_ERROR | JSON_PRESERVE_ZERO_FRACTION) . "\n";
if (strlen($json) > 262144) throw new RuntimeException('Capture exceeds record limit');
$epoch = strtotime($record['capturedAt'] ?? '');
if ($epoch === false) throw new RuntimeException('Capture timestamp missing');
$file = $directory . '/raw-' . gmdate('Ymd', $epoch) . '.jsonl';
$lockPath = $directory . '/.writer.lock';
if (is_link($lockPath) || is_link($file)) throw new RuntimeException('Symbolic link refused');
$lock = fopen($lockPath, 'c+b');
if ($lock === false) throw new RuntimeException('Capture lock unavailable');
try {
if (!flock($lock, LOCK_EX | LOCK_NB)) throw new RuntimeException('Capture writer busy');
clearstatcache(); $used = 0; $files = glob($directory . '/raw-*.jsonl');
if ($files === false || count($files) > 400) throw new RuntimeException('Data directory bounds exceeded');
foreach ($files as $p) {
if (is_link($p)) throw new RuntimeException('Symbolic journal refused');
$size = filesize($p); if ($size === false) throw new RuntimeException('Cannot size journal');
$used += $size;
}
if ($used + strlen($json) > $quota) throw new RuntimeException('Capture quota reached; no data deleted');
$free = disk_free_space($directory);
if ($free === false || $free < 104857600 + strlen($json)) throw new RuntimeException('Insufficient free disk; capture paused');
$h = fopen($file, 'c+b');
if ($h === false) throw new RuntimeException('Cannot open capture journal');
try {
fseek($h, 0, SEEK_END); $old = ftell($h);
if ($old === false || $old + strlen($json) > 67108864) throw new RuntimeException('Daily capture bounds exceeded');
if ($old > 0) {
fseek($h, -1, SEEK_END);
if (fread($h, 1) !== "\n") throw new RuntimeException('Incomplete journal tail; preserve for review');
}
fseek($h, 0, SEEK_END); $n = 0;
try {
while ($n < strlen($json)) {
$written = fwrite($h, substr($json, $n));
if ($written === false || $written === 0) throw new RuntimeException('Capture write incomplete');
$n += $written;
}
if (!fflush($h)) throw new RuntimeException('Capture flush failed');
} catch (Throwable $e) { ftruncate($h, $old); fflush($h); throw $e; }
if (!chmod($file, 0640)) throw new RuntimeException('Cannot restrict journal permissions');
} finally { fclose($h); }
return $file;
} finally { flock($lock, LOCK_UN); fclose($lock); }
}
}
tests/NetzfahrplanV4MessaufnahmeTest.php
+16
View File
@@ -0,0 +1,16 @@
<?php
declare(strict_types=1);
namespace Belevo\EnelixEMS\Tests;
use PHPUnit\Framework\TestCase;
final class NetzfahrplanV4MessaufnahmeTest extends TestCase
{
/** @dataProvider scripts */
public function testPassiveCapture(string $name): void
{
$pipes=[];$p=proc_open([PHP_BINARY,__DIR__.'/V4DataCapture/'.$name],[0=>['pipe','r'],1=>['pipe','w'],2=>['pipe','w']],$pipes);
self::assertIsResource($p);fclose($pipes[0]);$out=stream_get_contents($pipes[1]);$err=stream_get_contents($pipes[2]);fclose($pipes[1]);fclose($pipes[2]);
self::assertSame(0,proc_close($p),$out.$err);self::assertStringContainsString('TOTAL',$out);
}
public function scripts(): array { return [['checks.php'],['native_checks.php']]; }
}
tests/V4DataCapture/checks.php
+62
View File
@@ -0,0 +1,62 @@
<?php
declare(strict_types=1);
require_once __DIR__ . '/../../libs/NetzfahrplanV4Messaufnahme.php';
use Belevo\EnelixEMS\NetzfahrplanV4Messaufnahme as Capture;
$n = 0;
function ck(bool $ok, string $label): void { global $n; if (!$ok) throw new RuntimeException('FAIL ' . $label); $n++; echo 'PASS ' . $label . "\n"; }
function refuses(callable $f, string $label): void { try { $f(); } catch (Throwable $e) { ck(true,$label); return; } throw new RuntimeException('Expected refusal: '.$label); }
$now = 1790952000;
$roles=['grid','pv','physical_storage','controlled_virtual','external_virtual'];
$c=['version'=>1,'installationId'=>'e3a08f9e-af12-4695-99bd-8b51c0520021','managerId'=>17004,
'reportedInventorySha256'=>str_repeat('a',64),'accounting'=>['version'=>1,'maxSkewSeconds'=>30,'splitToleranceW'=>100.,'sources'=>[]],
'extraSources'=>[['key'=>'soc','variableId'=>106,'parentId'=>200,'ident'=>'soc','unit'=>'percent']]];
$values=[3000,5000,2000,1500,500]; $readings=[];
foreach($roles as $i=>$role){
$key='s'.$i;$id=101+$i;
$c['accounting']['sources'][]=['key'=>$key,'role'=>$role,'variableId'=>$id,'parentId'=>200,'ident'=>$key,'factorToW'=>1.,'maxAgeSeconds'=>60,'dependsOn'=>in_array($i,[3,4],true)?['s2']:[]];
$readings[$id]=['value'=>$values[$i],'updated'=>$now,'changed'=>$now-30,'parentID'=>200,'ident'=>$key];
}
$readings[106]=['value'=>0.,'updated'=>$now,'changed'=>0,'parentID'=>200,'ident'=>'soc'];
function take(array $c,array $r,int $now): array { return Capture::capture($c,static fn(int $id):array=>$r[$id],static fn():int=>$now); }
$r=take($c,$readings,$now);
ck($r['assessment']['baseLoadCandidateW']===6000.,'physical battery subtracted exactly once');
ck($r['raw']['soc']['value']===0. && $r['raw']['soc']['sourceChangedAt']===0,'real zero and unchanged timestamp retained');
ck(!$r['trainingEligible'] && !$r['controlEligible'] && !$r['measurementBoundaryVerified'],'algebra never enables training or control');
ck($r['futureSdl']==='unknown_no_assumption','no fabricated future SDL');
ck(count($r['raw'])===6,'raw extra sources retained separately');
$a=$readings;$a[103]['updated']=$now-90;$a[104]['updated']=$now;
$x=take($c,$a,$now);ck($x['assessment']['status']==='quality_hold','old physical measurement quarantines candidate');
ck($x['raw']['s2']['sourceUpdatedAt']===$now-90,'original old timestamp preserved');
ck(in_array('stale_origin:s3',$x['assessment']['issues'],true),'fresh virtual write cannot mask old dependency');
$a=$readings;$a[101]['value']=null;$x=take($c,$a,$now);ck($x['raw']['s0']['value']===null && $x['assessment']===null,'missing source not zero');
$a=$readings;$a[101]['value']='sensitive-test-text';$x=take($c,$a,$now);ck(!str_contains(json_encode($x),'sensitive-test-text'),'unexpected strings never logged');
foreach([true,NAN,INF,1.0e20] as $v){$a=$readings;$a[101]['value']=$v;$x=take($c,$a,$now);ck($x['raw']['s0']['value']===null && is_string(json_encode($x,JSON_THROW_ON_ERROR)),'invalid numeric represented safely');}
$a=$readings;$a[101]['parentID']=999;$x=take($c,$a,$now);ck($x['raw']['s0']['value']===null && $x['assessment']===null,'reused ID not recorded as correct measurement');
$a=$readings;$a[102]['updated']=$now+1;$x=take($c,$a,$now);ck($x['assessment']===null && in_array('s1:future_source',$x['issues'],true),'future timestamp marked');
$a=$readings;$a[101]['updated']=0;$x=take($c,$a,$now);ck($x['assessment']===null,'absent timestamp not invented');
$counts=[];$x=Capture::capture($c,static function($id)use(&$counts,$readings){$r=$readings[$id];$counts[$id]=($counts[$id]??0)+1;if($id===101 && $counts[$id]===2)$r['value']++;return $r;},static fn()=>$now);
ck($x['assessment']===null && $x['raw']['s0']['secondValue']===3001,'same-second concurrent change visible');
$x=Capture::capture($c,static function($id)use($readings){if($id===101)throw new RuntimeException('do-not-emit-secret');return $readings[$id];},static fn()=>$now);
ck(!str_contains(json_encode($x),'do-not-emit-secret') && $x['assessment']===null,'source errors bounded without secret messages');
$a=$readings;$a[106]['value']=null;$x=take($c,$a,$now);ck($x['assessment']!==null && !$x['trainingEligible'],'optional SOC error does not destroy available raw powers');
$a=$readings;$a[101]['value']=-10000;$x=take($c,$a,$now);ck($x['assessment']['baseLoadCandidateW']<0 && $x['assessment']['status']==='quality_hold','negative candidate not clipped');
$a=$c;$a['extraSources'][]=$a['extraSources'][0];refuses(fn()=>take($a,$readings,$now),'duplicate source rejected');
$a=$c;$a['accounting']['sources']=array_reverse($a['accounting']['sources']);ck(take($a,$readings,$now)['mappingSha256']===$r['mappingSha256'],'mapping identity stable');
$t=0;refuses(fn()=>Capture::capture($c,static fn($id)=>$readings[$id],static function()use(&$t,$now){return $now-$t++;}),'backwards clock rejected');
$base=sys_get_temp_dir().'/v4-raw-test-'.bin2hex(random_bytes(5));mkdir($base,0700);$dir=$base.'/data';mkdir($dir,0700);
try{
$file=Capture::append($dir,$r);$first=file_get_contents($file);Capture::append($dir,$r);
ck(file_get_contents($file)===$first.$first,'append retains existing observations');
ck(count(file($file))===2,'one record per line');
ck((fileperms($file)&0777)===0640,'data permissions restricted');
$old=file_get_contents($file);refuses(fn()=>Capture::append($dir,$r,1),'quota refuses without deleting data');ck(file_get_contents($file)===$old,'quota leaves journal unchanged');
file_put_contents($file,'partial',FILE_APPEND);refuses(fn()=>Capture::append($dir,$r),'crash-truncated tail not silently extended');ck(str_ends_with(file_get_contents($file),'partial'),'incomplete record retained for review');
unlink($file);file_put_contents($base.'/foreign','untouched');symlink($base.'/foreign',$file);refuses(fn()=>Capture::append($dir,$r),'journal symlink refused');ck(file_get_contents($base.'/foreign')==='untouched','unrelated file preserved');unlink($file);
$lock=fopen($dir.'/.writer.lock','c+b');flock($lock,LOCK_EX);refuses(fn()=>Capture::append($dir,$r),'parallel writer refused');flock($lock,LOCK_UN);fclose($lock);
$a=$r;$a['blob']=str_repeat('x',300000);refuses(fn()=>Capture::append($dir,$a),'oversized record refused');
}finally{foreach(new RecursiveIteratorIterator(new RecursiveDirectoryIterator($base,FilesystemIterator::SKIP_DOTS),RecursiveIteratorIterator::CHILD_FIRST)as$p){$p->isDir()&&!$p->isLink()?rmdir($p->getPathname()):unlink($p->getPathname());}rmdir($base);}
$source=file_get_contents(__DIR__.'/../../libs/NetzfahrplanV4Messaufnahme.php');
ck(!preg_match('/\b(?:IPS_|GetValue|SetValue|curl_|RequestAction|shell_exec|exec\s*\()/',$source),'capture core has no device or network calls');
echo "TOTAL $n checks passed; synthetic values and temporary files only.\n";
tests/V4DataCapture/native_checks.php
+61
View File
@@ -0,0 +1,61 @@
<?php
declare(strict_types=1);
// ALL IPS functions below are mocks. Never execute this test in a real Symcon kernel.
if (PHP_SAPI !== 'cli') throw new RuntimeException('CLI isolation required');
require_once __DIR__ . '/../../examples/V4DataCapture/install.php';
$count=0;$objects=[];$values=[];$scripts=[];$timers=[];$nextId=60000;$locks=[];$timerCalls=[];
function yes(bool $v,string $s):void{global$count;if(!$v)throw new RuntimeException('FAIL '.$s);$count++;echo 'PASS '.$s."\n";}
function IPS_GetKernelVersion(){return '8.0';}
function IPS_GetObject($id){return $GLOBALS['objects'][$id]??throw new RuntimeException('Missing object');}
function IPS_GetChildrenIDs($id){return array_keys(array_filter($GLOBALS['objects'],static fn($o)=>$o['ParentID']===$id));}
function IPS_InstanceExists($id){return $id===17004;}
function IPS_GetInstance($id){return ['ModuleInfo'=>['ModuleID'=>'{6F771B18-59D4-4C8A-B951-3B2FE9F6A2C4}']];}
function IPS_VariableExists($id){return isset($GLOBALS['values'][$id]);}
function IPS_GetVariable($id){$v=$GLOBALS['values'][$id];return ['VariableType'=>is_string($v)?3:2,'VariableUpdated'=>time(),'VariableChanged'=>time()-10];}
function GetValue($id){return $GLOBALS['values'][$id];}
function SetValue($id,$v){if($id<60000||IPS_GetObject($id)['ObjectIdent']!=='CaptureStatus')throw new RuntimeException('Foreign variable write');$GLOBALS['values'][$id]=$v;return true;}
function IPS_SemaphoreEnter($key,$wait){if($GLOBALS['locks'][$key]??false)return false;$GLOBALS['locks'][$key]=true;return true;}
function IPS_SemaphoreLeave($key){$GLOBALS['locks'][$key]=false;}
function makeObj($type){$id=$GLOBALS['nextId']++;$GLOBALS['objects'][$id]=['ObjectType'=>$type,'ObjectIdent'=>'','ParentID'=>0,'ObjectInfo'=>''];return $id;}
function IPS_CreateCategory(){return makeObj(0);}
function IPS_CreateScript($type){$id=makeObj(3);$GLOBALS['scripts'][$id]='<?php';return $id;}
function IPS_CreateVariable($type){if($type!==3)throw new RuntimeException('Only string status allowed');$id=makeObj(2);$GLOBALS['values'][$id]='';return $id;}
function setObj($id,$key,$v){if($id<60000)throw new RuntimeException('Existing object modified');$GLOBALS['objects'][$id][$key]=$v;return true;}
function IPS_SetParent($id,$v){return setObj($id,'ParentID',$v);}
function IPS_SetIdent($id,$v){return setObj($id,'ObjectIdent',$v);}
function IPS_SetName($id,$v){return setObj($id,'ObjectName',$v);}
function IPS_SetInfo($id,$v){return setObj($id,'ObjectInfo',$v);}
function IPS_GetScriptContent($id){return $GLOBALS['scripts'][$id];}
function IPS_SetScriptContent($id,$v){if($id<60000)throw new RuntimeException('Foreign script write');$GLOBALS['scripts'][$id]=$v;return true;}
function IPS_SetScriptTimer($id,$seconds){if($id<60000||IPS_GetObject($id)['ObjectIdent']!=='RawCapture')throw new RuntimeException('Foreign timer changed');$GLOBALS['timers'][$id]=$seconds;$GLOBALS['timerCalls'][]=[$id,$seconds];return true;}
// A separate fake stage with real candidate code and known synthetic numeric sources.
$base=sys_get_temp_dir().'/v4-native-capture-'.bin2hex(random_bytes(5));mkdir($base,0700);mkdir($base.'/data',0700);
$repo=dirname(__DIR__,2);$names=['libs/NetzfahrplanV4Bilanzierung.php','libs/NetzfahrplanV4Messaufnahme.php','examples/V4DataCapture/runtime.php','examples/V4DataCapture/install.php'];
$hashes=[];
foreach($names as $name){$p=$base.'/source/'.$name;if(!is_dir(dirname($p)))mkdir(dirname($p),0700,true);copy($repo.'/'.$name,$p);$hashes['source/'.$name]=hash_file('sha256',$p);}
$info=['source'=>'synthetic_test','physicalKwh'=>322,'evKwh'=>160];file_put_contents($base.'/USER_REPORTED_INVENTORY.json',json_encode($info));
$c=['version'=>1,'installationId'=>'e3a08f9e-af12-4695-99bd-8b51c0520021','managerId'=>17004,'reportedInventorySha256'=>hash_file('sha256',$base.'/USER_REPORTED_INVENTORY.json'),
'extraSources'=>[],'accounting'=>['version'=>1,'maxSkewSeconds'=>30,'splitToleranceW'=>100.,'sources'=>[]]];
foreach(['grid','pv','physical_storage','controlled_virtual','external_virtual']as$i=>$role){$id=101+$i;$c['accounting']['sources'][]=['key'=>'x'.$i,'variableId'=>$id,'parentId'=>200,'ident'=>'v'.$i,'role'=>$role,'factorToW'=>1,'maxAgeSeconds'=>60,'dependsOn'=>[]];$objects[$id]=['ParentID'=>200,'ObjectIdent'=>'v'.$i,'ObjectType'=>2,'ObjectInfo'=>'original'];$values[$id]=[1000,2000,500,500,0][$i];}
file_put_contents($base.'/config.json',json_encode($c));$hashes['config.json']=hash_file('sha256',$base.'/config.json');$hashes['USER_REPORTED_INVENTORY.json']=hash_file('sha256',$base.'/USER_REPORTED_INVENTORY.json');
file_put_contents($base.'/MANIFEST.json',json_encode(['scope'=>'passive_raw_capture_only','hashes'=>$hashes]));
$before=$objects;$beforeValues=$values;
try{
ob_start();$r=v4captureInstall($base);ob_end_clean();
yes($r['status']==='capture_installed','passive installation completes');
yes(count($r['createdObjects'])===4,'only category, two scripts and one status variable');
yes($timers[$r['scriptId']]===30,'only new capture timer runs at 30 seconds');
yes($r['firstCapture']['status']==='recorded'&&count(file($r['firstCapture']['journal']))===1,'first capture actually written before activation');
yes($r['firstCapture']['accountingQuality']==='consistent_candidate','candidate is not a training permit');
foreach($before as$id=>$o){yes($objects[$id]===$o && $values[$id]===$beforeValues[$id],'existing source preserved '.$id);}
$size=count($objects);ob_start();$again=v4captureInstall($base);ob_end_clean();yes(count($objects)===$size&&$again['createdObjects']===[],'installation idempotent, no duplicate timers');
$bad=$scripts[$r['scriptId']];$scripts[$r['scriptId']]="<?php // concurrent edit\n";ob_start();$err=v4captureInstall($base);ob_end_clean();yes($err['status']==='needs_review'&&$scripts[$r['scriptId']]!==$bad,'parallel script edit refused');$scripts[$r['scriptId']]=$bad;
$values[101]=null;$raw=v4captureRun($base,$r['categoryId']);yes($raw['status']==='recorded'&&$raw['accountingQuality']==='unavailable','missing measurement still records an explicit gap');$values[101]=1000;
v4captureStop($base,$r['categoryId']);yes($timers[$r['scriptId']]===0,'stop affects capture timer only');yes(file_exists($r['firstCapture']['journal']),'stop retains original journal');
ob_start();$resume=v4captureInstall($base);ob_end_clean();yes($resume['status']==='capture_installed'&&$timers[$r['scriptId']]===30,'restart by installer retains data');
$objects[$r['categoryId']]['ObjectInfo']='unrelated';ob_start();$err=v4captureInstall($base);ob_end_clean();yes($err['status']==='needs_review','foreign category identity not taken over');
foreach(['runtime.php','install.php']as$f){$s=file_get_contents($repo.'/examples/V4DataCapture/'.$f);yes(!preg_match('/\b(?:RequestAction|IPS_RequestAction|IPS_SetProperty|IPS_ApplyChanges|MC_ReloadModule|AC_SetLoggingStatus|curl_exec|shell_exec)\s*\(/',$s),'no control/network/module reload in '.$f);}
}finally{foreach(new RecursiveIteratorIterator(new RecursiveDirectoryIterator($base,FilesystemIterator::SKIP_DOTS),RecursiveIteratorIterator::CHILD_FIRST)as$p){$p->isDir()&&!$p->isLink()?rmdir($p->getPathname()):unlink($p->getPathname());}rmdir($base);}
echo "TOTAL $count native-mock checks passed; no Symcon kernel calls.\n";