diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..9f8651c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,37 @@ +name: Bug report +description: Report a reproducible problem in the repository's current behavior +title: "[Bug]: " +body: + - type: textarea + id: summary + attributes: + label: What happened? + description: Describe the problem in one or two clear sentences. + placeholder: The default sample run writes alerts.csv, but summary.json is missing. + validations: + required: true + + - type: textarea + id: reproduce + attributes: + label: How do we reproduce it? + description: Include the exact command, config, input path, and any relevant output. + placeholder: | + 1. Run `python -m telemetry_window_demo.cli run --config configs/default.yaml` + 2. Inspect `data/processed/` + 3. Observe ... + validations: + required: true + + - type: textarea + id: expected + attributes: + label: What did you expect? + placeholder: summary.json should be written alongside the CSV outputs. + + - type: input + id: environment + attributes: + label: Environment + description: OS, Python version, and anything else relevant. + placeholder: Windows 11, Python 3.12 diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..3ba13e0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..7032bff --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,28 @@ +name: Feature request +description: Suggest a small improvement or next demo for the repository +title: "[Feature]: " +body: + - type: textarea + id: proposal + attributes: + label: What would you like to add or improve? + description: Keep it concrete and close to the repository's current scope. + placeholder: Add a host-centric investigation demo that reuses the current sample outputs. + validations: + required: true + + - type: textarea + id: why + attributes: + label: Why does it help? + description: Explain how this improves the demo, docs, or portfolio narrative. + placeholder: It would make the rule outputs easier to connect to an analyst workflow. + validations: + required: true + + - type: textarea + id: scope + attributes: + label: Scope guardrails + description: Call out anything this request should not turn into. + placeholder: Keep it file-based; no hosted dashboard or alert-routing work. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..7ff06cd --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,14 @@ +## Summary + +- what changed +- why it changed + +## Validation + +- [ ] `pytest` +- [ ] README or docs updated if behavior, commands, or outputs changed + +## Scope Check + +- [ ] Small coherent change +- [ ] No production-readiness claims added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..554540b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,24 @@ +# Contributing + +Thanks for the interest. This is a solo-maintainer portfolio repository, so the most helpful contributions are small, scoped, and easy to review. + +## Good Contributions + +- bug reports with exact commands, configs, and sample inputs +- docs fixes when README behavior drifts from the actual CLI +- small tests around time parsing, window boundaries, and alert thresholds +- narrowly scoped pull requests that keep the repository local and file-based + +## Before Opening A Pull Request + +- keep the project boundary honest: this is not a production monitoring system +- prefer fixing or clarifying existing behavior over adding large new subsystems +- update README or docs if a user-visible command, output, or artifact changes +- run: + + ```bash + python -m pip install -e . + pytest + ``` + +If you want to propose a larger demo or roadmap item, opening an issue first is the easiest path. diff --git a/README.md b/README.md index 258d11f..437e779 100644 --- a/README.md +++ b/README.md @@ -2,72 +2,25 @@ [![CI](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml) -Latest release: [v0.3.0](https://github.com/stacknil/telemetry-lab/releases/tag/v0.3.0) adds more precise cooldown scoping, a richer bundled scenario pack, and machine-readable run summaries for reusable local demos. - -Small prototypes for telemetry analytics, monitoring, and detection-oriented signal processing. - -## Current demo - -`telemetry-window-demo` turns timestamped event streams into sliding-window telemetry features, simple rule-based alerts, and operator-friendly CSV and PNG outputs. - -## MVP workflow - -1. Install the package and its minimal dependencies: - - ```bash - python -m pip install -e . - ``` - -2. Run the bundled sample pipeline end-to-end: - - ```bash - python -m telemetry_window_demo.cli run --config configs/default.yaml - ``` - -The sample config reads `data/raw/sample_events.jsonl` and regenerates outputs in `data/processed/`. - -For a richer scenario pack that is easier to walk through in demos: +Small portfolio prototypes for telemetry analytics, monitoring, and detection-oriented signal processing. -```bash -python -m telemetry_window_demo.cli run --config configs/richer_sample.yaml -``` +## What This Repo Is -That scenario pack reads `data/raw/richer_sample_events.jsonl` and writes outputs to `data/processed/richer_sample/`. -It currently produces `28` normalized events, `24` windows, and `8` alerts. -Both sample paths also emit a compact `summary.json` alongside the CSV and PNG outputs. - -## Current behavior - -Default sample input: - -- JSONL event stream under `data/raw/sample_events.jsonl` - -Runtime input support: - -- `.jsonl` (default sample/demo format) -- `.csv` (also supported by the loader) - -Required fields for both formats on every row/record: - -- `timestamp` -- `event_type` -- `source` -- `target` -- `status` - -With the bundled sample data, the default run currently produces: +`telemetry-window-demo` is a local Python CLI that turns timestamped event streams into: -- `41` normalized events -- `24` windows -- `12` alerts after applying a `60` second cooldown +- sliding-window feature tables +- cooldown-reduced rule-based alerts +- PNG timeline plots +- machine-readable run summaries -The default config suppresses repeated alerts by cooldown key. The key is `rule_name` plus an entity scope when the rule input includes `entity`, `source`, `target`, or `host`; otherwise it falls back to `rule_name` alone. Different cooldown keys can still alert on the same window. +## Quick Run -The richer scenario pack uses a longer `120` second cooldown so the output stays compact enough to inspect as four phases: normal background activity, a login-failure burst, a high-risk configuration change with follow-on policy denials, and a rare malware-alert repeat sequence. - -## Outputs - -Running the default command regenerates: +```bash +python -m pip install -e . +python -m telemetry_window_demo.cli run --config configs/default.yaml +``` + +That command reads `data/raw/sample_events.jsonl` and regenerates: - `data/processed/features.csv` - `data/processed/alerts.csv` @@ -76,16 +29,79 @@ Running the default command regenerates: - `data/processed/error_rate_timeline.png` - `data/processed/alerts_timeline.png` -The summary artifact includes the input path, output directory, normalized event count, window count, feature row count, alert count, triggered rule names and counts, cooldown setting, and generated artifact paths. - -## Scope - -This repository is a portfolio prototype, not a production monitoring system. - -## Limitations - -- No real-time ingestion -- No streaming state management -- No alert routing or case management -- No dashboard or service deployment -- Sample-data driven only +With the bundled default sample, the current repo state produces: + +- `41` normalized events +- `24` windows +- `12` alerts after a `60` second cooldown + +Why it is worth a quick look: + +- it shows a full telemetry path from raw events to operator-facing outputs +- the sample inputs and outputs are reproducible in-repo +- a second bundled scenario gives a slightly richer walkthrough without changing the basic CLI flow + +![Default alert timeline](data/processed/alerts_timeline.png) + +## Demo Variants + +Default sample: + +- config: [`configs/default.yaml`](configs/default.yaml) +- input: `data/raw/sample_events.jsonl` +- outputs: `data/processed/` +- current summary: `41` events, `24` windows, `12` alerts, `summary.json` included + +Richer sample: + +- config: [`configs/richer_sample.yaml`](configs/richer_sample.yaml) +- input: `data/raw/richer_sample_events.jsonl` +- outputs: `data/processed/richer_sample/` +- current summary: `28` events, `24` windows, `8` alerts, `summary.json` included + +## Input Support + +Runtime input support: + +- `.jsonl` +- `.csv` + +Required fields for both formats on every row or record: + +- `timestamp` +- `event_type` +- `source` +- `target` +- `status` + +Cooldown behavior: + +- repeated alerts are keyed by `(rule_name, scope)` +- scope prefers the first available entity-like field in this order: `entity`, `source`, `target`, `host` +- when no entity-like field is present, cooldown falls back to per-`rule_name` behavior + +## Repo Guide + +- [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts +- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions +- [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form +- [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack +- [`tests/`](tests/) keeps regression coverage close to the CLI behavior and windowing logic + +## Next Demo Directions + +- strengthen JSONL and CSV validation so ingestion failures are clearer +- keep reducing repeated alert noise while preserving simple rule-based behavior +- keep sample-output docs and public repo presentation aligned with the checked-in demo state + +## Scope + +This repository is a portfolio prototype, not a production monitoring system. + +## Limitations + +- No real-time ingestion +- No streaming state management +- No alert routing or case management +- No dashboard or service deployment +- Sample-data driven only diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..eacab68 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,14 @@ +# Security Policy + +This repository is a local, sample-data portfolio prototype. It is not a hosted service and does not come with production security guarantees. + +## Reporting A Vulnerability + +Please avoid posting secrets, credentials, or detailed exploit steps in a public issue. + +- If GitHub private vulnerability reporting is available for this repository, use it for sensitive reports. +- Otherwise, open a minimal public issue that omits exploit details and asks for a private follow-up path. + +## What Is In Scope + +Useful reports are issues that could affect local file handling, sample-data processing, dependency safety, or accidental credential exposure in the repository. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..9c62246 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,27 @@ +# Roadmap + +This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out. + +## 1. Auth/Login Anomaly Triage Demo + +Goal: +Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration. + +Why it helps the portfolio: +This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation. + +## 2. Rule Evaluation And Dedup Demo + +Goal: +Add a demo or explainer that makes rule matches, cooldown behavior, and alert suppression easier to inspect side by side. + +Why it helps the portfolio: +This makes the repository's alert semantics more legible. It shows that the project is not only producing alerts, but also exposing how repeated windows become fewer, more explainable notifications. + +## 3. Config-Change Investigation Demo + +Goal: +Add a compact scenario centered on risky configuration changes, follow-on policy denials, and a short machine-readable investigation summary. + +Why it helps the portfolio: +This broadens the repo beyond auth-only behavior while staying inside the same local, file-based pipeline. It gives the project a second clear demo narrative that is still easy to explain from committed sample data. diff --git a/docs/sample-output.md b/docs/sample-output.md index 19d4b26..68865da 100644 --- a/docs/sample-output.md +++ b/docs/sample-output.md @@ -1,23 +1,51 @@ # Sample Output -Running the default pipeline produces: +The committed sample artifacts are intended to be reproducible from the bundled inputs and configs. + +## Default Sample + +Running `python -m telemetry_window_demo.cli run --config configs/default.yaml` produces: - a window feature table at `data/processed/features.csv` - an alert table at `data/processed/alerts.csv` +- a machine-readable summary at `data/processed/summary.json` - three timeline plots under `data/processed/` -On the bundled sample dataset, the default config produces: +On the bundled default sample dataset, the current repo state produces: -- `41` input events +- `41` normalized events - `24` sliding windows -- `53` alerts across rule categories +- `12` alerts after a `60` second cooldown + +The default summary currently reports these triggered rule counts: + +- `high_error_rate`: `3` +- `persistent_high_error`: `3` +- `high_severity_spike`: `2` +- `login_fail_burst`: `2` +- `source_spread_spike`: `1` +- `rare_event_repeat_malware_alert`: `1` + +## Richer Sample -The sample is intentionally bursty so the plots and alerts are visually obvious in a portfolio setting. +Running `python -m telemetry_window_demo.cli run --config configs/richer_sample.yaml` produces: + +- a window feature table at `data/processed/richer_sample/features.csv` +- an alert table at `data/processed/richer_sample/alerts.csv` +- a machine-readable summary at `data/processed/richer_sample/summary.json` +- three timeline plots under `data/processed/richer_sample/` + +On the richer bundled sample dataset, the current repo state produces: + +- `28` normalized events +- `24` sliding windows +- `8` alerts after a `120` second cooldown -Representative alert categories in the sample dataset: +Representative alert categories across the bundled samples: - elevated error rate during the login failure burst - repeated high-severity events around `malware_alert` -- sudden source spread as the number of distinct sources increases +- sudden source spread as the number of distinct sources increases in the default sample +- repeated rare-event alerts for both `malware_alert` and `policy_denied` in the richer sample -See the generated assets in `assets/` for README-friendly screenshots. +See the committed PNGs under `data/processed/` and `data/processed/richer_sample/` for GitHub-visible output snapshots.