Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Bug report
description: Report a reproducible problem in the repository's current behavior
title: "[Bug]: "
body:
- type: textarea
id: summary
attributes:
label: What happened?
description: Describe the problem in one or two clear sentences.
placeholder: The default sample run writes alerts.csv, but summary.json is missing.
validations:
required: true

- type: textarea
id: reproduce
attributes:
label: How do we reproduce it?
description: Include the exact command, config, input path, and any relevant output.
placeholder: |
1. Run `python -m telemetry_window_demo.cli run --config configs/default.yaml`
2. Inspect `data/processed/`
3. Observe ...
validations:
required: true

- type: textarea
id: expected
attributes:
label: What did you expect?
placeholder: summary.json should be written alongside the CSV outputs.

- type: input
id: environment
attributes:
label: Environment
description: OS, Python version, and anything else relevant.
placeholder: Windows 11, Python 3.12
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
blank_issues_enabled: false
28 changes: 28 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Feature request
description: Suggest a small improvement or next demo for the repository
title: "[Feature]: "
body:
- type: textarea
id: proposal
attributes:
label: What would you like to add or improve?
description: Keep it concrete and close to the repository's current scope.
placeholder: Add a host-centric investigation demo that reuses the current sample outputs.
validations:
required: true

- type: textarea
id: why
attributes:
label: Why does it help?
description: Explain how this improves the demo, docs, or portfolio narrative.
placeholder: It would make the rule outputs easier to connect to an analyst workflow.
validations:
required: true

- type: textarea
id: scope
attributes:
label: Scope guardrails
description: Call out anything this request should not turn into.
placeholder: Keep it file-based; no hosted dashboard or alert-routing work.
14 changes: 14 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
## Summary

- what changed
- why it changed

## Validation

- [ ] `pytest`
- [ ] README or docs updated if behavior, commands, or outputs changed

## Scope Check

- [ ] Small coherent change
- [ ] No production-readiness claims added
24 changes: 24 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Contributing

Thanks for the interest. This is a solo-maintainer portfolio repository, so the most helpful contributions are small, scoped, and easy to review.

## Good Contributions

- bug reports with exact commands, configs, and sample inputs
- docs fixes when README behavior drifts from the actual CLI
- small tests around time parsing, window boundaries, and alert thresholds
- narrowly scoped pull requests that keep the repository local and file-based

## Before Opening A Pull Request

- keep the project boundary honest: this is not a production monitoring system
- prefer fixing or clarifying existing behavior over adding large new subsystems
- update README or docs if a user-visible command, output, or artifact changes
- run:

```bash
python -m pip install -e .
pytest
```

If you want to propose a larger demo or roadmap item, opening an issue first is the easiest path.
164 changes: 90 additions & 74 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,72 +2,25 @@

[![CI](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml)

Latest release: [v0.3.0](https://github.com/stacknil/telemetry-lab/releases/tag/v0.3.0) adds more precise cooldown scoping, a richer bundled scenario pack, and machine-readable run summaries for reusable local demos.

Small prototypes for telemetry analytics, monitoring, and detection-oriented signal processing.

## Current demo

`telemetry-window-demo` turns timestamped event streams into sliding-window telemetry features, simple rule-based alerts, and operator-friendly CSV and PNG outputs.

## MVP workflow

1. Install the package and its minimal dependencies:

```bash
python -m pip install -e .
```

2. Run the bundled sample pipeline end-to-end:

```bash
python -m telemetry_window_demo.cli run --config configs/default.yaml
```

The sample config reads `data/raw/sample_events.jsonl` and regenerates outputs in `data/processed/`.

For a richer scenario pack that is easier to walk through in demos:
Small portfolio prototypes for telemetry analytics, monitoring, and detection-oriented signal processing.

```bash
python -m telemetry_window_demo.cli run --config configs/richer_sample.yaml
```
## What This Repo Is

That scenario pack reads `data/raw/richer_sample_events.jsonl` and writes outputs to `data/processed/richer_sample/`.
It currently produces `28` normalized events, `24` windows, and `8` alerts.
Both sample paths also emit a compact `summary.json` alongside the CSV and PNG outputs.

## Current behavior

Default sample input:

- JSONL event stream under `data/raw/sample_events.jsonl`

Runtime input support:

- `.jsonl` (default sample/demo format)
- `.csv` (also supported by the loader)

Required fields for both formats on every row/record:

- `timestamp`
- `event_type`
- `source`
- `target`
- `status`

With the bundled sample data, the default run currently produces:
`telemetry-window-demo` is a local Python CLI that turns timestamped event streams into:

- `41` normalized events
- `24` windows
- `12` alerts after applying a `60` second cooldown
- sliding-window feature tables
- cooldown-reduced rule-based alerts
- PNG timeline plots
- machine-readable run summaries

The default config suppresses repeated alerts by cooldown key. The key is `rule_name` plus an entity scope when the rule input includes `entity`, `source`, `target`, or `host`; otherwise it falls back to `rule_name` alone. Different cooldown keys can still alert on the same window.
## Quick Run

The richer scenario pack uses a longer `120` second cooldown so the output stays compact enough to inspect as four phases: normal background activity, a login-failure burst, a high-risk configuration change with follow-on policy denials, and a rare malware-alert repeat sequence.

## Outputs

Running the default command regenerates:
```bash
python -m pip install -e .
python -m telemetry_window_demo.cli run --config configs/default.yaml
```

That command reads `data/raw/sample_events.jsonl` and regenerates:

- `data/processed/features.csv`
- `data/processed/alerts.csv`
Expand All @@ -76,16 +29,79 @@ Running the default command regenerates:
- `data/processed/error_rate_timeline.png`
- `data/processed/alerts_timeline.png`

The summary artifact includes the input path, output directory, normalized event count, window count, feature row count, alert count, triggered rule names and counts, cooldown setting, and generated artifact paths.

## Scope

This repository is a portfolio prototype, not a production monitoring system.

## Limitations

- No real-time ingestion
- No streaming state management
- No alert routing or case management
- No dashboard or service deployment
- Sample-data driven only
With the bundled default sample, the current repo state produces:

- `41` normalized events
- `24` windows
- `12` alerts after a `60` second cooldown

Why it is worth a quick look:

- it shows a full telemetry path from raw events to operator-facing outputs
- the sample inputs and outputs are reproducible in-repo
- a second bundled scenario gives a slightly richer walkthrough without changing the basic CLI flow

![Default alert timeline](data/processed/alerts_timeline.png)

## Demo Variants

Default sample:

- config: [`configs/default.yaml`](configs/default.yaml)
- input: `data/raw/sample_events.jsonl`
- outputs: `data/processed/`
- current summary: `41` events, `24` windows, `12` alerts, `summary.json` included

Richer sample:

- config: [`configs/richer_sample.yaml`](configs/richer_sample.yaml)
- input: `data/raw/richer_sample_events.jsonl`
- outputs: `data/processed/richer_sample/`
- current summary: `28` events, `24` windows, `8` alerts, `summary.json` included

## Input Support

Runtime input support:

- `.jsonl`
- `.csv`

Required fields for both formats on every row or record:

- `timestamp`
- `event_type`
- `source`
- `target`
- `status`

Cooldown behavior:

- repeated alerts are keyed by `(rule_name, scope)`
- scope prefers the first available entity-like field in this order: `entity`, `source`, `target`, `host`
- when no entity-like field is present, cooldown falls back to per-`rule_name` behavior

## Repo Guide

- [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts
- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions
- [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form
- [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack
- [`tests/`](tests/) keeps regression coverage close to the CLI behavior and windowing logic

## Next Demo Directions

- strengthen JSONL and CSV validation so ingestion failures are clearer
- keep reducing repeated alert noise while preserving simple rule-based behavior
- keep sample-output docs and public repo presentation aligned with the checked-in demo state

## Scope

This repository is a portfolio prototype, not a production monitoring system.

## Limitations

- No real-time ingestion
- No streaming state management
- No alert routing or case management
- No dashboard or service deployment
- Sample-data driven only
14 changes: 14 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Security Policy

This repository is a local, sample-data portfolio prototype. It is not a hosted service and does not come with production security guarantees.

## Reporting A Vulnerability

Please avoid posting secrets, credentials, or detailed exploit steps in a public issue.

- If GitHub private vulnerability reporting is available for this repository, use it for sensitive reports.
- Otherwise, open a minimal public issue that omits exploit details and asks for a private follow-up path.

## What Is In Scope

Useful reports are issues that could affect local file handling, sample-data processing, dependency safety, or accidental credential exposure in the repository.
27 changes: 27 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Roadmap

This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out.

## 1. Auth/Login Anomaly Triage Demo

Goal:
Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration.

Why it helps the portfolio:
This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation.

## 2. Rule Evaluation And Dedup Demo

Goal:
Add a demo or explainer that makes rule matches, cooldown behavior, and alert suppression easier to inspect side by side.

Why it helps the portfolio:
This makes the repository's alert semantics more legible. It shows that the project is not only producing alerts, but also exposing how repeated windows become fewer, more explainable notifications.

## 3. Config-Change Investigation Demo

Goal:
Add a compact scenario centered on risky configuration changes, follow-on policy denials, and a short machine-readable investigation summary.

Why it helps the portfolio:
This broadens the repo beyond auth-only behavior while staying inside the same local, file-based pipeline. It gives the project a second clear demo narrative that is still easy to explain from committed sample data.
44 changes: 36 additions & 8 deletions docs/sample-output.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,51 @@
# Sample Output

Running the default pipeline produces:
The committed sample artifacts are intended to be reproducible from the bundled inputs and configs.

## Default Sample

Running `python -m telemetry_window_demo.cli run --config configs/default.yaml` produces:

- a window feature table at `data/processed/features.csv`
- an alert table at `data/processed/alerts.csv`
- a machine-readable summary at `data/processed/summary.json`
- three timeline plots under `data/processed/`

On the bundled sample dataset, the default config produces:
On the bundled default sample dataset, the current repo state produces:

- `41` input events
- `41` normalized events
- `24` sliding windows
- `53` alerts across rule categories
- `12` alerts after a `60` second cooldown

The default summary currently reports these triggered rule counts:

- `high_error_rate`: `3`
- `persistent_high_error`: `3`
- `high_severity_spike`: `2`
- `login_fail_burst`: `2`
- `source_spread_spike`: `1`
- `rare_event_repeat_malware_alert`: `1`

## Richer Sample

The sample is intentionally bursty so the plots and alerts are visually obvious in a portfolio setting.
Running `python -m telemetry_window_demo.cli run --config configs/richer_sample.yaml` produces:

- a window feature table at `data/processed/richer_sample/features.csv`
- an alert table at `data/processed/richer_sample/alerts.csv`
- a machine-readable summary at `data/processed/richer_sample/summary.json`
- three timeline plots under `data/processed/richer_sample/`

On the richer bundled sample dataset, the current repo state produces:

- `28` normalized events
- `24` sliding windows
- `8` alerts after a `120` second cooldown

Representative alert categories in the sample dataset:
Representative alert categories across the bundled samples:

- elevated error rate during the login failure burst
- repeated high-severity events around `malware_alert`
- sudden source spread as the number of distinct sources increases
- sudden source spread as the number of distinct sources increases in the default sample
- repeated rare-event alerts for both `malware_alert` and `policy_denied` in the richer sample

See the generated assets in `assets/` for README-friendly screenshots.
See the committed PNGs under `data/processed/` and `data/processed/richer_sample/` for GitHub-visible output snapshots.
Loading