Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## TLDR of changes


## What kind of changes does this PR introduce?

__Tick all that apply__

- [ ] fix: A bug fix. Correlates with PATCH in SemVer
- [ ] feat: A new feature. Correlates with MINOR in SemVer
- [ ] docs: Documentation only changes
- [ ] style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
- [ ] refactor: A code change that neither fixes a bug nor adds a feature
- [ ] perf: A code change that improves performance
- [ ] test: Adding missing or correcting existing tests
- [ ] build: Changes that affect the build system or external dependencies (example scopes: pip, docker, npm)
- [ ] ci: Changes to CI configuration files and scripts (example scopes: GitLabCI)


## Please check if the PR fulfills these requirements

- [ ] I have read and followed the [Contributing guidance](../CONTRIBUTE.md)
- [ ] Docs have been added / updated
- [ ] Tests and Linting in the CI are passing
- [ ] Changes have been reviewed and approved by a Project Maintainer
46 changes: 37 additions & 9 deletions CONTRIBUTE.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,42 @@
# DVE Contributing guidelines

# Developer information
__If you're planning to contribute to the DVE, please follow all the guidance below. Failure to follow the guidance in this document may result in your contributions being automatically rejected.__

## Getting started

To begin with I would recommend that users read all the documentation available within the [docs](./docs/). It gives an overview of how the DVE works and how to work with the dischema json document.
I would recommend that you read all the documentation available within the [docs](https://nhsdigital.github.io/data-validation-engine/). It gives an overview of how the DVE works and how to work with the dischema json document.

## General requirements
To start contributing to the DVE project you will need the following tooling available:

To start contributing to the DVE project you will need the tooling listed within the `.tool-versions` or `mise.toml`. The following tools are required because...

| Tool | Version | Reason |
| ---- | ------- | ------ |
| `Python` | 3.7.17 | Currently supported version of `Python` for the DVE. |
| `Poetry` | 1.4.2 | Build and venv tool used for the DVE. |
| `Python` | 3.11 | Latest version of Python supported by the DVE. |
| `Poetry` | 2.2.1 | Build and package manager tool used for the DVE. |
| `Java` | java liberica-1.8.0 | `Java` version required for `PySpark`. |
| `pre-commit` | 2.21.0 | Currently installed as part of the `poetry` venv but seperate installation is fine. |
| `commitizen` | 3.9.1 | Like `pre-commit`, installed as part of the `poetry` venv but seperate installation is fine. This is used to manage commits and automated semantic versioning. |
| `pre-commit` | 4.3.0 | Currently installed as part of the `poetry` venv but seperate installation is fine. |
| `commitizen` | 4.9.1 | Like `pre-commit`, installed as part of the `poetry` venv but seperate installation is fine. This is used to manage commits and automated semantic versioning. |
| `git-secrets` | Latest | Utilised as part of the `pre-commit` to ensure that no secrets are commited to the repository. There is a helper installation script within [scripts](/scripts/git-secrets/). |

Additionally, we have created a [asdf support](.tool-versions) and [mise-en-toml](.mise.toml) for those utilising `asdf` or `mise-en-toml` software.
You can install all the developer requirements with the following command:

```bash
poetry install --with dev
```

## Testing Requirements

Testing requirements are given in [pyproject.toml](./poetry.toml#48) under `tool.poetry.group.test.dependencies`. These are always pinned versions for consistency, but should be updated regularly if new versions are released. The following core packages are used for testing:
- [pytest](https://docs.pytest.org/en/stable/): Used for Python unit tests, and some small e2e tests which check coverage.
- [behave](https://github.com/behave/behave): Used for full, business-driven end-to-end tests.
- [coverage](https://coverage.readthedocs.io/en/7.10.7/): Used to get coverage for `pytest` tests.
- [coverage](https://coverage.readthedocs.io/en/): Used to get coverage for `pytest` tests.

You can install these requirements with the following command:

```bash
poetry install --with test
```

## Linting/Formatting/Type Checking Requirements

Expand All @@ -38,6 +50,12 @@ This mostly breaks down to:

We use these tools to ensure that code quality is not excessively compromised, even when working at pace.

You can install these requirements with the following command:

```bash
poetry install --with lint
```

## Installation for Development

We are utilising Poetry for build dependency management and packaging. If you're on a system that has `Make` available, you can simply run `make install` to setup a local virtual environment with all the dependencies installed (this won't install Poetry for you).
Expand All @@ -50,6 +68,16 @@ Tests should be run after installing the package for development as outlined abo
- To check the coverage run `poetry run coverage report -m`
- To run the behave tests, run `poetry run behave tests/features` (these are not included in coverage calculations)

## Committing

We use [commitizen](https://github.com/commitizen-tools/commitizen) to commit new changes. This ensures...

1. A consistent standard for the commit messages
2. Generation of changelog from the commit messages
3. Allows for automatic bumping of the version based on the changes

Please use `poetry run cz c` or `cz c` (if already in the venv).

## Submitting a pull request

If you want to contribute to the DVE then please follow the steps below:
Expand Down
1 change: 1 addition & 0 deletions docs/user_guidance/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ Once you have installed the DVE you are almost ready to use it. To be able to ru

| DVE Version | Python Version | DuckDB Version | Spark Version | Pydantic Version |
| ------------ | -------------- | -------------- | ------------- | ---------------- |
| >=0.7.2 | >=3.10,<3.12 | 1.1.* | 3.4.* | 1.10.16 |
| >=0.6 | >=3.10,<3.12 | 1.1.* | 3.4.* | 1.10.15 |
| >=0.2,<0.6 | >=3.10,<3.12 | 1.1.0 | 3.4.4 | 1.10.15 |
| 0.1 | >=3.7.2,<3.8 | 1.1.0 | 3.2.1 | 1.10.15 |
76 changes: 38 additions & 38 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 20 additions & 7 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
[tool.poetry]
[project]
name = "data-validation-engine"
version = "0.7.1"
dynamic = [ "version" ]
description = "`nhs data validation engine` is a framework used to validate data"
authors = ["NHS England <england.contactus@nhs.net>"]
readme = "README.md"
packages = [
{ include = "dve", from = "src" },
authors = [
{ name = "NHS England", email = "england.contactus@nhs.net" }
]
readme = "README.md"
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
Expand All @@ -15,6 +14,20 @@ classifiers = [
"Topic :: Software Development :: Libraries",
"Typing :: Typed",
]
license = "MIT"

[project.urls]
Homepage = "https://github.com/NHSDigital/data-validation-engine"
Documentation = "https://nhsdigital.github.io/data-validation-engine/"
Repository = "https://github.com/NHSDigital/data-validation-engine.git"
Issues = "https://github.com/NHSDigital/data-validation-engine/issues"
Changelog = "https://github.com/NHSDigital/data-validation-engine/blob/main/CHANGELOG.md"

[tool.poetry]
version = "0.7.1"
packages = [
{ include = "dve", from = "src" },
]

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
Expand All @@ -28,7 +41,7 @@ openpyxl = "^3.1"
pandas = "^2.2.2"
polars = "0.20.*"
pyarrow = "^17.0.0"
pydantic = "1.10.15"
pydantic = "1.10.16"
pyspark = "3.4.*"
typing_extensions = "^4.6.2"

Expand Down
Loading