Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions .github/PULL_REQUEST_TEMPLATE
Original file line number Diff line number Diff line change
@@ -1,16 +1,29 @@
## Summary
<!-- Brief description of what this PR does and why -->

<!-- What is this pull request for? Does it fix any issues? -->
## Changes Made
<!-- List the key changes -->

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change (existing functionality affected)
- [ ] Refactoring (no functional changes)
- [ ] Documentation
- [ ] CI/CD or build configuration
- [ ] Dependencies update

## Testing
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] All existing tests pass
- [ ] Manual testing performed

## Checklist
- [ ] If code changes were made, then they have been tested
- [ ] I have updated the documentation to reflect any changes made
- [ ] I have thought about how this code may affect other services
- [ ] This PR fixes an issue
- [ ] This PR is a breaking change (e.g. method, parameters, env variables)
- [ ] This PR adds something new (e.g. method, parameters, env variables)
- [ ] This PR change unit and integration tests
- [ ] This PR is **NOT** a code change (e.g. documentation, packages)
- [ ] Code follows the project's style and conventions
- [ ] Documentation updated (if applicable)
- [ ] No new warnings or linter errors introduced
- [ ] I have considered how this change may affect other services

## Reviewer
- [ ] I understand that approving this code, I am also responsible for it going into the codebase
- [ ] I understand that by approving this PR, I share responsibility for these changes
11 changes: 5 additions & 6 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ jobs:
uv sync --locked --all-extras --dev --no-install-package mysqlclient --no-install-package aiomysql
elif [[ '${{ matrix.os }}' == 'macos-latest' ]]; then
export PKG_CONFIG_PATH="$(brew --prefix mysql-client)/lib/pkgconfig"
uv sync --locked --all-extras --dev
uv sync --locked --all-extras --group dev
else
uv sync --locked --all-extras --dev
uv sync --locked --all-extras --group dev
fi
shell: bash

Expand All @@ -65,7 +65,7 @@ jobs:
with:
timeout_minutes: 2
max_attempts: 3
command: uv run --no-sync pytest tests/unit
command: uv run --no-sync coverage run -m pytest tests/unit && uv run --no-sync coverage report && uv run --no-sync coverage xml
shell: bash

- name: Upload coverage to Codecov
Expand Down Expand Up @@ -97,8 +97,7 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y default-libmysqlclient-dev pkg-config

- name: Install dependencies
run: uv sync --locked --all-extras --dev
shell: bash
run: uv sync --locked --all-extras --group dev

- name: Install ODBC driver for MSSQL
run: |
Expand All @@ -115,7 +114,7 @@ jobs:
with:
timeout_minutes: 3
max_attempts: 3
command: uv run --no-sync pytest tests/integration --no-cov
command: uv run --no-sync pytest tests/integration
shell: bash

build:
Expand Down
32 changes: 14 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,14 @@
<a href="https://www.paypal.com/ncp/payment/6G9Z78QHUD4RJ"><img src="https://img.shields.io/badge/Donate-PayPal-brightgreen.svg?style=plastic" alt="Donate"/></a>
<a href="https://github.com/sponsors/ddc"><img src="https://img.shields.io/static/v1?style=plastic&label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=ff69b4" alt="Sponsor"/></a>
<br>
<a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg?style=plastic" alt="Code style: black"/></a>
<a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json?style=plastic" alt="uv"/></a>
<a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json?style=plastic" alt="Ruff"/></a>
<br>
<a href="https://www.python.org/downloads"><img src="https://img.shields.io/pypi/pyversions/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="Python"/></a>
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=plastic" alt="License: MIT"/></a>
<a href="https://pepy.tech/projects/ddcDatabases"><img src="https://static.pepy.tech/badge/ddcDatabases?style=plastic" alt="PyPI Downloads"/></a>
<a href="https://pypi.python.org/pypi/ddcDatabases"><img src="https://img.shields.io/pypi/v/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="PyPi"/></a>
<br>
<a href="https://www.python.org/downloads"><img src="https://img.shields.io/pypi/pyversions/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="Python"/></a>
<a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json?style=plastic" alt="uv"/></a>
<a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json?style=plastic" alt="Ruff"/></a>
<br>
<a href="https://github.com/ddc/ddcDatabases/issues"><img src="https://img.shields.io/github/issues/ddc/ddcDatabases?style=plastic" alt="issues"/></a>
<a href="https://codecov.io/gh/ddc/ddcDatabases"><img src="https://codecov.io/gh/ddc/ddcDatabases/graph/badge.svg?token=XWB53034GI&style=plastic" alt="codecov"/></a>
<a href="https://sonarcloud.io/dashboard?id=ddc_ddcDatabases"><img src="https://sonarcloud.io/api/project_badges/measure?project=ddc_ddcDatabases&metric=alert_status&style=plastic" alt="Quality Gate Status"/></a>
Expand Down Expand Up @@ -48,7 +47,7 @@
- [Database Utilities](#database-utilities)
- [Available Methods](#available-methods)
- [Logging](#logging)
- [Development](#development)
- [Development and Testing](#development-and-testing)
- [Create DEV Environment and Running Tests](#create-dev-environment-and-running-tests)
- [Update DEV Environment Packages](#update-dev-environment-packages)
- [Building Wheel](#building-wheel)
Expand Down Expand Up @@ -89,13 +88,13 @@

Database classes use structured configuration dataclasses instead of flat keyword arguments:

| Class | Purpose | Fields |
|------------------------------|---------------------------------|-------------------------------------------------------------------------------------|
| `{DB}PoolConfig` | Connection pool settings | `pool_size`, `max_overflow`, `pool_recycle`, `connection_timeout` |
| `{DB}SessionConfig` | SQLAlchemy session settings | `echo`, `autoflush`, `expire_on_commit`, `autocommit` |
| `{DB}ConnectionRetryConfig` | Connection-level retry settings | `enable_retry`, `max_retries`, `initial_retry_delay`, `max_retry_delay` |
| `{DB}OperationRetryConfig` | Operation-level retry settings | `enable_retry`, `max_retries`, `initial_retry_delay`, `max_retry_delay`, `jitter` |
| `PersistentConnectionConfig` | Persistent connection settings | `idle_timeout`, `health_check_interval`, `auto_reconnect` |
| Class | Purpose | Fields |
|------------------------------|---------------------------------|-----------------------------------------------------------------------------------|
| `{DB}PoolConfig` | Connection pool settings | `pool_size`, `max_overflow`, `pool_recycle`, `connection_timeout` |
| `{DB}SessionConfig` | SQLAlchemy session settings | `echo`, `autoflush`, `expire_on_commit`, `autocommit` |
| `{DB}ConnectionRetryConfig` | Connection-level retry settings | `enable_retry`, `max_retries`, `initial_retry_delay`, `max_retry_delay` |
| `{DB}OperationRetryConfig` | Operation-level retry settings | `enable_retry`, `max_retries`, `initial_retry_delay`, `max_retry_delay`, `jitter` |
| `PersistentConnectionConfig` | Persistent connection settings | `idle_timeout`, `health_check_interval`, `auto_reconnect` |

**Note:** Replace `{DB}` with the database prefix: `PostgreSQL`, `MySQL`, `MSSQL`, `Oracle`, `MongoDB`, or `Sqlite`.

Expand Down Expand Up @@ -689,18 +688,15 @@ logging.getLogger("ddcDatabases").addHandler(logging.StreamHandler())
```


# Development
# Development and Testing

Must have [UV](https://uv.run/docs/getting-started/installation) installed.

## Create DEV Environment and Running Tests

> **Note:** All poe tasks automatically run ruff linter along with Black formatting

```shell
uv sync --all-extras --all-groups
poe test
poe test-integration
poe tests
```

## Update DEV Environment Packages
Expand Down
1 change: 1 addition & 0 deletions ddcDatabases/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ POSTGRESQL_PORT=5432
POSTGRESQL_USER=postgres
POSTGRESQL_PASSWORD=password
POSTGRESQL_DATABASE=postgres
# Comma-separated for multiple schemas (e.g. public,schema2)
POSTGRESQL_SCHEMA=public
# Session settings
POSTGRESQL_ECHO=false
Expand Down
2 changes: 1 addition & 1 deletion ddcDatabases/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@
__all__ = tuple(__all__)
__title__ = "ddcDatabases"
__author__ = "Daniel Costa"
__email__ = "ddcsoftwares@proton.me"
__email__ = "daniel@ddcsoftwares.com"
__license__ = "MIT"
__copyright__ = "Copyright 2024-present DDC Softwares"
__version__ = version(__title__)
Expand Down
2 changes: 1 addition & 1 deletion ddcDatabases/core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class SettingsMessages:
USERNAME_DESCRIPTION = "Database username"
PASSWORD_DESCRIPTION = "Database password"
NAME_DESCRIPTION = "Database name"
SCHEMA_DESCRIPTION = "Database schema"
SCHEMA_DESCRIPTION = "Database schema (comma-separated for multiple schemas, e.g. 'gw2,public')"
ASYNC_DATABASE_DRIVER_DESCRIPTION = "Async database driver"
SYNC_DATABASE_DRIVER_DESCRIPTION = "Sync database driver"

Expand Down
12 changes: 12 additions & 0 deletions ddcDatabases/core/persistent.py
Original file line number Diff line number Diff line change
Expand Up @@ -719,6 +719,7 @@ def __new__(
user: str | None = None,
password: str | None = None,
database: str | None = None,
schema: str | None = None,
async_mode: Literal[False] = False,
config: PersistentConnectionConfig | None = None,
connection_retry_config: BaseRetryConfig | None = None,
Expand All @@ -735,6 +736,7 @@ def __new__(
user: str | None = None,
password: str | None = None,
database: str | None = None,
schema: str | None = None,
*,
async_mode: Literal[True],
config: PersistentConnectionConfig | None = None,
Expand All @@ -751,6 +753,7 @@ def __new__(
user: str | None = None,
password: str | None = None,
database: str | None = None,
schema: str | None = None,
async_mode: bool = False,
config: PersistentConnectionConfig | None = None,
connection_retry_config: BaseRetryConfig | None = None,
Expand All @@ -765,7 +768,10 @@ def __new__(
user = user or _settings.user
password = password or _settings.password
database = database or _settings.database
schema = schema if schema is not None else _settings.schema
connection_key = f"postgresql://{user}@{host}:{port}/{database}" # NOSONAR
if schema and schema != "public":
connection_key += f"?schema={schema}"

# Build config from settings, allowing partial overrides
_cfg = config or PersistentConnectionConfig()
Expand Down Expand Up @@ -820,6 +826,9 @@ def __new__(
else:
async_connect_args["ssl"] = ssl_mode

if schema and schema != "public":
async_connect_args["server_settings"] = {"search_path": schema}

merged_kwargs = {**engine_kwargs}
if async_connect_args:
existing_connect_args = merged_kwargs.get("connect_args", {})
Expand Down Expand Up @@ -855,6 +864,9 @@ def __new__(
if ssl_client_key_path:
sync_connect_args["sslkey"] = ssl_client_key_path

if schema and schema != "public":
sync_connect_args["options"] = f"-c search_path={schema}"

merged_kwargs = {**engine_kwargs}
if sync_connect_args:
existing_connect_args = merged_kwargs.get("connect_args", {})
Expand Down
2 changes: 1 addition & 1 deletion ddcDatabases/core/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from pydantic_settings import BaseSettings, SettingsConfigDict
from typing import TypeVar

warnings.filterwarnings("ignore", message="Field name \"schema\".*shadows an attribute in parent")
warnings.filterwarnings("ignore", message='Field name "schema".*shadows an attribute in parent')

# Type variable for generic settings factory
T = TypeVar("T", bound=BaseSettings)
Expand Down
35 changes: 15 additions & 20 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ packages = ["ddcDatabases"]

[project]
name = "ddcDatabases"
version = "3.0.10"
version = "3.0.11"
description = "Simplified database ORM connections with support for multiple database engines"
urls.Repository = "https://github.com/ddc/ddcDatabases"
urls.Homepage = "https://pypi.org/project/ddcDatabases"
license = {text = "MIT"}
readme = "README.md"
authors = [
{name = "Daniel Costa", email = "ddcsoftwares@proton.me"},
{name = "Daniel Costa", email = "daniel@ddcsoftwares.com"},
]
maintainers = [
{name = "Daniel Costa"},
Expand Down Expand Up @@ -55,41 +55,40 @@ classifiers = [
requires-python = ">=3.10"
dependencies = [
"pydantic-settings>=2.11.0",
"sqlalchemy[asyncio]>=2.0.46",
"sqlalchemy[asyncio]>=2.0.47",
]

[project.optional-dependencies]
mongodb = ["motor>=3.7.1"]
oracle = ["oracledb>=3.4.2"]
mssql = ["pyodbc>=5.3.0", "aioodbc>=0.5.0"]
mysql = ["mysqlclient>=2.2.8", "aiomysql>=0.3.2"]
postgres = ["psycopg[binary]>=3.3.2", "asyncpg>=0.31.0"]
postgres = ["psycopg[binary]>=3.3.3", "asyncpg>=0.31.0"]
pgsql = ["ddcDatabases[postgres]"]
mariadb = ["ddcDatabases[mysql]"]

[dependency-groups]
dev = [
"coverage>=7.13.4",
"poethepoet>=0.42.0",
"pytest-asyncio>=1.3.0",
"pytest-cov>=7.0.0",
"ruff>=0.15.2",
"testcontainers[postgres,mysql,mssql,mongodb,oracle]>=4.14.1",
"poethepoet>=0.41.0",
"ruff>=0.15.0",
"black>=26.1.0",
]

[tool.poe.tasks]
linter.shell = "uv run ruff check --fix . && uv run black ."
profile.sequence = ["linter", {shell = "uv run python -m cProfile -o cprofile_unit.prof -m pytest tests/unit --no-cov"}]
profile-integration.sequence = ["linter", {shell = "uv run python -m cProfile -o cprofile_integration.prof -m pytest tests/integration --no-cov"}]
test.sequence = ["linter", {shell = "uv run pytest"}]
test-integration.sequence = ["linter", {shell = "uv run pytest tests/integration --no-cov"}]
linter.shell = "uv run ruff check --fix . && uv run ruff format ."
profile = "uv run python -m cProfile -o cprofile_unit.prof -m pytest tests/unit"
profile-integration = "uv run python -m cProfile -o cprofile_integration.prof -m pytest tests/integration"
test.sequence = [{ shell = "uv run coverage run -m pytest tests/unit" }, { shell = "uv run coverage report" }, { shell = "uv run coverage xml" }]
test-integration = "uv run pytest tests/integration"
tests.sequence = ["linter", "test", "test-integration"]
updatedev.sequence = ["linter", {shell = "uv lock --upgrade && uv sync --all-extras --group dev"}]
build.sequence = ["updatedev", "test", "test-integration", {shell = "uv build --wheel"}]
build.sequence = ["updatedev", "tests", {shell = "uv build --wheel"}]

[tool.pytest.ini_options]
addopts = "-v --cov --cov-report=term --cov-report=xml --junitxml=junit.xml"
addopts = "-v --import-mode=importlib --junitxml=junit.xml"
junit_family = "legacy"
testpaths = ["tests/unit"]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
markers = [
Expand Down Expand Up @@ -120,10 +119,6 @@ exclude_lines = [
"@(abc\\.)?abstractmethod",
]

[tool.black]
line-length = 120
skip-string-normalization = true

[tool.ruff]
line-length = 120
target-version = "py310"
Expand Down
28 changes: 21 additions & 7 deletions tests/integration/conftest.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
import pytest
import time
from sqlalchemy import Boolean, Identity, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

# Testcontainer image versions
POSTGRES_IMAGE = "postgres:18-alpine"
MYSQL_IMAGE = "mysql:9"
POSTGRES_IMAGE = "postgres:latest"
MYSQL_IMAGE = "mysql:latest"
MONGODB_IMAGE = "mongo:8.0"
MARIADB_IMAGE = "mariadb:latest"
ORACLE_IMAGE = "gvenzl/oracle-free:slim-faststart"
MSSQL_IMAGE = "mcr.microsoft.com/mssql/server:2022-latest"
MONGODB_IMAGE = "mongo:8"
MARIADB_IMAGE = "mariadb:12"
ORACLE_IMAGE = "gvenzl/oracle-free:23-slim"


class Base(DeclarativeBase):
Expand Down Expand Up @@ -50,8 +51,21 @@ def mssql_container():
def mongodb_container():
from testcontainers.mongodb import MongoDbContainer

with MongoDbContainer(MONGODB_IMAGE) as mongo:
yield mongo
max_attempts = 3
last_exc = None
for attempt in range(max_attempts):
try:
container = MongoDbContainer(MONGODB_IMAGE)
container.start()
break
except Exception as exc:
last_exc = exc
if attempt < max_attempts - 1:
time.sleep(2)
else:
raise last_exc
yield container
container.stop()


@pytest.fixture(scope="session")
Expand Down
Loading
Loading