Add django-db-connection-pool to fix PostgreSQL connection exhaustion by Copilot · Pull Request #1409 · scieloorg/core

Copilot · 2026-03-17T13:59:19Z

O que esse PR faz?

Resolve o erro FATAL: sorry, too many clients already em produção adicionando connection pooling via SQLAlchemy QueuePool ao backend de banco de dados.

Com 3 workers Gunicorn gevent (1000 connections cada), as conexões ao PostgreSQL eram ilimitadas. Agora são limitadas a POOL_SIZE(10) + MAX_OVERFLOW(5) = 15 por worker, totalizando ~45 conexões máximas.

Mudanças:

Adiciona django-db-connection-pool==1.2.6 em requirements/production.txt
Cria backend customizado core.db.backends.postgresql que combina connection pooling (PGDatabaseWrapperMixin) com métricas Prometheus (PrometheusDatabaseWrapper) via MRO
Atualiza ENGINE em production.py para usar o backend customizado
Adiciona PRE_PING: True ao POOL_OPTIONS para validar conexões antes do uso
Reduz MAX_OVERFLOW de 20 para 5

Já existente (sem alterações):

Celery signals (config/celery_signals.py) já fecham conexões em task_prerun/task_postrun
CONN_MAX_AGE=60 e CONN_HEALTH_CHECKS=True já configurados

Onde a revisão poderia começar?

core/db/backends/postgresql/base.py — backend customizado (5 linhas efetivas), depois config/settings/production.py linhas 19-28.

Como este poderia ser testado manualmente?

Deploy em staging com as variáveis DB_POOL_SIZE, DB_MAX_OVERFLOW, DB_RECYCLE configuradas
Monitorar conexões PostgreSQL: SELECT count(*) FROM pg_stat_activity WHERE datname = 'core';
Simular carga e verificar que conexões não excedem workers × (POOL_SIZE + MAX_OVERFLOW)
Verificar logs do Django para ausência de too many clients already

Algum cenário de contexto que queira dar?

O POOL_OPTIONS e as variáveis de ambiente (DB_POOL_SIZE, DB_MAX_OVERFLOW, DB_RECYCLE) já existiam no .envs/.production/.django e em production.py, mas o pacote django-db-connection-pool nunca foi instalado — a configuração era dead code.

O backend customizado usa herança múltipla para preservar métricas Prometheus:

class DatabaseWrapper(PGDatabaseWrapperMixin, PrometheusDatabaseWrapper):
    pass

PGDatabaseWrapperMixin primeiro no MRO para que get_new_connection() gerencie o pool; quando precisa criar conexões novas, delega para PrometheusDatabaseWrapper que adiciona tracking.

Screenshots

N/A

Quais são tickets relevantes?

N/A

Referências

Original prompt

This section details on the original issue you should resolve

<issue_title>scielo.org com dificuldade de manter up</issue_title>
<issue_description>### Descrição do problema

O log do django está com algumas mensagens em produção como:

cursor = self.connection.cursor() 2026-03-17T13:10:18.731109419Z ^^^^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731135526Z File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner 2026-03-17T13:10:18.731153429Z return func(*args, **kwargs) 2026-03-17T13:10:18.731169591Z ^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731186525Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 320, in cursor 2026-03-17T13:10:18.731203245Z return self._cursor() 2026-03-17T13:10:18.731218124Z ^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731233119Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 296, in _cursor 2026-03-17T13:10:18.731250537Z self.ensure_connection()
File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner 2026-03-17T13:10:18.731280895Z return func(*args, **kwargs) 2026-03-17T13:10:18.731297069Z ^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731312102Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 278, in ensure_connection 2026-03-17T13:10:18.731327108Z with self.wrap_database_errors: 2026-03-17T13:10:18.731342051Z File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ 2026-03-17T13:10:18.731356615Z raise dj_exc_value.with_traceback(traceback) from exc_value 2026-03-17T13:10:18.731370811Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection 2026-03-17T13:10:18.731385757Z self.connect() 2026-03-17T13:10:18.731402735Z File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner 2026-03-17T13:10:18.731418725Z return func(*args, **kwargs) 2026-03-17T13:10:18.731433473Z ^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731449340Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_prometheus/db/backends/postgresql/base.py", line 9, in get_new_connection
conn = super().get_new_connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731583998Z File "/usr/local/lib/python3.11/site-packages/django_prometheus/db/common.py", line 45, in get_new_connection
return super().get_new_connection(*args, **kwargs) 2026-03-17T13:10:18.731617109Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner 2026-03-17T13:10:18.731663564Z return func(*args, **kwargs) 2026-03-17T13:10:18.731684798Z ^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731705148Z File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection 2026-03-17T13:10:18.731723543Z connection = self.Database.connect(**conn_params) 2026-03-17T13:10:18.731740876Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731760124Z File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect 2026-03-17T13:10:18.731776242Z conn = _connect(dsn, connection_factory=connection_factory, **kwasync) 2026-03-17T13:10:18.731826098Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-03-17T13:10:18.731842735Z django.db.utils.OperationalError: connection to server at "192.168.2.190", port 5432 failed: FATAL: sorry, too many clients already 2026-03-17T13:10:18.731862247Z 2026-03-17T13:10:19.137316691Z WARNING 2026-03-17 13:10:19,136 log 16 139884299102432 Not Found: /wp-content/uploads/2018/08/bvs.gif 2026-03-17T13:10:21.318225371Z WARNING 2026-03-17 13:10:21,314 log 16 139884299102432 Not Found: /wp-content/uploads/2018/08/Logo_Fap-Unifesp.png
WARNING 2026-03-17 13:10:21,324 log 15 139884423662304 Not Found: /wp-content/uploads/2018/08/fapesp_patrocinadores.png
WARNING 2026-03-17 13:10:26,379 log 16 139884299102432 Not Found: /pt-br/apps/servicesplatform/client/controller/authentication/origin/aHR0cDovL3NjaWVsby5zbGQuY3Uvc2NpZWxvLnBocD9sbmc9ZXMmbnJtPWlzbyZwaWQ9UzIyMTgtMzYyMDIwMjMwMDA0MDA3NzUmc2NyaXB0PXNjaV9hcnR0ZXh0/
WARNING 2026-03-17 13:10:26,944 log 16 139884299102432 Not Found: /apple-touch-icon-precomposed.png
WARNING 2026-03-17 13:10:27,373 log 15 139884423662304 Not Found: /wp-content/uploads/2018/08/fapesp_patrocinadores.png
WARNING 2026-03-17 13:10:29,769 log 17 139884360979840 Not Found: /wp-content/uploads/2018/08/capes_patrocinadores.png 2026-03-17T13:10:29.775141592Z WARNING 2026-03-17 13:10:29,774 log 15 139884423662304 Not Found: /wp-content/uploads/2018/08/ops-oms_es_pt.gif 2026-03-17T13:10:29.781661078Z WARNING 2026-03-17 13:10:29,780 log 16 139884299102432 Not Found:...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes scieloorg/core#1407

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

- Add django-db-connection-pool==1.2.6 to production requirements - Change production DB ENGINE to dj_db_conn_pool.backends.postgresql - Add PRE_PING to POOL_OPTIONS for connection health verification - Reduce MAX_OVERFLOW default from 20 to 5 to prevent connection exhaustion - Update production .env with aligned pool configuration Celery signals for closing DB connections were already in place. Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>

…trics Create core.db.backends.postgresql backend that inherits from both PGDatabaseWrapperMixin (connection pooling) and PrometheusDatabaseWrapper (metrics), preserving Prometheus database monitoring while using SQLAlchemy QueuePool for connection management. Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>

Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>

Initial plan

db08e39

Copilot AI assigned Copilot and robertatakenaka Mar 17, 2026

Copilot started work on behalf of robertatakenaka March 17, 2026 13:59 View session

Copilot AI and others added 3 commits March 17, 2026 14:04

Add documentation for custom DB backend MRO and PRE_PING option

70cc4b3

Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix issues with maintaining Django connection to database~~ Add django-db-connection-pool to fix PostgreSQL connection exhaustion Mar 17, 2026

Copilot AI requested a review from robertatakenaka March 17, 2026 14:09

Copilot finished work on behalf of robertatakenaka March 17, 2026 14:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add django-db-connection-pool to fix PostgreSQL connection exhaustion#1409

Add django-db-connection-pool to fix PostgreSQL connection exhaustion#1409
Copilot wants to merge 4 commits intomainfrom
copilot/fix-django-connection-issues

Copilot AI commented Mar 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

O que esse PR faz?

Onde a revisão poderia começar?

Como este poderia ser testado manualmente?

Algum cenário de contexto que queira dar?

Screenshots

Quais são tickets relevantes?

Referências

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Mar 17, 2026 •

edited

Loading