Setup: configuration, PostgreSQL and Redis

Setup: configuration, PostgreSQL and Redis#

Connection settings, running the clean layer on PostgreSQL, and running Redis (for example in Docker on Windows). For why the system is split this way see Data architecture.

Running on PostgreSQL and Redis#

src/trade_engine/config.py resolves one connection string per logical database (Settings.from_env()), each overridable via an environment variable:

Database

Environment variable

Default

reference

REFERENCE_DB_URL

sqlite:///<root>/db/reference.db

market

MARKET_DB_URL

sqlite:///<root>/db/market.db

trade

TRADE_DB_URL

sqlite:///<root>/db/trade.db

risk

RISK_DB_URL

sqlite:///<root>/db/risk.db

TRADE_ENGINE_ROOT overrides <root> (used by tests to run in a temporary folder). Two more settings cover the other tiers:

Setting

Environment variable

Default

Staging Redis

REDIS_URL

redis://localhost:6379/0

DuckDB analytics file

ANALYTICS_DB_PATH

<root>/db/analytics.duckdb

To run the clean layer on PostgreSQL:

  1. Install the driver: pip install -e .[postgres] (psycopg 3).

  2. Start PostgreSQL. docker-compose.yml runs one server and creates the four databases (reference_db, market_db, trade_db, risk_db) from docker/postgres/init-databases.sql; set POSTGRES_PASSWORD in your shell first, then docker compose up -d.

  3. Point each database at it, using the postgresql+psycopg scheme, for example TRADE_DB_URL=postgresql+psycopg://trade_engine:<password>@localhost:5432/trade_db. Set all four variables, or the unset ones stay on SQLite.

  4. Run scripts/init_db.py.

Redis runs separately from PostgreSQL; see Running Redis in Docker on Windows. When Redis is unreachable, staging fails with an error and nothing is written to the clean databases.

No other code changes are needed, because all queries go through SQLAlchemy sessions, not SQLite-specific SQL. After that point, schema changes must go through migrations, not create_all.

Running Redis in Docker on Windows#

Redis has no native Windows build, so run it in a container (or inside WSL; see below). docker-compose.redis.yml starts one Redis 7 server with append-only persistence.

Prerequisites

  • Docker Desktop for Windows with the WSL 2 backend enabled (the default on current installs), and Docker Desktop running. docker --version and docker compose version must both work in PowerShell.

Start, check, stop (from the project root in PowerShell):

docker compose -f docker-compose.redis.yml up -d
docker compose -f docker-compose.redis.yml exec redis redis-cli ping   # PONG
docker compose -f docker-compose.redis.yml down                        # stop, keep data
docker compose -f docker-compose.redis.yml down -v                     # stop and delete data

The container publishes port 6379 on 127.0.0.1 only, so the default REDIS_URL=redis://localhost:6379/0 works from Windows with no configuration. The server has no password and must not be exposed beyond localhost; before using it on a shared host, add --requirepass (and put the password in REDIS_URL, not in source).

Persistence. Data lives in the redis_data Docker volume with append-only logging, so staged batches survive a container restart. down -v deletes them.

Inspecting the staging area:

docker compose -f docker-compose.redis.yml exec redis redis-cli XLEN trade_engine:staging:market
docker compose -f docker-compose.redis.yml exec redis redis-cli XLEN trade_engine:staging:market:rejected

The first is the number of batches waiting to be processed; the second holds batches that failed validation, with the reason.

Troubleshooting

Symptom

Cause and fix

Redis staging error: Error 10061 connecting to localhost:6379

Redis is not running. Start Docker Desktop, then run the up -d command above.

port is already allocated or address already in use on 6379

Another Redis (for example one running inside WSL) already uses the port. Stop it, or change the mapping to 127.0.0.1:6380:6379 in the compose file and set REDIS_URL=redis://localhost:6380/0.

docker is not recognized

Docker Desktop is not installed, or PowerShell was opened before it was added to PATH; reopen the terminal.

Alternative: Redis inside WSL. Install Redis in a WSL distribution and start it there (sudo service redis-server start). WSL 2 forwards localhost:6379 to Windows, so the same default REDIS_URL works. Use one or the other, not both, because they share the port.

Contents