Goal

Build several unrelated applications quickly with Django and a frontend, host them economically on one server, and preserve the option to move genuinely slow components to Go, Rust, or another runtime later.

Core architectural decision

Start with a modular Django monolith, not microservices. Organize code by business domain and keep important workflows behind explicit inputs and outputs.

Typical layers:

  • HTTP views/controllers: authentication, request parsing, status codes and responses.
  • Application use cases: workflows such as place_order() or convert_dataset().
  • Domain logic: business rules expressed with ordinary Python values where useful.
  • Adapters: Django ORM, object storage, email, queues and external APIs.
flowchart TD A[HTTP views and controllers] --> B[Application use cases] B --> C[Domain logic] B --> D[Adapters] D --> E[Django ORM] D --> F[Object storage] D --> G[Email] D --> H[Queues] D --> I[External APIs]

Django-specific code does not need to be eliminated. It only needs to be kept from spreading through every layer. Ordinary CRUD can use Django models directly; create stronger boundaries around complex workflows and likely performance hotspots.

Database and ORM guidance

Heavy use of Django ORM does not automatically mean it will later need to be rewritten as raw SQL. First measure the actual bottleneck. Common improvements are:

  • fixing N+1 queries with select_related() and prefetch_related();
  • adding or correcting indexes;
  • reducing result and response sizes;
  • rewriting only demanding queries as specialized query functions or raw SQL;
  • introducing read models for unusually complex reads.

Repository interfaces are useful selectively. They should represent business queries such as find_unpaid_orders_due_before(date), rather than becoming generic wrappers that imitate the ORM.

Keep important constraints in PostgreSQL as well as application validation. Stable IDs, explicit timestamps, transaction boundaries and database constraints make data easier to use from another language.

Portable contracts

The portable boundary is the protocol, not a Python class. Use explicit, versioned contracts such as:

  • HTTP with OpenAPI;
  • JSON Schema;
  • Protobuf/gRPC when internal binary contracts are justified;
  • versioned event schemas for asynchronous jobs.

Pydantic and Zod do not directly replace one another. Pydantic validates Python data and Zod validates JavaScript or TypeScript data. Both can implement the same OpenAPI or JSON Schema contract.

Do not expose accidental Django internals as public contracts, such as model serialization, database column names or framework-specific error formats.

Moving heavy processing to Go or Rust

Django can remain the control plane while another language performs CPU-intensive or independently scalable work.

Suggested workflow:

  1. The frontend submits a request.
  2. Django authenticates the user and validates a small metadata payload.
  3. Django creates a job record and publishes a language-neutral message.
  4. A Go, Rust or Python worker processes the job.
  5. The worker stores the result and updates job status or publishes a result event.
  6. The frontend polls with backoff or receives progress through SSE/WebSocket.
sequenceDiagram participant Client as Frontend/client participant Django participant DB as PostgreSQL participant Queue participant Worker as Go/Rust/Python worker participant Storage as Object storage Client->>Django: Submit job metadata Django->>Django: Authenticate and validate Django->>DB: Create job record Django->>Queue: Publish job message Django-->>Client: 202 Accepted with job id Worker->>Queue: Consume job Worker->>Storage: Read input and write result Worker->>DB: Update job status Client->>Django: Poll job status/result

Example message:

{
  "version": 1,
  "job_id": "uuid",
  "type": "dataset.convert",
  "payload": {
    "dataset_id": "uuid",
    "input_object": "uploads/dataset.gpkg",
    "output_format": "geojson"
  }
}

Large files and geometry collections should go directly to object storage. Queue messages should contain identifiers and metadata, not the file contents.

Queue options

  • RabbitMQ: strong default for cross-language jobs, acknowledgements, retries, routing and dead-letter queues.
  • Redis Streams: simpler when Redis is already required.
  • NATS JetStream: lightweight cross-language messaging with persistence.
  • Kafka: appropriate for large retained event streams, usually excessive for early products.
  • PostgreSQL jobs table: minimal infrastructure for modest workloads.

Avoid treating Celery’s internal task-message format as the cross-language contract. Celery may use RabbitMQ, but another-language worker is cleaner when consuming an application-owned JSON or Protobuf message.

A PostgreSQL worker can claim jobs using FOR UPDATE SKIP LOCKED. This avoids another service initially, but requires implementing retries, stale-lock recovery, cleanup and job state transitions.

Job handlers should be idempotent because most reliable queues provide at-least-once delivery. Use unique job IDs, acknowledge only after success, retry temporary errors with backoff, version schemas and move permanent failures to a dead-letter state.

For important jobs, use the Transactional Outbox pattern so a database change and creation of an outbound event happen in the same transaction. transaction.on_commit() is a simpler alternative but cannot guarantee delivery if publishing fails after the database commit.

Resource implications

If Django only performs authentication, small-payload validation, a few database operations, queue submission and response serialization, it should remain relatively inexpensive. Pydantic or other serializers are normally not the bottleneck for small JSON documents.

The expensive cases are:

  • large or deeply nested JSON;
  • huge GeoJSON payloads;
  • N+1 ORM queries;
  • large response serialization;
  • importing GIS, data science or AI libraries into every web worker;
  • clients polling job status too frequently.

Long jobs should return 202 Accepted immediately. Django should not hold an HTTP request open while a worker runs for tens of seconds. Use a job resource such as POST /jobs, GET /jobs/{id} and GET /jobs/{id}/result.

Why Django appears memory-heavy

The main cost is usually several Python worker processes, not Gunicorn itself. Every worker loads the Python interpreter, Django, the app registry, models, middleware, dependencies and process-local caches.

A small Django worker may use roughly 40-100 MB when idle; larger applications can use 150 MB or more per worker. Raw RSS can overstate total usage because shared pages are counted in multiple processes; proportional set size is a better measurement.

Minimal Python frameworks may save tens of megabytes, but once SQLAlchemy, migrations, authentication, validation, admin tools and monitoring are added, the difference often shrinks. Switching from Django to FastAPI solely for memory is therefore rarely compelling.

Go or Rust can provide a genuinely smaller runtime, but Django includes valuable product-development facilities: ORM, migrations, authentication, sessions, permissions and admin.

Hosting many unrelated apps on one server

Use a parking configuration for apps that have little traffic:

  • one shared Caddy instance;
  • one shared PostgreSQL cluster, with a separate database and database user for each serious app;
  • one small Django process per app;
  • static frontend builds served directly by Caddy;
  • no permanent Redis, RabbitMQ, Celery worker or scheduler unless the workload requires it.

For a WSGI app, a reasonable starting command is:

gunicorn project.wsgi:application \
  --worker-class gthread \
  --workers 1 \
  --threads 2 \
  --timeout 30 \
  --max-requests 1000 \
  --max-requests-jitter 100

For an ASGI app, a single Uvicorn process supervised by systemd can remove the small Gunicorn master-process cost. Changing Python servers does not remove Django’s or Python’s main memory usage.

One parked Django app may consume approximately 70-150 MB, depending heavily on imports. A carefully managed 4 GB server could plausibly host several, potentially around 8-15, very small, low-traffic apps, but this must be measured against the real applications.

PostgreSQL and SQLite strategy

Do not run one PostgreSQL server/container per app. Use one PostgreSQL installation with separate databases and users. Conservative initial settings can include a small shared_buffers, low work_mem and limited max_connections; tune from measurements rather than fixed rules.

SQLite is reasonable for disposable POCs and low-write experiments. Start directly with PostgreSQL when an app needs PostGIS, substantial concurrency, complex reporting or serious background processing.

Suggested progression:

  • disposable experiment: SQLite;
  • serious candidate product: shared PostgreSQL;
  • successful product: dedicated database resources when justified.

Avoid idle background infrastructure

Instead of permanently running Celery workers and Celery Beat for every app:

  • store pending jobs in the database;
  • run a management command from a systemd timer or cron;
  • process a bounded number of jobs and exit;
  • convert to a continuously running queue worker only when latency or volume requires it.

The same pattern can start a Go or Rust binary periodically. Keep GIS libraries such as GDAL, GeoPandas or Rasterio out of Django web-process imports; load them only in processing workers.

Migration strategy

Use the Strangler pattern rather than rewriting an entire application:

  1. Measure and identify an actual bottleneck.
  2. Stabilize its input/output contract.
  3. Add contract, behavior and benchmark tests.
  4. Reimplement that capability in Go, Rust or another runtime.
  5. Route a small portion of traffic or jobs to the new implementation.
  6. Compare correctness, latency, errors and resource usage.
  7. Increase traffic gradually and remove the old implementation after confidence is high.

Caddy can route most endpoints to Django and selected hot endpoints to another service. The frontend does not need to know which language serves an endpoint as long as the contract remains stable.

flowchart LR A[Frontend/client] --> B[Caddy] B --> C[Django monolith] B --> D[Go/Rust service] C --> E[(Shared database or API contract)] D --> E
  1. Build each product as a modular Django application with a static frontend frontend.
  2. Use one Django worker while the product is dormant.
  3. Use SQLite for disposable POCs or one shared PostgreSQL cluster for serious apps.
  4. Avoid Redis, Celery and RabbitMQ until they solve a demonstrated need.
  5. For early background work, use a database job table and scheduled one-shot workers.
  6. For reliable cross-language processing, graduate to RabbitMQ with application-owned JSON messages.
  7. Move only proven CPU-heavy or high-concurrency components to Go or Rust.
  8. Scale worker count, database resources and service separation only after an app gains traction.

The guiding principle is: optimize the deployment topology first, preserve explicit boundaries in the code, and rewrite only measured bottlenecks.