Dijon is built on a modular micro‑service framework that separates data ingestion, processing, and API delivery. Its core components communicate through lightweight messaging queues, allowing scalable and fault‑tolerant operation. This guide walks beginners through each layer and shows how to inspect the system in practice.

Diagram showing Dijon’s modular micro‑service architecture with ingestion, processing, and API layers
Diagram showing Dijon’s modular micro‑service architecture with ingestion, processing, and API layers

What Makes Up Dijon’s Architecture?

The platform consists of three primary layers: the Ingestion Layer, the Processing Layer, and the API Layer. Each layer runs as an independent service container, typically orchestrated by Kubernetes. This separation lets developers update one part without disrupting the others.

The Ingestion Layer pulls raw data from sources such as databases, webhooks, or file uploads. It normalizes the payload and forwards it to a message broker like Apache Kafka. The broker guarantees ordered delivery and buffers spikes in traffic.

Flowchart of a data record moving from source through Ingestor, Kafka, Processor, and API endpoint
Flowchart of a data record moving from source through Ingestor, Kafka, Processor, and API endpoint

Core Components and Their Roles

Below is a quick reference of the most important services in a standard Dijon deployment.

ComponentRoleTypical Technology
IngestorCollects and validates incoming dataNode.js, Python Flask
Message BrokerBuffers and routes messagesApache Kafka, RabbitMQ
ProcessorApplies business logic, transformationsJava Spring Boot, Go micro‑services
API GatewayExposes REST/GraphQL endpointsNGINX, Kong

All components expose health‑check endpoints that Kubernetes monitors. If a service fails, the orchestrator restarts it automatically, preserving overall uptime.

Data Flow from Source to Consumer

Understanding the data path helps beginners debug issues quickly. Follow these steps to trace a record through the system:

  1. Source system sends a JSON payload to the Ingestor’s HTTP endpoint.
  2. The Ingestor validates the schema, enriches metadata, and publishes the message to the Kafka topic “raw-events”.
  3. The Processor subscribes to “raw-events”, transforms the data, and writes the result to the “processed‑events” topic.
  4. The API Gateway reads from “processed‑events” and serves the data via a REST endpoint to client applications.

Each stage logs a unique transaction ID, making it easy to correlate logs across services. Tools like Grafana Loki or Elastic Stack can aggregate these logs for a single view.

Deploying and Scaling Dijon

For beginners, the easiest way to launch Dijon is with Docker Compose, which spins up all services on a local machine. Production environments typically use Helm charts to install the stack on a Kubernetes cluster.

Scaling is achieved by increasing the replica count of any service. Because the message broker decouples producers and consumers, you can add more Processor instances without touching the Ingestor. Horizontal Pod Autoscalers automatically adjust replicas based on CPU or queue length.

Maintenance Best Practices

Regularly update the schema definitions stored in a shared Git repository. This ensures every service validates against the same contract. Use semantic versioning for API changes to avoid breaking downstream clients.

Monitor queue lag in Kafka; a growing lag indicates that processors cannot keep up with incoming traffic. In such cases, either optimize the processing logic or add more Processor replicas.

Getting Started Today

To explore Dijon on your own machine, clone the official repository, run docker-compose up, and send a test payload with curl. Observe the logs, follow the transaction ID, and you’ll see the full path from ingestion to API response.

Frequently Asked Questions

What is the advantage of using a message broker in Dijon?

The broker decouples services, allowing each to scale independently and providing built‑in retry mechanisms for failed messages.

Can I replace Kafka with another broker?

Yes. Dijon’s architecture is broker‑agnostic; RabbitMQ or Amazon MSK work as long as they support the required topics and consumer groups.

How do I monitor health of individual services?

Each service exposes a /healthz endpoint that returns HTTP 200 when healthy. Kubernetes probes these endpoints and can restart unhealthy pods automatically.