Dijon is built on a modular micro‑service framework that separates data ingestion, processing, and API delivery. Its core components communicate through lightweight messaging queues, allowing scalable and fault‑tolerant operation. This guide walks beginners through each layer and shows how to inspect the system in practice.
What Makes Up Dijon’s Architecture?
The platform consists of three primary layers: the Ingestion Layer, the Processing Layer, and the API Layer. Each layer runs as an independent service container, typically orchestrated by Kubernetes. This separation lets developers update one part without disrupting the others.
The Ingestion Layer pulls raw data from sources such as databases, webhooks, or file uploads. It normalizes the payload and forwards it to a message broker like Apache Kafka. The broker guarantees ordered delivery and buffers spikes in traffic.
Core Components and Their Roles
Below is a quick reference of the most important services in a standard Dijon deployment.
| Component | Role | Typical Technology |
|---|---|---|
| Ingestor | Collects and validates incoming data | Node.js, Python Flask |
| Message Broker | Buffers and routes messages | Apache Kafka, RabbitMQ |
| Processor | Applies business logic, transformations | Java Spring Boot, Go micro‑services |
| API Gateway | Exposes REST/GraphQL endpoints | NGINX, Kong |
All components expose health‑check endpoints that Kubernetes monitors. If a service fails, the orchestrator restarts it automatically, preserving overall uptime.
Data Flow from Source to Consumer
Understanding the data path helps beginners debug issues quickly. Follow these steps to trace a record through the system:
- Source system sends a JSON payload to the Ingestor’s HTTP endpoint.
- The Ingestor validates the schema, enriches metadata, and publishes the message to the Kafka topic “raw-events”.
- The Processor subscribes to “raw-events”, transforms the data, and writes the result to the “processed‑events” topic.
- The API Gateway reads from “processed‑events” and serves the data via a REST endpoint to client applications.
Each stage logs a unique transaction ID, making it easy to correlate logs across services. Tools like Grafana Loki or Elastic Stack can aggregate these logs for a single view.
Deploying and Scaling Dijon
For beginners, the easiest way to launch Dijon is with Docker Compose, which spins up all services on a local machine. Production environments typically use Helm charts to install the stack on a Kubernetes cluster.
Scaling is achieved by increasing the replica count of any service. Because the message broker decouples producers and consumers, you can add more Processor instances without touching the Ingestor. Horizontal Pod Autoscalers automatically adjust replicas based on CPU or queue length.
Maintenance Best Practices
Regularly update the schema definitions stored in a shared Git repository. This ensures every service validates against the same contract. Use semantic versioning for API changes to avoid breaking downstream clients.
Monitor queue lag in Kafka; a growing lag indicates that processors cannot keep up with incoming traffic. In such cases, either optimize the processing logic or add more Processor replicas.
Getting Started Today
To explore Dijon on your own machine, clone the official repository, run docker-compose up, and send a test payload with curl. Observe the logs, follow the transaction ID, and you’ll see the full path from ingestion to API response.
Frequently Asked Questions
What is the advantage of using a message broker in Dijon?
The broker decouples services, allowing each to scale independently and providing built‑in retry mechanisms for failed messages.
Can I replace Kafka with another broker?
Yes. Dijon’s architecture is broker‑agnostic; RabbitMQ or Amazon MSK work as long as they support the required topics and consumer groups.
How do I monitor health of individual services?
Each service exposes a /healthz endpoint that returns HTTP 200 when healthy. Kubernetes probes these endpoints and can restart unhealthy pods automatically.