> Status: draft. Part of the Harmovela 0.2 transport-kafka profile.
Define how Harmovela runs over Apache Kafka, supporting topic-based publish/subscribe with partition-based ordering and offset-based replay.
Harmovela over Kafka uses JSON-encoded Harmovela events as Kafka message values:
The Kafka record key is derived from one of the following Harmovela envelope fields, in priority order:
| Priority | Field | Purpose |
|---|---|---|
| 1 | task_id | All events for a task land in the same partition |
| 2 | conversation_id | All events for a conversation are ordered |
| 3 | session_id | Events for a session are ordered |
| 4 | source | Events from the same producer are ordered |
| 5 | (none) | Round-robin across partitions |
| Header | Harmovela Field | Type |
|---|---|---|
aep-type | type | string |
aep-source | source | string |
aep-session | session_id | string |
aep-conversation | conversation_id | string |
aep-task | task_id | string |
aep-correlation | correlation_id | string |
aep-causation | causation_id | string |
aep-delivery-mode | delivery.mode | string |
Headers enable consumers to filter and route without deserializing the message body.
| Harmovela context | Kafka topic pattern | Example |
|---|---|---|
Topic tasks.task_01 | aep.topic.tasks.task_01 | aep.topic.tasks.task_01 |
Source agent:researcher | aep.source.agent.researcher | aep.source.agent.researcher |
Type pattern task.* | aep.type.task.* | Matches all task subtopics |
| Session | aep.sess.<session_id> | aep.sess.sess_01 |
| All events | aep.events | Single-topic deployment |
The default topic prefix is aep. Implementations should allow per-envelope topic routing or single-topic deployment with header-based filtering.
| Harmovela Delivery Mode | Kafka Mechanism |
|---|---|
best_effort | Fire-and-forget producer with acks=0 |
at_least_once | Producer with acks=1 or acks=all, consumer with manual commit after processing |
replayable | Consumer with auto.offset.reset=earliest, replay by seeking to offset |
enable.idempotence=true, acks=all prevents duplicates from producer retries.event.acknowledged is emitted. On rebalance, uncommitted offsets are redelivered.consumer_group:topic:partition:offset).consumer.seek(partition, offset).Harmovela sessions map to Kafka consumer groups:
| Session | Consumer Group |
|---|---|
sess_01 | aep-sess_01 |
| Multiple agents sharing a session | Same group — events distributed across members |
| Independent sessions | Separate groups — each receives full event stream |
Ordering is guaranteed within a partition, not across partitions:
For strict ordering, route all events through a single partition (num.partitions=1) or use a consistent key (e.g., session_id).
| Kafka State | Session State |
|---|---|
| Consumer connects and joins group | CREATED |
| Partition assignment received | OPENED |
| First event consumed | READY |
| Consumer leaves group (graceful) | CLOSED |
| Consumer session timeout | ERROR |
kafka-go (Go), kafkajs (Node.js), confluent-kafka (Python), kafka-clients (Java).linger.ms, batch.size).max.poll.records.