452 lines
16 KiB
Markdown
452 lines
16 KiB
Markdown
# DeployD Protocol Specification (_dproto_), version 0
|
|
|
|
## Table of Contents
|
|
|
|
1. [Abstract](#1-abstract)
|
|
2. [Conventions and Definitions](#2-conventions-and-definitions)
|
|
3. [Protocol Overview](#3-protocol-overview)
|
|
|
|
1. [Connection Establishment](#3-1-connection-establishment)
|
|
2. [Server Information Exchange](#3-2-server-information-exchange)
|
|
3. [Proof-of-Work Challenge](#3-3-proof-of-work-challenge)
|
|
4. [Proof-of-Work Solution](#3-4-proof-of-work-solution)
|
|
5. [Packet Phase](#3-5-command-phase)
|
|
6. [Termination](#3-6-termination)
|
|
7. [Chart](#3-7-chart)
|
|
|
|
4. [Packet Encoding](#4-packet-encoding)
|
|
|
|
1. [Generic Packet Structure](#4-1-generic-packet-structure)
|
|
2. [Token Types](#4-2-token-types)
|
|
3. [Error Codes and Formats](#4-3-error-codes-and-formats)
|
|
|
|
5. [`COMMAND` Packet](#5-command-packet)
|
|
6. [Keepalive: `PING` / `PING_REPLY`](#6-keepalive-ping--ping_reply)
|
|
7. [Access Control: `ALLOWED` / `READY`](#7-access-control-allowed--ready)
|
|
8. [Logging: `LOG` / `LOGS_END`](#8-logging-log--logs_end)
|
|
9. [`EXIT` Packet](#9-exit-packet)
|
|
10. [ `ERROR` Packet](#10-error-packet)
|
|
11. [ Token Generation Algorithm](#11-token-generation-algorithm)
|
|
12. [ Proof-of-Work Algorithm](#12-proof-of-work-algorithm)
|
|
13. [ Security Considerations](#13-security-considerations)
|
|
14. [ Authors](#14-authors)
|
|
|
|
## 1. Abstract
|
|
|
|
This document specifies dproto - a (near-)stateless protocol with a goal for cryptographic continuous deployment, DeployD's encrypted, authenticated, small, binary, little-endian, streaming protocol. It defines every packet format; enforces exact timeout values, strict length fields, and precise error-handling behaviours; and describes the proof-of-work handshake, command semantics, log streaming, and token generation with full determinism.
|
|
|
|
## 2. Conventions and Definitions
|
|
|
|
- **bool** :Exactly 1 octet. Permitted values: 0x00 (`false`), 0x01 (`true`). Any other value MUST be interpreted as `false`.
|
|
- **Endianness**: All multibyte integers are encoded in little-endian order.
|
|
- **Timeouts**: Both read and write timeouts SHALL be more or equal to 5 seconds. The client and the server's connection immediately terminates on exhaustion.
|
|
- **Connection**: A single SSL/TLS session over TCP, reused for the entire exchange.
|
|
- **Streaming**: Data may be sent in continuous back-to-back packets; boundaries are defined solely by the packet length fields.
|
|
|
|
## 3. Protocol Overview
|
|
|
|
### 3.1 Connection Establishment
|
|
|
|
1. Client performs standard SSL/TLS handshake.
|
|
2. Upon handshake completion, the client MUST immediately read the first packet of up to 256 bytes.
|
|
|
|
### 3.2 Server Information Exchange
|
|
|
|
- **Packet**
|
|
|
|
```c
|
|
struct {
|
|
uint8_t version; /* MUST be 0x00 (for this version) */
|
|
uint8_t server_info_len; /* MUST be in [1; 255] */
|
|
uint8_t server_info[server_info_len];
|
|
};
|
|
```
|
|
|
|
- **Requirements**
|
|
|
|
- `server_info_len` SHALL be above or equal to 1, but not more than 255.
|
|
- Server MUST send exactly `1 + server_info_len` octets.
|
|
- `server_info` contains raw data without any terminator.
|
|
- If fewer or more bytes arrive, the client MUST optionally initiate the `EXIT` packet and drop the connection.
|
|
- If the client does not support the given version, the client MUST optionally initiate the `EXIT` packet and drop the connection.
|
|
- The server MUST send the maximum supported version, with no gaps in support of lower versions. Dproto guarantees backwards compatibility.
|
|
- The client SHOULD choose the highest supported version, even if it is before `version`. Universal behaviour is guaranteed.
|
|
|
|
### 3.3 Proof-of-Work Challenge
|
|
|
|
- **Packet**
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type = 0x10; /* PING */
|
|
/* or */
|
|
uint8_t type = 0x13; /* READY */
|
|
};
|
|
```
|
|
|
|
(client may only send `PING` or `READY` during PoW stage)
|
|
|
|
- **Challenge Packet**
|
|
|
|
```c
|
|
struct {
|
|
uint8_t challenge[16]; /* 16 cryptographically secure random bytes */
|
|
uint8_t difficulty; /* MUST be above or equal to 1 */
|
|
uint8_t ones; /* MUST be above or equal to 1 */
|
|
};
|
|
```
|
|
|
|
- **Rules**
|
|
|
|
- Server sends exactly 16bytes for `challenge`, plus 2 bytes for `difficulty` and `ones`. These constitute PoW parameters.
|
|
- Client may issue up to 64 `PING`s, at most one per second. This is to avoid locking a worker/socket for too long and avoid spam.
|
|
- Each `PING` elicits a `PING_REPLY`; unmatched `PING`s trigger error code 0x2004.
|
|
- Excess `PING`s trigger error code=0x3000.
|
|
- The purpose of PoW is to avoid DoS/DDoS attacks by providing a robust layer against spam, hence why `difficulty` and `ones` MUST be at least `1`.
|
|
|
|
### 3.4 Proof-of-Work Solution
|
|
|
|
1. Client computes `nonce` in range of `[0; 18446744073709551615]` satisfying the challenge and difficulty.
|
|
2. Client sends:
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type = 0x13; /* READY */
|
|
uint8_t nonce[8]; /* 64-bit little-endian */
|
|
};
|
|
```
|
|
|
|
The nonce search algorithm is defined in § 12.
|
|
|
|
3. Server verifies:
|
|
|
|
- If valid: replies with `ALLOWED` (type=0x12).
|
|
- If invalid: replies with `ERROR` (code=0x3001), optionally followed by `EXIT`, and the connection is dropped.
|
|
- On any other errors a different `ERROR` may be issued.
|
|
|
|
### 3.5 Packet Phase
|
|
|
|
- After `ALLOWED`, client may send up to 64 packets in same connection. This is to avoid locking a worker/socket for too long and avoid spam.
|
|
- No interleaving proof-of-work messages allowed once in packet phase.
|
|
|
|
### 3.6 Termination
|
|
|
|
- Any side may send:
|
|
|
|
```c
|
|
struct { uint8_t type = 0x30; } /* EXIT */
|
|
```
|
|
|
|
- Upon sending or receiving `EXIT`, both sides MUST immediately close the connection; in-flight data may flush, in which case the server MUST not process further, and the client MAY choose to process specific packets, however, this is not a guarantee.
|
|
- The `EXIT` packet is optional, an "implicit" shutdown is done after a timeout or disconnection.
|
|
- At any point the server or the client can send a `PING` packet which MUST be followed by a `PING_REPLY` packet.
|
|
|
|
### 3.7 Chart
|
|
|
|
|
|
|
|
## 4. Packet Encoding
|
|
|
|
### 4.1 Generic Packet Structure
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type;
|
|
uint8_t *data; /* Length derived from packet type definition */
|
|
};
|
|
```
|
|
|
|
### 4.2 Token Types
|
|
|
|
| Name | Value | Direction | Content |
|
|
| ---------- | ------ | --------- | ----------------------- |
|
|
| COMMAND | `0x00` | _c2s_ | See §5 |
|
|
| PING | `0x10` | _bid._ | No payload |
|
|
| PING_REPLY | `0x11` | _bid._ | No payload |
|
|
| ALLOWED | `0x12` | _s2c_ | No payload |
|
|
| READY | `0x13` | _bid._ | As specified in context |
|
|
| LOG | `0x20` | _s2c_ | See §8 |
|
|
| LOGS_END | `0x21` | _s2c_ | No payload |
|
|
| EXIT | `0x30` | _bid._ | No payload |
|
|
| ERROR | `0xFF` | _bid._ | See §10 |
|
|
|
|
**Note:** "c2s" refers to "client to server", "bid." to "bidirectional (both c2s and s2c)", and "s2c" to "server to client", for brevity.
|
|
|
|
### 4.3 Error Codes and Formats
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type = 0xFF;
|
|
uint16_t msg_len; /* MUST be above or equal to 1, in bytes */
|
|
uint16_t error_code; /* See table */
|
|
uint8_t msg[msg_len]; /* UTF-8 text; no NULL terminator */
|
|
};
|
|
```
|
|
|
|
| Name | Value | Description |
|
|
| ------------------- | -------- | ---------------------------- |
|
|
| Internal | `0x0000` | Internal error |
|
|
| Type | `0x0001` | Unexpected packet type |
|
|
| Status | `0x0002` | Invalid status field |
|
|
| AuthToken | `0x1000` | Authentication token invalid |
|
|
| AuthKey | `0x1001` | Authentication key invalid |
|
|
| ProtoPacketTooShort | `0x2000` | Packet too short |
|
|
| ProtoDomainInvalid | `0x2001` | Invalid domain format |
|
|
| ProtoPacketTooLong | `0x2002` | Packet too long |
|
|
| ProtoDomainNotFound | `0x2003` | Domain not found |
|
|
| ProtoPacketInvalid | `0x2004` | Malformed or invalid packet |
|
|
| PowTooManyPings | `0x3000` | Too many PINGs during PoW |
|
|
| PowBadSolution | `0x3001` | Invalid PoW solution |
|
|
| DeployError | `0x4000` | Deployment execution error |
|
|
| InvalidCommand | `0x4001` | Invalid COMMAND sub-type |
|
|
|
|
## 5. `COMMAND` Packet
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type = 0x00;
|
|
uint8_t command; /* See table */
|
|
bool is_unsafe; /* 0x00 or 0x01 */
|
|
uint64_t id; /* Pre-shared semi-secret */
|
|
uint8_t domain_len; /* MUST be more or equal to 1 */
|
|
uint8_t domain[domain_len]; /* Raw bytes, no terminator */
|
|
uint8_t key[32]; /* Pre-shared secret */
|
|
uint8_t token[16]; /* From §11 */
|
|
};
|
|
```
|
|
|
|
| Command | Value | Description |
|
|
| -------- | ------ | ------------------------------ |
|
|
| Trigger | `0x00` | Trigger action |
|
|
| Teardown | `0x01` | Tear down deployment |
|
|
| Deploy | `0x02` | Initiate deployment |
|
|
| Rollback | `0x03` | Roll back to prior version |
|
|
| Cleanup | `0x04` | Clean up artefacts |
|
|
| Restart | `0x05` | Restart service |
|
|
| Sysadmin | `0x06` | System administration tasks |
|
|
| Logs | `0x07` | Request latest deployment logs |
|
|
|
|
- After COMMAND, server streams zero or more `LOG` packets, then `LOGS_END`.
|
|
- Commands lock the domain, and new deploys WILL be waited on if triggered when locked.
|
|
- The domain lock established by a `COMMAND` packet of type Deploy SHALL remain active for the entire duration of the associated deployment process. This lock MUST persist even if the initiating client disconnects, and it SHALL only be released upon successful or failed completion of the deployment.
|
|
- All secrets MUST be generated out-of-band.
|
|
|
|
## 6. Keepalive: `PING` / `PING_REPLY`
|
|
|
|
```c
|
|
struct { uint8_t type = 0x10; }; /* PING */
|
|
struct { uint8_t type = 0x11; }; /* PING_REPLY */
|
|
```
|
|
|
|
- No payload.
|
|
- Every `PING` MUST be followed by corresponding `PING_REPLY` by the receiver before any other message type.
|
|
|
|
## 7. Access Control: `ALLOWED` / `READY`
|
|
|
|
```c
|
|
struct { uint8_t type = 0x12; }; /* ALLOWED */
|
|
struct { uint8_t type = 0x13; }; /* READY */
|
|
```
|
|
|
|
- `READY` carries no payload except when used to send the 8-byte PoW nonce as specified in §3.4.
|
|
|
|
## 8. Logging: `LOG` / `LOGS_END`
|
|
|
|
```c
|
|
struct {
|
|
uint8_t type = 0x20;
|
|
uint16_t chunk_size; /* MUST be more or equal to 1 */
|
|
uint8_t chunk[chunk_size]; /* Opaque */
|
|
};
|
|
struct { uint8_t type = 0x21; }; /* LOGS_END */
|
|
```
|
|
|
|
- `chunk_size` is exact byte count; logs are raw UTF-8 or binary data.
|
|
- All `LOG` packets MUST be sent in order.
|
|
|
|
## 9. `EXIT` Packet
|
|
|
|
```c
|
|
struct { uint8_t type = 0x30; };
|
|
```
|
|
|
|
- No payload. Both peers MUST close the SSL connection immediately.
|
|
|
|
## 10. `ERROR` Packet
|
|
|
|
Defined in §4.3. Error packets may appear at any stage. On receiving `ERROR`, the receiver SHOULD cease current stage; on critical errors, server may follow with `EXIT`.
|
|
|
|
## 11. Token Generation Algorithm
|
|
|
|
1. **Inputs:**
|
|
|
|
- 64-bit UNIX epoch reference _T_ (seconds).
|
|
- 24-byte URL-safe secret (characters A-Z, a-z, 0-9, `_`, `-`).
|
|
|
|
2. **Compute:**
|
|
|
|
```text
|
|
delta = now() - T
|
|
counter = floor(delta / 300)
|
|
```
|
|
|
|
3. **Derive:**
|
|
|
|
- Initialize BLAKE2s with key=secret, digest_size=16.
|
|
- Update with counter as 8-octet little-endian.
|
|
- Finalize to obtain 16-octet token.
|
|
|
|
4. **Assuming:**
|
|
|
|
- All secrets are provisioned out-of-band.
|
|
- The clock is in-sync.
|
|
|
|
Example (Python):
|
|
|
|
```py
|
|
import hashlib
|
|
import struct
|
|
import time
|
|
|
|
|
|
def gen_token(secret: bytes, timestamp: int) -> bytes:
|
|
"""Generate a token"""
|
|
|
|
return hashlib.blake2s(
|
|
struct.pack("<Q", int((time.time() - timestamp) / 300)), # data
|
|
digest_size=16,
|
|
key=secret,
|
|
).digest()
|
|
```
|
|
|
|
The server MAY accept a counter drift +-1, but it is NOT guaranteed.
|
|
|
|
## 12. Proof-of-Work Algorithm
|
|
|
|
- **Inputs:**
|
|
|
|
- 16-byte `challenge` (opaque, secret, server-generated).
|
|
- 8-bit `difficulty`: number of required leading zero bits.
|
|
- 8-bit `ones`: number of required XOR-passing bytes.
|
|
|
|
- **Valid solution:**
|
|
|
|
A nonce is considered valid if:
|
|
|
|
- The 32-byte BLAKE2s digest of the ASCII decimal `nonce` (keyed with the `challenge`) has at least `difficulty` leading 0 bits.
|
|
- At least `ones` out of the first 32 digest bytes satisfy: XOR with the `challenge` byte (modulo 16) contains more or equal to 6 ones (in binary)
|
|
|
|
- **Compute:**
|
|
|
|
For a given nonce:
|
|
|
|
```text
|
|
nonce_string = string(nonce) # e.g., "12345"
|
|
hash = BLAKE2s(nonce_string, key=challenge)
|
|
```
|
|
|
|
Then verify:
|
|
|
|
```
|
|
has_leading_zero_bits(hash, difficulty) == true
|
|
AND
|
|
has_xor_ones(hash, ones, challenge) == true
|
|
```
|
|
|
|
- **Bit-Checking Logic:**
|
|
- Leading-zero bits are checked by verifying that:
|
|
- The full zero bytes match.
|
|
- The remaining prefix bits (if any) are zero via masking.
|
|
- XOR byte-pair test passes if:
|
|
- XOR of `challenge[i % 16]` and `digest[i % 32]` contains more or equal to 6 one-bits, where `i` is in range `[0;31]`
|
|
- Repeat until ones such bytes are found (or fail early)
|
|
|
|
Example (Python):
|
|
|
|
```py
|
|
import hashlib
|
|
|
|
|
|
def has_leading_zero_bits(h: bytes, difficulty: int) -> bool:
|
|
"""Check bit prefix requirements"""
|
|
|
|
full_bytes: int = difficulty // 8
|
|
remainder_bits: int = difficulty % 8
|
|
|
|
if h[:full_bytes] != b"\x00" * full_bytes:
|
|
return False
|
|
|
|
if remainder_bits > 0:
|
|
next_byte: int = h[full_bytes]
|
|
mask: int = 0xFF << (8 - remainder_bits) & 0xFF
|
|
|
|
if (next_byte & mask) != 0:
|
|
return False
|
|
|
|
return True
|
|
|
|
|
|
def count_ones(num: int) -> int:
|
|
"""Fast bit-counting using Hacker's Delight method"""
|
|
|
|
num = (num - ((num >> 1) & 0x55)) & 0xFF
|
|
num = ((num & 0x33) + ((num >> 2) & 0x33)) & 0xFF
|
|
return (num + (num >> 4)) & 0x0F
|
|
|
|
|
|
def has_xor_ones(h: bytes, ones: int, challenge: bytes) -> bool:
|
|
"""Check XOR requirements"""
|
|
|
|
ok_ones: int = 0
|
|
|
|
for idx in range(32):
|
|
if count_ones(challenge[idx % 16] ^ h[idx % 32]) >= 6:
|
|
ok_ones += 1
|
|
|
|
if ok_ones == ones:
|
|
return True
|
|
|
|
return False
|
|
|
|
|
|
def solve_pow(
|
|
difficulty: int, ones: int, challenge: bytes, nonce: int = 0, batch: int = 2**64 - 1
|
|
) -> t.Tuple[int, bool]:
|
|
"""Solve proof of work"""
|
|
|
|
for _ in range(batch):
|
|
if nonce > 2**64 - 1:
|
|
return nonce, False
|
|
|
|
h: bytes = hashlib.blake2s(str(nonce).encode("ascii"), key=challenge).digest()
|
|
|
|
if has_leading_zero_bits(h, difficulty) and has_xor_ones(h, ones, challenge):
|
|
return nonce, True
|
|
|
|
nonce += 1
|
|
|
|
return nonce, False
|
|
```
|
|
|
|
## 13. Security Considerations
|
|
|
|
- **Replay Protection**: Unique PoW nonces, time-bounded tokens.
|
|
- **Confidentiality & Integrity**: TLS mandatory; BLAKE2s keyed hashing.
|
|
- **DoS Mitigation**: Modern and dynamic PoW challenge; ping rate limits.
|
|
- **Strict Parsing**: All length fields and value ranges are enforced exactly.
|
|
- **Authentication Failures**: Explicit error codes for invalid keys or tokens.
|
|
- **Timeouts**: 5s read/write socket timeouts to prevent stalling.
|
|
- **Version Pinning**: Unknown protocol versions are rejected.
|
|
- **Domain Locking**: Prevents race conditions during client registration.
|
|
- **Immutable Logs**: No log mutation; entire stream SHOULD be read.
|
|
- **Silent Failure on Malformed Packets**: Unless required, no responses.
|
|
- **One-time Use Nonces**: Prevent precomputation or reuse.
|
|
- **Out-of-Band Trust Model**: Secrets MUST be securely provisioned.
|
|
|
|
## 14. Authors
|
|
|
|
- Arija A. \<<ari@ari.lt>\> - Author of dproto.
|
|
|
|
This protocol is licensed under AGPL-3.0-only and is part of the DeployD project which you can find at <https://git.ari.lt/ari/deployd>.
|
|
|
|
More information about the license can be found at <https://git.ari.lt/ari/deployd/raw/branch/main/LICENSE>.
|