111 lines
4.4 KiB
Markdown
111 lines
4.4 KiB
Markdown
# claude-tokenization-proxy
|
||
|
||
A local Data Loss Prevention (DLP) proxy that sits between `claude-code` and the
|
||
Anthropic API. It scans outgoing requests for secrets with
|
||
[GitGuardian's `ggshield`](https://github.com/GitGuardian/ggshield), swaps each
|
||
secret for an opaque placeholder token before it leaves your machine, then
|
||
restores the real secret in the streamed response on the fly — so Claude never
|
||
sees your secrets but tooling that echoes a placeholder still gets the real
|
||
value back.
|
||
|
||
```
|
||
claude-code ──POST /v1/messages──► proxy ──redacted──► api.anthropic.com
|
||
(ggshield scan, secret → <<<gg_token_…>>>) │
|
||
claude-code ◄──SSE (detokenized)─── proxy ◄──SSE stream──┘
|
||
(<<<gg_token_…>>> → real secret, streaming)
|
||
```
|
||
|
||
## How it works
|
||
|
||
1. **Request interception** — the JSON body of `POST /v1/messages` is scanned by
|
||
`ggshield`. Every detected secret is replaced with a unique
|
||
`<<<gg_token_[hex]>>>` placeholder, and the `token → secret` mapping is kept
|
||
in memory (`src/store.ts`). The redacted body is forwarded upstream.
|
||
2. **Streaming detokenization** — the Anthropic SSE response is piped through a
|
||
stateful `Transform` (`src/detokenize.ts`) that scans the byte stream for
|
||
placeholders and substitutes the original secret back. It holds back only the
|
||
few characters that could still become a token, so a placeholder split across
|
||
chunk boundaries (`…<<<gg_to` / `ken_abc>>>…`) is reassembled correctly.
|
||
|
||
Secrets live only in memory for the lifetime of the process — restart the proxy
|
||
and the mapping is gone.
|
||
|
||
## Requirements
|
||
|
||
- **Node.js ≥ 23.6** (the proxy runs TypeScript directly via Node's built-in
|
||
type stripping — no build step, no `ts-node`/`tsx`). On Node 22.6–23.5 run
|
||
with `node --experimental-strip-types src/index.ts`.
|
||
- **`ggshield`** on your `PATH` and authenticated (`ggshield auth login`). If
|
||
`ggshield` is missing or errors, the proxy logs a warning and forwards traffic
|
||
**unredacted** (fail-open) — it never blocks your requests.
|
||
|
||
## Setup
|
||
|
||
```bash
|
||
npm install # dev-only deps (TypeScript types); not needed to run
|
||
npm start # starts the proxy on http://127.0.0.1:8080
|
||
```
|
||
|
||
Point `claude-code` at the proxy:
|
||
|
||
```bash
|
||
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
|
||
claude
|
||
```
|
||
|
||
For the [fish shell](https://fishshell.com/):
|
||
|
||
```fish
|
||
set -gx ANTHROPIC_BASE_URL http://localhost:8080
|
||
claude
|
||
```
|
||
|
||
That's the only change `claude-code` needs — it keeps using your normal
|
||
`ANTHROPIC_API_KEY` / `x-api-key`, which the proxy forwards untouched.
|
||
|
||
## Configuration
|
||
|
||
| Env var | Default | Purpose |
|
||
| ----------------------- | ------------------ | -------------------------------- |
|
||
| `PORT` | `8080` | Port the proxy listens on |
|
||
| `ANTHROPIC_UPSTREAM_HOST` | `api.anthropic.com` | Upstream API host to forward to |
|
||
|
||
## Tell Claude to leave the tokens alone
|
||
|
||
So the model preserves placeholders verbatim (e.g. when writing them to files),
|
||
copy [`templates/CLAUDE.md`](templates/CLAUDE.md) into the `CLAUDE.md` of any
|
||
project you run `claude-code` in:
|
||
|
||
```markdown
|
||
# Security Constraints
|
||
|
||
Do not modify, remove, or evaluate strings formatted as `<<<gg_token_[a-z0-9]+>>>`.
|
||
Treat them as opaque identifiers and preserve them exactly as they appear when
|
||
writing or reading files.
|
||
```
|
||
|
||
## Scripts
|
||
|
||
| Command | What it does |
|
||
| ----------------- | ----------------------------------------------------- |
|
||
| `npm start` | Run the proxy |
|
||
| `npm run dev` | Run with `--watch` (restarts on file changes) |
|
||
| `npm test` | Run the detokenizer state-machine self-check |
|
||
| `npm run typecheck` | Type-check the source with `tsc --noEmit` |
|
||
|
||
## Limitations
|
||
|
||
- Detection is only as good as `ggshield`. It catches known secret patterns and
|
||
generic high-entropy strings, not arbitrary sensitive prose.
|
||
- Re-inserted secrets are assumed to be escape-neutral (typical API
|
||
keys/tokens). A secret literally containing a backslash or double-quote could
|
||
mis-escape inside the response JSON. See the `ponytail:` note in
|
||
`src/detokenize.ts`.
|
||
- The token map is in-memory only; it is not shared across proxy restarts or
|
||
multiple proxy processes.
|
||
|
||
## License
|
||
|
||
Copyright (C) 2026 Rootiest. Licensed under the
|
||
[GNU Affero General Public License v3.0 or later](LICENSE).
|