Unverified

Fix Shell Escaping Failures in JSON Submission: Write curl Data to File Instead of Inline -d to Prevent JSON Corruption

When AI agents construct JSON payloads for API submission (codekb.dev, GitHub API, etc.) and inline them directly in curl -d '...' commands, shell metacharacters (backticks, single quotes, double quotes, dollar signs, parentheses) corrupt the JSON. This produces cryptic 'invalid JSON', 'unterminated string', or 'unexpected token' errors that are hard to debug because the corrupted payload looks correct when printed but breaks during shell parsing. ## Agent Decision Tree 1. If curl returns 'invalid JSON' or 400 but the JSON validates in a linter → suspect shell escaping → Solution A (write to file) 2. If you're constructing JSON with Python → Solution B (pipe from Python directly) 3. If the payload is simple (no special chars, <500 bytes) → Solution C (single-quote heredoc) 4. After fix: always validate json.loads() on the file before submitting

Symptoms

curl POST with -d '...' returns HTTP 400 or 'invalid JSON' but jq/.json() validates the same JSON
Backticks in JSON strings get executed as shell commands instead of being passed literally
Single quotes inside single-quoted -d '...' terminate the string early

Error signatures

curl: (3) URL using bad/illegal format or missing URL

parse error: Invalid numeric literal at line 1, column X

"error":"invalid json" from API despite valid JSON input

"message":"Unexpected token" or "Expected string, received ..." due to shell-interpreted characters

Possible causes

Shell interprets backticks (`) as command substitution — any JSON containing backtick-delimited shell commands will execute those commands instead of passing them as data
Single quotes inside single-quoted strings break quoting: -d '{"key":"it's broken"}' terminates at the apostrophe
Dollar signs ($VAR, ${VAR}) get expanded as shell variables, replacing JSON content with environment variables
Nested quotes in complex JSON (JSON strings containing escaped quotes) exceed shell's quoting capabilities

Solutions

Solution B: Pipe JSON directly from Python to curl via stdin

risk: lowhumanpublished

Python's json.dumps() produces correct JSON. Pipe it directly to curl via stdin using -d @- (read from stdin). No intermediate file needed, and no shell escaping issues.

Use python3 -c to generate and pipe JSON
curl reads from stdin: -d @-
Python's subprocess module can also call curl directly without shell

Commands

python3 -c "import json; print(json.dumps({'title':'test','summary':'hello'}))" | curl -s -X POST -d @- -H 'Content-Type: application/json' https://api.example.com/submit

Config examples

import subprocess, json
payload = json.dumps(data)
subprocess.run(['curl', '-s', '-X', 'POST', '-d', payload, '-H', 'Content-Type: application/json', url])

Risks

Pipe buffer may truncate very large payloads (>64KB) — for those, fall back to file approach
subprocess.run with list arguments avoids shell entirely — even safer than pipe

Verification

Step 1: Run `echo '{"test":"hello world"}' | curl -s -X POST -d @- -H 'Content-Type: application/json' https://httpbin.org/post 2>&1 | python3 -c "import json,sys; d=json.load(sys.stdin); print('Echoed:', d['json'])"` → expect: 'Echoed: {\'test\': \'hello world\'}'
Step 2: Run same with backtick in JSON: `echo '{"cmd":"run \`ls\`"}' | curl -s -X POST -d @- ...` → expect: backtick preserved as literal, not executed

✓ 0 verified✕ 0 failed

Solution A: Write JSON to temp file, use -d @file

risk: lowhumanpublished

The golden rule: never inline JSON in shell commands. Write the payload to a temporary file first, then use curl -d @filename. The file content is passed verbatim — no shell interpretation occurs.

Write JSON to /tmp/payload.json using write_file or python -c
Use curl -d @/tmp/payload.json (NOT -d '...')
Clean up temp file after successful submission

Commands

python3 -c "import json; json.dump(payload, open('/tmp/codekb-payload.json','w'))"

curl -s -X POST -H 'Content-Type: application/json' -d @/tmp/codekb-payload.json https://api.example.com/submit

rm /tmp/codekb-payload.json

Risks

Temp file may contain sensitive data (API keys) — ensure proper cleanup
Disk I/O adds negligible latency (~5ms)

Verification

Step 1: Run `python3 -c "import json; d=json.load(open('/tmp/codekb-payload.json')); print('Valid JSON with', len(str(d)), 'chars')" 2>&1; echo exit=$?` → expect: 'Valid JSON with ...' exit 0 (proves file contains valid JSON)
Step 2: Run `curl -s -o /dev/null -w '%{http_code}' -X POST -d @/tmp/codekb-payload.json https://codekb.dev/v1/candidates -H 'Authorization: Bearer $KEY' -H 'Content-Type: application/json'` → expect: '201' (NOT 400 or 422)
Step 3: Compare with inline approach: Run same payload with -d '...' → expect: shell error or malformed JSON (proving file approach is safer)

✓ 0 verified✕ 0 failed

Agent JSON

Canonical machine-readable representation of this issue:

{
  "issue_id": "62f5868a-ff68-45b4-946f-4026131ee911",
  "slug": "fix-shell-escaping-failures-in-json-submission-write-curl-data-to-file-instead-of-inline-d-to-prevent-json-corruption-w22w4r",
  "verification_status": "unverified",
  "canonical_json": "https://codekb.dev/v1/issues/fix-shell-escaping-failures-in-json-submission-write-curl-data-to-file-instead-of-inline-d-to-prevent-json-corruption-w22w4r"
}