I have a Rails app that sends user-provided text to Claude for analysis and displays structured results in the UI. The response needs to be JSON so I can render it.

However longer inputs sometimes would generate errors. Longer inputs meant longer system prompts and longer responses.

The logs showed:

Analysis failed: expected ',' or '}' after object value

Claude was generating valid-looking JSON that wasn’t actually valid. A dropped comma deep in a large response object. The longer the response, the more likely this happened.

The Old Approach: Prompt Engineering + Defensive Parsing

My system prompt included a 28-line block demanding JSON output:

SYSTEM_PROMPT = <<~PROMPT
  ...
  CRITICAL INSTRUCTIONS:
  - You MUST ALWAYS respond with valid JSON. No exceptions. No explanations outside JSON.
  - NEVER respond with plain text - always use the JSON format.

  You MUST ALWAYS respond in JSON format with the following structure (no exceptions):
  {
    "score": 0-100,
    "level": "low|medium|high",
    "summary": "Brief overall assessment",
    "items": [
      {
        "title": "Issue name",
        "description": "What's wrong",
        "severity": "low|medium|high"
      }
    ],
    "recommendations": [
      "Specific actionable suggestion 1",
      "Specific actionable suggestion 2"
    ]
  }
PROMPT

Despite all the shouting in the prompt, Claude would sometimes:

• Wrap the JSON in markdown code fences (```json ... ```)
• Add explanatory text after the closing brace
• Drop commas in deeply nested objects on long responses
• Return plain text when it decided the input wasn’t suitable for analysis

So I built a pipeline of defensive code to handle all of this.

Step 1: Extract JSON from whatever Claude returned. A brace-matching parser that stripped markdown fences, found the first {, tracked nesting depth while respecting string escaping, and separated trailing notes:

def extract_json_and_note(text)
  text = text.strip
  if text.start_with?("```")
    text = text.sub(/A```(?:json|JSON)?s*/, "").sub(/s*```z/, "").strip
  end

  start_idx = text.index("{")
  return [text, nil, false] if start_idx.nil?

  brace_count = 0
  in_string = false
  escape_next = false

  text[start_idx..].each_char.with_index do |char, idx|
    if escape_next
      escape_next = false
      next
    end
    case char
    when "\" then escape_next = true if in_string
    when '"'  then in_string = !in_string unless escape_next
    when "{"  then brace_count += 1 unless in_string
    when "}"
      brace_count -= 1 unless in_string
      if brace_count == 0
        end_idx = start_idx + idx
        json_text = text[start_idx..end_idx]
        note = text[(end_idx + 1)..].strip.presence
        return [json_text, note, true]
      end
    end
  end

  [text, nil, false]
end

Step 2: Normalize missing fields because Claude might omit arrays for edge cases:

def normalize_response!(result)
  result["score"] ||= 0
  result["level"] ||= "unknown"
  result["summary"] ||= "Analysis complete"
  result["items"] ||= []
  result["recommendations"] ||= []
  result["score"] = result["score"].to_i if result["score"].is_a?(String)
end

Step 3: Validate the structure because even after parsing, I couldn’t trust it:

def validate_response_structure!(result)
  required_keys = %w[score level summary items recommendations]
  missing_keys = required_keys - result.keys
  raise "Invalid response structure: missing keys #{missing_keys.join(', ')}" if missing_keys.any?

  score = result["score"]
  unless score.is_a?(Integer) && score >= 0 && score <= 100
    raise "Invalid score: must be integer 0-100, got #{score.inspect}"
  end

  %w[items recommendations].each do |key|
    unless result[key].is_a?(Array)
      raise "Invalid #{key}: expected array, got #{result[key].class}"
    end
  end
end

All of this existed because I was asking an LLM to format its own output as JSON via natural language instructions. I was writing a fragile parser for a format the model was never constrained to produce.

The Fix: Tool Use

Anthropic’s tool use API (also called function calling) lets you define a JSON schema that Claude must conform to. Instead of asking Claude to output JSON, you tell the API: “call this function with these typed parameters.” Claude’s response is guaranteed to match the schema.

Here’s the schema definition:

ANALYSIS_TOOL = {
  name: "analyze",
  description: "Return the structured analysis results",
  input_schema: {
    type: "object",
    required: ["score", "level", "summary", "items", "recommendations"],
    properties: {
      score: { type: "integer", description: "Overall score from 0 (safe) to 100 (dangerous)" },
      level: { type: "string", enum: ["low", "medium", "high"] },
      summary: { type: "string", description: "Brief overall assessment" },
      items: {
        type: "array",
        items: {
          type: "object",
          required: ["title", "description", "severity"],
          properties: {
            title:       { type: "string" },
            description: { type: "string" },
            severity:    { type: "string", enum: ["low", "medium", "high"] }
          }
        }
      },
      recommendations: { type: "array", items: { type: "string" } }
    }
  }
}.freeze

The API call adds two parameters:

response = client.messages.create(
  model: model,
  max_tokens: max_tokens,
  system: [{ type: "text", text: system_prompt }],
  messages: [{ role: "user", content: user_message }],
  tools: [ANALYSIS_TOOL],
  tool_choice: { type: "tool", name: "analyze" }
)

tools: defines the schema. tool_choice: with type: "tool" forces Claude to use it — no chance of returning prose instead.

Response extraction is three lines:

tool_block = response.content.find { |b| b.type.to_s == "tool_use" }
raise "No tool_use block in response" unless tool_block
result = tool_block.input.transform_keys(&:to_s)

That’s it. tool_block.input is already a parsed hash. No JSON.parse, no brace matching, no markdown stripping, no comma repair.

The Result

Deleted: ~160 lines from the service, ~250 lines from tests..

Added: ~30 lines for the schema definition, 2 parameters on the API call, 3 lines of response extraction.

The system prompt shrank too. The 28 lines of “YOU MUST RESPOND IN JSON” instructions disappeared entirely. The prompt now focuses on what to analyze, not how to format the output.

The user message went from "Analyze this and respond with JSON only:" to just "Analyze this:".

When Should You Use This?

Any time you want structured output from an LLM. If you’re writing regex to fix JSON commas, building brace-matching parsers, or adding “RESPOND IN JSON ONLY” to your prompts: switch to tool use. The schema is self-documenting, the output is guaranteed valid, and you delete code instead of writing it.

The one caveat: tool use constrains the structure but not the content. Claude can still put whatever it wants in a string field. You still need to validate that a score is in a sensible range or that enum values match your expectations. But “validate the values” is a much smaller problem than “parse arbitrary text that might be JSON.”