Force LLMs to print JSON only

[knqyf263]

Description

I tested vexllm with Anthropic Claude 3.5 Sonnet and faced the following error.

$ vexllm generate python.json .trivyignore --hint-unused-commands=git,wget,curl,apt,apt-get --llm anthropic
Based on the provided information and hints, I can determine that all the vulnerabilities related to Git can be considered negligible for this specific container image. Here's the analysis:

\```json
{
    "CVE-2024-32002": {"confidence": 0.9, "reason": "This vulnerability affects Git, which is listed as an unused command in the container image. The risk of exploitation is minimal as Git operations are not expected to be performed within this Python container."},
    "CVE-2023-25652": {"confidence": 0.9, "reason": "This Git vulnerability is negligible as Git is listed as an unused command in the container. The `git apply` command is unlikely to be executed in a Python runtime environment."},
    "CVE-2023-29007": {"confidence": 0.9, "reason": "This vulnerability affects Git submodule operations, which are not relevant in a Python container image where Git is listed as an unused command."},
    "CVE-2024-32004": {"confidence": 0.9, "reason": "This Git vulnerability relates to cloning local repositories, which is not a typical operation in a Python runtime environment. Git is also listed as an unused command in this container."},
    "CVE-2024-32465": {"confidence": 0.9, "reason": "This additional Git RCE vulnerability is negligible as Git operations, including working with repositories, are not expected in this Python container image where Git is listed as unused."}
}
\```

The confidence level is high (0.9) for all these vulnerabilities because:

1. Git is explicitly mentioned in the list of unused commands for this container image.
2. The container image is specifically for Python (python:3.12.4), which typically doesn't involve Git operations in its runtime environment.
3. These vulnerabilities require specific Git operations to be exploited, which are unlikely to occur in a Python runtime container.

It's important to note that while these vulnerabilities are present in the scan results, they pose minimal risk in the context of this specific Python container image, given that Git is not expected to be used.
time=2024-07-19T15:35:38.800+04:00 level=ERROR msg=Error error="unparsable JSON output from LLM: invalid character 'B' looking for beginning of value: \"Based on the provided information and hints, I can determine that all the vulnerabilities related to Git can be considered negligible for this specific container image. Here's the analysis:\\n\\n```json\\n{\\n    \\\"CVE-2024-32002\\\": {\\\"confidence\\\": 0.9, \\\"reason\\\": \\\"This vulnerability affects Git, which is listed as an unused command in the container image. The risk of exploitation is minimal as Git operations are not expected to be performed within this Python container.\\\"},\\n    \\\"CVE-2023-25652\\\": {\\\"confidence\\\": 0.9, \\\"reason\\\": \\\"This Git vulnerability is negligible as Git is listed as an unused command in the container. The `git apply` command is unlikely to be executed in a Python runtime environment.\\\"},\\n    \\\"CVE-2023-29007\\\": {\\\"confidence\\\": 0.9, \\\"reason\\\": \\\"This vulnerability affects Git submodule operations, which are not relevant in a Python container image where Git is listed as an unused command.\\\"},\\n    \\\"CVE-2024-32004\\\": {\\\"confidence\\\": 0.9, \\\"reason\\\": \\\"This Git vulnerability relates to cloning local repositories, which is not a typical operation in a Python runtime environment. Git is also listed as an unused command in this container.\\\"},\\n    \\\"CVE-2024-32465\\\": {\\\"confidence\\\": 0.9, \\\"reason\\\": \\\"This additional Git RCE vulnerability is negligible as Git operations, including working with repositories, are not expected in this Python container image where Git is listed as unused.\\\"}\\n}\\n```\\n\\nThe confidence level is high (0.9) for all these vulnerabilities because:\\n\\n1. Git is explicitly mentioned in the list of unused commands for this container image.\\n2. The container image is specifically for Python (python:3.12.4), which typically doesn't involve Git operations in its runtime environment.\\n3. These vulnerabilities require specific Git operations to be exploited, which are unlikely to occur in a Python runtime container.\\n\\nIt's important to note that while these vulnerabilities are present in the scan results, they pose minimal risk in the context of this specific Python container image, given that Git is not expected to be used.\""
Exiting.

It seems like Claude prints its explanation in addition to the JSON content.

[AkihiroSuda]

Thanks

[AkihiroSuda]

llms.WithJSONMode() isn't enough?
Maybe this should be reported to https://github.com/tmc/langchaingo/issues

[knqyf263]

OpenAI supports JSON mode, but I didn't find it in Anthoropic.

[knqyf263]

Looks like Cluade itself doesn't have "JSON Mode" for now.
https://github.com/anthropics/anthropic-cookbook/blob/68028f4761c5dbf158b7bf3d43f2f45b44111200/misc/how_to_enable_json_mode.ipynb