You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
response_evaluation_score (COHERENCE) fails with INVALID_ARGUMENT: Error parsing JSON in all regions — vertexai 1.153.1 #6853
When using PrebuiltMetric.COHERENCE via client.evals.evaluate() (which backs response_evaluation_score in google-adk), the call consistently fails with:
400 INVALID_ARGUMENT: Error parsing JSON. Last error: Expecting property name
enclosed in double quotes: line 1 column 2 (char 1), Input: {* **STEP 1: ...}
The autorater LLM generates a free-text response starting with {* (bullet points), and the :evaluateInstances endpoint attempts to parse it as JSON and fails.
Reproduction
importvertexaiimportpandasaspdclient=vertexai.Client(project="<project>", location="us-central1")
df=pd.DataFrame([{
"prompt": "What slabs are in the model?",
"response": "The model has 17 C25 concrete slabs."
}])
result=client.evals.evaluate(
dataset=df,
metrics=[vertexai.types.PrebuiltMetric.COHERENCE],
)
Environment
google-cloud-aiplatform (vertexai): 1.153.1
Regions tested: us-central1, europe-west1, europe-west3 — all fail
The COHERENCE metric YAML (gs://vertex-ai-generative-ai-eval-sdk-resources/metrics/coherence/v1.yaml) produces free-text output, but the :evaluateInstances server endpoint attempts to parse the autorater response as JSON and fails
Expected behavior
Metric evaluation completes and returns a coherence score.
Actual behavior
num_cases_error=1, num_cases_valid=0 on every call. The full error from the server:
google.genai.errors.ClientError: 400 INVALID_ARGUMENT.
Error parsing JSON. Last error: Expecting property name enclosed in double quotes:
line 1 column 2 (char 1), Input: {* **STEP 1: Identify the purpose and audience:...
The autorater generates a valid free-text evaluation but the server-side JSON parser rejects it.
Activity