Creditsafe MCP - beta test report

[!IMPORTANT] The server is reachable and usable: 55 of 95 executed scenarios behave as expected, 37 work but can be improved and 3 fail. Most opportunities concern being LLM-agent friendly, in order of impact:

• 🔴 Functional behaviour differing from expectations (3)
• 🔴 Responses too large for an LLM context window (3)
• 🟠 Errors returned as successful results (26)
• 🟠 Tool and parameter documentation gaps (21)
• 🟠 Missing MCP metadata (annotations, outputSchema) (18)

Two complementary angles, applied to every tool:

1. Functional: does each tool return correct, complete data for realistic requests (search by name / SIREN / VAT / safe number, France, United Kingdom, Germany with legal reason_code, directors, group structure, beneficial owners, events, adverse information, financials, payment behaviour)?
2. MCP / LLM-agent oriented: is the server pleasant for an AI agent? Size of results versus context window, error contract (isError), actionable error messages, quality of tool definitions (descriptions, parameter docs, annotations, output schema), latency, noise in payloads, behaviour on invalid or hostile input.

• Black-box: a Python client speaks the real MCP protocol (Streamable HTTP) to the beta endpoint, exactly like an agent would. No access to server internals.
• Sequence mimics a human tester: discovery (list_tools), then searches, then every company tool on FR / GB / DE, directors, then edge cases (empty, very long, special characters, wrong types, unknown tool, invalid country / reason_code).
• Test company without stake: BearingPoint France (SIREN 443021241); UK and DE examples taken from Creditsafe's own tool descriptions.
• Each case opens a fresh session so a transport failure only affects that case.
• Each response is evaluated automatically against: expected outcome (success / error / not found), isError flag, presence of an actionable hint, size in characters and estimated tokens, share of empty fields, duration, JSON validity, raw technical text leaks.
• Credit-consuming calls (refresh=True, PDF, director report) are skipped unless explicitly enabled.
• Full requests/responses are kept in verbose.log and raw.jsonl; the API key is redacted everywhere.

#What works well

• 1.01 Account access and entitlements
• 1.02 Supported countries
• 1.03 Search fields for France
• 1.04 Search fields for several countries
• 1.06 Report languages, France
• 1.07 Report languages, UK
• 1.08 Report schema, France (summary)
• 1.09 Report schema, UK (summary)

• 2.01 Search by name, France
• 2.02 Search by registration number (SIREN)
• 2.03 Search by VAT number
• 2.04 Search by safe number, UK
• 2.05 Search by name, UK large company
• 2.06 Search, exact match
• 2.07 Search, multi-country
• 2.08 Search, city filter
• 2.09 Search, postcode filter
• 2.10 Search, status filter
• 2.11 Search, page_size=1
• 2.12 Search, page 2
• 2.14 Search with typo
• 2.15 Search with accents
• 2.18 Search, lowercase country
• 2.19 Search, unknown company

• 3.01 get_company_overview, French company
• 3.02 get_company_financials, French company
• 3.03 get_company_payment_behaviour, French company
• 3.05 get_company_adverse_information, French company
• 3.06 get_company_group_structure, French company
• 3.07 get_company_beneficial_owners, French company
• 3.08 get_company_directors, French company
• 3.13 Overview without country
• 3.14 Overview repeated (cache check)
• 3.15 Financials, years=1
• 3.16 Financials, years=5
• 3.17 Financials, global basis
• 3.18 Financials, English

• 4.01 get_company_overview, UK company
• 4.02 get_company_financials, UK company
• 4.03 get_company_payment_behaviour, UK company
• 4.04 get_company_events, UK company
• 4.05 get_company_adverse_information, UK company
• 4.06 get_company_group_structure, UK company
• 4.08 get_company_directors, UK company
• 4.09 get_company_directorships, UK company
• 4.12 Overview by registration number
• 4.13 Financials with global basis

• 5.06 FR overview with reason_code (should be ignored)

• 6.01 Search directors, UK
• 6.02 Search directors, France
• 6.03 Search directors, multi-country
• 6.04 Search directors with date of birth
• 6.05 Search directors, malformed date

• 7.10 Unknown extra argument
• 7.11 Unknown tool

#🔴 Functional behaviour differing from expectations

Why it matters: The tool does not return what a user would expect for a valid request.

[!TIP] Suggestion: See each case below.

• 🔴 FAIL - 3.11 get_company_overview: Unexpected error: { "error": { "code": "ambiguous", "message": "Multiple companies match '443021241'.", "hint": "Use the Creditsafe ID, safe number, registration number, VAT, or add postcode/city.", "candidates": [ { "
• 🔴 FAIL - 3.12 get_company_overview: Unexpected error: { "error": { "code": "not_found", "message": "No company found for 'FR96443021241'.", "hint": "Try safe number, registration number, VAT number, or name plus postcode." } }
• 🟠 WARN - 5.01 get_company_overview: Expected 'reason' in the response

#🔴 Responses too large for an LLM context window

Why it matters: An agent pastes every tool result into its context. Huge payloads crowd out the conversation, cost money on every following turn, or exceed the window entirely.

[!TIP] Suggestion: Return a compact view by default; add a fields/sections parameter and pagination; announce truncation explicitly; keep raw payloads behind a separate opt-in tool.

• 🔴 FAIL - 1.11 get_company_report_schema: Response of 870,747 chars (~248,785 tokens) exceeds typical LLM context windows (100,000 tokens): the tool is unusable by an agent in this mode
• 🟠 WARN - 3.04 get_company_events: Very large response (83,199 chars, ~23,771 tokens): consumes a large share of an LLM context
• 🟠 WARN - definition of (all tools): Tool definitions alone weigh ~11,122 tokens, paid on every LLM turn

#🟠 Errors returned as successful results

Why it matters: MCP clients and agent frameworks rely on isError=true to trigger retry/repair logic. An error body inside a normal result looks like a success.

[!TIP] Suggestion: Set isError=true on every failure, keeping the structured error body.

• 🟠 WARN - 1.05 get_company_search_fields: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 1.10 get_company_report_schema: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 2.16 search_companies: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 2.17 search_companies: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 3.09 get_company_directorships: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 3.10 get_company_overview: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 3.11 get_company_overview: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 3.12 get_company_overview: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 3.22 get_company_directorships: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 4.07 get_company_beneficial_owners: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 4.10 get_company_overview: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• 🟠 WARN - 4.11 get_company_overview: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
• ... and 14 more (see appendix / results.xlsx)

#🟠 Tool and parameter documentation gaps

Why it matters: The LLM chooses tools and fills arguments only from names and descriptions. Missing descriptions or valid values cause wrong calls.

[!TIP] Suggestion: Describe every parameter (format, example, allowed values), use enums/patterns in the JSON schema.

• 🟠 WARN - 3.09 get_company_directorships: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
• 🟠 WARN - 3.22 get_company_directorships: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
• 🟠 WARN - 4.07 get_company_beneficial_owners: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
• 🟠 WARN - definition of get_company_search_fields: Parameter 'countries' is a free string without enum/pattern: valid values are only discoverable by trial and error
• 🟠 WARN - definition of get_report_languages: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
• 🟠 WARN - definition of get_company_report_schema: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
• 🟠 WARN - definition of get_company_overview: Parameter 'language' has no description
• 🟠 WARN - definition of get_company_financials: Parameter 'years' has no description
• 🟠 WARN - definition of get_company_financials: Parameter 'language' has no description
• 🟠 WARN - definition of get_company_payment_behaviour: Parameter 'language' has no description
• 🟠 WARN - definition of get_company_events: Parameter 'language' has no description
• 🟠 WARN - definition of get_company_adverse_information: Parameter 'language' has no description
• ... and 9 more (see appendix / results.xlsx)

#🟠 Missing MCP metadata (annotations, outputSchema)

Why it matters: Annotations let clients auto-approve read-only calls and warn on costly ones; outputSchema helps downstream parsing.

[!TIP] Suggestion: Declare readOnlyHint / idempotentHint, flag credit-consuming tools, publish outputSchema.

• 🟠 WARN - definition of get_my_creditsafe_access: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_supported_countries: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_search_fields: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_report_languages: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_report_schema: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of search_companies: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_overview: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_financials: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_payment_behaviour: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_events: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_adverse_information: No outputSchema: the shape of the result is undocumented
• 🟠 WARN - definition of get_company_group_structure: No outputSchema: the shape of the result is undocumented
• ... and 6 more (see appendix / results.xlsx)

#🟠 Raw framework / validation errors exposed

Why it matters: Stack-like or pydantic texts are noisy, hard for a model to act on, and leak implementation details.

[!TIP] Suggestion: Map validation failures to short, friendly messages naming the field and the allowed range.

• 🟠 WARN - 2.13 search_companies: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 3.19 get_company_financials: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 3.20 get_company_financials: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 3.21 get_company_financials: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 5.04 get_company_overview: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 5.05 get_company_overview: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 6.07 search_directors: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 7.08 get_company_overview: Message exposes raw technical details (framework validation text / traceback / 5xx)
• 🟠 WARN - 7.09 get_company_financials: Message exposes raw technical details (framework validation text / traceback / 5xx)

#🟠 Errors that do not tell the model how to recover

Why it matters: A good MCP error says what was wrong and what to do next, so the LLM can self-correct without a human.

[!TIP] Suggestion: Always return code + message + hint (valid values, expected format, next tool to call).

• 🟠 WARN - 2.16 search_companies: Error has no 'hint': an LLM cannot tell how to retry
• 🟠 WARN - 2.17 search_companies: Error has no 'hint': an LLM cannot tell how to retry
• 🟠 WARN - 6.08 search_directors: No explicit 'not found' message for an unknown entity

#🟠 Calls rejected or failing at transport level

Why it matters: The client gets an HTTP/protocol exception instead of a tool result and cannot show a useful message to the model.

[!TIP] Suggestion: Return a tool-level error for unusual inputs; check that WAF rules do not block legitimate text.

• 🟠 WARN - 7.06 get_company_overview: Call raised at protocol level instead of returning a tool result: HTTP 403 Forbidden (transport level)
• 🟠 WARN - 7.07 search_companies: Call raised at protocol level instead of returning a tool result: HTTP 403 Forbidden (transport level)

#🔴 FAIL - 1.11 Report schema, include_raw

[!CAUTION] Response of 870,747 chars (~248,785 tokens) exceeds typical LLM context windows (100,000 tokens): the tool is unusable by an agent in this mode

{
  "schema": [
    {
      "title": "CompanyReportResponse",
      "$schema": "http://json-schema.org/draft-04/schema#",
      "type": "object",
      "properties": {
        "chargeReference": {
          "type": "string"
        },
        "messages": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "type": {
             ... [+870,347 chars]

#🔴 FAIL - 3.11 Overview by SIREN

[!CAUTION] Server error code: ambiguous - Multiple companies match '443021241'.; Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag; Unexpected error: { "error": { "code": "ambiguous", "message": "Multiple companies match '443021241'.", "hint": "Use the Creditsafe ID, safe number, registration number, VAT, or add postcode/city.", "candidates": [ { "

{
  "error": {
    "code": "ambiguous",
    "message": "Multiple companies match '443021241'.",
    "hint": "Use the Creditsafe ID, safe number, registration number, VAT, or add postcode/city.",
    "candidates": [
      {
        "creditsafe_id": "FR-X-44302124100072",
        "safe_no": "FR22340418",
        "registration_number": "44302124100072",
        "vat_numbers": [
          "FR964430212... [+3,741 chars]

#🔴 FAIL - 3.12 Overview by VAT number

[!CAUTION] Server error code: not_found - No company found for 'FR96443021241'.; Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag; Unexpected error: { "error": { "code": "not_found", "message": "No company found for 'FR96443021241'.", "hint": "Try safe number, registration number, VAT number, or name plus postcode." } }

{
  "error": {
    "code": "not_found",
    "message": "No company found for 'FR96443021241'.",
    "hint": "Try safe number, registration number, VAT number, or name plus postcode."
  }
}

What an LLM client sees when it discovers the tools (descriptions, parameter documentation, metadata).

• get_my_creditsafe_access: No outputSchema: the shape of the result is undocumented
• get_supported_countries: No outputSchema: the shape of the result is undocumented
• get_company_search_fields: No outputSchema: the shape of the result is undocumented
• get_company_search_fields: Parameter 'countries' is a free string without enum/pattern: valid values are only discoverable by trial and error
• get_report_languages: No outputSchema: the shape of the result is undocumented
• get_report_languages: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
• get_company_report_schema: No outputSchema: the shape of the result is undocumented
• get_company_report_schema: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
• search_companies: No outputSchema: the shape of the result is undocumented
• get_company_overview: No outputSchema: the shape of the result is undocumented
• get_company_overview: Parameter 'language' has no description
• get_company_financials: No outputSchema: the shape of the result is undocumented
• get_company_financials: Parameter 'years' has no description
• get_company_financials: Parameter 'language' has no description
• get_company_payment_behaviour: No outputSchema: the shape of the result is undocumented
• get_company_payment_behaviour: Parameter 'language' has no description
• get_company_events: No outputSchema: the shape of the result is undocumented
• get_company_events: Parameter 'language' has no description
• get_company_adverse_information: No outputSchema: the shape of the result is undocumented
• get_company_adverse_information: Parameter 'language' has no description
• get_company_group_structure: No outputSchema: the shape of the result is undocumented
• get_company_group_structure: Parameter 'language' has no description
• get_company_beneficial_owners: No outputSchema: the shape of the result is undocumented
• get_company_beneficial_owners: Parameter 'language' has no description
• get_company_directors: No outputSchema: the shape of the result is undocumented
• get_company_directors: Parameter 'language' has no description
• get_company_directors: Parameter 'limit' has no description
• get_company_directorships: No outputSchema: the shape of the result is undocumented
• get_company_directorships: Parameter 'language' has no description
• get_company_directorships: Parameter 'limit' has no description
• get_company_report_pdf: No outputSchema: the shape of the result is undocumented
• get_company_report_pdf: Parameter 'language' has no description
• search_directors: No outputSchema: the shape of the result is undocumented
• search_directors: Parameter 'countries' is a free string without enum/pattern: valid values are only discoverable by trial and error
• get_director_report: No outputSchema: the shape of the result is undocumented
• get_director_report: Parameter 'language' has no description
• (all tools): Tool definitions alone weigh ~11,122 tokens, paid on every LLM turn

Full requests and responses: verbose.log, raw.jsonl. Spreadsheet: results.csv / results.xlsx.