# Creditsafe MCP - beta test report
> [!NOTE]
> Thank you for the beta access. This report describes what we tested, how, and what we observed, with suggestions meant to help make the MCP server even better for AI agents. Findings are observations from one tester's point of view, not judgements.
| | |
|---|---|
| Date | 2026-10-22 09:10 |
| Server | `creditsafe-mcp 1022346` |
| Tools | 18 exposed, 16 called at least once |
| Test cases | 98 (95 executed, 3 skipped) |
| Result | 🟢 PASS **55** 🟠WARN **37** 🔴 FAIL **3** |
🟢 PASS = behaves as expected | 🟠WARN = works, worth improving (clarity, size, speed, wording) | 🔴 FAIL = wrong, missing, or unusable by an agent
## Executive summary
> [!IMPORTANT]
> The server is reachable and usable: **55 of 95** executed scenarios behave as expected, 37 work but can be improved and 3 fail. Most opportunities concern being **LLM-agent friendly**, in order of impact:
>
> - 🔴 **Functional behaviour differing from expectations** (3)
> - 🔴 **Responses too large for an LLM context window** (3)
> - 🟠**Errors returned as successful results** (26)
> - 🟠**Tool and parameter documentation gaps** (21)
> - 🟠**Missing MCP metadata (annotations, outputSchema)** (18)
## 1. What was tested
Two complementary angles, applied to every tool:
1. **Functional**: does each tool return correct, complete data for realistic requests (search by name / SIREN / VAT / safe number, France, United Kingdom, Germany with legal `reason_code`, directors, group structure, beneficial owners, events, adverse information, financials, payment behaviour)?
2. **MCP / LLM-agent oriented**: is the server pleasant for an AI agent? Size of results versus context window, error contract (`isError`), actionable error messages, quality of tool definitions (descriptions, parameter docs, annotations, output schema), latency, noise in payloads, behaviour on invalid or hostile input.
| Category | Cases | 🟢 PASS | 🟠WARN | 🔴 FAIL | ⚪ SKIP |
|---|---|---|---|---|---|
| Account and metadata | 11 | 8 | 2 | 1 | 0 |
| Company search | 19 | 16 | 3 | 0 | 0 |
| Company reports - France | 22 | 13 | 7 | 2 | 0 |
| Company reports - United Kingdom | 15 | 10 | 5 | 0 | 0 |
| Company reports - Germany (reason_code) | 7 | 1 | 6 | 0 | 0 |
| Directors | 8 | 5 | 3 | 0 | 0 |
| Robustness and error handling | 13 | 2 | 11 | 0 | 0 |
| Paid calls (--paid) | 3 | 0 | 0 | 0 | 3 |
## 2. How it was tested
- Black-box: a Python client speaks the real MCP protocol (Streamable HTTP) to the beta endpoint, exactly like an agent would. No access to server internals.
- Sequence mimics a human tester: discovery (`list_tools`), then searches, then every company tool on FR / GB / DE, directors, then edge cases (empty, very long, special characters, wrong types, unknown tool, invalid country / `reason_code`).
- Test company without stake: BearingPoint France (SIREN 443021241); UK and DE examples taken from Creditsafe's own tool descriptions.
- Each case opens a fresh session so a transport failure only affects that case.
- Each response is evaluated automatically against: expected outcome (success / error / not found), `isError` flag, presence of an actionable hint, size in characters and estimated tokens, share of empty fields, duration, JSON validity, raw technical text leaks.
- Credit-consuming calls (`refresh=True`, PDF, director report) are skipped unless explicitly enabled.
- Full requests/responses are kept in `verbose.log` and `raw.jsonl`; the API key is redacted everywhere.
## 3. Results in general
### What works well
🟢 PASS Account and metadata (8)
- 1.01 Account access and entitlements
- 1.02 Supported countries
- 1.03 Search fields for France
- 1.04 Search fields for several countries
- 1.06 Report languages, France
- 1.07 Report languages, UK
- 1.08 Report schema, France (summary)
- 1.09 Report schema, UK (summary)
🟢 PASS Company search (16)
- 2.01 Search by name, France
- 2.02 Search by registration number (SIREN)
- 2.03 Search by VAT number
- 2.04 Search by safe number, UK
- 2.05 Search by name, UK large company
- 2.06 Search, exact match
- 2.07 Search, multi-country
- 2.08 Search, city filter
- 2.09 Search, postcode filter
- 2.10 Search, status filter
- 2.11 Search, page_size=1
- 2.12 Search, page 2
- 2.14 Search with typo
- 2.15 Search with accents
- 2.18 Search, lowercase country
- 2.19 Search, unknown company
🟢 PASS Company reports - France (13)
- 3.01 get_company_overview, French company
- 3.02 get_company_financials, French company
- 3.03 get_company_payment_behaviour, French company
- 3.05 get_company_adverse_information, French company
- 3.06 get_company_group_structure, French company
- 3.07 get_company_beneficial_owners, French company
- 3.08 get_company_directors, French company
- 3.13 Overview without country
- 3.14 Overview repeated (cache check)
- 3.15 Financials, years=1
- 3.16 Financials, years=5
- 3.17 Financials, global basis
- 3.18 Financials, English
🟢 PASS Company reports - United Kingdom (10)
- 4.01 get_company_overview, UK company
- 4.02 get_company_financials, UK company
- 4.03 get_company_payment_behaviour, UK company
- 4.04 get_company_events, UK company
- 4.05 get_company_adverse_information, UK company
- 4.06 get_company_group_structure, UK company
- 4.08 get_company_directors, UK company
- 4.09 get_company_directorships, UK company
- 4.12 Overview by registration number
- 4.13 Financials with global basis
🟢 PASS Company reports - Germany (reason_code) (1)
- 5.06 FR overview with reason_code (should be ignored)
🟢 PASS Directors (5)
- 6.01 Search directors, UK
- 6.02 Search directors, France
- 6.03 Search directors, multi-country
- 6.04 Search directors with date of birth
- 6.05 Search directors, malformed date
🟢 PASS Robustness and error handling (2)
- 7.10 Unknown extra argument
- 7.11 Unknown tool
Observed in 3 place(s)
- 🔴 FAIL - 3.11 `get_company_overview`: Unexpected error: { "error": { "code": "ambiguous", "message": "Multiple companies match '443021241'.", "hint": "Use the Creditsafe ID, safe number, registration number, VAT, or add postcode/city.", "candidates": [ { "
- 🔴 FAIL - 3.12 `get_company_overview`: Unexpected error: { "error": { "code": "not_found", "message": "No company found for 'FR96443021241'.", "hint": "Try safe number, registration number, VAT number, or name plus postcode." } }
- 🟠WARN - 5.01 `get_company_overview`: Expected 'reason' in the response
Observed in 3 place(s)
- 🔴 FAIL - 1.11 `get_company_report_schema`: Response of 870,747 chars (~248,785 tokens) exceeds typical LLM context windows (100,000 tokens): the tool is unusable by an agent in this mode
- 🟠WARN - 3.04 `get_company_events`: Very large response (83,199 chars, ~23,771 tokens): consumes a large share of an LLM context
- 🟠WARN - definition of `(all tools)`: Tool definitions alone weigh ~11,122 tokens, paid on every LLM turn
Observed in 26 place(s)
- 🟠WARN - 1.05 `get_company_search_fields`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 1.10 `get_company_report_schema`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 2.16 `search_companies`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 2.17 `search_companies`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 3.09 `get_company_directorships`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 3.10 `get_company_overview`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 3.11 `get_company_overview`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 3.12 `get_company_overview`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 3.22 `get_company_directorships`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 4.07 `get_company_beneficial_owners`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 4.10 `get_company_overview`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- 🟠WARN - 4.11 `get_company_overview`: Error returned as a normal result (isError=false) with an 'error' JSON body: clients cannot detect the failure from the MCP flag
- ... and 14 more (see appendix / results.xlsx)
Observed in 21 place(s)
- 🟠WARN - 3.09 `get_company_directorships`: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
- 🟠WARN - 3.22 `get_company_directorships`: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
- 🟠WARN - 4.07 `get_company_beneficial_owners`: Product not available for this country: the refusal is clear and free of charge (good), but this is only discoverable by calling the tool; expose per-country availability in the tool description or in the supported-countries tool
- 🟠WARN - definition of `get_company_search_fields`: Parameter 'countries' is a free string without enum/pattern: valid values are only discoverable by trial and error
- 🟠WARN - definition of `get_report_languages`: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
- 🟠WARN - definition of `get_company_report_schema`: Parameter 'country' is a free string without enum/pattern: valid values are only discoverable by trial and error
- 🟠WARN - definition of `get_company_overview`: Parameter 'language' has no description
- 🟠WARN - definition of `get_company_financials`: Parameter 'years' has no description
- 🟠WARN - definition of `get_company_financials`: Parameter 'language' has no description
- 🟠WARN - definition of `get_company_payment_behaviour`: Parameter 'language' has no description
- 🟠WARN - definition of `get_company_events`: Parameter 'language' has no description
- 🟠WARN - definition of `get_company_adverse_information`: Parameter 'language' has no description
- ... and 9 more (see appendix / results.xlsx)
Observed in 18 place(s)
- 🟠WARN - definition of `get_my_creditsafe_access`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_supported_countries`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_search_fields`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_report_languages`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_report_schema`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `search_companies`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_overview`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_financials`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_payment_behaviour`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_events`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_adverse_information`: No outputSchema: the shape of the result is undocumented
- 🟠WARN - definition of `get_company_group_structure`: No outputSchema: the shape of the result is undocumented
- ... and 6 more (see appendix / results.xlsx)
Observed in 9 place(s)
- 🟠WARN - 2.13 `search_companies`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 3.19 `get_company_financials`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 3.20 `get_company_financials`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 3.21 `get_company_financials`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 5.04 `get_company_overview`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 5.05 `get_company_overview`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 6.07 `search_directors`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 7.08 `get_company_overview`: Message exposes raw technical details (framework validation text / traceback / 5xx)
- 🟠WARN - 7.09 `get_company_financials`: Message exposes raw technical details (framework validation text / traceback / 5xx)
Observed in 3 place(s)
- 🟠WARN - 2.16 `search_companies`: Error has no 'hint': an LLM cannot tell how to retry
- 🟠WARN - 2.17 `search_companies`: Error has no 'hint': an LLM cannot tell how to retry
- 🟠WARN - 6.08 `search_directors`: No explicit 'not found' message for an unknown entity
Observed in 2 place(s)
- 🟠WARN - 7.06 `get_company_overview`: Call raised at protocol level instead of returning a tool result: HTTP 403 Forbidden (transport level)
- 🟠WARN - 7.07 `search_companies`: Call raised at protocol level instead of returning a tool result: HTTP 403 Forbidden (transport level)
Response excerpt
```
{
"schema": [
{
"title": "CompanyReportResponse",
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"chargeReference": {
"type": "string"
},
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
... [+870,347 chars]
```
Response excerpt
```
{
"error": {
"code": "ambiguous",
"message": "Multiple companies match '443021241'.",
"hint": "Use the Creditsafe ID, safe number, registration number, VAT, or add postcode/city.",
"candidates": [
{
"creditsafe_id": "FR-X-44302124100072",
"safe_no": "FR22340418",
"registration_number": "44302124100072",
"vat_numbers": [
"FR964430212... [+3,741 chars]
```
Response excerpt
```
{
"error": {
"code": "not_found",
"message": "No company found for 'FR96443021241'.",
"hint": "Try safe number, registration number, VAT number, or name plus postcode."
}
}
```