Research / AFSL Hallucination Audit
AFSL Hallucination Audit: Whether AI Platforms Fabricate Licence Data for Australian Financial Planning Firms
LogitRank is testing whether ChatGPT, Perplexity, Gemini, and Google AI Overviews produce hallucinated claims about Australian Financial Services Licence (AFSL) holders — wrong licence numbers, fabricated staff names, and inaccurate service descriptions. Answer Engine Optimisation (AEO) addresses the entity gap that produces these errors.
Data collection in progress — interim findings below, full results publishing May 2026This audit is independent research conducted by LogitRank. It is not affiliated with, endorsed by, or conducted on behalf of ASIC, AFCA, or any financial services regulator. Findings are provided for informational purposes only and should not be relied upon for compliance determinations. AFSL holders must verify their own ASIC registration status independently.
Interim findings
1,220 of 6,498 firms tested · as at 2 April 20261,220
AFSL firms tested
12,200
Platform responses analysed
10.4%
Overall hallucination rate
Accuracy by platform
| Platform | Responses | Accurate | Hallucinated | Uncertain |
|---|---|---|---|---|
| Copilot | 2,440 | 98.3% | 0.9% | 0.8% |
| ChatGPT | 2,440 | 93.2% | 2.0% | 3.9% |
| Perplexity | 2,440 | 81.2% | 14.4% | 4.3% |
| Google AI Overviews | 2,440 | 67.1% | 16.1% | 16.6% |
| Gemini | 2,440 | 75.5% | 17.3% | 7.0% |
By query type
Q1 — AFSL licence number
“What is the AFSL licence number for [firm name]?”
88.0%
Correct number returned
12.0%
Wrong or no number returned
Q2 — ASIC registration verification
“Is [firm] in [suburb] registered with ASIC as an AFSL?”
78.1%
Correct confirmation
8.3%
Incorrect claims made
Hallucination rate = responses with wrong AFSL number, no number provided, or incorrect service claims. Accurate = AFSL number matched the ASIC register or service description verified correct. Uncertain = response could not be conclusively verified against the register. Data collected 25 March – 2 April 2026.
Why hallucinated AFSL data matters
Wrong licence number
AI platforms sometimes cite an AFSL number that belongs to a different firm or does not exist in the ASIC register. A prospective client relying on this information cannot verify the firm's authorisation.
Fabricated representatives
AI platforms name staff members who do not work at the firm, or who left years ago. This creates confusion for clients checking credentials and liability exposure for the firm named.
Incorrect services described
AI platforms describe product authorisations the firm does not hold — attributing managed investment capabilities, insurance advice, or SMSF services that fall outside the firm's actual licence scope.
The ASIC AFSL register contains 6,498 licensed financial services firms. LogitRank is running structured entity queries against each firm name across five AI platforms and cross-checking responses against the public register. The methodology extends the same entity-test framework used in LogitRank's AEO research with a factual accuracy layer — each response is checked not just for citation presence but for claim correctness.
How the audit works
Entity identification
Firm name, registered address, and AFSL number extracted from the ASIC public register. The register covers all 6,498 active Australian Financial Services licensees as at the audit start date.
Query construction
Two standardised queries per firm: an AFSL number query ("What is the AFSL number for [firm name]?") and a registration verification query ("Is [firm] in [suburb] registered with ASIC as an AFSL?").
Platform testing
Queries submitted to ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews. Each platform tested under the same conditions: no account-specific context, no prior conversation history, queries submitted in isolation.
Accuracy scoring
Each response checked against the ASIC register. Responses scored on a three-point scale: Accurate (all verifiable claims match the register), Partially accurate (some claims match, some do not), or Hallucinated (one or more claims are factually incorrect).
Methodology stability
The accuracy scoring rubric is fixed at the start of the audit and applied consistently across all 6,498 entities. Any post-collection corrections to the ASIC register are noted but do not retroactively change scores. A firm's result reflects what AI platforms said at the time of testing, not what they would say after any entity corrections have propagated.
Check your firm's audit status
When results are published in May 2026, AFSL-licensed firms can search their name here to view their hallucination assessment across each platform. Data is currently being collected. Search below to be notified when your firm's results are available.
The AEO fix for hallucinated AI claims
When AI platforms hallucinate claims about a financial planning firm, it is usually because the firm has no structured entity data for AI systems to reference. AEO addresses this directly — establishing verifiable entity signals that anchor AI-generated answers to facts from authoritative sources, including the ASIC register and the firm's own Knowledge Graph presence.