Approach to the architecture: what has emerged to date
A general-purpose LLM is good at producing plausible financial analysis. The harder problem is producing analysis that is consistently grounded, testable and sufficiently reliable to become working material for a professional analyst or finance team.
The design therefore starts from the premise that the LLM should be treated as a bounded inference component within a wider system, rather than as the system itself. Making this an explicit architectural principle has come through trial and error, as have many of the design choices throughout this process.
The architecture controls where probabilistic inference is permitted, what evidence an LLM can operate on, how wide its search space is, and what validation or evaluation takes place before an output is allowed to propagate downstream.
This is also why the system is deliberately decomposed. Tasks that can be performed reliably in code are kept outside the LLM. These include activities such as extracting numeric financial data and management statements, performing calculations and comparisons across reporting periods, and detecting predefined financial patterns. Where interpretation is required, the LLM receives tightly constrained inputs and a narrowly defined task. Where more open reasoning is required, as in the final synthesis stage, the model is given greater inferential freedom, but within a predefined analytical structure. These boundaries have largely emerged through trial and error.
Human-in-the-loop intervention is introduced when particular classes of error are detected and for targeted annotation tasks. Annotations are used by optimisation compilers to improve subsequent model performance.
The degree of inference is not uniform across the pipeline. It increases only where the analytical task requires it, with different controls applied at each stage.
Reasoning, architecture and data structures
The system can be viewed as divided into three forms of reasoning: fact extraction, bounded interpretation and abductive reasoning. Each represents a different stage in the pipeline and permits a different degree of inference.
Fact extraction
Fact extraction takes place in the data preparation and financial telemetry stages. This work is executed through deterministic programming.
Numeric data and natural language management statements are extracted and placed into well-defined, typed data structures with associated metadata. Arithmetic, chronology and financial pattern detection are performed before an LLM is involved. Deterministic validations are used to check the resulting data.
The LLM therefore does not need to discover the underlying financial movements or calculate them. It receives structured financial facts and associated narrative that have already been prepared upstream.
This separation also creates explicit data contracts between stages. Each component has a defined set of inputs and outputs, making the state of the analysis more explicit and testable.
Bounded interpretation
Bounded interpretation takes place in the first stages of the analysis engine.
A single financial pattern that has been triggered in the telemetry stage is passed to the LLM along with the numeric data supporting that pattern, the relevant management narrative, and a small set of predefined hypotheses associated with the pattern.
The LLM is not being asked to decide what financial signals matter or to generate an unrestricted set of explanations. Its task is narrower: determine which of the predefined hypotheses are supported by the available financial data and management statements, which are not supported, and which remain unresolved.
This takes place separately for each triggered pattern, usually resulting in 15–20+ bounded LLM calls rather than one broad analytical request.
Subsequent stages classify and organise the resulting findings and then write the prose used in the report. The prose stage cannot introduce additional hypotheses. Its role is to express the structured finding clearly and in grounded language.
Deterministic validators and LLM-based evaluations sit around these inference stages so that invalid outputs, unsupported statements and other errors can be identified before they become inputs to later stages.
Abductive reasoning
Abductive reasoning occurs at the final analytical stage of the pipeline, building on the preceding fact-extraction and bounded-interpretation stages.
At this point, the system begins synthesising the triggered patterns into a smaller number of higher-order tensions. This requires a different kind of reasoning. The LLM is being asked to infer plausible relationships between observed findings and identify what emerges from several findings taken together that could not be read from any one finding in isolation.
This is the least deterministic part of the pipeline and therefore the stage most vulnerable to misguided associations, accounting errors and outright hallucinations.
The answer is not to remove inference, because inference is the purpose of this stage. Instead, the search space is constrained before the model is asked to reason.
The system uses predefined neighbourhoods: groups of financial patterns whose potential relationship has been defined as analytically plausible. Rather than allowing the model to form associations freely across all triggered patterns, a neighbourhood typically represents four to six related findings within a predefined analytical theme.
The neighbourhood therefore acts as a constraint on abductive reasoning while still allowing the LLM to make associations that were not explicitly present in any individual finding.
The synthesis itself is decomposed into three stages. The first produces a concise tension narrative and collates the supporting evidence. The second identifies competing explanations that could account for that tension and states what evidence would distinguish between them. The third uses the committed outputs of the previous stages to generate a hypothesis-neutral diligence question for management.
Each stage has a distinct job. No single LLM call is asked to identify the tension, explain it, consider competing mechanisms and formulate the diligence question at the same time.
Design principles
The core principles that have emerged from this work combine established software engineering with practices specific to working with LLMs. My experience in building with LLMs has reinforced the importance of sound software engineering principles. Throughout this process, I found myself rediscovering many of them as I wrestled with hype and inconsistent output.
- Separation of concerns. Deterministic processing, interpretation and reasoning, synthesis, and presentation are treated as separate responsibilities. Arithmetic, chronology, validation and other deterministic tasks stay in deterministic code. LLMs are used where inference or synthesis makes sense.
- Bounded inference. LLMs are given tightly defined tasks, constrained inputs and clearly specified outputs. Larger analytical problems are decomposed into smaller inference steps that can be tested and evaluated independently.
- Explicit data contracts. Data passed between stages uses well-defined, typed structures rather than loosely structured text. Each component has a clear contract for what it receives and returns.
- Evaluation-driven development. LLM tasks should have defined criteria for what constitutes acceptable output. Evaluations may be deterministic or use an LLM as a judge, depending on the nature of the task.
- Optimisation is a design activity, not an afterthought. Optimisation is part of designing an inference task. The objective is both to refine the task against measurable criteria and to make it possible to use lower-cost models reliably. Once a task performs well on a lower-cost model, it can be tested against other models, including self-hosted open-weight models where data-sharing constraints make proprietary models unsuitable.
Engineering practices and technology choices
The design principles have also informed the engineering practices and technology choices used to build the system. In several cases, the technology choice helps enforce the principle rather than simply implementing it.
DSPy has had the greatest influence on how the LLM-based inference tasks are designed. Decomposing a larger analytical problem into discrete signatures and modules reinforces separation of concerns and bounded inference, and has encouraged greater clarity about each task's inputs, outputs and evaluation criteria. Evaluation and optimisation have become part of designing an inference task rather than an afterthought. DSPy’s optimisers provide a systematic way to improve these tasks against examples and measurable criteria, including testing whether lower-cost models can perform them reliably.
Decomposing what were originally monolithic queries into smaller inference tasks has allowed me to run many of them on Haiku-class models at roughly one-third the cost, with equal or better results than the original approach. Once a task is decomposed to that scale, it becomes straightforward to test against other lower-cost models as they emerge.
- Prefect has has forced me to think more precisely about what each stage is actually responsible for. This in turn has led to more rigorous evaluations. It has also repeatedly exposed work that had been bundled into LLM stages but could be handled deterministically. As the pipeline has evolved, this has helped separate orchestration from analytical logic while reducing unnecessary reliance on inference.
- BDD specifications is used as a thought exercise around what an LLM task is actually trying to accomplish. Writing real-world scenarios forces me to ask what responsibility belongs within its boundary. Scenarios then help shape the evaluations used to determine whether the LLM is performing that specific responsibility successfully.
- Unit testing has become increasingly important. Test suites provide a regression safety net as I refactor, decompose stages and make changes across the pipeline.
- MLflow is used for observability, tracing model calls, capturing inputs and outputs, and recording evaluation results, retries and model performance. This makes it possible to compare behaviour across models, versions and optimisation runs.
- Deterministic validation and LLM-based evaluation are used around model outputs. Deterministic checks are preferred where correctness can be established programmatically. LLM-based evaluation is used where judgement is required, but bounded by explicit criteria.
- Multi-LLM review is used for code and architectural review. Different models provide independent perspectives on implementation decisions and can surface weaknesses that may not be identified by a single model.
- Token usage is monitored in detail. Input and output tokens for each LLM call are logged and tracked. This cost history becomes an input into optimisation work.
The resulting architecture is vendor-agnostic at the inference layer. Individual tasks can be evaluated and optimised against different models rather than assuming that one model should perform every inference task. The objective is to use the model that provides the best balance of accuracy, reliability, cost and data protection for the work being performed.