Ongoing support
Managed AI services
An artificial intelligence system continues to change after it is implemented. The models it relies on are replaced by their providers, the systems it connects to are updated, and the information it works from goes out of date. Firmcraft's managed services keep each system working as intended through those changes.
Support covers three parts of every deployment: the models it uses, the harness that connects those models to the organization's systems, and the context the system works from.
The models
A deployment depends on one or more models, and the providers of those models change them on their own schedules. Models are retired on published timetables, in some cases with as little as two months’ notice. A newer model can cost more or less to run, respond differently to the same instructions, and perform better on some tasks and worse on others.
Firmcraft keeps a record of the models each deployment depends on and the dates by which each must be replaced. A candidate replacement is tested against the client’s own evaluation cases before any change is made, and the change is completed before a retirement date arrives. Model access is routed through a single point with a budget set for each client, so that a model can be changed in one place and usage can be measured.
Deployments are not tied to a single model provider. Where an open-source model suits the work, it can be run in the client’s own environment.
The harness
The harness is the software around the model. It holds the system’s instructions, its connections to the client’s systems, the permissions and approval steps that govern what it may do, its scheduled tasks, and the records of what it has done. Most of what makes an artificial intelligence system reliable in daily use sits in the harness.
The harness needs attention when a connected system changes how it returns data, when a permission needs adjusting, when a process changes, or when a newer model responds differently to instructions written for the one before it. Firmcraft maintains the harness as software. Each change is made under version control, tested before it is put into use, and can be reversed.
The harness is software built for the client, and the client owns it, under the arrangement described in Adaptable Software.
The context
Context is the information the system works from: its standing instructions, the organization’s documents and records, and the memory it accumulates through use. The quality of a system’s work depends on the quality of that information.
Context deteriorates gradually and is easy to overlook. Documents go out of date, prices and policies change, and accumulated memory collects errors and duplicates that the system then treats as fact. More material is not always better, because accuracy declines when a system is given more than the task requires.
Firmcraft keeps what each system draws on current. Outdated material is removed or replaced, errors in accumulated memory are corrected, and instructions are revised when the way the organization works changes. Client data remains in the client's own environment while this is done, as set out in Data Sovereignty.
Monitoring, evaluation, and cost
Monitoring, evaluation, and cost management apply across all three parts. Each system's output is assessed against an evaluation framework established during implementation, so that a decline in quality is found through measurement rather than through a complaint. Usage costs are tracked for each client against an agreed budget.
Results are reported against the measurements taken before the work began, as described in How We Work.
How support is provided
Support is provided as needs arise. Clients raise a request when something needs attention, and Firmcraft maintains the models, harness, and context of systems already in place, including refinement of existing workflows. New workflows are scoped as separate work.
The person who led the implementation remains responsible for support. Clients have a single point of accountability for the whole of their artificial intelligence capability, rather than a collection of vendors.
Managed services are not a condition of using the system. The client owns the software and its data, and can operate the system with its own staff or another firm.
Questions about managed services
What happens when a model is retired?
Firmcraft tracks the retirement dates of the models each deployment uses. A replacement is tested against the client’s own evaluation cases, and the change is completed before the retirement date.
What is an artificial intelligence harness?
The software around the model: its instructions, its connections to the organization’s systems, the permissions and approval steps that govern it, its scheduled tasks, and the records of what it has done.
Why does an artificial intelligence system become less accurate over time?
Usually because the information it works from has changed or deteriorated, or because a connected system or the model itself has changed. Managed services address each of these.
Is Firmcraft tied to a single model provider?
No. Deployments are designed so that the model can be changed without rebuilding the system, and open-source models are used where they suit the work.
What do managed services include?
Monitoring, evaluation, cost management, and the maintenance of the models, harness, and context of systems already in place, including refinement of existing workflows. New workflows are scoped as separate work.
Can we end managed services and keep the system?
Yes. The client owns the software and its data, and can operate it with its own staff or another firm.
Start a conversation
An initial conversation carries no cost and no obligation.