Automated synthesis workflows fail. This is not a pessimistic framing; it is an operational reality that any closed-loop discovery system has to be designed around from the start, not retrofitted to handle after the first bad run. Temperature overshoot, precursor contamination, phase separation during cooling, furnace sensor drift, weighing errors in the robotic dispenser: each of these produces a failed or ambiguous synthesis result. The question is what the system does next, and that question has more design surface than it first appears.
We have learned that the single most expensive mistake in closed-loop synthesis automation is not the synthesis failure itself. It is the silent retry: the system sees a failure, requeues the job with the same parameters, and runs it again without routing the failure information anywhere useful. You burn synthesis slots, you delay the feedback loop, and you learn nothing from the failure. Explicit failure routing, where every failure class has a defined downstream action, is what separates a self-correcting loop from an expensive random walk.
Classifying Failure Before Routing It
The first step is distinguishing failure types, because the right response to each is different. We use three broad categories:
Hardware faults
The synthesis hardware malfunctioned independently of the chemistry of the run. Temperature controller malfunction, robot arm calibration error, furnace power interruption, precursor vial empty when the system expected a full one. These failures provide no chemical information. The correct response is to halt the specific run, log the fault with the hardware telemetry, and requeue the synthesis job after the hardware issue is resolved. Retrying before resolution produces more failures of the same type.
Hardware faults require a separate notification pathway from chemistry failures. The platform needs to know whether a failure is a hardware issue (requires human intervention before retry) or a chemistry issue (informs the model). Conflating them corrupts the feedback signal.
Chemistry failures
The hardware ran correctly but the target phase did not form, or formed as a minority phase alongside other products. Phase separation, unintended secondary phases, amorphous product where a crystalline phase was expected, or volatile precursor loss during high-temperature processing all fall here. These failures are chemically informative: they tell you something about the synthesizability of that specific candidate under those specific conditions.
A chemistry failure is not a reason to discard the candidate. It is a reason to update the synthesis condition parameters (temperature ramp rate, dwell time, atmosphere, precursor stoichiometry) and potentially to update the feasibility score in the model for that composition class. Some chemistry failures reveal that the candidate is genuinely hard to synthesize as a pure phase; others reveal that the default synthesis conditions were not appropriate for this composition.
Characterization failures
The synthesis hardware ran correctly and a product formed, but the characterization output (XRD pattern, EDS map, or other measurement) is ambiguous or uninterpretable. This is different from a chemistry failure: the synthesis may have produced the target phase, but you cannot confirm it from the data you have. The correct response is human expert review before you update the model with the result. Auto-updating the model with uncertain characterization data will introduce label noise into the feedback loop.
Routing Logic in Practice
Building explicit routing means that every synthesis run outcome has a defined path. The paths we use are roughly:
- Hardware fault detected: pause synthesis queue, alert operator, do not requeue until hardware status is confirmed, log fault code and timestamp.
- Chemistry failure, condition-related: update synthesis condition parameters for this composition class, requeue with revised conditions if within retry budget, log failure with failure class label and original conditions.
- Chemistry failure, candidate-related: demote candidate in the queue, add negative synthesis evidence to the feedback dataset, update synthesizability score for this composition type.
- Characterization ambiguous: flag for human review, hold model update pending review decision, log ambiguity class.
- Successful synthesis, confirmed phase: promote candidate result, update model with positive synthesis evidence, log characterization data and conditions.
The routing logic requires that failure classification happen before routing, which means the system needs a failure classification layer between the raw synthesis output and the downstream decision logic. That classification can be automated for hardware faults (they usually produce hardware-specific error codes) and partially automated for chemistry failures (XRD phase identification is automatable for common phase libraries). Characterization ambiguity is harder to classify automatically and usually benefits from a human-in-the-loop step.
Retry Budget and Loop Hygiene
The retry budget is a parameter that tends to get set by default rather than by design, and the default is usually too high. A retry budget of three means that a chemistry failure on a given candidate will produce three synthesis runs before the system gives up and deprioritizes the candidate. If your synthesis throughput is limited (as it is for most small-scale automated platforms), a high retry budget means that failed candidates consume a disproportionate share of your synthesis capacity.
We recommend setting the retry budget based on your synthesis throughput and the value of the information returned by each retry. For condition-related failures where you have a specific hypothesis about what went wrong, a retry with revised conditions is usually worth the slot cost. For failures where you have no clear diagnostic (the phase simply did not form under any reasonable conditions you have tried), retrying consumes slots without generating useful model feedback. A retry budget of one for condition-adjusted retries and zero for undiagnosed chemistry failures is a reasonable starting point for most platforms.
Queue contamination
A related problem is queue contamination: failed candidates that keep cycling through the queue at low priority, consuming occasional synthesis slots without ever producing useful results, because the system never makes a definitive decision to retire them. Explicit retirement criteria, such as "retire candidate after two chemistry failures with no condition-change path" or "retire candidate if synthesis feasibility score drops below threshold after negative evidence update," prevent queue contamination. A synthesis queue that never retires anything will eventually be dominated by hard-to-synthesize candidates that scored well on thermodynamic stability but are practically unobtainable.
Feedback Quality and Model Updates
The whole point of a closed-loop system is that synthesis results improve the model that generates the next screening cycle. This only works if the feedback data is labeled correctly. A mislabeled synthesis outcome, such as a successful synthesis that was labeled as a failure due to characterization ambiguity, or a hardware-fault retry that was counted as a chemistry failure, degrades model quality in exactly the chemistry space you care most about.
We treat synthesis outcomes as noisy labels and apply a confidence weighting before any model update. Confirmed successful synthesis (clean XRD, phase-pure product, repeatable) gets full weight. Synthesis with minor secondary phases gets partial weight proportional to the estimated purity. Chemistry failures get a signed negative weight. Characterization-ambiguous results get zero weight until human review resolves the label.
This approach means the model updates more slowly on uncertain evidence, which is the right behavior for a system where each data point is expensive to generate. It is better to accumulate five clean data points than to update aggressively on fifteen noisy ones.
What We Have Found Hardest to Automate
The parts of error handling that resist full automation in our experience are: distinguishing condition-related from candidate-related chemistry failures when the failure mode is phase separation during cooling (both conditions and composition contribute), classifying XRD patterns for novel phases that are not in the reference library, and deciding when a candidate has enough negative evidence to retire rather than continue exploring conditions.
We are not claiming these are unsolvable, only that they are the current friction points where human judgment improves outcomes more than automation. We have built the routing logic, the retry budget, and the feedback labeling system to minimize the manual intervention load. The remaining manual steps are specifically the ones where we believe automated decision quality is not yet good enough to justify removing the human check. As the system accumulates more synthesis data, the classification models that support those steps will improve, and the manual intervention surface will shrink. That is the trajectory we expect, not a promise that the system is already fully autonomous.
If you are building or evaluating a closed-loop synthesis workflow and want to compare notes on error handling architecture, the contact page is the right starting point.