Improving the Reliability of AI Code Generation through Domain Knowledge

Introduction

Coding agents often struggle to generate accurate code in specific domains where they lack deep domain expertise. One example relevant to our customers is semiconductors. Without any domain knowledge, we expect LLMs to perform poorly on semiconductor coding tasks that require knowledge of STDF record structure and field semantics (e.g., which fields encode lot, wafer, and part identifiers), semiconductor-specific test and yield conventions (e.g., how yield, bin-failure rate, and WAT parameters are conventionally defined and computed), and domain terminology and data hierarchies (e.g., how lots, wafers, and parts relate to one another in a test flow).

At Emergence we've devoted significant effort to formalizing semiconductor domain knowledge from large public corpora. As one example, in the semiconductor manufacturing process, test information is stored in an industry-standard format known as the Standard Test Data Format (STDF for short). The STDF specification defines the format, including the file format and record types (such as the PRR, which lists the lot ID, wafer ID, part ID, and bins for each test instance), the semantics and valid values of individual fields, and the conventions governing how test results are recorded and organized across a manufacturing run. Further, individual companies have additional internal conventions and usage patterns, so a code generation system needs to capture this organization-specific domain knowledge in order to properly interface with their data. Formally representations of such knowledge, including in natural language rules, ontologies, and other formalizations, have the potential to improve the accuracy of coding agents on semiconductor code-generation tasks.

In this blog, we describe a set of experiments that aim to explore how much domain knowledge improves the accuracy of code generation, and comparing different representations of such knowledge. To understand these subjects more deeply, we created a semiconductor-focused coding benchmark and measured the performance of coding agents on this benchmark.

Experiment Setup

To measure how much domain knowledge actually improves LLM code generation on semiconductor test data and which representations of that knowledge are worth curating, we built a benchmark of 20 intents, natural-language descriptions of semiconductor coding tasks. These intents are split into 13 single standalone analyses (e.g., computing each lot's yield as a percentage and rank the lots from lowest to highest") which we call basic intents, and 7 sequential pipelines where each step feeds the next (e.g., relating each wafer's bin-failure rate to its WAT parameters as a Spearman correlation grid) which we call deep intents. We know that LLMs struggle with long, chained processes where a single mistake can carry forward and compound over future steps. Thus, deep intents serve as a crucial test of the LLM's ability to complete such complex, multi-step tasks.

Each intent ships with a reference Python implementation and synthetic raw STDF data. By running the reference Python implementation with the raw data as input, we produced ground-truth JSON files for each intent. Moreover, to ensure that any discrepancies between ground truth and generated JSON files were due to meaningful differences in content rather than formatting issues, we curated format-only sample JSON files that tell the coding agent the required output shape without revealing the actual numbers.

The basic flow of the experiment goes as follows. First, we set up a Docker sandbox where the raw data and domain knowledge documents, if applicable, are mounted. The coding agent is called from inside of the Docker sandbox, and it receives the intent and format file via the prompt. For the coding agent, we used Claude Code (CLI) with Opus 4.8 as the model. After the coding agent generates the Python implementation and accompanying JSON output produced by running the Python implementation with raw data as input, it moves all run artifacts to an external directory before processing the next intent. This protocol ensures that each intent receives a fresh start without results or accrued memory from the previous intent biasing the current intent's outputs. Then, this process repeats for each of the 20 intents.

Finally, the generated Python implementations and JSON outputs reach a deterministic evaluation script and error analysis agent to score the run and determine what the root causes of any discrepancies were. The deterministic evaluation script calculates the normalized score, which is a normalized, field-level metric: each intent's fraction of exactly-matching output fields, averaged so that every intent counts equally regardless of field count. This metric captures partial credit without letting field-heavy intents dominate the score. The error analysis agent calls Claude Code (CLI) with Opus 4.8 as the model to categorize discrepancies into common failure modes and has autonomy to declare new failure modes if a discrepancy does not belong to any existing failure modes. This final step allows us to compare the performance of different types of domain knowledge with each other and better understand how error modes differed across experimental conditions.

We ran this experiment across four conditions: LLM, LLM+spec, LLM+spec-derived NL rules, and LLM+NL rules. Note that "LLM" here refers to an LLM-based coding agent rather than a bare language model.

Experiment 1, the LLM branch, tested the innate LLM with no external domain knowledge. This is the baseline condition.

Experiment 2 tested the LLM with access to the STDF Specification, the industry-standard document defining the STDF file format and its record types. This is also a baseline condition in the sense that the spec is a pre-existing document requiring no additional curation on our part.

Experiment 3 tested the LLM with access to a set of 180 atomic natural language rules describing semiconductor data semantics that we curated. These natural language rules include rules from the spec as well as non-spec rules discovered during previous experimentation. An example of such a rule is "A die's identity is exactly the tuple (LOT_ID, WAFER_ID, X_COORD, Y_COORD)."

Experiment 4 tested the LLM with access to a subset of the 180 atomic natural language rules containing only the rules that were derived purely from the spec. If there is a delta between the performance of the LLM+NL rules and LLM+spec experiments, this experiment can provide insight into whether the delta can be attributed to the difference in knowledge representation or the additional knowledge that the NL rules contains.

We should note that the divide between spec-derived and non-spec-derived rules is blurry. Some rules are spec-adjacent but not purely derived from the spec, and some rules might even conflict with the spec depending on organizational preferences and conventions. For this experiment, we selected only the rules that were deemed as spec-derived rules by 2 independent coding agents. Future experiments may explore the effects of imposing less of a strict condition during spec-derived rule selection or including spec-adjacent rules.

For the code generation step, we constructed a deliberately minimal prompt template held constant across every arm. The only sentence that varied was an instruction to read a given knowledge artifact at a given path before starting. Everything else — task description, format example, save-file instructions — stayed fixed, so any performance delta could be attributed to the knowledge representation rather than prompt drift.

Two design choices mattered more than expected. First, early runs surfaced ambiguous prompts (rate as percent vs. decimal, hard bin vs. soft bin, Pearson vs. Spearman correlation) that the LLM silently resolved by guessing, so we manually disambiguated of 13 of 20 prompts before the final run. This is an inherent problem with coding benchmarks. Queries often contain small but critical ambiguities, which results in multiple answers being plausibly correct. Disambiguation with a human in the loop is an essential feature for any coding agent, which is why Emergence is currently developing agents to thoughtfully and efficiently disambiguate use cases with customers. Second, Claude Code repeatedly tried to read the reference implementation and ground truth directly, even from a different directory, so we isolated each run inside a Docker sandbox that mounts only the raw data and the relevant knowledge document, with intent and format details passed exclusively through the prompt. After each run, we moved all run artifacts out of the Docker container to give the next intent a fresh start.

Results

We evaluated all four conditions on both the basic and deep intent tiers, evaluating each with our normalized score metric and averaging 3 trials per condition.

On the basic tier, the ranking was clear. The innate LLM performed worst, as expected, scoring 0.76 with no domain knowledge at all. NL rules and SpecRules were effectively tied for first, both scoring ~0.9. This finding suggests that the subset of NL rules not derived from the spec added little value for this simpler tier of tasks. That NL rules nonetheless outperformed the raw STDF spec arm suggests the gap isn't really about how much knowledge the agent has access to, but about the form that knowledge takes — atomic, rule-shaped natural language appears easier for the LLM to consume and apply correctly than the same information embedded in a lengthy specification document.

The deep tier sharpened this picture. NL rules again performed best (0.95) and the innate LLM again performed worst (0.51) — a much larger gap than on the basic tier, consistent with our expectation that domain knowledge matters more as task complexity increases. In contrast with the basic intents tier, the ordering across knowledge representations was clearer: NL rules > SpecRules > STDF spec. Because SpecRules is by construction a strict subset of NL rules, this ordering tells us the extra, non-spec rules, many drawn from our previous bug-identification experiments, carry valuable information for complex, multi-step intents, even though they didn't affect performance on basic intents. The advantage of the NL rules arm over the raw spec appears to be jointly explained by (1) the more digestible form of the knowledge and (2) additional content the spec simply doesn't contain. We should flag one caveat: the deep tier contains only 7 intents, and the resulting standard deviations are large, highlighting the need to grow the benchmark. Relatedly, NL rules is already saturating this benchmark dataset even on the deep tier, so if we want a meaningful comparison against formal verification in the future, we'll need harder intents.

Key Takeaways

• NL rules outperforms the raw spec on the existing ground truth. Across basic and deep intents alike, our curated set of atomic natural-language rules beat the STDF spec.
• The most common bugs trace back to physical die identity and deduplication, and NL rules is the one arm that largely avoids them. Every other condition repeatedly misidentified die identity (using PART_ID instead of the (LOT_ID, WAFER_ID, coordinates) triple) or mishandled retested dies on PRR. NL rules sidesteps this because it states the rule explicitly. This is a clear example of a critical piece of domain knowledge that LLMs don't know but we need to capture in our domain knowledge-guided code generation pipelines.
• NL rules is already close to saturating this benchmark, even on the deep tier. That's good news for the value of curated rules, but it means we can't yet use this dataset to meaningfully compare domain knowledge-augmented code generation against formal verification. We need a larger, harder benchmark before that comparison is fair.
• Ambiguous prompts and organization-specific conventions are a recurring failure source. Between prompt ambiguity (rate as percent vs. decimal, Pearson vs. Spearman) and organization-dependent rules (how blank HBIN_PF is scored, where wafer-zone boundaries fall), a substantial share of errors were about the system silently guessing instead of asking. This is a direct argument for better elicitation as a first-class component of the pipeline.

This experiment has demonstrated the impact of different types of domain knowledge on LLM performance on semiconductor coding tasks, which has subsequently informed our creation of formalized ontologies and ongoing elicitation agent work. Ongoing and future works include automating the creation of curated domain knowledge from large domain corpora, and expanding the benchmark dataset to more challenging tasks.