The ISTQB Certified Tester Testing with Generative AI (CT-GenAI) v1.0 (CT-GenAI)

Embeddings are a fundamental concept in modern Natural Language Processing (NLP) and LLMs. They are high-dimensional numerical vectors—essentially lists of numbers—that represent the meaning (semantics) of a piece of text (a word, sentence, or document). Unlike traditional keyword matching, which looks for identical strings of characters, embeddings allow the model to understand the "closeness" of concepts. For example, in a vector space, the word "bug" would be mathematically closer to "defect" or "error" than to "feature" or "requirement." This captures the semantic relationship between terms. This technology is the backbone of Retrieval-Augmented Generation (RAG) used in testing: when a tester queries a documentation set, the system converts the query into an embedding and looks for other chunks of text with similar vector values. This allows the AI to retrieve relevant context even if the exact keywords do not match. It is not about logical rules (Option C) or groups of tokens (Option A), but rather a mathematical representation of language that enables machines to process human meaning.

A Retrieval-Augmented Generation (RAG) workflow is designed to "ground" an LLM's output in specific, verifiable data. The logical flow begins with an initial input or "focused query" (Step iv) that defines the tester's goal—in this case, generating cases for a new payments rule. The system then uses this query to perform a semantic search in a vector database (Step i) to find the most relevant "chunks" of the new API specification. Following this, the system retrieves complementary data from the relational database (Step iii), such as historical test cases that might provide structural patterns or regression context. Finally, all the retrieved information—the new specs and the historical context—is bundled together and "fed" into the LLM as part of an augmented prompt (Step ii). This ensures the LLM doesn't hallucinate rules but instead synthesizes the new requirements with established organizational testing standards. Following the order in Option B ensures that the model is provided with the most relevant and logically organized context prior to generating the final testware.

Tokenization is the foundational process by which an LLM breaks down raw text into smaller, manageable units called "tokens." These tokens can represent individual words, parts of words (sub-words), or even punctuation marks. This is a critical step because LLMs do not "read" words like humans do; they process numerical representations of these tokens. The way text is tokenized directly impacts the model's efficiency and its ability to understand complex technical terminology used in software testing. For example, a rare technical term might be broken into several sub-word tokens. This process is closely linked to the Context Window (Option C), which is the maximum number of tokens a model can "remember" or process at one time. While Embeddings (Option B) are the numerical vectors that represent the meaning of these tokens, and the Transformer (Option A) is the underlying architecture that processes them, tokenization is the specific mechanism for initial text decomposition. Understanding tokenization is vital for testers when managing long requirement documents to ensure they do not exceed the model's limits.

The primary differentiator between an LLM-powered test infrastructure and a traditional chatbot is the move from "deterministic" to "probabilistic" logic. Traditional chatbots (Option D) rely on "if-then" logic, decision trees, and predefined scripts. They can only respond to queries that match specific keywords or patterns mapped in their database. In contrast, an LLM-powered infrastructure utilizes the generative capabilities of Large Language Models to synthesize and create new content based on context. This allows it to dynamically generate test insights (Option A)—such as predicting potential regression risks based on unstructured code diffs or drafting test cases for a brand-new feature described in natural language. While traditional bots provide fixed, scripted responses (Option B), LLMs can "reason" through multi-step testing problems and provide nuanced explanations. This contextual awareness is powered by the model's training on vast amounts of technical documentation, enabling it to assist in exploratory testing and complex analysis that traditional, rule-based systems simply cannot handle.

Hallucinations—where an LLM generates factually incorrect or nonsensical information—occur primarily when the model lacks sufficient specific information and "fills in the gaps" using probabilistic patterns from its training data. The most effective mitigation strategy is "grounding," which involves providing the model with detailed, project-specific context. By including technical specifications, existing API schemas, business rules, and identified constraints within the prompt, the tester restricts the model’s operational space to the "project realities." This ensures the model does not have to guess or improvise details about the System Under Test (SUT). In contrast, randomizing prompts (Option B) or relying on generic examples (Option C) increases the likelihood of inconsistent and inaccurate outputs. Furthermore, using "longer" or higher temperature settings (Option D) actually encourages creativity and randomness, which is the opposite of the precision required for testing and significantly increases the risk of hallucinations. Therefore, rich contextual grounding is the technical foundation for reliable AI-assisted test analysis.

This scenario describes a specialized form of Data Exfiltration (specifically targeting the model's internal "weights" or training memory). While data exfiltration usually refers to stealing data from a database, in the context of LLMs, it can also refer to techniques that force the model to "reveal" sensitive information it was trained on or data that exists within its current context window. By using long, repetitive, or specifically "crafted" prompts to overwhelm the model's normal attention mechanisms or safety filters, an attacker may cause the model to output verbatim snippets of proprietary information, PII, or internal documentation that should have remained confidential. This is different from Request Manipulation (Option D), which aims to change the model's behavior, or Data Poisoning (Option A), which happens during training. In testing, this risk is high when models are fine-tuned on private company repositories. Testers must be aware that if a model is accessible to unauthorized users, those users might use adversarial prompting techniques to extract sensitive code or business logic through these types of data leakage attacks.

Prompt Chaining is a technique where a complex task is decomposed into several smaller, sequential steps, where the output of one step serves as the context or input for the next. This is far more reliable than a "one-shot" approach (Option A) because it reduces the cognitive load on the LLM and allows for intermediate verification. In the scenario of test analysis, the most logical and effective chain begins by extracting discrete test conditions from the raw requirements. Once these conditions are established, the next "link" in the chain is to prioritize them based on risk (impact and likelihood), which requires the model to reason specifically about the importance of each condition. The final step is to map these prioritized conditions back to the original requirements to identify any "coverage gaps." This systematic flow (Option B) mirrors the professional test analysis process defined in the ISTQB/CT-GenAI standards. By following this sequence, the tester ensures that the AI-generated output is logically derived and thorough, providing a clear "audit trail" from the initial requirement to the final prioritized test suite.

Classical Machine Learning (which includes algorithms like Random Forests, Support Vector Machines, and Linear Regression) is characterized by its reliance on Feature Engineering . This is the process where human experts manually select, extract, and transform raw data into a set of "features" or variables that the algorithm can process. For instance, in a classical ML model predicting software defects, a tester might have to manually define features like "lines of code changed" or "number of previous bugs." In contrast, Deep Learning and its subset, Generative AI (Options B and D), utilize "Representation Learning." This means the multi-layered neural networks automatically identify and extract the relevant features from raw, often unstructured data (like text or images) without explicit human instruction. Symbolic AI (Option A) is based on hard-coded logical rules rather than data-driven learning. Understanding this distinction is fundamental for testers, as it determines the level of data preparation required: Classical ML requires high human effort in data structuring, while GenAI requires high effort in prompt engineering and grounding.

According to the strategic frameworks for AI adoption (as detailed in the CT-GenAI and related ISO/IEC 42001 standards), the journey toward organizational AI maturity follows three primary phases. The Discovery phase involves identifying potential use cases, assessing current technical readiness, and understanding the legal/risk landscape. The Initiation and Usage Definition phase is where the organization sets the "ground rules"—defining which tools are approved, establishing system prompts, creating prompt libraries, and training the staff on prompt engineering. This phase transitions the AI from a novelty into a structured capability. Finally, the Utilization and Iteration phase is the ongoing process where GenAI is used in daily testing activities, and its outputs are constantly monitored, measured, and improved through feedback loops. This ensures the strategy remains dynamic and adapts to new model capabilities or changing project requirements. Options B, C, and D represent standard project management or IT lifecycles but do not capture the specific "learning and refinement" nature required for successful Generative AI integration in a testing department.