An Open GenAI Business Case Analysis Best Practice: A Methodology for Data Program Transformation

• Navigation: About | Audience | Background | Template Toolkit | Case Study | GenAI Facilitator | Download Full Method (PDF)
• Method Phases: Phase 1: Initiation & Scoping | Phase 2: Business Analysis | Phase 3: Technical & Data Analysis | Phase 4: Modernization Blueprint | Phase 5: Governance & Improvement

Phase 3: Technical and Data Analysis (“Assess Readiness”)

Objective

The objective of Phase 3 is to rigorously evaluate the technical and data feasibility of the prioritized GenAI use case. This phase is critical for grounding the “to-be” vision in reality by ensuring that the required data is available and of sufficient quality, the proposed technology is appropriate, and potential risks are identified and mitigated. The outcome of this phase determines whether a viable solution can be built.

Key Activities

1 Conduct a Data Readiness Assessment:

• Inventory and Categorize Data Sources: Identify and inventory all potential data sources required to train and operate the GenAI model. This includes a wide range of types, such as unstructured documents, tabular datasets, knowledge graphs, and raster collections. A Data Reference Model (DRM) should be used to categorize these sources into logical information classes, such as STRATEGIC GUIDANCE, TOPOGRAPHIC, WEATHER, and FIRE ANALYTIC PRODUCTS, as was done in the wildland fire reports.
• Assess Data Quality and Gaps: Evaluate each data source for its quality, availability, accessibility, and potential for bias. This assessment is crucial because the performance of any GenAI model is fundamentally dependent on its training data. This activity must also identify critical data gaps. For example, the analysis for the wildland fire use case identified a gap in national structures data, which is a key input for risk assessment.
• Define Data Governance and Security: Establish clear requirements for data governance, security, and the handling of any personally identifiable information (PII). This includes defining protocols for data provenance and traceability, which are essential for building trust and ensuring the ethical application of AI.

2 Conduct a Technology Assessment:

• Evaluate GenAI Components: A complete GenAI solution often requires augmenting a foundational Large Language Model (LLM) with other technologies to provide domain-specific context and accuracy. The assessment must evaluate a suite of components:
  • Retrieval-Augmented Generation (RAG): To enhance contextual understanding by retrieving relevant information from a specified knowledge base, which is critical for “knowledge-intensive” scenarios.
  • Generative Adversarial Networks (GANs): To generate or enhance synthetic imagery, which is valuable for tasks like damage assessment or predictive modeling from satellite and aerial imagery.
  • AI Agents: To orchestrate real-time API requests and call other specialized models, enabling the integration of live data feeds.
  • Natural Language Processing (NLP): To complement LLMs by performing nuanced analysis and extracting specific information, such as named entities, from unstructured text.
• Determine Architecture and Integration Needs: Assess how the proposed technology stack will integrate with existing enterprise systems addressing Data, Information, Knowledge, and Wisdom Integration. 

• Consider the full data, information, knowledge integration with AI Wisdom capabilities noted in Phase 4.

• Confirm Human-in-the-Loop Requirement: For high-stakes applications like disaster management, it is imperative to design the solution as a “human-in-the-loop” system. This ensures that AI-generated recommendations are reviewed and validated by human experts before action is taken, improving reliability and trust.

3 Perform a Comprehensive Risk Analysis:

• Identify and Categorize Risks: Proactively identify potential risks across multiple domains:
  • Technical Risks: These include model inaccuracies, “hallucinations,” lazy responses, goal drift, and the challenge of keeping models updated with real-time data.
  • Operational Risks: These include a lack of user trust, the need for new workforce skills, and cultural resistance to adopting new tools.
  • Ethical Risks: These include potential model bias derived from training data and a lack of transparency or traceability in AI-generated outputs.
  • Security Risks: These include protecting sensitive data and ensuring the system is resilient against adversarial attacks.
• Develop a Risk Mitigation Plan: For each identified risk, assess its likelihood and potential impact. Develop a clear mitigation strategy for high-priority risks. This plan is an essential component of the final business case.

Inputs for this Phase

• “To-Be” Conceptual Design Document (from Phase 2).
• Organizational data catalogs and system architecture diagrams.
• Enterprise security and data governance policies.

Outputs of this Phase

• Data Readiness Report: A comprehensive report detailing the available data sources, their quality assessment, and a clear analysis of any data gaps that must be addressed.
• Technology Recommendation Document: A document outlining the recommended GenAI technology stack, conceptual architecture, and integration plan.

Risk Assessment Matrix: A formal register of identified risks, their potential impact, and the corresponding mitigation strategies.