Today we cover the AI model lifecycle from a security management perspective. Your role: oversee the lifecycle to ensure security is embedded at every stage, not bolted on at the end.
The first lifecycle decision — where the model comes from — has significant security implications:
Build (in-house development) — Maximum control over model design, training data, and security properties. Highest cost and expertise requirement. Best for differentiated, high-risk AI applications. Risk: may lack the expertise to build safely.
Buy (commercial vendor) — Faster deployment. Vendor provides support and updates. Less control over model internals. Risk: vendor lock-in, limited transparency, dependency on vendor security practices.
Open-source (community models) — Low cost. Full transparency into model architecture and code. No vendor support guarantees. Risk: supply chain integrity, unknown training data provenance, maintenance responsibility falls on you.
Security considerations by approach:
- Build: Requires secure development practices, training environment security, and internal ML security expertise.
- Buy: Requires vendor risk assessment, contract provisions for security, and ongoing vendor monitoring.
- Open-source: Requires provenance verification, supply chain security assessment, and internal capability to maintain and secure the model.
The choice should be driven by risk appetite, use case criticality, available expertise, and total cost of ownership — not just initial cost.
Model training is a vulnerable phase that requires specific security controls:
Data integrity — Ensure training data hasn't been tampered with. Data poisoning — deliberately introducing malicious data into the training set — can cause models to learn incorrect or malicious behavior that's extremely difficult to detect.
Environment isolation — Training environments should be isolated from production. Access to training infrastructure should be limited to authorized personnel. GPU clusters are high-value targets.
Access controls — Who can modify training data? Who can initiate training runs? Who can access training outputs (model weights, metrics, logs)? Apply least privilege.
Poisoning prevention — Validate data sources, implement data quality checks, monitor for anomalous data entries, and test models against known poisoning attacks. Data from untrusted sources should be treated with suspicion.
Reproducibility — Maintain records of training data, hyperparameters, random seeds, and environment configurations. This enables auditing and troubleshooting.
Training security is often overlooked because it happens "before production." But a compromised training process produces a compromised production model.
Before a model reaches production, validation must confirm it meets security, performance, and governance requirements:
Performance validation — Does the model meet accuracy, precision, recall, and other performance metrics defined in the requirements? Test against held-out data that was not used in training.
Fairness validation — Does the model perform equitably across demographic groups? Test for disparate impact using defined fairness metrics (demographic parity, equalized odds, etc.).
Robustness validation — How does the model perform under adversarial conditions? Adversarial testing evaluates whether the model can be fooled or manipulated.
Security testing — Model-specific security testing: extraction attacks, membership inference, prompt injection (for LLMs), and data leakage testing.
Compliance validation — Does the model meet regulatory requirements? For high-risk AI: documentation, transparency, human oversight provisions.
Go/no-go criteria — Define clear, measurable criteria before training begins. The decision to deploy should be based on predefined criteria, not post-hoc rationalization.
As security manager, your role is to ensure the validation process exists, is followed, and includes security and fairness criteria — not to perform the validation yourself.
Transfer learning — building on pre-trained models — is the dominant approach in modern AI. It introduces inherited risks:
Base model risks — Any vulnerability, bias, or quality issue in the base model carries through to your fine-tuned version. You inherit risks you didn't create and may not fully understand.
Training data inheritance — The base model was trained on data you didn't select or approve. This data may contain biases, copyrighted material, toxic content, or personally identifiable information.
Behavioral inheritance — Pre-trained models have behaviors that persist through fine-tuning. A model with a tendency toward certain biases may retain those tendencies even after fine-tuning on balanced data.
Mitigation strategies:
- Choose base models from reputable sources with published model cards
- Evaluate base model performance on your specific use case and demographic groups
- Document the base model provenance as part of your model lineage records
- Monitor for inherited behaviors that fine-tuning didn't correct