AP Cybersecurity Unit 1 Lesson 5 Exercise 2
Exercise 2: AI Defense Failure Analysis
Harder applied questions: AI error classification, tool limitations, baseline contamination, and zero-day detection gaps. AP-level reasoning for Topic 1.5.
Each scenario describes an AI security tool in use. Identify the specific failure mode, classify it as false positive or false negative, and determine what human oversight step would have prevented it.
An AI email filter automatically deletes all emails scoring above 0.90. A hospital receives a critical patient lab result from a reference lab with an unusual formatting that causes the email to score 0.93. The email is silently deleted. The physician waiting for the result is not notified. The patient’s treatment is delayed by 18 hours.
Error type:
Human oversight step that prevents this:
Vantex deploys an AI-powered SOAR platform that automatically isolates any workstation showing anomalous file access patterns. During a quarterly business review, the finance team runs a large data extraction for the annual audit — a legitimate but unusual process. The SOAR platform isolates 14 finance workstations during the extraction. The annual audit preparation is delayed by two days.
Error type:
Human oversight step that prevents this:
Scenario 3: An AI code vulnerability scanner flags a function as a SQL injection vulnerability. The developer, trusting the AI completely, immediately rewrites the function using parameterized queries. The change breaks a critical reporting feature that relied on dynamic SQL generation for legitimate internal queries. Which failure occurred, and what should the developer have done differently?
You are a security consultant advising Maple Street Veterinary Clinic. They have just deployed three AI security tools. The clinic administrator asks you to explain what each tool does, what it cannot do, and what they still need to do themselves.
Tool A: AI phishing email filter (NLP-based, scores all inbound email 0.00-1.00, auto-blocks above 0.88)
Tool B: Anomaly-based user behavior monitoring (learns baselines over 30 days, alerts on deviations)
Tool C: AI code vulnerability scanner (reviews all code changes before deployment)
1. The administrator says: 'Now that we have Tool A, our staff doesn't need to worry about phishing anymore.' Is this correct? What should you tell her? (2 pts)
2. Tool B has been running for 5 days and has generated 847 alerts so far. Is this normal? What is likely happening? (2 pts)
3. Tool C flagged 3 vulnerabilities in the clinic's patient portal code. The developer fixed all 3 immediately and pushed the update to production. What important step was skipped? (2 pts)
Apply the complete AI defense framework to these harder scenario analysis questions.
Scenario 1: A hospital’s security team deploys an AI model for anomaly detection. After 3 months, they notice the model is generating very few alerts despite known security incidents occurring. Investigation reveals the model was trained during a period when the hospital was under a cyber attack — so the ‘normal’ baseline includes attack activity as part of normal behavior. What is this problem called and how is it fixed? (2 pts)
Scenario 2: Which statement MOST accurately describes why a Zero-Day exploit cannot be caught by a signature-based system on day one, but CAN potentially be caught by an anomaly-based system? (2 pts)
Scenario 3: A CISO says: 'Our AI security tools are fully automated and our security team can focus on other things.' Identify TWO specific risks this attitude creates. (2 pts)
The hardest AP exam questions about AI defense combine two concepts in one scenario: typically (1) why a specific AI tool has a blind spot, and (2) what human control compensates for it. If a question describes AI detecting something and a human doing nothing, the question is asking you to identify the missing human oversight step. If a question describes AI missing something, the question is asking you to identify what detection method or human process would catch what the AI missed.
Get in Touch
Whether you're a student, parent, or teacher — I'd love to hear from you.
Just want free AP CS resources?
Enter your email below and check the subscribe box — no message needed. Students get daily practice questions and study tips. Teachers get curriculum resources and teaching strategies.
Message Sent!
Thanks for reaching out. I'll get back to you within 24 hours.
Prefer email? Reach me directly at [email protected]