Carl Lochstampfor, Jr.

🛠️ Tools & Frameworks

🔬 Methodology Highlights

• Two independent LLM agents (attacker + victim) exchange turns iteratively, producing natural resistance and escalation dynamics across 8 elder-fraud categories and 16 prompt templates.
• Parameterized victim personas (age 65–85) with trust, scam-awareness, and tech-savviness traits; attacker knowledge tiers (cold-call 50% / partial 30% / full 20%).
• Initially prototyped with commercial LLM APIs (including Anthropic Claude); transitioned to local Qwen 2.5 14B inference via Ollama on NVIDIA RTX 4080 Super (16 GB VRAM) after safety guardrails blocked scam role-play scenarios — enabling unrestricted generation, zero API cost, and full data privacy.
• Built a negation-aware v2 label-audit pipeline after discovering a 49.8% initial mismatch, relabeling 1,594 conversations for outcome-label integrity.

📊 Notable Results

• Benchmarked 8 models (6 classical ML + 2 transformers); XGBoost + TF-IDF led at 72.5% accuracy / 0.691 macro-F1.
• TF-IDF lexical features beat 28 hand-engineered features by 16 points — quantifying how much discriminative signal lives in the raw text.
• Isolated input truncation and limited data scale as the transformer bottleneck — a hypothesis directly tested (and confirmed) in COVA-X.

🏗️ Pipeline Architecture

End-to-end pipeline: prompts → local LLM → multi-agent dialogue → quality checks → labeled dataset

🖼️ Pipeline in Action

Generating 10 bank conversations via Qwen 2.5 14B (8m 50s)

Sample output: skeptical victim resists social engineering

❮ ❯

Live pipeline screenshots: generation → sample dialogue

👤 My Contributions

Lead author. I designed and implemented the multi-agent generation pipeline that produces realistic attacker-victim dialogue exchanges. I engineered the 28-feature extraction system and TF-IDF representation, built the v2 label-audit tooling that corrected 1,594 mislabeled conversations, and trained and evaluated all eight baseline classifiers.

🛠️ Tools & Frameworks

🔬 Methodology Highlights

• Scaled the corpus 3.4× to 10,985 conversations (88,852 turns) with five distinct victim/attacker profiles per scam type for behavioral diversity.
• Engineered a three-role generation architecture for virtual-kidnapping scenarios, cutting artifact flag rates from 67.1% to 46.5%.
• Built a multi-stage quality lifecycle — contamination scanning, stage-direction stripping, and automated relabeling — driving a 12.7× label-consistency gain (49.8% → 3.9% correction rate).
• Characterized Qwen 2.5 14B capability limits under sustained emotional pressure, including a salience-bias effect where enumerating forbidden patterns increased their production.

📊 Notable Results

• Longformer overtook XGBoost on every metric (79.71% accuracy / 0.779 macro-F1 vs. 78.43% / 0.756), directly confirming the first paper's data-scale hypothesis.
• Transformers gained the most from scale (Longformer +11.2 macro-F1 points vs. XGBoost +6.5), with the biggest edge on the minority "complied" class.
• Key challenge identified: distinguishing partial compliance from rejection — victims who engage extensively before refusing present similar lexical patterns to those who partially comply, making this the dominant error boundary across all architectures.
• A pre/post-cleanup sensitivity analysis showed all three architectures improved — evidence the pipeline recovered genuine label-relevant signal, not architecture-specific noise.

🖼️ Scaling to 10K+

Quality scan: 94.1% clean rate, artifact taxonomy breakdown

Final dataset: 10,995 conversations across 8 fraud categories

❮ ❯

Scaling & QA screenshots: quality scan → final dataset

👤 My Contributions

Lead author. I scaled the generation pipeline across dual-GPU workstations (RTX 4080 Super + RTX 5060 Ti), designed the three-role virtual-kidnapping architecture that reduced artifact rates by 20 percentage points, and built the complete multi-stage quality-lifecycle tooling — including contamination scanning, stage-direction stripping, and automated relabeling. I then retrained and evaluated all classifiers to validate our data-scale hypothesis.