Show HN: Fine-tuned 110M encoder beat 7B LLMs and hybrid search for NIST mapping

An open-source machine learning framework for mapping technical software vulnerabilities (CVE descriptions) to formal cybersecurity compliance controls (NIST SP 800-53 Rev 5 and CMMC).

Developed by James Pusateri (Middle Coast Software Inc.) and published under the Applied Inference Lab initiative.

Correlating technical vulnerability scans with regulatory compliance frameworks is a persistent operational bottleneck in cybersecurity compliance programs. Technical vulnerability descriptions (such as memory corruption, buffer overflows, or authentication bypasses) inhabit a different vocabulary space than administrative compliance controls (such as flaw remediation, boundary protection, or audit logging).

This repository provides the complete model generation, contrastive fine-tuning, evaluation, and GGUF quantization pipeline developed for AutoRMF. Middle Coast Software Inc. has elected to open source this architecture as a contribution to the cybersecurity and machine learning communities.

Evaluating standard approaches against an isolated 2,000-sample golden validation dataset revealed critical insights that contradict common enterprise search assumptions:

- Zero-shot foundation models fail: Off-the-shelf embedding models (such as BGE-Large and GTE-Qwen2-7B) achieve less than 1.0% strict top-1 accuracy out of the box.

- Domain contrastive fine-tuning succeeds: Fine-tuning a compact 110-million parameter encoder (BGE-Base) using Multiple Negatives Ranking Loss (MNRL) increases strict top-1 retrieval to 77.45% and parent control roll-up top-3 retrieval to 87.15%.

- Generative LLM reranking degrades accuracy: Using a 7-billion parameter generative model (Qwen2.5-7B-Instruct) downstream of dense retrieval reduced top-1 accuracy from 77.45% to 72.00%, as the general-purpose model over-indexed on secondary operational cues.

- Lexical hybrid search causes rank collapse: Fusing sparse BM25 Okapi keyword matching with dense embeddings using Reciprocal Rank Fusion (RRF) caused strict top-1 accuracy to plummet from 77.45% to 13.65% due to lexical rank poisoning across mismatched vocabularies.

All evaluations were conducted against the full NIST SP 800-53 Rev 5 catalog (1,196 controls) using an isolated 2,000-sample validation dataset.

├── data/

│ ├── autormf.db # Standalone SQLite database containing control catalogs and mappings

│ ├── golden_validation_set.json # Isolated 2,000-item evaluation benchmark

│ ├── nist_seed.json # NIST SP 800-53 Rev 5 control catalog

│ ├── cwe_seed.json # Common Weakness Enumeration to NIST mappings

│ └── sp800-53r5-control-catalog.csv # Reference control catalog export

├── docs/

│ ├── 10_static_mapping_and_ml_pipeline_design.md

│ ├── 12_model_conversion_guide.md

│ ├── 13_model_retraining_and_overfitting_safeguards.md

│ └── 18_model_training_eval_and_cve_api_summary.md

├── models/

│ └── nist_control_embeddings_cache.json

├── scripts/

│ ├── init_db.py # SQLite database generator from catalog seeds

│ ├── train_embeddings.py # BGE-Base contrastive fine-tuning with MNRL

│ ├── train_qwen_embeddings.py # High-capacity GTE-Qwen2-7B LoRA adapter fine-tuning

│ ├── evaluate_local_model.py # Local evaluation suite for SentenceTransformers

│ ├── evaluate_mappings.py # Comparative evaluation harness for HTTP endpoints

│ ├── test_reranker.py # Local Cross-Encoder and remote LLM reranking testbed

│ ├── test_hybrid_search.py # Dense + BM25 Reciprocal Rank Fusion testbed

│ ├── offline_rag_mapper.py # Batch FAISS vector retrieval script

│ ├── generate_cve_mappings.py # Delta-aware mapping generation pipeline

│ ├── merge_lora.py # LoRA adapter weight merger for GGUF export

│ └── run_eval.ps1 # Automated evaluation runner

├── requirements.txt

├── LICENSE

└── README.md

All scripts require Python 3.10 or later and should be executed within a virtual environment.

On Windows (PowerShell):

python -m venv .venv

.\.venv\Scripts\Activate.ps1On Linux or macOS:

python3 -m venv .venv

source .venv/bin/activatepip install -r requirements.txtIf data/autormf.db is present, it is ready for immediate use. To initialize or verify the schema from catalog seed files:

python scripts/init_db.pyTo evaluate the fine-tuned model against the 2,000-sample golden validation dataset:

python scripts/evaluate_local_model.py --model-path models/bge-base-en-v1.5-fine-tunedTo run the automated PowerShell evaluation runner:

.\scripts\run_eval.ps1To reproduce the reranking experiment using a local Cross-Encoder:

python scripts/test_reranker.py localTo test with a remote OpenAI-compatible server hosting an instruction model (such as Qwen2.5-7B-Instruct):

python scripts/test_reranker.py remote --lm-studio-url http://localhost:1234/v1/chat/completions --llm-model-name qwen2.5-7b-instructTo reproduce the hybrid search experiment demonstrating lexical rank dilution:

python scripts/test_hybrid_search.py --rrf-k 60python scripts/train_embeddings.py --output-dir models/bge-base-en-v1.5-fine-tuned --epochs 3 --batch-size 8The script automatically isolates all queries present in data/golden_validation_set.json to prevent evaluation data leakage.

For high-capacity environments (such as machines with unified memory or high-memory GPUs):

python scripts/train_qwen_embeddings.py --output-dir models/gte-qwen2-7b-adjudicated-lora --epochs 3 --batch-size 32To merge LoRA weights back into the base architecture for export:

python scripts/merge_lora.py --adapter-path models/gte-qwen2-7b-adjudicated-lora --output-path models/gte-qwen2-7b-mergedTo convert fine-tuned Hugging Face weights to GGUF format for low-memory, in-process inference via llama.cpp or LLamaSharp:

- Clone llama.cpp:

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

pip install -r requirements-convert-hf-to-gguf.txt- Convert weights to GGUF:

python convert_hf_to_gguf.py ../models/bge-base-en-v1.5-fine-tuned/ --outfile ../models/bge-base-en-v1.5-fine-tuned.gguf- Quantize to Q8_0 precision:

./llama-quantize ../models/bge-base-en-v1.5-fine-tuned.gguf ../models/bge-base-en-v1.5-fine-tuned-q8_0.gguf Q8_0For comprehensive conversion details, consult docs/12_model_conversion_guide.md.

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.