How PBAP works.
A modular orchestrator that audits 26 published peptide-bioactivity prediction tools, runs the 10 that actually work under a unified output schema, and adds analytical layers — concordance, pathogen/commensal selectivity, hierarchical ranking — that no individual tool ships with.
Two phases, one philosophy.
PBAP operates in two independent modes. Phase 1 is what a user runs on their FASTA — sections 02–06 below describe it in detail. Phase 2 is the offline scientific audit that validates each tool against independent evaluation pools; its output is internal QC, not part of any user run.
User inference
FASTA → consolidated predictions + ranking. What the end user actually runs.
- FASTA batchinganti-OOM, default 100 peptides / batch
- Per-tool execution in sub-processesmicromamba run · 6 isolated tool envs
- Normalisation to the dual schemaclass_norm + score + extra_metrics
- Intra-category agreementconsensus_positive · split · single_tool
- APEX selectivity (pathogen vs commensal)post-processing of the 34 MICs
- Hierarchical rankingstructural_score → holistic_score
- Report generationHTML · MD · CSV · XLSX · JSON
Scientific audit — offline only
Per tool, validate against independent data and quantify its biases. Never invoked by a user run.
- Mining positives per bioactivityUniProt + 9 specialised databases
- Extracting the author's training setfrom each tool's repo
- Leakage analysis (CD-HIT-2D)40 / 60 / 80% identity bands
- Negative generation per toollength-stratified sampling
- Prediction & benchmarkingrecall, precision, MCC on the independent pool
- Taxonomic biasFisher exact · Wilson CI · BH-FDR
- Per-tool QCdistributions, AA composition, score_oor
- Global cross-tool reportJSON · TXT · XLSX
Anatomy of a run.
What happens between Inputs/peptides.fasta and Outputs/peptides_2026-05-08T1530/REPORT.html. Click on any step to inspect it.
10 tools, 7 categories.
Out of 26 published tools (2023–2025), only 10 are runnable end-to-end. Each one lives in its own conda environment — pinned, isolated — and emits into the common schema. Categories carry a polarity: GOOD adds to holistic, BAD subtracts.
Two axes, not mutually exclusive.
Some tools predict active / inactive (binary). Others predict continuous magnitudes — MIC in µM against 34 strains, in APEX's case. Forcing everything to binary discards information; using magnitudes only kills cross-tool comparison. PBAP keeps both axes alive.
Every classifier emits class_norm ∈ {positive, negative, null} with score ∈ [0, 1]. Feeds the cross-tool matrix and the intra-category agreement detection.
Magnitudes with a unit. Materialised as columns <tool>__<metric>__<unit>. Same magnitude + target + unit → same column; any difference → separate column.
Intra-category agreement — what happens when two tools disagree
Profile first, magnitude second.
Two levels so the user can prioritise peptides without a high score in one category masking a structurally weak profile. A peptide with a good profile across many categories always ranks above one with a single high score in few.
APEX — 34 strains, 4 labels, 2 means.
APEX predicts MIC against 34 bacterial strains. Some are pathogens (S. aureus, MRSA, K. pneumoniae). Others are gut-microbiome commensals (Akkermansia, Bacteroides). PBAP post-processes this into a selectivity label that feeds directly into the holistic_score.
not in user runs
Everything from here down is offline tooling.
The sections above describe what runs when a user invokes scripts/run_audit.py on their FASTA. The section below describes a separate orchestrator — bin/audit_pipeline.sh — which validates each tool against an independently constructed evaluation pool. Its artifacts live under Dataset_Bioactividad/Reports/ and are never emitted by a user run.
Scientific validation — Gold, Silver, Bronze, Red.
When PBAP audits a tool against an independent evaluation pool (Phase 2 — bin/audit_pipeline.sh), cd-hit-2d bins each evaluation peptide by its maximum identity to the tool's training set. These are identity bands, not a single confidence ladder — they map onto the Applicability Domain concept from QSAR (Tropsha & Golbraikh; OECD Principle 3). The right reading depends on what you are asking.
How good is the model at generalizing?
- Goldbest signaluncontaminated test of true generalization
- Silvercleaninside the AD, minimal leakage bias
- Bronzemildly inflatedclose-neighbor recognition leaks into the score
- Redexcludenear-memorization by interpolation; any decent model wins here trivially
Will this tool be right on this peptide?
- Goldlowpeptide is out-of-distribution; the model extrapolates
- Silverhighinside the applicability domain
- Bronzehighinterpolation from known territory
- Redhigh but trivialpractically correct because the model has seen a near-clone
Practical takeaway: for benchmarking a tool, Gold is the cleanest signal and Red is excluded. For estimating how much to trust a specific prediction the tool just made on a specific peptide, the order is roughly reversed — Bronze and Silver are the operational sweet spot, Gold is where the model is least trustworthy. Phase-2 audit reporting therefore publishes per-band metrics side by side rather than a single trust verdict.
What ends up in Outputs/.
Five formats, same data. The user picks the tool — the browser for rich visual inspection, Excel for hand-filtering, CSV/JSON to plug into other pipelines.