Source: docs/book/index.md

Xalec AutoML Services Handbook

This book is the curated reference for Xalec AutoML Services. It combines user guidance, workflow selection, data preparation, model methodology, algebraic definitions, billing behavior, explainability, AI reports, and operational troubleshooting.

The in-app Xalec assistant is also covered in this handbook. It is designed as a controlled handbook and job-artifact reader: answers should be grounded in approved documentation, service metadata, and saved job evidence.

Current App Status

As of May 30, 2026, the service website supports 20 active workflows and more than 250 configured model options. The upload workflow is readiness-gated: users upload data, map required roles, select features and classification positive class where needed, run Preview Data Readiness, and only then see the training configuration, model catalog, tuning, cost, and launch controls.

Completed jobs now use one artifact-backed review spine. The job detail page, AI Report tab, public demo workspace, and handbook all refer to the same saved evidence classes:

• readiness profile;
• service diagnostics;
• explainability status;
• reproducibility manifest;
• methods appendix.

Job detail pages include an embedded AI Report review surface, with a standalone focused reader available for longer review. The AI Report tab also links to the artifact-backed Methods Appendix when the saved job evidence is available.

The hypothetical data generator now writes one folder per service type. It accounts for all structured upload formats accepted by the shared dataset reader and keeps media workflows manifest-based.

The public guided demo workspace is available for prospective users. It replays the real product path without sign-in, uploads, or credit use: upload setup, readiness preview, a completed job page, saved plots, generated-report summary, methods appendix, prediction replay, and billing replay.

Who This Book Is For

• Users who need to upload data, choose a service, run jobs, and interpret results.
• Analysts and researchers who need methodological transparency, equations, model assumptions, metrics, and limitations.
• Developers and administrators who need artifact contracts, operational notes, and traceability guidance.

How To Read This Book

• Start with Product Overview and Getting Started if you are new to the platform.
• Use Choosing A Service when deciding which workflow fits your dataset.
• Use Service Reference for the 20 supported workflows and the current upload-visible model catalog.
• Use Technical Methodology for algebraic definitions, optimization objectives, assumptions, and metrics.
• Use Operations and Appendices for troubleshooting, billing, artifact layout, and source-document traceability.
• Use Publication Notes and Publisher Readiness And Editorial Audit when preparing the handbook for external review, rights review, or publisher consideration.
• Use Assistant And Controlled RAG to understand the simplified in-app assistant, evidence sources, quality gates, and feedback proposal review loop.
• Use Hypothetical Data Generator if you are a developer or administrator regenerating safe synthetic validation datasets for tests and workflow checks.
• Use Publication Style Sheet when editing the handbook for external publication.

Traceability

The handbook is maintained from source Markdown and cross-checked against the active service catalog, service documentation, pricing design, assistant steering documents, and synthetic validation dataset guidance. Generated HTML, PDF, and DOCX files are publication outputs rather than the source of truth.

Detailed service and model documentation remain available for audit and maintenance, while this book provides the coherent reader-facing structure.

Source: docs/book/chapters/01-product-overview.md

Product Overview

Xalec AutoML Services is a web-based automated machine-learning platform for training, comparing, explaining, and reporting models across structured data, longitudinal data, time series, text, media, survival analysis, anomaly detection, clustering, recommendation, ranking, network-analysis, and graphical-model workflows.

What The Platform Does

The platform gives users a guided path from raw data to model outputs. It handles the repetitive operational work around applied machine learning: schema checks, preprocessing, candidate model comparison, metric reporting, plots, saved artifacts, explainability, billing, and AI-assisted reporting.

The platform does not replace statistical judgment or domain review. It creates a consistent workflow so users can see what was trained, why a model was selected, what failed, what was skipped, what it cost, and which outputs were produced.

Product Surfaces

The main product surfaces are:

• Home and services catalog for service discovery.
• Upload pages for service-specific intake, role mapping, readiness preview, and post-readiness training configuration.
• Job workspace for monitoring running and completed jobs.
• Job detail pages for metrics, plots, artifacts, SHAP, embedded AI report review, and report actions.
• Prediction center for reusing trained models where supported.
• Billing and wallet views for reserve, refund, and net-charge history.
• AI report views for manuscript-style summaries and downloadable reports.
• Dockable Xalec assistant and full assistant workspace for handbook-grounded help and job-artifact diagnosis.

Core Workflow

1. Select a service.
2. Upload data and configure required columns, feature roles, and positive class where applicable.
3. Run Preview Data Readiness.
4. Review the comparison strategy, visible model catalog, tuning options, cost, and runtime expectations.
5. Run training.
6. Monitor job progress.
7. Review metrics, plots, artifacts, SHAP/explainability, embedded AI reports, and billing settlement.
8. Download or reuse outputs.

At any point, users can ask the assistant for workflow guidance or job-specific diagnosis. For job-specific claims, the assistant should cite saved job artifacts rather than relying on memory or general handbook text alone.

The 20-Service Scope

The current service set spans:

• three cross-sectional tabular workflows
• three longitudinal workflows
• three time-series workflows
• one text workflow
• three media workflows
• three specialized workflows
• four advanced-system workflows

These services are documented in the Service Reference part of this book and traced back to maintained workflow documentation.

The current upload-visible model registry contains more than 250 configured model options across these 20 workflows. The exact count is maintained in the service catalog and summarized in the upload forms and model registry appendix.

Product Principles

• Clarity: users should know what to do next.
• Confidence: users should see status, cost, and limitations.
• Speed to value: upload-to-insight should be short and guided.
• Trust: model behavior, metrics, warnings, and artifacts should be transparent.
• Consistency: service workflows should feel like one product.

Reader Promise

This book is intentionally practical. For each workflow, it answers four questions:

• What problem is this service meant to solve?
• What data shape does it expect?
• What model families and metrics should users expect?
• Which artifacts, assumptions, and source documents make the behavior auditable?

Related Chapters

• Choosing A Service
• Running Jobs
• Understanding Results
• Service Reference Overview
• Assistant And Controlled RAG

Source: docs/book/chapters/02-reading-paths-by-role.md

Reading Paths By Role

This chapter helps different readers move through the handbook without having to read it front to back. The book is designed to support several audiences at once: first-time product users, applied analysts, researchers writing methods sections, administrators supporting operations, and developers maintaining the platform.

The Fast Rule

If you are not sure where to start:

1. read the product overview
2. choose the workflow-selection chapter that matches your data
3. read the results and reproducibility chapters before trusting outputs

That path works for most readers.

Role 1: First-Time Product User

You likely want to answer:

• What service should I choose?
• What columns or files do I need?
• What outputs will I get?
• What will this likely cost?

Recommended reading order:

1. Product Overview
2. Getting Started
3. Choosing a Service
4. Billing and Credits
5. Understanding Results
6. Workflow Selection Matrix

Use Assistant and Controlled RAG if you want to understand what the in-app assistant can answer and why it may ask for a job ID or saved artifact evidence.

Role 2: Applied Analyst Or Data Scientist

You likely want to answer:

• Which workflow is methodologically right for this dataset?
• How are models compared and selected?
• What artifacts should I verify before presenting results?
• When should I trust SHAP, reports, or fallback outputs?

Recommended reading order:

1. Data Preparation
2. Choosing a Service
3. Metrics and Validation
4. Artifacts and Implementation
5. Reproducibility, Audit Trail, and Regeneration
6. Model Selection and Fallback Rules
7. Worked Case Studies

Use Assistant and Controlled RAG when checking whether an assistant answer is grounded in the right evidence rather than a raw retrieval dump.

Role 3: Researcher Or Manuscript Author

You likely want to answer:

• Which workflow assumptions should be stated in the paper?
• What mathematics and optimization rules sit behind the model families?
• How do I make the report, citations, and downloaded manuscript reproducible?

Recommended reading order:

1. Responsible Use, Scope, and Limitations
2. Explainability and AI Reports
3. Model Methodology Reference
4. Reproducibility, Audit Trail, and Regeneration
5. Per-Model Mathematical Appendix
6. Reproducibility Checklist

Role 4: Product Or Operations Administrator

You likely want to answer:

• What state transitions are trustworthy?
• How do billing reservation and settlement work?
• What should recovery or regeneration mean operationally?
• How do I tell a cosmetic issue from a durable-state issue?

Recommended reading order:

1. Running Jobs
2. Billing and Credits
3. Architecture and Execution Flow
4. Reproducibility, Audit Trail, and Regeneration
5. Troubleshooting
6. Model Selection and Fallback Rules

Role 5: Developer Or Maintainer

You likely want to answer:

• Where do service selection, billing, reporting, and artifact hydration connect?
• Which invariants must not drift?
• Which parts of the UX are backed by durable saved state versus derived views?

Recommended reading order:

1. Architecture and Execution Flow
2. EDA, Figures, and Reporting Reliability
3. Artifacts and Implementation
4. Reproducibility, Audit Trail, and Regeneration
5. Admin and Developer Appendix
6. Model Selection and Fallback Rules
7. Assistant and Controlled RAG

If You Are Reviewing A Real Job

Use this compact path:

1. Choosing a Service
2. Understanding Results
3. Metrics and Validation
4. Reproducibility, Audit Trail, and Regeneration
5. Worked Case Studies
6. Assistant and Controlled RAG

Deeper Reading

• Getting Started for the product user guide.
• Service Reference Overview for the service map.
• Billing and Credits for workflow cost and settlement.
• Architecture and Execution Flow for durable workflow state and recovery.
• Assistant and Controlled RAG for the assistant evidence policy.

Source: docs/book/chapters/03-getting-started.md

Getting Started

This chapter introduces the normal user path through the platform.

Before You Start

Before running a job, make sure you know:

• the dataset structure
• the target or outcome column
• whether rows are independent, repeated by subject, time-indexed, media files, text records, or graph/interaction data
• whether the job needs explainability, an AI report, or only model artifacts
• whether the workflow has a cost estimate that should be reviewed before execution

If the data structure is unclear, start with the Choosing A Service chapter before uploading.

Public Demo Workspace

Prospective users can open the public demo workspace before signing in. The demo uses read-only scenarios backed by completed source jobs with saved AI report artifacts. The same artifact-backed evidence spine shown in real job detail pages is used in the demo:

• readiness profile;
• service diagnostics;
• explainability status;
• reproducibility manifest;
• methods appendix.

A scenario can replay:

• upload setup and role mapping
• a completed job page
• saved plot artifacts from the mapped completed job
• a generated AI-report summary in the job detail AI Report tab
• the artifact-backed methods appendix linked from the AI Report tab
• prediction output review
• billing evidence from the source job

The demo never uploads data, starts training, runs SHAP, calls an LLM provider, executes predictions, reserves credits, settles credits, or exposes raw unauthenticated job result URLs. Use it to understand the workflow, then create an account for real data and wallet-backed compute.

Start Using The Platform

For hosted use, start at the services catalog, choose the workflow that matches your data, and follow the upload wizard. Administrators who run a private deployment should use the deployment guide and operating procedures for the active environment.

Developer startup commands and environment notes are kept in the Admin And Developer Appendix so first-time users can stay focused on the analysis workflow.

First Job

1. Open the services catalog.
2. Choose the service that matches your data structure and target.
3. Upload a supported file.
4. Select the target column and required metadata columns.
5. Select feature columns and the positive class for classification services where applicable.
6. Preview readiness so row counts, feature counts, role mappings, and common data issues are checked before training.
7. After readiness passes, review the first comparison strategy and adjust the visible model catalog if the study needs a broader, narrower, or more targeted comparison.
8. Open advanced tuning only when the study needs service-specific hyperparameters, runtime controls, or method settings.
9. Review the cost estimate.
10. Start training.
11. Watch the job workspace for progress.
12. Review metrics, plots, artifacts, SHAP, embedded AI report outputs, and billing settlement.

Upload And Model Selection Layout

Upload pages are designed to keep the first run guided without hiding the breadth of available methods.

• Required roles and dataset readiness are checked before training starts.
• Training configuration controls are not mounted until Preview Data Readiness succeeds.
• Presets are framed as comparison strategies, not as claims that only one small model set is useful.
• The model catalog remains visible after readiness review so users can explore alternatives across model families.
• Advanced tuning, runtime overrides, and detailed method settings remain available but secondary.
• Cost and expected outputs stay visible before launch.

First Job Success Criteria

A first job is successful when:

• the job reaches a completed or partially completed state
• at least one candidate model trains successfully
• the selected model has metrics that match the workflow objective
• skipped or failed models are explained rather than silently hidden
• artifacts can be opened from the job detail page
• billing shows reservation, settlement, and net charge clearly

First Interpretation Checklist

• Confirm the target column is correct.
• Confirm the train/test split is appropriate.
• Confirm the selected best model matches your goal metric.
• Check warnings for imbalance, missing data, small sample size, and unsupported models.
• Review SHAP/explainability only after verifying the model and data are sensible.
• Treat AI reports as drafts requiring domain review.

Common Beginner Mistakes

• Choosing tabular regression for a count target.
• Choosing tabular services for repeated subject data that needs longitudinal handling.
• Uploading time-series data without a time column.
• Leaving IDs, names, or post-outcome fields in the feature set.
• Trusting accuracy on an imbalanced classification problem.
• Treating an AI report as final scientific text without checking numbers, references, and limitations.

Related Chapters

• Choosing A Service
• Data Preparation
• Running Jobs
• Billing And Credits
• Responsible Use, Scope, And Limitations

Source: docs/book/chapters/04-data-preparation.md

Data Preparation

Good AutoML results depend on data that matches the selected service. This chapter summarizes cross-service preparation rules.

General Requirements

• Upload a tabular file, media manifest, or service-specific data format supported by the selected workflow.
• Use clear column names.
• Avoid leaking future information into predictors.
• Keep target columns separate from features.
• Use consistent units, coding, and missing-value conventions.
• Confirm class labels and positive-class definitions for classification.

For binary classification, choose the positive class deliberately before training. It should usually be the outcome you want to detect, flag, prioritize, or explain, such as churn, fraud, default, disease, failure, response, or another event of interest.

File And Schema Hygiene

Use stable, readable column names before upload. Avoid duplicate column names, hidden spreadsheet formulas, merged spreadsheet cells, mixed date formats, and columns whose meaning is only known from external context. If a column requires explanation, document it before running the job so the AI report context and downstream review are easier to validate.

Supported Upload Formats

Most structured-data workflows use one shared server-side reader. The reader validates the file extension, computes dataset rows and columns on the server, and passes a pandas-style table to the service-specific readiness and training logic.

The statistical-file metadata is retained in dataframe attributes and summarized with saved job metadata where supported. This helps reviewers keep original survey, cohort, and registry labels visible even after the modeling pipeline uses sanitized machine-safe column names.

Image, audio, and video workflows remain separate from statistical dataset uploads. They accept media folders or ZIP bundles plus a tabular label manifest. The manifest can use the same supported tabular formats as other upload forms, including CSV/TSV, Excel, Parquet, JSON, Stata, SPSS, SAS, and R data files when the deployment has the relevant readers installed. The canonical columns are filename and target; common aliases such as path, file_path, image_name, audio_name, video_name, label, class, and tag are also accepted.

Media manifests are ordinary tabular files. The statistical dataset formats above do not replace the manifest requirement for media training. Pickle-based manifests are not accepted.

Upload Readiness And Configuration

The application computes dataset rows and feature counts on the server before cost reservation or training. Users should use the readiness preview to confirm the selected service, required roles, target definition, and feature set before launching a job.

The current upload shell keeps intake and role mapping ahead of training configuration. Users should be able to choose target, time, subject or group columns, feature columns, and classification positive class before clicking Preview Data Readiness. Model selection, tuning controls, cost confirmation, and training launch then appear only after readiness succeeds.

For services with model choices, the upload form should show the first comparison strategy and the visible model catalog after readiness context is available. This preserves the main purpose of offering many models: users can start with a sensible strategy, then broaden, narrow, or target the comparison according to the study aim. Hyperparameters, runtime overrides, and detailed method settings belong in advanced tuning sections rather than in the primary intake path.

Dataset Setup And Supplied Splits

Users do not always arrive with the same data setup. AutoML Services supports three common paths:

Use the split roles this way: the training file or training rows fit preprocessing and models; validation rows check model choices, thresholds, calibration, or tuning when a service supports that step; test rows are final held-out performance evidence. If users upload train and validation files without a test file, the uploaded validation file remains model-checking evidence and an internally held-out portion of the training file becomes the final test split. The internal test creation should preserve the safest available structure: subject/group-aware for repeated entities, time-ordered for temporal workflows, and random or stratified random only as a fallback.

Expected outputs follow the split roles:

For classification-like services, the validation plot set is performance metrics, confusion matrix, ROC curve, and precision-recall curve when those split-specific artifacts are saved. Other services should attach service-specific best-model validation plots, such as actual-versus-predicted and residual diagnostics for regression, temporal/subject-aware plots for time-series or longitudinal workflows, and ranking/recommendation diagnostics for interaction services.

Not every service uses splits in the same way. Predictive services use the split for model evaluation. Media classification reads split roles from the label manifest. Labeled anomaly detection can evaluate on supplied held-out rows, but unlabeled anomaly detection remains a full-data unsupervised fit. Clustering, graphical models, and network analysis need service-specific stability or graph-aware validation rather than a generic row split.

Sampling Weights

If the modeling data are a sample of a larger population, users may provide a non-negative sampling-weight column. Treat weights as study-design information, not as predictors. The pipeline should validate the column, exclude it from features, persist a weight summary, and pass training weights to model families that support sample_weight.

When a model cannot consume weights directly, the job should continue only if that limitation is recorded clearly. Do not assume that every reported metric is weighted just because a weight column was uploaded.

Common Column Types

• Target column: outcome to predict or model.
• Feature columns: predictors.
• Subject column: repeated-measure or longitudinal identifier.
• Time column: time index for longitudinal or time-series workflows.
• Group column: grouping, clustering, panel, or multi-entity context.
• Event/time-to-event columns: survival-analysis inputs.
• User/item/rating columns: recommendation and ranking workflows.
• Split column: optional train/validation/test role indicator.
• Sampling-weight column: optional non-negative weight for sampled data.

Data Quality Checks

• missingness
• near-constant features
• high-cardinality categorical variables
• rare categories
• class imbalance
• duplicate records
• invalid timestamps
• insufficient events
• non-stationary time series
• unsupported media formats

Service-Specific Preparation

Leakage Risks

Avoid predictors that reveal the target directly or indirectly. Examples include post-outcome measurements, manually derived labels, future values in time series, and fields generated after the event of interest.

Minimum Pre-Upload Checklist

• Every selected required column exists and has the intended meaning.
• The target has enough non-missing values.
• Classification classes have enough examples for validation.
• Time and event columns are encoded consistently.
• Media or text files can be opened outside the app.
• IDs are excluded unless the selected service explicitly needs them as grouping keys.
• Sensitive or regulated fields are handled according to the deployment policy.

Related Source Docs

• Getting started
• Choosing a service
• Service reference

Source: docs/book/chapters/05-choosing-a-service.md

Choosing A Service

Use this chapter to choose the correct workflow before uploading data.

Service Selection Guide

Decision Tree

Use this decision tree when the right service is not obvious.

1. If the input is image, audio, video, or text, choose the matching unstructured-data service first.
2. If the outcome is time-to-event with censoring, choose survival.
3. If rows repeat for the same person, facility, customer, device, or entity, ask whether subject-level dependence matters. If yes, choose a longitudinal service.
4. If observations are ordered by time and temporal dependence is the modeling target, choose a time-series service.
5. If there is no target label and the goal is to discover groups, choose clustering.
6. If there is no target label and the goal is to detect unusual records, choose anomaly_detection.
7. If the goal is user-item matching or preference prediction, choose recommendation_system.
8. If the goal is ordered relevance for queries, candidates, or lists, choose ranking_models.
9. If records are connected by nodes and edges, choose network_analysis.
10. If the goal is conditional-dependence, hidden-state, or state-space structure among variables, choose graphical_models.
11. Otherwise, choose a cross-sectional tabular service based on the target type.

Target-Type Rules

Structure-Type Rules

Common Mistakes

• Do not use cross-sectional classification for repeated patient visits if subject-level leakage is likely.
• Do not use time-series regression just because a dataset has a date column; use it when temporal order is part of the signal.
• Do not use regression for count outcomes when zeros and overdispersion are central to the problem.
• Do not use clustering when you already have a validated target label.
• Do not interpret anomaly detection as causal diagnosis; it flags unusual patterns for review.
• Do not use recommendation workflows for generic multiclass classification unless there is a real user-item interaction structure.

Example Scenarios

Classification vs Regression vs Count Regression

• Use classification for discrete classes.
• Use regression for continuous numeric outcomes.
• Use count regression for non-negative integer outcomes, especially when counts are skewed or overdispersed.

Cross-Sectional vs Longitudinal vs Time Series

• Cross-sectional workflows assume independent rows.
• Longitudinal workflows model repeated measures, subject IDs, and within-subject structure.
• Time-series workflows model ordered temporal dependence.

If Two Services Seem Plausible

Prefer the service that preserves the strongest dependency in the data:

• repeated person/entity structure beats ordinary cross-sectional treatment
• time ordering beats shuffled-row treatment when the deployment use case is temporal
• censoring/time-to-event structure beats ordinary binary classification
• user-item interaction structure beats ordinary tabular classification
• graph connectivity beats flattened node tables when edges carry signal
• inferred conditional-dependence structure beats ordinary graph analysis when the edges are not observed but estimated from variables

Related Source Docs

• Service reference overview
• Cross-sectional services
• Longitudinal services
• Time-series services

Source: docs/book/chapters/06-responsible-use-scope-and-limitations.md

Responsible Use, Scope, And Limitations

This chapter explains what the platform is for, what it is not for, and how to use its outputs responsibly. The platform can automate a large amount of modeling, comparison, explainability, and reporting work, but it does not remove the need for human judgment, domain review, or methodological caution.

What The Platform Is Good At

The platform is strongest when the user needs:

• a guided workflow from upload to trained model outputs
• comparison across multiple candidate models
• durable artifacts, plots, metrics, and report-ready summaries
• transparent warnings about data quality, missingness, imbalance, or unsupported models
• explainability and manuscript-style reporting grounded in saved artifacts

In those settings, the platform helps teams move faster while preserving more audit trail than a loose collection of notebooks or ad hoc scripts.

What The Platform Is Not

The platform is not:

• a guarantee that the selected model is causally valid
• a substitute for subject-matter expertise
• a substitute for clinical, legal, or policy review
• proof that a result is deployable just because it scored well on one validation split
• proof that an AI-generated report is publication-ready or regulator-ready

It is best understood as a high-structure applied ML workflow system, not as an automatic truth engine.

Use Outputs As Decision Support, Not As Unquestioned Truth

Model outputs, SHAP values, rankings, survival curves, anomaly flags, and AI reports should all be treated as decision-support artifacts. They may surface useful patterns, but they do not resolve:

• causal interpretation
• external validity
• fairness or bias questions
• regulatory acceptability
• operational fit in the real deployment environment

The strongest use pattern is to combine platform outputs with domain review, independent plausibility checks, and explicit governance for the final decision.

Scope Boundaries By Workflow

Different service families create different kinds of risk.

Cross-Sectional And Longitudinal Prediction

These workflows are good for prediction and structured comparison, but they do not by themselves justify causal claims. Leakage, repeated-subject splitting mistakes, and threshold misinterpretation can make a model look stronger than it really is.

Time Series

Time-series workflows are especially vulnerable to future leakage, horizon confusion, and accidental use of random splits where chronological validation was needed.

Text And Media

Text and media models can appear strong while relying on artifacts of labeling, file formatting, class imbalance, or spurious visual/textual cues. Per-class review matters more than a single overall metric.

Survival

Survival outputs require careful interpretation of censoring, follow-up time, and risk scores. A survival model is not automatically a clinical decision rule.

Anomaly Detection And Clustering

These workflows are especially easy to over-interpret. Clusters are not necessarily meaningful groups, and anomaly scores are not ground truth labels unless independently verified.

Recommendation, Ranking, And Network Analysis

These workflows need extra care around popularity bias, group/query definitions, graph structure validity, and whether the evaluation setup reflects the real product question.

AI Reports Need Human Review

AI-generated reports should be treated as structured drafts, not final scientific or operational documents.

Users should verify:

• that references are relevant to the actual study topic
• that numerical claims match saved metrics and plots
• that figures render both in the app and in downloadable outputs
• that unresolved placeholders, declarations, and metadata fields have been completed
• that the wording does not overstate what the underlying workflow actually established

If the report sounds more certain than the saved evidence supports, the report is wrong, even if it reads well.

Explainability Has Limits

SHAP and related explanations describe model behavior, not causal effect. They can help answer:

• which inputs moved the prediction
• which features mattered globally
• whether the model seems to rely on plausible signals

They do not answer:

• whether changing a feature would change the outcome in the real world
• whether the model is fair across subgroups
• whether the feature importance is stable under correlated predictors or distribution shift

Explainability is useful, but it is not an epistemic shortcut.

Common Misuse Patterns

Users should actively avoid:

• choosing the wrong workflow for the data structure
• treating accuracy as sufficient on imbalanced outcomes
• leaving identifier or post-outcome features in the dataset
• interpreting predictive models as causal evidence
• treating anomaly or clustering outputs as self-validating
• shipping AI report text without checking the saved evidence underneath it

When To Stop And Escalate

Pause and escalate to deeper review when:

• the selected workflow may be wrong for the data shape
• metrics are suspiciously perfect or unstable
• key plots are missing or inconsistent with the result tables
• billing, recovery, or completion state looks contradictory
• the domain is high-stakes enough that false confidence would be costly
• the report context or reference list looks generic, incomplete, or off-topic

This is not a sign of failure. It is part of responsible use.

A Practical Trust Checklist

Before relying on a platform result:

• confirm the workflow matches the data and target
• confirm the validation design is appropriate
• review the best model against alternative candidates
• inspect warnings, skipped models, and missing artifacts
• verify SHAP or explanation outputs only after the model itself is plausible
• verify the AI report against saved metrics, figures, and declarations
• decide explicitly whether the output is suitable for exploration, internal decision support, publication drafting, or production deployment

Deep References

• Getting started
• Metrics and validation
• Understanding results
• Architecture and execution flow
• Explainability and AI reports
• Billing and credits

Source: docs/book/chapters/07-running-jobs.md

Running Jobs

Jobs are the execution unit for training, explainability, reporting, and generated artifacts.

What A Job Contains

A job records the service type, uploaded data reference, selected columns, model options, budget estimate, execution state, model results, plots, artifacts, and add-on actions such as SHAP or AI report generation.

The job ID is the durable handle used by the job workspace, result directory, billing ledger, report routes, and downloadable artifacts.

Artifact-Backed Review Spine

The current job detail page, AI Report tab, public demo workspace, and handbook use the same review vocabulary. A completed or partially completed job should preserve these evidence classes when possible:

Older completed jobs can be backfilled from saved result files without retraining. Backfilling these review artifacts does not change the trained model or wallet ledger; it makes the saved evidence easier to inspect.

For active jobs, model/training_results.json is an intermediate training snapshot, not final completion proof by itself. The job should remain active until final pipeline evidence such as job_results.json or report.json confirms that model visuals, diagnostics, reports, and settlement context have finished writing. This prevents a result page from racing ahead while the last model-comparison or diagnostic plots are still being generated.

Job Lifecycle

1. Created
2. Queued
3. Running
4. Generating artifacts
5. Completed, failed, cancelled, or partially completed

Expected Job Execution Workflow

A normal training workflow should move through these user-visible steps:

1. Choose the service that matches the data structure and study aim.
2. Upload the dataset, media bundle, or manifest required by that service.
3. Map required roles such as target, features, subject ID, time, event, query, user, item, node, or edge columns.
4. Choose whether evaluation comes from an internal split, a split column, or separate train/validation/test files, and map any sampling-weight column.
5. Preview readiness and cost before committing credits.
6. After readiness succeeds, choose the first comparison strategy and adjust model selections if the study needs a broader or narrower comparison.
7. Confirm training and let the platform create a durable job ID.
8. Monitor the job workspace or job detail page for status, progress, logs, billing events, and model-level outcomes.
9. Review the completed result package: model comparison, metrics, confidence intervals, plots, warnings, saved artifacts, billing settlement, and optional add-ons such as SHAP, AI report generation, or prediction.

This workflow is service-aware, but the contract is consistent: upload, readiness, cost review, training, persisted artifacts, interpretation, and optional regeneration.

The full model catalog should remain visible where a service supports model choice. Presets provide the first comparison strategy; they should not hide the broader set of models that a user may want to test. Service-specific hyperparameters and runtime controls are secondary tuning decisions.

What Happens During Training

Training normally includes:

• validating required columns and target values
• preparing preprocessing transforms
• splitting or validating the dataset according to service rules, including user-supplied split columns or train/validation/test files when provided
• creating an internal final test split from uploaded training rows when the user supplies train and validation files but no test file
• applying sampling weights to compatible model families and recording when a selected backend cannot use them directly
• running candidate models
• collecting metrics and failure reasons
• selecting the best model under the service objective
• saving model, metadata, plots, and result summaries
• settling training credits against actual work

For larger services, models may run in sequence or by guarded groups. A skipped model can be a healthy outcome if the runtime knows the model is unavailable, too expensive, or incompatible with the dataset.

For regression and count-regression services, Model Results should surface the model-evaluation plot set when available: actual-versus-predicted, model comparison, feature importance, residual diagnostics, residual distribution or QQ plot as appropriate, and training-time comparison. If only one plot appears, inspect the saved plot inventory and completion state before assuming the service failed to generate diagnostics.

When a validation split is present, the job detail page should include a dedicated Validation Dataset section. That section reports validation performance metrics separately from the main Model Performance table and shows validation-specific best-model visuals when the service saved them. For classification-like services, those visuals are confusion matrix, ROC curve, and precision-recall curve. For other services, the visuals should be the service-specific validation diagnostics rather than borrowed classification plots.

Runtime Signals

The job UI should expose:

• current stage
• percent complete where credible
• latest event
• successful and failed models
• output artifacts
• cost and settlement state
• next recommended action

Live monitoring uses two delivery paths:

• live snapshots and log events for immediate updates
• polling against persisted job state as the fallback and catch-up path

The persisted job manifest is the authority. If the browser reconnects or misses a burst of logs, the monitor should catch up from progress, current_step, state_version, last_heartbeat, and recent live_events.

For long jobs, compute-heavy training is offloaded from the interactive web request path. This keeps polling and live-log traffic responsive during workflows such as survival, longitudinal, time-series, ranking, and media classification.

Some upload forms submit inline and keep the user on the upload page while the job starts. Those forms should show the same live-monitor contract as the job detail page: current stage, status text, recent log events, and a progress bar fed by persisted job state. The live log is a convenience surface; the job manifest remains the authoritative source.

Upload pages should not show training configuration as the primary intake step. Required role mapping and readiness preview come first. Model selection, advanced tuning, cost confirmation, and launch controls are mounted after the readiness preview succeeds so the user is choosing models with the dataset shape and guardrail context already visible.

Completed Versus Partially Completed

Completed means the workflow produced the expected primary outputs. Partially completed usually means the main job produced useful outputs but one or more optional parts failed, such as a candidate model, plot, explainability add-on, or report artifact.

Users should look at the successful model list, failed model list, warnings, and generated artifacts before deciding whether to rerun.

How To Evaluate Successful Job Execution

Do not judge a job only from the final status badge. A successful execution should have enough saved evidence to support review and reuse:

• Terminal state is completed or an explicitly explained partial-completion state.
• At least one candidate model trained successfully, unless the selected workflow is an unsupervised diagnostic whose primary output is not a trained predictor.
• Failed or skipped models have reasons such as dependency unavailable, guardrail skipped, efficiency skipped, incompatible data, or fit failure.
• Best-model selection used a metric that matches the service objective.
• Performance tables include the expected metrics, and confidence intervals are present where the workflow supports them.
• Plots, model artifacts, metadata, and result summaries exist in the job result directory and are reachable from the job detail page.
• Active-job recovery did not promote an intermediate training snapshot to a terminal status before final job_results.json or report.json evidence was written.
• Readiness profile, service diagnostics, explainability status, reproducibility manifest, and methods appendix are available or the UI states which evidence class is missing.
• Live logs do not continue past a terminal state except as read-only replay or add-on activity for SHAP, report generation, or prediction.
• Billing shows reservation, settlement, refund, and net charge consistently with the wallet ledger.
• Optional SHAP, AI report, and prediction actions are evaluated separately from the main training job because they are add-on actions.

For a specific job, inspect the saved job record, training results, plot inventory, logs, and wallet-backed billing ledger before making a claim.

Failure Handling

Model failures do not always mean the whole job failed. Some models may be skipped because dependencies are unavailable, runtime guardrails apply, or model-specific fitting fails. The report should distinguish:

• preflight unavailable
• efficiency-skipped
• failed during training
• completed successfully

If the application server is stopped or reloaded while training is still running, the background task may be cancelled before the service writes final artifacts. The platform should persist a terminal interrupted state rather than leaving the job running forever. User-requested cancellations should become cancelled; shutdown or reload interruptions should become failed with an interruption message and refund handling where billing has already reserved credits.

Reruns And Regeneration

Rerun training when the data, target, service choice, or model options were wrong. Regenerate add-ons when the training output is correct but derived artifacts need refreshing, such as SHAP, AI reports, manuscript references, or report figures.

Regeneration should use the saved job payload and result artifacts, not a generic fallback context.

Related Chapters

• Billing And Credits
• Artifacts And Implementation
• Reproducibility, Audit Trail, And Regeneration
• Runtime Guardrails

Source: docs/book/chapters/08-understanding-results.md

Understanding Results

Completed jobs generate metrics, plots, model artifacts, metadata, and optional reports.

Core Outputs

• best model and model-comparison table
• performance metrics
• train/test counts
• feature list
• diagnostic warnings
• plots
• serialized models
• report.json
• job_results.json
• optional SHAP outputs
• optional AI report artifacts

Reading The Result Page

Start with the selected model and the comparison table, then move outward to diagnostics:

1. Confirm the service and target are correct.
2. Check how many rows, classes, events, or files were used.
3. Review the selected metric and whether it matches the goal.
4. Inspect failed or skipped models so the comparison is not misunderstood.
5. Confirm whether the main Model Performance table is internal, validation, uploaded test, or internally generated test evidence.
6. Review plots and per-class/per-group diagnostics.
7. Open explainability only after the model itself looks plausible.
8. Generate or review the AI report after numbers and figures are stable.

Reading Supplied Splits

dataset_split_contract.json is the result-page source of truth for upload combinations. If validation and test are both present, the main Model Performance table should use test evidence and the Validation Dataset section should show validation evidence separately. If validation is uploaded without a test file, AutoML creates an internal test split from the uploaded training file before fitting; the main table should use that generated test split and the Validation Dataset section should still show the uploaded validation metrics.

Do not treat validation and test as interchangeable. Validation evidence is for model checking, threshold review, or tuning where a service supports it. Test evidence is the final performance estimate. Reports should name the split used for each claim and should state when the test split was generated internally.

For classification-like services, validation visuals should include validation performance metrics, confusion matrix, ROC curve, and precision-recall curve when those files are saved. Regression, count, longitudinal, time-series, survival, ranking, recommendation, anomaly, and network services should use the validation plots that match their own metric family and data structure.

Interpreting Best Model Selection

Best model selection depends on the service and metric. Classification workflows often prioritize PR-AUC [@DavisGoadrich2006] or ROC-AUC [@HanleyMcNeil1982] for imbalanced data. Regression workflows use error and goodness-of-fit metrics. Specialized workflows use service-specific criteria.

The best model should be interpreted with:

• metric confidence intervals
• class balance
• runtime and model complexity
• threshold behavior
• explainability availability
• domain plausibility

The best model is not always the most complex model. A simpler model can be preferred when performance is similar, threshold behavior is more stable, explainability is stronger, or deployment cost is lower.

Common Classification Metrics

• Accuracy
• Precision
• Recall
• F1
• ROC-AUC
• PR-AUC
• specificity
• balanced accuracy
• Youden's J

Binary Cutoff And Threshold Plots

For binary classification services, the platform standardizes the operating cutoff used by plots, reports, and prediction outputs. When a best model exposes probabilities, Xalec computes an optimal cutoff from Youden's J, defined as:

Youden's J = sensitivity + specificity - 1

The saved threshold_performance.png plot shows accuracy, precision, recall, specificity, F1, balanced accuracy, and Youden's J across candidate thresholds. It marks the selected optimal cutoff with a vertical line. The roc_curve.png plot can also mark the same operating point so users can see where the chosen threshold sits on the ROC curve.

Use this cutoff as the model's deployed decision threshold unless there is a documented domain reason to use a different operating point. A different cutoff may be appropriate when false positives and false negatives have asymmetric costs, but that decision should be documented separately from the automated model comparison.

Common Regression Metrics

• MAE
• MSE
• RMSE
• R-squared
• residual plots

Reading Weak Regression Fit

Low or negative R-squared does not automatically make a job useless, but it does mean the model has little demonstrated lift over a simple baseline on the validation split. Interpret it alongside RMSE and MAE on the target scale, the target standard deviation, confidence intervals, and residual plots.

A near-zero R-squared with RMSE close to the target standard deviation usually means the available features are not explaining much held-out variation. Such a job can still be useful as:

• a baseline showing that the current data is weak for prediction
• evidence that more features, better timing variables, or data-quality review are needed
• a diagnostic result for residual patterns, outliers, and feature influence
• a reason to defer deployment until validation improves

It is not a strong deployment candidate unless the absolute error is acceptable for the user's decision context and later validation confirms the model remains useful.

Result Interpretation By Workflow

When To Rerun

Rerun the job when the selected service was wrong, the target was mis-specified, obvious leakage exists, too many candidate models failed for fixable dependency reasons, or metrics do not answer the real user question.

Do not rerun only because one model family failed if the comparison still produced a trustworthy winner and the failure is clearly documented.

Related Source Docs

• Model methodology reference
• Service reference overview
• Metrics and validation

References

Source: docs/book/chapters/09-explainability-and-ai-reports.md

Explainability And AI Reports

The platform supports SHAP-style explainability outputs and LLM-assisted AI reports.

The Xalec assistant is documented separately in 10-assistant-and-controlled-rag.md. AI reports generate report drafts from saved job context, while the assistant answers user questions through controlled retrieval from the handbook, service metadata, and job artifacts.

Explainability Goal

Explainability should help users understand which features influenced model behavior and whether the model is learning plausible patterns. It is not proof of causality, clinical validity, fairness, or deployment readiness.

SHAP And Surrogate Explainability

SHAP explains model predictions by attributing contribution values to features [@LundbergLee2017]. Exact SHAP support varies by model family and runtime dependency. The app may use:

• tree explainers for tree models [@LundbergLee2020]
• kernel or permutation explainers for non-tree models
• precomputed SHAP values when available
• surrogate feature-importance outputs when native SHAP is unavailable [@Ribeiro2016]

For neural models such as mlp, keras_mlp, and keras_cnn, explainability may use model weights or prediction-response permutation fallback [@Fisher2019] when native SHAP cannot run reliably.

For time-series jobs, explainability should be panel-aware. When the fitted forecaster does not expose a native tabular SHAP contract, the user-initiated SHAP action can write a pooled panel surrogate bundle over the saved job artifacts. The metadata should say whether the surrogate explained model prediction response or fell back to observed target response, and should record the series identifier, time column, sampled series, row counts, and the pooled_global panel scope. This is distinct from training-time diagnostics: training may prepare metadata and ordinary plots, but it should not silently run billable SHAP.

SHAP is a user-initiated action from the completed job page or API. Training does not imply that SHAP has already run, and a completed training job can still show no SHAP artifacts until the user launches the explainability action.

SHAP Interpretation Rules

• Confirm the model and metric are acceptable before interpreting SHAP.
• Treat feature contribution as model behavior, not causal effect.
• Check whether SHAP is native, kernel/permutation-based, or surrogate-style.
• For encoded categorical variables, map features back to human-readable terms where possible.
• For correlated predictors, interpret rankings cautiously because attribution can be shared or unstable.

AI Reports

AI reports summarize study context, data, methods, model results, explainability, limitations, and references. They should be treated as structured drafts requiring human review.

AI reports should be grounded in saved artifacts. A report should not invent study context, overstate causal conclusions, cite unrelated topic references, or describe figures that do not render.

AI report generation and regeneration are also user-initiated actions. They use the saved job artifacts available at launch time; they are not automatically generated by successful training.

The normal review surface is the AI Report tab on the job detail page. That tab embeds the same focused-reader report surface used by /llm-report/<job_id>, so users can read the manuscript and executive brief without leaving the job context. The standalone route remains useful for focused reading or sharing a direct report link when access rules allow it.

Artifact-Backed Methods Appendix

Every report should be able to point back to the saved evidence that shaped its methods language. The platform writes a compact methods-evidence artifact from saved job files rather than from free-form prose alone. In generated manuscripts, the main Methods section keeps only a short signpost and the detailed model methodology evidence is rendered as Appendix A so the Methods narrative stays readable. The appendix uses:

• readiness profile;
• model preset and selected/effective model evidence;
• service diagnostics;
• explainability status;
• reproducibility manifest;
• report and training result artifacts.

The AI Report tab links to the methods-evidence appendix when it exists. For older jobs, the job detail route can rebuild the markdown appendix from saved artifacts. This backfill is a review artifact; it does not retrain models or alter billing.

Manuscript-Style Reports

The AI manuscript/report workflow should align:

• title and abstract with the detected study domain
• introduction and discussion with topic-relevant references
• methods with the actual service, target, model comparison, and validation approach
• results with saved metrics, plots, SHAP outputs, and selected model metadata
• references with the resolved reference list displayed in the manuscript
• downloadable HTML with the same figures and edited text shown in the app

Report Trust Checklist

• references match the study topic
• no prompt leakage
• no unresolved placeholders
• numeric claims match saved tables
• figures render in the app and downloaded report
• SHAP section reflects the actual explainability method
• methods appendix agrees with the saved readiness, model-selection, explainability, and reproducibility artifacts
• limitations disclose data and model constraints

Human Review Checklist

• Are all numerical claims traceable to saved result files?
• Does the report avoid implying causality from prediction alone?
• Are model failures or skipped methods disclosed when relevant?
• Are domain-specific references appropriate for the study topic?
• Are author declarations, funding, ethics, data availability, and conflicts complete?
• Does the downloaded report match the in-app preview after edits?

Related Source Docs

• Understanding results
• Metrics and validation
• Billing and credits
• Assistant and controlled RAG
• Artifacts and implementation

References

Source: docs/book/chapters/10-assistant-and-controlled-rag.md

Assistant And Controlled RAG

Xalec includes an in-app assistant for handbook-grounded help and job-artifact diagnosis. The assistant is not a generic chatbot, broad analyst, strategy advisor, or autonomous reasoning agent. The current live assistant is a simple, controlled retrieval-augmented generation system: it retrieves approved handbook chapters, service metadata, or saved job evidence, summarizes only that evidence, cites exact sources, and says when evidence is missing.

The earlier V1 and bounded-agentic assistant architectures have been retired from the live runtime. Historical support modules may remain for release evidence or export utilities, but the canonical runtime export mlagent_website.assistant.AssistantRAG uses the guarded implementation in mlagent_website/assistant/simple_rag_v2.py. There is no V1 runtime fallback.

The assistant is available from the dockable app widget and the full assistant workspace at /assistant.

The readable handbook surface is served from the rendered Quarto book at /book/. Legacy /handbook links remain as a Markdown fallback and compatibility path, but normal product links should open /book/ so users get the complete book navigation, search assets, and generated export styling.

What The Assistant Helps With

The assistant is intended to help users:

• choose the right service for a dataset
• understand required columns, readiness warnings, presets, and model families
• understand dataset setup choices such as split columns, separate train/validation/test files, validation versus test evidence, and sampling weights, including train-plus-validation uploads that generate an internal test split
• interpret metrics, confidence intervals, SHAP outputs, plots, and AI reports
• diagnose a specific job from saved artifacts and logs
• explain runtime guardrails, skipped models, and backend availability only from handbook or job evidence
• understand billing, refunds, charges, and add-on actions from recorded evidence
• navigate prediction workflows and known-subject versus new-subject behavior
• explain where saved artifacts live and what evidence is missing

It should answer in plain language, cite the evidence it used, and avoid dumping raw retrieved chunks or JSON unless the user explicitly asks for implementation detail. It should not analyze unseen datasets, invent product strategy, make generic business/medical/legal/financial recommendations, or present itself as a replacement for a human analyst.

Evidence Sources

The assistant uses a controlled source hierarchy:

1. Saved job artifacts and logs for a specific job question.
2. Running service metadata, model catalog, routes, pricing, and configuration for platform behavior.
3. This handbook for stable product and methodology guidance.

For user-facing handbook answers, the approved source is the handbook Markdown. Navigation files, legacy investigation notes, and internal design notes should not be presented as user-facing evidence unless the question is explicitly about developer or administrator internals.

The assistant should retrieve from handbook source Markdown, not from generated HTML, PDF, or DOCX outputs. Citations may point users to the rendered book for reading, but the evidence boundary remains the source handbook and saved job artifacts.

Job-Specific Answers

For job-specific questions, the assistant should inspect saved evidence such as:

• job manifest
• model training-results record
• final job-results record
• report record
• job metadata record
• dataset_split_contract.json, saved dataset_split_policy, and split_evaluation_metrics
• runtime_guardrails.json for service-family cost estimates, disabled models, and runtime-budget warnings
• reproducibility_manifest.json for dataset fingerprints, split settings, model roster, runtime controls, and artifact inventory
• report_validation.json, prompt_package.json, manuscript_context.json, report_tables.json, and llm_telemetry.json for AI report quality, prompt-snapshot, author-completion, table, and provider evidence
• model/validation_policy.json, model/best_model_evaluation.json, and model/optimal_cutoff.json for validation semantics, best-model held-out predictions, and threshold/cutoff evidence
• per-job logs
• plot inventory and SHAP artifacts
• billing, action, prediction, or report artifacts where available

If an artifact is absent, the assistant should say that the evidence is missing. It should not infer a refund, plot, SHAP completion, model success, prediction output, or report generation unless a saved artifact proves it.

The assistant should not index every service implementation file or pipeline module for ordinary user answers. Those files define behavior, but user-facing job answers should flow through stable saved contracts such as split, guardrail, reproducibility, report-quality, validation-policy, and threshold artifacts. This keeps answers auditable and avoids exposing implementation details as if they were job evidence.

For train/validation/test questions, the assistant should first explain the standard split contract from the handbook and then inspect job artifacts for a specific job. Train plus validation without uploaded test means AutoML creates an internal test split from the uploaded training file before fitting; uploaded validation remains validation evidence and should appear in the Validation Dataset section when validation metrics or split-specific validation plots were saved.

Current Simple RAG Runtime

The assistant follows one controlled loop:

1. classify the question into an intent and question archetype
2. build an evidence plan with allowed source types
3. match the question against AutoML ontology concepts
4. retrieve approved handbook, service metadata, or job-artifact evidence
5. rank sources so substantive handbook chapters and saved artifacts outrank index/navigation chunks
6. synthesize a concise answer with citations
7. attach source-inspection metadata that explains why each cited source was selected
8. format the answer as sanitized HTML with MathJax-compatible notation for equations
9. verify citation validity, source policy, redaction, and unsupported-claim risks
10. return a compact answer-quality audit for review

This is why the assistant may refuse to answer a high-trust claim when the needed evidence is not present. A refusal with the missing evidence named is preferable to a confident unsupported answer.

The live assistant does not run an autonomous tool-planning loop. For high-evidence questions, such as job diagnosis, guardrails, billing, prediction, plots, SHAP, reports, or readiness, it chooses the appropriate approved retriever directly and applies the same evidence, citation, and verification rules. If a job id is absent or the saved artifact is missing, it should say what evidence is missing rather than inferring the answer.

For unsupported questions, the assistant uses a strict scope fallback. It should say that it can only answer from the Xalec AutoML Services Handbook, service metadata, or saved job artifacts, then name the kind of evidence needed. It must not turn a broad advice question into a loosely related handbook summary.

For signed-in users, assistant turns may be saved as durable threads through /api/assistant/threads. The saved thread record stores sanitized question and answer text plus compact source metadata; it does not copy row-level uploaded data into the assistant memory.

Optional Semantic Recall

The assistant can use a semantic recall layer, such as local hash embeddings, Chroma, OpenAI, or Ollama embeddings, to improve handbook retrieval. Semantic search is a recall booster, not an autonomous reasoning engine.

By default, semantic indexing should be limited to handbook content under docs/book/**. Job logs, billing events, saved artifacts, uploaded datasets, and row-level data should not be sent to external embedding providers unless a separate redaction and tenant-safety policy is explicitly implemented.

When semantic indexing is enabled, the vector index carries a freshness manifest. The manifest records the embedding configuration and approved handbook source fingerprints so the quality gate can detect a missing or stale index before deployment.

Retrieval results should expose visible retrieval signals where available. These include semantic score, lexical or metadata match signals, source kind, and source-quality ranking notes. These signals are for traceability and review; they should not be treated as user-facing proof unless the cited source itself supports the claim.

Traceability Matrix

The assistant maintains simple steering and evaluation coverage linking intents to question archetypes, ontology concepts, required source kinds, preferred source paths, and evaluation cases. High-risk intents such as billing, runtime guardrails, job diagnosis, SHAP or plot claims, prediction guidance, dataset split setup, sampling weights, and labeled anomaly evaluation must have explicit evidence contracts and evaluation coverage before the RAG readiness gate can pass.

Regenerate the matrix when intents, ontology concepts, source policies, or evaluation fixtures change:

python scripts/build_assistant_rag_traceability_matrix.py

Quality Dashboard And Feedback Loop

The assistant quality cockpit is available at /assistant/quality. It shows:

• latest answer-quality report status
• coverage-gate status
• RAG readiness status
• vector-index freshness status when semantic retrieval is enabled
• simple-RAG runtime and steering status for high-risk answer contracts
• real-job fixture coverage
• feedback-driven improvement proposals
• proposal gate state

When a user marks an answer as insufficient, the app stores a redacted feedback event and creates a pending improvement proposal. Proposals can be reviewed from the quality cockpit, opened in a detail view, approved, rejected, reset to pending, or exported as staged eval-case candidates.

Approved proposals are exported to reports/assistant_rag_feedback/staged_approved_eval_cases.json. Exporting approved proposals does not automatically change the live ontology, answer policies, source rules, or golden evaluation fixtures. Human review and normal code review are still required.

Evaluation Gates

The assistant should be evaluated on more than retrieval success. The quality harness checks:

• ontology and expected-source coverage
• answer quality and citation validity
• pressure safety for unsupported refund, SHAP, plot, model-success, and job-outcome claims
• real-job fixture behavior from saved folder shapes
• source redaction and cross-user job-access protection
• user-response quality, including clarity, usefulness, actionability, and raw-dump avoidance

The main local gate is:

python scripts/run_assistant_rag_quality_gate.py

The assistant should not be treated as deployment-ready after documentation or retriever changes unless these quality gates pass.

Good Questions To Ask

Useful assistant questions include:

• "Where should I start?"
• "What service should I use for repeated subjects over time?"
• "What does this confidence interval mean?"
• "Why is SHAP unavailable for this job?"
• "What happened in job_...?"
• "What runtime guardrails apply to longitudinal services?"
• "Why was this model skipped by the guardrails?"
• "Was this job refunded?"
• "Why is this plot missing?"
• "Can this trained model support known-subject prediction?"

For job questions, include the job id or ask from the job detail page so the assistant can retrieve the right artifacts.

Related Source Docs

• Reading paths by role
• Understanding results
• Service reference overview
• Artifacts and implementation
• Operations and maintainer guidance

Source: docs/book/chapters/11-billing-and-credits.md

Billing And Credits

Billing uses one simple story across the product:

1. Subscription gives platform access and an included credit allowance.
2. Top-ups buy extra compute when the plan allowance is not enough.
3. Jobs and add-ons consume credits through estimate, reserve, settlement, and refund.

This story applies to training, SHAP, AI reports, large batch prediction, and any future billable compute action.

Simple Pricing Story

Use the same terms in the pricing page, billing workspace, job detail page, assistant answers, and support documentation:

• Subscription access covers the workspace, service access, retention, collaboration or governance features, and the included plan credits for the active billing period.
• Included plan credits refresh according to the billing cycle. They are not cash, do not represent purchased wallet balance, and normally do not roll forward indefinitely.
• Purchased top-up credits are separate wallet credits bought for extra compute. They remain available until used unless account terms or an enterprise contract defines an expiry rule.
• Billable work shows an estimate before launch, reserves the current estimate, then settles to a final net charge or refund after the platform knows what was produced.
• Trial credits, when enabled, are an optional one-time registration grant. They are not a subscription allowance and should not be described as a default paid-plan benefit.

Pricing Options

Xalec uses a hybrid billing model: subscription access plus included credits, top-ups for extra compute, and settled usage for compute-heavy work.

The current plan posture is:

• Starter: for solo pilots, lightweight experimentation, and core tabular or lower-cost workflows.
• Professional: for regular analyst, consultant, or applied-research work that needs broader service access, SHAP, advanced diagnostics, and report generation.
• Team: for shared workspaces with pooled credits, seats, role-based review, admin visibility, and workspace spend caps.
• Enterprise: for institutions needing invoice billing, private deployment, SSO/governance, custom retention, contract-defined usage bands, and configurable approval thresholds.

Plan choice should start from expected workload, collaboration needs, governance requirements, and tolerance for variable compute cost. It should not be chosen only from the service name. A small cross-sectional run can fit a lighter plan, while repeated large longitudinal, media, ranking, recommendation, graph, SHAP, and report-heavy work may need Professional, Team, or Enterprise controls.

Credits And Dollar Equivalents

Credits are the internal unit for billable work. The app should show the estimated credits and dollar equivalent before the user commits to a charge. Current self-serve top-ups are documented as starting at $0.10 per credit, while subscription plans include product access and bundled monthly credits.

The plan fee should not be read as a pure credit-resale price. Plan fees can also cover product access, support, queueing, artifact retention, collaboration, and governance features.

Subscription Credit Lifecycle

Subscription plans combine platform access with an included credit allowance. Wallet top-ups are separate compute purchases. The product should keep these balances understandable because they behave differently.

• Included plan credits are bundled into a monthly, yearly, Team, or Enterprise plan. They support the current billing period and normally reset at the next cycle rather than rolling forward indefinitely.
• Purchased wallet credits are top-up credits bought separately for extra compute. They remain in the wallet until used, unless a future account term or enterprise contract defines a specific expiry period.
• Monthly plans refresh the included credit allowance at the start of each billing cycle. Unused included credits do not create a cash balance, refund, or unlimited future compute liability.
• Yearly plans may provide credits as a monthly or quarterly allowance, or as a contract-defined annual pool. Team and Enterprise annual pools should use admin spend caps and approval thresholds so one early workflow does not consume the full contract unintentionally.
• If credits remain at month end, included plan credits reset according to the plan cycle, while purchased wallet credits remain available.
• If credits are depleted before month end, the user can buy top-up credits, upgrade the plan, wait for the next allowance refresh, or request an admin-approved workspace spend-cap increase.
• If a user cancels or downgrades, unused included plan credits should not be cashed out. Purchased wallet credits should follow the account terms for reactivation, expiry, or refund handling.

The billing UI should label these balances separately: plan allowance, purchased wallet credits, reserved credits, settled net charges, refunds, and remaining available balance.

Credit Source Labels

Ledger labels should keep the source of credits clear:

• subscription_allowance: included plan credits for the active billing period.
• top_up_purchase: purchased wallet credits from a payment provider or approved invoice.
• trial_credit: optional one-time registration or promotional credit.
• service_debit: reserved or direct charge for training, SHAP, reports, prediction, or another billable action.
• service_refund: credits returned after failure, cancellation, or settlement below the reserved amount.

Historical rows may still contain legacy labels such as initial_credit or payment_credit. Product copy should normalize those rows to the same story: trial credits are optional grants, and payment credits are top-up purchases.

Choosing A Cost Strategy

For exploratory work, start with Preview Data Readiness, a modest model set, and a low-cost or balanced preset. Expand model coverage after the first run shows the dataset is suitable and the early model comparison is useful.

For production or shared workspaces, prefer plans that provide admin review, spend caps, and pooled credit visibility. Team and Enterprise workspaces should use spend caps and approval thresholds for high-cost jobs, premium workflow families, SHAP, report generation, and large batch prediction.

Break-Even Framing

Per-job break-even depends on settled credits, not the initial estimate alone.

job revenue = settled credits * price per credit
job breaks even when job revenue >= variable job cost

At the current self-serve top-up anchor of $0.10 per credit, a job that settles at 100 credits produces $10 of gross revenue. If the variable compute, storage, provider, and support cost for that job is $3, the job has a 70% gross margin:

gross margin = (revenue - variable cost) / revenue
gross margin = ($10 - $3) / $10 = 70%

For a target gross margin, the required revenue is:

required revenue = variable cost / (1 - target gross margin)
required credits = required revenue / credit price

Company-level break-even also depends on fixed monthly costs such as hosting, support, development, monitoring, payments, and administration:

required monthly revenue = fixed monthly costs / blended gross margin

This is why shadow pricing, model-family weights, dataset/media-size factors, and settlement logs matter. They show whether plan allowances and per-job charges are commercially sustainable before pricing changes are turned on for paid users.

Why Reserve And Settle Exists

Jobs can vary in runtime, model availability, optional add-ons, and artifact generation. The platform debits the current estimate before work begins, then settles to the actual charge after it knows what completed.

This avoids charging users for outputs that were not produced while still making estimated cost visible before expensive work starts.

Typical Billing Flow

1. Debit the current estimated credits before execution.
2. Run the workflow.
3. Settle based on actual successful work.
4. Refund unused credits when applicable.
5. Refund failed add-ons when outputs are not produced.

If training is interrupted, cancelled, or fails before producing a billable trained output, the training reservation should be refunded. The refund must appear as a wallet ledger event before the product can say the user was actually refunded.

Billable Actions

• training jobs
• SHAP/explainability add-ons
• AI report generation
• batch prediction at scale
• storage or retention where configured

Training charges should be based on successful trained outputs after settlement, not simply the number of models originally requested. If no billable trained output is produced, the training reservation should settle to zero or be refunded according to the active failure path.

Manual prediction is free for single-record previews from a completed job. Batch prediction is metered after the included row/item allowance and may reserve credits based on the uploaded row/item count, upload or media size, service multipliers, and selected model-family factors.

Pricing Rollout State

Training pricing now records both the current charged estimate and a shadow enhanced estimate. The enhanced estimate considers model family, dataset size, media size, and runtime signals. By default, this runs in shadow mode so real job deltas can be reviewed before changing live charges.

Important fields:

• training_estimated_credits and training_reserved_credits: the amount used for the current wallet reservation.
• training_shadow_estimated_credits: the enhanced price that would apply if the rollout mode were active.
• training_settlement_cost_details: the actual settlement calculation after successful models and runtime are known.
• training_billing_summary: a compact job-level summary of estimate, reservation, settlement, refund, and charge-reference fields.

The enhanced pricing switch is controlled by MLAGENT_TRAINING_PRICING_MODE. Use shadow while inspecting margins and set it to active only after the observed deltas look acceptable.

Model-family weights should be treated as calibrated product policy, not as a universal mathematical truth. Good weights are anchored in objective signals such as runtime, memory pressure, GPU/provider cost, artifact size, dependency risk, support burden, and observed failure/refund rates. They still require business judgment because plan positioning, user expectations, and margin goals also matter. Keep enhanced pricing in shadow mode until real job telemetry shows that the proposed deltas are explainable, stable, and fair across service families.

Late-Failure Policy

Early validation failures, missing files, platform errors, and dependency failures should still refund the reservation. For late failures that occur after meaningful compute has already been spent, the platform records a shadow partial charge. The late-failure policy is controlled separately by MLAGENT_LATE_FAILURE_BILLING_MODE.

In shadow mode, the user still receives the normal refund while the job stores what the partial charge would have been. In active mode, eligible late failures can keep a bounded minimum charge and refund the rest. This should be enabled only after reviewing real failure patterns.

Reading A Ledger Entry

Ledger entries should be read as a sequence:

• Up-front debit: estimated credits were deducted before execution.
• Settled: final charge was computed after execution.
• Refunded: unused reservation was returned.
• Net charge: the final amount the user actually paid.

For example, if a workflow debits 50 credits and later refunds 50 credits, the final net charge is 0 credits. If it debits 13 credits and refunds 7 credits, the final net charge is 6 credits.

User-Facing Explanation

Billing language should make net charge clear:

• up-front debit amount
• refunded amount
• final net charge
• reason for settlement or refund
• dollar equivalent when a credit-to-dollar conversion is configured

For job-level billing displays, the wallet ledger is the source of truth for actual debits, refunds, and settlements. Job-local billing mirrors can help with context and troubleshooting, but duplicate local mirror rows should not be rendered as additional wallet charges.

Add-On Billing

Add-ons such as SHAP or AI report generation should only settle as a net charge when the requested output is produced according to the product contract. Failed or unavailable add-ons should be refunded or settled to zero according to the active billing rules.

What Users Should Check

• The estimate before starting the action.
• Whether failed models changed the final charge.
• Whether optional add-ons produced artifacts before settling.
• The final net charge after any refund.
• Whether the wallet balance changed by the expected amount.

Failed Or Pending Refunds

A failed job can have a saved debit and a failed refund attempt at the same time. In that case, the billing workspace should show refund attention from saved job metadata even though the wallet ledger has not yet recorded the refund. Treat this as unresolved billing state, not as a completed refund.

Common fields to inspect are:

• training_refund_status
• training_refund_error
• training_refund_reference
• billing_summary.refund_status
• wallet ledger entries with entry_type=service_refund

Administrators can retry a failed training refund through the training-refund reconciliation endpoint after correcting the underlying issue. A common cause is a mismatch between the web app and auth/billing service INTERNAL_REFUND_SECRET values. The secret must match in the web app environment and auth_payment_service environment, and the auth/billing service must be restarted after a change.

Related Source Docs

• Running jobs
• Choosing a service
• Service reference overview

Source: docs/book/chapters/12-architecture-and-execution-flow.md

Architecture And Execution Flow

This chapter gives the system-level view of the canonical product. It explains how the FastAPI/Jinja app, service pipelines, job persistence, monitoring, billing, and AI reporting fit together as one operating workflow rather than as separate tools.

Canonical Product Surface

The source of truth for the product experience is the FastAPI/Jinja web application. In practice, that means:

• application routes that coordinate pages and APIs
• shared templates that keep workflows consistent
• shared CSS and JavaScript that provide one visual system
• one application entrypoint that starts the product shell

Legacy or adjunct interfaces may exist in the repository, but the main product shell, jobs flow, billing flow, and report flow are centered on this route-driven web app.

Architecture Layers

The system can be understood in six layers.

flowchart TD
    A[User and Browser] --> B[FastAPI and Jinja App]
    B --> C[Routes and APIs]
    C --> D[Job Manager and Pipeline]
    D --> E[Domain Services]
    E --> F[Saved Artifacts and Results]
    F --> G[Job Detail and Download Surfaces]
    F --> H[SHAP and AI Report Actions]
    C --> I[Billing and Auth Services]
    H --> F
    I --> G

Figure: High-level product architecture from user actions through routes, pipelines, saved artifacts, reporting, and billing.

1. Presentation Layer

The presentation layer is the user-facing shell built from Jinja templates, shared upload forms, job detail views, and report/billing surfaces.

Primary responsibilities:

• guide service selection and upload
• display long-running state clearly
• surface artifacts, plots, SHAP, billing, and AI reports
• keep service workflows feeling like one product

2. Route And Application Layer

The route layer translates user actions into pipeline operations, job launches, report generation, wallet/billing actions, and result retrieval.

Primary responsibilities:

• register pages and APIs
• validate requests and normalize inputs
• map UI actions to durable job and artifact workflows
• expose websocket or polling state to the frontend

3. Pipeline And Job Layer

The pipeline/job layer creates durable job records and runs the shared workflow stages that every service depends on.

Primary responsibilities:

• create/update job manifests
• track progress, stage, and terminal state
• run shared preprocessing, EDA, and readiness steps
• orchestrate service-specific training and post-processing

4. Domain Service Layer

The domain service layer contains the actual ML workflow implementations for tabular, longitudinal, time-series, text, media, survival, specialized, and advanced-system services.

Primary responsibilities:

• service-specific preprocessing
• model registration and training
• evaluation, comparison, and plot generation
• prediction and artifact persistence

5. Reporting And LLM Layer

The reporting layer converts saved job evidence into structured report context and manuscript-style outputs.

Primary responsibilities:

• assemble structured report inputs from saved job artifacts
• resolve references and study-domain context
• generate manuscript-style reports and executive summaries
• persist rendered report outputs and validation metadata

6. Billing, Auth, And Operations Layer

The operational layer handles wallet logic, reservation/settlement, user identity, and workflow monitoring semantics.

Primary responsibilities:

• reserve and settle credits
• expose wallet and ledger history
• connect users to their jobs and report actions
• preserve operational truth across live monitoring and recovery

Core Execution Flows

Training Flow

The normal training flow is:

1. The user selects a service and uploads data.
2. The route layer validates the request and stages input files.
3. A durable job record is created.
4. The pipeline runs shared EDA/readiness and dispatches the chosen domain service.
5. Candidate models train, evaluate, and persist artifacts.
6. Results, plots, metrics, and metadata are written into the job result directory.
7. Billing settles against actual completed work.
8. The job detail page presents the final outputs.

flowchart LR
    A[Service Selection and Upload] --> B[Route Validation and Staging]
    B --> C[Job Created]
    C --> D[EDA and Readiness]
    D --> E[Model Training and Comparison]
    E --> F[Artifacts and Metrics Saved]
    F --> G[Billing Settlement]
    G --> H[Job Detail, Downloads, and Prediction Surfaces]

Figure: Operational training flow showing the durable path from upload to settled results.

Reporting Flow

The reporting flow is deliberately downstream of training:

1. The user explicitly launches AI report generation or regeneration.
2. The reporting route builds structured manuscript context from saved job payloads and artifacts.
3. The report generator writes manuscript HTML, metadata, validation files, and resolved references.
4. The report surface reads those durable outputs rather than inventing context on the fly.
5. Billing settles only after the requested report action reaches a durable outcome.

Monitoring Flow

Live monitoring is a delivery problem, not a truth problem.

The operating rule is:

• persisted state is truth
• websocket and polling are transports
• raw log lines are supporting evidence, not status authority

flowchart TD
    A[Persisted Job and Artifact State] --> D[Visible Status Truth]
    B[Websocket Updates] --> D
    C[Polling and Log Tails] --> D
    B -. transport only .-> A
    C -. supporting evidence only .-> A

Figure: State-truth rule for monitoring. Durable persisted state outranks transports and log-derived impressions.

That distinction matters because users should not see stale progress labels, duplicate launches, false recovery messages, or billing confidence that the stored job state does not actually support.

Upload-page monitors use the same state-truth rule. When a workflow starts without immediately navigating away from the form, the embedded progress area should initialize only after its markup exists, then catch up through the shared job monitor. This avoids a form that says only "Processing..." while the real job is already advancing elsewhere.

Event-Loop Responsiveness For Long Jobs

Some service methods are declared as async but internally run long synchronous model-fitting work. Those services must not execute their full training loop on the FastAPI event loop, because doing so can starve websocket sends, polling requests, auth-status refreshes, and heartbeat traffic until the model fit returns.

The shared pipeline therefore offloads CPU-heavy async training for longitudinal, time-series, ranking, survival, and media-classification workflows. Survival is included because lifelines, scikit-survival, and neural survival model fits can run for minutes while still being wrapped by an async service entry point.

For users, the intended behavior is:

• the upload-page progress bar should advance from queued, through EDA, through model training, and into completion
• /api/job-status/<job_id> should remain responsive while the job is active
• websocket snapshots and polling should show the same persisted state
• if the websocket reconnects, the job manifest and live_events are used to catch the UI up

If a reload or shutdown cancels a background task, cancellation is treated as a first-class terminal runtime outcome. Instrumentation should record the stage as cancelled, and the job route should mark the job as cancelled or interrupted instead of allowing a None pipeline result to masquerade as successful completion.

Persistence Model

Each job has two durable homes:

• a job manifest that records its identity and state
• a job-specific artifact directory that stores results, plots, and evidence

Typical persisted outputs include:

• model and preprocessor artifacts
• plots and EDA summaries
• training results and comparison tables
• SHAP artifacts and plot files
• manuscript/report artifacts
• billing and validation metadata

Long-running services should persist core metrics and model results before optional finalization work such as plots, feature importance, or report assembly. That snapshot gives recovery and job-detail pages a durable source of truth even if the server is interrupted while optional artifacts are still being generated.

During an active run, that early snapshot must not be treated as terminal completion on its own. model/training_results.json can exist while final model-comparison plots, service diagnostics, reproducibility metadata, billing settlement context, and report.json are still being written. Active-job rehydration may merge model facts from the snapshot, but it should promote the job to completed only when final pipeline evidence such as job_results.json or report.json is present.

This persistence model is what makes regeneration, report rebuilding, downstream prediction, and auditability possible.

Invariants That Matter

Several workflow truths are more important than any one UI rendering:

• one authoritative training payload should win over stale recovery fragments
• active jobs should not become terminal from intermediate training snapshots alone
• completed outputs should not be mislabeled as recovered unless recovery actually happened
• one user click should produce one action launch
• billing UI should reflect authoritative ledger state, not placeholders
• downloadable report outputs should match the saved manuscript state shown in the app

These are not cosmetic concerns. They determine whether the system feels trustworthy under interruption, regeneration, or partial failure.

Why This Architecture Matters For The Book

This book is not only a service catalog. It is also a map of how the product behaves end to end. The architecture view helps readers connect:

• upload pages to pipelines
• pipelines to saved artifacts
• saved artifacts to SHAP and AI reports
• live job state to billing and recovery behavior

Without that systems view, the service chapters can look self-contained when, in reality, they are part of one shared platform contract.

Deep References

• Running jobs
• Artifacts and implementation
• Reproducibility, audit trail, and regeneration
• Runtime guardrails
• Operations and maintainer guidance

Source: docs/book/chapters/13-eda-figures-and-reporting-reliability.md

EDA, Figures, And Reporting Reliability

EDA and figure generation are the first trust layer of the product. If the statistical summary is shallow, the plots are missing, the plot manifest is stale, or the report context is only partially populated, every later stage becomes harder to trust.

Why This Layer Matters

Users do not experience the platform only as model metrics. They experience:

• what warnings the system gave before training
• which plots appeared on the job page
• whether SHAP images actually display
• whether downloaded reports match the embedded manuscript
• whether live monitoring and billing messages reflect the true state

That means the reliability contract is broader than "model finished."

The EDA Contract

The non-negotiable product target is that every active service returns the same high-level EDA structure:

{
  "statistics": {},
  "warnings": [],
  "readiness_checks": {},
  "recommended_models": [],
  "plots": ["eda/plot_a.png", "eda/plot_b.png"]
}

This does not require identical diagnostics across services. It requires that every service produce a truthful, consistent manifest the pipeline, UI, caching layer, and report generator can rely on.

Feature-distribution panels should be materialized as independent plot entries where possible. A combined "all features" image is harder to inspect, harder to reuse in reports, and easier to lose in route-level filtering. The EDA tab should therefore be able to display each feature-distribution plot as its own card across active services.

Model-evaluation plots are a separate contract. Actual-versus-predicted, residual, QQ or residual-distribution, feature-importance, model-comparison, and training-time plots belong in the Model Results surface rather than being mixed into EDA simply because they are PNG files.

Service-Aware EDA

The right EDA depends on the service family:

• classification needs target balance and class diagnostics
• regression needs target shape and residual-oriented readiness checks
• count regression needs zero-inflation [@Lambert1992] and over-dispersion [@CameronTrivedi1998] visibility
• longitudinal workflows need subject/time structure checks, including subject coverage, attrition, observations per subject, target observation or missingness over time, dropout pattern, last observed visit distribution, covariate missingness over time, average outcome trends, class prevalence over time for categorical targets, and time-gap diagnostics
• time-series workflows need order-aware and lag-aware diagnostics
• survival needs event/censoring and time-to-event summaries
• media/text need file/content readiness checks rather than only table plots

For longitudinal workflows, these are EDA figures rather than model-specific diagnostics. ROC curves, confusion matrices, threshold plots, residual plots, feature-importance plots, and model-comparison plots belong in model-results or appendix contexts. They should not be promoted as the primary longitudinal EDA story merely because they are available as image files.

Generic plots are better than nothing, but they are not enough for product-grade trust.

Course-Alignment Contract

The MIT-IDSS notebook review is now represented as a service-family contract in mlagent_website/utils/course_alignment.py. The contract does not replace the live service catalog. Instead, it names the EDA checks, model families, evaluation metrics, plot groups, and implementation checks that each service family should make visible.

Readiness profiles include a course_alignment section so a preview, saved job, report, or audit can inspect what the workflow was expected to check. This keeps course-derived expectations out of one-off prose and gives the product a stable way to explain why a service should show, for example, residual diagnostics for regression, zero-count cues for count regression, cold-start diagnostics for recommendation, or graph integrity checks for network analysis.

Figure Reliability Problems The Repo Already Exposed

The repo history already shows several concrete failure patterns.

1. Plots Generated But Not Registered

A plot can exist on disk and still not appear in the job UI if the authoritative plot manifest is not updated. The SHAP display investigation showed exactly this pattern: SHAP images were generated, but eda_results.json was missing the corresponding plots/shap/... entries, so the template never surfaced them.

2. Metrics Exist But Diagnostic Plots Do Not

An anomaly detection workflow can compute metrics successfully and still fail to save the model file required for later diagnostic plotting. If that save failure is swallowed silently, the later plot stage sees "file missing" and the UI ends up looking incomplete without explaining why.

3. Route-Level Mitigation Is Not Enough

Filesystem rescans and route-level plot merging can hide some persistence problems, but they are not a durable substitute for writing the right authoritative manifest during the workflow itself.

4. Early Completion Can Race Final Plot Writes

Intermediate training snapshots can appear before final model-comparison plots or report metadata are saved. The UI should not treat those snapshots as terminal completion for active jobs. A job is ready for full result review only after final pipeline evidence such as job_results.json or report.json is available and the plot inventory can be hydrated from disk.

5. Report Figures Can Drift From Manuscript State

If report regeneration, reference resolution, or manuscript context loading is incomplete, the embedded report can look updated while the downloadable output or validation metadata tells a different story. This is especially risky for legacy-job regeneration.

Monitoring Reliability Is Part Of Reporting Reliability

Users interpret logs, status pills, progress labels, recovery notices, and billing messages as part of the result. That means live logging and workflow-state invariants are part of report reliability, not a separate concern.

Important operational rules include:

• persisted state must beat stale frontend inference
• one action launch should create one action
• terminal states should not regress because of old live fragments
• recovery labels should appear only when real recovery happened
• billing states should not imply authoritative settlement when the ledger does not support it

Practical Reliability Checklist

Before calling a workflow or report output trustworthy, verify:

• the EDA payload includes a real plots manifest
• plot paths are relative and job-local
• plot files on disk match the manifest
• SHAP plots are registered after they are generated
• report context files are populated from the saved job payload
• validation JSON warns when structured report context is partial or missing
• the in-app report and downloadable report reference the same figure set
• live status, billing state, and action recovery labels match durable backend truth
• active jobs were not marked completed from intermediate model/training_results.json alone

What Product-Grade Looks Like

For this platform, product-grade reliability means:

• no active service returns placeholder-only EDA
• every active service emits domain-appropriate plots
• figure paths are durable and renderable in both app and report surfaces
• report generation is grounded in saved artifacts, not generic fallback context
• warnings are explicit when context or figures are incomplete

Deep References

• Data preparation
• Understanding results
• Explainability and AI reports
• Architecture and execution flow
• Reproducibility, audit trail, and regeneration
• Troubleshooting

References

Source: docs/book/chapters/14-service-reference-overview.md

Service Reference Overview

The service reference covers 20 active service workflows and the current upload-visible model catalog. Model counts are generated from service metadata so the handbook does not become stale when a workflow adds, hides, or retires a model.

Service Families

• Cross-sectional: 3 services
• Longitudinal: 3 services
• Time series: 3 services
• Text: 1 service
• Media: 3 services
• Specialized: 3 services
• Advanced systems: 4 services

20-Service Quick Cards

The complete generated list and a plain-language definition for every configured model appear in the authoritative service and model registry appendix. The upload/catalog parity summary explains the related UI contract.

Per-Service Documentation Pattern

Each service chapter should help readers answer:

• What question does this service answer?
• What data shape does it require?
• Which columns are required?
• Which models can run?
• Which models may be unavailable due to optional dependencies?
• Which metrics should users trust most?
• Which artifacts are produced?
• Which failure modes are expected and recoverable?
• Where can a developer find the supporting implementation?

Model Documentation Pattern

Each model entry should include:

• plain-language purpose
• algebraic definition
• objective or loss function
• assumptions
• preprocessing path
• implementation mapping
• metrics emitted
• interpretability behavior
• runtime guardrails
• artifacts persisted
• known limitations

Detailed Source Files

• Cross-sectional services
• Longitudinal services
• Time-series services
• Text and media services
• Specialized services
• Advanced systems
• Running jobs
• Billing and credits

Reading Pattern

Each service family chapter should eventually include:

• when to use the service
• required data structure
• preprocessing
• model inventory
• model equations and objectives
• metrics
• artifacts
• limitations
• implementation mapping

Source: docs/book/chapters/15-cross-sectional-services.md

Cross-Sectional Services

Cross-sectional services cover tabular datasets where each row is treated as one observation. They are the right starting point when the dataset has one record per person, transaction, sample, facility, image summary, or event and there is no required subject-level time ordering.

Services Covered

Data Contract

The common contract is a rectangular table with a target column and predictor columns. Predictors may include numeric, categorical, boolean, and date-derived fields. The workflow is most reliable when the target is cleanly encoded, column names are stable, and train/test leakage fields such as future outcomes, identifiers, or post-event summaries are removed before upload.

Classification requires a target with at least two classes. Regression requires a numeric target. Count regression expects integer-like non-negative outcomes and is the better choice when the response is skewed, zero-inflated, or naturally represents counts.

Cross-sectional services can use an internal split, a user-supplied split column, or separate train/validation/test files. The split role and any sampling-weight column are control columns, not predictors. When weights are provided, compatible model families receive the training weights; unsupported families should make the limitation visible rather than treating the weight column as a normal feature.

If users upload train and validation files without a test file, the shared split contract creates an internal test holdout from the training file before fitting. The main Model Performance table should use that generated test split, while the Validation Dataset section keeps uploaded validation metrics and best-model validation plots separate.

Shared Pipeline

The cross-sectional pipeline generally follows this pattern:

1. Validate upload schema, target selection, and service-specific requirements.
2. Split the dataset into training and validation/test partitions.
3. Build preprocessing for missing values, categorical encoding, scaling where needed, and feature alignment.
4. Train candidate model families under a consistent evaluation contract.
5. Select the strongest model using service-appropriate metrics and guardrails.
6. Persist model artifacts, metrics, comparison tables, plots, and report-ready metadata.
7. Optionally run SHAP or compatible explainability on the selected model or the strongest explainable fallback.

Model Families

Classification and regression compare a broad set of classical and modern tabular methods:

• Linear, logistic, ridge [@Hoerl1970], lasso [@Tibshirani1996], and elastic-net style models [@ZouHastie2005].
• Naive Bayes and related probabilistic classifiers where applicable.
• Decision trees [@Breiman1984], random forests [@Breiman2001], extra trees [@Geurts2006], and bagged ensembles.
• AdaBoost [@FreundSchapire1997], gradient boosting [@Friedman2001], XGBoost [@Chen2016], LightGBM [@Ke2017], and CatBoost [@Prokhorenkova2018].
• K-nearest neighbors [@CoverHart1967] and support vector machines [@Cortes1995].
• Multilayer perceptrons [@Rumelhart1986], Keras MLPs, and compatible neural tabular paths.

Count regression adds count-specific approaches:

• Poisson generalized linear models [@McCullaghNelder1989].
• Negative binomial models for over-dispersed counts [@CameronTrivedi1998].
• Zero-inflated and hurdle-style approaches where supported [@Lambert1992].
• Tree, SVM, KNN, MLP, and boosting regressors exposed with count-specific runtime IDs such as svm_count, knn_count, mlp_count, gradient_boosting_count, and ada_boost_count.
• Keras MLP/CNN count variants exposed as keras_mlp_count and keras_cnn_count when the TensorFlow/Keras runtime is available.
• GAM-style or smooth count models where available through the runtime [@HastieTibshirani1990].

Metrics And Selection

Classification emphasizes discrimination and calibration signals such as ROC-AUC, PR-AUC, accuracy, balanced accuracy, precision, recall, F1, log loss, confusion matrices, and threshold behavior. For imbalanced problems, PR-AUC [@DavisGoadrich2006] and threshold sanity are especially important.

Regression emphasizes error and explained-variation metrics such as RMSE, MAE, R-squared, residual plots, and prediction-vs-observed summaries. Count regression should be interpreted through count-aware error patterns, zero handling, dispersion, and whether the model produces plausible non-negative predictions.

Artifacts

Typical outputs include:

• model comparison tables
• selected model artifact
• preprocessing artifact
• metrics JSON/CSV
• prediction summaries
• feature importance or SHAP outputs when available
• report context used by the AI manuscript/report workflow

Common Failure Modes

• Target leakage from columns that encode the answer after the fact.
• Identifier columns acting as memorization features.
• Severe class imbalance without enough minority examples.
• Count targets routed through ordinary regression when count regression would be more appropriate.
• Neural models producing a strong metric but a pathological cutoff or unstable threshold behavior.
• Optional dependency gaps for XGBoost, LightGBM, CatBoost, TensorFlow, or SHAP-compatible explainers.

Deep Reference

See the cross-sectional model mathematics for the full model inventory, algebraic definitions, and implementation notes.

References

Source: docs/book/chapters/16-longitudinal-services.md

Longitudinal Services

Longitudinal services handle repeated observations per subject, unit, facility, patient, animal, device, or cohort member. They are appropriate when the same entity appears more than once and the correlation among repeated records matters.

Services Covered

Data Contract

Longitudinal workflows require more structure than ordinary tabular workflows. At minimum, the dataset should identify:

• a subject, unit, or grouping column
• a time, visit, wave, or ordering column
• a target column
• predictor columns measured before or at the same time as the target

The workflow is strongest when each subject has multiple observations and the time variable has a meaningful order. A dataset with only one row per subject should usually use a cross-sectional service instead.

Longitudinal validation must state whether held-out performance represents new subjects or later visits from known subjects. The saved split policy should record the split strategy, train/test subjects, test size, and whether the evaluation rows came from a user-supplied split. For regression and count regression, downstream plots such as actual-versus-predicted and residual diagnostics should read from saved held-out predictions where available rather than re-predicting with a different subject/time context.

Train, Validation, And Test Files

When users provide separate files for a longitudinal workflow, every file should use the same long-format structure: one row per subject or entity at one visit, time point, or ordered observation. The train, validation, and test files should have compatible column names and data types, including the same subject ID column, time or visit column, target column, and predictor columns. Optional sampling-weight columns should be numeric, non-negative, and excluded from the model feature set.

The right split structure depends on what the user wants to prove:

• For new-subject generalization, subjects in train, validation, and test should be mutually exclusive. This answers whether the model works for new people, facilities, devices, customers, or other entities not seen during fitting.
• For future-visit prediction among known subjects, the same subject may appear in more than one split, but validation and test rows should be later observations than that subject's training rows. This answers whether earlier history can predict later outcomes for subjects already known to the system.

Avoid an ordinary random row split unless that is explicitly justified. Random row splitting can put different visits from the same subject into both train and test, which can leak subject history and make performance look stronger than it is.

If users provide train and test files without a validation file, the train file is still the fitting source. A service may carve an internal validation subset from the training rows for tuning, threshold selection, or early stopping, but the uploaded test file should remain final held-out evidence. If users provide train and validation files without a test file, the uploaded validation file remains model-checking evidence and Xalec should create a subject-aware internal test holdout from the uploaded training file before fitting. If users provide a single file with a split column, the split roles should follow the same subject/time logic and the split column should be excluded from model features. The job page should keep longitudinal validation metrics and validation plots in the Validation Dataset section rather than mixing them with final test evidence.

Method Families

The longitudinal family combines statistical panel methods, mixed-effects methods, and machine-learning methods adapted to repeated measurements:

• Fixed-effects and random-effects style models.
• Generalized estimating equations for correlated outcomes [@LiangZeger1986].
• Linear and generalized mixed-effects models [@LairdWare1982].
• Longitudinal random forests and historical forests.
• REEM/MERF-style mixed-effects ensembles [@Hajjem2011; @SelaSimonoff2012].
• GPBoost-style boosted mixed-effects models where available [@Sigrist2022].
• Count models adapted to subject/time structure.
• Tree and boosting fallbacks when panel-native models are unavailable.

MERF, REEMforest, and MECB remain available as advanced mixed-effects variants for smaller panels, but they are not normal default choices. Preview Data Readiness estimates their interactive fit score and disables them with a warning when the estimate exceeds the 500,000 balanced guardrail. The backend keeps the same budget enforcement so bypassing the upload UI does not launch unsafe long runs.

Upload forms keep Balanced as the interactive default. Users who need package-style reference behavior can opt into the CRAN/reference preset, which uses heavier defaults and should be expected to increase runtime and credit estimates. The reference preset is intended for parity checks, audit work, or method-comparison runs, not for every first-pass analysis.

The native longitudinal package path now expects both local packages to be installed and importable from the application virtual environment:

• longiturf supplies MERT, MERF, MECB, REEMTree, REEMForest, and REEMCatBoost.
• pyhtree supplies HTB, HRF, and HCB historical-tree backends.

Startup should report pyhtree (HTB/HRF/HCB) available when these backends are loaded. If those imports fail, the affected historical-tree models should be treated as unavailable rather than silently shown as trainable.

HTB and HRF follow the historical-tree workflow and control behavior exposed by the local pyhtree bridge. HCB is the CatBoost historical-tree variant in the Python package and should use the same subject/time-aware historical contract, even though it is not a separate CRAN htree export.

For longitudinal count regression, advanced count GEE and count GLMM variants are production-available only when their registered wrappers and runtime dependencies are present end to end. The app should declare and install statsmodels and patsy, register the count wrappers in the service catalog and upload form, and keep train, predict, save/load, metrics, SHAP, and report compatibility covered by tests. If those dependencies are missing at runtime, the models should be preflight-unavailable rather than appearing as broken checkboxes.

Standalone randomforest is not exposed as a longitudinal service model. The longitudinal services instead expose named panel-aware families such as HRF, MERF, REEMforest, and their CatBoost variants. Those families may use tree or forest learners internally, but the user-facing model key should preserve the longitudinal method family rather than appearing as a generic random forest.

What Makes The Workflow Different

Longitudinal workflows must protect against treating repeated rows as independent when they are not. The important design questions are:

• Should validation split by subject rather than by row?
• Does the model use group identifiers as random effects, fixed effects, or ordinary predictors?
• Does the time column represent visits, calendar time, exposure duration, or follow-up?
• Are predictors measured before the target, or do some fields leak future information?

These questions affect interpretation as much as model performance. A high score from a row-wise split can be misleading if the same subject appears in both training and validation records.

Metrics And Selection

Classification, regression, and count metrics mirror the cross-sectional services, but the interpretation should be subject-aware. Strong validation should consider whether performance holds across subjects, visits, and time periods rather than only across pooled rows.

For repeated outcomes, a single pooled metric is not enough. The first visual layer should describe the panel itself: subject coverage, attrition, observation density, target observation or missingness over time, time gaps, average outcome trends, class prevalence for categorical outcomes, and feature-stratified outcome trends where relevant. Fitted-model diagnostics can support model review, but they should not replace these EDA plots as the main longitudinal evidence. The book treats the selected model as an empirical prediction model unless the method and study design support a stronger causal or inferential interpretation.

Artifacts

Typical outputs include:

• subject/time-aware validation metadata when available
• model comparison tables
• selected model and preprocessing artifacts
• group/time configuration captured in job metadata
• longitudinal EDA visuals suitable for report generation, including subject coverage, attrition, observations per subject, target observation or missingness over time, dropout pattern, last observed visit distribution, covariate missingness over time, average outcome trends, class prevalence over time where applicable, and time-gap diagnostics
• model-results visuals such as actual-versus-predicted, residual diagnostics, feature-importance, model-comparison, and training-time plots when generated, kept separate from the EDA evidence layer
• AI report context describing the repeated-measures structure

Common Failure Modes

• Missing or incorrectly selected subject/time columns.
• Only one observation per subject.
• Leakage from post-outcome measurements.
• Row-wise validation that overstates performance for repeated subjects [@Roberts2017].
• Optional package gaps for advanced mixed-effects or longitudinal ensemble methods.
• Sparse panels where some subjects have too few observations for stable subject-specific patterns.
• Broken local editable installs for longiturf or pyhtree, which can make native longitudinal backends disappear at startup.

Deep Reference

See the longitudinal model mathematics for package-specific implementation notes, runtime guards, and the full model inventory.

References

Source: docs/book/chapters/17-time-series-services.md

Time-Series Services

Time-series services handle ordered sequences, temporal dependence, lagged structure, and forecast-style features. They are appropriate when the order of observations is part of the signal rather than a nuisance variable.

Services Covered

Data Contract

Time-series workflows need a time column or an explicit sequence structure. Depending on the service, the dataset may represent one series, many parallel series, or multiple entities with their own time-indexed records. The key requirement is that the workflow can determine temporal order and avoid using future observations to predict the past.

Good inputs make clear:

• time index, timestamp, period, or sequence order
• target column or class label
• series identifier when multiple series are present
• covariates available at prediction time
• forecast horizon or windowing assumptions when relevant

When common panel identifiers such as series_id, subject_id, entity_id, or similar columns are present but not explicitly mapped, the workflow should infer them where possible. This keeps grouped time-series plots from collapsing parallel series into one duplicate-time trace.

Multi-Series Workflow

When a time-series upload contains multiple known series, Xalec treats it as a pooled panel workflow by default rather than as many unrelated single-series jobs. The workflow should:

• detect or confirm the series identifier, usually series_id, subject_id, entity_id, or a similar grouped-series column
• validate per-series length, duplicate (series, time) rows, missing periods, and irregular gaps
• hold out the final horizon or validation windows inside each series when the task is forecasting known future periods
• train one pooled model per selected algorithm
• rank algorithms using aggregate per-series validation error across the pooled holdout, primarily RMSE with MAE as the next tie-breaker
• save per-series error summaries and representative forecast plots for inspection
• render actual_vs_predicted.png as a series-aware validation plot rather than connecting unrelated series into one line
• write by-series diagnostic companions for ACF/PACF, rolling statistics, and target drift when grouped regression data are available
• default prediction to all trained series, while still allowing a focused forecast for one selected series and a user-selected horizon

A separate single-series upload remains the simpler workflow for one sequence. For a multi-series job, the model performance table is expected to have one row per algorithm, not one row per series. Series-level diagnostics explain where the pooled model is strong or weak.

Forecast Horizon Contract

time_series_regression and time_series_count_regression now expose an explicit forecast horizon before Preview Data Readiness. The horizon is the number of future time periods the user wants the model to forecast. During readiness preview, Xalec validates that the horizon is positive and shorter than the usable ordered history. During training, the same value defines the chronological validation holdout: the last N ordered observations are held out, the model trains on the earlier history, and prediction defaults to N future periods unless the prediction request supplies a different horizon.

For grouped or multi-entity count time series, the horizon is interpreted per series when possible so each entity keeps historical training rows and the final N periods form the validation segment. If no horizon is supplied, the service falls back to the existing test_size holdout while recording the implicit holdout in the saved job artifacts. The prediction form defaults to forecasting all trained series for that horizon unless the user chooses a specific series. Classification remains a sequence/window labeling workflow and does not use this forecasting horizon contract.

Prediction Workspace Display

For time_series_regression and time_series_count_regression, the prediction workspace treats prediction as a forecast-window action rather than as one isolated future point. After a successful prediction, the History and Next Prediction panel should show the selected series' observed target history and overlay the returned future forecast rows for the chosen horizon. In multi-series jobs, the prediction table and download may contain forecasts for all trained series, but the trend plot focuses on the currently selected series so series are not visually mixed.

Historical fitted predictions are shown only when the saved model/artifact contract provides one prediction per historical time point. Native forecasting models often return a future horizon rather than row-wise in-sample fitted values; in that case, the app should not draw a misleading historical predicted line. The caption should make clear that the displayed red forecast segment is the future prediction window, while the blue series is the observed history.

Classification Window Contract

time_series_classification uses a related but distinct contract. The user sets a lookback window and a classification horizon before Preview Data Readiness:

• Lookback window: how many recent periods the classifier may use as historical context when creating lag and rolling features.
• Classification horizon: how far ahead the class label is defined. A horizon of 0 means classify the current row label; a horizon of 1 means use history at time t to classify the label at t + 1.
• Validation windows: the final chronological windows held out for testing. If left blank, Xalec derives the holdout from the configured test size.

For grouped time-series classification, validation windows are taken from the end of each series where possible so short series still keep earlier rows for training. Saved job artifacts record the lookback, horizon, label mode, validation mode, and usable labeled row count so reports and predictions remain interpretable.

Method Families

The restored documentation covers both feature-based time-series models and sequence-native methods:

• ROCKET [@Dempster2020] and MiniROCKET [@Dempster2021] feature transforms.
• InceptionTime-style deep classifiers [@FawazInceptionTime2020].
• LSTM [@HochreiterSchmidhuber1997], GRU [@Cho2014], CNN-1D, CNN-LSTM, and temporal convolutional networks [@Bai2018].
• Transformer-style sequence encoders [@Vaswani2017].
• ARIMA and auto-ARIMA style statistical forecasting [@BoxJenkins1976].
• Exponential smoothing and trend/seasonality models [@Holt1957; @Winters1960].
• VAR for multivariate temporal dependence [@Sims1980].
• GARCH-style volatility models [@Engle1982; @Bollerslev1986].
• Prophet-style decomposable time-series models where available [@TaylorLetham2018].
• Count-adapted temporal models for event frequencies.

The sequence-classification inventory is deliberately narrower than the forecasting inventory. ARIMA, Auto ARIMA, VAR, GARCH, Exponential Smoothing, and Prophet are available for numeric or count forecasting services, while /time-series-classification exposes models that directly produce class labels or class probabilities.

Time-series count regression includes native count forecasters and sequence wrappers that do not behave like ordinary scikit-learn estimators. During final training cleanup, the workflow should persist model metrics before optional plot/report finalization, generate model-comparison and training-time plots from the saved metrics, and skip expensive permutation feature-importance fallbacks for native count forecasters. Feature importance should be shown only when the fitted model exposes a meaningful and tractable importance contract.

Validation Discipline

Time-series validation should respect time. Random splits can leak future information into training and inflate performance. The safer pattern is chronological validation, rolling-origin evaluation, or blocked validation that preserves order [@Bergmeir2012].

For multi-series data, validation should also consider whether the task is future prediction for known series, generalization to new series, or both.

If users supply train/validation/test files or a split column, the service must still preserve chronological meaning. A supplied test split should represent future periods, held-out windows, or another defensible temporal evaluation design. For grouped time series, the preferred evaluation holds out the tail of each known series when forecasting future periods for those same series. If users provide train and validation files without a test file, the internally created test split should be time-ordered from the uploaded training file and the uploaded validation file should remain separate model-checking evidence. The job page should present time-series validation metrics and validation forecast diagnostics in the Validation Dataset section when those artifacts are saved.

Metrics And Selection

Classification uses the same class metrics as other classification services, but should be read across windows or horizons. Regression uses MAE, RMSE, MAPE-like summaries where appropriate [@HyndmanKoehler2006], residual plots, and forecast error by horizon. For multiple-series forecasting jobs, the persisted best-model ranking metric is aggregate per-series RMSE, with per-series MAE as the next tie-breaker. R-squared remains a useful diagnostic, but it is not the primary winner for panel forecasts because short per-series holdouts can make it unstable. Count regression follows the same per-series RMSE/MAE ranking pattern while keeping Poisson deviance as a count-specific diagnostic and later tie-breaker.

Artifacts

Typical outputs include:

• forecast or sequence prediction tables
• model comparison metrics
• selected model artifact
• temporal preprocessing/windowing metadata
• plots of observed versus predicted values
• panel-aware trend plots for grouped time-series inputs
• per-series validation error breakdowns for grouped forecasting inputs
• representative forecast plots for selected grouped series
• prediction_history.csv with row-aligned holdout predictions for the prediction workspace when the service can save one historical prediction per validation time point
• ACF/PACF diagnostics when enough ordered observations are available, including acf_pacf_by_series.png for grouped inputs
• by-series rolling-statistics and target-drift diagnostics for grouped time-series regression jobs
• horizon-level or window-level summaries when available
• report context describing temporal assumptions

For time-series count jobs, model-comparison artifacts should include a primary performance comparison and a separate training-time comparison when timing data are available. If optional finalization is interrupted, the saved training_results.json snapshot should still contain the trained model metrics and model inventory needed for job-detail recovery.

Common Failure Modes

• Missing or unsorted time column.
• Future leakage through rolling summaries computed incorrectly.
• Too few time points for sequence models.
• Irregular time spacing without explicit handling.
• Multiple series mixed without a series identifier.
• Optional dependency gaps for specialized time-series libraries.
• Optional finalization work, such as feature importance or plot generation, taking longer than the core model fit. Core metrics should already be saved before that work begins.

Deep Reference

See the time-series model mathematics for detailed model notes, algebraic definitions, and implementation mapping.

References

Source: docs/book/chapters/18-text-and-media-services.md

Text And Media Services

Text and media services cover unstructured inputs such as documents, images, audio, and video. These services differ from tabular services because the feature representation is learned or extracted from raw content rather than supplied directly as spreadsheet columns.

Services Covered

Text Data Contract

Text workflows typically require one text field and one target label. The best datasets include enough examples per class, consistent language, and minimal label ambiguity. Text may come from notes, descriptions, survey responses, abstracts, support tickets, messages, or document excerpts.

Text classification uses the shared tabular upload reader. The text and label columns may be supplied in CSV/TSV/TXT, Excel, Parquet, Feather, JSON/JSONL/NDJSON, Stata, SPSS, SAS/XPORT, or R data files when the deployment has the relevant optional readers installed.

Text classification can use the same dataset setup choices as other supervised table-based services: internal split, split column, or separate train/validation/test files. The split role and any sampling-weight column are excluded from model text/features.

When text users provide train and validation files without a test file, Xalec creates an internal test split from the uploaded training file before fitting. The uploaded validation data remains validation evidence and should be reported separately from the generated final test evidence.

The workflow may tokenize text, truncate or pad sequences, build embeddings, fine-tune transformer-style encoders [@Howard2018], or use compatible fallback classifiers depending on runtime availability and configured model options.

Text Method Families

The restored text documentation includes transformer and encoder families such as:

• BERT-style encoders [@Devlin2019].
• RoBERTa [@Liu2019RoBERTa].
• XLM-RoBERTa for multilingual contexts [@Conneau2020].
• ELECTRA, ALBERT, XLNet, MobileBERT, and MPNet variants.
• Classical text vectorization and classifier fallbacks where the runtime uses lighter paths.

Media Data Contract

Media workflows require file inputs and labels. They are sensitive to file organization, supported formats, image/audio/video quality, sample size, class imbalance, and whether the uploaded media can be decoded consistently in the runtime environment.

Image classification usually works from image files and class folders or metadata. Audio classification usually needs clips plus labels and may extract spectrogram or acoustic features. Video classification may sample frames, use temporal features, or use video-native models depending on available dependencies.

Media workflows do not use the statistical dataset reader for the media files themselves. Upload a folder or ZIP bundle of media files, then upload a tabular label manifest. The manifest can use the same supported tabular formats as other upload forms, including CSV/TSV, Excel, Parquet, JSON, Stata, SPSS, SAS, and R data files when the deployment has the relevant readers installed. It should include one media-path column and one target-label column. Canonical names are filename and target, while common aliases such as path, file_path, image_name, audio_name, video_name, label, class, and tag are accepted. Manifest paths should match the relative paths inside the folder or ZIP bundle.

Media manifests may also include a split indicator such as split, partition, or role. Those values are normalized to the internal split role so evaluation can use the user's prepared train/validation/test assignment instead of an automatic split.

For media classification, validation-specific outputs should follow the classification convention when artifacts are saved: validation performance metrics, confusion matrix, ROC curve, and precision-recall curve.

Media Method Families

The restored media documentation covers:

• Image backbones such as ResNet [@He2016], EfficientNet [@Tan2019], DenseNet [@Huang2017], ConvNeXt [@Liu2022ConvNeXt], Inception [@Szegedy2015], ViT [@Dosovitskiy2021], Swin [@Liu2021Swin], and DeiT [@Touvron2021].
• Audio feature models, CNNs, CRNNs [@Choi2017], recurrent models, and attention-based variants.
• Video feature pipelines, frame-based classifiers, 3D CNNs [@Tran2015], and transformer-style video encoders.

Metrics And Selection

Because these services are classification-oriented, common metrics include accuracy, balanced accuracy, precision, recall, F1, ROC-AUC where meaningful, PR-AUC for imbalanced classes, confusion matrices, and per-class summaries. For text and media, per-class performance is often more useful than a single overall score because rare or visually/textually ambiguous classes can be hidden by the average.

Artifacts

Typical outputs include:

• trained classifier or fine-tuned model artifact
• label mapping
• preprocessing or tokenizer metadata
• model comparison table where multiple candidates run
• confusion matrix and class metrics
• selected examples or prediction summaries
• report context for AI-generated interpretation

Common Failure Modes

• Too few examples per class.
• Corrupt or unsupported files.
• Labels encoded only in filenames without a consistent manifest.
• Very long text fields truncated before important content.
• Media files too large for configured runtime limits.
• Optional dependency gaps for transformer, TensorFlow, PyTorch, image, audio, or video backends.

Deep References

• Model methodology reference
• Text model mathematics
• Media model mathematics

References

Source: docs/book/chapters/19-specialized-services.md

Specialized Services

Specialized services cover survival analysis, anomaly detection, and clustering. These workflows are included separately because their targets, evaluation logic, or user goals differ from ordinary supervised prediction.

Services Covered

Survival Data Contract

Survival workflows require at least a duration or follow-up time column and an event indicator. The event indicator should distinguish observed events from censored observations. Predictors should be measured before or at baseline relative to the survival interval.

Survival interpretation depends heavily on censoring. A model can predict risk or survival patterns, but the manuscript and report should remain clear about whether the analysis is predictive, descriptive, or inferential.

Survival Method Families

The survival documentation includes:

• Kaplan-Meier style baseline survival summaries [@KaplanMeier1958].
• Cox proportional hazards models [@Cox1972].
• Penalized and machine-learning survival variants.
• Random survival forests [@Ishwaran2008].
• Survival SVM methods where available.
• DeepSurv [@Katzman2018] and DeepHit [@Lee2018] style neural survival models.
• Survival CNN and transformer variants when supported by runtime dependencies.

Anomaly Detection Data Contract

Anomaly detection can be unsupervised or semi-supervised depending on whether labels exist. The common input is a feature table where unusual records should receive high anomaly scores. The workflow is strongest when obvious identifiers are removed and features represent behavior or measurements relevant to abnormality.

When a label or anomaly indicator is available, the service can honor a user-supplied split column or separate train/validation/test files and compute held-out metrics such as AUROC, AUPRC, confusion matrix, and precision/recall on the supplied evaluation rows. Without labels, anomaly detection should not pretend to have supervised validation; it fits on the available feature table and relies on score distribution, flagged-record review, and domain inspection.

If train and validation files are supplied without a test file, the shared split contract creates an internal test holdout from training before fitting. Validation evidence remains a separate model-checking section, while survival and anomaly services should attach their own split-specific validation diagnostics when saved.

Anomaly Method Families

The restored specialized documentation includes:

• Isolation Forest [@Liu2008].
• One-Class SVM [@Scholkopf2001].
• Local Outlier Factor [@Breunig2000].
• Robust covariance and minimum covariance determinant.
• PCA-style reconstruction or subspace methods.
• HBOS [@Goldstein2012], COPOD [@Li2020], RRCF [@Guha2016], Feature Bagging, DeepSVDD [@Ruff2018], DeepSAD, GANomaly, autoencoder, and VAE-style approaches [@Kingma2014] where supported.

Clustering Data Contract

Clustering expects features but not a supervised target. The user should identify columns that represent the segmentation problem and remove identifiers or downstream labels that would make clusters artificial.

Because clustering is unsupervised, a train/test split is not the main evidence contract. Better validation uses cluster stability, profile reproducibility, holdout assignment where available, and domain review of whether the clusters mean something useful.

Clustering Method Families

The clustering inventory includes:

• k-means and related centroid methods [@MacQueen1967].
• DBSCAN [@Ester1996], HDBSCAN [@Campello2013], and density-based clustering.
• Hierarchical clustering [@Ward1963].
• Spectral clustering [@ShiMalik2000].
• Gaussian mixture models [@Dempster1977].
• OPTICS and BIRCH.

Metrics And Interpretation

Survival analysis uses time-to-event metrics such as concordance-style summaries [@Harrell1982] and survival curves when available. Anomaly detection may use ranking metrics if labels exist, but otherwise relies on score distributions, flagged-record inspection, and domain review. Clustering uses internal validation metrics such as silhouette-style summaries [@Rousseeuw1987] and practical interpretability of the discovered groups.

Artifacts

Typical outputs include:

• survival curves, risk scores, and event/censoring summaries
• anomaly scores and flagged-record tables
• cluster assignments and cluster-profile summaries
• metrics and diagnostic plots
• selected model or pipeline artifacts
• report context describing assumptions and limitations

Common Failure Modes

• Survival jobs missing duration or event columns.
• Event indicator encoded inconsistently.
• Anomaly detection run on identifier-heavy data.
• Clustering interpreted as ground-truth classification.
• Optional dependency gaps for survival, PyOD-style, HDBSCAN, or deep anomaly models.
• Too few events, too few anomalies, or too little feature variation for stable conclusions.

Deep References

• Survival model mathematics
• Specialized model mathematics

References

Source: docs/book/chapters/20-advanced-systems.md

Advanced Systems

Advanced systems cover recommendation, ranking, network analysis, and graphical models. These services move beyond one-row, one-target prediction into relational, preference, graph-structured, and probabilistic dependency workflows.

Services Covered

Recommendation Data Contract

Recommendation workflows usually require users/entities, items, and interactions. Interactions may be ratings, purchases, clicks, views, co-occurrence, clinical actions, service use, or other preference signals. Content features can be included when the workflow supports content-based or hybrid recommendation.

The core design question is whether recommendations should be based mostly on similar users, similar items, item content, popularity, or a hybrid of these signals.

Prepared split roles are supported when the interaction file marks train, validation, or test rows. The split should reflect how recommendations will be used, often by holding out later interactions or held-out user-item events rather than random individual cells without regard to time or cold-start risk. If train and validation are supplied without test, AutoML creates an internal test holdout from training before fitting and keeps validation metrics as model-checking evidence.

Recommendation Method Families

The restored advanced-systems documentation covers:

• Popularity and baseline recommendation.
• User-based and item-based collaborative filtering [@Goldberg1992; @Sarwar2001].
• Matrix factorization and latent-factor models [@Koren2009].
• Content-based recommendation.
• Hybrid recommenders.
• Clustering-based recommendation.

Ranking Data Contract

Ranking workflows require query/group identifiers or comparable candidate sets, candidate features, and relevance labels or preference outcomes. A ranking dataset should make clear which rows compete with each other. Without group structure, ranking can collapse into ordinary regression or classification.

When users supply a split, ranking validation must preserve query or group structure. A split that places competing candidates from the same query in both train and test can overstate ranking quality. When the shared split contract generates an internal test split, ranking outputs should state that final test evidence came from that generated holdout and keep validation diagnostics separate.

Ranking Method Families

The ranking inventory includes:

• XGBoost learning-to-rank models: ranker, pairwise, and listwise variants [@Burges2005; @Cao2007].
• LightGBM learning-to-rank models: ranker, pairwise, and listwise variants.
• CatBoost learning-to-rank models: YetiRank, PairLogit, and QuerySoftMax variants [@Prokhorenkova2018].
• Pointwise relevance baselines that learn relevance scores first and are then evaluated as ranked lists within held-out query groups.

The pointwise relevance baselines are deliberately labeled as baselines rather than full learning-to-rank backends. They include Random Forest, Extra Trees, Gradient Boosting, AdaBoost, KNN, and MLP relevance scorers.

Network Data Contract

Network workflows require graph structure. The upload form supports edge lists with source/target columns and adjacency matrices with node columns. Depending on the task, the dataset may also include node features, edge weights, labels, communities, timestamps, or graph-level metadata. A phase or wave column can be mapped when the user wants CAVIAR-style phase comparison; in that case training uses the aggregate graph while EDA reports per-phase size, density, connectedness, and key-actor centrality trends. The graph structure should be meaningful, not merely a transformed table for convenience.

Network validation should be graph-aware. Node, edge, link-prediction, and phase-comparison tasks can require different holdout rules; a generic row split may break the graph or leak structural information through adjacent nodes.

Network Method Families

The network analysis documentation includes:

• Graph convolutional networks [@KipfWelling2017].
• GraphSAGE [@Hamilton2017].
• Graph attention networks [@Velickovic2018].
• Graph-feature baselines such as Random Forest, Gradient Boosting, Logistic Regression, MLP, SVM, and KNN on structural features.
• Graphormer-style graph transformers [@Ying2021].
• node2vec and graph embedding methods [@Grover2016].
• Variational graph autoencoders [@KipfWelling2016VGAE].
• Broader graph neural and graph embedding variants as supported by dependencies.

Graphical Models Data Contract

Graphical-model workflows use ordinary tabular/statistical datasets but interpret selected columns as variables in a probabilistic system. A state column can be mapped for Markov transition diagnostics. A time or sequence column is optional; when present, it orders states and enables hidden-state and state-space diagnostics. A target or outcome column can be mapped as an anchor variable, but this service does not treat it as an ordinary supervised label.

This service is intentionally separate from network_analysis. Network analysis starts with observed nodes and edges. Graphical models infer dependency structure among measured variables and should not be used to claim causality without a causal design.

Graphical-model evaluation should focus on dependency stability, held-out likelihood or reconstruction diagnostics where available, and bootstrap uncertainty rather than ordinary supervised train/test metrics.

Graphical Models Method Families

The first Graphical Models rollout includes:

• Gaussian graphical lasso for sparse precision-matrix structure.
• Partial-correlation networks from empirical precision matrices.
• Ising pseudolikelihood for binary or median-binarized variables.
• Bayesian-network proxy structure from bounded pairwise dependency screening.
• Markov chain transition matrices and transition graphs from ordered state columns.
• Gaussian hidden-state sequence proxy for ordered records.
• Kalman local-level state-space trace for ordered numeric signals.

Graphical Models Mathematics And Diagnostics

Graphical-model outputs are dependency and state-dynamics diagnostics, not ordinary supervised predictions. The main models use the following contracts:

Gaussian graphical lasso estimates a sparse precision matrix $\Theta$ from the empirical covariance matrix $S$:

$$ \min_{\Theta \succ 0} \left{-\log\det(\Theta) + \operatorname{tr}(S\Theta) + \lambda \lVert \Theta \rVert_1\right}. $$

Partial-correlation networks compute:

$$ \rho_{ij \cdot -ij} = -\frac{\Theta_{ij}}{\sqrt{\Theta_{ii}\Theta_{jj}}}, $$

so larger absolute values indicate stronger conditional association after accounting for the other measured variables.

Ising pseudolikelihood uses nodewise logistic conditionals for binary or median-binarized variables:

$$ P(X_j = 1 \mid X_{-j}) = \sigma\left(\alpha_j + \sum_{k \ne j}\beta_{jk}X_k\right). $$

Markov transition diagnostics estimate:

$$ \hat P_{ij} = \frac{N_{ij}}{\sum_k N_{ik}}, $$

where $N_{ij}$ counts observed transitions from state $i$ to state $j$.

The hidden-state proxy uses Gaussian emission clusters, $x_t \mid z_t=k \sim \mathcal{N}(\mu_k,\Sigma_k)$, while the Kalman local-level diagnostic uses $y_t = \alpha_t + \epsilon_t$ and $\alpha_t = \alpha_{t-1} + \eta_t$.

Confidence intervals are reported where the implementation can compute them honestly: dependency metrics use bounded nonparametric bootstrap intervals, transition metrics resample transition counts, and state-space RMSE uses a fitted-proxy interval. Missing intervals should be explained rather than hidden.

Metrics And Selection

Recommendation and ranking systems should be evaluated with ranking-aware metrics when possible, such as precision-at-k, recall-at-k, NDCG [@JarvelinKekalainen2002], MAP, hit rate, and coverage. Network tasks depend on the specific target: node classification uses classification metrics, link prediction uses ranking or binary metrics, and graph clustering/community discovery uses structure-aware diagnostics. Graphical Models uses a compact primary diagnostic set: structure score, edge count, graph density, mean absolute edge weight, and variables/states. Markov transition entropy and self-transition rate, hidden-state AIC/BIC, transition sparsity/density, and state-space RMSE are secondary family-specific diagnostics rather than universal comparison columns. Fitting uses bounded regularization discipline: graphical lasso cross-validates sparse precision strength, partial-correlation networks use shrinkage covariance before inversion, Ising pseudolikelihood uses regularized nodewise logistic models, and hidden-state proxies choose state count by BIC.

Artifacts

Typical outputs include:

• top-k recommendation tables
• relevance/ranking metrics
• user/item or query/candidate metadata
• graph summaries and embeddings
• centrality rankings and phase-comparison diagnostics
• graphical-model dependency graphs, precision/partial-correlation heatmaps, and edge-list artifacts
• one deduplicated EDA Variable Correlation visual for measured variables
• transition matrices and transition graphs for Markov state-sequence runs
• hidden-state and Kalman trace plots for ordered variables
• selected model artifacts
• report context describing relational assumptions

Common Failure Modes

• Recommendation data without enough repeat interactions.
• Ranking data missing query/group identifiers.
• Leakage from future interactions into training.
• Popularity dominating recommendations without useful personalization.
• Graphs with isolated nodes, missing features, or malformed edge lists.
• Graphical-model inputs with too few variables, too few rows relative to variables, heavy missingness, missing state columns for Markov transition models, or no meaningful time order for dynamic models.
• Treating graphical-model edges as causal effects rather than conditional-dependence diagnostics.
• Optional dependency gaps for ranking libraries or graph neural network frameworks.

Deep Reference

See the advanced-systems model mathematics for full implementation notes, canonical references, and model inventories.

References

Source: docs/book/chapters/21-worked-case-studies.md

Worked End-To-End Case Studies

This chapter turns real workflow patterns and real investigations into end-to-end reading examples. The aim is to show how a user should move from workflow choice to interpretation, and also how the team should reason when the saved evidence and the visible UI drift apart.

How To Read These Cases

Each case is structured around the same questions:

1. Why was this workflow the right choice?
2. What does a healthy job outcome look like?
3. What should the user verify before trusting the output?
4. What failure modes should the team watch for?

Each case also includes:

• the most important evidence points to capture
• the trust question that should be answered before export or publication

Case Study 1: Cross-Sectional Prediction From A Clean Tabular Dataset

Typical Scenario

A user has a single-row-per-observation dataset and wants to predict a categorical or numeric target. The key decision is to choose among:

• tabular_classification
• tabular_regression
• tabular_count_regression

Workflow Choice

This is the right family when:

• rows are independent observations
• there is no required subject-level repeated structure
• there is no time-ordered forecasting requirement
• the target type can be clearly identified as class, continuous value, or count

Healthy End-To-End Flow

1. The user selects the tabular service that matches the target type.
2. Upload validation confirms the target and key predictors.
3. EDA surfaces missingness, imbalance, skew, and leakage risks.
4. Candidate models train and compare under the service metric.
5. The best model is selected with visible runner-ups and failed/skipped models.
6. SHAP can be launched if the selected or fallback explainable model supports it.
7. An AI report can summarize the saved outputs into manuscript-style text.

What The User Should Check

• Was the correct service chosen for the target?
• Are there leakage columns or identifiers left in the feature set?
• Does the selected metric match the actual use case?
• Do plots, metrics, and SHAP outputs tell a consistent story?

Evidence Points To Capture

• the final best-model table
• the comparison view showing runner-ups
• the core validation plots or metrics table
• the SHAP or feature-importance summary used for interpretation

Trust Question

Would a reviewer looking only at the saved outputs agree that the chosen service and winning metric fit the actual study question?

Lessons

This is the most straightforward workflow family, but it is also the easiest place to become overconfident. Good scores on a clean split can still hide leakage, imbalance, or a mismatch between the chosen metric and the real decision goal.

Case Study 2: Repeated-Measure Data That Needs A Longitudinal Workflow

Typical Scenario

A user has repeated observations per subject, patient, facility, or unit across visits or time points. The temptation is often to run a simpler tabular workflow, but that can overstate performance if repeated records leak across train/test partitions.

Workflow Choice

The right family is:

• longitudinal_classification
• longitudinal_regression
• longitudinal_count_regression

This is appropriate when the same subject or unit appears more than once and the grouping/time structure matters.

Healthy End-To-End Flow

1. The user selects the longitudinal service and specifies the subject/group and time columns.
2. Readiness checks verify that repeated observations actually exist and that the time column is meaningful.
3. The workflow compares panel-aware or repeated-measure-compatible models where available.
4. Metrics are interpreted with attention to subject-level validation and repeated-measure leakage risk.
5. The results and report explain the repeated-measures structure rather than treating rows as independent.

What The User Should Check

• Is there truly repeated structure, not just one row per subject?
• Did the workflow capture the right subject and time columns?
• Are the reported metrics plausible under grouped validation?
• Does the manuscript/report language reflect the longitudinal structure honestly?

Evidence Points To Capture

• the configured subject and time fields
• the longitudinal readiness or validation summary
• the final metric table with grouped or panel-aware evaluation
• the report language that describes repeated-measure structure

Trust Question

If the same dataset were flattened into ordinary tabular rows, would the apparent performance increase for the wrong reason?

Lessons

The most important move here is not a model trick. It is choosing the right workflow and validating it correctly. A weaker but correctly structured longitudinal result is more trustworthy than a strong-looking result from the wrong cross-sectional setup.

Case Study 3: Anomaly Detection That Looked Complete But Was Not Fully Healthy

Typical Scenario

An anomaly detection job completed, a best model was selected, and metrics were visible. But deeper inspection showed missing diagnostic plots, missing AUROC/AUPRC for one deep model, and hidden save/load problems that affected downstream diagnostics.

Workflow Choice

The anomaly detection workflow itself was appropriate. The issue was not the chosen service. The issue was that downstream outputs were incomplete even though the page looked mostly successful.

What Happened

Three different failures illustrated three different layers of trust:

• a broken decision_function() caused missing AUROC/AUPRC for DeepSVDD
• silent model-save failures prevented later diagnostic plots from being generated
• plots and comparison outputs needed better logging and verification

Why This Is An End-To-End Case

This case matters because it spans the full lifecycle:

• model fitting
• metric generation
• artifact persistence
• plot generation
• job result presentation

In other words, the job did not fail at a single point. The visible problem emerged because several downstream contracts were weaker than the surface UI implied.

Evidence Points To Capture

• the best-model summary
• the metrics that were present
• the plots or artifacts that were missing
• the logs or validation files that explain the missing downstream outputs

Trust Question

Did the job truly complete cleanly, or did the UI overstate completeness relative to the saved artifacts?

Lessons

• a completed status is not enough
• metrics, model files, and plot outputs must all agree
• missing downstream artifacts should be explained explicitly, not discovered by the user by accident

Case Study 4: SHAP Outputs Existed But The UI Could Not See Them

Typical Scenario

A tabular job generated SHAP plots successfully on disk, but the job detail page did not display them.

What Happened

The issue was not missing images. It was a broken registration path:

• SHAP plots were written to plots/shap/
• the authoritative EDA plot manifest was not updated
• the template depended on the manifest, not a raw disk listing

Why This Matters For Users

From the user perspective, this can feel like "SHAP failed." In reality, SHAP succeeded and the display contract failed. That distinction is important because the right fix is to repair the durable manifest path, not to rerun the workflow blindly.

Evidence Points To Capture

• the SHAP images on disk
• the saved plot manifest
• the rendered job detail page
• the action or artifact state describing SHAP completion

Trust Question

Is the missing visualization evidence a computation failure or a registration failure?

Lessons

• generated artifacts must be registered in the saved manifest
• route-level rescans are a fallback, not the source of truth
• report and UI layers depend on the same underlying artifact contract

Case Study 5: Workflow State And Billing Must Follow Durable Truth

Typical Scenario

Training, SHAP, or AI report actions appear complete in the interface, but stale recovery fragments, duplicate action triggers, or billing placeholders make the state feel inconsistent or confusing.

What Happened

The workflow-state audit surfaced problems such as:

• transient recovery payloads competing with final training results
• completed add-on outputs being mislabeled as recovered
• duplicate click handlers causing repeated action starts
• billing messages implying settled activity without authoritative ledger evidence

Why This Is A Product Case Study

This is not only a backend correctness issue. It is a user-trust issue. The platform's meaning is carried by:

• status labels
• recovery pills
• action cards
• billing lines
• downloadable outputs

When those signals drift away from durable state, the user is forced to guess which layer is telling the truth.

Evidence Points To Capture

• the billing timeline
• the ledger entries
• the job state labels and recovery badges
• the actual add-on outputs present on disk

Trust Question

Could an external reviewer reconstruct the true sequence of execution, settlement, and regeneration from the saved state alone?

Lessons

• terminal state must beat stale live fragments
• one click should create one action
• billing should clearly distinguish reserved, refunded, and net charge
• recovery language should only be used when actual recovery happened

Case Study 6: Text Classification Where The Encoder Family Matters

Typical Scenario

A user has labeled notes, documents, or free-text records and wants a classifier that respects the language content instead of flattening everything into a manually engineered feature table.

Workflow Choice

The right family is:

• text_classification

This is appropriate when the text itself carries the predictive signal and the user wants a true NLP pipeline rather than a tabular proxy.

Healthy End-To-End Flow

1. The user confirms there is a clean text column and a label column.
2. The service tokenizes text and fine-tunes one or more transformer encoders.
3. The comparison table makes clear which encoder family actually ran.
4. The winning model is interpreted with awareness of tokenizer and encoder-family tradeoffs.
5. The report explains the text workflow in language that is faithful to the saved model family.

What The User Should Check

• Was the text column clean and meaningful, rather than mostly IDs or placeholders?
• Did the service really run the intended encoder family?
• Are the results plausible for dataset size and label balance?
• Does the report describe a text-classification workflow rather than generic tabular learning?

Evidence Points To Capture

• the final model comparison table
• the winning encoder identifier
• the metrics table for the winning model
• the report section describing the text workflow

Trust Question

Is the report grounded in the actual transformer family that ran, or is it generic enough that the same prose would fit almost any workflow?

Lessons

Text workflows are easy to oversimplify in prose. A healthy output should preserve the distinction between encoder families, tokenization-heavy NLP training, and ordinary structured-data classification.

Case Study 7: Survival Analysis Where Simpler Classification Would Be Misleading

Typical Scenario

A dataset contains an event indicator, a time-to-event column, and censoring. A simpler classification framing may look tempting, but it would throw away the timing structure and censoring information.

Workflow Choice

The right family is:

• survival

This is appropriate when the timing of the event and the presence of censoring are central to the study question.

Healthy End-To-End Flow

1. The user confirms that time and event columns are valid.
2. The service trains a censoring-aware survival workflow.
3. The results present survival-specific summaries rather than only generic classification scores.
4. The report keeps the survival framing explicit.

What The User Should Check

• Are time values positive and event indicators valid?
• Is censoring present and handled explicitly?
• Do the outputs include survival-native artifacts rather than ordinary tabular stand-ins?
• Does the manuscript language preserve the survival framing?

Evidence Points To Capture

• the survival workflow configuration
• the survival curves or survival-model summaries
• the chosen survival model family
• the report language describing event-time analysis

Trust Question

Would a reader of the exported report understand that this was a time-to-event analysis with censoring, not an ordinary binary prediction task?

Lessons

Workflow choice is especially important here because the wrong simplification removes part of the science, not just part of the model sophistication.

Case Study 8: Recommendation And Ranking Need Structure, Not Just More Features

Typical Scenario

A team wants to recommend items or rank candidates and is tempted to flatten user, item, or query identifiers into a generic supervised table.

Workflow Choice

The right families are:

• recommendation_system
• ranking_models

This is appropriate when user-item interactions or query-level relevance groups are part of the task definition.

Healthy End-To-End Flow

1. The user confirms the presence of user-item or query-group structure.
2. The service preserves that structure during training and evaluation.
3. The final metrics are interpreted as recommendation or ranking metrics, not ordinary classification accuracy.
4. The report explains how group structure shaped the workflow.

What The User Should Check

• Were the correct grouping identifiers supplied?
• Were cold-start or sparsity limitations made visible?
• Do the metrics correspond to recommendation or ranking quality?
• Does the output avoid pretending this was an ordinary tabular problem?

Evidence Points To Capture

• the configured identifiers such as user_id, item_id, or query_id
• the ranking or recommendation metric summary
• the winning model family
• the report section describing the structural task

Trust Question

Did the workflow preserve the real interaction or ranking structure, or was a richer systems problem reduced to a convenient but weaker tabular proxy?

Lessons

Advanced-system workflows are valuable precisely because they preserve structure that simpler supervised pipelines would erase.

Cross-Case Pattern

Across all of these cases, the same principle keeps returning:

The durable workflow contract matters more than a superficially plausible page.

That contract includes:

• correct workflow choice
• correct validation design
• correct metrics
• correct artifact persistence
• correct plot manifests
• correct state semantics
• correct billing semantics
• correct workflow-specific language in the exported report

If those elements stay aligned, the platform feels trustworthy. If they drift, even polished UI can mislead.

How To Use These Cases

Use these cases when:

• onboarding new users to the workflow logic
• reviewing a new service patch
• deciding whether a problem is cosmetic or structural
• writing regression tests for artifacts, SHAP, reporting, or billing
• checking whether an AI report is grounded in saved evidence
• deciding whether a workflow choice error is more serious than a model-performance issue

Deep References

• Running jobs
• Understanding results
• Explainability and AI reports
• Specialized services
• Reproducibility, audit trail, and regeneration

Source: docs/book/chapters/22-model-methodology-reference.md

Model Methodology Reference

This chapter is the book-level index for algebraic model definitions, optimization objectives, assumptions, and interpretation.

The per-model mathematical appendices provide model-identifier lookup tables and family-specific objectives. This chapter gives readers a common notation layer before those detailed appendices.

Notation

• $x_i \in \mathbb{R}^p$: feature vector for observation $i$
• $y_i$: target value or class label
• $X$: design matrix
• $\theta$: model parameters
• $\hat{y}_i$: prediction
• $L(\theta)$: loss/objective function

Linear Regression

Prediction:

$$ \hat{y}_i = \beta_0 + x_i^\top \beta $$

Ordinary least squares objective:

$$ \min_\beta \sum_{i=1}^{n}(y_i - \beta_0 - x_i^\top\beta)^2 $$

Regularized variants add penalties such as:

$$ \lambda ||\beta||_2^2 $$

for ridge regression [@Hoerl1970], or:

$$ \lambda ||\beta||_1 $$

for lasso [@Tibshirani1996].

Logistic Regression

Binary class probability:

$$ P(Y_i = 1 \mid x_i) = \sigma(\beta_0 + x_i^\top\beta) $$

where:

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

Cross-entropy objective:

$$ \min_\beta -\sum_i \left[y_i\log(p_i) + (1-y_i)\log(1-p_i)\right] + \lambda R(\beta) $$

Poisson Count Regression

Poisson regression belongs to the generalized linear model family and models count outcomes with equi-dispersed variance [@McCullaghNelder1989].

Mean count:

$$ \lambda_i = \exp(\beta_0 + x_i^\top\beta) $$

Poisson likelihood:

$$ P(Y_i=y_i)=\frac{e^{-\lambda_i}\lambda_i^{y_i}}{y_i!} $$

Negative Binomial Count Regression

Negative binomial models are useful when count outcomes are overdispersed relative to Poisson variance assumptions [@CameronTrivedi1998].

One common parameterization uses mean $\mu_i$ and dispersion $\alpha$:

$$ Var(Y_i \mid x_i) = \mu_i + \alpha\mu_i^2 $$

with:

$$ \mu_i = \exp(\beta_0 + x_i^\top\beta) $$

Naive Bayes

Naive Bayes applies Bayes' rule with a conditional-independence assumption:

$$ P(y \mid x_1,\ldots,x_p) \propto P(y)\prod_{j=1}^{p}P(x_j \mid y) $$

Gaussian Naive Bayes assumes continuous features are normally distributed within class. Multinomial Naive Bayes assumes non-negative count-like feature representations.

Support Vector Machines

Support vector machines find the maximum-margin separating hyperplane [@Cortes1995]. The hard-margin formulation solves:

$$ \min_{w,b}\frac{1}{2}||w||^2 $$

subject to:

$$ y_i(w^\top x_i + b) \geq 1 $$

The soft-margin form introduces slack variables:

$$ \min_{w,b,\xi}\frac{1}{2}||w||^2 + C\sum_i \xi_i $$

subject to:

$$ y_i(w^\top x_i + b) \geq 1-\xi_i,\quad \xi_i \geq 0 $$

Kernel SVMs replace inner products with a kernel function $K(x_i,x_j)$.

K-Nearest Neighbors

KNN predicts from nearby training examples. For classification:

$$ \hat{y}(x)=\operatorname{mode}{y_i: x_i \in N_k(x)} $$

For regression:

$$ \hat{f}(x)=\frac{1}{k}\sum_{x_i \in N_k(x)}y_i $$

where $N_k(x)$ is the set of $k$ nearest training observations under the chosen distance metric.

Decision Trees

A tree partitions feature space into regions $R_m$ and predicts:

$$ \hat{f}(x) = \sum_{m=1}^{M} c_m I(x \in R_m) $$

Splits are chosen to reduce impurity or loss [@Breiman1984].

Random Forests

For classification:

$$ \hat{y} = \operatorname{mode}{T_b(x)}_{b=1}^{B} $$

For regression:

$$ \hat{f}(x) = \frac{1}{B}\sum_{b=1}^{B}T_b(x) $$

Each tree $T_b$ is trained on a bootstrap sample with random feature subsampling [@Breiman2001].

Gradient Boosting

Boosting builds an additive model:

$$ F_M(x) = \sum_{m=1}^{M}\nu h_m(x) $$

where each weak learner $h_m$ is fit to improve the current objective.

For differentiable loss $L$, many gradient-boosting implementations fit each new learner to pseudo-residuals:

$$ r_{im} = -\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]{F=F} $$

The general gradient boosting framework is due to @Friedman2001. XGBoost [@Chen2016], LightGBM [@Ke2017], and CatBoost [@Prokhorenkova2018] are optimized implementations with different tree-growth, categorical-handling, and regularization strategies.

Multilayer Perceptron

Hidden layer:

$$ h = \phi(W_1x + b_1) $$

Output:

$$ \hat{y} = g(W_2h + b_2) $$

Classification uses cross-entropy; regression typically uses squared or absolute error.

Convolutional Neural Networks

Convolutional models learn filters over local structure. A one-dimensional convolution can be written as:

$$ z_t = \phi\left(b + \sum_{k=0}^{K-1}w_k x_{t+k}\right) $$

Convolutional architectures were originally developed for image recognition and document analysis [@LeCun1998]. For tabular CNN variants in this project, the encoded feature vector may be reshaped to a one-dimensional sequence before convolution. For image, audio, video, and time-series workflows, convolutional structure maps more naturally to pixels, spectrograms, frames, or temporal windows.

Recurrent Neural Networks

A basic recurrent update is:

$$ h_t = \phi(W_x x_t + W_h h_{t-1} + b) $$

LSTM and GRU variants add gates to control memory, forgetting, and update behavior [@HochreiterSchmidhuber1997]. These models are used for sequence and time-dependent workflows where ordering matters.

Transformer Models

Transformer models use multi-head self-attention to combine contextual information [@Vaswani2017]. Scaled dot-product attention is:

$$ \operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V $$

Text, time-series, image, video, and graph workflows may use transformer-style models when the implementation and dependencies are available.

Survival Models

Cox proportional hazards [@Cox1972]:

$$ h(t \mid x) = h_0(t)\exp(x^\top\beta) $$

Survival workflows require careful censoring and event-time interpretation.

Clustering

The k-means algorithm [@MacQueen1967] minimises within-cluster variance:

$$ \min_{{C_k}}\sum_{k=1}^{K}\sum_{x_i \in C_k}||x_i-\mu_k||^2 $$

Density and hierarchical methods use different assumptions and diagnostics.

Anomaly Detection

Anomaly detectors assign an unusualness score $s(x)$. A threshold $\tau$ converts scores to labels:

$$ \hat{y}(x)=I(s(x)>\tau) $$

Different anomaly models define $s(x)$ differently: isolation depth, local density, reconstruction error, robust covariance distance, or learned representation distance.

Recommendation And Ranking

Matrix factorization [@Koren2009] decomposes the user-item interaction matrix into user and item latent factors:

$$ \hat{r}_{ui}=p_u^\top q_i $$

Ranking objectives optimize pairwise or listwise ordering rather than point prediction.

Pairwise ranking commonly minimizes a loss over item pairs:

$$ \sum_{(i,j)} \ell(f(q,i)-f(q,j), y_{ij}) $$

where $q$ is a query or user context and $i,j$ are candidate items.

Graph And Network Models

Graph neural networks [@KipfWelling2017] update node representations by aggregating neighbourhood information:

$$ h_v^{(k)} = \sigma\left(W^{(k)} \cdot \operatorname{AGG}{h_u^{(k-1)}:u\in \mathcal{N}(v)\cup v}\right) $$

Different architectures vary the aggregation, attention, normalization, and positional encoding strategies.

SHAP And Surrogate Explainability

SHAP [@LundbergLee2017] expresses a prediction as an additive attribution model:

$$ g(z')=\phi_0+\sum_{j=1}^{p}\phi_j z'_j $$

where $\phi_j$ is the contribution assigned to feature $j$.

When exact SHAP is unavailable, this project may use surrogate feature-importance outputs based on:

• native tree importances
• linear coefficients
• MLP layer weights
• prediction-response permutation fallback for neural or otherwise unsupported models

These fallbacks should be labeled as surrogate explainability rather than exact SHAP values.

Implementation Mapping

For model-specific implementation details, see:

• Cross-sectional model mathematics
• Longitudinal model mathematics
• Time-series model mathematics
• Text model mathematics
• Media model mathematics
• Survival model mathematics
• Specialized model mathematics
• Advanced-systems model mathematics

References

Source: docs/book/chapters/23-metrics-and-validation.md

Metrics And Validation

Metrics should be interpreted in the context of the task, class balance, sample size, and deployment goal.

General Rule

No metric is meaningful without knowing the target, data split, class/event balance, and intended use. The platform can compute metrics, but users must decide which metric matches the decision they plan to make.

Classification

• Accuracy
• Precision
• Recall
• F1
• ROC-AUC
• PR-AUC
• confusion matrix
• threshold diagnostics

For imbalanced outcomes, PR-AUC, recall, precision, and threshold behavior often matter more than accuracy. The ROC-AUC summarises classifier performance across all decision thresholds [@HanleyMcNeil1982]; PR-AUC is preferable when positive events are rare [@DavisGoadrich2006].

Positive Class

For binary classification, the positive class is the event or outcome of interest. Selecting it tells Xalec which label should be treated as the "yes", "event", or "case" side when computing class-specific metrics and threshold diagnostics.

This matters because precision, recall, F1, ROC/PR curves, confusion-matrix interpretation, and threshold tuning all depend on which label is considered positive. If the positive class is reversed, a model can appear to have strong recall or precision for the wrong outcome, especially when the rare or costly event is the one the user actually cares about.

Thresholds

Many classification models output probabilities or scores. The threshold that converts a score into a class should be checked against the use case. A high AUC can still produce a poor default cutoff when prevalence is low or costs of false positives and false negatives differ.

For binary classification, Xalec stores a standardized best-model cutoff when probabilities are available. The current default selection rule maximizes Youden's J:

Youden's J = sensitivity + specificity - 1

This gives one operating threshold that balances sensitivity and specificity. The threshold-performance plot shows accuracy, precision, recall, specificity, F1, balanced accuracy, and Youden's J across the threshold grid, with the selected cutoff marked. Prediction workflows then reuse the saved cutoff so the reported confusion matrix, ROC operating point, AI report language, and later predicted classes are based on the same decision rule.

Users may still choose a different threshold for a deployment policy when the decision context has asymmetric costs. In that case, the report should state that the deployment threshold differs from the automated Youden cutoff.

Regression

• MAE
• MSE
• RMSE
• R-squared
• residual diagnostics
• actual-vs-predicted plots

Regression metrics should be interpreted on the target scale. A low RMSE may still be unacceptable if the decision requires small absolute error, and a strong R-squared can hide poor performance for rare high-value cases.

Count Regression

• MAE/RMSE-style error metrics
• deviance-style summaries where implemented
• mean-variance diagnostics
• zero-inflation checks

Count models should respect the non-negative, integer-like nature of the outcome. Over-dispersion and excess zeros often make ordinary regression a poor fit even when it appears to produce a numeric prediction.

Survival

• concordance index
• survival curves
• censoring-aware diagnostics
• event-time calibration where available

Survival validation must account for censoring and follow-up time. The concordance index (C-index) measures discriminative ability in the presence of censoring [@Harrell1982]. A risk score is not the same as a direct probability of an event unless the model and output are calibrated for that interpretation.

Clustering

• silhouette score
• cluster-size diagnostics
• dimensionality-reduction visualizations
• domain review of cluster meaning

Clustering metrics can suggest structure, but they do not prove the clusters are useful. The silhouette score measures how well each point fits its assigned cluster relative to neighbouring clusters [@Rousseeuw1987]. Domain review and cluster-profile interpretation are required.

Recommendation And Ranking

• hit rate
• NDCG
• MAP
• precision@k
• recall@k

Ranking metrics should be computed within query, user, or candidate groups. NDCG weights correctly ranked relevant items by their position, applying a logarithmic discount to lower ranks [@JarvelinKekalainen2002]. If the grouping is wrong, the metric can look strong while the ranking problem is not actually being evaluated.

Validation Split Patterns

Supplied Validation And Test Data

If users supply a split column or separate train, validation, and/or test files, metrics should be interpreted according to that contract. Training rows fit the models. Validation rows check choices such as tuning, threshold selection, or calibration when the service has that step; validation is not the final score. Test rows are final held-out evidence. For explicit train and validation uploads without an uploaded test file, AutoML creates an internal test holdout from the uploaded training file before fitting; reports should describe the uploaded validation metrics beside the internally generated test metrics and state that the test split was created internally. If a saved contract truly has validation rows but no test rows, validation is the evaluation split and reports should say so.

AI manuscript tables should avoid duplicating the main model-comparison table. The split-specific model-performance table is shown only when saved artifacts contain separate validation/test evidence that is not already represented by the primary Model Performance table. When an ordinary internal holdout is the only evaluation split, Table 1 is the single source of truth for model performance.

For artifact-backed review, look for dataset_split_contract.json and the saved dataset_split_policy in result metadata. Those files explain how many rows came from each role and which split was used for displayed metrics. Newer split contracts also include an output_contract block that records the primary evaluation role, whether a Validation Dataset section is expected, and whether validation visuals should be classification-style or service-specific. The job-detail page should surface uploaded validation evidence in a dedicated Validation Dataset section. Classification services should attach validation confusion-matrix, ROC, and precision-recall plots when those split-specific artifacts are saved. Regression, count, longitudinal, time-series, survival, ranking, recommendation, anomaly, and other services should use their own best-model validation performance plots rather than borrowing classification visuals.

Weighted Samples

Sampling weights change how training examples represent a larger population. They are not ordinary features. When weights are uploaded, the safest reading is model-specific: some estimators use weights directly, some fall back to unweighted fitting, and some services may only preserve the weight summary for audit. Reports should not claim a fully weighted analysis unless the saved model evidence shows the relevant model consumed the weights.

Model Drift And Distribution Shift

Model drift means a model becomes less reliable after training because the data, outcome process, or operating context changes relative to the data used for validation. A completed Xalec training job shows performance on the uploaded validation design; it does not prove that future production data will behave the same way.

Related terms should be kept separate:

• data or feature drift: the distribution of input predictors changes after training
• target or prevalence drift: the outcome rate or class balance changes over time
• concept drift: the relationship between predictors and the target changes
• prediction drift: the distribution of model scores or predicted classes changes after deployment

The practical response is monitoring, not blind trust. Compare future input distributions, prediction distributions, calibration, and performance metrics against the training or validation baseline. When true labels arrive later, recompute task metrics by time period, site, group, or segment. If drift is material, review the data-generating process, retrain or recalibrate when justified, and document the change before using the output for decisions.

For a specific job, drift cannot be diagnosed from the original training artifacts alone. The assistant needs post-training evidence such as saved prediction history, new labeled outcomes, monitoring summaries, or a later validation dataset before claiming that drift occurred.

Validation Caveats

• Cross-validation may be inappropriate when subject/time leakage is possible [@Kohavi1995].
• Random train/test splits can be misleading for temporal data.
• Metrics on extremely imbalanced data need threshold and prevalence context.
• Automated reports should not overstate causal or clinical conclusions.

Reporting Metrics Honestly

Reports should state the validation design, the selected metric, the best model, and any caveats that materially affect interpretation. When a metric is unavailable or unreliable, the report should say so rather than filling the gap with generic language.

Related Source Docs

• Responsible use, scope, and limitations
• Understanding results
• Model methodology reference
• Service reference overview

References

Source: docs/book/chapters/24-artifacts-and-implementation.md

Artifacts And Implementation

This chapter describes the major persisted outputs and implementation locations.

Artifact Contract

Artifacts are the durable evidence of a job. They should let a user or developer answer:

• what data and configuration were used
• which models ran, failed, or were skipped
• which model was selected
• which metrics and plots were produced
• what context was available to AI reports
• whether downloadable reports match the in-app preview
• which evidence the assistant can cite when diagnosing a job

Common Artifact Types

• serialized model files
• report.json
• job_results.json
• model/training_results.json
• model/best_model_evaluation.json
• model/optimal_cutoff.json for binary classification jobs with probabilities
• dataset_split_contract.json for user-supplied split and weight setup including output_contract, which records primary evaluation role, generated internal test status, and Validation Dataset expectations
• dataset_split_policy in service results or metadata where the service consumes explicit train/validation/test roles
• runtime_guardrails.json for service-family runtime estimates, disabled models, and interactive-budget warnings
• reproducibility_manifest.json for dataset fingerprints, split settings, model roster, runtime controls, package versions, and artifact inventory
• report_validation.json, prompt_package.json, manuscript_context.json, report_tables.json, and llm_telemetry.json for AI report quality review, prompt snapshots, manuscript context, table provenance, and provider telemetry
• model/validation_policy.json for service-specific validation semantics, especially longitudinal known-subject versus new-subject evidence
• held-out prediction artifacts for selected predictive models where available
• metadata.json
• plots
• SHAP outputs
• AI report outputs
• assistant feedback and staged improvement proposals
• logs and status files

Artifact Principles

• Saved artifacts should be generated from the real job payload, not generic defaults.
• model/training_results.json is allowed to appear before the outer pipeline has finished all visuals and report metadata. It can support active-job display and interrupted-job recovery, but active jobs should not be marked completed from that file alone.
• job_results.json and report.json are final pipeline evidence for model results, plot inventory, diagnostics, and report context.
• Regeneration should preserve the same job identity and update derived outputs consistently.
• Downloadable reports should render the same text, references, figures, and saved edits shown in the app.
• Validation metadata should warn when structured report context is missing or partially empty.

Common Stored Evidence

• the job manifest store
• the saved-results store
• the fitted-model evidence for a specific job
• the job plot inventory
• the optional SHAP explanation inventory

Model-evaluation plots should live under plots/, not in the EDA namespace. Regression and count-regression jobs commonly produce actual-versus-predicted, model-comparison, feature-importance, residual, residual-distribution or QQ, and training-time visuals. EDA feature-distribution panels should be registered as independent EDA plots so the EDA tab can display each feature separately without hiding model diagnostics.

Binary classification jobs that expose best-model probabilities should persist the same cutoff evidence used by plots, reports, and prediction. The key files are:

• model/best_model_evaluation.json: saved labels, predictions, probabilities, class names, best-model identity, and threshold metadata.
• model/optimal_cutoff.json: compact saved cutoff metadata, including the method, positive-class index and label, selected cutoff, threshold grid, and selected metrics.
• plots/threshold_performance.png: accuracy, precision, recall, specificity, F1, balanced accuracy, and Youden's J across thresholds, with the selected cutoff marked.
• plots/roc_curve.png: ROC curve, optionally annotated with the saved operating point.

When validation rows are saved separately, split-specific validation plots should use a recognizable role path or manifest entry, such as plots/validation/confusion_matrix.png or split_evaluation_visuals.validation. Classification-like services should save validation confusion-matrix, ROC, and precision-recall plots when possible. Other services should save the best-model validation diagnostics that match their own metric family.

When these artifacts exist, downstream prediction should reuse the saved cutoff instead of silently falling back to argmax or a default 0.5 threshold.

Deployment Refit Contract

For supervised workflows, the evaluation estimator and a production deployment estimator are separate artifacts. deployment_refit_contract.json records the validation-selected model, its selection metric and source, the permitted training roles (train and validation), and the locked test role that must never be included in a refit. It also records preprocessing, calibration, and threshold lineage.

The contract starts as pending_refit after valid selection evidence is saved. It becomes ready only when a service writes a distinct, versioned deployment artifact using train plus validation rows and preserves the immutable evaluation artifact. A saved evaluation model must not be described as a deployment refit until that transition is recorded.

For explicit split jobs, model-results plots and reports should prefer saved held-out prediction artifacts over a second prediction pass. This is especially important for longitudinal and time-series services because subject IDs, time order, known-series context, and validation mode affect what a prediction means.

Important Report Artifacts

AI-report workflows may produce files such as:

• manuscript HTML
• structured report metadata
• validation report JSON
• resolved references JSON
• manuscript context JSON
• editable declaration/completion state
• figure assets referenced by the manuscript

The route and downloadable report should agree on these files. If a report is regenerated for a legacy job, the workflow should rebuild context from the saved job payload and warn when important fields are missing.

Core Implementation Areas

• application entrypoint
• job routes
• service routes
• shared training pipeline
• service registry and service catalog
• AI-report generation
• assistant runtime and assistant routes
• assistant workspace and quality review surface

Implementation Traceability

For product behavior, the strongest source is the running FastAPI/Jinja app. Documentation should follow code when they disagree. For algorithm descriptions, use the detailed service methodology and then the relevant service modules, route handlers, pipeline functions, and tests.

For assistant behavior, the strongest runtime sources are the simplified RAG runtime under mlagent_website/assistant/simple_rag.py, the assistant routes, and the assistant evaluation fixtures and scripts. The handbook chapter 10-assistant-and-controlled-rag.md defines the user-facing evidence rules and safety expectations.

Pipeline source files should not be treated as ordinary assistant evidence. Each service should instead persist compact job-level contracts that describe what happened in that run. The assistant can then answer across services by reading the same artifact contracts used by the UI, reports, logs, and audit tools.

Traceability Rule

Every technical claim in the book should be traceable to:

• a service source file
• a route or pipeline implementation
• a saved artifact contract
• a test
• or a clearly marked design document

Source: docs/book/chapters/25-reproducibility-audit-trail-and-regeneration.md

Reproducibility, Audit Trail, And Regeneration

This chapter explains how the platform preserves evidence, how a result can be traced back to its saved state, and what regeneration is expected to do. Reproducibility in this product is not only about rerunning code. It is also about preserving the durable artifacts that explain what happened, what was selected, and what was later edited or regenerated.

The Core Principle

Every meaningful result should be traceable through a durable chain:

1. job creation and configuration
2. persisted job state
3. saved results and artifacts
4. optional add-on actions such as SHAP or AI report generation
5. validation metadata that records whether the derived outputs are complete and trustworthy

If one of those links is missing, the workflow may still be useful, but it is less reproducible and less auditable.

The Job ID Is The Anchor

The job_id is the primary handle that ties the workflow together. It connects:

• the job manifest
• the result directory
• model and plot artifacts
• billing records
• SHAP outputs
• manuscript/report outputs
• regeneration actions

In practice, the job ID is the shortest path to reconstructing what the platform knew and produced for a given run.

flowchart TD
    A[job_id] --> B[Job Manifest]
    A --> C[Result Directory]
    A --> D[Billing Records]
    B --> E[Status, Timestamps, Action State]
    C --> F[Models, Metrics, Plots]
    C --> G[SHAP Artifacts]
    C --> H[Manuscript, References, Validation]
    H --> I[Downloaded Report]
    H --> J[In-App Report View]

Figure: Job-centered audit trail linking manifests, result artifacts, billing records, SHAP outputs, and report outputs through one durable identifier.

The Two Durable Homes Per Job

The platform relies on two major durable locations.

1. Job Manifest

The job manifest under mlagent_website/jobs/ stores the persisted state used by the frontend and workflow monitors.

Typical content includes:

• service type
• timestamps and status
• progress/current stage
• result summaries
• action-level state for SHAP or AI report workflows
• billing summaries

2. Result Directory

The canonical result directory under mlagent_website/results/<job_id>/ stores the artifacts produced by training and post-processing.

Typical content includes:

• saved models and preprocessors
• metrics and comparison results
• plots and EDA summaries
• SHAP files and images
• report artifacts
• metadata, validation files, and logs

Together, the manifest and result directory form the audit trail.

What Needs To Be Reproducible

For this platform, reproducibility means more than "can retrain if needed." It means a reviewer can inspect:

• which service was used
• what the target and key columns were
• which models ran, failed, or were skipped
• which model was selected
• what metrics and plots were available
• what contextual information grounded the AI report
• whether a later regeneration changed only derived artifacts or changed the underlying training evidence

Important Artifact Classes

The most important artifact classes are:

• training results and comparison files
• EDA summaries and plot manifests
• model artifacts and preprocessors
• SHAP outputs and plot images
• report metadata and manuscript HTML
• reference manifests and validation JSON
• logs that explain failures, skips, and recovery behavior

The more these classes remain aligned, the more reproducible the workflow feels in practice.

Current Review Artifacts

The product now standardizes a smaller set of review artifacts that connect the upload flow, completed job page, AI Report tab, public demo workspace, and handbook.

These artifacts are allowed to be generated after the original training run when older jobs lack them. That kind of backfill improves auditability but does not create new analytical evidence. If a claim depends on model performance, training results, billing, SHAP, or predictions, the original saved job artifacts and ledger remain the authority.

Regeneration Versus Retraining

The platform should distinguish clearly between retraining and regeneration.

flowchart TD
    A[Need To Change Something] --> B{Underlying training evidence still valid?}
    B -- No --> C[Retrain]
    B -- Yes --> D[Regenerate]
    C --> E[New models, metrics, and artifacts]
    D --> F[Refresh report, SHAP, or validation outputs]
    E --> G[New analytical result]
    F --> H[Same analytical basis, newer derived outputs]

Figure: Decision split between retraining and regeneration. Retraining changes the analytical result; regeneration refreshes derived outputs from saved evidence.

Retraining

Retraining is appropriate when:

• the wrong workflow was selected
• the data or target were wrong
• the model options were wrong
• the training run itself produced incomplete or untrustworthy evidence

Retraining changes the underlying analytical result.

Regeneration

Regeneration is appropriate when:

• the underlying training result is still the right basis
• a report needs refreshed references or context
• SHAP outputs need rebuilding
• downloadable manuscript output drifted from the saved in-app state
• structured validation files need to be rebuilt from the saved job payload

Regeneration should refresh derived artifacts while preserving the identity and evidence of the original job.

Regeneration Must Use Real Saved Context

The product has already surfaced why this matters. Legacy or interrupted jobs can drift into generic output if regeneration is allowed to fall back to incomplete or placeholder context.

The durable rule should be:

• regenerate from the real saved job payload and artifact set
• warn explicitly when manuscript context or validation context is partial or missing
• preserve a clear trail of what was regenerated and why

If regeneration rebuilds a report from generic fallback context, the output may look polished while becoming less trustworthy.

Report Reproducibility

For AI-report workflows, reproducibility means that these layers agree:

• manuscript HTML shown in the app
• downloadable report output
• resolved references
• validation metadata
• saved completions or declarations
• figure assets referenced by the report

If the user edits declarations or completions inline, the saved manuscript and the downloadable output should stay synchronized. Otherwise the interface becomes visually convincing but operationally unreliable.

flowchart LR
    A[Saved Job Context] --> B[Structured Manuscript Context]
    B --> C[Rendered Manuscript HTML]
    C --> D[In-App Report View]
    C --> E[Downloaded DOCX and PDF]
    B --> F[Resolved References]
    B --> G[Validation Metadata]
    H[Saved Completions and Declarations] --> C
    F --> D
    G --> D

Figure: Report reproducibility contract. Edited completions, saved manuscript state, validation metadata, and downloaded outputs should remain aligned.

Audit Questions A Reviewer Should Be Able To Answer

For any completed job, a reviewer should be able to answer:

• what service and target were used?
• what data structure assumptions did the workflow make?
• which models succeeded, failed, or were skipped?
• why was the best model selected?
• what validation setup was used: internal split, split column, or supplied train/validation/test files?
• were sampling weights supplied, and did the selected model use them?
• what warnings or readiness checks were recorded?
• what add-on actions were run later?
• whether billing and final output state reflect the actual completed work?
• whether regenerated report outputs still match the original saved evidence?

Practical Audit Checklist

• Start with the job detail page, but verify against saved files when something looks off.
• Prefer final persisted training results over transient recovery fragments.
• Check whether plot manifests match actual plot files.
• Confirm SHAP and report outputs are tied to the correct job.
• Review validation files for missing or partial structured context.
• Distinguish retraining from regeneration when interpreting changes over time.

Deep References

• Architecture and execution flow
• EDA, figures, and reporting reliability
• Artifacts and implementation
• Runtime guardrails
• Troubleshooting

Source: docs/book/chapters/26-troubleshooting.md

Troubleshooting

This chapter collects common failure modes and recovery actions.

First Triage

Start by identifying where the problem appears:

• upload or schema validation
• job startup
• candidate model training
• artifact generation
• SHAP/explainability
• AI report generation
• billing/settlement
• route/template rendering
• local development environment

Then check the job detail page, result directory, logs, and any validation JSON produced by the workflow.

Training Fails

Check:

• missing required columns
• unsupported file format
• invalid target values
• insufficient rows
• dependency not installed
• efficiency guardrail skipped a model
• Windows file locks or OneDrive interference

If some models succeeded, treat the job as partially useful and inspect the successful model list before rerunning.

If a job flips from running to failed during a server reload, check whether the error says training was interrupted during shutdown or restart. That is a runtime cancellation, not necessarily a model-quality failure. Restart the services cleanly, confirm package imports and billing refund status, and then rerun the job when appropriate.

Live Monitor Appears Stuck

If the upload-page live log or progress bar appears stuck at an early value such as 5%, first check the saved job manifest under mlagent_website/jobs/<job_id>.json.

If the manifest shows later progress, current_step, last_heartbeat, or terminal status, then training continued and the issue is transport responsiveness rather than model execution. Check:

• whether /api/job-status/<job_id> is responding during training
• whether websocket reconnect warnings appear in the browser log
• whether the service is an async wrapper around synchronous CPU-heavy fits
• whether that service is included in the pipeline async-training offload policy

Survival, longitudinal, time-series, ranking, and media-classification workflows are expected to be offloaded so the FastAPI event loop can continue serving polling and websocket updates during long fits.

Keras Models Unavailable

Keras models require TensorFlow/Keras. If TensorFlow is not installed, Keras MLP and Keras CNN will be skipped or recorded as unavailable.

Native Longitudinal Backends Unavailable

Longitudinal historical-tree and mixed-effects models depend on local editable packages:

• pyhtree for HTB, HRF, and HCB
• longiturf for MERT, MERF, MECB, REEMTree, REEMForest, and REEMCatBoost

From the same virtual environment used to run the app, verify:

python -c "from pyhtree import hrf, htb; from pyhtree.hcb import hcb; from longiturf import MERT, MECB; print('ok')"

If pyhtree imports as an unknown-location namespace package or lacks hrf, the editable install is incomplete. If longiturf works only from the repo root but not from mlagent_website/, the editable path is pointing at the wrong package directory.

SHAP Fails

SHAP may fail because:

• the model is not supported by the native SHAP explainer
• the model lacks tree or linear importance attributes
• categorical preprocessing cannot be reconstructed
• the process times out
• output files are blocked by filesystem locks

Fallback behavior may generate surrogate SHAP-style outputs when native SHAP is unavailable.

AI Report Looks Generic

Check whether the report context resolver detected the correct domain and whether manuscript_context.json was populated from the real saved job payload. Generic titles, irrelevant references, duplicated References sections, or empty structured context usually point to a context-loading or post-processing problem.

Report Figures Do Not Display

Check whether the report is opened inside the app or as a local downloaded file. Downloaded reports may require inlined assets or absolute /results/<job_id>/... paths.

Billing Looks Confusing

Look for reserve, refund, and final net charge. A zero-net add-on can show both a charge and refund in the ledger.

Local Python Or Git Problems

Common local issues include:

• virtual environments pointing to a removed Python install
• uvicorn.exe launchers from the wrong environment
• OneDrive cloud placeholders corrupting .git/index
• generated runtime logs cluttering git status
• locked result files from a still-running Python process
• interrupted uv sync runs leaving .venv partially pruned
• local editable packages pointing to the wrong source directory

Use python -m uvicorn ... from the intended virtual environment and keep the repository hydrated locally when working inside OneDrive.

For this repository, treat the root pyproject.toml as local package metadata for longiturf and related local sources, not as a complete manifest for every FastAPI runtime dependency. Running uv sync can remove packages the web app still needs if the full application dependency set is not represented. After an interrupted sync or failed startup, run:

uv sync --inexact
uv pip install -r mlagent_website\requirements-deploy.txt
uv pip install -r auth_payment_service\requirements.txt

For normal local startup, prefer the repo script because it checks imports and repairs the selected environment before launching uvicorn:

.\start-dev.ps1
.\start-dev.ps1 -MainOnly
.\start-dev.ps1 -AuthOnly

Then validate:

python -m pip check
python -c "import main; print('main import ok')"

from mlagent_website/, and:

python -c "import app.main; print('auth app import ok')"

from auth_payment_service/.

Recovery Pattern

1. Preserve user-authored work before cleanup.
2. Separate generated/runtime files from source changes.
3. Reproduce the failure on the smallest route, job, or test.
4. Patch the durable workflow, not only the displayed page.
5. Regenerate affected artifacts.
6. Re-review the app surface and downloaded output.

Related Source Docs

• Understanding results
• Explainability and AI reports
• EDA, figures, and reporting reliability
• Specialized services

Source: docs/book/chapters/27-runtime-guardrails.md

Runtime Guardrails

Runtime guardrails are the checks Xalec applies before and during training to keep an interactive job within a practical analysis budget. They are not a judgment that the research question is invalid. They mean the selected data size, model family, preset, or local dependency state would make one or more requested models too expensive, too slow, or too unreliable for the current run.

Guardrails should be interpreted alongside the saved job artifacts. For a specific job, use the request metadata, readiness output, training_results.json, skipped-model entries, and log snippets before stating which rule fired.

The guiding principle is that guardrails should guide rather than block. Most guardrails should warn, explain the likely runtime or billing consequence, and suggest a practical next step. A hard stop should be used only when the job is very likely to fail, exhaust memory or local compute resources, produce unreliable execution behavior, or create an unfair billing/runtime experience for the user.

Guardrail Types

Xalec uses four common guardrail families:

• Readiness guardrails check whether the uploaded data has the required role columns, usable target values, compatible feature columns, and enough non-missing observations.
• Plan and billing guardrails check whether the workflow is allowed for the current plan and whether the estimated cost requires confirmation.
• Runtime budget guardrails check whether the selected model configuration is likely to fit within an interactive job budget.
• Dependency and availability guardrails check whether a model backend is available in the current environment.

Shared Service Runtime Guardrail Contract

All service families expose the same runtime-guardrail evidence contract. The source module is mlagent_website/utils/service_runtime_guardrails.py. Upload readiness preview, backend training, job logs, saved report artifacts, AI report generation, and assistant/RAG answers should all read the same service_runtime_guardrails payload rather than recomputing separate estimates.

The shared payload includes:

• the guardrail registry version,
• the service family,
• the selected preset,
• the workload dimensions used for scoring,
• the family-specific score formula,
• the estimated fit score and interactive budget,
• the severity/status,
• model-level fit estimates,
• disabled or warning model lists,
• warnings and recommended next actions.

For completed jobs, this evidence may appear in runtime_guardrails.json, request.json, model/training_results.json, and report.json. When these sources disagree, use the completed job artifact that was written closest to training execution, usually runtime_guardrails.json or model/training_results.json.

The current service-family formulas are:

• Tabular services: rows x features x selected model count x confidence-interval/bootstrap multiplier x SHAP multiplier x model complexity.
• Anomaly detection: rows x features x selected model count x deep/GAN/kernel-model complexity.
• Time-series services: rows x features x selected model count x forecast horizon x lookback/window count x series count x deep-model complexity.
• Survival services: rows x features x selected model count x Cox, random-survival-forest, or DeepSurv complexity.
• Text and media services: file/item count x size or duration x embedding/deep-model complexity.
• Ranking, recommendation, network, and graphical-model services: entities x edges/interactions x features x model-family complexity.
• Clustering: rows x features x selected model count x clustering-model complexity.

These scores are advisory workload estimates for interactive execution. They help users start with a practical model set and then expand once the dataset, split strategy, and candidate model family are understood. They are not a scientific judgment about whether the study question is worth modeling. When the estimate is elevated but still feasible, the preferred behavior is a warning, an explanation, and a lower-cost path. Blocking should be reserved for cases where continued execution would probably fail or would be unreasonable for the user to pay for or wait on.

Longitudinal Runtime Budgets

Longitudinal services can be computationally heavier than ordinary cross-sectional services because the same subject, unit, patient, facility, device, or cohort member can appear multiple times. The service must preserve the repeated-measures structure rather than treating every row as independent.

Longitudinal services use a specialized model-cost registry underneath the shared service_runtime_guardrails contract so the upload readiness preview, backend training, job logs, AI report generator, and assistant all describe the same runtime contract. Every longitudinal model is routed through this registry, but the score is model-specific rather than a blunt row-count cap.

For each longitudinal model, Xalec estimates an interactive fit score from:

• the number of training rows,
• the configured iteration count,
• the configured tree count or depth setting,
• and a model-family multiplier when the method is known to be more expensive.

If the estimated score exceeds the preset's runtime budget, the affected high-cost model can be downshifted, disabled in readiness, or skipped before fitting. This is a controlled guardrail decision, not a model crash. Lower-cost fixed-effects, GEE, GLMM, GPBoost-style, and other baseline models are still scored through the registry, but they usually remain inside the same budget because their model-specific cost profiles are lighter.

For the shared longitudinal upload experience, the readiness-time interactive guardrail is currently 500,000 in the default balanced posture. This value is intended for a high-end development workstation or a first production server with comparable resources, such as roughly 64 GB RAM, many CPU threads, adequate free disk, and controlled job concurrency. Faster presets are stricter, while broader accuracy-oriented or reference presets can permit more work. The saved job artifacts include the runtime-guardrail registry version, selected preset, fit-score budget, and budgeted model list when a longitudinal service uses this contract.

The balanced guardrail should evolve with measured infrastructure rather than being raised blindly. Future server upgrades should update the guardrail registry through a versioned policy that considers hardware and concurrency:

effective_budget =
  base_budget
  x hardware_factor
  x concurrency_factor
  x service_family_factor
  x preset_factor

As a practical starting policy, a 64 GB RAM / 32-thread machine with a single active heavy longitudinal job can use a balanced budget of about 500,000. A 128 GB RAM server with more CPU threads and an explicit worker queue may justify a balanced budget around 750,000 to 900,000. Larger servers can raise the cap further, but only after saved telemetry shows that runtime, memory, queue delay, and billing remain reasonable. Multi-user servers should usually scale throughput with queues and workers before raising every per-job cap.

Every guardrail increase should be accompanied by saved evidence: estimated fit score, actual runtime, peak memory when available, skipped models, selected preset, and the runtime-guardrail registry version. After enough successful jobs, update the registry version and document the new default.

Affected Longitudinal Model Families

The guardrails most often apply to advanced longitudinal model families such as HTB, HRF, HCB, MECB, MERF, REEM tree, REEM forest, and REEM CatBoost. These methods can be useful for repeated-measures data, but they are also more sensitive to row count, subject count, tree count, depth, and iteration settings.

Fixed-effects, GEE, GLMM, GPBoost-style, and simpler baseline models may still train successfully in the same job when an advanced tree ensemble is skipped. A completed job with skipped models should therefore be read as a partial model set, not automatically as a failed analysis.

Presets And Interpretation

Training presets change the practical runtime budget. A faster or lower-cost preset should be expected to skip more expensive model configurations sooner. A broader accuracy-oriented preset permits more work, but it still has guardrails so that an interactive job does not run indefinitely.

When a model is skipped, the job evidence should say which model was skipped and why. The clearest evidence usually appears in saved training results, failed/skipped model metadata, and logs with text such as skipped before fit, runtime budget, interactive guardrail, or an estimated fit score.

HTB, HRF, And Local Backends

If a message says HTB or HRF is not available in the environment, interpret that as a backend availability guardrail. It means the local historical-tree implementation, bridge, or required dependency was not available to that service at run time. It is different from a runtime-budget skip and different from a fitted model performing poorly.

How To Ask The Assistant

Good guardrail questions include:

• what runtime guardrails apply to longitudinal services?
• why was reem_catboost skipped in job_...?
• did HTB fail or was it unavailable in job_...?
• which models were skipped by the interactive budget?

For general guardrail questions, the assistant should cite this handbook chapter. For job-specific guardrail questions, it should also cite the saved job artifacts and logs.

Source: docs/book/chapters/28-admin-developer-appendix.md

Admin And Developer Appendix

This appendix supports administrators and contributors.

Canonical App

The canonical app consists of:

• the FastAPI application entrypoint
• FastAPI routes for product pages and APIs
• shared Jinja templates
• shared CSS and JavaScript assets

Secondary Surfaces

Other interfaces or scripts may exist in the repository, but product UX changes should default to the FastAPI/Jinja app unless a documented architecture decision says otherwise. This prevents drift between multiple partial frontends.

Development Rules

• Favor shared templates and partials.
• Keep service metadata configuration-driven.
• Update docs when behavior changes.
• Avoid introducing parallel frontends unless explicitly planned.
• Keep runtime/generated artifacts out of git.

Commit Hygiene

Before committing:

• inspect git status --short
• separate source/docs changes from generated logs and results
• avoid staging .env, local virtual environments, temporary repro files, and runtime artifacts
• prefer docs-only commits when the change is documentation-only
• run the smallest relevant tests or render checks

CI Gate Strategy

The GitHub workflow keeps the core contract job on push so ordinary commits still get fast feedback. Heavier gates such as SHAP validation, AI report validation, Assistant RAG safety, and UI validation are intended for pull requests or manual workflow_dispatch runs when dependency, browser, or model runtime noise would otherwise produce frequent email failures on every push.

If notification noise returns, do not delete the gates blindly. First check whether the failing gate is required for the current change. If it is a heavy validation gate, run it manually or on a pull request. If it is the core contract gate, treat it as a release-blocking regression until reviewed.

Book Maintenance

The book source lives under docs/book/. The detailed service references remain under docs/services/. When a service workflow changes, update both the source service documentation and the relevant book chapter or appendix.

Quarto configuration is in docs/book/_quarto.yml. If chapters are added or renamed, update the chapter list and rerun a render check.

Synthetic Validation Data

The repository includes a developer-only synthetic dataset generator under hypothetical_data_generator/. It can regenerate safe validation datasets for upload, readiness, training, reporting, and assistant-evaluation checks without using private uploaded data.

The generator is documented in the appendix docs/book/appendices/hypothetical-data-generator.md. Treat it as an internal validation pipeline, not as a source of production model evidence.

The current generator writes one folder per canonical service type under hypothetical_data_generator/datasets/. Structured services are exported across the tabular/statistical formats accepted by the shared dataset reader when the local writer dependencies are available; each folder records written and skipped formats in dataset_manifest.json. Media services stay separate and use manifest-based fixtures (labels.csv) rather than statistical table exports.

Public Demo System

The public demo workspace is an onboarding system, not a compute system. It is served from:

• /demo
• /demo/{demo_id}
• /demo/{demo_id}/upload
• /demo/jobs/detail/{demo_id}
• /demo/{demo_id}/predict
• /demo/{demo_id}/billing

The primary source is three allowlisted completed jobs with saved AI report artifacts under mlagent_website/jobs/ and mlagent_website/results/, loaded through mlagent_website/services/demo_workspace.py and routed by mlagent_website/routes/demo_routes.py. The legacy mlagent_website/demo/demo_manifest.json remains only as a fallback when a matching completed source job is not available.

Demo completed-job pages reuse mlagent_website/templates/job_status.html with demo_mode=True. The demo can show upload setup, readiness, model metrics, saved plots from the mapped completed job, generated-report summary, prediction replay, and billing replay. It must not call training, SHAP, LLM report-generation, prediction, billing mutation, or job-admin mutation APIs.

Public demo plot requests under /results/demo-*/plots/* are routed through the demo alias. They serve the mapped completed job's plot when present and use a generated SVG fallback only when the saved file is missing.

Admin Jobs Workspace

The administrative route is /admin/jobs, backed by /api/admin/jobs and /api/admin/jobs/{job_id}/cancel. It is separate from the user /jobs/workspace route because administrators need cross-user visibility and stalled-job controls.

Admin identity:

• auth_payment_service.users.is_superuser=true is exposed by /users/me as is_superuser=true, is_admin=true, and roles=["user", "admin"]
• the web app accepts either is_admin=true or an admin-like role/scope
• selected accounts can be promoted through POST /users/admin/promote on the auth service with the dedicated X-Admin-Promotion-Secret header
• ADMIN_PROMOTION_SECRET must be separate from INTERNAL_REFUND_SECRET; leave it unset to disable the promotion endpoint

Implemented behavior:

• require an authenticated user with an admin claim
• list active, queued, stalled, failed, recently completed, and recently cancelled jobs across all users
• show owner, service, status, progress, current step, created time, last event, runtime, worker/process metadata where available, and billing/refund/ settlement state
• provide filters for owner, service, status, and stalled-heartbeat threshold
• mark a job as stalled when it is active but has exceeded a heartbeat or progress-update threshold
• require a reason before cancellation
• call the same cancellation/finalization path used by user-owned jobs, but with admin authorization
• trigger refund/settlement reconciliation after cancellation
• write an audit event with admin identity, reason, prior status, final status, and billing outcome

This route should not replace the owner-facing job workspace. It should be an operations console for support, recovery, and incident response.

Local Package And Environment Notes

The repository contains local package sources that are used by the canonical FastAPI app:

• root package metadata for longiturf
• pyhtree/ as a local editable package
• the canonical application package

The root pyproject.toml is therefore not the whole web-application dependency contract. Before running uv sync, confirm that all FastAPI/runtime dependencies used by mlagent_website/ and auth_payment_service/ are represented, or use a repair path rather than allowing sync to prune the environment. For the shared local .venv, prefer:

uv sync --inexact
uv pip install -r mlagent_website\requirements-deploy.txt
uv pip install -r auth_payment_service\requirements.txt

The local startup helper performs these checks before launching:

.\start-dev.ps1

A clean local startup check should include:

python -m pip check

from the repository root, then import checks from the two app directories:

python -c "import main; print('main import ok')"
python -c "import app.main; print('auth app import ok')"

When validating longitudinal native backends, also check:

python -c "from pyhtree import hrf, htb; from pyhtree.hcb import hcb; from longiturf import MERT, MECB; print('longitudinal backends ok')"

Billing Refund Secret

Training, SHAP, report, and batch-prediction refunds are internal service-to-service calls from the web app to the auth/billing service. The INTERNAL_REFUND_SECRET value must match in:

• the repository-level environment configuration
• the web-application environment configuration
• the authentication and billing-service environment configuration

If the web app and auth service disagree, failed-job refund attempts will be rejected with 401 Invalid internal refund secret. After correcting the auth service .env, restart the auth/billing service before retrying refund reconciliation.

Billing Configuration Notes

Keep pricing configuration aligned with the product's simple story:

1. Subscription gives access and an included credit allowance.
2. Top-ups buy extra compute.
3. Jobs and add-ons consume credits through estimate, reserve, settlement, and refund.

Registration-time grants should use TRIAL_CREDIT only when a deliberate trial or promotion is enabled. INITIAL_TOKEN_CREDIT remains a backward-compatible local/dev alias and should stay 0.00 in production. Payment-provider credits should be recorded as top-up purchases, not as subscription allowances.

Useful Further Reading

• Reading paths by role
• Product overview
• Service reference overview
• Architecture and execution flow
• Artifacts and implementation
• Runtime guardrails

Testing

Use the smallest relevant tests first and add deterministic tests for route or workflow regressions.

For documentation changes, useful checks include:

• validating that Quarto chapter references exist
• rendering HTML locally when Quarto is installed
• checking links to restored service workflow files
• confirming that the 20-service checklist remains consistent with the service reference overview

Source: docs/book/chapters/29-glossary.md

Glossary

AutoML

Automated machine-learning workflows that handle preprocessing, model training, comparison, evaluation, and artifact generation.

Artifact

A saved output from a job, such as a model file, plot, metadata file, report, or SHAP output.

Cross-Sectional Data

Data where each row is treated as one independent observation.

Longitudinal Data

Data containing repeated observations per subject or entity over time.

Time Series

Data indexed by ordered time, usually with temporal dependence.

SHAP

SHapley Additive exPlanations, a family of feature-attribution methods for interpreting model predictions.

PR-AUC

Area under the precision-recall curve. Often useful for imbalanced classification.

ROC-AUC

Area under the receiver operating characteristic curve.

Leakage

A modeling error where predictors contain information that would not be available at prediction time.

Censoring

In survival analysis, incomplete observation of the event time.

Dataset Split Contract

The saved record of how a job divided data for evaluation. It can describe an internal split, a user-supplied split column, or separate train/validation/test files, plus row counts and optional sampling-weight metadata.

Settlement

The billing step where a reserved charge is converted into a final net charge or refund.

Add-On

An optional job action such as SHAP analysis, AI report generation, report regeneration, or prediction-at-scale.

AI Report

An LLM-assisted report generated from saved job artifacts, metrics, context, figures, and references. It should be reviewed by a human before use.

Assistant

The in-app Xalec copilot that answers questions using the simplified controlled RAG runtime over the handbook, service metadata, and saved job artifacts.

Best Model

The model selected by the workflow according to the configured service objective, metrics, and guardrails.

Candidate Model

A model attempted during training and included in the comparison as successful, skipped, unavailable, or failed.

Count Regression

Regression for non-negative integer-like outcomes such as event counts, visits, cases, or incidents.

Feature

A predictor variable used by a model.

Guardrail

A rule that prevents a model or action from running when it is unavailable, unsafe, too expensive, incompatible, or unlikely to produce a valid result.

Controlled RAG

A retrieval-augmented generation pattern where allowed sources, evidence requirements, answer shape, citations, and unsupported-claim checks are constrained before an answer is returned. In the current Xalec runtime, this is a simplified handbook-first RAG rather than an autonomous agentic tool loop.

Manuscript Context

Structured job and study metadata used to ground an AI manuscript/report.

Net Charge

The final amount charged after reservation, settlement, and refund events are reconciled.

Positive Class

The class treated as the event or outcome of interest in binary classification.

Positive-class selection controls the direction of class-specific metrics, threshold diagnostics, ROC/PR interpretation, and confusion-matrix labels.

Regeneration

Rebuilding derived artifacts such as SHAP outputs or AI reports from an existing saved job.

Semantic Recall

An optional retrieval layer that uses embeddings to find relevant handbook chunks. In Xalec, semantic recall is a helper for retrieval, not a replacement for controlled source rules and answer verification.

Sampling Weight

A non-negative value that says how much a row represents in a sampled dataset. It is study-design information, not a predictor. Models use it only when their fitting backend supports weighted training.

Service

A product workflow for a particular data shape and objective, such as tabular classification, survival analysis, or network analysis.

Target

The outcome column or label the workflow is trying to predict, classify, explain, or model.

Validation Split

The way data are divided or resampled to estimate model performance on data not used for fitting.

Validation Dataset Section

The job-detail section shown when a saved split contract contains validation rows. It keeps validation metrics and validation-specific best-model visuals separate from the main Model Performance table so model-checking evidence is not confused with final test evidence.

Internally Generated Test Split

A final test split created by AutoML from the uploaded training file when a user uploads train and validation files but does not upload a test file. The uploaded validation file remains validation evidence, while the generated test split becomes the primary final performance evidence.

Canonical Acronym Index

This generated index is authoritative for product, metric, and configured-model acronyms. Xalec Assistant retrieves these definitions before broader handbook search.

Source: docs/book/chapters/30-publication-notes.md

Publication Notes

This chapter records publication-facing metadata and use conditions for the Xalec AutoML Services Handbook. It is intended for readers, reviewers, libraries, partners, and publishing teams who need to understand the status and permitted use of the handbook as a reference work.

Edition And Status

This edition describes Xalec AutoML Services as documented in the handbook source at the time of rendering. The handbook is maintained as a living technical and user reference, so the current source Markdown is the content of record.

Generated HTML, PDF, and DOCX versions are publication outputs. They should be treated as editioned snapshots rather than the editing source. If a generated export differs from the source Markdown, the source Markdown takes priority.

The handbook covers active platform workflows, user-facing outputs, methodology references, billing concepts, explainability behavior, AI report behavior, and administrative notes. It does not certify that every deployment environment, dependency, model runtime, or third-party integration is identical to the editioned snapshot.

Suggested Citation

For internal reports, reviews, and technical appendices, cite the handbook as:

Xalec. Xalec AutoML Services Handbook: User Guide, Service Reference, Mathematical Methods, and Implementation Notes. Xalec AI, 2026.

When citing a specific model, workflow, or methodology section, include the chapter or appendix title and the edition date shown in the rendered handbook.

Rights And Permissions

Copyright © 2026 Xalec AI. All rights reserved unless a separate written license or publishing agreement states otherwise.

The handbook may reference open-source libraries, statistical methods, machine learning algorithms, and third-party services. Those references do not transfer third-party rights, licenses, trademarks, warranties, or permissions into this handbook. Users and publishers should respect the licenses and terms governing the referenced software and services.

Permission requests for reproduction, translation, adaptation, commercial distribution, or publisher-hosted editions should be directed to Xalec AI or the designated rights holder for the edition being used.

Scope Of Use

This handbook is a product and methodology reference for Xalec AutoML Services. It is appropriate for:

• onboarding users to the platform;
• selecting service workflows;
• preparing datasets and interpreting readiness feedback;
• reviewing metrics, plots, explainability outputs, reports, and artifacts;
• understanding billing, credits, reservations, settlement, and refunds;
• auditing model methodology at a high level.

The handbook is not a substitute for professional statistical review, legal review, clinical review, financial advice, regulatory approval, institutional review, or publication peer review. Results generated by the platform should be treated as decision-support evidence that requires human review in the relevant domain context.

Editorial Conventions

The handbook uses consistent terminology across workflows:

• service means a product workflow for a specific data shape and objective;
• job means a saved training or analysis run with durable artifacts;
• readiness means pre-training evidence about whether the uploaded data can support the requested workflow;
• artifact means saved evidence such as metrics, plots, logs, reports, manifests, model files, and billing records;
• AI report means a reviewable draft grounded in saved job evidence;
• assistant means the controlled handbook and job-artifact reader, not a general autonomous analyst.

Equations are typeset in display math where possible. Model sections should state the plain meaning, mathematical or probabilistic formulation, optimization objective or estimator, assumptions, and interpretation guidance when these are applicable.

Disclaimer

Xalec AutoML Services can assist with machine-learning workflow selection, model training, evaluation, explainability, prediction, reporting, and audit trails. The platform and handbook do not guarantee that a model is correct, fair, causal, clinically valid, legally compliant, production-ready, or appropriate for a particular decision.

Users remain responsible for checking data provenance, consent, governance, privacy, security, statistical validity, domain assumptions, downstream use, and any regulatory or institutional requirements that apply to their work.

The handbook may describe billing concepts, runtime behavior, dependencies, and third-party integrations. Actual deployed behavior may vary by environment, configuration, plan, available dependencies, account status, and service provider availability.

Source: docs/book/chapters/31-publisher-readiness-and-editorial-audit.md

Publisher Readiness And Editorial Audit

This chapter defines the editorial audit used before a handbook edition is released for external readers, partners, institutional review, or publisher consideration. It complements the Publication Style Sheet appendix by turning style rules into practical review checkpoints.

Editorial Readiness Standard

A publisher-ready edition should read as one coherent reference work. It should not feel like a collection of implementation notes, issue reports, or internal engineering fragments.

Before release, the edition should satisfy five standards:

• the product narrative is consistent across overview, workflow, service, and methodology chapters;
• user-facing concepts are explained before implementation details;
• formulas and objectives are readable in HTML, PDF, and DOCX outputs;
• service counts, model counts, billing behavior, and workflow descriptions match the active platform;
• limitations, scope, rights, and review responsibilities are explicit.

Content Audit

The content audit checks whether the handbook answers the reader's practical questions in a clear order:

• What does the platform do?
• Who should use it?
• Which workflow should a user choose?
• What data does each workflow require?
• What happens during readiness preview, training, explainability, reporting, prediction, and billing?
• How should outputs be reviewed before use?
• What evidence is saved for audit and reproducibility?

Any chapter that relies on internal repository language, unresolved TODOs, local development paths, or duplicate product narratives should be revised before publication.

Methodology And Formula Audit

Every active service family should have a methodology path from service description to model mathematics. Where a model has a standard objective, estimator, likelihood, split criterion, score, distance, ranking loss, or diagnostic statistic, the handbook should name it.

The formula audit checks that:

• formulas render as math, not stacked plain text;
• each model or model family explains the quantity being estimated or optimized;
• count, classification, regression, ranking, clustering, survival, time-series, media, network, recommendation, anomaly, and graphical-model workflows use service-appropriate metrics;
• model limitations and assumptions are stated without overstating certainty;
• planned or conditional behavior is not presented as guaranteed active behavior.

If a model is implementation-specific or does not have one clean textbook objective, the handbook should explain the estimator or decision rule rather than forcing a misleading optimization equation.

Figures, Tables, And Accessibility Audit

Publication outputs should support both screen reading and print review.

The figure and table audit checks that:

• important figures have meaningful captions or surrounding explanation;
• plots are described as evidence, not conclusions;
• tables fit in PDF and DOCX layouts where possible;
• image assets render in HTML, PDF, and DOCX builds;
• model comparison, readiness, billing, and audit tables use consistent terminology;
• decorative assets do not obscure the reading order.

Where a plot or metric is workflow-specific, the handbook should explain why it matters for that service rather than implying that all workflows produce the same evidence.

Evidence And Traceability Audit

The handbook should make clear which claims are product behavior, methodology background, billing policy, or operational guidance.

The traceability audit checks that:

• service descriptions align with the active service catalog;
• upload and data-format guidance matches the shared dataset reader and media manifest workflows;
• assistant and AI report descriptions match their controlled, evidence-grounded role;
• billing chapters describe credits, reservations, settlement, refunds, and ledger evidence consistently;
• job-artifact claims use saved evidence rather than broad inference;
• generated handbook exports are treated as outputs rather than source files.

Editorial Risk Review

The editorial risk review identifies areas that require careful human approval before public distribution:

• medical, clinical, public-health, financial, legal, or regulated use cases;
• claims about fairness, causality, compliance, deployment readiness, or scientific validity;
• pricing, refund, rights, and permission language;
• third-party trademarks, library references, and service-provider descriptions;
• AI-generated report language that might be mistaken for peer-reviewed findings.

High-risk language should be conservative, source-grounded, and explicit about human review.

Release Checklist

Before publishing an edition:

1. Confirm the source Markdown is the source of truth.
2. Run model-objective and publication-readiness checks.
3. Render HTML, PDF, and DOCX from Quarto.
4. Open the rendered HTML and confirm the book structure is intact.
5. Spot-check PDF math, figures, tables, and page breaks.
6. Spot-check DOCX headings, equations, lists, and tables.
7. Confirm service counts and model catalog summaries match the active app.
8. Confirm Publication Notes, rights language, scope, and disclaimer are present.
9. Confirm the edition date and citation guidance are current.
10. Record any known limitations that remain before external release.

Readiness Classification

Use these labels during editorial review:

• Internal draft: suitable for product development and review, but not for external publication.
• External review draft: coherent enough for selected reviewers, but still subject to editorial and legal review.
• Publisher review candidate: complete enough to submit for editorial, design, rights, and production review.
• Published edition: approved, rendered, archived, and distributed under the applicable rights and permissions.

The label should be assigned to the edition as a whole. Individual chapters may have different maturity levels, but the public edition should carry the most conservative classification that applies.

Source: docs/book/appendices/service-workflow-checklist.md

Service Workflow Checklist

This appendix mirrors the restored 20-service coverage checklist.

The operational checklist is maintained with the service workflow documentation.

Live-Monitor Runtime Guardrail

For service workflows whose async training entry points wrap long synchronous model fits, the shared pipeline must offload training from the FastAPI event loop. This guardrail currently applies to:

• longitudinal services
• time-series services
• ranking_models
• survival
• media classification services

This keeps websocket updates, polling fallback, auth-status checks, and progress bar tracking responsive while model training continues.

Dataset Evaluation Contract Coverage

The shared explicit-split contract applies to supervised and evaluation-oriented workflows where held-out rows have a clear meaning. Tabular, text, longitudinal, time-series, survival, ranking, and recommendation services consume split roles directly. Media classification consumes split roles through the label manifest. Labeled anomaly detection can evaluate on supplied validation/test rows; unlabeled anomaly detection remains unsupervised.

Clustering, graphical models, and network analysis should use their own validation contracts: stability and profile review for clustering, dependency stability or held-out likelihood for graphical models, and graph-aware node/edge/phase validation for network analysis.

Source: docs/book/appendices/workflow-selection-matrix.md

Workflow Selection Matrix

This appendix gives a one-page decision aid for choosing the right service before training. It is meant to complement the narrative workflow-choice chapter with a faster lookup table.

Fast Selection Questions

Use these questions in order:

1. Is the data tabular, repeated over time, purely time-ordered, text, media, graph-structured, or interaction-based?
2. Is the goal prediction, grouping, ranking, anomaly detection, recommendation, or time-to-event modeling?
3. Does the target exist already, and if so is it categorical, continuous, count, event, or ranking relevance?
4. Are repeated subject structure or temporal order part of the truth, not just extra columns?

Service Matrix

Practical Choice Heuristics

• If repeated subjects appear in the data, prefer longitudinal over ordinary tabular unless you have a strong reason not to.
• If time order changes what counts as leakage or what should be predicted next, prefer time-series services over ordinary tabular ones.
• If there is no target and the goal is discovery, start with clustering or anomaly detection rather than forcing supervised training.
• If censoring matters, use survival rather than recoding the problem into a simpler target and losing the survival structure.
• If user-item interaction is the core data object, use recommendation instead of general tabular modeling.
• If network phases or waves are the analytic question, map the phase column explicitly instead of treating it as an ordinary feature.
• If edges are inferred among variables rather than observed between actors/nodes, use Graphical Models instead of Network Analysis.

Red Flags That Usually Mean The Wrong Workflow Was Chosen

• The same subject appears in both train and test, but the workflow is cross-sectional.
• A forecasting task is being run without a real time index.
• A survival task has event and time columns but is being treated as plain classification.
• Raw text is heavily compressed into ad hoc numeric features before the text service is even considered.
• Ranking is attempted through ordinary classification labels without query grouping.
• Recommendation data is flattened into a tabular prediction problem with user and item IDs as plain features.
• A phased network is uploaded with the phase column left in the feature list, which hides the phase-comparison diagnostics.
• A graphical-model question is flattened into ordinary regression even though the main output should be an interpretable dependency graph.

Minimal Review Checklist Before Launch

• Confirm the target type really matches the service.
• Confirm the required identifying columns exist and are clean.
• Confirm whether evaluation should use an internal split, a split column, or separate train/validation/test files.
• Confirm whether a sampling-weight column is needed because the modeling file represents a sampled population.
• Confirm the product outputs you need are supported by the service family.
• Confirm the workflow reflects the real structure of the study, not just the convenient shape of the CSV.

Deep References

• Choosing a service
• Service reference overview

Source: docs/book/appendices/model-family-matrix.md

Model Family Matrix

This appendix maps major method families to service families.

Use this as a navigation aid, not a substitute for the detailed service chapters.

Source: docs/book/appendices/upload-forms-model-registry-summary.md

Upload Forms And Model Registry Summary

This appendix summarizes the current relationship between upload workflows and the service catalog model registry.

The implementation source of truth is the live service catalog, the shared upload templates, and the service-specific media model registries. This appendix is the reader-facing summary of that live inventory.

The current catalog-backed inventory is:

In public product copy, this can be summarized as 250+ model configurations across 20 workflows. The precise current count is 268.

Per-Service Counts

Interpretation

The service catalog now separates two ideas:

• supported_models: the full model list represented in the upload workflow and catalog.
• preset_model_values: curated subsets for Fast, Balanced, Best Accuracy, and Low Cost setup choices.

Therefore, a preset count is not the same as a service's full model count. The full count should come from supported_models.

Upload Catalog UX Contract

Model variety is part of the product value. Upload forms that expose models_to_train should mount the model catalog after a successful readiness preview instead of hiding it behind a generic advanced drawer or showing it before the user has mapped the dataset roles.

The expected pattern is:

1. upload and map required roles
2. preview readiness and server-computed dataset shape
3. show the first comparison strategy
4. show the visible model catalog for broadening, narrowing, or targeting model families
5. keep hyperparameters, runtime overrides, and detailed method settings in secondary tuning sections
6. confirm cost and launch training

Regression coverage should continue to check that upload-visible model values match SERVICE_CATALOG, that model controls are clearly labeled, and that the live-monitor bootstrap does not regress on inline upload forms.

The cross-sectional count regression workflow exposes 22 configured options, including count-specific aliases for the added classical and neural comparison set: svm_count, knn_count, mlp_count, gradient_boosting_count, ada_boost_count, keras_mlp_count, keras_cnn_count, and keras_rnn_count. Legacy submitted names such as gradient_boosting, ada_boost, keras_mlp, and keras_cnn are canonicalized to these count-specific IDs before training.

The time-series classification service currently exposes 15 configured options: lstm_classifier, gru_classifier, cnn_1d, cnn_lstm, transformer_ts, tcn_classifier, inception_time, arima_classifier, auto_arima_classifier, var_classifier, garch_classifier, exponential_smoothing_classifier, prophet_classifier, rocket, and minirocket.

Maintenance Rule

When a model is added to an upload form, it must also be added to the matching supported_models list in SERVICE_CATALOG. The regression test tests/test_service_catalog_upload_model_parity.py checks this alignment without importing heavy ML runtime modules.

Source: docs/book/appendices/authoritative-model-registry.md

Authoritative Service And Model Registry

This appendix is generated from the live product registry. Do not edit model lists here by hand.

It covers 20 service workflows, 268 service/model configurations, and 223 unique model values.

A model may appear in more than one service because its objective, preprocessing, validation, and saved artifacts differ by workflow. PDF page numbers can change between builds, so related handbook sections are recorded instead.

Service Configuration Counts

Canonical Acronym Dictionary

These definitions come from the same runtime registry used by Xalec Assistant. Repo-defined longitudinal names use the expansion implemented by the model wrappers; they are not inferred from general web usage.

Canonical Local Model Packages

Package definitions are maintained with the model registry so the handbook and Xalec Assistant describe local longitudinal backends consistently.

Canonical Service/Model Configuration Registry