{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Learning Curve generation for: RPR \u00d7 T \u00d7 MFML\n\nIn this example we will demonstrate a full set of learning curves for the following\ncases of single fidelity and multifidelity ML models. We will be using the KRR basic\narchitecture.\n\n Single-fidelity: MFML:\n 1. Isotropic 5. MFML Isotropic\n 2. RPR + Isotropic 6. MFML RPR + Isotropic\n 3. Coupled T 7. MFML Coupled T\n 4. RPR + Coupled T 8. MFML RPR + Coupled T\n\nX-axis: number of training samples at the highest fidelity.\nMFML uses all 5 QeMFi fidelities with 2x scaling per level down.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import copy\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom mfml_qc.datasets import load_nitrophenol_data\nfrom mfml_qc.representations import compute_flat_coulomb\nfrom mfml_qc.utils import top_down_subsetting\nfrom mfml_qc.vector_krr import VectorKRR, CoupledVectorKRR\nfrom mfml_qc.rpr import RPRVectorKRR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load Data\nWe load the QeMFi nitrophenol dataset and split into a fixed test set of\n1000 geometries and a training pool of the remaining 14000. The test set\nuses the highest-fidelity (def2-TZVP) electronic dipole moments as\nreference.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "dataset = load_nitrophenol_data()\nR = dataset[\"R\"]\nZ = dataset[\"Z\"]\nX_CM = dataset[\"X_CM\"]\nDPe = dataset[\"DPe\"]\n\nNUM_FIDS = 5\nN_TEST = 1000\n\nrng = np.random.RandomState(42)\nperm = rng.permutation(len(R))\ntest_idx = perm[:N_TEST]\npool_idx = perm[N_TEST:]\n\nDPe_test_ref = DPe[test_idx, -1, :]\nX_test_flat = R[test_idx].reshape(N_TEST, -1)\nCM_test = X_CM[test_idx]\nCM_pool = X_CM[pool_idx]\nR_pool = R[pool_idx]\nDPe_pool = DPe[pool_idx]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Configuration\nWe sweep over highest-fidelity training sizes from 16 to 256. Each model\nvariant is defined by a tuple of ``(name, use_rpr, use_T, is_mfml)``.\nThe ``eval_preds`` helper computes overall MAE, per-component MAE, cosine\nsimilarity, and mean angular error.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "N_HF_VALUES = [16, 32, 64, 128, 256]\nKERNEL = \"gaussian\"\nSIGMA = 250.0\nREG = 1e-10\n\n\ndef eval_preds(pred, true):\n mae = np.mean(np.abs(pred - true))\n comp_mae = np.mean(np.abs(pred - true), axis=0)\n mag_p, mag_t = np.linalg.norm(pred, axis=-1), np.linalg.norm(true, axis=-1)\n dot = np.sum(pred * true, axis=-1)\n cos_sim = np.clip(dot / np.maximum(mag_p * mag_t, 1e-12), -1.0, 1.0)\n angle = np.mean(np.degrees(np.arccos(cos_sim)))\n return {\n \"mae\": mae,\n \"comp_mae\": comp_mae,\n \"cos_sim\": np.mean(cos_sim),\n \"angle\": angle,\n \"pred\": pred,\n }\n\n\ndef make_model(use_rpr, use_T):\n if use_T:\n inner = CoupledVectorKRR(\n kernel_type=KERNEL,\n sigma=SIGMA,\n reg=REG,\n val_fraction=0.15,\n max_opt_iter=200,\n )\n else:\n inner = VectorKRR(kernel_type=KERNEL, sigma=SIGMA, reg=REG)\n if use_rpr:\n return RPRVectorKRR(inner, Z, compute_flat_coulomb)\n return inner\n\n\n# recall that the variants are labelled by (name, use_rpr, use_T, is_mfml)\nVARIANTS = [\n (\"Iso\", False, False, False),\n (\"RPR+Iso\", True, False, False),\n (\"T\", False, True, False),\n (\"RPR+T\", True, True, False),\n (\"MFML Iso\", False, False, True),\n (\"MFML RPR+Iso\", True, False, True),\n (\"MFML T\", False, True, True),\n (\"MFML RPR+T\", True, True, True),\n]\n\nresults = {v[0]: [] for v in VARIANTS}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Single-Fidelity Models\nTrains a single model on the ``n_hf`` highest-fidelity samples. Non-RPR\nmodels receive pre-computed Coulomb matrices; RPR models receive flattened\nCartesian coordinates and compute canonicalized descriptors internally.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def run_single_fidelity(n_hf, use_rpr, use_T):\n Y_train = DPe_pool[:n_hf, -1, :]\n Y_mean = np.mean(Y_train, axis=0)\n model = make_model(use_rpr, use_T)\n if use_rpr:\n model.train(R_pool[:n_hf].reshape(n_hf, -1), Y_train - Y_mean)\n pred = model.predict(X_test_flat) + Y_mean\n else:\n model.train(CM_pool[:n_hf], Y_train - Y_mean)\n pred = model.predict(CM_test) + Y_mean\n return eval_preds(pred, DPe_test_ref)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## MFML Models\nTrains 2N-1 = 9 sub-models following the SGCT scheme. The hierarchy uses\n``n_hf`` samples at the highest fidelity and doubles at each lower level.\nUpper sub-models learn fidelity-i targets on the fidelity-i subset; lower\nsub-models learn fidelity-i targets on the fidelity-(i+1) subset.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def run_mfml(n_hf, use_rpr, use_T):\n n_trains = [\n min(n_hf * 2 ** (NUM_FIDS - 1 - i), len(R_pool)) for i in range(NUM_FIDS)\n ]\n\n y_trains_full = np.zeros(NUM_FIDS, dtype=object)\n indexes_full = np.zeros(NUM_FIDS, dtype=object)\n for i in range(NUM_FIDS):\n vals = DPe_pool[:, i, :]\n y_trains_full[i] = vals - np.mean(vals, axis=0)\n indexes_full[i] = np.column_stack(\n [np.arange(len(R_pool)), np.arange(len(R_pool))]\n )\n\n subset_y, subset_idx = top_down_subsetting(\n y_trains_full, indexes_full, n_trains, seed=42\n )\n\n n_sub = 2 * NUM_FIDS - 1\n sub_X, sub_Y, sub_Y_mean = [], [], []\n\n for i in range(NUM_FIDS):\n idx_i = subset_idx[i][:, 0]\n Y_i = DPe_pool[idx_i, i, :]\n m_i = np.mean(Y_i, axis=0)\n sub_Y.append(Y_i - m_i)\n sub_Y_mean.append(m_i)\n sub_X.append(\n R_pool[idx_i].reshape(len(idx_i), -1) if use_rpr else CM_pool[idx_i]\n )\n\n for i in range(NUM_FIDS - 1):\n idx_ip1 = subset_idx[i + 1][:, 0]\n Y_lower = DPe_pool[idx_ip1, i, :]\n m_i = np.mean(Y_lower, axis=0)\n sub_Y.append(Y_lower - m_i)\n sub_Y_mean.append(m_i)\n sub_X.append(\n R_pool[idx_ip1].reshape(len(idx_ip1), -1) if use_rpr else CM_pool[idx_ip1]\n )\n\n sub_models = []\n for s in range(n_sub):\n model_s = make_model(use_rpr, use_T)\n model_s.train(sub_X[s], sub_Y[s])\n sub_models.append(model_s)\n\n X_pred = X_test_flat if use_rpr else CM_test\n pred = np.zeros((N_TEST, 3))\n for i in range(NUM_FIDS):\n pred += sub_models[i].predict(X_pred) + sub_Y_mean[i]\n for i in range(NUM_FIDS - 1):\n pred -= sub_models[NUM_FIDS + i].predict(X_pred) + sub_Y_mean[NUM_FIDS + i]\n\n return eval_preds(pred, DPe_test_ref)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run All Variants\nFor each highest-fidelity training size, all 8 variants are trained and\nevaluated. Results are stored in a dictionary keyed by variant name.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "for n_hf in N_HF_VALUES:\n for name, use_rpr, use_T, is_mfml in VARIANTS:\n r = (\n run_mfml(n_hf, use_rpr, use_T)\n if is_mfml\n else run_single_fidelity(n_hf, use_rpr, use_T)\n )\n results[name].append(r)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overall MAE and Angular Error Learning Curves\nDashed lines represent single-fidelity models; solid lines represent MFML.\nColour encodes the (RPR, T) combination. The left panel shows overall MAE\non a log-log scale; the right panel shows mean angular error.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "combo_colors = {\n \"Iso\": plt.cm.viridis(0.1),\n \"RPR+Iso\": plt.cm.viridis(0.4),\n \"T\": plt.cm.viridis(0.65),\n \"RPR+T\": plt.cm.viridis(0.9),\n}\n\n\ndef _base_key(use_rpr, use_T):\n return (\"RPR+\" if use_rpr else \"\") + (\"T\" if use_T else \"Iso\")\n\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\n\nfor name, use_rpr, use_T, is_mfml in VARIANTS:\n maes = [results[name][i][\"mae\"] for i in range(len(N_HF_VALUES))]\n angles = [results[name][i][\"angle\"] for i in range(len(N_HF_VALUES))]\n color = combo_colors[_base_key(use_rpr, use_T)]\n ls = \"-\" if is_mfml else \"--\"\n marker = \"s\" if is_mfml else \"o\"\n ax1.loglog(\n N_HF_VALUES,\n maes,\n marker=marker,\n linestyle=ls,\n color=color,\n lw=2,\n ms=6,\n label=name,\n )\n ax2.semilogx(\n N_HF_VALUES,\n angles,\n marker=marker,\n linestyle=ls,\n color=color,\n lw=2,\n ms=6,\n label=name,\n )\n\nfor ax, ylabel, title in [\n (ax1, \"Overall MAE (Debye)\", \"Learning Curve: MAE\"),\n (ax2, \"Mean Angular Error (\u00b0)\", \"Learning Curve: Angular Error\"),\n]:\n ax.set_xlabel(\"N (highest fidelity training samples)\")\n ax.set_ylabel(ylabel)\n ax.set_title(title)\n ax.legend(fontsize=7, ncol=2)\n ax.grid(True, alpha=0.3, which=\"both\")\n ax.set_xticks(N_HF_VALUES)\n ax.set_xticklabels([str(n) for n in N_HF_VALUES])\n\nplt.tight_layout()\nplt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Per-Component Learning Curves\nThe same learning curves broken down by dipole component (\u03bc_x, \u03bc_y, \u03bc_z).\nThis reveals which components benefit most from RPR, coupled T, or MFML.\nUnlike the last example, we DO NOT train individual component-wise MFML models\nbut only look at the separate components post hoc.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "comp_labels = [\"\u03bc_x\", \"\u03bc_y\", \"\u03bc_z\"]\nfig, axes = plt.subplots(1, 3, figsize=(15, 4))\n\nfor c in range(3):\n for name, use_rpr, use_T, is_mfml in VARIANTS:\n vals = [results[name][i][\"comp_mae\"][c] for i in range(len(N_HF_VALUES))]\n color = combo_colors[_base_key(use_rpr, use_T)]\n ls = \"-\" if is_mfml else \"--\"\n marker = \"s\" if is_mfml else \"o\"\n axes[c].loglog(\n N_HF_VALUES,\n vals,\n marker=marker,\n linestyle=ls,\n color=color,\n lw=2,\n ms=5,\n label=name,\n )\n axes[c].set_xlabel(\"N (highest fidelity)\")\n axes[c].set_ylabel(\"MAE (Debye)\")\n axes[c].set_title(f\"Learning Curve: {comp_labels[c]}\")\n axes[c].grid(True, alpha=0.3, which=\"both\")\n axes[c].set_xticks(N_HF_VALUES)\n axes[c].set_xticklabels([str(n) for n in N_HF_VALUES])\n if c == 0:\n axes[c].legend(fontsize=6, ncol=2)\n\nplt.tight_layout()\nplt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parity Plots at Largest N_hf\nPredicted vs. reference dipole components for all 8 variants at the largest\ntraining size. Points are coloured by the true dipole magnitude. Each row\ncorresponds to one model variant; columns are the three Cartesian components.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "n_hf_last = N_HF_VALUES[-1]\nmag_true = np.linalg.norm(DPe_test_ref, axis=-1)\ncmap = plt.cm.viridis\nvmin, vmax = mag_true.min(), mag_true.max()\n\nfig, axes = plt.subplots(\n len(VARIANTS),\n 3,\n figsize=(5 * 3, 4 * len(VARIANTS)),\n gridspec_kw={\"hspace\": 0.4, \"wspace\": 0.3},\n)\n\nfor row, (name, _, _, _) in enumerate(VARIANTS):\n pred = results[name][-1][\"pred\"]\n comp_mae = np.mean(np.abs(pred - DPe_test_ref), axis=0)\n for c in range(3):\n sc = axes[row, c].scatter(\n DPe_test_ref[:, c],\n pred[:, c],\n s=2,\n alpha=0.4,\n c=mag_true,\n cmap=cmap,\n vmin=vmin,\n vmax=vmax,\n )\n lo = min(DPe_test_ref[:, c].min(), pred[:, c].min())\n hi = max(DPe_test_ref[:, c].max(), pred[:, c].max())\n axes[row, c].plot([lo, hi], [lo, hi], \"r--\", lw=1.5)\n axes[row, c].set_xlabel(f\"True {comp_labels[c]} (Debye)\")\n axes[row, c].set_ylabel(f\"Predicted {comp_labels[c]}\")\n axes[row, c].set_title(f\"{name} \u2014 {comp_labels[c]} (MAE: {comp_mae[c]:.4f})\")\n\nfig.subplots_adjust(right=0.88)\ncbar_ax = fig.add_axes([0.90, 0.05, 0.015, 0.9])\nfig.colorbar(sc, cax=cbar_ax).set_label(\"|\u03bc| (Debye)\")\nplt.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.22" } }, "nbformat": 4, "nbformat_minor": 0 }