1. Purpose

Block 4 developed the asymptotic theory of Path 1 (hierarchical Bayesian via Stan). Block 5 developed the parallel theory of Path 2 (varying-coefficient via penalized splines). This block addresses Path 3 (hypernetwork via torch), which is the least mature of the three paths from the standpoint of formal asymptotic theory.

The honest position is the following. The asymptotic theory of Bayesian neural networks (BNNs) and, more broadly, of neural-network-based estimators in the regime relevant to Path 3 is an active area of research with substantial open questions. Several useful partial results exist —NTK consistency in the lazy regime, PAC-Bayes generalization bounds, infinite-width Gaussian-process limits— but a closed asymptotic theory parallel to Theorems 4A-4C of Block 4 or Theorems 5A-5C of Block 5 is not currently available for the hypernetwork form of Path 3.

This block adopts a deliberately modest format:

• §3 identifies the three asymptotic layers (consistency, rate, BvM analog) and explicitly notes which layers Path 3 covers in part and which remain open.
• §4 nominates the standing hypotheses specific to neural-network estimation (architecture, prior, optimizer, regime).
• §5-§8 state four partial results —each adapted from the cited literature— that cover specific regimes of Path 3 with explicit hypotheses and explicit limitations.
• §10 is a substantive list of seven open questions that the framework does not close, with explicit reference to active research and recommendations for users.
• §11 ties the partial results to the empirical discrimination protocol of Block 1 §6.8.1, which is the operational mechanism by which Path 3’s gaps are surfaced.

The block does not pretend to give a complete asymptotic theory of Path 3. It gives what is currently established, marks what is conjectural, and explicitly recognizes the open frontier.

Implementation status (gdpar 0.0.0.9001). Path 3 (hypernetwork via torch) is not currently implemented in the package. A call to gdpar(..., path = "hyper") aborts with a gdpar_unsupported_feature_error (see R/gdpar.R). The asymptotic theory developed below is reference-grade for the framework’s hypernetwork target; the implementation notes in §11 are written in prospective voice (“the planned implementation will…”) because the operational layer is queued for a future version of gdpar and the present block does not assume its availability.

---

2. Setting and Notation

3. Three Asymptotic Layers and Their Status for Path 3

Parallel to Blocks 4-5:

• (L1) Function-level consistency. As \(n \to \infty\), \(\widehat\Phi_n \to \Phi_*\) in \(d_{L^2}(\mu \otimes \pi_\Theta)\) or in a weaker function-space metric.
• (L2) Convergence rate. A sequence \(r_n \to 0\) with \(d_{L^2}(\widehat\Phi_n, \Phi_*) = O_P(r_n)\).
• (L3) BvM analog. A limiting distributional statement for \(\widehat\Phi_n\) (Bayesian posterior; or asymptotic distribution of the trained estimator).

Status of each layer for Path 3:

Layer	Status	Reference for partial result	Open questions
(L1)	Partial: established under NTK regime (Theorem 6A) and under fixed architecture with strong conditions (Proposition 6D). General \(L^2(\mu \otimes \pi_\Theta)\) consistency for arbitrary architecture and SGD optimizer is not closed.	Jacot et al. 2018; Hron et al. 2020	(O1-Path3)
(L2)	Partial: NTK regime gives a kernel-regression rate (Theorem 6A). Finite-sample PAC-Bayes generalization bounds (Theorem 6B) provide non-asymptotic risk control but not contraction rates. Adaptive contraction rates for general BNNs are not closed.	Jacot et al. 2018; Dziugaite and Roy 2017	(O2-Path3), (O3-Path3)
(L3)	Largely open. Infinite-width Gaussian-process limit (Proposition 6C) characterizes the prior rather than the posterior. Function-level BvM for the trained network is not closed.	Lee et al. 2018; Matthews et al. 2018	(O4-Path3)

The framework reports this status explicitly to the user when Path 3 is invoked, and supplements the partial asymptotic theory with the empirical discrimination protocol of Block 1 §6.8.1 (four-step test of stability across seeds, OOS comparison, calibration, component-wise divergence) as the operational substitute for closed asymptotic guarantees.

---

4. Standing Asymptotic Hypotheses for Path 3

In addition to (C1)-(C6), (LIN), (D-ID) of Block 1 (with (LIN) failing for Path 3 —as already noted in Proposition 1F— so identifiability is at the function level), and (HOM), (REG), (IID) of Block 2:

(NN-ARCH) Architecture. The hypernetwork has finite depth \(L\) and per-layer widths \(W_1, \ldots, W_L\). Activation functions are smooth (e.g., GELU, softplus) or piecewise-linear (ReLU); the framework defaults to GELU for differentiability of the activation in the asymptotic analysis.

(NN-PRIOR-SCALE) Prior scaling. Network weights at layer \(\ell\) have prior variance scaling as \(\sigma_\ell^2 / W_\ell\) (NTK parametrization) or \(\sigma_\ell^2 / W_\ell^2\) (mean-field parametrization). The choice of scaling determines the limiting regime as widths grow.

(NN-OPT) Optimization regularity. The optimizer (SGD with momentum, Adam, or Hamiltonian Monte Carlo) converges to a stationary point or, in the Bayesian variant, samples from the posterior asymptotically. This is a non-trivial assumption for non-convex losses; in practice, multiple restarts and convergence diagnostics are used as proxy verification.

(NN-LIM) Limiting regime. One of:

• (NN-LIM-NTK) NTK regime: width \(\to \infty\) with NTK parametrization, training step size \(\to 0\), training time finite. Lazy training; trained network behaves like kernel regression.
• (NN-LIM-MF) Mean-field regime: width \(\to \infty\) with mean-field parametrization. Different limiting dynamics.
• (NN-LIM-FIX) Fixed architecture: \(L\) and \(W_\ell\) fixed; standard parametric asymptotics in \(\phi \in \mathbb{R}^P\).
• (NN-LIM-INF-W) Infinite-width Gaussian process: \(W_\ell \to \infty\), prior on \(\phi\) induces Gaussian process prior on \(\Phi_\phi\).

The four sub-regimes are mutually exclusive and exhaustive of the regimes treated in the literature. The framework does not assume one over the others; the user (or an empirical diagnostic) selects the regime appropriate to the problem and architecture.

---

5. Theorem 6A: Consistency under the NTK Regime (Partial)

Theorem 6A. Stated under (R-RANDOM) and (NN-LIM-NTK). Suppose:

• The network architecture (NN-ARCH) and prior scaling (NN-PRIOR-SCALE) are NTK-compatible.
• Training proceeds via gradient descent (or its Bayesian analog) in the lazy regime: width \(\to \infty\), step size \(\to 0\), \(T\) training iterations finite.
• Standard regularity on the loss and the data distribution.

Then two distinct limits hold, with distinct hypotheses:

Limit (i): Trained network \(\to\) NTK kernel-regression solution.

The trained network’s realized function converges, in the NTK limit, to the kernel-regression solution: \[\widehat\Phi_n \;\xrightarrow{P}\; \Phi^{\mathrm{NTK}}_n \;=\; \mathop{\arg\min}_{\Phi \in \mathrm{RKHS}(\mathrm{NTK})} \;\frac{1}{n} \sum_i \mathcal{L}(Y_i, \Phi(X_i, \theta_{\text{ref}})) + \tau \|\Phi\|^2_{\mathrm{RKHS}},\] for an appropriate regularization \(\tau\) determined by the early-stopping schedule. This convergence is the content of Jacot, Gabriel, and Hongler (2018, Theorem 1) and uses only NTK-regime hypotheses on training dynamics.

Limit (ii): NTK kernel-regression solution \(\to \Phi_*\).

As \(n \to \infty\), \(\Phi^{\mathrm{NTK}}_n \to \Phi^{\mathrm{NTK}}_*\) in \(L^2(\mu \otimes \pi_\Theta)\), where \(\Phi^{\mathrm{NTK}}_*\) is the best \(L^2\)-approximation of \(\Phi_*\) within the closure of \(\mathrm{RKHS}(\mathrm{NTK})\): \[\Phi^{\mathrm{NTK}}_* \;=\; \mathop{\arg\min}_{\Phi \in \overline{\mathrm{RKHS}(\mathrm{NTK})}} \;d_{L^2}(\Phi, \Phi_*).\] The further conclusion \(\Phi^{\mathrm{NTK}}_* = \Phi_*\) —i.e., that the kernel-regression limit recovers the true function— requires the additional source/smoothness hypothesis that \(\Phi_* \in \mathrm{RKHS}(\mathrm{NTK})\) with sufficient smoothness to satisfy the kernel’s source condition (Caponnetto and De Vito 2007). If \(\Phi_* \notin \mathrm{RKHS}(\mathrm{NTK})\), only convergence to the projection \(\Phi^{\mathrm{NTK}}_*\) holds, and the gap \(d_{L^2}(\Phi^{\mathrm{NTK}}_*, \Phi_*)\) is the kernel approximation error —an irreducible bias of the NTK regime for the chosen architecture.

Reference and limitations. The result is a specialization of Jacot, Gabriel, and Hongler (2018, Theorem 1) and Bach (2017, Theorem 5) to the AMM hypernetwork context. Three substantive limitations:

1. The NTK regime is the lazy training regime: it requires width \(\to \infty\) with appropriate parameter scaling and small initial parameters. Practical hypernetworks of finite width and standard SGD operate outside the NTK regime; Theorem 6A gives an approximation whose quality depends on the gap between practical training and lazy training.

2. The RKHS of the NTK is restrictive: it is a specific Hilbert space determined by the architecture. Functions \(\Phi_*\) outside this RKHS may not be representable by the trained network in the NTK regime, even though the network has universal approximation capacity in the wide-width limit (Hornik 1991).

3. The rate of \(\Phi^{\mathrm{NTK}}_n \to \Phi_*\) depends on the NTK and on the smoothness of \(\Phi_*\) in this specific RKHS. There is no universal rate analogous to the Stone-optimal \(n^{-\beta/(2\beta+d)}\).

Practical implication. Theorem 6A is a useful conceptual statement: it says that, in the wide-width lazy regime, the hypernetwork behaves like a specific kernel regression. But it does not give a universally applicable contraction rate or a finite-sample guarantee for typical Path 3 fits.

---

6. Theorem 6B: PAC-Bayes Generalization Bound (Non-Asymptotic)

Theorem 6B. Finite-sample PAC-Bayes bound (McAllester 1999; Dziugaite and Roy 2017). Let \(\pi(\phi)\) be a prior on \(\phi\) chosen before observing the data, and \(\widehat\Pi_n(\phi)\) any (data-dependent) posterior over \(\phi\). Let \(L(\phi) = \mathbb{E}_{(X, Y)}[\ell(Y, \Phi_\phi(X))]\) be the population risk and \(\widehat L_n(\phi) = n^{-1} \sum_i \ell(Y_i, \Phi_\phi(X_i))\) the empirical risk. Suppose \(\ell \in [0, 1]\) (or appropriately bounded). Then with probability at least \(1 - \delta\) over the sample: \[\mathbb{E}_{\widehat\Pi_n}[L(\phi)] \;\leq\; \mathbb{E}_{\widehat\Pi_n}[\widehat L_n(\phi)] \;+\; \sqrt{\frac{\mathrm{KL}(\widehat\Pi_n \,\|\, \pi) + \log(2 \sqrt{n} / \delta)}{2(n - 1)}}.\]

Argument. Standard PAC-Bayes via concentration of the Rényi divergence; see Dziugaite and Roy (2017) for the deep-learning specialization with \(\widehat\Pi_n\) realized as a Gaussian variational posterior.

What this gives Path 3.

• A finite-sample bound on the gap between empirical and population risk that the user can evaluate after fitting. Concretely: after running the SVI variational fit, the user computes the empirical risk on training data, the KL divergence between variational posterior and prior, and obtains a numerical upper bound on the population risk.
• The bound is tight enough to be informative in many practical regimes (Dziugaite and Roy 2017 obtained vacuous-bound improvements that gave non-vacuous test-error bounds for MNIST-scale problems).

What this does not give.

• An asymptotic contraction rate. Theorem 6B is a finite-sample statement and does not directly yield a sequence \(r_n \to 0\).
• A guarantee about the realized function \(\Phi_\phi\) converging to \(\Phi_*\). It bounds population risk, which is a one-dimensional functional of \(\Phi_\phi\), and does not control \(d_{L^2}(\widehat\Phi_n, \Phi_*)\) directly.
• Uniform validity in \(\phi\). The bound is on the expected risk under \(\widehat\Pi_n\), not on individual draws.

Operational use. The library reports the PAC-Bayes bound as a finite-sample quality certificate alongside Path 3 fits. A small bound is evidence that the trained network has not overfit; a large bound is a warning that the empirical-risk minimizer may not generalize.

---

7. Proposition 6C: Infinite-Width Gaussian Process Limit (Characterization of the Prior)

Proposition 6C. Characterization of the prior in the infinite-width limit. Stated under (NN-LIM-INF-W). Suppose the network has an infinite-width limit with NTK parametrization (or appropriate analog). Then the prior \(\pi(\phi)\) on the network weights induces, in this limit and at the level of the prior over functions, a Gaussian process prior on the realized function: \[\Phi_\phi \;\sim\; \mathrm{GP}(0, K_{\mathrm{NNGP}}) \qquad \text{(prior over functions, infinite-width limit)},\] where \(K_{\mathrm{NNGP}}\) is the Neural Network Gaussian Process kernel determined recursively by the architecture and the prior weight variances (Lee et al. 2018; Matthews et al. 2018).

Important: this is a statement about the prior, not about the posterior at finite width. The infinite-width Gaussian process limit characterizes how the weight prior \(\pi(\phi)\) translates into a function-space prior. The posterior under this prior, given finite data, is a separate question; in the infinite-width limit the posterior coincides with Gaussian-process-regression posterior (whose theory is mature), but at finite width the posterior is not generally Gaussian and the gap to the GP posterior is the subject of ongoing research (Hron et al. 2020 and (O5-Path3) below).

Reference. Lee, Bahri, Novak, Schoenholz, Pennington, Sohl-Dickstein (2018) for the closed-form recursion of \(K_{\mathrm{NNGP}}\); Matthews, Rowland, Hron, Turner, Ghahramani (2018) for the rigorous infinite-width limit theorem.

Implication for Path 3. In the infinite-width limit, Path 3 is asymptotically equivalent to a Gaussian process regression with the NNGP kernel. The asymptotic theory of GP regression then transfers to Path 3 in this limit:

• Posterior consistency at the GP regression rate (van der Vaart and van Zanten 2008).
• Posterior contraction at smoothness-adaptive rates under appropriate kernel scaling.
• Bernstein-von Mises for the parametric component of the GP posterior under standard regularity.

Crucial caveat: this is a characterization of the prior, not the posterior at finite width. The infinite-width GP limit holds for the prior over functions induced by the network architecture and weight prior. The posterior at finite width —which is what the user actually fits with torch— has additional finite-width corrections that are not fully characterized. Hron et al. (2020) survey the partial results on finite-width Bayesian deep networks; the gap between infinite-width and finite-width is the subject of (O5-Path3).

---

8. Proposition 6D: Function-Level Consistency under Fixed Architecture (Partial)

Proposition 6D. Stated under (R-RANDOM) and (NN-LIM-FIX). Suppose:

• The hypernetwork has fixed architecture (depth \(L\), widths \(W_1, \ldots, W_L\), total parameters \(P\)).
• The prior \(\pi(\phi)\) has positive density on a compact subset of \(\mathbb{R}^P\).
• The negative log-likelihood is smooth in \(\phi\) on this subset.
• The optimizer (or MCMC sampler) converges to some stationary point or stationary distribution of the empirical risk surface (no claim is made that the stationary point is a global minimum).

Then, as \(n \to \infty\) with the architecture fixed, the trained network’s realized function converges to a stationary limit \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) within the function class realizable by the architecture: \[\widehat\Phi_n \;\xrightarrow{P}\; \Phi^{(L, W_{1:L}), \text{stat}}_n,\] where \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) corresponds to a (not necessarily global) minimizer of the empirical risk reachable by the optimizer from its initialization.

The further claim that \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) converges to the global best \(L^2(\mu \otimes \pi_\Theta)\)-approximation \[\Phi^{(L, W_{1:L})}_* \;=\; \mathop{\arg\min}_{\Phi_\phi : \phi \in \mathbb{R}^P} \;d_{L^2}(\Phi_\phi, \Phi_*)\] requires the additional global-optimality hypothesis (NN-OPT-GLOBAL): the optimizer (or sampler) reaches a global minimum of the empirical risk, not just a stationary point. (NN-OPT-GLOBAL) is a strong condition for non-convex losses (the typical case for neural networks) and is not generically satisfied; it can fail at saddle points and at non-global local minima created by the network’s symmetries (Proposition 1F (ii)).

Reference and discussion. This is M-estimator consistency (van der Vaart 1998, §5) applied to neural-network estimation under fixed architecture, modified to honor the gap between stationary points and global optima in non-convex landscapes.

Three points of substance:

1. The unconditional limit is a stationary point, not the best architecture-realizable approximation. The two coincide only under (NN-OPT-GLOBAL).

2. Even under (NN-OPT-GLOBAL), the limit \(\Phi^{(L, W_{1:L})}_*\) is not \(\Phi_*\) in general. It is the projection of \(\Phi_*\) onto the realizable function class. If the architecture is too narrow or too shallow, the projection may differ substantially from \(\Phi_*\); the gap is the architecture approximation error.

3. For the architecture approximation error to vanish, \(W_\ell\) must grow with \(n\). This is the growing-architecture regime, which is a hybrid of (NN-LIM-FIX) and (NN-LIM-INF-W); the asymptotic theory in this regime is not closed and is part of (O3-Path3) below.

4. Convergence rate to the stationary limit is parametric (\(n^{-1/2}\)) in \(\phi\) under standard regularity, but network symmetries (Proposition 1F (ii)) make this convergence at the level of \(\phi\) ambiguous; convergence at the level of \(\Phi_\phi\) holds at parametric rate when the stationary point is locally identifiable up to symmetries.

Practical implication. Proposition 6D is a useful conceptual statement with two layers of caveat: first, the empirical optimization may not find the global minimum (the architecture-realizable best approximation within the realizable class may itself be unreached); second, even if it is reached, the architecture may be too rigid to approach \(\Phi_*\) (architecture approximation error of the realizable class). The framework reports both layers of error separately when feasible (using restart-based estimates of the gap to global optimum, plus comparison to NTK-regime fits as a check on architecture sufficiency). This is the formal version of two related practical concerns: “your optimizer might not have converged to a global minimum” and “your network might not be deep/wide enough”.

---

9. Specialization to Block 3 Special Cases

Path 3 covers Proposition 3.7 of Block 3 (hypernetworks). The asymptotic results of this block specialize as:

Special Case	AMM Level	(L1) Consistency	(L2) Rate	(L3) BvM
Hypernetwork (Proposition 3.7)	\(\infty\)	Theorem 6A (NTK) or Proposition 6D (fixed) under regime-specific hypotheses	NTK-determined or PAC-Bayes finite-sample (Theorem 6B); no universal rate	Open (O4-Path3); Proposition 6C characterizes the prior, not the posterior

The Path 1 and Path 2 special cases (Theorems 3.1-3.6) are not treated by Path 3 because they fall within the linear-class scope (LIN) of Block 1; using a hypernetwork for those cases is technically possible but operationally unnecessary, and the asymptotic theory of the simpler paths is more complete.

---

10. Open Questions

The open questions for Path 3 are substantively more numerous than for Paths 1-2, reflecting the genuinely open frontier of the asymptotic theory of neural networks.

(O1-Path3) General consistency outside the NTK regime. Theorem 6A requires the NTK lazy regime; Proposition 6D requires fixed architecture. The realistic regime —finite-but-large width, SGD with momentum and weight decay— is not covered by either. General function-level consistency in this regime is an open problem.

(O2-Path3) Convergence rate under realistic optimization. Most rate results assume gradient descent at vanishing step size or full-batch gradient. SGD with momentum, Adam, and other modern optimizers introduce additional dynamics whose asymptotic effects on the realized function are partially understood (Hardt, Recht, Singer 2016 for SGD generalization) but not fully characterized.

(O3-Path3) Adaptive contraction rates under unknown smoothness. The rates derivable from Theorem 6A depend on the smoothness of \(\Phi_*\) in the NTK RKHS, which is typically unknown. Adaptive priors and architectures that achieve the optimal rate without specifying the smoothness are an open question (relevant work: Polson and Sokolov 2017; Suzuki 2018 for adaptive deep learning).

(O4-Path3) Function-level Bernstein-von Mises. A full BvM for the posterior over \(\Phi_\phi\) in a function-space metric is open. The infinite-width GP limit (Proposition 6C) gives BvM only for the GP-projected functional, not the full network posterior.

(O5-Path3) Finite-width corrections to the infinite-width GP limit. Proposition 6C holds in the infinite-width limit. At finite width, the network behaves differently; characterizing the gap is the subject of recent work (Hron et al. 2020; Naveh, Ringel 2021) but is not closed.

(O6-Path3) Quantification of network symmetries. Proposition 1F (ii) of Block 1 notes that \(\phi\) is non-identifiable due to permutation, sign, and rescaling symmetries. The quantitative effect of these symmetries on optimizer convergence —specifically, how they create local minima or saddle points and how SGD navigates them— is not fully characterized.

(O7-Path3) Out-of-distribution behavior of \(\Phi_\phi\). All asymptotic results are stated under the i.i.d. data-generating assumption. The behavior of the trained network on covariates outside \(\mathrm{supp}(\mu)\) is not controlled by the asymptotic theory and is a separate domain (out-of-distribution generalization; recent work: Arjovsky et al. 2019; Nagarajan, Andreassen, Neyshabur 2021).

The framework reports these open questions explicitly to the user when Path 3 is invoked, alongside the partial results above. The empirical discrimination protocol of Block 1 §6.8.1 provides operational diagnostics that compensate, in part, for the asymptotic gaps: stability across seeds (testing for symmetry-induced non-identifiability, related to (O6)); cross-validated OOS performance (testing generalization in the absence of asymptotic guarantees, related to (O1)-(O3)); calibration (testing posterior reliability in the absence of BvM, related to (O4)); Path 1 vs. Path 3 comparison (testing how much the gap to closed theory matters in practice).

---

11. Implementation Implications for Path 3 (torch)

The planned Path 3 implementation in gdpar will use the R torch package (LibTorch backend), supporting both SGD-based ERM training and SVI Bayesian variational inference. The current release (0.0.0.9001) does not ship this path; the design notes below specify the contract the operational layer will honour when it lands.

12. Summary

This block has established for Path 3:

1. Three asymptotic layers —function-level consistency (L1), convergence rate (L2), BvM analog (L3)— and an explicit assessment that none of the three is closed in general for Path 3, with (L1) and (L2) covered partially by regime-specific results and (L3) largely open.

2. Standing asymptotic hypotheses —(NN-ARCH), (NN-PRIOR-SCALE), (NN-OPT), (NN-LIM)— nominated, with (NN-LIM) further decomposed into four mutually exclusive sub-regimes (NTK, mean-field, fixed, infinite-width).

3. Theorem 6A (NTK consistency) in the lazy regime, with three substantive limitations explicitly recognized.

4. Theorem 6B (PAC-Bayes bound) as a finite-sample non-asymptotic quality certificate, with explicit recognition that it does not yield a contraction rate.

5. Proposition 6C (Infinite-width GP limit) as a characterization of the prior, not the posterior at finite width, with the gap to finite-width acknowledged.

6. Proposition 6D (Fixed-architecture consistency) with the limit being the best architecture-realizable approximation of \(\Phi_*\), not \(\Phi_*\) itself.

7. Seven explicitly recognized open questions (general consistency outside NTK, realistic optimizer dynamics, adaptive rates, function-level BvM, finite-width corrections, quantification of symmetries, OOD behavior).

8. Implementation diagnostics for Path 3 in torch, including the NTK eigenvalue spectrum, the PAC-Bayes bound report, and cross-reference to the empirical discrimination protocol of Block 1 §6.8.1 as the operational substitute for closed asymptotic guarantees.

The block does not pretend to give a complete asymptotic theory of Path 3 —none currently exists— and is honest about what is established, what is conjectural, and what remains genuinely open. The framework’s recommendation is that Path 3 be used as a complement to Path 1, not as a stand-alone path with full asymptotic backing, except in regimes where the partial results of this block apply (lazy NTK regime; PAC-Bayes finite-sample bound that the user evaluates explicitly; infinite-width GP limit when justified by the architecture).

---

13. Connections to Subsequent Blocks

• Block 7 (Empirical Bayes vs. Fully Bayes) interacts with this block via the choice of how \(\theta_{\text{ref}}\) is treated under Path 3: either as a hyperparameter fit empirically alongside the network parameters (EB), or as a parameter with its own prior fit jointly (FB). The asymptotic implications differ between the two.
• Block 8 (CATE/ITE positioning) places Path 3 in the context of neural-network-based heterogeneous treatment effect estimators (e.g., causal forests with neural extensions; deep IV); the asymptotic theory of these estimators inherits the gaps of Path 3.

---

Appendix A. Asymptotic Notation for Path 3

Symbol	Meaning
\(\Phi_\phi\)	Realized function of the hypernetwork
\(\widehat\Phi_n = \Phi_{\widehat\phi_n}\)	Trained realized function
\(\Phi_*\)	True realized function
\(\Phi^{(L, W_{1:L})}_*\)	Best architecture-realizable approximation of \(\Phi_*\)
\(\Phi^{\mathrm{NTK}}_n\)	NTK kernel-regression solution
\(\mathrm{NTK}\)	Neural Tangent Kernel
\(\mathrm{NNGP}\)	Neural Network Gaussian Process kernel
\(K_{\mathrm{NNGP}}\)	NNGP kernel function
\(\widehat\Pi_n(\phi)\)	(Variational) posterior on \(\phi\)
\(L(\phi), \widehat L_n(\phi)\)	Population and empirical risk at parameter \(\phi\)
\(\mathrm{RKHS}(\mathrm{NTK})\)	RKHS associated with the NTK
\(P\)	Number of network parameters
\(L\)	Network depth
\(W_\ell\)	Width of layer \(\ell\)

---

Appendix B. Asymptotic Hypothesis Table for Path 3

Hypothesis	Content	Used by
(R-RANDOM)	Random design as in Block 4 §2.1.1	Default for §5-8
(R-COND)	Fixed/conditional design	Variant for §5-8
(NN-ARCH)	Finite depth \(L\), per-layer widths \(W_\ell\), smooth or piecewise-linear activations	All theorems
(NN-PRIOR-SCALE)	Per-layer prior variance \(\sigma_\ell^2 / W_\ell\) (NTK) or \(\sigma_\ell^2 / W_\ell^2\) (mean-field)	Theorems 6A, 6B, 6C, 6D
(NN-OPT)	Optimizer / sampler converges to a stationary point / distribution (not necessarily global optimum)	All theorems
(NN-OPT-GLOBAL)	Optimizer / sampler reaches the global minimum of the empirical risk; strong condition for non-convex losses	Proposition 6D when limit is the best architecture-realizable approximation
(NN-LIM-NTK)	Width \(\to \infty\), lazy regime	Theorem 6A
(NN-LIM-MF)	Mean-field regime	Not stated; reference to literature
(NN-LIM-FIX)	Fixed architecture	Proposition 6D
(NN-LIM-INF-W)	Infinite-width Gaussian process limit	Proposition 6C

---

References Cited in This Block

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv:1907.02893.

Bach, F. (2017). Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18, 1–53.

Caponnetto, A., and De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3), 331–368.

Chizat, L., and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. NeurIPS 2018.

Dziugaite, G. K., and Roy, D. M. (2017). Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Proceedings of UAI.

Hardt, M., Recht, B., and Singer, Y. (2016). Train faster, generalize better: stability of stochastic gradient descent. ICML 2016.

Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 251–257.

Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. (2020). Infinite attention: NNGP and NTK for deep attention networks. ICML 2020.

Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: convergence and generalization in neural networks. NeurIPS 2018.

Lee, J., Bahri, Y., Novak, R., Schoenholz, S., Pennington, J., and Sohl-Dickstein, J. (2018). Deep neural networks as Gaussian processes. ICLR 2018.

Matthews, A. G. de G., Rowland, M., Hron, J., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. ICLR 2018.

McAllester, D. A. (1999). PAC-Bayesian model averaging. Proceedings of COLT.

Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. PNAS, 115(33), E7665–E7671.

Nagarajan, V., Andreassen, A., and Neyshabur, B. (2021). Understanding the failure modes of out-of-distribution generalization. ICLR 2021.

Naveh, G., and Ringel, Z. (2021). A self-consistent theory for finite-width neural networks. NeurIPS 2021.

Polson, N. G., and Sokolov, V. (2017). Deep learning: a Bayesian perspective. Bayesian Analysis, 12(4), 1275–1304.

Suzuki, T. (2018). Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. ICLR 2018.

van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.

van der Vaart, A. W., and van Zanten, J. H. (2008). Rates of contraction of posterior distributions based on Gaussian process priors. Annals of Statistics, 36(3), 1435–1463.