Protein Autoregressive Modeling
via Multiscale Structure Generation

We construct the multi-scale representations of protein structures via hierarchical downsampling to serve as training context and targets for PAR.

The input structure \(\mathbf{x} \in \mathbb{R}^{L \times 3}\) is decomposed into multi-scale representations \(\{ \mathbf{x}^{1}, \dots, \mathbf{x}^{n} \}\) through a scale configuration \(\{ \texttt{size(1)}, \texttt{size(2)}, \dots, \texttt{size(n)} \}\), where \(\text{Down}(\mathbf{x}, \texttt{size(i)}) \in \mathbb{R}^{\texttt{size(i)} \times 3}\) denotes a downsampling operation that interpolates \(\mathbf{x}\) along the sequence dimension, producing \(\texttt{size(i)}\) 3D centroids that provide a coarse structural layout.

Scale configuration can be defined in two ways:

• By length: scales are chosen as hyperparameters, e.g., \(\mathbf{S} = \{64, 128, 256\}\). If \(\mathbf{L} \in (\texttt{size(i)}, \texttt{size(i+1)}]\), the protein can be generated with only \((i+1)\) autoregressive steps. Empirically, defining scales by length yields slightly better results in modeling data distributions.
• By ratio: scales are adaptively determined based on protein length, e.g., \(\mathbf{S} = \{L/4, L/2, L\}\). This approach adjusts automatically for different protein lengths.

We adopt length-based scales as the default configuration. This design enables training PAR with flexible scale configurations. In the following sections, we describe how this hierarchy of representations is modeled using the autoregressive transformer and backbone decoder.

To formulate the autoregressive order, we leverage the hierarchical nature of proteins, where a protein structure spans multiple levels of representation from coarse tertiary topology to the finest atomic coordinates. We adopt next-scale prediction to model the per-scale distribution based on prior coarser scales, ensuring that the bidirectional dependencies of residues are captured at each scale. We train our autoregressive model (Fig. 2, left), a non-equivariant transformer 𝒯_θ, to produce scale-wise conditioning embeddings \(\mathbf{z}^{i}\) for scale i depending on prior scales \(\mathbf{X^{\lt i}}=\{\mathbf{x}^{1}, \ldots, \mathbf{x}^{i-1}\}\):

where \(\mathbf{bos} \in \mathbb{R}^{\texttt{size(1)} \times 3}\) is a learnable embedding, and \(\mathrm{Up}(\mathbf{x}^{i-1}, \texttt{size(i)})\) interpolates \(\mathbf{x}^{i-1}\) to \(\texttt{size(i)}\) 3D points. All inputs are concatenated along the sequence dimension before being fed into \(\mathcal{T}_{\theta}\). The embedding \(\mathbf{z}^{i}\) is then used to condition the flow-matching decoder to predict the backbone coordinates \(\mathbf{x}^{i}\).

We enable PAR to directly model Cα positions \(\mathbf{x}\), wherein \({p}_{\theta}(\mathbf{x} \mid \mathbf{z}^{i})\) is parameterized by an atomic decoder \(\mathcal{v}_{\theta}\) with flow matching, which maps a standard normal distribution to the target data distribution. We condition \(\mathcal{v}_{\theta}\) with scale-wise embeddings \(\mathbf{z}^{i}\) predicted by the AR transformer \(\mathcal{T}_{\theta}\) at each scale \(i\) (Fig. 2, right). During training, we sample noise \(\boldsymbol{\varepsilon}^{i} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) and a time variable \(t^{i} \in [0,1]\), and compute the interpolated sample as \(\mathbf{x}_{t^{i}}^{i} = t^{i} \cdot \mathbf{x}^{i} + (1 - t^{i}) \cdot \boldsymbol{\varepsilon}^{i}\). As such, we can jointly train \(\mathcal{v}_{\theta}\) and \(\mathcal{T}_{\theta}\) with a flow-matching (FM) objective.

where \({p}_{D}(\mathbf{x})\) denotes the training data distribution and p(t) denotes the \(t\)-sampling distribution in training. The conditioning embedding \(\mathbf{z}^{i}\) is injected into the atomic decoder network \(\mathcal{v}_{\theta}\) through adaptive layer norms.

Training AR models typically uses teacher forcing, where ground-truth data are fed as context to stabilize learning. However, during inference the model is conditioned on its own predictions, creating a training-inference mismatch known as exposure bias. Errors can then accumulate across autoregressive steps, degrading output quality. Our preliminary study shows that teacher forcing greatly reduces the designability of generated structures. To mitigate this, we adapt Noisy Context Learning (NCL) and Scheduled Sampling (SS), techniques from language and image AR modeling, for PAR.

At inference, the autoregressive transformer first produces \(\mathbf{z}^{1}\) at the coarsest scale, which conditions the flow-matching decoder \(\mathcal{v}_{\theta}\) to generate \(\mathbf{x}^{1}\) either via ODE or SDE sampling. We upsample \(\mathbf{x}^{1}\) using \(\texttt{Up}(\mathbf{x}^{1}, \texttt{size(2)})\) and send it back into the autoregressive transformer to predict the next-scale embedding \(\mathbf{z}^{2}\). This coarse-to-fine process iterates \(n\) times until the flow-matching model generates the full-resolution backbone \(\mathbf{x}\). KV cache is applied throughout the autoregressive process for efficiency.

Leveraging the learned flow network \(\mathcal{v}_{\theta}\), sampling can be performed at each scale through the ordinary differential equation (ODE): \(\mathrm{d} \mathbf{x}_{t} = \mathcal{v}_{\theta}(\mathbf{x}_{t}, t)\, \mathrm{d}t\), with the scale superscript \(i\) omitted for simplicity. Moreover, we can define the stochastic differential equation (SDE) for sampling:

where \(g(t)\) is a time-dependent scaling function for the score function \(s_{\theta}(\mathbf{x}_t, t)\), and the noise term \(\gamma\) is a noise scaling parameter, and \(\mathcal{W}_t\) is a standard Wiener process. The score function, defined as the gradient of the log-probability of the noisy data distribution at time \(t\), could be computed as:

We compare PAR with other baselines on unconditional backbone generation benchmark as shown in Table 1. To better reflect the goal of unconditional protein generation as modeling the full data distribution, we adopt FPSD, which jointly measures quality and diversity by comparing generated and reference distributions, analogous to FID in image generation. As shown in Table 1, PAR generates samples that closely match the reference data distribution and maintains competitive designability. On FPSD, PAR achieves scores of 211.8 against AFDB and 231.5 against PDB. By reducing the noise scaling parameter \(\gamma\) from 0.45 to 0.3 in SDE sampling, we reduce sampling stochasticity and further improve sample quality, improving the designability from \(88.0\%\) to \(96.0\%\). After fine-tuning, PAR achieved \(96.6\%\) designability and 161.0 FPSD against the PDB, highlighting its superior distributional fidelity compared to pure diffusion-based baselines.

Proteins possess hierarchical and complex structures, which makes it challenging to directly specify a target shape and design proteins accordingly. By leveraging PAR’s coarse-to-fine generation, a simple prompt (e.g., 16 points) can specify a protein’s coarse layout, from which the model generates the complete structure as shown in Fig. 3. In particular, we first obtain a 16-point input prompt either by downsampling a real protein structure from the test set or by specifying the points manually (the top row in Fig. 3). Using a 5-scale PAR (S = {16, 32, 64, 128, 256}), we initialize the first-scale prediction with the 16-point prompt and autoregressively upsample until the full protein structure is generated, as illustrated in the bottom row of Fig. 3.

📌 Following this process, PAR can generate a new structure that preserves the coarse structural layout (first five examples), and explore entirely novel structures (last three examples).

Besides the point-based layout, PAR can preserve finer-grained prompts like atomic coordinates. Fig. 4 highlights the zero-shot motif scaffolding capabilities of PAR. Using a 5-scale PAR, we downsample a raw protein structure into five scales and teacher-force the ground-truth motif coordinates at each scale before propagating into the next scale. To avoid clashes or discontinuities, we superimpose the ground-truth motif residues and the generated motif segments before replacement.

📌 With no fine-tuning and no conditioning, PAR generates plausible scaffolds that preserve motif structures with high fidelity. This stands in contrast to diffusion or flow-based frameworks, which typically require fine-tuning on additional conditions such as masks or motif coordinates, or rely on decomposition strategies.

Moreover, the generated scaffolds differ substantially from the input structure, showing that PAR generates structurally diverse scaffolds rather than merely copying. For example, the leftmost example in Fig. 4 preserves the yellow motif helix while introducing new secondary structure elements such as β-sheet and loops, in contrast to the original helices. We provide benchmarking results in the appendix.

PAR demonstrates favorable behavior when scaling both model size and training duration, effectively improving its ability to capture the protein data distribution with FPSD scores of 187 against PDB and 170 against AFDB (first two columns in Fig. 5). Further, the fS scores, which reflect quality and diversity, increase with larger model sizes and greater computational budgets.

Meanwhile, we empirically observe that scaling the autoregressive transformer has minimal impacts on the evaluation results. This allows us to reduce computational costs and prioritize increasing the backbone decoder's model capacity that effectively improves generation quality. We provide more discussion on varying model sizes in the appendix.

While Tab. 3 reports results using a uniform number of sampling steps across scales, the multi-scale formulation of PAR actually offers advantages in sampling efficiency. More specifically, (1) sampling at the coarser scale (e.g., first scale) is more efficient than sampling at finer scales (e.g., 2nd scale) due to shorter sequence length; (2) we can use fewer sampling steps at finer scales than coarser scales.

📌 Efficient sampling at finer scales: As shown in Tab. 4, by using SDE sampling only at the first scale, and switching to ODE sampling for the remaining scales, PAR could dramatically reduce the diffusion steps from 400 to 2 steps at the last two scales without harming designability (97%), yielding a 2× inference speedup. This is possible because a high-quality coarse topology places the model near high-density regions, enabling efficient refinement with ODE sampling.

📌 2.5× inference speedup versus single-scale baselines. Compared to the single-scale 400-step baseline, PAR achieves 1.96× and 2.5× sampling speedup at length 150 and 200, respectively. This improvement is driven by speeding up the final scales, where the longer sequence lengths cause computational costs to grow quadratically in transformer architectures. Moreover, the computational costs remain constant at the first scale because it has a fixed size 64, even when generating longer sequences. For single-scale models, naively reducing the SDE sampling steps significantly harms designability.

📌 Orchestrating SDE and ODE sampling. Crucially, SDE sampling at the first scale is necessary for establishing a reliable global topology, given that ODE-only sampling exhibits poor designability.

We visualize the attention maps of the autoregressive transformer at each scale (Fig. 6). We average the attention scores within each scale, normalize them such that the scores across scales sum to 1, and average them over 50 test samples to obtain the scale-level attention distribution during inference. We summarize three key observations:

(i) Most scales barely attend to the first scale, since the input to this scale, a bos token, carries little structural signal.

(ii) Each scale primarily attends to the previous scale, which typically contains richer contextual and structural information.

(iii) Despite focusing most heavily on the current scale, the model still retains non-negligible attention to earlier scales.

📌 This indicates that PAR effectively integrates information across multiple scales and maintains structural consistency during generation.