How to Run Upgrade Readiness Reviews

An Upgrade Readiness Review (URR) is a formal, structured process to validate a smart contract upgrade before it goes live on a mainnet. It is a critical governance checkpoint, distinct from initial security audits, that focuses on the specific changes in the new version. The core goal is to answer one question: Is this upgrade safe and ready for production? This involves verifying the upgrade's correctness, assessing its impact on the existing system and users, and ensuring all operational procedures are in place. For teams using upgradeable proxy patterns like those from OpenZeppelin, this review is non-negotiable for managing protocol risk.

The review process typically follows a phased approach. First, the development team prepares a release candidate, complete with the new contract bytecode, a comprehensive changelog, and the proposed upgrade transaction. This bundle is then subjected to internal review, where engineers verify the code diff, run integration tests against forked mainnet state, and simulate the upgrade on a testnet. Key checks include ensuring storage layout compatibility, validating initialization functions, and confirming that all state migrations (if any) are correct and gas-efficient. Tools like Slither for static analysis and Hardhat or Foundry for fork testing are indispensable here.

Following internal validation, the upgrade proposal enters the external review phase. This often involves a time-locked, on-chain governance proposal on platforms like Tally or Snapshot, giving the protocol's community and stakeholders time to scrutinize the changes. The review materials—including the audit report for the new code, the full diff, and a detailed impact analysis—should be publicly accessible. This transparency allows token holders, integrators, and whitehat researchers to assess risks such as new attack vectors, economic model changes, or unintended interactions with other DeFi protocols.

A successful review culminates in the execution phase. Once the governance vote passes and any timelock expires, a designated party (e.g., a multisig wallet) executes the upgrade transaction. It is a best practice to have a verified rollback plan or emergency downgrade procedure ready in case critical issues are discovered post-upgrade. Monitoring tools like OpenZeppelin Defender Sentinels or Tenderly alerts should be configured immediately after deployment to watch for anomalous activity, completing the upgrade lifecycle from review to live operation.

The core requirement is a production-ready codebase that is feature-complete and stable. The smart contract system should have its core logic fully implemented, with all major features integrated and functional. The code submitted for review must be the exact version intended for deployment, as auditing a moving target is ineffective. Ensure all external dependencies, such as oracle interfaces or token standards, are clearly defined and integrated. A frozen codebase allows auditors to conduct a thorough, static analysis without the noise of ongoing development.

Comprehensive technical documentation is non-negotiable. This includes a detailed technical specification or whitepaper explaining the system architecture, data flows, access controls, and key invariants. You must also provide complete, up-to-date NatSpec comments within the Solidity code. For complex protocols, architectural diagrams (e.g., sequence diagrams for key transactions) are invaluable. This documentation acts as the single source of truth for auditors to understand your system's intended behavior, which is crucial for identifying deviations that could be vulnerabilities.

A robust testing suite demonstrates the code's reliability and your team's diligence. This should include high-coverage unit tests (aim for >90% for critical paths) using frameworks like Foundry or Hardhat, integration tests for cross-contract interactions, and, for upgradeable contracts, specific tests for the upgrade mechanism itself. Include tests for edge cases and failure modes. Providing the test suite and its coverage report allows auditors to verify the system's behavior and quickly identify untested, potentially risky code paths.

For projects using upgradeable proxy patterns (e.g., Transparent, UUPS), you must have the upgrade infrastructure in place and tested. This includes the deployed proxy admin contract (if applicable), a clearly defined and access-controlled upgrade multisig or DAO, and a verified implementation of the upgrade function. The review will scrutinize the security of the upgrade process itself—ensuring initialization is protected, storage layouts are compatible, and admin keys are properly secured is a prerequisite for a meaningful assessment.

Finally, establish clear organizational readiness. Designate a primary technical point of contact from your team who can answer detailed questions during the audit. Prepare a staging environment (e.g., a testnet deployment) where auditors can interact with the contracts. Have a plan for triaging and addressing findings, including a process for critical issue remediation. Being prepared logistically ensures the review process is efficient and that high-severity issues can be addressed and verified promptly.

An upgrade readiness review is a formal, structured assessment conducted by a protocol's core development team or security council before executing a governance-approved upgrade. Its primary goal is to verify that the new code is production-ready and that all deployment risks have been identified and mitigated. This process is critical for protocols using upgradeable proxy patterns, like the Transparent Proxy or UUPS (EIP-1822), where a single bug can compromise the entire system. The review acts as the final quality gate, ensuring the upgrade payload matches the audited code and that the deployment process is safe.

The review should follow a documented checklist. Key technical items include: verifying the bytecode hash of the deployment artifact matches the audited version, confirming all storage layout changes are backward compatible to prevent data corruption, and validating that the new implementation address is correctly set in the proxy admin contract. For complex upgrades, teams should conduct a dry-run on a forked mainnet testnet (using tools like Foundry's forge create --fork-url) to simulate the exact deployment transaction and gas costs. This step often uncovers environment-specific issues missed in unit tests.

Beyond code, the review must assess operational readiness. This includes confirming that multisig signers or governance executors are available, that the transaction calldata is correctly formatted, and that emergency pause mechanisms or rollback procedures are documented and understood. A common practice is to maintain an upgrade runbook—a living document detailing every step, from pre-flight checks to post-deployment verification. For example, after upgrading a Compound-style interest rate model, the runbook would specify commands to call checkAssets() and accrueInterest() on a sample market to verify the new logic is active.

Finally, the review concludes with a formal sign-off from designated technical leads. The output is not just an approval but a risk acknowledgment document that records any known residual risks, such as dependencies on external oracle states or un-audited peripheral changes. This creates accountability and a clear audit trail. For major protocols like Aave or Uniswap, this process is often transparently summarized in governance forums, providing confidence to token holders that the upgrade has undergone rigorous internal scrutiny before their final on-chain vote.

A final code audit is a targeted security review of the specific upgrade payload. Unlike a full protocol audit, it focuses exclusively on the new or modified code in the upgrade. The goal is to identify logic errors, security vulnerabilities, and unintended side effects introduced by the changes. This review should be conducted by experienced smart contract auditors who understand the protocol's architecture and the upgrade's objectives. For high-value protocols, engaging a separate, reputable third-party audit firm for this final check provides an essential layer of independent verification.

Simultaneously, a diff review is a line-by-line comparison between the current on-chain contract and the proposed upgrade contract. This is typically done using tools like git diff or dedicated blockchain explorers for verified contracts. The diff review ensures transparency and correctness by answering key questions: Are the changes exactly what the community or governance approved? Have any last-minute, unauthorized modifications been introduced? Does the diff match the audit report? This step prevents deployment errors and malicious code injections, serving as the final gate before the upgrade transaction is signed.

To perform an effective diff review, start by obtaining the exact bytecode or source code of the live contract from the blockchain (e.g., via Etherscan for Ethereum). Compare it against the compiled bytecode of the new implementation. For source code comparisons, ensure you are comparing the exact commit hash that was audited. Scrutinize every changed line, paying special attention to state variable layouts, function visibility changes, and any modifications to access control or critical math logic. A single misplaced character in a storage layout can lead to catastrophic data corruption.

Common pitfalls to look for include storage collisions (where new variables incorrectly overwrite existing data), broken function selectors due to parameter changes, and inconsistencies in inheritance or interface implementations. Also, verify that all require, revert, and event statements related to security and access control are correctly updated. Document every change and its justification. This documented diff becomes a crucial artifact for governance communication and post-upgrade verification.

For developers, integrating this into a CI/CD pipeline is best practice. Use tools like slither-check-upgradeability or solc with storage layout output to automate parts of the review. However, automation does not replace manual review. The final step must involve senior developers and auditors manually signing off on the diff. This rigorous process, combining automated checks and expert human analysis, is your primary defense against upgrade-related exploits, which have led to losses exceeding hundreds of millions of dollars in past incidents.

After initial unit and integration testing, the next critical phase is deploying your upgrade candidate to a long-running testnet. Unlike ephemeral development networks, these testnets (like Ethereum's Goerli, Sepolia, or Holesky, or a dedicated network for your protocol) operate continuously with multiple validators and simulate real-world network conditions, including transaction load and validator churn. This environment is essential for catching issues that only appear under sustained operation, such as memory leaks, state growth problems, or consensus instability over thousands of blocks.

The deployment process mirrors your mainnet procedure. Use your existing deployment scripts or CI/CD pipeline to target the testnet's RPC endpoint. Key actions include: deploying the new Implementation contract, calling the upgradeTo function on your proxy (like an OpenZeppelin TransparentUpgradeableProxy or UUPS proxy), and verifying the upgrade via the proxy's implementation() function. Always perform a dry-run on a forked version of the testnet first using tools like Hardhat or Foundry to simulate the transaction and catch any immediate revert errors.

Once deployed, initiate a structured testing regimen. This goes beyond basic functionality checks. You must monitor: network performance (block production time, gas usage trends), smart contract metrics (event logs for errors, state size), and validator/client health (if applicable). Run automated load tests that mimic expected mainnet traffic patterns against the upgraded contracts. For at least 48-72 hours, observe the chain's stability, ensuring no forks occur and that the upgrade is fully synced across all nodes.

This phase is also when you conduct final integration testing with dependent systems. If your protocol interacts with oracles (Chainlink), bridges (LayerZero, Wormhole), or major DeFi applications, verify that all external calls and price feeds function correctly post-upgrade. Test edge cases like high-gas scenarios, failed transactions, and interactions with other recently upgraded contracts on the testnet. Document any anomalies and their resolutions meticulously.

The outcome of this step is a Testnet Upgrade Report. This document should include the deployment transaction hash, verification of the new implementation address, a summary of all tests performed, key metrics collected during the observation period, and a list of any issues found and mitigated. This report becomes a core artifact for stakeholder review and is mandatory before proceeding to a mainnet proposal. A clean, stable run on a long-running testnet is the strongest technical indicator of upgrade readiness.

An upgrade readiness review is a systematic audit of the external services and infrastructure your dApp depends on. This includes RPC providers, indexers, oracles, and wallet SDKs. These components are often the first point of failure during a network upgrade if they haven't updated their node software or API endpoints. For example, if your dApp uses The Graph for querying data, you must verify their subgraph is compatible with the new chain state. Similarly, an oracle like Chainlink must support the upgraded network to provide accurate price feeds.

Start by creating an inventory of all third-party dependencies. For each, check their official channels—GitHub, Discord, or status pages—for upgrade announcements. Use a structured checklist to track their status: Pending, In Progress, Ready, or Not Supported. For RPC providers, test the new network endpoints on a testnet. A simple cURL command can verify endpoint responsiveness and correct chain ID: curl -X POST https://new-rpc-endpoint -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_chainId","id":1}'. This confirms the provider is serving the upgraded network.

For indexers and oracles, review their documentation for any required contract address updates or data schema changes. If a critical service declares it won't support the upgrade, you must identify and integrate an alternative before the mainnet fork. This process should conclude with a go/no-go decision based on the compatibility status of your core infrastructure. Documenting this review provides a clear audit trail and is essential for communicating upgrade readiness to your team and users, ensuring a seamless transition.

A rollback plan is not a theoretical exercise; it is a step-by-step operational manual for executing an emergency downgrade. Its primary purpose is to minimize system downtime and financial loss by providing a clear, pre-approved path to restore a known-good state. The plan must be developed before the upgrade deployment and should be reviewed and signed off by all key stakeholders, including developers, security auditors, and protocol governance. Treating this as a formality is a common and dangerous mistake.

The core of the plan is the rollback procedure. This is a sequential list of all on-chain transactions required to revert the system. It must specify: the exact contract addresses and ABI to use, the precise function calls and calldata (e.g., upgradeToAndCall(previousImplementation, "")), the required transaction signers and their multisig thresholds, and the exact order of operations. For complex systems, this may involve multiple steps like pausing contracts, migrating state, and re-initializing modules. Document every parameter; ambiguity during a crisis leads to errors.

Beyond the on-chain steps, the plan must include pre-conditions and triggers. Clearly define what constitutes a "failed upgrade" that warrants a rollback. Triggers can be automated (e.g., a monitoring bot detects a critical bug or fund drain) or manual (e.g., a governance vote). The plan should also list verification steps to confirm the rollback was successful, such as checking that the proxy's implementation address has reverted and that core user functions are operating correctly on the previous version.

Finally, integrate the rollback plan into your broader incident response protocol. Designate a response team, establish communication channels (e.g., a war room), and define escalation paths. The plan should be stored in an accessible, version-controlled location like a GitHub repository. Conduct a tabletop exercise where the team walks through the plan using a testnet fork to identify gaps. A documented and tested rollback plan transforms a potential catastrophe into a managed recovery event.

To operationalize this process, establish a formal URR checklist for your team. This should be a living document that evolves with your protocol. Core items include: verifying the upgrade's on-chain target address, confirming the deployer's administrative rights, reviewing all code changes line-by-line against the diff, simulating the upgrade on a forked testnet, and documenting the rationale for each change. Using a tool like OpenZeppelin Defender can automate many of these checks and provide an audit trail.

The next step is to integrate this review into your CI/CD pipeline. For example, you can create a GitHub Action that runs Slither or other static analyzers on the proposed upgrade diff, automatically deploys it to a temporary forked environment using Hardhat or Foundry, and executes a suite of integration tests. This "upgrade simulation" should test not just the new functionality but also that existing storage layouts are preserved and that user funds remain secure.

Finally, consider the governance and communication aspects. For decentralized protocols, the upgrade proposal and its accompanying URR report should be published for community review well in advance of a vote. Transparency builds trust. For your next upgrade, challenge yourself to go beyond basic functionality: run a failure mode analysis to ask "what if the upgrade fails mid-execution?" and have a verified rollback plan ready. Continuous improvement of your URR process is the best way to ensure the long-term security and upgradability of your protocol.