How to Coordinate a Cosmos SDK Chain Software Upgrade

A software upgrade is a fundamental process for any live blockchain. In the Cosmos ecosystem, upgrades are not just a technical deployment but a formal on-chain governance proposal. This process, defined by the x/upgrade module, allows a decentralized network to coordinate a hard fork, introducing new features, patching security vulnerabilities, or implementing consensus-breaking changes. Unlike a simple server restart, a chain upgrade requires explicit consensus from the network's validators to switch to a new binary at a predetermined block height.

The upgrade lifecycle follows a clear sequence: a proposal is drafted, voted on by the community's staked token holders, and, if passed, is scheduled for execution. Validators must then prepare by building the new binary and configuring their nodes. At the designated block height, the chain halts, and validators must manually restart their nodes with the new software. Failure for a supermajority (greater than 2/3) of voting power to upgrade results in a chain halt, making coordination paramount. This guide details each step, from drafting a SoftwareUpgradeProposal to post-upgrade validation.

Key participants have distinct roles. Governance proposers draft and submit the upgrade proposal. Validators are responsible for running the upgraded software; their preparedness directly impacts network liveness. Full node operators must also upgrade to continue syncing and serving data. Developers maintain the new codebase in a version control system like GitHub, tagging the release. Tools like Cosmovisor can automate the binary swap, but understanding the manual process is essential for troubleshooting and validator operations.

A successful upgrade hinges on precise communication and testing. Before the mainnet proposal, the new software should undergo rigorous testing on a testnet that mirrors the mainnet state. The upgrade height must be calculated carefully, allowing ample time for voting and validator preparation. Clear communication channels—such as validator forums, Discord, and commonwealth chats—are used to broadcast the upgrade plan, binary hashes, and instructions. This guide provides the technical checklist and commands validators need to execute a smooth transition.

Before initiating a chain upgrade, you must have operational control over a live network node. This means running a full node, typically a validator, with access to the cosmovisor binary manager and the node's data directory. You will need the ability to stop and restart the node's service, which requires appropriate system permissions (e.g., systemctl on Linux). Ensure you have a secure, backed-up environment, as the upgrade process involves replacing the core binary and can be a point of failure.

A deep understanding of the upgrade proposal and governance process is critical. Upgrades are enacted via on-chain governance proposals of type SoftwareUpgradeProposal. You must know how to draft a proposal with the correct parameters: a unique name, the target height for the upgrade, and the info field containing the upgrade plan metadata, often a link to the new binary. Familiarize yourself with your chain's specific deposit requirements, voting periods, and quorum thresholds using the gaiad or junod CLI tools.

You must have the new, compiled binary ready for deployment. This involves either building the new version from the official source code repository or verifying a pre-compiled release from a trusted source. The binary must be compatible with the chain's current state and database. It is a security best practice to verify the checksum of the downloaded binary against the checksum published by the core development team to prevent supply-chain attacks.

The use of Cosmovisor is highly recommended, if not mandatory, for seamless upgrade automation. Cosmovisor monitors the blockchain for the agreed-upon upgrade height and automatically swaps the running binary. You must install and configure Cosmovisor before the upgrade, setting up its directory structure ($DAEMON_HOME/cosmovisor) with the genesis and upgrades folders. The new binary must be placed in the correct path (e.g., $DAEMON_HOME/cosmovisor/upgrades/<upgrade-name>/bin/) before the target height is reached.

Finally, comprehensive testing in a simulated environment is non-negotiable. You should run the upgrade procedure end-to-end on a testnet or a local simd instance that mirrors your mainnet state. This tests the binary compatibility, the Cosmovisor configuration, and the migration scripts (x/upgrade module). Testing helps identify issues with state migrations, consensus breaking changes, or unexpected halts, preventing catastrophic failures on the live network.

A software upgrade proposal is a specific type of governance proposal defined by the x/upgrade module in the Cosmos SDK. Its primary purpose is to formally signal to the network the intention to halt the chain at a predetermined block height and switch to a new binary. The proposal must be submitted by a validator or a delegator with sufficient staked tokens to meet the minimum deposit requirement, which is defined by the chain's governance parameters (e.g., min_deposit). This deposit is temporarily locked and acts as a spam-prevention mechanism.

The proposal's JSON payload is critical. It must specify the upgrade name (a unique identifier like v12 or gamma), the height at which the old chain will halt, and an info field often containing a link to the release notes or upgrade instructions. Optionally, it can include metadata for automated upgrade tools. Here is a simplified example of the command to submit a proposal using the cosmovisor-ready binary format:

bashCopy
d tx gov submit-proposal software-upgrade v12 \
  --title "Upgrade to v12" \
  --description "Upgrade to enable IBC rate limiting" \
  --upgrade-height 10000000 \
  --upgrade-info '{"binaries":{"linux/amd64":"https://github.com/chain/chaind/releases/download/v12.0.0/chaind"}}' \
  --deposit 1000000stake \
  --from validator-key

Once submitted, the proposal enters a deposit period. The initial deposit from the proposer may not be enough; other community members must contribute to the deposit until it reaches the minimum threshold within a set timeframe (e.g., 14 days). If the deposit is not met, the proposal is rejected and deposits are refunded. This period allows for initial community vetting before the proposal proceeds to a formal vote. It is common for core development teams or validator groups to coordinate to ensure the deposit is met promptly.

After the deposit threshold is reached, the proposal moves to a voting period, typically lasting 1-2 weeks. During this time, stakers (validators and delegators) vote Yes, No, NoWithVeto, or Abstain. A successful upgrade proposal requires: a quorum (minimum percentage of total voting power participating), a simple majority of Yes over No votes, and less than 33.4% NoWithVeto votes (which signals a harmful proposal). Validators should actively monitor and vote, as their voting power directly influences the outcome.

A successful vote does not trigger the upgrade automatically. It simply authorizes the upgrade to occur at the specified block height. Validators and node operators are now responsible for preparing their systems. They must download the new binary, verify its checksum, and configure their process manager (like cosmovisor) before the target height. The chain will panic and halt at the upgrade height if a validator's node is not running the correct new version, emphasizing the importance of coordination and preparation following a passed proposal.

An UpgradePlan is a governance proposal of type SoftwareUpgradeProposal. It contains the definitive instructions for the network to halt at a specific height, switch binaries, and resume. The primary parameters you must define are name, height, and info. The name acts as a unique identifier for the upgrade and must match the upgrade handler defined in your application's app.go. The height specifies the exact block number at which the chain will halt, which must be sufficiently in the future to allow all validators time to prepare.

The info field is a string field for metadata, typically used to store the URI for the new binary or upgrade instructions. While not parsed by the chain, it is a critical coordination tool. A common practice is to store a link to a verified release page (e.g., a GitHub release) or an IPFS hash. This ensures all participants can independently fetch and validate the correct software. You can also include checksums here for added security. Never rely solely on the info field for critical logic; the upgrade name is the programmatic identifier.

Two optional but important parameters are time and upgraded_client_state. The time parameter can be used instead of height to schedule the upgrade based on wall-clock time, though height is more common for its predictability. The upgraded_client_state is used in IBC-enabled chains for coordinated client upgrades. If your upgrade includes changes to the IBC light client, this field must contain the new client state to be committed at the upgrade height.

Here is the protobuf definition for the Plan type from cosmos.upgrade.v1beta1:

protoCopy
message Plan {
  string name = 1;
  google.protobuf.Timestamp time = 2;
  int64 height = 3;
  string info = 4;
  google.protobuf.Any upgraded_client_state = 5;
}

When drafting your proposal, you will embed this Plan within a SoftwareUpgradeProposal message. Ensure the chosen height provides a minimum safety margin (e.g., 10,000 blocks) after the proposal passes, accounting for voting and propagation delays.

Before submission, validate your plan parameters against your chain's constants and application code. Confirm the name corresponds to an upgrade keeper's ScheduleUpgrade call or an upgrade handler defined in your app.go. Test the upgrade process on a long-running testnet using identical parameters to surface timing or coordination issues. A well-defined plan is the foundation for a smooth, deterministic transition.

The core of a software upgrade is the new binary that validators must run. Begin by checking out the specific release tag (e.g., git checkout v2.0.0) from the official repository. Build the binary using the project's standard procedure, typically make install or go install ./cmd/<daemon-name>. Verify the build by checking the version output: ./<daemon-name> version. This should match the release tag and the upgrade-info planned for the governance proposal. It is critical that the binary is built from the exact, audited commit that was proposed.

Before distribution, rigorous testing is non-negotiable. Test the binary on a testnet that mirrors the mainnet state and upgrade height. Validate that the chain halts at the target height and that the x/upgrade module triggers the automatic binary switch. Test the migration logic for any state-breaking changes defined in the upgrade handler. Additionally, run integration tests for core functionality like staking, governance, and IBC. Successful testnet execution builds confidence and identifies edge cases before mainnet deployment.

Create a reproducible build environment for validators. Document the exact Go version (e.g., Go 1.21), dependencies, and build flags used. Provide clear build instructions in the release notes. The compiled binary should be distributed via multiple trusted channels to ensure availability and integrity. Primary distribution points include the project's official GitHub Releases page and a decentralized storage service like IPFS. Always provide SHA256 checksums for all distributed files so validators can verify the binary was not tampered with during download.

Validators must be given ample time and clear instructions to prepare. Announce the final binary release well before the upgrade height. The communication should include: the download URLs, the SHA256 checksum, the target block height, and the exact command to set the upgrade-info flag in their systemd service file (e.g., --upgrade-info "{\"name\": \"v2\", \"height\": 5000000}"). Encourage validators to test the upgrade process on a backup node or local testnet. Proactive communication minimizes last-minute confusion and reduces the risk of validators being unprepared at the critical height.

For chains using Cosmovisor, the process is streamlined. Validators simply need to place the new binary in the correct upgrades/<name>/bin directory before the upgrade height, where <name> matches the upgrade plan's name field. The release announcement should specify the exact path and confirm the binary name. Cosmovisor automates the switch, but validators must still ensure the binary is executable (chmod +x) and tested. Using Cosmovisor significantly reduces operational overhead and is the recommended upgrade manager for Cosmos SDK chains.

Once the governance proposal passes and the upgrade height is finalized, node operators must prepare their systems. The core action is to replace the current binary with the new version before the chain reaches the specified block. This is a manual process; the chain will halt if a validator's node is still running the old software. Operators should download the new binary, verify its checksum against the official release, and stop the cosmovisor service or node process. It's critical to have a rollback plan, such as keeping the old binary accessible, in case the new version fails to start.

For chains using Cosmovisor, the upgrade process is largely automated. You place the new binary in the upgrades/<upgrade-name>/bin directory within Cosmovisor's home folder. Cosmovisor will automatically detect the upgrade height, stop the current process, swap the binary, and restart. The command structure is cosmovisor run start. Ensure Cosmovisor is configured with the correct DAEMON_NAME (e.g., gaiad, osmosisd) and DAEMON_HOME environment variables. Without Cosmovisor, you must manually stop the node at the correct height, replace the binary, and restart the service.

Coordination among validators is essential to minimize chain downtime. Operators should monitor the block height using RPC endpoints or block explorers. Communication channels like Discord or Telegram are used to confirm peers are ready. When the chain halts at the upgrade height, each validator restarts their node with the new binary. The network resumes once more than 2/3 of the voting power has upgraded. Post-upgrade, verify the node is syncing and participating in consensus. Check logs for errors and confirm the new software version is active using the --version flag on your binary.

A state migration is a function that programmatically transforms the blockchain's state from the format expected by the old software version to the format expected by the new version. This is required when an upgrade introduces breaking changes to the way data is stored. Common scenarios include: renaming or restructuring module keys in the KVStore, changing the data structure of a protobuf type, or altering the logic for how certain parameters are derived. Without a migration, the new binary will fail to read the existing state, causing a consensus failure at the upgrade height.

Migrations are implemented within the affected Cosmos SDK modules. The primary entry point is the RegisterMigration method in a module's AppModule. You must specify a from-version and a to-version, and provide a migration function. For example, to migrate the bank module from version 1 to 2:

goCopy
app.upgradeKeeper.SetModuleVersionMap(ctx, app.mm.GetVersionMap())
app.mm.GetModule("bank").(module.AppModuleBasic).RegisterMigration(
    ctx,
    app.appCodec,
    1,
    2,
    func(sdk.Context) error {
        // Migration logic here
        return nil
    },
)

The upgrade handler's Apply function must call moduleManager.RunMigrations to execute all registered migrations for the target version.

The migration function receives a sdk.Context and has full read/write access to the KVStore for its module. A typical task is iterating over old data, transforming it, and saving it under a new key. Always perform migrations on a testnet first and use the cosmovisor export command to verify the post-migration state. Critical considerations include: ensuring the migration is idempotent (safe to run multiple times), gas consumption (long-running migrations may hit block gas limits), and thorough testing with a state export from the live network to catch edge cases.

For complex upgrades like Cosmos SDK v0.45 to v0.46 (which introduced the x/authz and x/feegrant module migrations), the upgrade handler coordinates migrations across multiple modules. The process is: 1) Halt the chain at the upgrade height, 2) The new binary starts and runs the Apply function in the upgrade handler, 3) RunMigrations is called, executing all registered migrations sequentially, 4) The app.StoreLoader is updated to load the migrated state, 5) The chain resumes consensus. Failed migrations will halt the chain permanently, making pre-upgrade testing on a snapshot absolutely essential.

You can validate your migration logic using the simapp testing utilities or by writing a standalone test that loads a genesis state, applies the migration, and asserts the new state. Resources include the official Cosmos SDK Upgrade Guide and example migrations in the cosmwasm and Osmosis codebases. Remember, a well-tested migration is the difference between a smooth upgrade and a chain halt.

The most critical factor in a successful upgrade is communication. A clear, public upgrade proposal on-chain is mandatory, but it is not sufficient. You must proactively inform your community through all available channels: governance forums, social media, Discord, and Telegram. Publish a detailed upgrade guide that includes the block height, the new binary version, the migration process for node operators, and instructions for users. Transparency about the changes, especially breaking changes to APIs or client expectations, builds trust and minimizes support requests during the critical upgrade window.

Testing cannot be overstated. Before proposing the upgrade on your mainnet, you must conduct exhaustive testing in a controlled environment. This involves: - Spinning up a testnet that mirrors your mainnet's state and parameters. - Running the upgrade procedure end-to-end multiple times. - Testing all critical user pathways (transactions, queries, IBC transfers) post-upgrade. Utilize the Cosmos SDK's built-in upgrade module on your testnet to simulate the halt and binary swap. Tools like simd for simulation testing and state export/import for creating realistic test environments are invaluable for uncovering edge cases.

For node operators, automation is key to reliability and reducing human error. Instead of manual binary swaps, implement an automated upgrade handler. This can be a shell script that monitors the chain for the target height, gracefully stops the cosmovisor or node process, swaps the binary, and restarts. Using cosmovisor is a best practice, as it is designed specifically for this purpose and can handle both planned and emergency upgrades. Ensure your upgrade documentation provides explicit commands, including the full git tag or release URL for the new binary to prevent version mismatch issues.

Finally, establish a clear rollback and contingency plan. Despite thorough testing, unforeseen issues can arise. Before the upgrade, ensure a majority of validators have backed up their data directory and priv_validator_key.json. Agree on a rollback block height in case of a critical failure. Monitor network health closely after the upgrade using explorers and validator tools. Be prepared to provide immediate support in public channels. A successful upgrade is a testament to a chain's governance and operational maturity, reinforcing confidence among developers, validators, and users in the network's long-term stability.