My pulumi stack was created with an older plugin version that I can't install on M1 mac - pulumi

Question

I have a Pulumi project which I haven't touched (deployed etc) for a while.
Now I need to make some changes but I get the "403 HTTP error fetching plugin" described here
The description in the docs makes sense: I have bought a new Apple M1 laptop since last time I worked on it, my stack was built with the digitalocean provider v3.1.1 but I can't install that version on my new laptop.
The docs say that if you have access to an Intel system, set up your project there, update the providers and run pulumi up. The implication is that you can install both old and latest versions of the provider plugin, pulumi up will update the stack using the latest version. After that I should be able to manage the stack from my new laptop using the latest provider version.
I asked on Pulumi Slack about this and confirmed the above.
But it doesn't seem to work for me.
First I started a Docker container with --platform=linux/amd64 to emulate Intel. Inside that I checked out my project, installed pulumi and the old and new provider versions.
In my docker container pulumi plugin ls shows:
NAME KIND VERSION SIZE INSTALLED LAST USED
cloudflare resource 4.7.0 38 MB n/a 1 day ago
cloudflare resource 2.8.0 46 MB n/a 1 day ago
digitalocean resource 4.14.0 42 MB n/a 1 day ago
digitalocean resource 3.1.1 45 MB n/a 1 day ago
From there I successfully ran pulumi up.
However if I go back to my local shell and try pulumi preview I get:
error: could not load plugin for digitalocean provider 'urn:pulumi:staging::myproject::pulumi:providers:digitalocean::default': no resource plugin 'pulumi-resource-digitalocean' found in the workspace at version v3.1.1 or on your $PATH, install the plugin using pulumi plugin install resource digitalocean v3.1.1
So despite updating the stack Pulumi is still trying to use the old provider version that I can't install.
How do I get around this?

Answer 1

You also need to update your pulumi program's dependencies.
Simply installing the new plugin isn't enough.
If you do pulumi stack export you'll see a JSON file with all your resources. Those resources have a provider attached to them with a specific version of the plugin. As an example:
{
"version": 3,
"deployment": {
"manifest": {
"time": "2022-06-23T12:03:30.071863-07:00",
"magic": "eccb7d9cc1cab43d7465783c52b0648063d5e7228dd3bb2fc7600583a8bca5d5",
"version": "v3.34.1"
},
"secrets_providers": {
"type": "service",
"state": {
"url": "https://api.pulumi.com",
"owner": "jaxxstorm",
"project": "s3_event_bridge",
"stack": "dev"
}
},
"resources": [
{
"urn": "urn:pulumi:dev::s3_event_bridge::pulumi:pulumi:Stack::s3_event_bridge-dev",
"custom": false,
"type": "pulumi:pulumi:Stack",
"outputs": {
"bucketName": "test001-c86ab36"
},
"sequenceNumber": 1
},
{
"urn": "urn:pulumi:dev::s3_event_bridge::pulumi:providers:aws::default_5_9_1",
"custom": true,
"id": "484b75ed-d5cd-4ee1-96d1-b3f641236ab6",
"type": "pulumi:providers:aws",
"inputs": {
"region": "us-west-2",
"version": "5.9.1"
},
"outputs": {
"region": "us-west-2",
"version": "5.9.1"
},
"sequenceNumber": 1
},
{
"urn": "urn:pulumi:dev::s3_event_bridge::aws:s3/bucket:Bucket::test001",
"custom": true,
"id": "test001-c86ab36",
"type": "aws:s3/bucket:Bucket",
"inputs": {
"__defaults": [
"bucket",
"forceDestroy"
],
"acl": "private",
"bucket": "test001-c86ab36",
"forceDestroy": false,
"tags": {
"Environment": "Dev",
"Name": "My bucket",
"__defaults": []
}
},
"outputs": {
"accelerationStatus": "",
"acl": "private",
"arn": "arn:aws:s3:::test001-c86ab36",
"bucket": "test001-c86ab36",
"bucketDomainName": "test001-c86ab36.s3.amazonaws.com",
"bucketRegionalDomainName": "test001-c86ab36.s3.us-west-2.amazonaws.com",
"corsRules": [],
"forceDestroy": false,
"grants": [],
"hostedZoneId": "Z3BJ6K6RIION7M",
"id": "test001-c86ab36",
"lifecycleRules": [],
"loggings": [],
"objectLockConfiguration": null,
"region": "us-west-2",
"replicationConfiguration": null,
"requestPayer": "BucketOwner",
"serverSideEncryptionConfiguration": null,
"tags": {
"Environment": "Dev",
"Name": "My bucket"
},
"tagsAll": {
"Environment": "Dev",
"Name": "My bucket"
},
"versioning": {
"enabled": false,
"mfaDelete": false
},
"website": null
},
"parent": "urn:pulumi:dev::s3_event_bridge::pulumi:pulumi:Stack::s3_event_bridge-dev",
"provider": "urn:pulumi:dev::s3_event_bridge::pulumi:providers:aws::default_5_9_1::484b75ed-d5cd-4ee1-96d1-b3f641236ab6",
"propertyDependencies": {
"acl": null,
"tags": null
},
"sequenceNumber": 1
},
{
"urn": "urn:pulumi:dev::s3_event_bridge::aws:s3/bucketNotification:BucketNotification::bucketNotification",
"custom": true,
"id": "test001-c86ab36",
"type": "aws:s3/bucketNotification:BucketNotification",
"inputs": {
"__defaults": [],
"bucket": "test001-c86ab36",
"eventbridge": false
},
"outputs": {
"bucket": "test001-c86ab36",
"eventbridge": false,
"id": "test001-c86ab36",
"lambdaFunctions": [],
"queues": [],
"topics": []
},
"parent": "urn:pulumi:dev::s3_event_bridge::pulumi:pulumi:Stack::s3_event_bridge-dev",
"dependencies": [
"urn:pulumi:dev::s3_event_bridge::aws:s3/bucket:Bucket::test001"
],
"provider": "urn:pulumi:dev::s3_event_bridge::pulumi:providers:aws::default_5_9_1::484b75ed-d5cd-4ee1-96d1-b3f641236ab6",
"propertyDependencies": {
"bucket": [
"urn:pulumi:dev::s3_event_bridge::aws:s3/bucket:Bucket::test001"
],
"eventbridge": null
},
"sequenceNumber": 1
}
]
}
}
If you look at my BucketNotification resource, you can see a provider field which has a version in it for the AWS provider I've used:
"provider": "urn:pulumi:dev::s3_event_bridge::pulumi:providers:aws::default_5_9_1::484b75ed-d5cd-4ee1-96d1-b3f641236ab6"
Which in this case is 5.9.1
So, in order to fix this problem, you need to update your resources to have a new version of the provider.
To do this, it depends on the language you're using with Pulumi.
If you're using TypeScript or JavaScript, update your #pulumi/digitalocean dependency in your package.json
If you're using Python, update pulumi_digitalocean in your requirements.txt
Make sure you update with your package manager with npm update or pip3 upgrade
The same applies if you're using DotNet, Go and Java.
then you need to run a successful pulumi up. Pulumi will update the provider version associated with each resource as you saw above, you can verify this by again doing a pulumi stack export
From here, you should be able to successfully use your M1 mac without the legacy plugins.