I have registered a model to mlflow model registry.
When I call ‘load_model’ function to try to fetch the model from model registry and try to make prediction, mlflow cannot find the model from the artifact path I provided:
model_name = "sample-ann-1"
version = 1
loaded_model = mlflow.pyfunc.load_model("models:/{}/{}".format(model_name, version))
And return the following error:
"mlflow.exceptions.MlflowException: The following failures occurred while downloading one or more artifacts from s3://{bucket}/5/8429aef5d8304990ae035c638db093e7/artifacts/../saved-model/model20/: {'': "ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')"}"
When I open s3 browser to check the file in artifact path (s3://{bucket}/5/8429aef5d8304990ae035c638db093e7/artifacts/../saved-model/model20/), I found the model is in the path, not sure why mlflow return 404 not found error
The artifact you provided in loaded_model is not the artifact path you specified in S3. You should try this:
logged_model = "s3://{bucket}/5/8429aef5d8304990ae035c638db093e7/artifacts/../saved-model/model20/"
loaded_model = mlflow.pyfunc.load_model(logged_model)
Related
I try to execute the following command line:
mssparkutils.fs.ls("abfss://mycontainer#myadfs.dfs.core.windows.net/myfolder/")
I get the error:
Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls.
: java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation.", 403, GET, https://myadfs.dfs.core.windows.net/mycontainer?upn=false&resource=filesystem&maxResults=5000&directory=myfolder&timeout=90&recursive=false, AuthorizationFailure, "This request is not authorized to perform this operation.
I followed the steps described in this link
by granting access to me and my Synapse workspace the role of "Storage Blob Data Contributor" in the container or file system level:
Even that, I still get this persistent error. Am I missing other steps?
I got the same kind of error in my environment. I just followed this official document and done the repro, now it's working fine for me. You can follow the below code it will solve your problem.
Sample code:
from pyspark.sql import SparkSession
account_name = 'your_blob_name'
container_name = 'your_container_name'
relative_path = 'your_folder path'
linked_service_name = 'Your_linked_service_name'
sas_token = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name)
Access to Blob Storage
path = 'wasbs://%s#%s.blob.core.windows.net/%s' % (container_name,account_name,relative_path)
spark.conf.set('fs.azure.sas.%s.%s.blob.core.windows.net' % (container_name,account_name),sas_token)
print('Remote blob path: ' + path)
Sample output:
Updated answer
Reference to configure Spark in pyspark notebook:
https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/notebook-this-request-is-not-authorized-to-perform-this/ba-p/1712566
My source is parquet files in ADLS gen2. All the parquet files are part files of size 10-14 MB. The total size should be around 80 GB
Sink is Azuresynapse table.
Copy method is Polybase. Getting below error within 5 sec of execution like below:
ErrorCode=PolybaseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse. Operation: 'Create external table'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message:
HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, URL',Source=.Net SqlClient Data Provider,SqlErrorNumber=105019,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=105019,State=1,Message=External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message:
HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD,
I've seen this error due to failed authentication, check whether the authorization header and/or signature is wrong.
For example, create the scope credential using your ADLS Gen2 storage account access key:
CREATE DATABASE SCOPED CREDENTIAL [MyADLSGen2Cred] WITH
IDENTITY='user',
SECRET='zge . . . 8V/rw=='
The external data source is created as follows:
CREATE EXTERNAL DATA SOURCE [MyADLSGen2] WITH (
TYPE=HADOOP,
LOCATION='abfs://myblob#pabechevb.dfs.core.windows.net',
CREDENTIAL=[MyADLSGen2Cred])
You can specify wasb instead of abfs, and if you're using SSL, specify it as abfss. Then the external table is created as follows:
CREATE EXTERNAL TABLE [dbo].[ADLSGen2] (
[Content] varchar(128))
WITH (
LOCATION='/',
DATA_SOURCE=[MyADLSGen2],
FILE_FORMAT=[TextFileFormat])
You can find additional information in my book "Hands-On Data Virtualization with Polybase".
Setup
On Yocto environment, I have a bitbake recipe which fetch a patch on github:
SRC_URI = " \
[...]
https://raw.githubusercontent.com/openembedded/meta-oe/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch;name=patch \
"
When fetching this patch I have the error: ERROR 404: Not Found (since a few days).
Issue
WARNING: Failed to fetch URL https://raw.githubusercontent.com/openembedded/meta-oe/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch;name=patch, attempting MIRRORS if available
ERROR: Fetcher failure: Fetch command failed with exit code 8, output:
https://raw.githubusercontent.com/openembedded/meta-oe/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch:
2020-08-19 12:21:44 ERROR 404: Not Found.
ERROR: Function failed: Fetcher failure for URL: 'https://raw.githubusercontent.com/openembedded/meta-oe/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch;name=patch'. Unable to fetch URL from any source.
Tests
I tried to access the link directly on internet, it doesn't work.
Also tried with the following link, doesn't work (without raw.githubusercontent.com option):
https://github.com/openembedded/meta-oe/blob/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch
I tried to find this patch on github with no success:
Find in openembedded/meta-openembeded not openembedded/meta-oe (https://github.com/openembedded/meta-openembedded/blob/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch) not the same patch
Find in http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-edison/tree/meta-intel-edison-distro/recipes-support/tcpdump/tcpdump-4.6.1/unnecessary-to-check-libpcap.patch?h=master not the same project
Thanks
please find the new link below:
https://raw.githubusercontent.com/openembedded/meta-openembedded/master/meta-networking/recipes-support/tcpdump/tcpdump/unnecessary-to-check-libpcap.patch
Seems like the url changed.
Using Terraform 11.14
My terraform file contains the following resource:
resource "google_storage_bucket" "assets-bucket" {
name = "${local.assets_bucket_name}"
storage_class = "MULTI_REGIONAL"
force_destroy = true
}
And this bucket has already been created (it exists on the infrastructure based on a previous apply)
However the state (remote on gcs) is inconsistent and doesn't seem to include this bucket.
As a result, terraform apply fails with the following error:
google_storage_bucket.assets-bucket: googleapi: Error 409: You already own this bucket. Please select another name., conflict
How can I reconcile the state? (terraform refresh doesn't help)
EDIT
Following #ydaetskcoR's response, I did:
terraform import module.bf-nathan.google_storage_bucket.assets-bucket my-bucket
The output:
module.bf-nathan.google_storage_bucket.assets-bucket: Importing from ID "my-bucket"...
module.bf-nathan.google_storage_bucket.assets-bucket: Import complete! Imported google_storage_bucket (ID: next-assets-bf-nathan-botfront-cloud)
module.bf-nathan.google_storage_bucket.assets-bucket: Refreshing state... (ID: next-assets-bf-nathan-botfront-cloud)
Error: module.bf-nathan.provider.kubernetes: 1:11: unknown variable accessed: var.cluster_ip in:
https://${var.cluster_ip}
The refreshing step doesn't work. I ran the command from the project's root where a terraform.tfvars file exists.
I tried adding -var-file=terraform.tfvars but no luck. Any idea?
You need to import it into the existing state file. You can do this with the terraform import command for any resource that supports it.
Thankfully the google_storage_bucket resource does support it:
Storage buckets can be imported using the name or project/name. If the project is not passed to the import command it will be inferred from the provider block or environment variables. If it cannot be inferred it will be queried from the Compute API (this will fail if the API is not enabled).
e.g.
$ terraform import google_storage_bucket.image-store image-store-bucket
$ terraform import google_storage_bucket.image-store tf-test-project/image-store-bucket
Failed to deploy web package to IIS website.
Error: An error was encountered when processing operation 'Create File' on D:\Websites\project\project.pdb.
Error: The error code was 0x800704C8. Error: The requested operation
cannot be performed on a file with a user- mapped section open.
I have tried to add this
MSDEPLOY_RENAME_LOCKED_FILES = 1 on variables