I would like to move files that are available in the system working directory in the azure pipeline to the Kubernetes pod.
Method one (Kubectl cp command)
kubectl cp D:\a\r1\a\test-files\westus\test.txt /test-745f6564dd:/inetpub/wwwroot/
D:\a\r1\a\test-files\westus\test.txt -- my system working directory file location
(name-space)/test-745f6564dd:/inetpub/wwwroot/ -- kubernetes pod location
I have tried to use kubectl cp command but facing an error.
error: one of src or dest must be a local file specification
Method two command line tool in azure devops
Even i tried to use command line to copy files from one directory to another directory.
cd C:\inetpub\wwwroot>
copy C:\inetpub\wwwroot\west\test.txt C:\inetpub\wwwroot\
Once this task is executed in the azure pipeline, its throwing error.
he syntax of the command is incorrect.
Method three azure cli
I have tried to use azure cli and login into Kubernetes and tried to try one of the below codes. But not throwing any errors even file is not copied too.
az aks get-credentials --resource-group test --name test-dev
cd C:\inetpub\wwwroot
dir
copy C:\inetpub\wwwroot\west\test.txt C:\inetpub\wwwroot\
Is there any way do this operation.
For the first error:
error: one of src or dest must be a local file specification
Try to run the kubectl cp command from the same directory where your file is there and instead of giving the whole path try like below:
kubecto cp test.txt /test-745f6564dd:/inetpub/wwwroot/test.txt
I uploaded a folder structure with a single file inside to an existing gcloud storage bucket.
C:\Users\Administrator\Desktop>gcloud alpha storage cp -r testfolder gs://auction-engine-upload
Copying file://testfolder\testSubfolder\MAXPOWER.png to gs://auction-engine-upload/testfolder/testSubfolder/MAXPOWER.png
Completed files 1/1 | 10.0kiB/10.0kiB
Then I tried to verify the file was uploaded by using the ls command:
gcloud alpha storage ls gs://auction-engine-upload
This lists about 40 directories that are not the /testfolder directory, so I tried a few different ways to get only the /testfolder to list:
gcloud alpha storage ls gs://auction-engine-upload/testfolder
gcloud alpha storage ls gs://auction-engine-upload/testfolder/
gcloud alpha storage ls gs://auction-engine-upload/testfolder/*
But I keep getting this error:
ERROR: (gcloud.alpha.storage.ls) One or more URLs matched no objects.
Am I screwing up syntax or is the file actually not uploaded?
I don't have access to change the permissions in the bucket, so I had to have the account owner create another bucket and give me permission to create the file there.
I have this command
gsutil rsync -r -x '".*.jpg$"' File Share\data\Home Drive gs://sdefs01/Home Drive
this is to exclude any .jpg file to be copied to my google bucket.
however, it returns an error:
commandexceptions: the rsync command accept at most 2 arguments.
the command example that I refer to is from google cloud support page.
please help.
You need to put the source directory path inside double quotes as it contains spaces.
I have backup files in different directories in one drive. Files in those directories can be quite big up to 800GB or so. So I have a batch file with a set of scripts which upload/syncs files to S3.
See example below:
aws s3 sync R:\DB_Backups3\System s3://usa-daily/System/ --exclude "*" --include "*/*/Diff/*"
The upload time can vary but so far so good.
My question is, how do I edit the script or create a new one which checks in the s3 bucket that the files have been uploaded and ONLY if they have been uploaded then deleted them from the local drive, if not leave them on the drive?
(Ideally it would check each file)
I'm not familiar with aws s3, or aws cli command that can do that? Please let me know if I made myself clear or if you need more details.
Any help will be very appreciated.
Best would be to use mv with --recursive parameter for multiple files
When passed with the parameter --recursive, the following mv command recursively moves all files under a specified directory to a specified bucket and prefix while excluding some files by using an --exclude parameter. In this example, the directory myDir has the files test1.txt and test2.jpg:
aws s3 mv myDir s3://mybucket/ --recursive --exclude "*.jpg"
Output:
move: myDir/test1.txt to s3://mybucket2/test1.txt
Hope this helps.
As the answer by #ketan shows, Amazon aws client cannot do batch move.
You can use WinSCP put -delete command instead:
winscp.com /log=S3.log /ini=nul /command ^
"open s3://S3KEY:S3SECRET#s3.amazonaws.com/" ^
"put -delete C:\local\path\* /bucket/" ^
"exit"
You need to URL-encode special characters in the credentials. WinSCP GUI can generate an S3 script template, like the one above, for you.
Alternatively, since WinSCP 5.19, you can use -username and -password switches, which do not need any encoding:
"open s3://s3.amazonaws.com/ -username=S3KEY -password=S3SECRET" ^
(I'm the author of WinSCP)
So I run the following:
gsutil -m cp -R file.png gs://bucket/file.png
And I get the following error message:
Copying file://file.png [Content-Type=application/pdf]...
Uploading file.png: 42.59 KiB/42.59 KiB
AccessDeniedException: 401 Login Required
CommandException: 1 files/objects could not be transferred.
I'm not sure what the problem is since I ran config and I can see all my buckets. Does anyone know what I need to do?
Note: I do not have gcloud, I just installed gsutil and ran the config.
Login to Google Cloud is needed for accessing any Cloud service. You need to use below command which will guide you through login steps like typing verification code you generate by opening browser link given in console.
gcloud auth login
I was getting a similar response, and was able to solve this problem by looking at the read permissions on the .boto file. In my case, I was using a service account and the .boto file that was created by
gsutil config -e
only had read permissions set for user. Since it was being read by a service running as a different user, it wasn't able to read the file and yielding a 401 Login Required error. I fixed it by adding read permissions for the service's group.
In the least sophisticated case, you could fix it by giving any user read permission with
chmod a+r .boto
A more detailed explanation for troubleshooting
To get more information, run the same command with a -D flag, like:
gsutil -m -D cp ....
In the debug output, look at:
Command being run: /path/to/gsutil
config_file_list: /path/to/boto/config
Create your login credentials using the executable at /path/to/gsutil, not gcloud auth or any other gsutil executable on the machine, using:
/path/to/gsutil config
For a service account, use:
/path/to/gsutil config -e
These should create a .boto config file in your home directory, $HOME/.boto. If you are running the gsutil command this file should be referenced in the config_file_list variable in the debug output. If not, see below to change it.
Running gsutil under a service account or as another user
If you are running as another user, and need to reference a newly-created config file, set the environment variable BOTO_CONFIG (don't forget to export it):
BOTO_CONFIG=/path/to/$HOME/.boto
export BOTO_CONFIG
By setting this variable, when you execute gsutil, it will reference the config file you have placed in BOTO_CONFIG. You can confirm that you are referencing the correct config file by looking at the config_file_list variable in the gsutil -D command's output.
make sure the referenced .boto file is readable by the user who is executing the gsutil command
Running the /path/to/gsutil with the BOTO_CONFIG variable set allowed me to execute gsutil as another user, referencing an arbitrary BOTO_CONFIG file that was set up with a service-account's credentials.
To set up the service account:
https://console.cloud.google.com/permissions/serviceaccounts
The key file from the service account set-up process needs to be downloaded, and the path to it is requested during the gsutil config -e step.
This may be an issue with how gsutil/boto handles the OS path separators on Windows, as referenced here. This should eventually get merged into the sdk tools, but until then the following should work:
Go to
google-cloud-sdk\platform\gsutil\third_party\boto\boto\pyami\config.py
and replace the line:
for path in os.environ['BOTO_PATH'].split(':'):
with:
for path in os.environ['BOTO_PATH'].split(os.path.pathsep):
Next, go to
google-cloud-sdk\bin\bootstrapping\gsutil.py
replace the lines that use ':'
if boto_config:
boto_path = ':'.join([boto_config, gsutil_path])
elif boto_path:
# this is ':' for windows as well, hardcoded into the boto source.
boto_path = ':'.join([boto_path, gsutil_path])
else:
path_parts = ['/etc/boto.cfg',
os.path.expanduser(os.path.join('~', '.boto')),
gsutil_path]
boto_path = ':'.join(path_parts)
with
if boto_config:
boto_path = os.path.pathsep.join([boto_config, gsutil_path])
elif boto_path:
# this is ':' for windows as well, hardcoded into the boto source.
boto_path = os.path.pathsep.join([boto_path, gsutil_path])
else:
path_parts = ['/etc/boto.cfg',
os.path.expanduser(os.path.join('~', '.boto')),
gsutil_path]
boto_path = os.path.pathsep.join(path_parts)
Restart cmd and now the error should go away.