I'm running a bash script as postCreateCommand when building a VSCode remote development container. In this bash script the command is az login. The problem is that there is not output from this command and therefore I cannot login the Azure CLI. When ran manually in the terminal, it works.
devcontainer.json has this command:
"postCreateCommand": "bash .devcontainer/install.sh",
install.sh has only one line:
az login
The expected output is to be asked by Azure CLI to login on microsoft.com/devicelogin using a code.
The VSCode output doesn't show any Azure CLI output; it only tells that the install.sh script is being executed and it waits until the az login command gives a timeout error and finishes.
The CLI command az login is an interactive command. If you want to use it in the script without input, then you need to change it into a non-interactive command. I recommend you use the service principal to achieve it:
az login --service-principal -u username -p password --tenant tenantId
This CLI command will execute directly without input and give the output if the service principal has no problem.
When I start confluent local start I get /users/myusername/.confluent/config.json: No such file or directory.
Does anyone have any idea how to solve this?
you need to create the .confluent directory
1)Install the Confluent CLI, confluent, using the following script. The parameter must be in your PATH (e.g. /usr/local/bin).
On Microsoft Windows, an appropriate Linux environment may need to be installed in order to have the curl and sh commands available, such as the Windows Subsystem for Linux https://learn.microsoft.com/en-us/windows/wsl/about.
curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /path-to-cli
and then go to the path-to-cli folder and run below command
./confluent local start
I am having problems running commands on an EC2 Instance from my Bamboo server.
I have a command generated from the Run Command in the AWS Console. I place that command in a script on my bamboo server and run it:
aws ssm send-command --document-name "AWS-RunPowerShellScript" --targets '{\"Key\":\"tag:Name\",\"Values\":[\"Auto-Scaling-Group\"]}' --parameters '{\"commands\":[\"$fileEXE = \\\"C:\\\\Program Files (x86)\\\\NUnit\\\\NUnit.ConsoleRunner.3.7.0\\\\tools\\\\nunit3-console.exe\\\\\\\"\",\"$testDll = \\\"C:\\\\TestFramework\\\\TestFramework\\\\Tests\\\\bin\\\\Debug\\\\TESTS.dll\\\"\",\"[System.Diagnostics.Process]::Start($fileEXE,$testDll)\"]}' --comment "Run Test UI Testing" --timeout-seconds 600 --region us-east-1
It does run the tests. But it runs the Chrome.exe browser AND the chromedriver.exe as background processes. This throws a NoSuchWindowException because there is no browser showing up...
I can run the same command in PowerShell on the instance locally: (*Note that this is the same command I pasted into the Run Command console to generate the code mentioned above.)
$fileEXE = "C:\Program Files (x86)\NUnit\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe\"
$testDll = "C:\TestFramework\TestFramework\Tests\bin\Debug\TESTS.dll"
[System.Diagnostics.Process]::Start($fileEXE,$testDll)
It works just fine. chromedriver.exe is a background process and chrome.exe (the browser) is a regular app that works like normal.
I believe my problem is how Run Command is running my test program.
What is the difference between Run Command (send-command) and running the PowerShell commands locally? Shouldn't it do the same thing?
I think there is a mess up with quotes and the way how they're escaped.
See: How to escape a double quote inside double quotes?
This version should look much simpler:
CMD='$fileEXE = "C:\Program Files (x86)\NUnit\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe";'
CMD+='$testDll = "C:\TestFramework\TestFramework\Tests\bin\Debug\TESTS.dll";'
CMD+='[System.Diagnostics.Process]::Start($fileEXE,$testDll);'
aws ssm send-command --document-name "AWS-RunPowerShellScript" \
--filters "Name=tag:Name,Values=Auto-Scaling-Group" \
--comment "Run Test UI Testing" --timeout-seconds 600 --region us-east-1 \
--parameters commands="'$CMD'"
Note: Run it in the Bash shell.
As user, gsutil works nice.
gsutil works nice when called from crontab (user).
As root, gsutil says:
Caught non-retryable exception while listing gs://....: ServiceException: 401 Anonymous users does not have storage.objects.list access to bucket ...."
gsutil does not work when called from Anacron (root).
Other scripts called from Anacron run nice.
The ~/.boto file contains credentials, and is located in user HOME directory.
So maybe that is causing the exception.
I tried setting BOTO_CONFIG, but it didn't change results:
$ gsutil -D ls 2>&1 | grep config_file_list
config_file_list: ['/home/wolfv/.boto']
$ sudo gsutil -D ls 2>&1 | grep config_file_list
config_file_list: []
$ BOTO_CONFIG="/root/.boto"
$ sudo gsutil -D ls 2>&1 | grep config_file_list
config_file_list: []
How to setup gsutil to run from Anacron?
$ gsutil -D
gsutil version: 4.22
checksum: 2434a37a663d09ae21d1644f64ce60ca (OK)
boto version: 2.42.0
python version: 2.7.13 (default, Jan 12 2017, 17:59:37) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
OS: Linux 4.9.11-200.fc25.x86_64
multiprocessing available: True
using cloud sdk: True
config path: /home/wolfv/.boto
gsutil path: /home/wolfv/Downloads/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /home/wolfv/Downloads/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=redacted -D
config_file_list: ['/home/wolfv/.config/gcloud/legacy_credentials/redacted/.boto', '/home/wolfv/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'redacted')]
UPDATE_1
export BOTO_CONFIG worked for the terminal:
$ sudo -s
[root] # export BOTO_CONFIG=/home/wolfv/.boto
[root] # gsutil -D ls 2>&1 | grep config_file_list
config_file_list: ['/home/wolfv/.boto']
[root] # vi /root/.bashrc
add this line to end of .bashrc:
export BOTO_CONFIG=/home/wolfv/.boto
exit
open new terminal and test the new BOTO_CONFIG in bash.rc
$ sudo -s
[root] # gsutil -D ls 2>&1 | grep config_file_list
config_file_list: ['/home/wolfv/.boto']
exit
Unfortunately export BOTO_CONFIG in /root/.bashrc did not help Anacron call gsutil.
The backup log shows that Anacron called the backup script, and the backup script call to gsutil failed.
Does it matter in which initialization script sets path BOTO_CONFIG?
To make the path permanently accessible to Anacron (root), in which file should set BOTO_CONFIG?:
/etc/profile
/root/.bash_profile
/root/.bashrc
UPDATE_2
My credentials are now invlalid, probably from some change I made.
Here is my attempt at houglum's suggestions for BOTO_CONFIG.
First authorize login to get that out of the way:
$ gcloud auth login
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&prompt=select_account&response_type=code&client_id=redacted.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&access_type=offline
Created new window in existing browser session.
WARNING: `gcloud auth login` no longer writes application default credentials.
If you need to use ADC, see:
gcloud auth application-default --help
You are now logged in as [edacted].
Your current project is [redacted]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
Defining BOTO_CONFIG inline does not work:
$ BOTO_CONFIG=/home/wolfv/.boto gsutil ls
Your credentials are invalid. Please run
$ gcloud auth login
Exporting BOTO_CONFIG does not work:
$ export BOTO_CONFIG=/home/wolfv/.boto; gsutil ls
Your credentials are invalid. Please run
$ gcloud auth login
Sourcing bashrc does not work:
$ ls /home/wolfv/.bashrc
/home/wolfv/.bashrc
$ . /home/wolfv/.bashrc; gsutil ls
Your credentials are invalid. Please run
$ gcloud auth login
UPDATE_3
My credentials work if I remove my credentials from .boto, and use auth login instead (based on Your credentials are invalid. Please run $ gcloud auth login)
$ gcloud auth login redacted#email.com
WARNING: `gcloud auth login` no longer writes application default credentials.
If you need to use ADC, see:
gcloud auth application-default --help
You are now logged in as [redacted#email.com].
Your current project is [redacted-123]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
After using auth login, gsutil works from the terminal:
$ gsutil ls
gs://redacted/
gs://redacted/
gs://redacted/
And the backup script that calls gsutil also works from the terminal:
$ ~/scripts/backup_to_gcs/backup_to_gcs.sh
backup_to_gcs.sh in progress ...
backup_to_gcs.sh completed successfully
However, backup_to_gcs.sh fails when called from crontab.
How to run gsutil from crontab?
UPDATE_4
This is in my anacron file:
1 10 anacron_test_id BOTO_PATH=/home/wolfv/.config/gcloud/legacy_credentials/wolfvolpi#gmail.com/.boto:/home/wolfv/.boto /home/wolfv/scripts/backup_to_gcs/backup_to_gcs.sh
anacron runs the backup_to_gcs.sh script as expected, but the backup fails.
When backup_to_gcs.sh script is called from command line, it works fine.
Probably because gsutil runs as user, but does not run as root:
$ gsutil ls
gs://wolfv/
gs://wolfv-test-log/
gs://wolfv2/
gs://wolfvtest/
$ BOTO_PATH=/home/wolfv/.config/gcloud/legacy_credentials/wolfvolpi#gmail.com/.boto:/home/wolfv/.boto gsutil ls
gs://wolfv/
gs://wolfv-test-log/
gs://wolfv2/
gs://wolfvtest/
$ sudo BOTO_PATH=/home/wolfv/.config/gcloud/legacy_credentials/wolfvolpi#gmail.com/.boto:/home/wolfv/.boto gsutil ls
sudo: gsutil: command not found
$ sudo gsutil ls
sudo: gsutil: command not found
Two days ago root was able to run gsutil.
Since then I used dnf history rollback to uninstall a different software.
Could that have effected gsutil authentication?
UPDATE_5
I followed the instructions on https://cloud.google.com/storage/docs/authentication#gsutilauth
USING SERVICE ACCOUNT
$ gcloud auth activate-service-account --key-file=/home/wolfv/REDACTED.json
Activated service account credentials for: [REDACTED#appspot.gserviceaccount.com]
But still, root could not run gsutil:
$ sudo gsutil ls
sudo: gsutil: command not found
$ gsutil ls -la gs://wolfvtest/test_lifecycle/
CommandException: You have multiple types of configured credentials (['Oauth 2.0 User Account', 'OAuth 2.0 Service Account']), which is not supported. One common way this happens is if you run gsutil config to create credentials and later run gcloud auth, and create a second set of credentials. Your boto config path is: ['/home/wolfv/.boto', '/home/wolfv/.config/gcloud/legacy_credentials/my-project#appspot.gserviceaccount.com/.boto']. For more help, see "gsutil help creds".
The help referse to a page that no longer mentions "auth" https://developers.google.com/cloud/sdk/gcloud/#gcloud.auth
So I have one too many credentials:
$ gsutil -D
...
config_file_list: ['/home/wolfv/.boto', '/home/wolfv/.config/gcloud/legacy_credentials/my-project#appspot.gserviceaccount.com/.boto']
Are any of these credentials used by root (for anacron)?
They are not in the root directory.
Should credintals needed for anacron be in the root directory?
UPDATE_5
I tried again after installing Fedora 26 on How to authorize root to run gsutil?
When you execute BOTO_CONFIG=<value> in the shell, you're not actually defining an environment variable, but rather a local shell variable (see this thread for more details). You want to either define the variable inline with the command:
BOTO_CONFIG=/path/to/config gsutil ls
or first export the BOTO_CONFIG environment variable, then run the gsutil command:
export BOTO_CONFIG=/path/to/config; gsutil ls
EDIT:
I just noticed that in addition to your own $HOME/.boto file, you're relying on gcloud's credentials that get set up from gcloud auth login. When you run this, gcloud creates another .boto file for you, and when you run gsutil from gcloud's wrapper script, it loads that .boto file first, followed by whatever .boto file(s) you specify with either the BOTO_CONFIG or BOTO_PATH environment variable.
If you want to run as root (which the cron job does) and use both those .boto files, you'll need to instead use the BOTO_PATH variable to list them, separated by colons, also making sure the BOTO_CONFIG environment variable is not set (BOTO_CONFIG takes precedence over BOTO_PATH... the gsutil docs mention this briefly):
BOTO_PATH=/home/wolfv/.config/gcloud/legacy_credentials/REDACTED/.boto:/home/wolfv/.boto gcloud ls
EDIT 2:
1) When you get the error "sudo: gsutil: command not found", it means that the root user cannot find the gsutil executable in its PATH. You should use the absolute path to the gsutil executable instead -- from your post, it looks like this is /home/wolfv/Downloads/google-cloud-sdk/platform/gsutil/gsutil.
2) When you activate service account credentials, the gcloud wrapper for gsutil will create a separate .boto file (with a path containing legacy_credentials/myproject#appspot[...]), and prefer to use this one if it's present. It contains the attribute gs_service_key_file, while your other .boto file probably contains gs_oauth2_refresh_token -- loading multiple .boto files with multiple credentials attributes like this will result in the error you're seeing.
If you want to use gcloud to manage your auth credentials, you generally shouldn't put anything under the [Credentials] section of your $HOME/.boto file.
ss.exe command -I- won't run using local service
If I run the below command as a network user it works.
"C:\Program Files\SourceSafe\win32\SS.EXE" GET $/JavaSource -R -W -GL "d:\Release\com\fitltd" -GF- -O- -I- Set d:\Release\com\fitltd as the default folder for project $/JavaSource?(Y/N)N
However if this command is called using a local service user (Jenkins service) the command stops waiting for input (you can't see the question it is waiting (see above) for unless you run the same thing using a network user).
so the output looks like: "C:\Program Files\SourceSafe\win32\SS.EXE" GET $/JavaSource -R -W -GL "d:\Release\com\fitltd" -GF- -O- -I-
I am running Windows 2008 R2.