I have created (using terraform resource google_artifact_registry_repository) a python repository on Google Artifact Registry. Here's my terraform code that created it:
resource "google_artifact_registry_repository" "pypi" {
provider = google-beta
project = var.project_id
location = var.region
repository_id = "dataplatformpypi"
description = "PyPi repo for use by dataplatform"
format = "PYTHON"
}
here is that repository:
I am now following the quickstart at https://cloud.google.com/artifact-registry/docs/python/quickstart, specifically the Configure authentication section which instructs me to issue gcloud artifacts print-settings python. I actually modify that slightly to issue:
gcloud --project myproject artifacts print-settings python --repository dataplatformpypi --location europe-west2
and I get error:
ERROR: (gcloud.artifacts.print-settings.python) Invalid repository type PYTHON. Valid type is PYPI.
I haven't specified the repository type as part of that command so I can only assume that "repository type PYTHON" refers to the format of the repository:
However given that the repository has been created successfully and PYTHON is (according to the terraform resource documentation) a valid value for the repository format I am struggling to understand what the problem is here.
I would appreciate any advice.
It doesn't appear to be a user-specific problem. Other users have also encountered the issue. There's a similar issue ongoing in GitHub. You can follow the thread here.
Related
Being new to Ansible collections I’m hoping I’ve missed something obvious here in my attempt to refactor some old Ansible roles into collections using private GitHub repositories.
I have GitHub setup with 2 linked accounts. I’ll call the main personal account GITHUB_AC_P. The personal account is linked to a child organizational account I’ll call GITHUB_AC_O. I can switch between these accounts in the GitHub web UI and use the following single entry in ~/.ssh/config to access both accounts with git clients:
Host GITHUB_AC_P.github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_github_REDACTED_GITHUB_A
I first added Ansible Galaxy collection files to a new GitHub repository named ansible.common in account GITHUB_AC_O. I plan to reuse this collection in other Ansible Galaxy collections. It currently has a single role and the following galaxy.yml file:
namespace: REDACTED_NS
name: common
version: 0.0.1
description: "Common Ansible collection"
readme: README.md
authors:
- REDACTED_AUTHOR
The following command reports “installed successfully” and I see the collection in ~/.ansible/collections/ansible_collections/REDACTED_NS/common:
ansible-galaxy collection install git#GITHUB_AC_P.github.com:GITHUB_AC_O/ansible.common.git,main
I then created a second Ansible Galaxy collection in a new GitHub repository named ansible.harden_host. This is also in account GITHUB_AC_O. This currently has no roles and uses the following galaxy.yml file to reference the above common collection (the value of REDACTED_NS is the same in both galaxy.yml files):
namespace: REDACTED_NS
name: harden_host
version: 0.0.1
description: "Ansible collection to harden hosts"
readme: README.md
authors:
- REDACTED_AUTHOR
dependencies: {
REDACTED_NS.common: git#GITHUB_AC_P.github.com:GITHUB_AC_O/ansible.common.git,main
}
But when I run the following:
ansible-galaxy collection install --verbose git#GITHUB_AC_P.github.com:GITHUB_AC_O/ansible.harden_host.git,main
It fails with message:
Starting galaxy collection install process
Process install dependency map
ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>
Why is this trying to hit galaxy.ansible.com instead of my GitHub account?
When I add --ignore-certs and run the following:
ansible-galaxy collection install --ignore-certs git#GUTHUB_AC_P.github.com:GITHUB_AC_O/ansible.harden_host.git,main
It fails with this different message:
ERROR! Failed to find collection REDACTED_NS.common:git#GITHUB_AC_P.github.com:GITHUB_AC_O/ansible.common.git
I pasted the URI from this error (right of the colon) into a ansible-galaxy collection install command to verify there’s no typo in the URI. This worked fine.
The string REDACTED_NS does not equal the value of GITHUB_AC_P or GITHUB_AC_O.
If someone could please explain what’s wrong here and how the issue can be fixed that would be much appreciated.
Solved; it seems the answer was hiding in plain site in Ansible's Using collections document, which says to use the following form for git based dependencies:
dependencies: {'git#github.com:organization/repo_name.git': 'devel'}
The form I was using was for Galaxy servers, hence it was hitting galaxy.ansible.com (unless I overrode the default with e.g. --server localhost).
So the following form works (git repo followed by git reference):
namespace: REDACTED_NS
name: harden_host
version: 0.0.1
description: "Ansible collection to harden hosts"
readme: README.md
authors:
- REDACTED_AUTHOR
dependencies: {
'git#GITHUB_AC_P.github.com:GITHUB_AC_O/ansible.common.git': 'main'
}
I have configured a webhook between github and terraform enterprise correctly, so each time I push a commit, the terraform module gets executed. Why I want to achieve is to use part of the branch name where the push was made and pass it as a variable in the terraform module.
I have read that the value of a variable can be a HCL code, but I am unable to find the correct object to access the payload (or at least, the branch name), so at this moment I think it is not possible to get that value directly from the workspace configuration.
if you get a workaround for this, it may also work from me.
At this point the only idea I get is to call the terraform we hook using an API Call
Thanks in advance
Ok, after several try and error I found out that it is not possible to get any information in the terraform module if you are using the VCS mode. So, in order to be able to get the branch, I got these options:
Use several workspaces
You can configure a workspace for each branch, so you may create a variable a select that branch in each workspace. The problem is you will be repeating yourself with this option
Use Terraform CLI and a GitHub action
I used these fine tutorial from Hashicorp for creating a Github action that uses Terraform Cloud. It gets you done the 99% of the job. For passing a varible you must be aware that there are two methods, using a file or using an enviromental variable (check that information on the Hashicorp site here). So using a:
terraform apply -var="branch=value"
won't work. In my case I used the tfvars approach, so in my Github Action I put this snippet:
- name: Setup Terraform variables
id: vars
run: |-
cat > terraform.auto.tfvars <<EOF
branch = "${GITHUB_REF#refs/*/}"
EOF
I defined a variable within terraform called branch, I was able to get and work with this value
I have been working on an installation of OKD on an air-gapped environment. The first major step has been mirroring the OKD images so that they can be moved over to the new environment and pulled locally. I've been following a combination of the OpenShift documentation and this article, as well as this resource for getting my certificates set up. I have been making slow but consistent progress.
However, I am now having trouble when attempting to actually mirror the files using
oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}
I get the following, encouraging response:
info: Mirroring 120 images to host.okd-registry.dns:5000/ocp4/openshift4 ...
followed by blobs: and manifests: lines, and finally the line
stats: shared=0 unique=7 size=105.3MiB ratio=1.00
I then get about 50 lines stating
error: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev manifest
sha256:{some value}: unauthorized: access to the requested resource is not authorized
I have a quay account but I am not sure if that is required even after my research, and if it is, where or how I would log into it. I have attempted doing so using oc login followed by various addresses within the release structure, but if this is the solution, I may be using the wrong arguments as I have not been able to find any instructions on doing this.
I have also tried the command with sudo. I doubt that is an issue but I tried it anyway.
I suppose the issue could be with my certificates, but I am not sure how to determine if this is the case.
Any guidance or suggestions would be much appreciated.
It has been determined that the OKD documentation is inaccurate at the time that I am posting this answer, and was instructing readers to pull from the OCP image repository rather than the OKD repository, which apparently requires additional credentials. A bug has been logged and the documentation will hopefully be updated soon.
The correct environment variables and full command to mirror the images are as follows:
LOCAL_REGISTRY=localhost:5000 (or your local domain name and port for the registry)
LOCAL_REPOSITORY=okd
LOCAL_SECRET_JSON=<full path to your pull secret>
OCP_RELEASE=4.5.0-0.okd-2020-10-15-235428
PRODUCT_REPO=openshift
RELEASE_NAME=okd
ARCHITECTURE=not-used-in-okd
oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} --dry-run
I am working in a GCP lab (Securing Google Cloud with CFT Scorecard). All instructions for the lab are given.
First I have to run the following two commands to set environment variables
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
export CAI_BUCKET_NAME=cai-$GOOGLE_PROJECT
In the second command given above I don't know what to replace with my own credentials? May be that is the reason I am getting error.
Now I have to enable the "cloudasset.googleapis.com" gcloud service. For this they gave the following command.
gcloud services enable cloudasset.googleapis.com \
--project $GOOGLE_PROJECT
Error for this is given in the screeshot attached herewith:
Error in the serviec enabling command
Next step is to clone the policy: The given command for that is:
git clone https://github.com/forseti-security/policy-library.git
After that they said: "You realize Policy Library enforces policies that are located in the policy-library/policies/constraints folder, in which case you can copy a sample policy from the samples directory into the constraints directory".
and gave this command:
cp policy-library/samples/storage_blacklist_public.yaml policy-library/policies/constraints/
On running this command I received this:
error on running the directory command
Finally they said "Create the bucket that will hold the data that Cloud Asset Inventory (CAI) will export" and gave the following command:
gsutil mb -l us-central1 -p $GOOGLE_PROJECT gs://$CAI_BUCKET_NAME
I am confused in where to replace my own credentials like in the place of project_Id I wrote my own project id.
Also I don't know these errors are ocurring. Kindly help me.
I'm unable to access the tutorial.
What happens if you run the following:
echo ${DEVSHELL_PROJECT_ID}
I suspect you'll get an empty result because I think this environment variable isn't actually set.
I think it should be:
echo ${DEVSHELL_GCLOUD_CONFIG}
Does that return a result?
If so, perhaps try using that variable instead:
export GOOGLE_PROJECT=${DEVSHELL_GCLOUD_CONFIG}
export CAI_BUCKET_NAME=cai-${GOOGLE_PROJECT}
It's not entirely clear to me why this tutorial is using this approach but, if the above works, it may get you further along.
We're you asked to create a Google Cloud Platform project?
As per the shared error, this seems to be because your env variable GOOGLE_PROJECT is not set. You can verify it by using echo $GOOGLE_PROJECT and seeing whether it returns the project ID or not. You could also use echo $DEVSHELL_PROJECT_ID. If that returns the project ID and the former doesn't, it means that you didn't export the variable as stated at the beginning.
If the problem is that GOOGLE_PROJECT doesn't have any value, there are different approaches on how to solve it.
Set the env variable as you explained at the beginning. Obviously this will only work if the variable DEVSHELL_PROJECT_ID is also set.
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
Manually set the project ID into that variable. This is far from ideal because in Qwiklabs they create a new temporal project on every lab, so this would've only worked if you were still on that project. The project ID can be seen on both of your shared screenshots.
export GOOGLE_PROJECT=qwiklabs-gcp-03-c6e1787dc09e
Avoid using the argument --project. According to the documentation, the aforementioned argument is optional and if none is used the command will take the one by default, which will be on the configuration settings. You can get the current project by using this:
gcloud config get-value project
If the previous command matches the project ID you want to use, you can simply issue the following command:
gcloud services enable cloudasset.googleapis.com
Notice that the project ID is not being explicitly mentioned using --project.
Regarding your issue with the GitHub file, I have checked the repository and the file storage_blacklist_public.yaml doesn't seem to be in the directory policy-library/samples. There seems to be a trace that it was once there, but it isn't anymore, they should probably update the lab as it isn't anymore.
About your credentials confusion, you don't have to use your own project ID, just the one given on your lab. If I recall properly all the needed data should be on the left side of the lab. Still, you shouldn't need to authenticate in a normal situation as you are already logged in your temporal project if you are accessing it form the Cloud Shell, which is where you should be doing all this.
Adding this for the later versions
in the gcloud shell you can set a temp variable for the current project id with
PROJECT_ID="$(gcloud config get-value project)"
then use like
--project ${PROJECT_ID}
Can someone been able to integrate Dataproc,Datalab and Source code repo? As many of us have seen that when you call an init action to install datalab, it does not create the source code repo. I am trying to achieve a full end-to-end solution where a user logs into to a datalab notebook, interact with Dataproc through Pyspark and check-in the notebooks to the Source code repo. I have not been able to do this with the init action like i pointed out earlier. I also tried installing dataproc and then datalab as a separate install ( this time it creates the source repo) , however, I can't run any spark code on this datalab notebook. Can someone please give me some pointers on how to achieve this? Any and all is appreciated.
Code in Datalab
from pyspark.sql import HiveContext
hc=HiveContext(sc)
hc.sql("""show databases""").show()
hc.sql("""CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
(SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
STORED AS PARQUET
LOCATION 'gs://my-exercise-project-2019016-ds-team/datasets/invoices'""")
hc.sql("""select * from invoices limit 10""").show()
Error
Py4JJavaError: An error occurred while calling o55.sql.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3208)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3240)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3291)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3259)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:470)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$or
Unfortunately, it takes some pre-work to be able to create the datalab-notebooks repository in Cloud Source Repositories from an init action.
The reason is that creating the repository requires the service account for the VM to have the "source.repos.create" IAM permission on the project, which is not true by default.
You can either grant that permission to the service account, and then create the repository via gcloud source repos create datalab-notebooks, or manually create the repository before creating the cluster.
Then, to clone the repository inside of your startup script, add the following lines:
mkdir -p ${HOME}/datalab
gcloud source repos clone datalab-notebooks ${HOME}/datalab/notebooks
If you are modifying the canned init action for Datalab, then I would suggest adding these lines here