I have created Azure Batch pool with Linux Machine and specified Application Package for the Pool.
My command line is
command='python $AZ_BATCH_APP_PACKAGE_scriptv1_1/tasks/XXX/get_XXXXX_data.py',
python3: can't open file '$AZ_BATCH_APP_PACKAGE_scriptv1_1/tasks/XXX/get_XXXXX_data.py':
[Errno 2] No such file or directory
when i connect to node and look at working directory non of the Application Package files are present there.
How do i make sure that files from Application Package are available in working directory or I can invoke/execute files under Application Package from command line ?
Make sure that your async operation have proper await in place before you start using the package in your code.
Also please share your design \ pseudo-code scenario and how you are approaching it as a design?
Further to add:
Seems like this one is pool level package.
The error seems like that the application env variable is either incorrectly used or there is some other user level issue. Please checkout linmk below and specially the section where use of env variable is mentioned.
This seems like user level issue because In case of downloading the package resource, if there will be an error it will be visible to you via exception handler or at the tool level is you are using batch explorer \ Batch-labs or code level exception handling.
https://learn.microsoft.com/en-us/azure/batch/batch-application-packages
Reason \ Rationale:
If the pool level or the task application has error, an error-list will come back if there was an error in the application package then it will be returned as the UserError or and AppPackageError which will be visible in the exception handle of the code.
Key you can always RDP into your node and checkout the package availability: information here: https://learn.microsoft.com/en-us/azure/batch/batch-api-basics#connecting-to-compute-nodes
I once created a small sample to help peeps around so this resource might help you to checkeout the use here.
Hope rest helps.
On Linux, the application package with version string is formatted as:
AZ_BATCH_APP_PACKAGE_{0}_{1}
On Windows it is formatted as:
AZ_BATCH_APP_PACKAGE_APPLICATIONID#version
Where 0 is the application name and 1 is the version.
$AZ_BATCH_APP_PACKAGE_scriptv1_1 will take you to the root folder where the application was unzipped.
Does this "exact" path exist in that location?
tasks/XXX/get_XXXXX_data.py
You can see more information here:
https://learn.microsoft.com/en-us/azure/batch/batch-application-packages
Edit: Just saw this question: "or can I invoke/execute files under Application Package from command line"
Yes you can invoke and execute files from the application package directory with the environment variable above.
If you type env on the node you will see the environment variables that have been set.
Related
I am working in a GCP lab (Securing Google Cloud with CFT Scorecard). All instructions for the lab are given.
First I have to run the following two commands to set environment variables
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
export CAI_BUCKET_NAME=cai-$GOOGLE_PROJECT
In the second command given above I don't know what to replace with my own credentials? May be that is the reason I am getting error.
Now I have to enable the "cloudasset.googleapis.com" gcloud service. For this they gave the following command.
gcloud services enable cloudasset.googleapis.com \
--project $GOOGLE_PROJECT
Error for this is given in the screeshot attached herewith:
Error in the serviec enabling command
Next step is to clone the policy: The given command for that is:
git clone https://github.com/forseti-security/policy-library.git
After that they said: "You realize Policy Library enforces policies that are located in the policy-library/policies/constraints folder, in which case you can copy a sample policy from the samples directory into the constraints directory".
and gave this command:
cp policy-library/samples/storage_blacklist_public.yaml policy-library/policies/constraints/
On running this command I received this:
error on running the directory command
Finally they said "Create the bucket that will hold the data that Cloud Asset Inventory (CAI) will export" and gave the following command:
gsutil mb -l us-central1 -p $GOOGLE_PROJECT gs://$CAI_BUCKET_NAME
I am confused in where to replace my own credentials like in the place of project_Id I wrote my own project id.
Also I don't know these errors are ocurring. Kindly help me.
I'm unable to access the tutorial.
What happens if you run the following:
echo ${DEVSHELL_PROJECT_ID}
I suspect you'll get an empty result because I think this environment variable isn't actually set.
I think it should be:
echo ${DEVSHELL_GCLOUD_CONFIG}
Does that return a result?
If so, perhaps try using that variable instead:
export GOOGLE_PROJECT=${DEVSHELL_GCLOUD_CONFIG}
export CAI_BUCKET_NAME=cai-${GOOGLE_PROJECT}
It's not entirely clear to me why this tutorial is using this approach but, if the above works, it may get you further along.
We're you asked to create a Google Cloud Platform project?
As per the shared error, this seems to be because your env variable GOOGLE_PROJECT is not set. You can verify it by using echo $GOOGLE_PROJECT and seeing whether it returns the project ID or not. You could also use echo $DEVSHELL_PROJECT_ID. If that returns the project ID and the former doesn't, it means that you didn't export the variable as stated at the beginning.
If the problem is that GOOGLE_PROJECT doesn't have any value, there are different approaches on how to solve it.
Set the env variable as you explained at the beginning. Obviously this will only work if the variable DEVSHELL_PROJECT_ID is also set.
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
Manually set the project ID into that variable. This is far from ideal because in Qwiklabs they create a new temporal project on every lab, so this would've only worked if you were still on that project. The project ID can be seen on both of your shared screenshots.
export GOOGLE_PROJECT=qwiklabs-gcp-03-c6e1787dc09e
Avoid using the argument --project. According to the documentation, the aforementioned argument is optional and if none is used the command will take the one by default, which will be on the configuration settings. You can get the current project by using this:
gcloud config get-value project
If the previous command matches the project ID you want to use, you can simply issue the following command:
gcloud services enable cloudasset.googleapis.com
Notice that the project ID is not being explicitly mentioned using --project.
Regarding your issue with the GitHub file, I have checked the repository and the file storage_blacklist_public.yaml doesn't seem to be in the directory policy-library/samples. There seems to be a trace that it was once there, but it isn't anymore, they should probably update the lab as it isn't anymore.
About your credentials confusion, you don't have to use your own project ID, just the one given on your lab. If I recall properly all the needed data should be on the left side of the lab. Still, you shouldn't need to authenticate in a normal situation as you are already logged in your temporal project if you are accessing it form the Cloud Shell, which is where you should be doing all this.
Adding this for the later versions
in the gcloud shell you can set a temp variable for the current project id with
PROJECT_ID="$(gcloud config get-value project)"
then use like
--project ${PROJECT_ID}
I have a ZF3 project with a controller which opens excel-files and compares them with an template which will be openened, too.
On my development notebook (xampp) everything works fine, at my production system (ubuntu) the phpspreadsheet causes errors (I think it is the one).
here a snippet from my code:
$fileName="./public/files/" . $fileName; //.\ neu
echo $fileName;
$template= new Spreadsheet();
$importdcl= new Spreadsheet();
$template= \PhpOffice\PhpSpreadsheet\IOFactory::load('./public/files/Template_DCL_final.xlsx');
$importdcl= \PhpOffice\PhpSpreadsheet\IOFactory::load( $fileName);
echo "filename geladen";
I already have the folders in non relative paths because basePath() doesn't work, it won't give any result.
The echo statement is just because the server log won't give any errors. On my development system I get the echo text on my production system the error seems to be at the load statements.
First question: How could I use relative paths in here?
Second question: How can I get an idea wether is something wrong with the spreadsheet class?
This is what composer loaded:
"phpoffice/phpspreadsheet" : "dev-develop",
Is it a problem, because it has this dev version? At this point I'm quite confused because I played with the pathes of the files, I changed the rights manually in the folder, I checked server logs and now I don't have any idea left.
Here the rights in the folder:
Any helpful suggestions appreciated.
Answer (hopefully) to the first question: if your application is based on ZendSkeletonApplication, then you can use paths relative to your application root (from index.php):
/**
* This makes our life easier when dealing with paths. Everything is relative
* to the application root now.
*/
chdir(dirname(__DIR__));
So if your data files are located in <application root>/public/files, then you should be able to read them from controller/service/etc. using public/files/<file name> path. You can test it with eg. the file_exists function.
I’m unable to answer your second question, but here are some suggestions (OK, questions…):
what is the status code of your (production) server’s response?
Do you have read/write permissions to the data files from the server’s account (www-data?)?
Does phpspreadsheet depend on any PHP extensions? Do you have them installed on both your development machine and the server?
What PHP version is installed on the server? Do you or phpspreadsheet use any features that may be unavailable on that version?
Try running your project not through Apache, but with PHP’s builtin server (but don’t do it long-term) and try to reproduce the issue.
how can I change the directory where capistrano puts its log files? I could not find in the docs.
Currently the logs appear in myapp/log/... on my dev machine. However, since I am using laravel, and there is a log directory myapp/storage/logs I would like capistranos logs to appear here as well.
Do you mean the capistrano.log file that is created and appended to whenever you deploy?
You can specify the location by adding the following to deploy.rb:
set :format_options, log_file: "storage/logs/capistrano.log"
This tells Airbrussh (the default logging implementation in Capistrano 3.5.0+) where to place the log file. More information here: https://github.com/mattbrictson/airbrussh#configuration
I create a step of database script which access to SQL Database Service in Workload Scheduler Service. When I run the process, the step encountered the error below.
error message
AWKDBE018E Cannot access required JDBC Driver folder
message information
http://www-01.ibm.com/support/knowledgecenter/SSGSPN_9.2.0/com.ibm.tivoli.itws.doc_9.2/common/src_ms/awsmsawkdbe.htm?lang=en
AWKDBE018E Cannot access required JDBC Driver folder
Explanation
The job was not able to access a JDBC Driver folder, you might not
have enough permissions.
System action
The operation is not performed.
Operator response
Verify that you have enough permissions.
This message seems to ask me to grant the proper authority to the job user. But there is no property to specify the job user of Workload Automation Agent. I use a Workload Automation Agent provisioned by Bluemix automatically.
Could you teach me which parameters are needed ?
Database script step information
JDBC driver class path info
I checked the path by the following "ls -lR" command step's log.
it seems to have a problem with the agent, I tried to replicate the same job type but it is not working with the same error message (even using different solutions for jdbc driver path).
If you are using the Workload Automation Agent that is created for you then you could open a support ticket to have the Workload team look at that agent.
Edit after having support from service team:
in the jar classpath field for a predefined workload scheduler process you have to put only the path to the directory containing jar files, without putting the jar file name to use.
So, according to current Workload Scheduler documentation, you have to use the following value:
/home/wauser/utils
By this way the database script works fine.
(screenshot added)
It looks like it is having issues referencing the location to the JDBC class path for DB2. Can you please double check the location for the class path for the DB2 driver?
Even though old, I wanted to make some quick checks.
This is tested on a 9.5 FP1 dynamic agent, part of the container delivery. The path values are the standard values for the container.
Try 1 - full path - SUCCESS
<jsdldatabase:driverPath>/opt/wa/TWS/jdbcdrivers/db2/</jsdldatabase:driverPath>
= Status Message: Success
= Exit Status : 0
Try 2 - relative path - FAIL
<jsdldatabase:driverPath>./jdbcdrivers/db2/</jsdldatabase:driverPath>
Job status : FAIL
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
Try3 - variable in path - FAIL
<jsdldatabase:driverPath>${UNISONHOME}/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
Try4 - variable in path - FAIL
<jsdldatabase:driverPath>$UNISONHOME/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
So put short you need an absolute path into that parameter.
BUT, you can set the path in a config file global to the agent
Try5 - variable in agent config -
Inside IWSDATA Home : wadata/JavaExt/cfg/DatabaseJobExecutor.properties, write the following line
jdbcDriversPath=/opt/wa/TWS/jdbcdrivers
then remove the xml element about driver from the job, so no line
<jsdldatabase:driverPath>/opt/wa/TWS/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
= Exit Status : 0
Note that in this case the jdbcdrivers/db2 is not needed. It will search for subdirectories.
What kind of permission needs to be given to the Error.log file in the server (running IIS 7.5) so that errors are written to the Error.log file?
I have created a directory named ErrorLog and set up the listener in web.config like below.
The windows identity in play (depends on how you authenticate)
var principal = ClaimsPrincipal.Current; //normally this reverts to Thread.CurrentPrincipal, but can be changed
return principal.Identity.Name;
or
var windowsIdentity = WindowsIdentity.GetCurrent();
if (windowsIdentity != null)
{
return windowsIdentity.Name;
}
The Executing windows identity Will need create permission in directory and write permission to the file specified in your EL config.
You can place file anywhere you like using config. Default for file without path is Start project directory containing assembly dll.
See In IIS the Application Pool used by the website. The pool determines the windows Identity. See the advanced settings tab for the Pool. The identity approach to be used is defined here.
I don't see your configuration example, but..
I find that it is best not to use Flat File outside of development. The Event Log or Database listeners are more suitable for a web application in production.
If you must use Flat File, you will need to give the account (for the application pool that is assigned to the web application) write access to the directory that the file should be created in. By default I think that is the web application's bin directory, unless you include a path in the file name in your configuration.
It probably depends on what context your app in operating in. The app pool thread is responsible for writing the log so give the read/write folder permissions for the same identity as your app pool. I think this is NETWORKSERVICE by default.