I need to create a data factory pipeline to move data from sftp to blob storage. At this point, I'm only doing a POC, and I'd like know how I would read configuration settings and kick off my pipeline based on those configuration settings.
Example of config settings would be (note that there would be around 1000 of these):
+--------------------+-----------+-----------+------------------------+----------------+
| sftp server | sftp user | sftp pass | blob dest | interval |
+--------------------+-----------+-----------+------------------------+----------------+
| sftp.microsoft.com | alex | myPass | /myContainer/destroot1 | every 12 hours |
+--------------------+-----------+-----------+------------------------+----------------+
How do you kick off a pipeline using some external configuration file/store?
Take a look at Lookup activity and linked service Parameterize
Related
I am trying to find a place to upload a binary that has to be executed in all the workflows that we run.
The problem that I am having is finding such a place.
I tried uploading the binary as a release. I am able to download this binary via the REST API but I have to pass in a personal access token to do so.
Is there a way to download a release artifact without passing in a personal access token on an internal repository that is visible to all the users in the organization?
-----------
| |
| bin.exe |
| |
-----------
/ | \
/ | \
----------- ------------ ------------
| | | | | |
| Repo 1 | | Repo 2 | | Repo 3 |
| | | | | |
----------- ------------ ------------
We have hundreds of repos and we want to pull the release artifact that is hosted in a different repo without passing in a personal access token because adding a personal access token is not feasible on hundreds of repos. We also do not want to host the binary publicly.
If you have been in a similar situation, please let me know on how you resolved it.
because adding a personal access token is not feasible on hundreds of repos
That means using one key (a deploy key, associated to the unique repository with the binary, instead of a PAT associated to a user) in order to access said binary.
Using gh secret set, you can add that key to hundred of repository through a script.
Then a common GitHub action can reference that secret when executed in each repository.
Lets say I have a scenario in my demo.feature file
Scenario Outline: Gather and load all submenus
Given I will login using <username> and <password>
When I will click all links
Examples :
| username | password |
| user1 | pass1 |
| use2 | pass2 |
lets say i have a file called users.json
How can i get those usernames and passwords from that external file to my demo.feature ?
Can I catch the file by passing parameters to my npm script like below ?
npm run cucumber -- --params.environment.file=usernames.json
I recommend having the login step access that json file within the step definition. Just make sure not to check it into the repo and instead always expect it to be in a location but only locally and not in the repository.
Doing the above is useful for a couple of reasons:
- An engineer running your tests does not need to know that a param must be passed in from the command line
- The code is self-descriptive in that step as to how it logs in
- You can add better error handling
- You can use multiple user files if needs be by having hooks define paths etc based on tags
I'm trying to create Project and Task in TAC using MetaServletCaller.bat file.
I'm able to create a project using the bat file, but didn't get how to link or assign jobs to that project.
How to create project with the jobs using MetaServletCaller.bat file?
Talend MetaServletCaller API doesn't provide any command for creating a job from an export file. The only way to do this would be to do it in Talend studio, or programmatically using the commandline importItems command which allows you to import an exported job (while logged in to the project):
| importItems source (dir|.zip) imports items |
| -if (--item-filter) filterExpr item filter expression |
| -im (--implicit) import implicit |
| -o (--overwrite) overwrite existing items |
| -s (--status) import the status |
| -sl (--statslogs) import stats & logs params |
You can find the commandline API reference here.
I'm attempting to start OrientDB in distributed mode on AWS.
I have an auto scaling group that creates new nodes as needed. When the nodes are created, they start with a default config without a node name. The idea is that the node name is generated randomly.
My problem is that the server starts up and ask for user input.
+---------------------------------------------------------------+
| WARNING: FIRST DISTRIBUTED RUN CONFIGURATION |
+---------------------------------------------------------------+
| This is the first time that the server is running as |
| distributed. Please type the name you want to assign to the |
| current server node. |
| |
| To avoid this message set the environment variable or JVM |
| setting ORIENTDB_NODE_NAME to the server node name to use. |
+---------------------------------------------------------------+
Node name [BLANK=auto generate it]:
I don't want to set the node name because I need a random name and the server never starts because it's waiting for user input.
Is there a parameter I can pass to dserver.sh that will pass this check and generate a random node name?
You could create a random string to pass to OrientDB as node name with the ORIENTDB_NODE_NAME variable. Example:
ORIENTDB_NODE_NAME=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
For more information about this, look at: https://gist.github.com/earthgecko/3089509
the server that a magento install was on crashed for (currently reason unknown) and now the backend is inaccessible. I either get no error or I get access denied. I am able to preform the password reset.
I have tried:
clearing the browser cookies and cache
clearing the session, tmp and cache folders in magento
using the magento database repair tool - no errors were found
creating a new user manually in the admin_users table (this gets access denied)
rebooting the server again (last resort but still no)
The main admin user gets a redirect loop.
Any ideas are welcome, I am now stumped.
EDIT:I am really after any way to recover a magento admin login? or failing this any way to export the data (without exporting the bugged section)
I have employed a hack in one of the core magento files detailed here: http://blog.chapagain.com.np/magento-admin-login-problem/
I have had to comment out lines in "app/code/core/Mage/Core/Model/Session/Abstract/Varien.php":
(please note this is for 1.6 - check link posted for advice on 1.4)
$cookieParams = array(
'lifetime' => $cookie->getLifetime(),
'path' => $cookie->getPath()//,
//'domain' => $cookie->getConfigDomain(),
//'secure' => $cookie->isSecure(),
//'httponly' => $cookie->getHttponly()
);
also line 104 comment out :
//call_user_func_array('session_set_cookie_params', $cookieParams);
It seems to be when the server went down something happened to the session, as the new install also failed.
You can get your data by performing a database backup via cPanel or mysqldump via SSH access. Your hosting provider should be able to help you. Additionally, you can unzip the admin files from your Magento installed version. Download it to your computer and unzip the installation... then re-zip the folder for app/design/adminhtml ... upload and unpack this on your website in the proper place, and this will reinstall the PHP pages.
Double check the file app/etc/local.xml this usually has the database login stored in it.
Inside the mysql database for your magento installation, there is the table core_config_data and in it, some path values might interfere with using admin if intentionally set incorrectly:
mysql> select config_id, path, left(value,30) from core_config_data WHERE path LIKE '%admin%';
+-----------+-------------------------------------------+--------------------------------+
| config_id | path | left(value,30) |
+-----------+-------------------------------------------+--------------------------------+
| 50 | dev/translate_inline/active_admin | 0 |
| 220 | web/secure/use_in_adminhtml | 0 |
| 687 | admin/startup/page | catalog/products |
| 690 | admin/security/use_form_key | 0 |
| 691 | admin/security/use_case_sensitive_login | 1 |
| 692 | admin/security/session_cookie_lifetime | 36000 |
| 745 | admin/url/use_custom | 0 |
| 746 | admin/url/custom | |
+-----------+-------------------------------------------+--------------------------------+