In order to generalize the python functions, I wanted to add functions to python libraries so that I can use these function across the multiple repositories. Anyone please answer the below questions.
1) How to create our own python libraries
2) how to import those libraries across multiple repositories
How to create python libraries in Palantir Foundry?
To create a new library, you can do so by creating a new repository. When prompted to initialise the repository, you should have an option that reads:
Python Library
Template for publishing a Python library package. Consuming new libraries has changed,
please read README in library repository.
The readme will contain instructions on how to publish a library. It is recommended you understand how conda publishing channels work for this.
A note, avoid using _ in the library name, since it can cause problems. - is safe to use though.
How to import a library in code authoring?
Once your library is publishing, you can add it to your conda recipe of the repository you want to consume the library in. You can find this in: transforms-python/conda_recipe/meta.yaml
Afterwards just add it to the list of under
requirements:
run:
- python
- pandas
- your-library-name
In addition to fmsf's answer, in the second step you might have to add the following content to the build.gradle of your transforms-python directory:
transformsPython {
sharedChannels "libs"
}
Related
My organization is fairly big, we use node and are looking into potentially using Cypress. We have like 100 different projects and each have their own Github repository, but we like to share functions and methods and import them. So my question is, is there a way to do that with Cypress custom commands so that it can be a dependency and you can just import them?
Is it as easy as just adding a Github URL in the dependency?
You will have to create a library/package that has all those commands implemented. And then in every project you should install Cypress and then import those commands. For example:
In the library you would have something like this in commands.ts if you're using Typescript or commands.js if you're using JavaScript.
Cypress.Commands.add('take', (testSelector) => {
return cy.get(`[data-test='${testSelector}']`);
});
Then, in the repo you want to use it you'll have to import it from the library.
In your cypress/support/index file you should add:
import '#organization/library-name/path/to/commands';
Note that the name of the package can depend on how you choose to generate it.
I'm trying to find a way to make VSCode Python intellisense work with Airflow Plugins.
Following the code example the import path of plugin operators could be:
from airflow.operators import MyPluginOperator
VSCode cannot resolve this import because it will only be valid at runtime through the airflow plugin system.
Is there any way to configure VSCode to resolve this import?
Airflow loads plugins dynamically by searching the airflow/plugins folder for AirflowPlugin subclasses and add them in airflow namespace in a runtime. Here is the code from airflow/operators/__init__.py:
# Imports operators dynamically while keeping the package API clean,
# abstracting the underlying modules
...
def _integrate_plugins():
"""Integrate plugins to the context"""
from airflow.plugins_manager import operators_modules
for operators_module in operators_modules:
sys.modules[operators_module.__name__] = operators_module
globals()[operators_module._name] = operators_module
VS Code can't handle it. Even "big" Python IDEs like PyCharm has problems with it. It is impossible for VS Code to know that a piece of code in particular folder will transform in airflow.operator later. "python.autoComplete.extraPaths" will not help too. You should only hope that someone will write a VS Code extension for Airflow somewhere :)
Since version 2.0.0 the way that Airflow handles plugins imports has changed:
Importing operators, sensors, hooks added in plugins via airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported, and these extensions should just be imported as regular python modules. For more information, see: Modules Management and Creating a custom Operator
The next important thing to consider is that Airflow adds dags/, plugins/, and config/ directories to PYTHONPATH env. source
Following the specifications from the docs, you could create your MyCustomOperator
in airflow_home/plugins/custom_operator/ directory. Then you could use it like this:
from custom_operator.my_custom_operator import MyCustomOperator
with dag:
sample_custom_task = MyCustomOperator(task_id='sample-task')
So far so good, the dag will run, but according to my experience, the VSCode IntelliSense won't work yet. To make it work, you need to add the path to /plugins as same as Airflow will do when it runs. Depending on your setup there are a few ways to do this. The goal is to add an extra path to the python interpreter VSCode is "using" (make sure to select the interpreter related to your project)
A common solution could be to use the env PYTHONPATH to append our path to the path the interpreter knows. In my case, I'm using a virtual environment, so following the explained in this post, I created a .pth file with the path I wanted to add and locate that file on airflow_home/venv/lib/my_python_version/site-packages/
Following the path on our example, this will create such a file:
cd $(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
echo airflow_home/plugins > airflow_plugins_example.pth
Once that is done, reload VSCode (could also just change to another interpreter and then come back) and you should see the intelliSense working properly.
Hope that works!
As of today, in the IBM Bluemix docs for the IBM Bluemix OpenWhisk service I could not find any clues as to how to use libraries.
How am I missing the obvious that all apps invariably require a library and therefore why isn't that at least mentioned in the OpenWhisk docs?
If libraries are called something else or the concept doesn't apply in the usual way (such as maybe Libraries need to be converted into "OpenWhisk Packages"?), the OpenWhisk docs should SAY SOMETHING about the word/term/concept "libraries".
You can use webpack to bundle all your dependencies and create the final .js file you'll use as your OpenWhisk action.
See this example:
These are all the actions before webpack build: https://github.com/IBM-Bluemix/logistics-wizard-recommendation/tree/dev/actions
Invoking webpack: https://github.com/IBM-Bluemix/logistics-wizard-recommendation/blob/dev/package.json webpack --config webpack.config.js
Here is another more simpler example: https://github.com/IBM-Bluemix/openwhisk-webpack
To cover another language for anyone who finds this question…
For Swift, OpenWhisk comes with the Kitura-net, SwiftyJSON & swift-watson-sdk packages (Swift term for libraries) built in.
If you want to include any other packages then you have to either build your own Docker container for your action or concatenate all the Swift source files that are in the packages together with your action file to create a single .swift file for upload with wsk action update. I've used cat to do this:
cat lib/Redis/Redis*.swift actions/_common.swift actions/counts.swift > build/counts.swift
which creates a single build/counts.swift containing Kitura-Redis, some common code and my counts action.
I'm using ember-cli and I made a custom component using ember-cli syntax & naming conventions. This is a highly reusable component and I'd like to know what is the better way to put it all into a "package" so it's easy to integrate into other projects.
My component use a .js file for the Ember.Component subclass along with a .hbs file for the template and yet another couple of .js files for the necessary Ember.View subclasses. Right now, every file is in its respective folder along with the files for the rest of my project.
How can I isolate the files related to the component and package them for reuse? In Ruby on Rails I use gems for this matter, and in jQuery I used to write plugins by extending $.fn in a single file.
Take advantage of Ember CLI addon system. It's been designed for cases like this one. The process should be easy if you are familiar with Ember CLI already. As Ember CLI addon system's been reworked in the recent past and it's API was changing it's possible that older articles or guides on this topic became out of sync.
The most comprehensive and kept in sync guide on this topic is kristianmandrup's gist Converting libraries to Ember CLI addons.
There is also an Addons tutorials section on the official Ember CLI site.
I've compiled Goauth so that I can use OAuth in my Go Google App Engine project. Where do I put the goauth.a file so that I can both use it in the project, and have it available when deploying to the GAE servers? I can get it working locally if I put it in a subfolder of $GOROOT/pkg, but then it can't be found when compiling at deployment time.
GoClipse sets up a project with lots of folders, I'm not really sure what their purpose is, where should I put goauth.a and how do I import it?
To fix this I ended up including the source for the package in the directory tree for my app, as mentioned in this thread on the google-appengine-go group http://groups.google.com/group/google-appengine-go/browse_thread/thread/1fe745debc678afb
Here is the important part of the thread:
You may include as many packages as necessary. Packages are imported
by path relative to the base directory (the one that has your app.yaml
file), so if you have the following:
helloworld/app.yaml
helloworld/hello/hello.go // package hello
helloworld/world/world.go // package world
you can import "world" in hello and import "hello" in world.
If you are including a third-party library, it might look something like this:
helloworld/app.yaml
helloworld/hello/hello.go // package hello
helloworld/world/world.go // package world
helloworld/goprotobuf.googlecode.com/proto/*.go // package proto
Then you can, as normal, import "goprotobuf.googlecode.com/proto".