Suppose I have 2 notebooks of which the first is the main and the second is for testing.
In the main, I have the following
dbutils.notebook.run("testing", timeoutSeconds = 60, arguments = Map("var" -> "1234"))
In testing:
%scala
println(s"Donut price = $var")
And in Main run the notebook. There is error:
You can pass arguments to DataImportNotebook and run different notebooks (DataCleaningNotebook or ErrorHandlingNotebook) based on the result from DataImportNotebook.
val status = dbutils.notebook.run("DataImportNotebook", timeoutSeconds
= 60, argumenrs = Map ("x" -> "1234"))
println("Status: " + status)
In scala, the variables are declared as follows:
The following are examples of value definitions:
var $price = 1234
println("Donut price:" + $price)
For more details, refer "Scala - How to declare variables" and "Databricks - Notebook workflows".
Hope this helps.
Related
I have the below test_dss.py file which is used for pytest:
import dataikuapi
import pytest
def setup_list():
client = dataikuapi.DSSClient("{DSS_URL}", "{APY_KEY}")
client._session.verify = False
project = client.get_project("{DSS_PROJECT}")
# Check that there is at least one scenario TEST_XXXXX & that all test scenarios pass
scenarios = project.list_scenarios()
scenarios_filter = [obj for obj in scenarios if obj["name"].startswith("TEST")]
return scenarios_filter
def test_check_scenario_exist():
assert len(setup_list()) > 0, "You need at least one test scenario (name starts with 'TEST_')"
#pytest.mark.parametrize("scenario", setup_list())
def test_scenario_run(scenario, params):
client = dataikuapi.DSSClient(params['host'], params['api'])
client._session.verify = False
project = client.get_project(params['project'])
scenario_id = scenario["id"]
print("Executing scenario ", scenario["name"])
scenario_result = project.get_scenario(scenario_id).run_and_wait()
assert scenario_result.get_details()["scenarioRun"]["result"]["outcome"] == "SUCCESS", "test " + scenario[
"name"] + " failed"
My issue is with setup_list function, which able to get only hard coded values for {DSS_URL}, {APY_KEY}, {PROJECT}. I'm not able to use PARAMS or other method like in test_scenario_run
any idea how I can pass the PARAMS also to this function?
The parameters in the mark.parametrize marker are read at load time, where the information about the config parameters is not yet available. Therefore you have to parametrize the test at runtime, where you have access to the configuration.
This can be done in pytest_generate_tests (which can live in your test module):
#pytest.hookimpl
def pytest_generate_tests(metafunc):
if "scenario" in metafunc.fixturenames:
host = metafunc.config.getoption('--host')
api = metafuc.config.getoption('--api')
project = metafuc.config.getoption('--project')
metafunc.parametrize("scenario", setup_list(host, api, project))
This implies that your setup_list function takes these parameters:
def setup_list(host, api, project):
client = dataikuapi.DSSClient(host, api)
client._session.verify = False
project = client.get_project(project)
...
And your test just looks like this (without the parametrize marker, as the parametrization is now done in pytest_generate_tests):
def test_scenario_run(scenario, params):
scenario_id = scenario["id"]
...
The parametrization is now done at run-time, so it behaves the same as if you had placed a parametrize marker in the test.
And the other test that tests setup_list now has also to use the params fixture to get the needed arguments:
def test_check_scenario_exist(params):
assert len(setup_list(params["host"], params["api"], params["project"])) > 0,
"You need at least ..."
The following code (not mine) is able to run NotebookA and NotebookB concurrently. I need some help to figure out how to pass multiple arguments to the same notebooks.
I want to pass this list of arguments to each notebook:
args = {}
args["arg1"] = "some value"
args["arg2"] = "another value"
If I wanted to pass the arguments above to each of the running notebooks, what will I need to amend in the code below?
Here is the working code:
from multiprocessing.pool import ThreadPool
pool = ThreadPool(10)
inputs = [("NotebookA", "NotebookB") ]
run_in_parallel = lambda x: dbutils.notebook.run(x, 1800)
from concurrent.futures import ThreadPoolExecutor, wait
pool = ThreadPoolExecutor(3)
results = []
with ThreadPoolExecutor(3) as pool:
for x in inputs:
results.extend(pool.map(run_in_parallel, list(x)))
The dbutils.notebook.run accepts the 3rd argument as well, this is a map of parameters (see documentation for more details). So in your case, you'll need to change definition of the run_in_parallel to something like this:
run_in_parallel = lambda x: dbutils.notebook.run(x, 1800, args)
and the rest of the code should be the same.
If you'll want to pass different arguments to different notebooks, then you'll need to have a list of tuples, and pass this list to a map, like this:
data = [('notebook 1', {'arg1':'abc'}), ('notebook2', {'arg1': 'def', 'arg2': 'jkl'})]
...
run_in_parallel = lambda x: dbutils.notebook.run(x[0], 1800, x[1])
with ThreadPoolExecutor(3) as pool:
results.extend(pool.map(run_in_parallel, data))
How can I do a function to get the current year with 4 digits using ELM 0.19.1? I have read something but nothing works with 0.19.1.
Signature:
getCurrentYear : Int
Execution:
getCurrentYear => 2020
Edit:
Maybe executing new Date().getFullYear() javascript code?
The simplest way would be to pass the year in via flags when you start the app, since the current year isn't likely to change in the course of the application running. In that case, you can use the snippet of JavaScript you suggested (ellie example):
Elm.Main.init({
node: document.querySelector('main'),
flags: {
year: new Date().getFullYear(),
}
});
module Main exposing (main)
import Browser
import Html exposing (Html, p, text)
type alias Flags =
{ year : Int }
main : Program Flags Model Msg
main =
Browser.element
{ init = \flags -> ( Model flags.year, Cmd.none )
, view = view
, update = update
, subscriptions = \_ -> Sub.none
}
type alias Model =
{ year : Int }
type Msg
= NoOp
update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
case msg of
NoOp ->
( model, Cmd.none )
view : Model -> Html Msg
view model =
p [] [ text "The year is ", text (String.fromInt model.year) ]
Alternatively, you can use Time.now to request the current time, as Robin Zigmond's answer suggests, however that is pointing to Elm 0.18 documentation (for elm-lang/core instead of elm/time). For 0.19, you need both a Time.Posix and a Time.Zone in order to call Time.toYear. You can chain Time.now (a Task producing a Posix value) and Time.here (a Task producing a Zone with the current time zone offset) to retrieve those values in one Cmd. Here's an example (also on ellie)
module Main exposing (main)
import Browser
import Html exposing (Html, p, text)
import Task exposing (Task)
import Time
type alias Flags =
{ year : Int }
main : Program () Model Msg
main =
Browser.element
{ init = \() -> ( Model 0, whatYearIsIt |> Task.perform GotYear )
, view = view
, update = update
, subscriptions = \_ -> Sub.none
}
whatYearIsIt : Task x Int
whatYearIsIt =
Task.map2 Time.toYear Time.here Time.now
type alias Model =
{ year : Int }
type Msg
= GotYear Int
update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
case msg of
GotYear year ->
( { model | year = year }, Cmd.none )
view : Model -> Html Msg
view model =
p [] [ text "The year is ", text (String.fromInt model.year) ]
As I already said in my comment, it's impossible to define a function in Elm that returns the current year. You have to get such information from the Elm runtime system (which is basically JavaScript, but you don't have to write it yourself). This happens via commands, where you tell the runtime system to do something for you. But note that you can't simply retrieve the "return value" of that command and get it back into your Elm code. Instead you have to pass it into a function that can convert it into a "message" (see basic Elm Architecture tutorial here, it's fundamental to understand this before you can do anything with Elm) - this then allows you to store the value in your Model and thereby display it in your app.
These patterns do take some getting your head around, especially if you're not used to pure functional programming - but once you get used to it the benefits are huge, including a near guaranteed absence of runtime errors, and greatly enhanced ability to reason about your code.
For getting the year specifically, it looks like you need this library, which gives you (as now) a Task rather than a Cmd. You can use Task.perform to convert it to a command, which is documented here - in fact it even gives an example that matches your use case quite closely - I'll copy it here for posterity:
import Time -- elm install elm/time
import Task
type Msg
= Click
| Search String
| NewTime Time.Posix
getNewTime : Cmd Msg
getNewTime =
Task.perform NewTime Time.now
You'll have to fill this in to fit your own use case, in particular your own Msg type. But it gives a good basic outline. To get the user's current year, you need to replace the Time.Posix type with Int, and the Time.now command with (Task.map2 Time.toYear Time.here Time.now), as explained by #bdukes in his answer.
I'm trying to write a simple package for Atom.io editor. This is my first experience with Coffeescript.
So I'm probably missing something trivial.
Anyway, this is my index.coffee
module.exports =
activate: ->
atom.workspaceView.command "md-utils:unorderedList", => #unorderedList()
unorderedList: ->
out = ""
editor = atom.workspace.activePaneItem
selection = editor.getSelection()
lines = selection.getText().split "\n"
for line in lines
out += "- " + line + "\n"
console.log(lines)
selection.insertText(out)
And here it is my index-spec.coffee
{WorkspaceView} = require 'atom'
describe "Markdown Utilities", ->
[editor, editorView] = []
unorderedList = (callback) ->
editorView.trigger "md-utils:unorderedList"
runs(callback)
beforeEach ->
atom.workspaceView = new WorkspaceView
atom.workspaceView.openSync()
editorView = atom.workspaceView.getActiveView()
editor = editorView.getEditor()
describe "when text is selected", ->
it "formats it correctly", ->
console.log = jasmine.createSpy("log")
editor.setText """
a
b
c
d
"""
editor.selectAll()
unorderedList ->
expect(console.log).toHaveBeenCalled()
expect(editor.getText()).toBe """
- a
- b
- c
- d
"""
Now, when I run the spec looks like the method in the index.coffee is not even called.
Both expectations failed :
Expected spy log to have been called.
Expected 'a b c d' to be '-a -b -c -d"
The method in itself works , so I do not understand why test fails.
Any suggestion is much appreciated
Actually your specs lack of a package activation, which is generally done with something like this:
beforeEach ->
atom.workspaceView = new WorkspaceView
atom.workspaceView.openSync()
editorView = atom.workspaceView.getActiveView()
editor = editorView.getEditor()
# Package activation is done within a promise so this will wait for the end of it
# before actually running the tests.
waitsForPromise -> atom.packages.activatePackage('your-package-name')
As your package is never activated, the command you define in the activate method is never registered, so the event triggered in your unorderedList helper never reach it.
I am trying to use Scala ScriptEngine (2.11) to run Scala script in Java.
The script uses the bindings provided to the engine.
Script is used multiple times with different bindings.
For this, I am using CompiledScript.
The script runs fine for the first time and uses the bindings.
But, when the same CompiledScript is rerun using new bindings, the result does not change.
What I observed is that the script actually does not run the second time. It just uses the stored engine state.
The following is the code snippet:
ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("scala");
Bindings bindings = engine.createBindings();
bindings.put("firstVal", 209);
bindings.put("secondVal", 30);
bindings.put("sumUtil", new Sum());
CompiledScript script = ((Compilable)engine).compile(
"var sum = sumUtil.asInstanceOf[com.myutils.Sum]\n"+
"var firstInt = firstVal.asInstanceOf[Integer]\n"+
"var secondInt = secondVal.asInstanceOf[Integer]\n"+
"sum.add(firstInt, secondInt)\n"
);
Integer res1 = (Integer) script.eval(bindings);
System.out.println("Result 1: "+res1);
bindings = engine.createBindings();
bindings.put("firstVal", 2);
bindings.put("secondVal", 3);
Integer res2 = (Integer) script.eval(bindings);
System.out.println("Result 2: "+res2);
The output is:
firstVal: Object = 209
secondVal: Object = 30
sumUtil: Object = com.myutils.Sum#71136646
Result 1: 239
firstVal: Object = 2
secondVal: Object = 3
Result 2: 239
The expectation is that Result 2 is "5" instead of 239.
Am I doing anything wrong here?
Thanks in advance