I've got the below code and I'm not sure why it doesn't work. It says celery.beat.SchedulingError: Couldn't apply scheduled task Foo: f() missing 1 required positional argument: 'x'
If I take out the function f out from the Foo class, then it works.
from celery import Celery
from celery.schedules import crontab
app = Celery('test', broker='amqp://guest#localhost//')
class Foo:
def f(self, x: int) -> int:
return x
foo = Foo()
f = foo.f
f = app.task(f)
#app.on_after_configure.connect
def run_task(**kwargs):
app.add_periodic_task(
crontab(minute='*/1'),
f.s(),
args=(60,),
name='Foo'
)
app.add_periodic_task(
crontab(minute='*/2'),
f.s(),
args=(32,),
name='Bar'
)
To use class-based tasks, inherit from app.Task.
from celery import Celery, Task
from celery.schedules import crontab
app = Celery('test', broker='amqp://guest#localhost//')
class Foo(app.Task):
def run(self, x: int, *args, **kwargs) -> int:
return x
app.register_task(Foo())
#app.on_after_configure.connect
def run_task(**kwargs):
app.add_periodic_task(
crontab(minute='*/1'),
Foo().s(),
args=(60,),
name='Foo'
)
app.add_periodic_task(
crontab(minute='*/2'),
Foo().s(),
args=(32,),
name='Bar'
)
Related
When doing stateful testing with ScalaCheck the library can shrink the commands needed to find a certain bug. Like in the counter example from the userguide: https://github.com/typelevel/scalacheck/blob/master/doc/UserGuide.md. But what if the commands took arguments and i wanted ScalaCheck to shrink the data inside the commands as well? See the scenario where i am testing the Counter below:
package Counter
case class Counter() {
private var n = 1
def increment(incrementAmount: Int) = {
if (n%100!=0) {
n += incrementAmount
}
}
def get(): Int = n
}
The counter is programmed with a bug. It should not increment with the given amount if n%100 == 0. So if the value of n is x*100, where x is any positive integer, the counter is not incremented. I am testing the counter with the ScalaCheck stateful test below:
import Counter.Counter
import org.scalacheck.commands.Commands
import org.scalacheck.{Gen, Prop}
import scala.util.{Success, Try}
object CounterCommands extends Commands {
type State = Int
type Sut = Counter
def canCreateNewSut(newState: State, initSuts: Traversable[State],
runningSuts: Traversable[Sut]): Boolean = true
def newSut(state: State): Sut = new Counter
def destroySut(sut: Sut): Unit = ()
def initialPreCondition(state: State): Boolean = true
def genInitialState: Gen[State] = Gen.const(1)
def genCommand(state: State): Gen[Command] = Gen.oneOf(Increment(Gen.chooseNum(1, 200000).sample.get), Get)
case class Increment(incrementAmount: Int) extends UnitCommand {
def run(counter: Sut) = counter.increment(incrementAmount)
def nextState(state: State): State = {state+incrementAmount}
def preCondition(state: State): Boolean = true
def postCondition(state: State, success: Boolean) = success
}
case object Get extends Command {
type Result = Int
def run(counter: Sut): Result = counter.get()
def nextState(state: State): State = state
def preCondition(state: State): Boolean = true
def postCondition(state: State, result: Try[Int]): Prop = result == Success(state)
}
}
Everytime the increment command is chosen it is given some arbitrary integer between 1 and 200000 as argument. Running the test gave the following output:
! Falsified after 28 passed tests.
> Labels of failing property:
initialstate = 1
seqcmds = (Increment(1); Increment(109366); Increment(1); Increment(1); Inc
rement(104970); Increment(27214); Increment(197045); Increment(1); Increm
ent(54892); Get => 438600)
> ARG_0: Actions(1,List(Increment(1), Increment(109366), Increment(1), Incr
ement(1), Increment(104970), Increment(27214), Increment(197045), Increme
nt(1), Increment(54892), Get),List())
> ARG_0_ORIGINAL: Actions(1,List(Get, Get, Increment(1), Increment(109366),
Get, Get, Get, Get, Increment(1), Get, Increment(1), Increment(104970),
Increment(27214), Get, Increment(197045), Increment(1), Increment(54892),
Get, Get, Get, Get, Get, Increment(172491), Get, Increment(6513), Get, I
ncrement(57501), Increment(200000)),List())
ScalaCheck did indeed shrink the commands needed to find the bug (as can be seen in ARG_0) but it did not shrink the data inside the commands. It ended up with a much larger counter value (438600) than what is actually needed to find the bug. If the first increment command was given 99 as argument the bug would be found.
Is there any way in ScalaCheck to shrink the data inside the commands when running stateful tests? The ScalaCheck version used is 1.14.1.
EDIT:
I tried simplifying the bug (and only incrementing if n!=10) and added the shrinker that was suggested by Levi and could still not get it to work. The whole runnable code can be seen below:
package LocalCounter
import org.scalacheck.commands.Commands
import org.scalacheck.{Gen, Prop, Properties, Shrink}
import scala.util.{Success, Try}
case class Counter() {
private var n = 1
def increment(incrementAmount: Int) = {
if (n!=10) {
n += incrementAmount
}
}
def get(): Int = n
}
object CounterCommands extends Commands {
type State = Int
type Sut = Counter
def canCreateNewSut(newState: State, initSuts: Traversable[State],
runningSuts: Traversable[Sut]): Boolean = true
def newSut(state: State): Sut = new Counter
def destroySut(sut: Sut): Unit = ()
def initialPreCondition(state: State): Boolean = true
def genInitialState: Gen[State] = Gen.const(1)
def genCommand(state: State): Gen[Command] = Gen.oneOf(Increment(Gen.chooseNum(1, 40).sample.get), Get)
case class Increment(incrementAmount: Int) extends UnitCommand {
def run(counter: Sut) = counter.increment(incrementAmount)
def nextState(state: State): State = {state+incrementAmount}
def preCondition(state: State): Boolean = true
def postCondition(state: State, success: Boolean) = success
}
case object Get extends Command {
type Result = Int
def run(counter: Sut): Result = counter.get()
def nextState(state: State): State = state
def preCondition(state: State): Boolean = true
def postCondition(state: State, result: Try[Int]): Prop = result == Success(state)
}
implicit val shrinkCommand: Shrink[Command] = Shrink({
case Increment(amt) => Shrink.shrink(amt).map(Increment(_))
case Get => Stream.empty
})
}
object CounterCommandsTest extends Properties("CounterCommands") {
CounterCommands.property().check()
}
Running the code gave the following output:
! Falsified after 4 passed tests.
> Labels of failing property:
initialstate = 1
seqcmds = (Increment(9); Increment(40); Get => 10)
> ARG_0: Actions(1,List(Increment(9), Increment(40), Get),List())
> ARG_0_ORIGINAL: Actions(1,List(Increment(9), Increment(34), Increment(40)
, Get),List())
Which is not the minimal example.
You should be able to define a custom Shrink for Command along these lines:
implicit val shrinkCommand: Shrink[Command] = Shrink({
case Increment(amt) => shrink(amt).map(Increment(_))
case Get => Stream.empty
}
Note that, because Stream is deprecated in Scala 2.13, you may need to disable warnings in Scala 2.13 (Scalacheck 1.15 will allow LazyList to be used to define shrinks).
So I have a webworker:
package pWorker
import scala.scalajs.js._
import org.scalajs.dom._
object WorkScript extends scala.scalajs.js.JSApp
{
def main(): Unit =
{
val x = 4
val y = 8
val z = x + y
println("Worker x + y =" -- z.toString)
}
}
When launched from the primary JavaScript thread the worker prints to the console as desired. But I can't find a way to call the equivalents of:
self.addEventListener('message', function(e) {}
self.postMessage(e.data);
that I would call from a straight Javascript file.
As always in Scala.js, if you do not find existing types for a JavaScript API, you have the option to go with dynamic types or to write your own.
Dynamic types
val self = js.Dynamic.global
self.addEventListener("message", { (e: dom.MessageEvent) =>
...
}
self.postMessage(someData)
Static types
#js.native
object WorkerGlobal extends js.GlobalScope {
def addEventListener(`type`: String, f: js.Function): Unit = js.native
def postMessage(data: js.Any): Unit = js.native
}
WorkerGlobal.addEventListener("message", { (e: dom.MessageEvent) =>
...
}
WorkerGlobal.postMessage(someData)
I'm trying implement section 17.9 Workers in Scala by Example book with Scala version 2.11.7.
Import statements:
import scala.concurrent._, scala.concurrent.ops._
was error "ops not is member of scala.concurrent". I did google and known that concurrent.ops is deprecated and rather by future, change import statement to:
import scala.concurrent._, scala.concurrent.Future._
Entire class source:
import scala.concurrent._
import scala.concurrent.Future._
class ComputeServer(n: Int) {
private abstract class Job {
type T
def task: T
def res(x: T)
}
private val openJobs = new Channel[Job]()
private def processor(i: Int) {
while(true) {
val job = openJobs.read
job.res(job.task)
}
}
def future[A](p: => A): () => A = {
val reply = new SyncVar[A]()
openJobs.write{
new Job{
type T = A
def task = p
def res(x: A) = reply.put(x)
}
}
() => reply.get
}
spawn(replicate(0, n){processor})
}
But occurs errors in line: spawn(replicate(0, n){processor})
not found: value spawn
not found: value replicate
missing arguments for method processor in class ComputeServer; follow this method with `_' if you want to treat it as a partially applied function
What're spawn, replicate, processor function in version 2.11.7?
one can create method spawn like this:
def spawn(p: => Unit) {
val t = new Thread() { override def run() = p }
t.start()
}
I have defined the following macros to get file, line and object/class from current location:
http://pastebin.com/UsNLemnK
Using SBT, I have defined two projects, in order to compile the macros first, then the actual project using these macros.
The purpose of these macros are to be be used in a log method:
def log( msg: Any, srcFile: String = "", srcLine: String = "", srcClass:String = "")
I am then using this log method as follows:
log(msg, s"$F_",s"$L_",s"$C_")
where F_, L_ and C_ are defined in the macro.
Now, I would like to create a shortcut to avoid this boilerplate and just call:
log(msg)
which should automatically be replaced by
log(msg, s"$F_",s"$L_",s"$C_")
I could define a macro to do this:
def log_(msg: String) : Unit = macro logImpl
def logImpl( c: Context )(msg: c.Expr[String]): c.Expr[Unit] = {
import c.universe._
reify( log(msg.splice, srcFile=s"$F_", srcLine=s"$L_", srcClass=s"$C_") )
}
but again, this macro needs to be compiled before the project, where the log function itself is defined... So I don't see how to solve the compilation dependencies cycle...
Any suggestion about how to do this?
Thanks
Barring the use of macro annotations (which would necessarily and significantly alter your API's syntax), the problem you have to face is that you need the type-checked identifier of your log function.
Since you can't import the entire log implementation, a solution would be to:
wrap the method into a trait,
define this trait in the "macro" project,
add an implicit parameter to the log_ method,
in your "main" project, create an implementation of this trait, and instantiate this implementation in an implicit val visible everywhere you'd like to use the log_ macro (in the package object for example).
Of course, you could also use a simple FunctionN here and avoid the trait definition and implementation, but this way you'll avoid potential conflicts with other same-typed implicits.
In general, your code would resemble the following:
//"macro" project
trait EncapsulatingTrait {
def yourMethod(...)
}
object Macros {
def myMacro(...)(implicit param: EncapsulatingTrait) = macro myMacroImpl
def myMacroImpl( c: Context )(...)
(param: c.Expr[EncapsulatingTrait]): c.Expr[...] = {
import c.universe._
reify(param.splice.yourMethod(...))
}
}
//--------------------------
//"main" project
class Impl extends EncapsulatingTrait {
def yourMethod(...)
}
...
implicit val defaultParam = new Impl
import Macros.myMacro
myMacro(...)
In your specific case, here's how an implementation could look like:
//"macro" project
package yourpackage
import java.io.File
import language.experimental.macros
import scala.reflect.macros.Context
trait LogFunction {
def log( msg: Any, srcFile: String = "", srcLine: Int = -1, srcClass:String = "")
}
object Macros {
// get current line in source code
def L_ : Int = macro lineImpl
def lineImpl( c: Context ): c.Expr[Int] = {
import c.universe._
val line = Literal( Constant( c.enclosingPosition.line ) )
c.Expr[Int]( line )
}
// get current file from source code (relative path)
def F_ : String = macro fileImpl
def fileImpl( c: Context ): c.Expr[String] = {
import c.universe._
val absolute = c.enclosingPosition.source.file.file.toURI
val base = new File( "." ).toURI
val path = Literal( Constant( c.enclosingPosition.source.file.file.getName() ) )
c.Expr[String]( path )
}
// get current class/object (a bit sketchy)
def C_ : String = macro classImpl
def classImpl( c: Context ): c.Expr[String] = {
import c.universe._
val class_ = Literal( Constant( c.enclosingClass.toString.split(" ")( 1 ) ) )
c.Expr[String]( class_ )
}
def log_(msg: String)(implicit logFunc: LogFunction) : Unit = macro logImpl
def logImpl( c: Context )(msg: c.Expr[String])(logFunc: c.Expr[LogFunction]): c.Expr[Unit] = {
import c.universe._
reify( logFunc.splice.log(msg.splice, srcFile=fileImpl(c).splice, srcLine=lineImpl(c).splice, srcClass=classImpl(c).splice) )
}
}
//--------------------------
//"main" project
import yourpackage.LogFunction
class LogImpl extends LogFunction {
def log( msg: Any, srcFile: String = "", srcLine: Int = -1, srcClass:String = "") {
println(List(msg,srcFile,srcLine,srcClass).mkString("|"))
}
}
object testLog {
def main(args: Array[String]): Unit = {
implicit val defaultLog = new LogImpl
import yourpackage.Macros.log_
log_("blah")
}
}
(note that I had to correct the signature of log_ and tweak the macro call a bit)
I have an external process that I would like to treat as a
function from String=>String. Given a line of input, it will respond with a single line of output. It seems that I should use
scala.sys.process, which is clearly an elegant library that makes many
shell operations easily accessible from within scala. However, I
can't figure out how to perform this simple use case.
If I write a single line to the process' stdin, it prints the result
in a single line. How can I use sys.process to create a wrapper so I
can use the process interactively? For example, if I had an
implementation for ProcessWrapper, here is a program and it's output:
// abstract definition
class ProcessWrapper(executable: String) {
def apply(line: String): String
}
// program using an implementation
val process = new ProcessWrapper("cat -b")
println(process("foo"))
println(process("bar"))
println(process("baz"))
Output:
1 foo
2 bar
3 baz
It is important that the process is not reloaded for each call to process because there is a significant initialization step.
So - after my comment - this would be my solution
import java.io.BufferedReader
import java.io.File
import java.io.InputStream
import java.io.InputStreamReader
import scala.annotation.tailrec
class ProcessWrapper(cmdLine: String, lineListenerOut: String => Unit, lineListenerErr: String => Unit,
finishHandler: => Unit,
lineMode: Boolean = true, envp: Array[String] = null, dir: File = null) {
class StreamRunnable(val stream: InputStream, listener: String => Unit) extends Runnable {
def run() {
try {
val in = new BufferedReader(new InputStreamReader(this.stream));
#tailrec
def readLines {
val line = in.readLine
if (line != null) {
listener(line)
readLines
}
}
readLines
}
finally {
this.stream.close
finishHandler
}
}
}
val process = Runtime.getRuntime().exec(cmdLine, envp, dir);
val outThread = new Thread(new StreamRunnable(process.getInputStream, lineListenerOut), "StreamHandlerOut")
val errThread = new Thread(new StreamRunnable(process.getErrorStream, lineListenerErr), "StreamHandlerErr")
val sendToProcess = process.getOutputStream
outThread.start
errThread.start
def apply(txt: String) {
sendToProcess.write(txt.getBytes)
if (lineMode)
sendToProcess.write('\n')
sendToProcess.flush
}
}
object ProcessWrapper {
def main(args: Array[String]) {
val process = new ProcessWrapper("python -i", txt => println("py> " + txt),
err => System.err.println("py err> " + err), System.exit(0))
while (true) {
process(readLine)
}
}
}
The main part is the StreamRunnable, where the process is read in a thread and the received line is passed on to a "LineListener" (a simple String => Unit - function).
The main is just a sample implementation - calling python ;)
I'm not sure, but you want somethings like that ?
case class ProcessWrapper(executable: String) {
import java.io.ByteArrayOutputStream
import scala.concurrent.duration.Duration
import java.util.concurrent.TimeUnit
lazy val process = sys.runtime.exec(executable)
def apply(line: String, blockedRead: Boolean = true): String = {
process.getOutputStream().write(line.getBytes())
process.getOutputStream().flush()
val r = new ByteArrayOutputStream
if (blockedRead) {
r.write(process.getInputStream().read())
}
while (process.getInputStream().available() > 0) {
r.write(process.getInputStream().read())
}
r.toString()
}
def close() = process.destroy()
}
val process = ProcessWrapper("cat -b")
println(process("foo\n"))
println(process("bar\n"))
println(process("baz\n"))
println(process("buz\n"))
println(process("puz\n"))
process.close
Result :
1 foo
2 bar
3 baz
4 buz
5 puz
I think that PlayCLI is a better way.
http://blog.greweb.fr/2013/01/playcli-play-iteratees-unix-pipe/ came across this today and looks exactly like what you want
How about using an Akka actor. The actor can have state and thus a reference to an open program (in a thread). You can send messages to that actor.
ProcessWrapper might be a typed actor itself or just something that converts the calls of a function to a call of an actor. If you only have 'process' as method name, then wrapper ! "message" would be enough.
Having a program open and ready to receive commands sounds like an actor that receives messages.
Edit: Probably I got the requirements wrong. You want to send multiple lines to the same process. That's not possible with the below solution.
One possibility would be to add an extension method to the ProcessBuilder that allows for taking the input from a string:
implicit class ProcessBuilderWithStringInput(val builder: ProcessBuilder) extends AnyVal {
// TODO: could use an implicit for the character set
def #<<(s: String) = builder.#<(new ByteArrayInputStream(s.getBytes))
}
You can now use the method like this:
scala> ("bc":ProcessBuilder).#<<("3 + 4\n").!!
res9: String =
"7
"
Note that the type annotation is necessary, because we need two conversions (String -> ProcessBuilder -> ProcessBuilderWithStringInput, and Scala will only apply one conversion automatically.