I have a workflow with ZonedDatetime of UTC as an argument. During the unit test with test env, I see the UTC zone is dropped while running the workflow. What would be the cause of this behavior?
[UPDATE] It turns out the timezone is not dropped, but somehow converted to Zone id of "Z" instead of the original "UTC", although they're basically the same. Sample code,
data class WorkflowArgs(
val currentTime: ZonedDateTime
)
class WorkflowImpl {
override fun execute(workflowArgs: WorkflowArgs) {
activityStub.doSomeActivity(
workflowArgs.currentTime // Zone ID of "Z"
)
}
}
WorkflowClient.start(workflowStub::execute, WorkflowArgs(ZonedDateTime.now(ZoneId.of("UTC"))) // Zone ID of "UTC"
You need to customize the DataConverter.
DataConvert is for converting between your argument(Class) and byte array so that your client can send the argument to Cadence server, and then server send it to workflow worker.
For example, you can do something like below.
NOTE that this is just an example for you to implement DataConverter. To encode/decode ZonedDataTime you need to find out a way to do it correctly. You may need to refer to Spring Data JPA - ZonedDateTime format for json serialization
data class WorkflowArgs(
val currentTime: ZonedDateTime
)
fun main1() { // this doesn't work
val dataConverter = JsonDataConverter.getInstance()
val args = WorkflowArgs( ZonedDateTime.now())
println(args.currentTime.zone)
val encoded = dataConverter.toData(args)
val decoded: WorkflowArgs = dataConverter.fromData(encoded, args.javaClass, args.javaClass)
println(decoded.currentTime.zone)
/**
* Results:
America/Los_Angeles
Exception in thread "main" com.uber.cadence.converter.DataConverterException: when parsing:"{"currentTime":{"dateTime":{"date":{"year":2021,"month":9,"day":13},"time":{"hour":9,"minute":14,"second":46,"nano":517000000}},"offset":{"totalSeconds":-25200},"zone":{"id":"America/Los_Angeles"}}}" into following types: [class com.uber.cadence.converter.WorkflowArgs]
at com.uber.cadence.converter.JsonDataConverter.fromData(JsonDataConverter.java:111)
at com.uber.cadence.converter.TestZonedDatatimeKt.main(TestZonedDatatime.kt:14)
at com.uber.cadence.converter.TestZonedDatatimeKt.main(TestZonedDatatime.kt)
Caused by: java.lang.RuntimeException: Failed to invoke java.time.ZoneId() with no args
at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:113)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:212)
...
at com.uber.cadence.converter.JsonDataConverter.fromData(JsonDataConverter.java:109)
... 2 more
*/
}
fun main() { // you need to implement another DataConverter like this
val dataConverter = MyDataConvert()
val args = WorkflowArgs( ZonedDateTime.now())
println(args.currentTime.zone)
val encoded = dataConverter.toData(args)
val decoded: WorkflowArgs = dataConverter.fromData(encoded, args.javaClass, args.javaClass)
println(decoded.currentTime.zone)
/**
Resutls:
America/Los_Angeles
America/Los_Angeles
**/
}
class MyDataConvert:DataConverter{
val originalDataConverter = JsonDataConverter.getInstance()
override fun toData(vararg value: Any?): ByteArray {
if( (value.size == 1) && (value[0] is WorkflowArgs)){
val v = value[0] as WorkflowArgs
val zoneId = v.currentTime.zone.id
val br = zoneId.toByteArray()
return br
}else{
return originalDataConverter.toData(value)
}
}
override fun <T : Any?> fromData(content: ByteArray?, valueClass: Class<T>?, valueType: Type?): T {
if(valueClass!!.canonicalName.contains("WorkflowArgs") ){
val zi = String(content!!)
val zoneId: ZoneId = ZoneId.of(zi)
val dt = ZonedDateTime.now(zoneId)
return WorkflowArgs(dt) as T
}else{
return originalDataConverter.fromData(content, valueClass, valueType)
}
}
override fun fromDataArray(content: ByteArray?, vararg valueType: Type?): Array<Any> {
if( (valueType.size == 1) && (valueType[0]!!.typeName.contains("WorkflowArgs"))){
val zi = String(content!!)
val zoneId: ZoneId = ZoneId.of(zi)
val dt = ZonedDateTime.now(zoneId)
val arr = arrayOf(WorkflowArgs(dt))
return arr as Array<Any>
}
return originalDataConverter.fromDataArray(content, *valueType)
}
}
This is another example of implmenting DataConverter: https://github.com/uber/cadence-java-samples/pull/37
Related
I'm trying to assert date and time displayed on the page
Problem is it's returning value of "2017-03-11T09:00" instead of "2017-03-11 09:00:00" and I'm confused why as the pattern = yyyy-MM-dd HH:mm:ss
Any ideas?
def getDate :String = {
val timeStamp = find(xpath("//*[#id=\"content\"]/article/div/div/table/tbody/tr[5]/td/div/p[4]")).get.underlying.getText
val stripDate: Array[String] = timeStamp.split("Timestamp:\n")
stripDate(1)
}
def datePattern(date: String): LocalDateTime = {
val pattern: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
val result = LocalDateTime.parse(date, pattern)
result
}
def checkDatePattern() = datePattern(getDate).toString shouldBe getDate
The DateTimeFormatter only gets used for the parse operation. It doesn't influence the result of toString. If you want to convert your LocalDateTime to a String in a certain format you have to call
date.format(pattern)
I've managed to get the result I wanted by just deleting some parts of the code. As long as the date is in displayed in the correct format, the test passes if it's displayed in an incorrect format it fails, which is good enough for me. Thanks for your input. CASE CLOSED
def getDate :String = {
val timeStamp = find(xpath("//*[#id=\"content\"]/article/div/div/table/tbody/tr[5]/td/div/p[4]")).get.underlying.getText
val stripDate: Array[String] = timeStamp.split("Timestamp:\n")
stripDate(1)
}
def datePattern(date: String): LocalDateTime = {
val pattern: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
LocalDateTime.parse(date, pattern)
}
def checkDatePattern() = datePattern(getDate)
I am having a hard to come up with a solution to time individual functions in a Scala map operation. Let's say I have the following line in my code:
val foo = data.map(d => func1(d).func2())
where data is a Seq of N elements. How would I go about timing how long my program has executed func1 in total and func2 in total? Since it is a map operation, these functions will be called N times, so each time record should be added to a cumulative time record.
How can I do this without breaking the Scala syntax?
NB: I want to end up with totalTime_inFunc1 and totalTime_inFunc2.
Let's say, func2() returns YourType. Then, you need to return tuple (YourType, Long, Long) from function inside map, where second tuple element is execution time of func1 and third element is exec time of func2. After that, you can easily get execution time from seq of tuples using sum:
val fooWithTime = {
data.map(d => {
def now = System.currentTimeMillis
val beforeFunc1 = now
val func1Result = func1(d)
val func1Time = now - beforeFunc1
val beforeFun2 = now
val result = func1Result.func2()
(result, func1Time, now - beforeFun2)
}
}
val foo = fooWithTime.map(_._1)
val totalTimeFunc1 = fooWithTime.map(_._2).sum
val totalTimeFunc2 = fooWithTime.map(_._3).sum
Also, you can easily use your preferred method of calculating execution time instead of System.currentTimeMillis().
Look at the Closure. You will declare your functions and make them refer to variable in scope, then pass them to map, and make them increment variable from scope.
Edit
code with closures
object closure {
var time1 = 0L
var time2 = 0L
def time[R](block: => R)(time: Int): R = {
val t0 = System.nanoTime()
val result = block // call-by-name
val t1 = System.nanoTime()
if (time==1)
time1 += t1-t0
else
time2 += t1-t0
result
}
def fun1(i: Int): Int = {
time{i+1}(1)
}
def fun2(i: Int): Int = {
time{i+2}(2)
}
}
val data = List(1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7)
val foo = data.map(d => closure.fun2(closure.fun1(d)))
closure.time1 // res4: Long = 22976
closure.time2 // res5: Long = 25438
Edit 2
object closure {
var time1 = 0L
var time2 = 0L
def time[R](block: => R)(time: Int): R = {
val t0 = System.nanoTime()
val result = block // call-by-name
val t1 = System.nanoTime()
if (time==1)
time1 += t1-t0
else
time2 += t1-t0
result
}
val data = List(1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7)
val test = new test;
val foo = data.map(d => {
val fun1 = time{test.fun1(d)}(1)
time{fun1.fun2(d)}(2)
})
}
val s = Seq(1, 2, 3)
val (mappedSeq, totalTime) = s.map(x => {
// call your map methods here and time
// x is the mapped element
val timing = 5.5
// then return the tuple with mapped element and time taken for the map function
(x, timing)
}).foldLeft((Seq.empty[Int], 0d))((accumulator, pair) => (accumulator._1 :+ pair._1, accumulator._2 + pair._2))
println(totalTime)
println(mappedSeq.mkString(", "))
Trying to use streaming api of 0.10.0 flink version in scala 2.10.4. While trying to compile this first version:
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala.DataStream
import org.apache.flink.streaming.api.windowing.time._
object Main {
def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val text = env.socketTextStream("localhost", 9999)
val words : DataStream[String] = text.flatMap[String](
new Function[String,TraversableOnce[String]] {
def apply(line:String):TraversableOnce[String] = line.split(" ")
})
env.execute("Window Stream wordcount")
}
}
I am getting compile time error:
[error] found : String => TraversableOnce[String]
[error] required: org.apache.flink.api.common.functions.FlatMapFunction[String,String]
[error] new Function[String,TraversableOnce[String]] { def apply(line:String):TraversableOnce[String] = line.split(" ")})
[error] ^
And in decompiled version of DataStream.class that I have included to project there are functions that accept such type (the last one):
public <R> DataStream<R> flatMap(FlatMapFunction<T, R> flatMapper, TypeInformation<R> evidence$12, ClassTag<R> evidence$13) {
if (flatMapper == null) {
throw new NullPointerException("FlatMap function must not be null.");
}
TypeInformation outType = (TypeInformation)Predef..MODULE$.implicitly(evidence$12);
return package..MODULE$.javaToScalaStream((org.apache.flink.streaming.api.datastream.DataStream)this.javaStream.flatMap(flatMapper).returns(outType));
}
public <R> DataStream<R> flatMap(Function2<T, Collector<R>, BoxedUnit> fun, TypeInformation<R> evidence$14, ClassTag<R> evidence$15) {
if (fun == null) {
throw new NullPointerException("FlatMap function must not be null.");
}
Function2<T, Collector<R>, BoxedUnit> cleanFun = this.clean((F)fun);
.anon flatMapper = new /* Unavailable Anonymous Inner Class!! */;
return this.flatMap((FlatMapFunction<T, R>)flatMapper, evidence$14, evidence$15);
}
public <R> DataStream<R> flatMap(Function1<T, TraversableOnce<R>> fun, TypeInformation<R> evidence$16, ClassTag<R> evidence$17) {
if (fun == null) {
throw new NullPointerException("FlatMap function must not be null.");
}
Function1<T, TraversableOnce<R>> cleanFun = this.clean((F)fun);
.anon flatMapper = new /* Unavailable Anonymous Inner Class!! */;
return this.flatMap((FlatMapFunction<T, R>)flatMapper, evidence$16, evidence$17);
}
What could be wrong here? I would be grateful if you could give some insight.
Thank you in advance.
The problem is that you are importing the Java StreamExecutionEnvironment of Flink: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.
You have to use the Scala variant of the StreamExecutionEnvironment like this: import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.
With that change, everything is successfully building!
Original answer:
The problem is that you are passing a Function to the flatMap() method. However flatMap() expects a FlatMapFunction.
val words : DataStream[String] = text.flatMap[String](
new FlatMapFunction[String,String] {
override def flatMap(t: String, collector: Collector[String]): Unit = t.split(" ")
})
my application takes in a string like this "2002-10-15 10:55:01.000000". I need to validate inside scala script that the string is a valid for a db2 timestamp.
In general (I'd guess) you would do it moslty the same way as in java with either java.text.DateFormat or joda.time.DateTimeFormat (see Joda time).
A simple example:
import java.text.SimpleDateFormat
import java.util.Date
import scala.util.Try
val date = "2002-10-15 10:55:01.000000"
val formatter = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss.SSSS")
val test = Try[Date](formatter.parse(date))
would give you:
test: scala.util.Try[java.util.Date] = Success(Tue Oct 15 10:55:01 CEST 2002)
Then you could match:
test match {
case Success(date) => // ok
case Failure(exception) => // not ok
}
You should use SimpleDateFormat of java to do that in scala :
object DateParser {
def isValid(f : String, d : String) = {
try {
val format = new java.text.SimpleDateFormat(f)
format.parse(d)
}catch(java.text.ParseException e) {
false
}
}
def main(args : Array[String]) {
val format = "yyyy-MM-dd k:m:s"
println(isValid(format,"2002-10-15 10:55:01.000000"))
println(isValid(format,"2002-10-1510:55:01.000000"))
}
}
To embed Scala as a "scripting language", I need to be able to compile text fragments to simple objects, such as Function0[Unit] that can be serialised to and deserialised from disk and which can be loaded into the current runtime and executed.
How would I go about this?
Say for example, my text fragment is (purely hypothetical):
Document.current.elements.headOption.foreach(_.open())
This might be wrapped into the following complete text:
package myapp.userscripts
import myapp.DSL._
object UserFunction1234 extends Function0[Unit] {
def apply(): Unit = {
Document.current.elements.headOption.foreach(_.open())
}
}
What comes next? Should I use IMain to compile this code? I don't want to use the normal interpreter mode, because the compilation should be "context-free" and not accumulate requests.
What I need to get hold off from the compilation is I guess the binary class file? In that case, serialisation is straight forward (byte array). How would I then load that class into the runtime and invoke the apply method?
What happens if the code compiles to multiple auxiliary classes? The example above contains a closure _.open(). How do I make sure I "package" all those auxiliary things into one object to serialize and class-load?
Note: Given that Scala 2.11 is imminent and the compiler API probably changed, I am happy to receive hints as how to approach this problem on Scala 2.11
Here is one idea: use a regular Scala compiler instance. Unfortunately it seems to require the use of hard disk files both for input and output. So we use temporary files for that. The output will be zipped up in a JAR which will be stored as a byte array (that would go into the hypothetical serialization process). We need a special class loader to retrieve the class again from the extracted JAR.
The following assumes Scala 2.10.3 with the scala-compiler library on the class path:
import scala.tools.nsc
import java.io._
import scala.annotation.tailrec
Wrapping user provided code in a function class with a synthetic name that will be incremented for each new fragment:
val packageName = "myapp"
var userCount = 0
def mkFunName(): String = {
val c = userCount
userCount += 1
s"Fun$c"
}
def wrapSource(source: String): (String, String) = {
val fun = mkFunName()
val code = s"""package $packageName
|
|class $fun extends Function0[Unit] {
| def apply(): Unit = {
| $source
| }
|}
|""".stripMargin
(fun, code)
}
A function to compile a source fragment and return the byte array of the resulting jar:
/** Compiles a source code consisting of a body which is wrapped in a `Function0`
* apply method, and returns the function's class name (without package) and the
* raw jar file produced in the compilation.
*/
def compile(source: String): (String, Array[Byte]) = {
val set = new nsc.Settings
val d = File.createTempFile("temp", ".out")
d.delete(); d.mkdir()
set.d.value = d.getPath
set.usejavacp.value = true
val compiler = new nsc.Global(set)
val f = File.createTempFile("temp", ".scala")
val out = new BufferedOutputStream(new FileOutputStream(f))
val (fun, code) = wrapSource(source)
out.write(code.getBytes("UTF-8"))
out.flush(); out.close()
val run = new compiler.Run()
run.compile(List(f.getPath))
f.delete()
val bytes = packJar(d)
deleteDir(d)
(fun, bytes)
}
def deleteDir(base: File): Unit = {
base.listFiles().foreach { f =>
if (f.isFile) f.delete()
else deleteDir(f)
}
base.delete()
}
Note: Doesn't handle compiler errors yet!
The packJar method uses the compiler output directory and produces an in-memory jar file from it:
// cf. http://stackoverflow.com/questions/1281229
def packJar(base: File): Array[Byte] = {
import java.util.jar._
val mf = new Manifest
mf.getMainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
val bs = new java.io.ByteArrayOutputStream
val out = new JarOutputStream(bs, mf)
def add(prefix: String, f: File): Unit = {
val name0 = prefix + f.getName
val name = if (f.isDirectory) name0 + "/" else name0
val entry = new JarEntry(name)
entry.setTime(f.lastModified())
out.putNextEntry(entry)
if (f.isFile) {
val in = new BufferedInputStream(new FileInputStream(f))
try {
val buf = new Array[Byte](1024)
#tailrec def loop(): Unit = {
val count = in.read(buf)
if (count >= 0) {
out.write(buf, 0, count)
loop()
}
}
loop()
} finally {
in.close()
}
}
out.closeEntry()
if (f.isDirectory) f.listFiles.foreach(add(name, _))
}
base.listFiles().foreach(add("", _))
out.close()
bs.toByteArray
}
A utility function that takes the byte array found in deserialization and creates a map from class names to class byte code:
def unpackJar(bytes: Array[Byte]): Map[String, Array[Byte]] = {
import java.util.jar._
import scala.annotation.tailrec
val in = new JarInputStream(new ByteArrayInputStream(bytes))
val b = Map.newBuilder[String, Array[Byte]]
#tailrec def loop(): Unit = {
val entry = in.getNextJarEntry
if (entry != null) {
if (!entry.isDirectory) {
val name = entry.getName
// cf. http://stackoverflow.com/questions/8909743
val bs = new ByteArrayOutputStream
var i = 0
while (i >= 0) {
i = in.read()
if (i >= 0) bs.write(i)
}
val bytes = bs.toByteArray
b += mkClassName(name) -> bytes
}
loop()
}
}
loop()
in.close()
b.result()
}
def mkClassName(path: String): String = {
require(path.endsWith(".class"))
path.substring(0, path.length - 6).replace("/", ".")
}
A suitable class loader:
class MemoryClassLoader(map: Map[String, Array[Byte]]) extends ClassLoader {
override protected def findClass(name: String): Class[_] =
map.get(name).map { bytes =>
println(s"defineClass($name, ...)")
defineClass(name, bytes, 0, bytes.length)
} .getOrElse(super.findClass(name)) // throws exception
}
And a test case which contains additional classes (closures):
val exampleSource =
"""val xs = List("hello", "world")
|println(xs.map(_.capitalize).mkString(" "))
|""".stripMargin
def test(fun: String, cl: ClassLoader): Unit = {
val clName = s"$packageName.$fun"
println(s"Resolving class '$clName'...")
val clazz = Class.forName(clName, true, cl)
println("Instantiating...")
val x = clazz.newInstance().asInstanceOf[() => Unit]
println("Invoking 'apply':")
x()
}
locally {
println("Compiling...")
val (fun, bytes) = compile(exampleSource)
val map = unpackJar(bytes)
println("Classes found:")
map.keys.foreach(k => println(s" '$k'"))
val cl = new MemoryClassLoader(map)
test(fun, cl) // should call `defineClass`
test(fun, cl) // should find cached class
}