Error ocurred when run scala script based on Joern! [duplicate] - scala

I want to analyze the if-else structure of the demo.c file through joern and try to write a script with scala, but I found that the running results of scala1 and scala2 are different (the scala1 and scala2 want to express the same meaning). The difference between them is that a function fun is defined in scala1 (actually I want to use this function to achieve other repeated functions), but the result is wrong. Why? Is the parameter type (Traversal [ControlStructure]) of function Fun passed incorrectly?
--------- demo.c ---------
#include <stdio.h>
int main () {
/* local variable definition */
int a = 100;
/* check the boolean condition */
if(a>10) {
if(a>100){
printf("a > 100");
}else{
printf("10<a<100");
}
}else if( a <= 10 && a > 0) {
printf("Value of a is 20\n" );
}else {
printf("None of the values is matching\n" );
}
printf("Exact value of a is: %d\n", a );
if ( x > 10 ) {
printf("111");
}else{
printf("222")
}
while(x++ < MAX) {
if(x!=0) {
int y = 2*x;
sink(y);
}
}
return 0;
}
--------scala script 1--------
open("demo")
def Outermost_layer_branch =
cpg.method("main").block.astChildren.isControlStructure.controlStructureType("IF")
def fun(node:Traversal[ControlStructure]){
def node1 = node.astChildren.isControlStructure.controlStructureType("ELSE")
println(node1.size)
// the result is 1, it's right
def node2 =
node1.astChildren.filter(_.isBlock).astChildren.isControlStructure.controlStructureType("IF")
println(node2.size)
//the result is 0, it's wrong!!
}
fun(Outermost_layer_branch.order(3))
--------scala 2--------
open("demo")
def Outermost_layer_branch=
cpg.method("main").block.astChildren.isControlStructure.controlStructureType("IF")
def node1 = Outermost_layer_branch.order(3).astChildren.isControlStructure.controlStructureType("ELSE")
println(node1.size)
// the result is 1,and it is right!
def node2 = node1.astChildren.filter(_.isBlock).astChildren.isControlStructure.controlStructureType("IF")
println(node2.size)
// the result is 1,and it is right!
If I want to achieve the correct results through scala1, how can I pass parameters? Or is there any other way?

Related

Run scala script based on joern, function parameter error!

I want to analyze the if-else structure of the demo.c file through joern and try to write a script with scala, but I found that the running results of scala1 and scala2 are different (the scala1 and scala2 want to express the same meaning). The difference between them is that a function fun is defined in scala1 (actually I want to use this function to achieve other repeated functions), but the result is wrong. Why? Is the parameter type (Traversal [ControlStructure]) of function Fun passed incorrectly?
--------- demo.c ---------
#include <stdio.h>
int main () {
/* local variable definition */
int a = 100;
/* check the boolean condition */
if(a>10) {
if(a>100){
printf("a > 100");
}else{
printf("10<a<100");
}
}else if( a <= 10 && a > 0) {
printf("Value of a is 20\n" );
}else {
printf("None of the values is matching\n" );
}
printf("Exact value of a is: %d\n", a );
if ( x > 10 ) {
printf("111");
}else{
printf("222")
}
while(x++ < MAX) {
if(x!=0) {
int y = 2*x;
sink(y);
}
}
return 0;
}
--------scala script 1--------
open("demo")
def Outermost_layer_branch =
cpg.method("main").block.astChildren.isControlStructure.controlStructureType("IF")
def fun(node:Traversal[ControlStructure]){
def node1 = node.astChildren.isControlStructure.controlStructureType("ELSE")
println(node1.size)
// the result is 1, it's right
def node2 =
node1.astChildren.filter(_.isBlock).astChildren.isControlStructure.controlStructureType("IF")
println(node2.size)
//the result is 0, it's wrong!!
}
fun(Outermost_layer_branch.order(3))
--------scala 2--------
open("demo")
def Outermost_layer_branch=
cpg.method("main").block.astChildren.isControlStructure.controlStructureType("IF")
def node1 = Outermost_layer_branch.order(3).astChildren.isControlStructure.controlStructureType("ELSE")
println(node1.size)
// the result is 1,and it is right!
def node2 = node1.astChildren.filter(_.isBlock).astChildren.isControlStructure.controlStructureType("IF")
println(node2.size)
// the result is 1,and it is right!
If I want to achieve the correct results through scala1, how can I pass parameters? Or is there any other way?

Efficient way to optimise a Scala code to read large file that doesn't fit in memory

Problem Statement Below,
We have a large log file which stores user interactions with an application. The entries in the log file follow the following schema: {userId, timestamp, actionType} where actionType is one of two possible values: [open, close]
Constraints:
The log file is too big to fit in memory on one machine. Also assume that the aggregated data doesn’t fit into memory.
Code has to be able to run on a single machine.
Should not use an out-of-the box implementation of mapreduce or 3rd party database; don’t assume we have a Hadoop or Spark or other distributed computing framework.
There can be multiple entries of each actionType for each user, and there might be missing entries in the log file. So a user might be missing a close record between two open records or vice versa.
Timestamps will come in strictly ascending order.
For this problem, we need to implement a class/classes that computes the average time spent by each user between open and close. Keep in mind that there are missing entries for some users, so we will have to make a choice about how to handle these entries when making our calculations. Code should follow a consistent policy with regards to how we make that choice.
The desired output for the solution should be [{userId, timeSpent},….] for all the users in the log file.
Sample log file (comma-separated, text file)
1,1435456566,open
2,1435457643,open
3,1435458912,open
1,1435459567,close
4,1435460345,open
1,1435461234,open
2,1435462567,close
1,1435463456,open
3,1435464398,close
4,1435465122,close
1,1435466775,close
Approach
Below is the code I've written in Python & Scala, which seems to be not efficient and upto the expectations of the scenario given, I'd like to feedback from community of developers in this forum how better we could optimise this code as per given scenario.
Scala implementation
import java.io.FileInputStream
import java.util.{Scanner, Map, LinkedList}
import java.lang.Long
import scala.collection.mutable
object UserMetrics extends App {
if (args.length == 0) {
println("Please provide input data file name for processing")
}
val userMetrics = new UserMetrics()
userMetrics.readInputFile(args(0),if (args.length == 1) 600000 else args(1).toInt)
}
case class UserInfo(userId: Integer, prevTimeStamp: Long, prevStatus: String, timeSpent: Long, occurence: Integer)
class UserMetrics {
val usermap = mutable.Map[Integer, LinkedList[UserInfo]]()
def readInputFile(stArr:String, timeOut: Int) {
var inputStream: FileInputStream = null
var sc: Scanner = null
try {
inputStream = new FileInputStream(stArr);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
val line: String = sc.nextLine();
processInput(line, timeOut)
}
for ((key: Integer, userLs: LinkedList[UserInfo]) <- usermap) {
val userInfo:UserInfo = userLs.get(0)
val timespent = if (userInfo.occurence>0) userInfo.timeSpent/userInfo.occurence else 0
println("{" + key +","+timespent + "}")
}
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
}
def processInput(line: String, timeOut: Int) {
val strSp = line.split(",")
val userId: Integer = Integer.parseInt(strSp(0))
val curTimeStamp = Long.parseLong(strSp(1))
val status = strSp(2)
val uInfo: UserInfo = UserInfo(userId, curTimeStamp, status, 0, 0)
val emptyUserInfo: LinkedList[UserInfo] = new LinkedList[UserInfo]()
val lsUserInfo: LinkedList[UserInfo] = usermap.getOrElse(userId, emptyUserInfo)
if (lsUserInfo != null && lsUserInfo.size() > 0) {
val lastUserInfo: UserInfo = lsUserInfo.get(lsUserInfo.size() - 1)
val prevTimeStamp: Long = lastUserInfo.prevTimeStamp
val prevStatus: String = lastUserInfo.prevStatus
if (prevStatus.equals("open")) {
if (status.equals(lastUserInfo.prevStatus)) {
val timeSelector = if ((curTimeStamp - prevTimeStamp) > timeOut) timeOut else curTimeStamp - prevTimeStamp
val timeDiff = lastUserInfo.timeSpent + timeSelector
lsUserInfo.remove()
lsUserInfo.add(UserInfo(userId, curTimeStamp, status, timeDiff, lastUserInfo.occurence + 1))
} else if(!status.equals(lastUserInfo.prevStatus)){
val timeDiff = lastUserInfo.timeSpent + curTimeStamp - prevTimeStamp
lsUserInfo.remove()
lsUserInfo.add(UserInfo(userId, curTimeStamp, status, timeDiff, lastUserInfo.occurence + 1))
}
} else if(prevStatus.equals("close")) {
if (status.equals(lastUserInfo.prevStatus)) {
lsUserInfo.remove()
val timeSelector = if ((curTimeStamp - prevTimeStamp) > timeOut) timeOut else curTimeStamp - prevTimeStamp
lsUserInfo.add(UserInfo(userId, curTimeStamp, status, lastUserInfo.timeSpent + timeSelector, lastUserInfo.occurence+1))
}else if(!status.equals(lastUserInfo.prevStatus))
{
lsUserInfo.remove()
lsUserInfo.add(UserInfo(userId, curTimeStamp, status, lastUserInfo.timeSpent, lastUserInfo.occurence))
}
}
}else if(lsUserInfo.size()==0){
lsUserInfo.add(uInfo)
}
usermap.put(userId, lsUserInfo)
}
}
Python Implementation
import sys
def fileBlockStream(fp, number_of_blocks, block):
#A generator that splits a file into blocks and iterates over the lines of one of the blocks.
assert 0 <= block and block < number_of_blocks #Assertions to validate number of blocks given
assert 0 < number_of_blocks
fp.seek(0,2) #seek to end of file to compute block size
file_size = fp.tell()
ini = file_size * block / number_of_blocks #compute start & end point of file block
end = file_size * (1 + block) / number_of_blocks
if ini <= 0:
fp.seek(0)
else:
fp.seek(ini-1)
fp.readline()
while fp.tell() < end:
yield fp.readline() #iterate over lines of the particular chunk or block
def computeResultDS(chunk,avgTimeSpentDict,defaultTimeOut):
countPos,totTmPos,openTmPos,closeTmPos,nextEventPos = 0,1,2,3,4
for rows in chunk.splitlines():
if len(rows.split(",")) != 3:
continue
userKeyID = rows.split(",")[0]
try:
curTimeStamp = int(rows.split(",")[1])
except ValueError:
print("Invalid Timestamp for ID:" + str(userKeyID))
continue
curEvent = rows.split(",")[2]
if userKeyID in avgTimeSpentDict.keys() and avgTimeSpentDict[userKeyID][nextEventPos]==1 and curEvent == "close":
#Check if already existing userID with expected Close event 0 - Open; 1 - Close
#Array value within dictionary stores [No. of pair events, total time spent (Close tm-Open tm), Last Open Tm, Last Close Tm, Next expected Event]
curTotalTime = curTimeStamp - avgTimeSpentDict[userKeyID][openTmPos]
totalTime = curTotalTime + avgTimeSpentDict[userKeyID][totTmPos]
eventCount = avgTimeSpentDict[userKeyID][countPos] + 1
avgTimeSpentDict[userKeyID][countPos] = eventCount
avgTimeSpentDict[userKeyID][totTmPos] = totalTime
avgTimeSpentDict[userKeyID][closeTmPos] = curTimeStamp
avgTimeSpentDict[userKeyID][nextEventPos] = 0 #Change next expected event to Open
elif userKeyID in avgTimeSpentDict.keys() and avgTimeSpentDict[userKeyID][nextEventPos]==0 and curEvent == "open":
avgTimeSpentDict[userKeyID][openTmPos] = curTimeStamp
avgTimeSpentDict[userKeyID][nextEventPos] = 1 #Change next expected event to Close
elif userKeyID in avgTimeSpentDict.keys() and avgTimeSpentDict[userKeyID][nextEventPos]==1 and curEvent == "open":
curTotalTime,closeTime = missingHandler(defaultTimeOut,avgTimeSpentDict[userKeyID][openTmPos],curTimeStamp)
totalTime = curTotalTime + avgTimeSpentDict[userKeyID][totTmPos]
avgTimeSpentDict[userKeyID][totTmPos]=totalTime
avgTimeSpentDict[userKeyID][closeTmPos]=closeTime
avgTimeSpentDict[userKeyID][openTmPos]=curTimeStamp
eventCount = avgTimeSpentDict[userKeyID][countPos] + 1
avgTimeSpentDict[userKeyID][countPos] = eventCount
elif userKeyID in avgTimeSpentDict.keys() and avgTimeSpentDict[userKeyID][nextEventPos]==0 and curEvent == "close":
curTotalTime,openTime = missingHandler(defaultTimeOut,avgTimeSpentDict[userKeyID][closeTmPos],curTimeStamp)
totalTime = curTotalTime + avgTimeSpentDict[userKeyID][totTmPos]
avgTimeSpentDict[userKeyID][totTmPos]=totalTime
avgTimeSpentDict[userKeyID][openTmPos]=openTime
eventCount = avgTimeSpentDict[userKeyID][countPos] + 1
avgTimeSpentDict[userKeyID][countPos] = eventCount
elif curEvent == "open":
#Initialize userid with Open event
avgTimeSpentDict[userKeyID] = [0,0,curTimeStamp,0,1]
elif curEvent == "close":
#Initialize userid with missing handler function since there is no Open event for this User
totaltime,OpenTime = missingHandler(defaultTimeOut,0,curTimeStamp)
avgTimeSpentDict[userKeyID] = [1,totaltime,OpenTime,curTimeStamp,0]
def missingHandler(defaultTimeOut,curTimeVal,lastTimeVal):
if lastTimeVal - curTimeVal > defaultTimeOut:
return defaultTimeOut,curTimeVal
else:
return lastTimeVal - curTimeVal,curTimeVal
def computeAvg(avgTimeSpentDict,defaultTimeOut):
resDict = {}
for k,v in avgTimeSpentDict.iteritems():
if v[0] == 0:
resDict[k] = 0
else:
resDict[k] = v[1]/v[0]
return resDict
if __name__ == "__main__":
avgTimeSpentDict = {}
if len(sys.argv) < 2:
print("Please provide input data file name for processing")
sys.exit(1)
fileObj = open(sys.argv[1])
number_of_chunks = 4 if len(sys.argv) < 3 else int(sys.argv[2])
defaultTimeOut = 60000 if len(sys.argv) < 4 else int(sys.argv[3])
for chunk_number in range(number_of_chunks):
for chunk in fileBlockStream(fileObj, number_of_chunks, chunk_number):
computeResultDS(chunk, avgTimeSpentDict, defaultTimeOut)
print (computeAvg(avgTimeSpentDict,defaultTimeOut))
avgTimeSpentDict.clear() #Nullify dictionary
fileObj.close #Close the file object
Both program above gives desired output, but efficiency is what matters for this particular scenario. Let me know if you've anything better or any suggestions on existing implementation.
Thanks in Advance!!
What you are after is iterator usage. I'm not going to re-write your code, but the trick here is likely to be using an iterator. Fortunately Scala provides decent out of the box tooling for the job.
import scala.io.Source
object ReadBigFiles {
def read(fileName: String): Unit = {
val lines: Iterator[String] = Source.fromFile(fileName).getLines
// now you get iterator semantics for the file line traversal
// that means you can only go through the lines once, but you don't incur a penalty on heap usage
}
}
For your use case, you seem to require a lastUser, so you're dealing with groups of 2 entries. I think you you have two choices, either go for iterator.sliding(2), which will produce iterators for every pair, or simply add recursion to the mix using options.
def navigate(source: Iterator[String], last: Option[User]): ResultType = {
if (source.hasNext) {
val current = source.next()
last match {
case Some(existing) => // compare with previous user etc
case None => navigate(source, Some(current))
}
} else {
// exit recursion, return result
}
}
You can avoid all the code you've written to read the file and so on. If you need to count occurrences, simply build a Map inside your recursion, and increment the occurrences at every step based on your business logic.
from queue import LifoQueue, Queue
def averageTime() -> float:
logs = {}
records = Queue()
with open("log.txt") as fp:
lines = fp.readlines()
for line in lines:
if line[0] not in logs:
logs[line[0]] = LifoQueue()
logs[line[0]].put((line[1], line[2]))
else:
logs[line[0]].put((line[1], line[2]))
for k in logs:
somme = 0
count = 0
while not logs[k].empty():
l = logs[k].get()
somme = (somme + l[0]) if l[1] == "open" else (somme - l[0])
count = count + 1
records.put([k, somme, count//2])
while not records.empty():
record = records.get()
print(f"UserId={record[0]} Avg={record[1]/record[2]}")

Linked List Parameterized constructor

I tried to do a Parameterized constructor for a linked list my program is about to implement a queue by using a liked list so i want to do a parameterized constructor like Queue(int value , int size) and it dose not run or doing a list
this is my code for this problem
Queue(int value,int _size)
{
for(int i = 0; i < _size; ++i)
{
Node* temp = new Node;
temp->data = value;
temp->next = nullptr;
if(head == nullptr)
{
head = tail = temp;
}
else
{
tail->next = temp;
tail = temp;
}
}
}
i expected that the result is to fill the lest by value times size like if i run this function Queue x(20,3) the linked list should be
20 20 20
Since that this is a constructor, The head and tail are not properly initialized to use them. I would suggest adding head = tail = nullptr just before the loop and see what happens.
Follow this code after your node creation. I hope this will work. And do use i++ instead of ++i, as the later will make the loop for size-1 times.
if(head == NULL)
head = temp;
else{
Node *x;
x= head;
while(x->next != NULL)
x = x->next;
x->next = temp;
}

Specify Variable Initialization Order in Scala

I have a special class Model that needs to have its methods called in a very specific order.
I tried doing something like this:
val model = new Model
new MyWrappingClass {
val first = model.firstMethod()
val second = model.secondMethod()
val third = model.thirdMethod()
}
The methods should be called in the order listed, however I am seeing an apparently random order.
Is there any way to get the variable initialization methods to be called in a particular order?
I doubt your methods are called in the wrong order. But to be sure, you can try something like this:
val (first, second, third) = (
model.firstMethod(),
model.secondMethod(),
model.thirdMethod()
)
You likely have some other problem with your code.
I can run 100 million loops where it never gets the order wrong, as follows:
class Model {
var done = Array(false,false,false);
def firstMethod():Boolean = { done(0) = true; done(1) || done(2) };
def secondMethod():Boolean = { done(1) = true; !done(0) || done(2) };
def thirdMethod():Boolean = { done(2) = true; !done(0) || !done(1) };
};
Notice that these methods return a True if done out of order and false when called in order.
Here's your class:
class MyWrappingClass {
val model = new Model;
val first = model.firstMethod()
val second = model.secondMethod()
val third = model.thirdMethod()
};
Our function to check for bad behavior on each trial:
def isNaughty(w: MyWrappingClass):Boolean = { w.first || w.second || w.third };
A short program to test:
var i = 0
var b = false;
while( (i<100000000) && !b ){
b = isNaughty(new MyWrappingClass);
i += 1;
}
if (b){
println("out-of-order behavior occurred");
println(i);
} else {
println("looks good");
}
Scala 2.11.7 on OpenJDK8 / Ubuntu 15.04
Of course this doesn't prove it impossible to have wrong order, only that correct behavior seems highly repeatable in a fairly simple case.

Access non static function from static function

Here is some insight: I am working with UnityScript in Unity 4.6.3. I have one script called Pause.js and it contains this function:
function fadeMusicOut () {
while (audio.volume >= 0.005) {
yield WaitForSeconds(0.1);
Debug.Log("Loop Entered: " + audio.volume);
audio.volume = (audio.volume - 0.015);
}
Another script GameManager.js has this function:
static function Score (wallName : String) {
if (wallName == "rightWall") {
playerScore01 += 1;
}
else {
playerScore02 += 1;
}
if (playerScore01 == SettingsBack.scoreLimit || playerScore02 == SettingsBack.scoreLimit)
{
startParticles = 1;
SettingsBack.gameOver = 1;
BallControl.fadeSound = 1;
yield WaitForSeconds(4);
Camera.main.SendMessage("fadeOut");
Pause.fadeMusic = 1;
SettingsBack.soundVolume = 0;
yield WaitForSeconds(2);
playerScore01 = 0;
playerScore02 = 0;
SettingsBack.soundVolume = oldSoundVol;
Application.LoadLevel("_Menu");
}
}
So pretty much I want to call the fadeMusicOut() function from static function Score, but it will not let me because it says it needs an instance of that object.
The Pause.js script is not attached to any game objects, but it is attached to 2 buttons that call their specific functions. The GameManager.js script is attached to an object called GM. So how can I go about calling fadeMusicOut() from the Score function?
I have tried setting new vars that import the game object but still no luck. I tried making fadeMusicOut() a static function, but it creates many errors.
Any help at all is appreciated.