Using a Beakerx Custom Magic - beaker

I've created a custom Magic command with the intention of generating a spark query programatically. Here's the relevant part of my class that implements the MagicCommandFunctionality:
MagicCommandOutcomeItem execute(MagicCommandExecutionParam magicCommandExecutionParam) {
// get the string that was entered:
String input = magicCommandExecutionParam.command.substring(MAGIC.length())
// use the input to generate a query
String generatedQuery = Interpreter.interpret(input)
MIMEContainer result = Text(generatedQuery);
return new MagicCommandOutput(MagicCommandOutcomeItem.Status.OK, result.getData().toString());
}
This works splendidly. It returns the command that I generated. (As text)
My question is -- how do I coerce the notebook into evaluating that value in the cell? My guess is that a SimpleEvaluationObject and TryResult are involved, but I can't find any examples of their use
Rather than creating the MagicCommandOutput I probably want the Kernel to create one for me. I see that the KernelMagicCommand has an execute method that would do that. Anyone have any ideas?

Okay, I found one way to do it. Here's my solution:
You can ask the current kernelManager for the kernel you're interested in,
then call PythonEntryPoint.evaluate. It seems to do the job!
#Override
MagicCommandOutcomeItem execute(MagicCommandExecutionParam magicCommandExecutionParam) {
String input = magicCommandExecutionParam.command.substring(MAGIC.length() + 1)
// this is the Scala code I want to evaluate:
String codeToExecute = <your code here>
KernelFunctionality kernel = KernelManager.get()
PythonEntryPoint pep = kernel.getPythonEntryPoint(SCALA_KERNEL)
pep.evaluate(codeToExecute)
pep.getShellMsg()
List<Message> messages = new ArrayList<>()
//until there are messages on iopub channel available collect them into response
while (true) {
String iopubMsg = pep.getIopubMsg()
if (iopubMsg == "null") break
try {
Message msg = parseMessage(iopubMsg) //(I didn't show this part)
messages.add(msg)
String commId = (String) msg.getContent().get("comm_id")
if (commId != null) {
kernel.addCommIdManagerMapping(commId, SCALA_KERNEL)
}
} catch (IOException e) {
log.error("There was an error: ${e.getMessage()}")
return new MagicKernelResponse(MagicCommandOutcomeItem.Status.ERROR, messages)
}
}
return new MagicKernelResponse(MagicCommandOutcomeItem.Status.OK, messages)
}

Related

rxjava2 - how to zip Maybe that can be empty?

I am attempting to make 3 web services calls (e.g.: getPhoneNumber, getFirstName, getLastName) and collect the answers into a common object Person. Any of the web services calls can return a Maybe.empty().
When attempting to zip the response together, rxjava2 skips over the zip operation and terminate normally (without having aggregated my answer).
For a simplified example, see below:
#Test
public void maybeZipEmptyTest() throws Exception {
Maybe<Integer> a = Maybe.just(1);
Maybe<Integer> b = Maybe.just(2);
Maybe<Integer> empty = Maybe.empty();
TestObserver<String> observer = Maybe.zip(a, b, empty, (x, y, e) -> {
String output = "test: a "+x+" b "+y+" empty "+e;
return output;
})
.doOnSuccess(output -> {
System.out.println(output);
})
.test();
observer.assertNoErrors();
}
How can we collect empty values within a zip operation instead of having the zip operation skipped/ignored? If this is the wrong pattern to solve this problem, how would you recommend solving this?
For most use cases, leveraging the defaultIfEmpty method is the right way to go.
For representing something which is ultimately optional (doesn't even use default), I used Java 8 Optional type to represent.
For example
#Test
public void maybeZipEmptyTest() throws Exception {
Maybe<Optional<Integer>> a = Maybe.just(Optional.of(1));
Maybe<Optional<Integer>> b = Maybe.just(Optional.of(2));
Maybe<Optional<Integer>> empty = Maybe.just(Optional.empty());
TestObserver<String> observer = Maybe.zip(a, b, empty, (x, y, e) -> {
String output = "test: a "+toStringOrEmpty(x)+" b "+toStringOrEmpty(y)+" empty "+toStringOrEmpty(e);
return output;
})
.doOnSuccess(output -> {
System.out.println(output);
})
.test();
observer.assertNoErrors();
}
private String toStringOrEmpty(Optional<Integer> value){
if(value.isPresent()){
return value.get().toString();
}
else {
return "";
}
}

Relational Queries In parse.com (Unity)

I found example in parse.com. I have 2 objects : Post and Comment, in the Comment objects have a collumn: "parent" pointer to Post obj and I want to join them:
var query = ParseObject.GetQuery ("Comment");
// Include the post data with each comment
query = query.Include("parent");
query.FindAsync().ContinueWith(t => {
IEnumerable<ParseObject> comments = t.Result;
// Comments now contains the last ten comments, and the "post" field
// contains an object that has already been fetched. For example:
foreach (var comment in comments)
{
// This does not require a network access.
string o= comment.Get<string>("content");
Debug.Log(o);
try {
string post = comment.Get<ParseObject>("parent").Get<string>("title");
Debug.Log(post);
} catch (Exception ex) {
Debug.Log(ex);
}
}
});
It worked!
And then, I have 2 objects: User and Gamescore, in the Gamescore objects have a collumn: "playerName" pointer to Post obj I want join them too:
var query = ParseObject.GetQuery ("GameScore");
query.Include ("playerName");
query.FindAsync ().ContinueWith (t =>{
IEnumerable<ParseObject> result = t.Result;
foreach (var item in result) {
Debug.Log("List score: ");
int score = item.Get<int>("score");
Debug.Log(score);
try {
var obj = item.Get<ParseUser>("playerName");
string name = obj.Get<string>("profile");
//string name = item.Get<ParseUser>("playerName").Get<string>("profile");
Debug.Log(name);
} catch (Exception ex) {
Debug.Log(ex);
}
}
});
but It isn't working, Please help me!
Why didn't you do the following like you did your first example:
query = query.Include ("playerName");
you just have -
query.Include ("playerName");
One solution would be to ensure that your ParseUser object is properly fetched. ie:
var obj = item.Get<ParseUser>("playerName");
Task t = obj.FetchIfNeededAsync();
while (!t.IsCompleted) yield return null;
Then you can do this without worrying:
string name = obj.Get<string>("profile");
But that will be another potential request to Parse, which is unfortunate. It seems that query.Include ("playerName") isn't properly working in the Unity version of Parse?
I believe you're supposed to use multi-level includes for this, like .Include("parent.playerName") in your first query.

' ', hexadecimal value 0x1F, is an invalid character. Line 1, position 1

I am trying to read a xml file from the web and parse it out using XDocument. It normally works fine but sometimes it gives me this error for day:
**' ', hexadecimal value 0x1F, is an invalid character. Line 1, position 1**
I have tried some solutions from Google but they aren't working for VS 2010 Express Windows Phone 7.
There is a solution which replace the 0x1F character to string.empty but my code return a stream which doesn't have replace method.
s = s.Replace(Convert.ToString((byte)0x1F), string.Empty);
Here is my code:
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
using (var reader = new StreamReader(e.Result))
{
int[] counter = { 1 };
string s = reader.ReadToEnd();
Stream str = e.Result;
// s = s.Replace(Convert.ToString((byte)0x1F), string.Empty);
// byte[] str = Convert.FromBase64String(s);
// Stream memStream = new MemoryStream(str);
str.Position = 0;
XDocument xdoc = XDocument.Load(str);
var data = from query in xdoc.Descendants("user")
select new mobion
{
index = counter[0]++,
avlink = (string)query.Element("user_info").Element("avlink"),
nickname = (string)query.Element("user_info").Element("nickname"),
track = (string)query.Element("track"),
artist = (string)query.Element("artist"),
};
listBox.ItemsSource = data;
}
}
XML file:
http://music.mobion.vn/api/v1/music/userstop?devid=
0x1f is a Windows control character. It is not valid XML. Your best bet is to replace it.
Instead of using reader.ReadToEnd() (which by the way - for a large file - can use up a lot of memory.. though you can definitely use it) why not try something like:
string input;
while ((input = sr.ReadLine()) != null)
{
string = string + input.Replace((char)(0x1F), ' ');
}
you can re-convert into a stream if you'd like, to then use as you please.
byte[] byteArray = Encoding.ASCII.GetBytes( input );
MemoryStream stream = new MemoryStream( byteArray );
Or else you could keep doing readToEnd() and then clean that string of illegal characters, and convert back to a stream.
Here's a good resource for cleaning illegal characters in your xml - chances are, youll have others as well...
https://seattlesoftware.wordpress.com/tag/hexadecimal-value-0x-is-an-invalid-character/
What could be happening is that the content is compressed in which case you need to decompress it.
With HttpHandler you can do this the following way:
var client = new HttpClient(new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip
| DecompressionMethods.Deflate
});
With the "old" WebClient you have to derive your own class to achieve the similar effect:
class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
Above taken from here
To use the two you would do something like this:
HttpClient
using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
{
using (var stream = client.GetStreamAsync(url))
{
using (var sr = new StreamReader(stream.Result))
{
using (var reader = XmlReader.Create(sr))
{
var feed = System.ServiceModel.Syndication.SyndicationFeed.Load(reader);
foreach (var item in feed.Items)
{
Console.WriteLine(item.Title.Text);
}
}
}
}
}
WebClient
using (var stream = new MyWebClient().OpenRead("http://myrss.url"))
{
using (var sr = new StreamReader(stream))
{
using (var reader = XmlReader.Create(sr))
{
var feed = System.ServiceModel.Syndication.SyndicationFeed.Load(reader);
foreach (var item in feed.Items)
{
Console.WriteLine(item.Title.Text);
}
}
}
}
This way you also recieve the benefit of not having to .ReadToEnd() since you are working with the stream instead.
Consider using System.Web.HttpUtility.HtmlDecode if you're decoding content read from the web.
If you are having issues replacing the character
For me there were some issues if you try to replace using the string instead of the char. I suggest trying some testing values using both to see what they turn up. Also how you reference it has some effect.
var a = x.IndexOf('\u001f'); // 513
var b = x.IndexOf(Convert.ToString((byte)0x1F)); // -1
x = x.Replace(Convert.ToChar((byte)0x1F), ' '); // Works
x = x.Replace(Convert.ToString((byte)0x1F), " "); // Fails
I blagged this
I had the same issue and found that the problem was a  embedded in the xml.
The solution was:
s = s.Replace("", " ")
I'd guess it's probably an encoding issue but without seeing the XML I can't say for sure.
In terms of your plan to simply replace the character but not being able to, because you have a stream rather than a text, simply read the stream into a string and then remove the characters you don't want.
Works for me.........
string.Replace(Chr(31), "")
I used XmlSerializer to parse XML and faced the same exception.
The problem is that the XML string contains HTML codes of invalid characters
This method removes all invalid HTML codes from string (based on this thread - https://forums.asp.net/t/1483793.aspx?Need+a+method+that+removes+illegal+XML+characters+from+a+String):
public static string RemoveInvalidXmlSubstrs(string xmlStr)
{
string pattern = "&#((\\d+)|(x\\S+));";
Regex regex = new Regex(pattern, RegexOptions.IgnoreCase);
if (regex.IsMatch(xmlStr))
{
xmlStr = regex.Replace(xmlStr, new MatchEvaluator(m =>
{
string s = m.Value;
string unicodeNumStr = s.Substring(2, s.Length - 3);
int unicodeNum = unicodeNumStr.StartsWith("x") ?
Convert.ToInt32(unicodeNumStr.Substring(1), 16)
: Convert.ToInt32(unicodeNumStr);
//according to https://www.w3.org/TR/xml/#charsets
if ((unicodeNum == 0x9 || unicodeNum == 0xA || unicodeNum == 0xD) ||
((unicodeNum >= 0x20) && (unicodeNum <= 0xD7FF)) ||
((unicodeNum >= 0xE000) && (unicodeNum <= 0xFFFD)) ||
((unicodeNum >= 0x10000) && (unicodeNum <= 0x10FFFF)))
{
return s;
}
else
{
return String.Empty;
}
})
);
}
return xmlStr;
}
Nobody can answer if you don't show relevant info - I mean the Xml content.
As a general advice I would put a breakpoint after ReadToEnd() call. Now you can do a couple of things:
Reveal Xml content to this forum.
Test it using VS Xml visualizer.
Copy-paste the string into a txt file and investigate it offline.

A better way of representing File Attachment into a list(c#3.0)

I have written
List<Attachment> lstAttachment = new List<Attachment>();
//Check if any error file is present in which case it needs to be send
if (new FileInfo(Path.Combine(errorFolder, errorFileName)).Exists)
{
Attachment unprocessedFile = new Attachment(Path.Combine(errorFolder, errorFileName));
lstAttachment.Add(unprocessedFile);
}
//Check if any processed file is present in which case it needs to be send
if (new FileInfo(Path.Combine(outputFolder, outputFileName)).Exists)
{
Attachment processedFile = new Attachment(Path.Combine(outputFolder, outputFileName));
lstAttachment.Add(processedFile);
}
Working fine and is giving the expected output.
Basically I am attaching the file to the list based on whether the file is present or not.
I am looking for any other elegant solution than the one I have written.
Reason: Want to learn differnt ways of representing the same program.
I am using C#3.0
Thanks.
Is it looks better?
...
var lstAttachment = new List<Attachment>();
string errorPath = Path.Combine(errorFolder, errorFileName);
string outputPath = Path.Combine(outputFolder, outputFileName);
AddAttachmentToCollection(lstAttachment, errorPath);
AddAttachmentToCollection(lstAttachment, outputPath);
...
public static void AddAttachmentToCollection(ICollection<Attachment> collection, string filePath)
{
if (File.Exists(filePath))
{
var attachment = new Attachment(filePath);
collection.Add(attachment);
}
}
How about a little LINQ?
var filenames = new List<string>()
{
Path.Combine(errorFolder, errorFilename),
Path.Combine(outputFolder, outputFilename)
};
var attachments = filenames.Where(f => File.Exists(f))
.Select(f => new Attachment(f));

IronRuby performance issue while using Variables

Here is code of very simple expression evaluator using IronRuby
public class BasicRubyExpressionEvaluator
{
ScriptEngine engine;
ScriptScope scope;
public Exception LastException
{
get; set;
}
private static readonly Dictionary<string, ScriptSource> parserCache = new Dictionary<string, ScriptSource>();
public BasicRubyExpressionEvaluator()
{
engine = Ruby.CreateEngine();
scope = engine.CreateScope();
}
public object Evaluate(string expression, DataRow context)
{
ScriptSource source;
parserCache.TryGetValue(expression, out source);
if (source == null)
{
source = engine.CreateScriptSourceFromString(expression, SourceCodeKind.SingleStatement);
parserCache.Add(expression, source);
}
var result = source.Execute(scope);
return result;
}
public void SetVariable(string variableName, object value)
{
scope.SetVariable(variableName, value);
}
}
and here is problem.
var evaluator = new BasicRubyExpressionEvaluator();
evaluator.SetVariable("a", 10);
evaluator.SetVariable("b", 1 );
evaluator.Evaluate("a+b+2", null);
vs
var evaluator = new BasicRubyExpressionEvaluator();
evaluator.Evaluate("10+1+2", null);
First Is 25 times slower than second. Any suggestions? String.Replace is not a solution for me.
I do not think the performance you are seeing is due to variable setting; the first execution of IronRuby in a program is always going to be slower than the second, regardless of what you're doing, since most of the compiler isn't loaded in until code is actually run (for startup performance reasons). Please try that example again, maybe running each version of your code in a loop, and you'll see the performance is roughly equivalent; the variable-version does have some overhead of method-dispatch to get the variables, but that should be negligible if you run it enough.
Also, in your hosting code, how come you are holding onto ScriptScopes in a dictionary? I would hold onto CompiledCode (result of engine.CreateScriptSourceFromString(...).Compile()) instead -- as that will help a lot more in repeat runs.
you can of course first build the string something like
evaluator.Evaluate(string.format("a={0}; b={1}; a + b + 2", 10, 1))
Or you can make it a method
if instead of your script you return a method then you should be able to use it like a regular C# Func object.
var script = #"
def self.addition(a, b)
a + b + 2
end
"
engine.ExecuteScript(script);
var = func = scope.GetVariable<Func<object,object,object>>("addition");
func(10,1)
This is probably not a working snippet but it shows the general idea.