A set of macros, jars and properties to be set in every pig latin script file. Have moved all these to common.pig.
Objective is to include this common pig file in all the pig latin files.
Tried the below approaches as suggested in Invoke Pig Latin script from other Pig script
Approach 1 : Using RUN, is working as expected.
Usage : RUN common.pig;
Approach 2 : Using IMPORT is resulting in an error.
Usage : IMPORT 'common.pig';
Error message :
ERROR 1200: <file common.pig, line 1, column 0> Syntax error, unexpected symbol at or near 'SET'
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. <file common_macro.pig, line 1, column 0> Syntax error, unexpected symbol at or near 'SET'
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1608)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1547)
I am trying to understand why the second approach is failing. My understanding of IMPORT is that it writes the imported file directly in to pig script in place of IMPORT statement, if this is the case, ideally it should not have thrown the above error.
Any inputs/ thoughts on this.
Also from code modularity/ maintainability/ execution perspective is it better to go for EXEC or RUN command to call common.pig file in the above use case.
Files :
common.pig
SET job.priority HIGH;
-- SET few others ...;
REGISTER snappy-java-1.0.4.1.jar;
-- REGISTER piggybank, avro and other required jars
test_import.pig
IMPORT 'common.pig';
A = load 'test/part*' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
DUMP A;
SET is not supported in an import file.
Read the IMPORT Macro show that Grunt Shell command are not supported - see below
Usage Use the IMPORT command to import a macro defined in a separate file into your Pig script.
IMPORT adds the macro definitions to the Pig Latin namespace; these
macros can then be invoked as if they were defined in the same file.
Macros can only contain Pig Latin statements; Grunt shell commands are
not supported.
Related
I have a Julia module in which I would like to import the Python function sympy.physics.wigner.wigner_9j. My minimal example module is as follows:
module my_module
using PyCall
using SymPy
export test
test()=sympy.physics.wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
end
Then in my Julia notebook running:
using my_module
test()
gives
KeyError: key :physics not found
However adding to the notebook
#pyimport sympy.physics.wigner as sympy_wigner
sympy_wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
gives the correct output. For some reason using #pyimport inside modules gives errors, which I typically avoid by using an __init__ inside my module, e.g adding to my_module.jl
const camb=PyNULL()
function __init__() # this should probably go in SFBBispectrum.jl
copy!(camb, pyimport_conda("camb", "camb", "conda-forge"))
pars=camb.CAMBparams()
end
which allows me to access camb.CAMBparams. Unfortunately I am failing to do something similar for sympy.physics.wigner.wigner_9j.
It has been awhile, but does this help https://github.com/JuliaPy/SymPy.jl/blob/master/src/physics.jl
Let's define a simple test class
classdef test_file < matlab.unittest.TestCase
methods(Test)
function test_function(testCase)
import some_package.some_function
testCase.verifyEqual(true,some_function(0));
end
end
end
It is irrelevant what some_function does. Function some_package.some_function does not exist in my path (for example I forgot to add it when pushing a commit). Whenever I try to run the test file above the test test_function is skipped with warning:
Warning: "test_file.m" was excluded.
Caused by:
Error: File: test_file.m Line: 4 Column: 20
The import statement 'import sc_force_models.apparent_accel' cannot be found or cannot be imported. Imported names must
end with '.*' or be fully qualified.
Since the test is skipped the problem is undetected and test runner returns 0 errors. In case someone forgets to commit a file I'd still like to detect this problem during testing, so expected behavior is to fail the test case.
As a workaround I've tried using the 'Strict',True argument to testrunner but it neither detects the issue. I've also tried putting the import statement between try, catch statements but it seems any code in the file is not executed.
Any ideas how to detect incorrect import statements in test cases?
import numpy as np
from statistics import mean
x=[1,2,3,4,5]
y=[6,7,8,9,10]
m=((mean(x)*mean(y)-mean(x*y))/(mean(x)**2)-(mean(x**2)))
print(m)
In the above the(or any other code where I run numpy), Firstly I am getting an input request when running the program. Something like this:
PS D:\Codes\Python> python practice.py
Enter no.: 1
Enter: 1
which should not happen as values are initialized. I saw in other forums regarding how the file should not be named after a Python module(which you can see, it isn't). Even after that I'm getting error:
"C:\Users\KIIT\AppData\Local\Programs\Python\Python36\lib\logging\__init__.py", line 28, in <module>
from string import Template
ImportError: cannot import name 'Template'
Can someone please tell me what to do about it?
Edit:
This problem is only powershell centric. The problem is faced when I run program through powershell. It works fine in IDLE.
There must be a file named "string.py" in your D:\Codes\Python.
You can rename it to solve this issue.
You should not name the file to the modules in Python Standard Library.
I am getting the java.lang.ClassNotFoundException: error inside Postgres when running a function that calls a JAR file I have loaded. I have installed and configured PL/JAVA (including the delivered examples) in my database and can run the examples to success. I am not attempting to load/install my first JAR, but I am doing something wrong.
My host controls the OS version: CentOS 6.8. Postgres is version 8.4.
I am attempting to install my own very simple java class, which is a derivative of the delivered example Parameters.addOne class. All my code is in /tmp. Here are the steps I've followed:
Doug.java:
package com.msmetric;
import java.math.BigDecimal;
import java.sql.Date;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Time;
import java.sql.Timestamp;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.TimeZone;
import java.util.logging.Logger;
public class Doug {
public static int addOne(int value) {
return value + 1;
}
}
Compile Doug.java using 'javac Doug.java' succeeds.
Create JAR file with Doug.class file in it using 'jar -cvf Doug.jar Doug.class. This works fine.
Now I load the JAR file into Postgres (public schema), change the classpath, create the function that calls the JAR, then attempt to run at psql prompt.
Run sqlj.install_jar from psql:
select sqlj.install_jar('file:/tmp/Doug.jar','Doug',false);
Set the classpath inside Postgres (from psql prompt postgres=#):
select sqlj.set_classpath('public','Doug');
Create the function that calls the JAR. This create function code is taken directly from the examples.ddr file that came with PL/JAVA. I simply changed org.postgres to com.msmetric.
create or replace function addone(int) returns int as 'com.msmetric.Doug.addOne(java.lang.Integer)' language java;
Now with the JAR loaded and function created, I attempt to run it. This function should simply add 1 to the number provided.
select addone(3);
Results:
ERROR: java.lang.ClassNotFoundException: com.msmetric.Doug
Thoughts?
I'm very sorry I didn't see your question sooner. Underneath all the exotic details (PostgreSQL, PL/Java, schemas, classpaths...), there's just a bit of basic Java going on here: if a jar file contains a class Doug.class in package com.msmetric, its path within the jar has to reflect that: it has to be com/msmetric/Doug.class. Otherwise, it won't be found.
You can set up that whole structure step by step:
javac Doug.java
mkdir com
mkdir com/msmetric
mv Doug.class com/msmetric/
jar -cvf Doug.jar com/msmetric/Doug.class
Or, you can let javac do more of the work for you:
mkdir classes
javac -d classes Doug.java
jar -cvf Doug.jar -C classes .
When you give javac a -ddirectory option, instead of just writing class files next to their .java sources, it will put them all in their proper places under the directory you named, and then you can just tell jar to change into that directory and slurp them all up (don't overlook the . at the end of that jar command).
Once you fix that, if you retry your original steps, you'll see that you now get a different error:
ERROR: Unable to find static method com.msmetric.Doug.addOne with signature (Ljava/lang/Integer;)I
That happens because you declared the function in Doug.java with int addOne(int value) (that is, taking a primitive int argument), but you declared it in SQL with returns int as 'com.msmetric.Doug.addOne(java.lang.Integer)' taking an Integer object.
Once you correct that:
create or replace function addone(int) returns int as 'com.msmetric.Doug.addOne(int)' language java;
you'll be able to see:
# select addone(3);
addone
--------
4
(1 row)
If you happen to see this belated answer, may I ask what version of PL/Java you are using? That's one detail you didn't mention. If it is older than 1.5.0, there are newer features that can help you out. For one, you can just annotate that function:
#Function
public static int addOne(int value) {
return value + 1;
}
and have javac spit out not only the Doug.class file but also a pljava.ddr file with your SQL function declaration already written correctly (no mixing up argument types!). There is a way to include that .ddr file into the jar you create so that you can just call sqlj.install_jar with the last parameter true so it runs the commands in the .ddr and your functions are ready to use. There's a Hello, world example in the docs that shows more of how it's done.
Cheers,
-Chap
While trying to use the sphinx matlab domain I can't get the MWE to work, provided on the extensions pypi site
There is always this Can't import module error. I'd guess, that the extension kind of generates pseudo modules from the m-code, but up to know I actually could not figure out, how this mechanism works.
The dir structure looks like this
root
|--test_data
| |--MyHandleClass.m
|
|--doc
|--------conf.py
|--------Makefile
|--------index.rst
The files MyHandleClass.m and index.rst contain the example code given on the package site and the conf.py starts like this
import sys, os
sys.path.append(os.path.abspath('.'))
sys.path.append(os.path.abspath('./test_data'))
# -- General configuration -----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
"sphinxcontrib.matlab",
"sphinx.ext.autosummary",
"sphinx.ext.autodoc"]
autodoc_default_flags = ['members','show-inheritance','undoc-members']
autoclass_content = 'both'
mathjax_path = 'http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default'
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
Error msg
WARNING: autodoc: failed to import module u'test_data'; the following exception was raised:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\sphinx\ext\autodoc.py", line 335, in import_object
__import__(self.modname)
ImportError: No module named test_data
E:\ME\doc\index.rst:13: WARNING: don't know which module to import for autodocumenting u'MyHandleClass' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)
After varying this and that maybe somebody out there has a clue?
Thanks for trying the matlabdomain sphinxcontrib extension. In order to use Sphinx to document MATLAB m-files, you need to add matlab_src_dir in conf.py as described in the Configuration section of the documenation. This is because the Python interpreter can't import a MATLAB m-file. Therefore you should not add your MATLAB root to the Python sys.path, or you will get the error you received. Instead set matlab_src_dir to the path containing the folder of your MATLAB project which you want to document.
Given your file structure, in order to document test_data use a conf.py with the following:
import os
# NOTE: don't add MATLAB m-files to `sys.path`
#sys.path.insert(0, os.path.abspath('.'))
# instead add them to `matlab_src_dir
matlab_src_dir = os.path.abspath('..') # MATLAB
Hope that does it! Please feel free to ask any more questions. I'm happy to help!