I'm having trouble importing magellan-1.0.4-s_2.11 in spark notebook. I've downloaded the jar from https://spark-packages.org/package/harsha2010/magellan and have tried placing SPARK_HOME/bin/spark-shell --packages harsha2010:magellan:1.0.4-s_2.11 in the Start of Customized Settings section of the spark-notebook file of the bin folder.
Here are my imports
import magellan.{Point, Polygon, PolyLine}
import magellan.coord.NAD83
import org.apache.spark.sql.magellan.MagellanContext
import org.apache.spark.sql.magellan.dsl.expressions._
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
And my errors...
<console>:71: error: object Point is not a member of package org.apache.spark.sql.magellan
import magellan.{Point, Polygon, PolyLine}
^
<console>:72: error: object coord is not a member of package org.apache.spark.sql.magellan
import magellan.coord.NAD83
^
<console>:73: error: object MagellanContext is not a member of package org.apache.spark.sql.magellan
import org.apache.spark.sql.magellan.MagellanContext
I then tried to import the new library like any other library by placing it into the main script like so:
$lib_dir/magellan-1.0.4-s_2.11.jar"
This didn't work and I'm left scratching my head wondering what I've done wrong. How do I import libraries such as magellan into spark notebook?
Try evaluating something like
:dp "harsha2010" % "magellan" % "1.0.4-s_2.11"
It will load the library into Spark, allowing it to be imported - assuming it can be obtained though the Maven repo. In my case it failed with a message:
failed to load 'harsha2010:magellan:jar:1.0.4-s_2.11 (runtime)' from ["Maven2 local (file:/home/dev/.m2/repository/, releases+snapshots) without authentication", "maven-central (http://repo1.maven.org/maven2/, releases+snapshots) without authentication", "spark-packages (http://dl.bintray.com/spark-packages/maven/, releases+snapshots) without authentication", "oss-sonatype (https://oss.sonatype.org/content/repositories/releases/, releases+snapshots) without authentication"] into /tmp/spark-notebook/aether/b2c7d8c5-1f56-4460-ad39-24c4e93a9786
I think file was to big and connection was interrupted before whole file could be downloaded.
Workaround
So I downloaded the JAR manually from:
http://dl.bintray.com/spark-packages/maven/harsha2010/magellan/1.0.4-s_2.11/
and copied it into the:
/tmp/spark-notebook/aether/b2c7d8c5-1f56-4460-ad39-24c4e93a9786/harsha2010/magellan/1.0.4-s_2.11
And then :dp command worked. Try Calling it first, and if it will fail copy JAR into the right path to make things work.
Better solution
I should investigate why download failed to fix it in the first place... or put that library in my local M2 repo. But that should get you going.
I would suggest to check this:
https://github.com/spark-notebook/spark-notebook/blob/master/docs/metadata.md#import-download-dependencies
and
https://github.com/spark-notebook/spark-notebook/blob/master/docs/metadata.md#add-spark-packages
I think the :dp magic command is depreciated, instead you should add your custom dependencies in the notebook metadata. You can go in the menu Edit > Edit notebook metadata, there add something like:
"customDeps": [
"harsha2010 % magellan % 1.0.4-s_2.11"
]
Once done, you will need to restart the kernel, you can check in the browser console if the package is being downloaded properly.
The easy way, you should set or add the EXTRA_CLASSPATH environnent variable to point to your .jar file downloaded :
export EXTRA_CLASSPATH = </link/to/your.jar> or set EXTRA_CLASSPATH= </link/to/your.jar> in wondows OS. Here find the detailed solution.
Related
I'm trying to parse json in an action script for a form component.
I tried to:
import javax.json.JsonObject;
And get the following on a test compile:
Failed to compile script
1. ERROR in /private/var/folders/3t/l3dvn7tx1j76wsx17xfjcpww0000gn/T/script10053200329813627000.java.dir/code/Completion.java (at line 1)
import javax.json.JsonObject;
The import javax.json cannot be resolved
How do I achieve the equivalent of this import so I can parse json? I started going down the rabbit hole of create a JMOD for javax.json but that's starting to seem like the wrong path.
The package javax.json is not part of the JRE, but a JEE 7 API. You can download the JAR file from
https://mvnrepository.com/artifact/javax.json/javax.json-api/1.1.4
and add it on the "Installer->Screens & Actions->Custom code" step. Then the class will be available in all scripts.
I use PyCharm and Eclipse with PyDev.
To be specific, I am using Odoo and setting up project.
https://github.com/odoo/odoo
Here is the folder structure.
odoo-12
|-addons
| '-web
| '-...
|-odoo
'-addons
'-...
In source code for example:
addons/purchase/controllers/portal.py
# Unresolved yet this is the official source code
from odoo.addons.web.controllers.main import Binary
# Resolved perfectly
from addons.web.controllers.main import Binary
I understand the reason why this one works
from addons.web.controllers.main import Binary
but how can I make this works instead?
from odoo.addons.web.controllers.main import Binary
I cannot and should not modify any Odoo source code to make IDE resolving path correctly
In Eclipse/PyDev you should be able to set your source folder to the folder containing odoo and it should work...
I think there is a bug with respect to how PyDev (version 4.6) recognizes intra-package imports when selecting Grammar 3.x for the project preferences. I have a project like this:
foobar
mypack
__init__.py
mod1.py
mod2.py
mod2.py simply says
from mod1 import fun1
mod1.py simply says
def fun1():
print("Hey we are in fun1 in mod1")
If the project Python project preferences are set to use Grammar 3.0-3.5, with a Python 3.4 interpreter, and I open up mod2.py the line from mod1 import fun1 is highlighted with an error Unresolved import: fun1. If I change the Python project preferences to use Grammar 2.7, close the file mod2.py and reopen it, the error disappears. Just by changing the grammar back and forth, and closing/reopening the file, I can make the error appear/disappear.
So it seems that setting the Grammar to 3.x in PyDev causes intra-package imports to be incorrectly flagged as having an import error.
Any suggestions?
PyDev is doing right... on Python 3, relative imports must be written as:
from .mod1 import fun1
If the import doesn't start with a dot, it'll consider it an absolute import (and will properly show the error for you as with that absolute path the imported file cannot be resolved).
So my real issue was getting PyDev to not report errors about the imports, and to be able to debug modules buried in packages that have a main() in there for debugging. The solution for me was to use relative imports (as said in Fabio's answer), and then to do the following for debugging purposes. Let's say I want to run a module pack1.subpack2.subpack3.subpack4.modtodebug using PyDev, that has relative imports, and has a main() function. At the top level of my project, I have a module debugmain.py that reads
from pack1.subpack2.subpack3.subpack4.modtodebug import main as debugmain
if __name__ == '__main__':
debugmain()
I then have one run configuration for debugmain.py and each time I want to debug a different module, I just have to edit the code in debugmain.py to point to that module.
I hope this helps someone else with this kind of problem.
In scala it is common practice to stack package statements to allow shorter imports, but when I load a file using stacked packages into the scala ide and I attempt to use an import starting with the same organization I get a compiler error from what appears to be the presentation compiler. The code compiles fine in sbt outside of the IDE.
An example code snippet is as follows:
package com.coltfred
package util
package time
import com.github.nscala_time.time.Imports._
On the import I get the error object github is not a member of package com.coltfred.util.com.
If I move the import to a single line the error will go away, but we've used this practice frequently in our code base so changing them all to be single line package statements would be a pain.
Why is this happening and is there anything I can do to fix it?
Edit:
I used the eclipse-sbt plugin to generate the eclipse project file for this. The directory structure is what it should be and all of the dependencies are in the classpath.
Edit 2:
It turns out there was a file in the test tree of the util package (which should have been in the same package), but had a duplicate package statement at the top. I didn't check the test tree because it shouldn't affect the compilation of the main tree, but apparently I was wrong.
Not sure why the Scala IDE is not liking this, but you can force the import to start at the top level using _root_:
import _root_.com.github.nscala_time.time.Imports._
See if that avoids irritating the IDE.
This is a common annoyance that annoyed paulp into an attempt to fix it. His idea was that a dir that doesn't contribute class files shouldn't be taken as a package. If you can take util as scala.util, you should do so in preference to foo.util where that util is empty.
The util dir is the usual suspect, because who doesn't have a util dir lying around, and in particular, ./util?
apm#mara:~/tmp/coltfred$ mkdir -p com/coltfred/util/time
apm#mara:~/tmp/coltfred$ mkdir -p com/coltfred/util/com
apm#mara:~/tmp/coltfred$ vi com/coltfred/util/time/test.scala
apm#mara:~/tmp/coltfred$ scalac com/coltfred/util/time/test.scala
./com/coltfred/util/time/test.scala:5: error: object github is not a member of package com.coltfred.util.com
import com.github.nscala_time.time._
^
one error found
apm#mara:~/tmp/coltfred$ cat com/coltfred/util/time/test.scala
package com.coltfred
package util
package time
import com.github.nscala_time.time._
class Test
apm#mara:~/tmp/coltfred$
To debug, find out where the offending package is getting loaded from.
I am working on a project in Eclipse Juno. I wrote a class called Character in a package named chargen.py. There's a red X next to from chargen import Character:
Unresolved import: Character
Character Found at: Avarice_v0.PlayAvarice_v0
from chargen import Character
Yet the import works. The entire code at the moment is simply this:
from chargen import Character
def main():
PLAYER = Character("")
print(PLAYER)
if __name__ == '__main__':
main()
This code results in the printing of the __str__ proving the Character("") ran. Also, it generates no errors when running. Why does Eclipse label this Unresolved import: Character?
I figured out how to get rid of the error. I moved everything one directory up. Settings for the project showed the PYTHONPATH included the main directory, but not the nested one. By moving it all up and deleting the now empty original folder, I have no unresolved import errors. This helped me to understand more about the PYTHONPATH choices offered at the initial setup of the project in PyDev.
In the properties for your project, there's a pane called "PyDev - PYTHONPATH", with a sub-pane called "External Libraries". You can add source folders (any folder that has an init.py) to the path using that pane. Your project code will then be able to import modules from those source folders.