I am trying to compute the SVD of a large matrix (4799 x 53130) using Pycharm as my IDE. The following returns the exit code -1073741819 (0xC0000005), but no error message is displayed. I have looked up this error code on here and people have found a mixture of things causing it. I have re-installed Pycharm and all dependencies, yet this issue persists. I have tried also using the numpy version, which returns the same error. I'm confident this problem is related to the lapack package, since both use it. Any direction would be greatly appreciated.
import numpy as np
import pandas as pd
from scipy.linalg import svd
df = pd.read_pickle(r'filepath\data.pkl')
df = df.to_numpy()
svd(df)
I tried using df=np.ones((4799,53130)) too in case it was my data that was the problem, and this yields the same issue. It isn't the amount of RAM, my machine has 120GB and the process doesn't go near the limit.
See this thread for a discussion of the problem https://github.com/scipy/scipy/issues/14001
There is no solution for this yet.
Related
I'm trying to use the DeBerta model and I'm first trying to implement some of the code I found here (an example of how to use the model), just to make sure I have the right dependencies and know what imports I need. I'm working in Scala 2.4.0, Zeppelin 0.8.1. I've added "com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.4.4" as a dependency, which I believe should give me spark-nlp 3.4.4. So far most of the imports I got from the sample code are working:
import com.johnsnowlabs.nlp.embeddings
import com.johnsnowlabs.nlp.annotator._
import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.training.CoNLL
import com.johnsnowlabs.nlp.util.io.ResourceHelper
import com.johnsnowlabs.util.Benchmark
import org.apache.spark.ml.Pipeline
import org.apache.spark.sql.functions.{col, explode, size}
Although two of the imports get errors
but I'm not sure I need these to load the model. But when I try to load
val embeddings = DeBertaEmbeddings.pretrained("deberta_v3_base", "en")
.setInputCols("sentence", "token")
.setOutputCol("embeddings")
.setMaxSentenceLength(512)
I get an error saying
java.lang.IllegalArgumentException: requirement failed: Can not find deberta_v3_base inside public/models to download. Please make sure the name and location are correct!
at scala.Predef$.require(Predef.scala:224)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadResource(ResourceDownloader.scala:441)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:499)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:492)
at com.johnsnowlabs.nlp.HasPretrained$class.pretrained(HasPretrained.scala:44)
at com.johnsnowlabs.nlp.annotator.package$DeBertaEmbeddings$.com$johnsnowlabs$nlp$embeddings$ReadablePretrainedDeBertaModel$$super$pretrained(annotator.scala:532)
at com.johnsnowlabs.nlp.embeddings.ReadablePretrainedDeBertaModel$class.pretrained(DeBertaEmbeddings.scala:329)
at com.johnsnowlabs.nlp.annotator.package$DeBertaEmbeddings$.pretrained(annotator.scala:532)
at com.johnsnowlabs.nlp.annotator.package$DeBertaEmbeddings$.pretrained(annotator.scala:532)
at com.johnsnowlabs.nlp.HasPretrained$class.pretrained(HasPretrained.scala:47)
at com.johnsnowlabs.nlp.annotator.package$DeBertaEmbeddings$.com$johnsnowlabs$nlp$embeddings$ReadablePretrainedDeBertaModel$$super$pretrained(annotator.scala:532)
at com.johnsnowlabs.nlp.embeddings.ReadablePretrainedDeBertaModel$class.pretrained(DeBertaEmbeddings.scala:326)
at com.johnsnowlabs.nlp.annotator.package$DeBertaEmbeddings$.pretrained(annotator.scala:532)
... 51 elided
I've tried to go through the documentation on the DeBerta model, and from what I've seen I have adequate versions of scala and spark-nlp according to John Snow Labs models hub. I saw on this stackoverflow thread that I might need to put "_noncontrib" after my model name. I tried that even though it says I wouldn't need it after spark-nlp 2.4.0+. So now I'm pretty sure "deberta_v3_base" is the right name, but I'm not sure what the error means by the location being incorrect. Do I need to specify a location for the pretrained model per the config helper? Does anyone know what's going on here? I apologize if I've left anything out, I'm new to Zeppelin and spark-nlp. Thanks!
I want to run evolutionary algorithms like GA,PSO using pyspark on spark.How to do this using MLLib using Deap python library.Is there any other library available to perform same task.
Have a look at my answer on how to use DEAP with Spark and see if it works for you.
Here is an example of how to configure the DEAP toolbox to replace the map function by a custom one using Spark.
from pyspark import SparkContext
sc = SparkContext(appName="DEAP")
def sparkMap(algorithm, population):
return sc.parallelize(population).map(algorithm)
toolbox.register("map", sparkMap)
In https://github.com/DEAP/deap/issues/268 they show how to do this in the DEAP package. However this is an issue. but they mention there is a pull request (https://github.com/DEAP/deap/pull/76), and is seems the fixed code/branch is from a forked repo.
It sounds like if you rebuild the package with that code it should resolve the issue.
Another resource I found, haven't tried it, is https://apacheignite.readme.io/docs/genetic-algorithms.
Also came across this https://github.com/paduraru2009/genetic-algorithm-with-Spark-for-test-generation
I'm new to flash development, so I'm watching a tutorial on how to use FlashDevelop. The video recommended I use Box2D and explained how to use it as a global classpath, which I have done.
I was messing around with the code using what the person in the video was showing, just trying to get an output. As I typed, FlashDevelop was adding in the import statements for me.
import Box2D.Collision.Shapes.b2CircleShape;
import Box2D.Common.Math.b2Vec2;
import Box2D.Dynamics.b2BodyDef;
import Box2D.Dynamics.b2FixtureDef;
import Box2D.Dynamics.b2World;
import Box2D.Dynamics.b2Body;
When I run the program though, it's returning this:
col: 31 Error: Definition Box2D.Collision.Shapes:b2CircleShape could not be found.
It's returning a variation of that for each import.
I've checked and the files are indeed there. I'm really not certain what this could be; it's possible I just missed a step.
Any ideas?
(Sorry if I formatted this question incorrectly, I'm new to this site.)
It's maybe cause you are using an old version
I think these are your choices :
1) you have to do an update
or
2) use "b2CircleDef"
See the code source in this link the change are commented
http://www.emanueleferonato.com/2010/01/27/box2dflash-2-1a-released-what-changed/
Hope that was helpful !
Good luck
I'm in the midst of trying to create my own radon transform function, for which I need to rotate the simple image I've created. According to the documentation, the function is in scipy.misc.
However,
from scipy.misc import imrotate
gives me "could not import name imrotate"
and
import scipy.misc
scipy.misc.imrotate(myImage,theta)
says that scipy.misc does not have a module named 'imrotate' when I try to call the function.
I've tried removing the '--pylab inline' arguments from my launch, and I've made sure the PIL/Image libraries aren't imported, because I've heard that there were problems with that in other threads, but nothing seems to make it work.
I've gotten around it for now by using a different suite, the scikit-image library, but I'd prefer to use the scipy version if I can, because it's more commonly used.
scipy.misc.imrotate requires the library PIL and I guess that is your problem. Ether it is not installed at all or not installed correctly.
You can use scipy.ndimage.interpolation.rotate instead.
I am a student who is envolving in a research about robust visual tracking.
And these days ,I had met a problem in my study.
The teacher gave me a project of matlab code about the research, when I try to run this code, and the program error is as follows:
??? Attempt to execute SCRIPT mexLasso as a function:
F:\L1_Tracking_standard_car\mexLasso.m
Error in ==> L1Tracking_release at 95
c = mexLasso(Y(:,i), [A fixT], param);
Error in ==> demo at 46
tracking_res = L1Tracking_release( s_frames, sz_T, n_sample, init_pos,
res_path, fcdatapts);
When I go to the program tracking, I found that mexLasso function does not exist, Only get an empty mexLasso.m file and a mexLasso.mexw32 files.
My OS version is Windows 7 64bit,and the matlab is matlab 7.12.0 r2011a
Does anybody here knows the causes of my problem?
Anymore, I wonder if anybody knows who has the source code of the binary file mexLasso.mexw32.Because I thought that if I can get the source code of the file mexLasso.mexw32,then I could compile its 64 bit version myself.(I doubt that my os could not recognize the .mexw32 file.)
I hope my express clear enough to let you come to help me ,thanks a lot!
I think your analysis is basically correct - mexLasso is intended to be a MEX function, but MATLAB is finding only mexLasso.m which presumably contains help text. Unfortunately, the error you're getting isn't terribly helpful. As I see it, you have two options:
Obtain the source code for mexLasso and recompile on WIN64
Run the WIN32 version of MATLAB on your WIN64 machine
the function mexLasso comes from the SPAMS toolbox
http://www.di.ens.fr/willow/SPAMS/.
You can find the sources there and compile the mex file corresponding to your OS.
First, you need to find mexLasso.cpp file in http://spams-devel.gforge.inria.fr/downloads.html as Marial has already mentioned.
Then you can find and download the recent version of SPAMS.
A following stage is just to execute compile.m file on your MATLAB.
Finally you can find mexLasso.mexw64 in the build folder.
Good luck.