In Orange 2 it was possible to perform MDS and project data onto multidimensional space rather than a plane when using the scripting library as documented in the docs.
How to do that in Orange 3? Is the feature still supported as I cannot find the MDS projection class in the data mining library, just in the visual programming part?
The following should work (set the number of desired components with n_components):
import Orange
data = Orange.data.Table("iris")
mds = Orange.projection.MDS(n_components=3)
projection = mds(data)
print(projection.embedding_[:5])
Orange3 borrows this functionality from sklearn.
Related
I've got a network with undirected, weighted edges. I'm interested in whether nodes A and B are in the same community. When I run "modularity" in Gephi, the modularity class for node A is usually, though not always, distinct from that of B. However, when I switch over to Python and run, on the exact same underlying data, either louvain_communities() (from the networkx.algorithms.community module) or community_louvain.best_partition() (from the community module), A is always in the same community as B. I've tried this at various resolutions, but keep getting similar results: the Python modules are grouping A and B significantly more often than Gephi.
My question is: What is the Gephi method doing differently? My understanding is that Gephi uses the Louvain method; the variables I can see (resolution, using weights, etc.) seem to be the same. Why the disparity?
Edit:
I was asked to provide some code for this.
Basically I have edge tuples with weights like so:
edge_tuples = [(A,B,5),(A,C,11),(B,C,41),…]
I have nodes as a list:
nodes = [A,B,C,…]
I use networkx to make a graph:
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_weighted_edges_from(edge_tuples)
If I’m using community, I get the partition like so:
partition = community.community_louvain.best_partition(G,resolution=.7)
The resolution could be whatever, but .7 is in line with what I've tried before.
In Gephi, I'm just using ordinary node and edge tables. These are generated and exported as csv's in the process of creating the edge_tuples described above (i.e. I make them both out of the same data and just export a csv before making the networkx graph), so I don't see where the underlying data would be differing, although I'm certainly open to correction.
I would like to implement Siril's Midtones Function Transfer algorithm in a GIMP plugin. Ideally, it would get 2 sliders (black level, midtones value) and a preview.
But I can't find a non-obsolete tutorial/example to implement a pixel operation with minimum GUI (sliders, preview...). Do you have an idea where to start ? Maybe a up-to-date example of plugin with pixel by pixel operation ? or with widget like sliders ?
# Siril's algortihm :
MTF(x;m) = ((m-1)x)/((2m-1)x-m)
# x : pixel value (float)
# m : midtones values float[0, 1]
https://free-astro.org/siril_doc-en/#Histograms
https://free-astro.org/index.php?title=Siril:Processing_tutorial
Siril is a great free software to process astrophotography, but it lacks some tools like layers, selection...
For a full-capability plugin written in C the best place is Gimp's own source. Some filters are still defined as plugins in Gimp. See here.
Otherwise you can do a quick proof-of-concept in Python, with an auto-generated dialog for the two sliders (but no preview). The python interface defines "pixel regions" that map a python list directly over the Gimp data.
You can even use Numpy.
For a Python POC:
Official Gimp-python doc is here.
Numpy link has an example script.
Large set of example scripts here.
An example of all the UI widgets here
Some hints about debugging your stuff here
Good afternoon,
I'm a newbie to H3. Before reading deeply the documentation and go further on tests with H3, I'm taking the liberty to ask you general questions regarding H3.In advance sorry if my questions seem naive or clumsy.
Which bindings are recommended for using H3? Is there one more suitable one for each fonctionality ? data integration ? display? Raster supported ? Sampling/quantification? : python? geopandas with jupyter notebook? postgis? R ? Bigquery ? js,etc.?
We wonder about the possibility with H3 to consider DGGS maritime trafficability shorter path analysis with some constraints. I past below a screen shot.
Does H3 allows the integration/fusion/combine of data? We would like do some test with multi-source/multi-date data fusion combination for the creation of a DTM (topographic or bathymetric)?
Is it possible to assign a weight to the THR data (importance flag in order to not decimate the Very Hihgt Resolution). So, Is it possible to manage and define metadata ?
Which type of data the tool is able to integrate ? (raster ? polygon? line? point ? point cloud?).
Does the tool offer different methods in terms of sampling and quantification? Is it possible for the user to decide at what level in the hierarchy of cells it is possible to assign the data?
Finally does H3 is compliant with OGC DGGS abstract standard. If no, do you know the existing gap ?
In advance, thank you very much for your useful replies.
Kind regards.
Best-effort answers to your questions:
A. Bindings: The bindings we're aware of are listed here. The bindings for Java, JavaScript, and Python are probably the best-maintained (though Python has been undergoing a major refactor and might be best used when this is finished).
B. Path Analysis: I haven't worked with this, but this tutorial suggests that all you need to implement this in a hex grid are neighbors and a distance function. Neighbors in H3 are available via kRing(origin, 1) and distance can be calculated via h3Distance(origin, target) (with some limitations at present - the two cells cannot be too far apart and the path cannot cross a pentagon).
C. Merging Data Sources: H3 is an excellent choice as a common unit for analysis that merges multiple data sources - you can convert multiple sources into H3 and then e.g. perform cell-based raster arithmetic to get a value for each hexagon. The H3 library itself only offers conversion functions, not data merging functions.
D. I don't fully understand this question, but it would be outside the purview of the H3 library.
E. Data Type Conversion: The library provides strong support for converting polygon data (via polyfill) and point data (via h3ToGeo). Raster data would probably need to be converted into a grid of points for conversion to cells. H3 uses a spherical plane that doesn't consider altitude, so it can't be used to convert a 3d point cloud without external logic about how to project the points onto the surface. Note that the H3 library itself has no logic to deal with file formats, etc.
F. Sampling/Quantification: The choice of resolution is user-specified, but otherwise the H3 library does not explicitly deal with sampling or quantification. Points are assigned to the cells in which they are found; when using polyfill, cells are assigned to polygons in which their centers are found. Further sampling choices are left to the user.
G. Adherence to DGGS Standard: See this paper for an assessment of H3 and an alternative DGGS in relation to the standard.
I am new to orange tool. My project is to implement FP-Growth algorithm in orange tool and generate graph using data set. But there is no widget for that.
Please any one can help me
The old Orange 2 had widgets for association rules built-in. In Orange 3, however, they are available in an add-on: Orange3-Associate.
The algorithms Orange3-Associate contains seem to already include FP-growth algorithm you seek.
Different teams use different libraries to train and run neural networks (caffe, torch, theano...). This makes sharing difficult: each library has its own format to store networks and you have to install a new library each time you want to test other teams' work.
I am looking for solutions to make this less tedious:
- Is there a preferred (shared?) format to store neural networks?
- Is there a service or library that can help handle different types of networks / or transform one type into another?
Thank you!
Is there a preferred (shared?) format to store neural networks?
Each library / framework has its own serialization, e.g. Caffe uses Protocol Buffers, Torch has a built-in serialization scheme and Theano objects can be serialized with pickle.
In some cases like OverFeat or darknet the weights and biases are stored on-disk in binary format via plain fwrite-s of the corresponding float(or double) contiguous arrays (see this answer for more details). Note that this does not cover the architecture of the network / model which has to be known or represented separately (like declared explicitly at load time).
Also: a library like libccv stores the structure and the weights in a SQLite database.
Is there a service or library that can help handle different types of networks / or transform one type into another?
I don't think there is a single (meta) library that claims to do so. But it exists distinct projects that provide convenient converters.
Some examples (non exhaustive):
Caffe -> Torch: https://github.com/szagoruyko/loadcaffe
Torch -> Caffe: https://github.com/facebook/fb-caffe-exts
Caffe -> TensorFlow: https://github.com/ethereon/caffe-tensorflow
--
UPDATE (2017-09): two noticeable initiatives are:
(1) the ONNX format (a.k.a. Open Neural Network Exchange):
[...] a standard for representing deep learning models that enables models to be transferred between frameworks
See these blog posts.
(2) the CoreML format introduced by Apple:
[...] a public file format (.mlmodel) for a broad set of ML methods [...] Models in this format can be directly integrated into apps through Xcode.
ONNX (Open Neural Network Exchange)
ONNX is an open-source AI ecosystem that provides a common format for neural networks.
It helps converting a deep learning model to another.
Generally, model conversion takes weeks/months without ONNX:
ONNX offers simpler and faster conversion process:
For all supported conversions, see here.
It makes deployment easier, models stored in a much preferable way:
In the above image, ONNX acts as a data ingestion layer, transforms each input model to the same format. Otherwise, all models would be like a bunch of puzzle pieces that do not fit each other.
How to Use ONNX - Keras Conversion Example
Let's say you have your Keras model and you want to transform it to ONNX:
model = load_model("keras_model.hdf5") # h5 is also OK!
onnx_model = keras2onnx.convert_keras(model, model.name)
onnx_model_file = 'output_model.onnx'
onnx.save_model(onnx_model, onnx_model_file)
Then load & run saved model :
onnx_model = onnx.load_model('output_model.onnx')
content = onnx_model.SerializeToString()
sess = onnxruntime.InferenceSession(content)
x = x if isinstance(x, list) else [x]
feed = dict([(input.name, x[n]) for n, input in enumerate(sess.get_inputs())])
# Do inference
pred_onnx = sess.run(None, feed)
This example uses keras2onnx to convert the Keras model and onnxruntime to do inference.
Note: There are also lots of pre-trained models in the ONNX format. Check this out!
References:
1. https://towardsdatascience.com/onnx-made-easy-957e60d16e94
2. https://blog.codecentric.de/en/2019/08/portability-deep-learning-frameworks-onnx/
3. http://on-demand.gputechconf.com/gtc/2018/presentation/s8818-onnx-interoperable-deep-learning-presented-by-facebook.pdf
4. https://devblogs.nvidia.com/tensorrt-3-faster-tensorflow-inference/