Get more of the metadata from the Neurotransmitter study using Allen SDK - allen-sdk

I am downloading all the images from the Neurotransmitter study of the Allen Brain Atlas using this script:
from allensdk.api.queries.image_download_api import ImageDownloadApi
from allensdk.config.manifest import Manifest
import pandas as pd
import os
#getting transmitter study
#product id from http://api.brain-map.org/api/v2/data/query.json?criteria=model::Product
nt_datasets = image_api.get_section_data_sets_by_product([27])
#an instance of Image Api for downloading
image_api = ImageDownloadApi()
for index, row in df_nt.iterrows():
#get section dataset id
section_dataset_id= row['id']
#each section dataset id has multiple image sections
section_images = pd.DataFrame(
image_api.section_image_query(
section_data_set_id=section_dataset_id)
)
for section_image_id in section_images['id'].tolist():
file_path = os.path.join('/path/to/save/dir/',
str(section_image_id) + '.jpg' )
Manifest.safe_make_parent_dirs(file_path)
image_api.download_section_image(section_image_id,
file_path=file_path,
downsample=downsample)
This script downloads presumably all the available ISH experiments. However, I am wondering what would be the best way to get more of the metadata as follows:
1) type of ISH experiment, known as "gene" (for example whether an image is MBP-stained, Nissl-stained or etc). Shown in red circle below.
2) Structure and correspondence to the atlas image (annotations, for example to see to which part of brain a section belongs to). I think this could be acquired with tree_search but not sure how. Shown in red circles below from two different webpages on Allen website.
3) The scale of the image, for example how big one pixel is in the downloaded image (e.g., 0.001x0.001 mm). I would require this for image analysis with respect to MRI, for example. Shown below in the red circle.
All the above information are somehow available on the website, my question is whether you could help me to do this programmatically via the SDK.
EDIT:
Also would be great to download "Nissl" stains programmatically, as they do not show using the above loop iteration. The picture is shown below.

To access this information, you'll need to formulate a somewhat complex API query.
from allensdk.api.queries.rma_api import RmaApi
api = RmaApi()
data_set_id = 146586401
data = api.model_query('SectionDataSet',
criteria='[id$eq%d]' % data_set_id,
include='section_images,treatments,probes(gene),specimen(structure)')
print("gene symbol: %s" % data[0]['probes'][0]['gene']['acronym'])
print("treatment name: %s" % data[0]['treatments'][0]['name'])
print("specimen location: %s" % data[0]['specimen']['structure']['name'])
print("section xy resolution: %f um" % data[0]['section_images'][0]['resolution'])
gene symbol: MBP
treatment name: ISH
specimen location: Cingulate Cortex
section xy resolution: 1.008000 um
Without doing a deep dive on the API data model, SectionDataSets have constituent SectionImages, Treatments, Probes, and source Specimens. Probes target Genes, and Specimens can be associated with a Structure. The query is downloading all of that information for a single SectionDataSet into a nested dictionary.
I don't remember how to find the specimen block extent. I'll update the answer if I find it.

Related

Save and re-load a weighted graph from OSMnx for NetworKX

I am using OSMnx to get a graph and add a new edge attribute (w3) representing a custom weight for each edge. Then I can successfully find 2 different shortest paths between 2 points using NetworkX and 'length', 'w2'. Everything works fine, this is my code:
G = ox.graph_from_place(PLACE, network_type='all_private', retain_all = True, simplify=True,truncate_by_edge=False) ```
w3_dict = dict((zip(zip(lu, lv, lk),lw3)))
nx.set_edge_attributes(G, w3_dict, "w3")
route_1 = nx.shortest_path(G, node_start, node_stop, weight = 'length')
route_2 = nx.shortest_path(G, node_start, node_stop, weight = 'w3')
Now I would like to save G to disk and reopen it, to perform more navigation tasks later on. But after saving it with:
ox.save_graph_xml(G, filepath='DATA/network.osm')
and reopen it with:
G = ox.graph_from_xml('DATA/network.osm')
my custom attribute w3 has disappeared. I have followed the instructions in the docs but with no luck. It feels like I'm missing something really obvious but I don't understand what it is..
Use the ox.save_graphml and ox.load_graphml functions to save/load full-featured OSMnx/NetworkX graphs to/from disk for later use. The save xml function exists only to allow serialization to the .osm file format for applications that require it, and has many constraints to conform to that.
import networkx as nx
import osmnx as ox
ox.config(use_cache=True, log_console=True)
# get a graph, set 'w3' edge attribute
G = ox.graph_from_place('Piedmont, CA, USA', network_type='drive')
nx.set_edge_attributes(G, 100, 'w3')
# save graph to disk
ox.save_graphml(G, './data/graph.graphml')
# load graph from disk and confirm 'w3' edge attribute is there
G2 = ox.load_graphml('./data/graph.graphml')
nx.get_edge_attributes(G2, 'w3')

Plot a graph with ipycytoscape (and networkx)

Following the instructions of ipycitoscape I am not able to plot a graph using ipycitoscape.
according to: https://github.com/QuantStack/ipycytoscape/blob/master/examples/Test%20NetworkX%20methods.ipynb
this should work:
import networkx as nx
import ipycytoscape
G2 = nx.Graph()
G2.add_nodes_from([*'ABCDEF'])
G2.add_edges_from([('A','B'),('B','C'),('C','D'),('E','F')])
print(G2.nodes)
print(G2.edges)
cytoscapeobj = ipycytoscape.CytoscapeWidget()
cytoscapeobj.graph.add_graph_from_networkx(nx_graph)
G2 is a networkx graph example and it looks ok since print(G2) gives the networkx object back and G2.nodes and G2.edges can be printed.
The error:
ValueError: invalid literal for int() with base 10: 'A'
Why should a node be an integer?
More general what to do if the starting data point if a pandas dataframe with a million rows edges those being strings like ProcessA-ProcessB, processC-processD etc
Also having a look to the examples it is to be noted that the list of nodes is composed of a dictionary data for every node. that data including an "id" per node and also "Atribute". The surprise here is that the networkx Graph should have all those properties.
thanks
This problem was fixed. See attachment.
Please let me know if it's still happening. Feel free to open an issue: https://github.com/QuantStack/ipycytoscape/
I'm just playing around with ipycytoscape myself, so I could be way off-base, but, shouldn't the line be:
cytoscapeobj.graph.add_graph_from_networkx(G2) # your graph name goes here
Trying to generate a cytoscape object built on a graph that doesn't exist might trigger a ValueError because it can't find any nodes.

How to read info on voltage/beam energy, imaging mode, acquisition date/timestamp, etc. from image meta-data? (Tags)

DM scripting beginner here, almost no programming skills.
I would like to know the commands to access all the metadata of DM images/spectra.
I realized that all my STEM images at 80 kV taken between 2 dates (let's say 02.11.2017-05.04.2019) have the scale calibration wrong by the same factor (scale of all such images needs to be multiplied by 1.21).
I would like to write a script which multiplies the scale value by a factor only for images in scanning mode at 80 kV taken during a period for all images in a folder with subfolders or for all images opened in DM and save the new scale value.
I checked this website http://digitalmicrograph-scripting.tavernmaker.de/other%20resources/Old-DMHelp/AllFunctions.html but only found how to call the scale value (ImageGetDimensionCalibration). I have a general idea how to write the script based on other scripts if I find out how to call the metadata.
If anyone can write the whole script for me I would greatly appreciate your effort.
All general meta-data is organized in the image tag-structure
You can see this, if you open the Image Display Info of an image. (Via the menu, or by pressing CTRL + D) and then browse to the "Tags" section:
All info on the right are image tags and they are organized in a hierarchical tree.
How this tree looks like, and what information is written where, is totally open and will depend on what GMS version you are using, how the hardware is configured etc. Also custom scripts might alter this information.
So for a scripting start, open the data you want to modify and have a look in this tree.
Hint: The following min-script can be useful. It opens a tag-browsing window for the front-most image but as a modeless dialog (i.e. you can keep it open and interact with other parts):
GetFrontImage().ImageGetTagGroup().TagGroupOpenBrowserWindow(0)
The information you need to check against is most probably found in the Microscope Info sub-tree. Here, usually all information gathered from the microscope during acquisition is stored. What is there, will depend on your system and how it is set up.
The information of the STEM image acquisition - as far as the scanning engine and detector is concerned - is most probably in the DigiScan sub-tree.
The Data Bar sub-tree usually contains date and time of creation etc.
Calibration values are not stored in the image tag-structure
What you will not find in this tag-structure is the image calibration, i.e. the values actually used by DM to display calibrated values. These values are "one level up" so to speak here:
This is important to know in the following for your script, because you will need different commands for both the "meta-data" from the tags, and the "calibration" you want to change.
Accessing meta-data by script
The script-commands you need to read from the tags are all described in the F1 help documentation here:
Essentially, you need a command to get the "root" TagGroup of an image, which is ImageGetTagGroup() and then you traverse within this tree.
This might seem confusing - because there are a lot of slightly different commands for the different types of stored tags - but the essential bits are easy:
All "Paths" through the tree are just the individual names (typed exactly)
For each "branch" you have to use a single colon :
The commands to set/get a tag-value all require as input the "root" tagGroup object and the "path" as a string. The get commands require a variable of matching type to store the value in, the set commands need the value which should be written.
= The get commands themeselves return true or false depending on whether or not a tag-path could be found and the value could be read.
So the following script would read the "Imaging Mode" from the tags of the image shown as example above:
string mode
GetFrontImage().ImageGetTagGroup().TagGroupGetTagAsString( "Microscope Info:Imaging Mode", mode )
OKDialog( "Mode: " + mode )
and in a little more verbose form:
string mode // variable to hold the value
image img // variable for the image
string path // variable/constant to specify the where
TagGroup tg // variable to hold the "tagGroup" object
img := GetFrontImage() // Use the selected image
tg = img.ImageGetTagGroup() // From the image get the tags (root)
path = "Microscope Info:Imaging Mode" // specify the path
if ( tg.TagGroupGetTagAsString( path, mode ) )
OKDialog( "Mode: " + mode )
else
Throw( "Tag not found" )
If the tag is not a string but a value, you will need the according commands, i.e.
TagGroupGetTagAsNumber().

postgis shape file import problems

Hi I'm trying to import a shape file from
http://www.nyc.gov/html/dcp/html/bytes/bytesarchive.shtml
into a postgis database. the above files creates MULTIPOLYGONS when i import using shp2pgsql.
then i'm trying to simply determine if lat/long points are contained in my multipolygons
however my select's are not working, and when i print out the poitns of my the_geom column it seems to be very broken.
select st_astext(geom) from (select (st_dumppoints(the_geom)).* from nybb where borocode =1) foo;
gives the result...
st_astext
------------------------------------------
POINT(1007193.83859999 257820.786899999)
POINT(1007209.40620001 257829.435100004)
POINT(1007244.8654 257833.326199993)
POINT(1007283.3496 257839.812399998)
POINT(1007299.3502 257851.488900006)
POINT(1007320.1081 257869.218500003)
POINT(1007356.64669999 257891.055800006)
POINT(1007385.6197 257901.432999998)
POINT(1007421.94509999 257894.084000006)
POINT(1007516.85959999 257890.406100005)
POINT(1007582.59110001 257884.7861)
POINT(1007639.02150001 257877.217199996)
POINT(1007701.29170001 257872.893099993)
...
for points in nyc, this is very off.. what am i doing wrong?
The points are not of. The spatial data that is referred to is NOT in lat/long. This is why numbers are different from what you expect. If you need it to be in long/lat it must be reprojected. See more here: http://postgis.refractions.net/news/20020108/
The projection of the data seems to be in the NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet coordinate system (according to the metadata - see code.).
<spref>
<horizsys>
<planar>
<planci>
<plance Sync="TRUE">coordinate pair</plance>
<coordrep>
<absres Sync="TRUE">0.000000</absres>
<ordres Sync="TRUE">0.000000</ordres>
</coordrep>
<plandu Sync="TRUE">survey feet</plandu>
</planci>
<mapproj><mapprojn Sync="TRUE">Lambert Conformal Conic</mapprojn><lambertc><stdparll Sync="TRUE">40.666667</stdparll><stdparll Sync="TRUE">41.033333</stdparll><longcm Sync="TRUE">-74.000000</longcm><latprjo Sync="TRUE">40.166667</latprjo><feast Sync="TRUE">984250.000000</feast><fnorth Sync="TRUE">0.000000</fnorth></lambertc></mapproj></planar>
<geodetic>
<horizdn Sync="TRUE">North American Datum of 1983</horizdn>
<ellips Sync="TRUE">Geodetic Reference System 80</ellips>
<semiaxis Sync="TRUE">6378137.000000</semiaxis>
<denflat Sync="TRUE">298.257222</denflat>
</geodetic>
<cordsysn>
<geogcsn Sync="TRUE">GCS_North_American_1983</geogcsn>
<projcsn Sync="TRUE">NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet</projcsn>
</cordsysn>
</horizsys>
</spref>
If you work much with spatial data I suggest that you read more about map projection.
I think this is not issue with PostGIS. I checked input esri Shape file nybb.shp with AvisMap Free Viewer and as you see points are weird itself:
However there is something interesting in nybb.shp.xml metadata file:
<spdom>
<bounding>
<westbc Sync="TRUE">-74.257465</westbc>
<eastbc Sync="TRUE">-73.699450</eastbc>
<northbc Sync="TRUE">40.915808</northbc>
<southbc Sync="TRUE">40.495805</southbc>
</bounding>
<lboundng>
<leftbc Sync="TRUE">913090.770096</leftbc>
<rightbc Sync="TRUE">1067317.219904</rightbc>
<bottombc Sync="TRUE">120053.526313</bottombc>
<topbc Sync="TRUE">272932.050103</topbc>
</lboundng>
</spdom>
I am not familiar with those toolkit (ESRI ArcCatalog), but most probably you need to rescale your points after import using that metadata.

pygraphviz/networkx external label (xlabel)

I have a question regarding external labels in pygraphviz. Sadly, I haven't found anything regarding this on the internet.
I want to use networkx to create/parse a graph in tree structure and then use pygraphviz/pydot to draw it. I need external labels on top of the normal labels for the nodes because I want to display values for the nodes + the node name itself.
Let's say I have the following graph (very simplified example of what I'm doing later):
g = nx.Graph()
g.add_edges_from([('A','B'), ('A','C')])
p = nx.drawing.nx_pydot.to_pydot(g)
So I'm using the last line to generate a tree like hraph and I need external labels for B and C.
How to do it?
For pygraphviz you can pass through arbitrary graphviz attributes, so you can do:
dot.node('point', '', shape='point', xlabel='Label')