Unable To Plot Graph From PostgreSQL Query Results In Dash App - postgresql

I am attempting to write a simple code to simply plot a bar graph of some fruit names in the x-axis vs corresponding sales units. The aim of this code is just to understand how to query postgres results from heroku hosted database through a dash app.
Below is the code,
from dash import dash
import dash_core_components as dcc
import dash_html_components as html
import plotly.graph_objs as go
import psycopg2
import os
DATABASE_URL = os.environ['DATABASE_URL']
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
cur = conn.cursor()
cur.execute("SELECT fruits FROM pgrt_table")
fruits1=cur.fetchall()
#print(fruits1)
cur.execute("SELECT sales FROM pgrt_table")
sales1=cur.fetchall()
app = dash.Dash()
app.layout = html.Div(children=[
html.H1(
children='Hello Dash'
),
html.Div(
children='''Dash: A web application framework for Python.'''
),
dcc.Graph(
id='example-graph',
figure=go.Figure(
data=[
go.Bar(
x=fruits1, y=sales1, name='SF'),
#{'x': [1, 2, 3], 'y': [2, 4, 5], 'type': 'bar', 'name': u'Montréal'},
],
#'layout':{
# 'title': 'Dash Data Visualization'
#}
)
)
])
if __name__ == '__main__':
app.run_server(debug=True)
The output is below,
Output to the above code
The corresponding output is just the axes with no bar graphs. The connection with the db is working since printing fruits1 or sales1 gives me the values from the columns in postgres. The only issue is the plotting.
NOTE: This question has been heavily modified since the previous draft was extremely vague without any code to show for.

Example:
fruits1 = [('apple',), ('banana',),
('mango',), ('pineapple',),
('peach',), ('watermelon',)]
The output of your database cannot be used directly:
xData = [fruit[0] for fruit in fruits1]
# gives ['apple', 'banana', 'mango', 'pineapple', 'peach', 'watermelon']
yData = [sales[0] for sales in sales1]
You have to assign your data to the go.Bar object:
go.Bar(x=xData, y=yData, name='SF')

Related

Consolidating multiple Python Charts into one Dashboard with Plotly-Dash

I have two different Python Dashboards, both of which visualize different types of financial data. I would like to have both figures on one single dashboard, one above the other. Would you happen to know if that is possible? If so, I'm sure one has to extend the entire app structure, including the layout and, more importantly, callback part. Has anyone any experience with merging two apps into one dashboard? Below you'll find my code I've assembled so far.
First Dashboard:
# import relevant packages
import pandas as pd
import numpy as np
import matplotlib as mpl
import plotly
import dash
import pyodbc
import plotly.express as px
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output
data = [['2020-01-31', 100, 100, 100], ['2020-02-28', 101, 107, 99], ['2020-03-31', 104, 109, 93], ['2020-04-30', 112, 115, 94], ['2020-05-31', 112, 120, 89]]
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['DATE', 'A', 'B', 'C'])
df = df.set_index('DATE')
df
# create the Dash app
app = dash.Dash()
# Set up the app layout
app.layout = html.Div(children=[
html.H1(children='Index Dashboard'),
html.P('''Pick one or more stocks from the dropdown below.'''),
dcc.Dropdown(id='index-dropdown',
options=[{'label': x, 'value': x}
for x in df.columns],
value='A',
multi=True, clearable=True),
dcc.Graph(id='price-graph')
])
# Set up the callback function
#app.callback(
Output(component_id='price-graph', component_property='figure'),
[Input(component_id='index-dropdown', component_property='value')],
)
def display_time_series(selected_index):
dff = df[selected_index] # Only columns selected in dropdown
fig = px.line(dff, x=df.index, y=selected_index, labels={'x': 'x axis label'})
fig.update_layout(
title="Price Index Development",
xaxis_title="Month",
yaxis_title="Price",
font=dict(size=13))
return fig
# Run local server
if __name__ == '__main__':
app.run_server(debug=True, use_reloader=False)
Second Dashboard:
data2 = [['A', 'B', 0.4], ['A', 'C', 0.5], ['A', 'D', 0.1], ['X', 'Y', 0.15], ['X', 'Z', 0.85]]
df2 = pd.DataFrame(data2, columns = ['BM_NAME', 'INDEX_NAME', 'WEIGHT'])
df2
barchart = px.bar(
data_frame=df2,
x=df2.BM_NAME,
y="WEIGHT",
color="INDEX_NAME",
opacity=0.9,
barmode='group')
barchart
# create the Dash app
app = dash.Dash()
# set up app layout
app.layout = html.Div(children=[
html.H1(children='BM Composition'),
dcc.Dropdown(id='BM-dropdown',
options=[{'label': x, 'value': x}
for x in df2.BM_NAME.unique()],
value='A',
multi=False, clearable=True),
dcc.Graph(id='bar-chart')
])
# set up the callback function
#app.callback(
Output(component_id="bar-chart", component_property="figure"),
[Input(component_id="BM-dropdown", component_property="value")],
)
def display_BM_composition(selected_BM):
filtered_BM = df2[df2.BM_NAME == selected_BM] # Only use unique values in column "BM_NAME" selected in dropdown
barchart = px.bar(
data_frame=filtered_BM,
x="BM_NAME",
y="WEIGHT",
color="INDEX_NAME",
opacity=0.9,
barmode='group')
return barchart
# Run local server
if __name__ == '__main__':
app.run_server(debug=True, use_reloader=False)
Many thanks in advance!

How I can overcome the _geoslib problem in this code?

This code is specified to visualize the CALIPSO satellite atmospheric profiles
The input files are .HDF
The code is copyrighted to the HDF group.
In the begining, I struggled with installing the basemap,
finally I installed it using .whl file on my windows10.
Now, this error is reached when I run the script:
SystemError:
execution of module _geoslib raised unreported exception.
I have looked a lot in google, but nothing done.
Can you please help me?
Cheers
"Copyright (C) 2014-2019 The HDF Group
Copyright (C) 2014 John Evans
This example code illustrates how to access and visualize a LaRC CALIPSO file
in file in Python.
If you have any questions, suggestions, or comments on this example, please use
the HDF-EOS Forum (http://hdfeos.org/forums). If you would like to see an
example of any other NASA HDF/HDF-EOS data product that is not listed in the
HDF-EOS Comprehensive Examples page (http://hdfeos.org/zoo), feel free to
contact us at eoshelp#hdfgroup.org or post it at the HDF-EOS Forum
(http://hdfeos.org/forums).
Usage: save this script and run
$python CAL_LID_L2_VFM-ValStage1-V3-02.2011-12-31T23-18-11ZD.hdf.py
The HDF file must either be in your current working directory
or in a directory specified by the environment variable HDFEOS_ZOO_DIR.
Tested under: Python 2.7.15::Anaconda custom (64-bit)
Last updated: 2019-01-25
"""
import os
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.basemap import Basemap
from matplotlib import colors
USE_NETCDF4 = False
def run(FILE_NAME):
# Identify the data field.
DATAFIELD_NAME = 'Feature_Classification_Flags'
if USE_NETCDF4:
from netCDF4 import Dataset
nc = Dataset(FILE_NAME)
# Subset the data to match the size of the swath geolocation fields.
# Turn off autoscaling, we'll handle that ourselves due to presence of
# a valid range.
var = nc.variables[DATAFIELD_NAME]
data = var[:,1256]
# Read geolocation datasets.
lat = nc.variables['Latitude'][:]
lon = nc.variables['Longitude'][:]
else:
from pyhdf.SD import SD, SDC
hdf = SD(FILE_NAME, SDC.READ)
# Read dataset.
data2D = hdf.select(DATAFIELD_NAME)
data = data2D[:,1256]
# Read geolocation datasets.
latitude = hdf.select('Latitude')
lat = latitude[:]
longitude = hdf.select('Longitude')
lon = longitude[:]
# Subset data. Otherwise, all points look black.
lat = lat[::10]
lon = lon[::10]
data = data[::10]
# Extract Feature Type only through bitmask.
data = data & 7
# Make a color map of fixed colors.
cmap = colors.ListedColormap(['black', 'blue', 'yellow', 'green', 'red', 'purple', 'gray', 'white'])
# The data is global, so render in a global projection.
m = Basemap(projection='cyl', resolution='l',
llcrnrlat=-90, urcrnrlat=90,
llcrnrlon=-180, urcrnrlon=180)
m.drawcoastlines(linewidth=0.5)
m.drawparallels(np.arange(-90.,90,45))
m.drawmeridians(np.arange(-180.,180,45), labels=[True,False,False,True])
x,y = m(lon, lat)
i = 0
for feature in data:
m.plot(x[i], y[i], 'o', color=cmap(feature), markersize=3)
i = i+1
long_name = 'Feature Type at Altitude = 2500m'
basename = os.path.basename(FILE_NAME)
plt.title('{0}\n{1}'.format(basename, long_name))
fig = plt.gcf()
# define the bins and normalize
bounds = np.linspace(0,8,9)
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
# create a second axes for the colorbar
ax2 = fig.add_axes([0.93, 0.2, 0.01, 0.6])
cb = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, spacing='proportional', ticks=bounds, boundaries=bounds, format='%1i')
cb.ax.set_yticklabels(['invalid', 'clear', 'cloud', 'aerosol', 'strato', 'surface', 'subsurf', 'no signal'], fontsize=5)
# plt.show()
pngfile = "{0}.py.png".format(basename)
fig.savefig(pngfile)
if __name__ == "__main__":
# If a certain environment variable is set, look there for the input
# file, otherwise look in the current directory.
hdffile = 'CAL_LID_L2_VFM-ValStage1-V3-02.2011-12-31T23-18-11ZD.hdf'
try:
fname = os.path.join(os.environ['HDFEOS_ZOO_DIR'], ncfile)
except KeyError:
fname = hdffile
run(fname)
Please try miniconda and use basemap from conda-forge:
conda install -c conda-forge basemap

Receiving key error on Networkx color_map

I'm having trouble getting the color_map to work with my networkx graph. It's fairly simply code but won't seem to work. I've looked through other similar threads but no the solutions don't seem to work.
I have data that look like this:
edgelist_manual = [{'source': 'ABE', 'target': 'ATL', 'value': 851},
{'source': 'ABE', 'target': 'BHM', 'value': 1},
{'source': 'ABE', 'target': 'CLE', 'value': 805}]
edgelist = pd.DataFrame(edgelist_manual)
nodelist_manual = [{'source': 'ABE', 'value': '4807', 'group': 0},
{'source': 'ABI', 'value': '2660', 'group': 4},
{'source': 'ABQ', 'value': '41146', 'group': 2}]
nodelist = pd.DataFrame(nodelist_manual)
I run the code below, but my color_map keep screwing up. I just get a key error on the 'group' reference.
import itertools
import copy
import networkx as nx
import pandas as pd
import matplotlib.pyplot as plt
nodelist = pd.read_csv('final_nodes.csv')
edgelist = pd.read_csv('final_edges.csv')
g = nx.Graph()
for i, elrow in edgelist.iterrows():
g.add_edge(elrow[0], elrow[1], attr_dict=elrow[2:].to_dict())
for i, nlrow in nodelist.iterrows():
g.node[nlrow['source']].update(nlrow[1:].to_dict())
color_map = {0: 'r', 1:'b', 2:'r', 3:'b', 4:'r', 5:'b'}
colors = [color_map[g.node[node]['group']] for node in g]
nx.draw(g, node_color=colors)
ax = plt.gca()
ax.collections[0].set_edgecolor("#555555")
plt.show()
The only difference from this and my code is that rather than creating the data manually I'm loading it from .csv. I've checked for trailing whitespaces on the feature labels but nothing. I don't understand indices well so I wonder if those are messing it up. Any ideas?
Thanks!

ValueError: Cannot feed value of shape (1, 2048, 2048, 1) for Tensor 'image_tensor:0', which has shape '(?, ?, ?, 3)'

Using TensorFlow I am trying to detect one object(png and grayscale image). I have trained and exported a model.ckpt successfully. Now I am trying to restore the saved model.ckpt for prediction. Here is the script:
import numpy as np
import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import zipfile
from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image
if tf.__version__ != '1.4.0':
raise ImportError('Please upgrade your tensorflow installation to v1.4.0!')
# This is needed to display the images.
#matplotlib inline
# This is needed since the notebook is stored in the object_detection folder.
sys.path.append("..")
from utils import label_map_util
from utils import visualization_utils as vis_util
MODEL_NAME = 'melon_graph'
# Path to frozen detection graph. This is the actual model that is used for the object detection.
PATH_TO_CKPT = MODEL_NAME + '/frozen_inference_graph.pb'
# List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = os.path.join('training', 'object_detection.pbtxt')
NUM_CLASSES = 1
detection_graph = tf.Graph()
with detection_graph.as_default():
od_graph_def = tf.GraphDef()
with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
serialized_graph = fid.read()
od_graph_def.ParseFromString(serialized_graph)
tf.import_graph_def(od_graph_def, name='')
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)
def load_image_into_numpy_array(image):
(im_width, im_height) = image.size
return np.array(image.getdata()).reshape((im_height, im_width, 1)).astype(np.float64)
# For the sake of simplicity we will use only 2 images:
# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.
PATH_TO_TEST_IMAGES_DIR = 'test_images'
TEST_IMAGE_PATHS = [ os.path.join(PATH_TO_TEST_IMAGES_DIR, 'te_data{}.png'.format(i)) for i in range(1, 336) ]
# Size, in inches, of the output images.
IMAGE_SIZE = (12, 8)
with detection_graph.as_default():
with tf.Session(graph=detection_graph) as sess:
# Definite input and output Tensors for detection_graph
image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
# Each box represents a part of the image where a particular object was detected.
detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
# Each score represent how level of confidence for each of the objects.
# Score is shown on the result image, together with the class label.
detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')
num_detections = detection_graph.get_tensor_by_name('num_detections:0')
for image_path in TEST_IMAGE_PATHS:
image = Image.open(image_path)
# the array based representation of the image will be used later in order to prepare the
# result image with boxes and labels on it.
image_np = load_image_into_numpy_array(image)
# Expand dimensions since the model expects images to have shape: [1, None, None, 3]
image_np_expanded = np.expand_dims(image_np, axis=0)
# Actual detection.
(boxes, scores, classes, num) = sess.run([detection_boxes, detection_scores, detection_classes, num_detections], feed_dict={image_tensor: image_np_expanded})
# Visualization of the results of a detection.
vis_util.visualize_boxes_and_labels_on_image_array(image_np,np.squeeze(boxes),np.squeeze(classes).astype(np.float64), np.squeeze(scores), category_index, use_normalized_coordinates=True, line_thickness=5)
plt.figure(figsize=IMAGE_SIZE)
plt.imshow(image_np)
and this is the error
Traceback (most recent call last): File "cochlear_detection.py",
line 81, in
(boxes, scores, classes, num) = sess.run([detection_boxes, detection_scores, detection_classes, num_detections],
feed_dict={image_tensor: image_np_expanded}) File
"/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py",
line 889, in run
run_metadata_ptr) File "/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py",
line 1096, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape()))) ValueError: Cannot feed value of shape (1, 2048, 2048, 1) for Tensor
'image_tensor:0', which has shape '(?, ?, ?, 3)'

How to add meta_data to Pandas dataframe?

I use Pandas dataframe heavily. And need to attach some data to the dataframe, for example to record the birth time of the dataframe, the additional description of the dataframe etc.
I just can't find reserved fields of dataframe class to keep the data.
So I change the core\frame.py file to add a line _reserved_slot = {} to solve my issue. I post the question here is just want to know is it OK to do so ? Or is there better way to attach meta-data to dataframe/column/row etc?
#----------------------------------------------------------------------
# DataFrame class
class DataFrame(NDFrame):
_auto_consolidate = True
_verbose_info = True
_het_axis = 1
_col_klass = Series
_AXIS_NUMBERS = {
'index': 0,
'columns': 1
}
_reserved_slot = {} # Add by bigbug to keep extra data for dataframe
_AXIS_NAMES = dict((v, k) for k, v in _AXIS_NUMBERS.iteritems())
EDIT : (Add demo msg for witingkuo's way)
>>> df = pd.DataFrame(np.random.randn(10,5), columns=list('ABCDEFGHIJKLMN')[0:5])
>>> df
A B C D E
0 0.5890 -0.7683 -1.9752 0.7745 0.8019
1 1.1835 0.0873 0.3492 0.7749 1.1318
2 0.7476 0.4116 0.3427 -0.1355 1.8557
3 1.2738 0.7225 -0.8639 -0.7190 -0.2598
4 -0.3644 -0.4676 0.0837 0.1685 0.8199
5 0.4621 -0.2965 0.7061 -1.3920 0.6838
6 -0.4135 -0.4991 0.7277 -0.6099 1.8606
7 -1.0804 -0.3456 0.8979 0.3319 -1.1907
8 -0.3892 1.2319 -0.4735 0.8516 1.2431
9 -1.0527 0.9307 0.2740 -0.6909 0.4924
>>> df._test = 'hello'
>>> df2 = df.shift(1)
>>> print df2._test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "D:\Python\lib\site-packages\pandas\core\frame.py", line 2051, in __getattr__
(type(self).__name__, name))
AttributeError: 'DataFrame' object has no attribute '_test'
>>>
This is not supported right now. See https://github.com/pydata/pandas/issues/2485. The reason is the propogation of these attributes is non-trivial. You can certainly assign data, but almost all pandas operations return a new object, where the assigned data will be lost.
Your _reserved_slot will become a class variable. That might not work if you want to assign different value to different DataFrame. Probably you can assign what you want to the instance directly.
In [6]: import pandas as pd
In [7]: df = pd.DataFrame()
In [8]: df._test = 'hello'
In [9]: df._test
Out[9]: 'hello'
I think a decent workaround is putting your datafame into a dictionary with your metadata as other keys. So if you have a dataframe with cashflows, like:
df = pd.DataFrame({'Amount': [-20, 15, 25, 30, 100]},index=pd.date_range(start='1/1/2018', periods=5))
You can create your dictionary with additional metadata and put the dataframe there
out = {'metadata': {'Name': 'Whatever', 'Account': 'Something else'}, 'df': df}
and then use it as out[df]