I am using the "lists" widget in dashing to display a string and a value (in this case the name of the user and number of jobs running on the batch system).
I would like this displayed sorted by value (number of jobs) in descending order. It currently reads these in from a csv file that contains the string (username) and value (number of jobs), sorted in descending order.
CSV.foreach('/path/to/file.csv') do |row|
user = row[0]
numberOfJobs = row[1]
SGE_list[user] = { label: user, value: numberOfJobs }
end
As soon as dashing starts, and reads the file for the first time, this is correct. However when it re-reads the file (which is continuously updated) then it keeps the original order, (regardless of the order in the csv file).
Any suggestions?
Full jobs file:
require 'csv'
JOB_list = Hash.new({ value: 0 })
SCHEDULER.every '5m' do
groups = ["user1", "user2", "user3", "user4", "user5", "user6", "user7", "user8"]
# Read in to get order
CSV.foreach('/opt/dashing/sweet_dashboard_project/jobs/qusage.csv') do |row|
user = row[0]
numberOfJobs = row[1]
JOB_list[user] = { label: user, value: numberOfJobs }
end
# blank all values
for g in groups
JOB_list[g] = { label: g, value: 0 }
end
CSV.foreach('/opt/dashing/sweet_dashboard_project/jobs/qusage.csv') do |row|
user = row[0]
numberOfJobs = row[1]
JOB_list[user] = { label: user, value: numberOfJobs }
end
send_event('batch_jobs', { items: JOB_list.values })
end
The csv file can vary. it could be:
user7, 1000
user2, 987
user8, 800
user6, 400
user5, 200
user4, 122
user1, 89
user3, 2
or, if the user is not running a job, they are not listed so:
user6, 340
user5, 123
user4, 101
it is always sorted by the 2nd column.
Seems like your hash order becomes "burned in" sorted by [user].
Hashes enumerate their values in the order that the corresponding keys were inserted.
http://www.ruby-doc.org/core-2.1.2/Hash.html
Therefore "emptying" the values won't cure this. You could add some sorting to your hash. But if you trust the sorting in your CSV, and it loads correctly with a new Hash object, why not reinitialize the Hash every run? Quick & Easy
require 'csv'
SCHEDULER.every '5m' do
JOB_list = Hash.new
# Trust the CSV file to sort by row[1], a.k.a. NumberOfJobs
CSV.foreach('/opt/dashing/sweet_dashboard_project/jobs/qusage.csv') do |row|
user = row[0]
numberOfJobs = row[1]
JOB_list[user] = { label: user, value: numberOfJobs }
end
send_event('batch_jobs', { items: JOB_list.values })
end
I'd recommend deleting the widget data before repopulating it. Just before the send_event in your script, you could add this line:
Sinatra::Application.settings.history.delete('batch_jobs')
This will erase the widget hash you're working with, without sending an update to any dashboards that are using the widget. Since you're immediately repopulating it, the widget will get the update from that, which will then display the new (and newly sorted) list.
It's not elegant, but it should work for you.
Related
What I need is to insert rows according to the integer value in a column while parsing through the values in the remaining columns to separate their values on the new inserted rows.
I have a table like this
ID
Household
User Count
Show 1
Show 2
Show 3
Show 4
123
House 1
2
Shooter
Dark
1234
House 2
4
Awake
Arrow
Lou
Ozark
And I need an expanded table where each row represents an individual user
ID
Household
User Count
Show 1
Show 2
Show 3
Show 4
123
House 1
1
Shooter
123
House 1
1
Dark
1234
House 2
1
Awake
1234
House 2
1
Arrow
1234
House 2
1
Lou
1234
House 2
1
Ozark
I need to solve this problem using either Google Apps Script or PostgreSQL.
In your situation, when Google Apps Script is used, how about the following sample script?
Sample script:
Please copy and paste the following script to the script editor of Spreadsheet and save the script. And, please set the source and destination sheet names.
function myFunction() {
const srcSheetName = "Sheet1"; // Please set source sheet name.
const dstSheetName = "Sheet2"; // Please set source sheet name.
const ss = SpreadsheetApp.getActiveSpreadsheet();
const sheet = ss.getSheetByName(srcSheetName);
const [header, ...values] = sheet.getDataRange().getValues();
const res = [header, ...values.flatMap(([a, b, c, ...d]) => {
const len = d.length;
return [...Array(c)].map((_, i) => {
const temp = [...Array(i).fill(null), d[i]];
return [a, b, 1, ...temp, ...Array(len - temp.length).fill(null)];
});
})];
ss.getSheetByName(dstSheetName).getRange(1, 1, res.length, res[0].length).setValues(res);
}
When this script is run, the values are retrieved from source sheet, and the values are converted, and then, the converted values are put to the destination sheet. In this case, when your sample input table is used, the sample output table can be obtained.
If you want to use this script as a custom function, how about the following sample script? When your showing input table is used, please put a custom function like =SAMPLE(A1:G3) to a cell. By this, the result values are returned.
function SAMPLE(v) {
const [header, ...values] = v;
return [header, ...values.flatMap(([a, b, c, ...d]) => {
const len = d.length;
return [...Array(c)].map((_, i) => {
const temp = [...Array(i).fill(null), d[i]];
return [a, b, 1, ...temp, ...Array(len - temp.length).fill(null)];
});
})];
}
Note:
This sample script is prepared from your sample input and output tables. So, when you changed the table or your actual Spreadsheet is different from your sample input table, the script might not be able to be used. Please be careful about this.
Reference:
map()
I have a script that looks into values in column G and if the correspondent cell in column A is empty, sends me an email.
--- WHAT WORKS --
It works ok for static values: it sends one email per each not empty cell in column G for which there is no value in column A
--- WHAT DOESN'T WORK --
It sends several emails for what I assume it's every Column G cell (empty or not) when the column A values are fetched from another tab. That way it's like all G and A cells have data, so I get multiple unwanted emails.
This is the script code:
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName('Sheet to send emails');
const data = sh.getRange('A2:G'+sh.getLastRow()).getValues();
data.forEach(r=>{
let overdueValue = r[0];
if (overdueValue === ""){
let name = r[6];
let message = 'Text ' + name;
let subject = 'TEXT.'
MailApp.sendEmail('myemail#gmail.com', subject, message);
}
});
}
And this is the link to the test sheet:
https://docs.google.com/spreadsheets/d/1OKQlm0PjEjDB7PXvt34Og2fa4vPZWnvLazTEawEtOXg/edit?usp=sharing
In this test case, I "should" only get one email, related to ID 55555. With the script as is, I get one related to 55555 and several others "undefined".
To avoid e-mail spam, I didn't add the script to that sheet but it shows the "Vlookup" idea.
Can anyone give me a hand, please?
Thank you in advance
Issue:
The issue with your original script is that the sh.getLastRow returns 1000 (it also processes those rows that doesn't have contents, result to undefined)
Fix 1: Get specific last row of column G:
const gValues = sh.getRange('G1:G').getValues();
const gLastRow = gValues.filter(String).length;
or
Fix 2: Filter data
const data = sh.getRange('A2:G' + sh.getLastRow()).getValues().filter(r => r[6]);
Note:
As Kris mentioned in the comments, there is a specific case where getting the last row above will fail (same with getNextDataCell). This will not properly get the last row WHEN there are blank rows in between the first and last row of the column. If you have this kind of data, then use the 2nd method which is filtering the data.
If your data in column G does not have blank cells in between the first and last row, then any method should work.
I checked your test sheet, and sh.getLastRow() is 1000.
OPTION 1
If column G won't have empty cells between filled ones, then you can do this:
const ss = SpreadsheetApp.getActive();
const sheet = ss.getSheetByName("Sheet to send emails");
// get the first cell in column G
var gHeader = sheet.getRange(1, 7);
// equivelent of using CTRL + down arrow to find the last da
var lastRow = gcell.getNextDataCell(SpreadsheetApp.Direction.DOWN).getRow();
const data = sheet.getRange(2, 1, lastRow, 7).getValues();
OPTION 2
Add another condition to your code - like this:
data.forEach(r=>{
let overdueValue = r[0];
let name = r[6]
// check if the value in col A is blankd and col G is not blank
if (overdueValue === "" && name !== ""){
let message = 'Text ' + name;
let subject = 'TEXT.'
MailApp.sendEmail('myemail#gmail.com', subject, message);
}
});
And to speed it up, use a named range to limit how many rows it has to iterate through:
const ss = SpreadsheetApp.getActive();
const data = ss.getRangeByName("Your_NamedRange_Here").getValues();
I was working on my unittests which for each cases the database should be reset.
Because that the indexes should be kept but the data should be cleared, what will be a faster way to do that reset?
I am using pymongo, not sure if the driver will heavily impacts the performance.
There are 2 ways that come up to my mind
Simply executes delete_many()
Drop the whole collection and recreate the indexes
I found that:
for collections containing < 25K entries, the first way is faster
for collections containing ~ 25K entries, two ways are similar
for collections containing < 25K entries, the second way is faster.
Below is my script to test and run.
import time
import pymongo
from bson import ObjectId
from extutils import exec_timing_result
client = pymongo.MongoClient("mongodb://192.168.50.33:27017/?readPreference=primary&ssl=false")
def test_drop_recreate(col):
col.drop()
col.create_index([("mike", pymongo.DESCENDING)])
def test_delete_many(col):
col.delete_many({})
def main():
db = client.get_database("_tests")
STEP = 1000
to_insert = []
for count in range(0, STEP * 101, STEP):
# Get the MongoDB collection
col = db.get_collection("a")
# Insert dummy data
to_insert.extend([{"_id": ObjectId(), "mike": "ABC"} for _ in range(STEP)])
if to_insert:
col.insert_many(to_insert)
# Record the execution time of dropping the databㄇse then recreate indexes of it
_start = time.time()
test_drop_recreate(col)
ms_drop = time.time() - _start
# Insert dummy data
if to_insert:
col.insert_many(to_insert)
# Record the execution time of simply executes `delete_many()`
_start = time.time()
test_delete_many(col)
ms_del = time.time() - _start
if ms_drop > ms_del:
print(f"{count},-{(ms_drop / ms_del) - 1:.2%}")
else:
print(f"{count},+{(ms_del / ms_drop) - 1:.2%}")
if __name__ == '__main__':
main()
After I ran this script a few times, I generated a graph to visualize the result using the output.
(Above 0) means deletion takes longer
(Below 0) means dropping and recreating takes longer
The value represents additional time consumed.
For example: +20 means deletion takes 20 times longer than drop & create.
I tried to "save" all the existing indexes and recreate them after the collection drop, but there are tons of edge cases and hidden parameters to the index object. Here is my shot:
def convert_key_to_list_index(son_index):
"""
Convert index SON object to a list that can be used to create the same index.
:param SON son_index: The output of "list_indexes()":
SON([
('v', 2),
('key', SON([('timestamp', 1.0)])),
('name', 'timestamp_1'),
('expireAfterSeconds', 3600.0)
])
:return list: List of tuples: (field, direction) to use for pymonog function create_index
"""
key_index_list = []
index_key_definitions = son_index["key"]
for field, direction in index_key_definitions.items():
item = (field, int(direction))
key_index_list.append(item)
return key_index_list
def drop_and_recreate_indexes(db, collection_name):
"""
Use list_indexes() and not index_information() to get the "key" as a SON object instead of regular dict,
Because the SON object is an ordered dict, which the order of the key of the index are important for the
recreation action.
"""
son_indexes = list(db[collection_name].list_indexes())
db[collection_name].drop()
# Re-create the collection indexes
for son_index in son_indexes:
# Skip the default index for the field "_id"
# Notice: For collection without any index except of the default "_id_" index,
# the collection will not be recreated (Recreate the index recreate the collection).
if son_index['name'] == '_id_':
continue
recreate_index(db, collection_name, son_index)
def recreate_index(db, collection_name, son_index):
"""
Get a SON object that represent an index from the db, and create the same index in the db.
:param pymongo.database.Database db:
:param str collection_name:
:param SON son_index: The object that returned from the call to pymongo.collection.Collection.list_indexes()
# Notice: Currently supported only these fields: ["name", "unique", "expireAfterSeconds"]
# For more fields/options see: https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html
# ?highlight=create_index#pymongo.collection.Collection.create_index
:return:
"""
list_index = convert_key_to_list_index(son_index)
# This dict is defined to send parameter if exists to the "create_index" function.
create_index_kwargs = {}
# "name" will always be part of the SON object that represent the index.
create_index_kwargs["name"] = son_index.get("name")
# "unique" may exist or not in the SON object
unique = son_index.get("unique")
if unique:
create_index_kwargs["unique"] = unique
# "expireAfterSeconds" may exist or not in the SON object
expire_after_seconds = son_index.get("expireAfterSeconds", None)
if expire_after_seconds:
create_index_kwargs["expireAfterSeconds"] = expire_after_seconds
db[collection_name].create_index(list_index, **create_index_kwargs)
Im trying to return the values from a 'repeating item' input. But 'inputs.value' doesn't work. I think I need to creat a loop and index for every item on the list but not sure.
I think I need to create a loop and index for every item on the list..
Yes. You are right on that. The 'Repeating Item' is internally stored as an object array. So you need to iterate that array to process it. The individual items are objects and hence you will not be available on the inputs object directly, but via inputs.lineitems, where lineitems is the property name of the repeating item prototype.
For example:
You are creating a repeating items list of items which you want to order. So, you have two inputs inside the repeating items prototype, say itemName and itemQuantity. You name the repeating items property name as LineItems. You want to display it as an output table and also display total quantity ordered. The output table is named Orders and the total label is named Total.
You could then iterate this to process it further, like this:
var result = [], totalItems = 0;
Where result is an array that you would want to map to your output table, and totalItems is where you would cache the total quantity.
inputs.LineItems.forEach(function(item, idx) {
totalItems += item.itemQuantity;
result.push({
'ItemNumber': idx + 1,
'Item': item.itemName,
'Quantity': item.itemQuantity
});
});
Where, you are iterating the repeating items via inputs.LineItems and increment total accordingly . You also prepare the result array to map to the Orders table later on.
This is what you return:
return {
Total: totalItems,
Orders: result
};
Where, Total is the output label you defined earlier, and Orders is the out put table name you defined earlier.
Here is a demo for you to understand it better:
https://jscalc.io/calc/YicDJYCSlYTGYFMS
To see the source, just click on the ellipsis (three dots shown after the 'Powered by JSCalc.io' text) and click "make a copy".
Hope this helps.
I'm comparing 2 users of twitter on who receives the most tweets on a particular day, and put this in a line graph of Highcharts, the following mysql code I have:
<?php
require('mysql_connect.php');
$artist1 = $_POST['dj1'];
$artist2 = $_POST['dj2'];
$dates_result = mysqli_query($con,"SELECT DISTINCT(tweetDate) FROM tb_tweetDetails");
while ($row = mysqli_fetch_array($dates_result)) {
$dates[] = $row['tweetDate'];
}
$artist1_tweetsADay_result = mysqli_query($con,"SELECT tweetDate,COUNT('tweetIds') AS 'amountTweets' FROM tb_tweetDetails WHERE tweetId IN
(SELECT tweetId FROM tb_tweetLink LEFT JOIN tb_artists ON
(tb_tweetLink.artistId = tb_artists.artistId) where artistName = '$artist1') GROUP BY tweetDate");
while ($row = mysqli_fetch_array($artist1_tweetsADay_result)) {
$artist1_tweetsADay_date[] = "'" . $row['tweetDate'] . "'";
$artist1_tweetsADay_amount[] = $row['amountTweets'];
}
$artist1_tweetsADay_result = mysqli_query($con,"SELECT tweetDate,COUNT('tweetIds') AS 'amountTweets' FROM tb_tweetDetails WHERE tweetId IN
(SELECT tweetId FROM tb_tweetLink LEFT JOIN tb_artists ON
(tb_tweetLink.artistId = tb_artists.artistId) where artistName = '$artist2') GROUP BY tweetDate");
while ($row = mysqli_fetch_array($artist1_tweetsADay_result)) {
$artist2_tweetsADay_date[] = "'" . $row['tweetDate'] . "'";
$artist2_tweetsADay_amount[] = $row['amountTweets'];
}
?>
I use this to collect all the available dates I collected tweet data (so also from other then the selected 2 artists)
Then I get the amount of tweets the user received that day, together with the date.
This all works nicely, and the output is as I expected.
Now when I put it in the graph, I put the array of the dates, as the xAxis Categories.
And put the tweetAmount for both artists in the data inputs to create both lines.
The problem is:
Artist 1 has data on 06-04-2013,08-04-2013 & 10-04-2013
Artist 2 has data on 07-04-2013,08-04-2013, 09-04-2013 & 10-04-2013 (so everyday that actually is in my database)
Artist 2 would have his data of 07-04-2013 at 06-04-2013 (since that value comes first)
Artist 1 & 2 have 08-04-2013 under the categorie 07-04-2013 since that is the second available date.
etc. etc.
Is it possible I could use the dates array, to fix the arrays of the amount of tweets, so that every missing date would have 0 assigned to it so the line stays correct with the date.
The two things that I would do to accomplish this:
1) use a datetime x axis
2) while pulling the data from your table, create an array of every date returned in addition to the arrays you are already building. loop through the array of dates, and for each date, check the individual arrays for a value. if no value exists, create it.
You will then have two arrays with the same dates, and you can plot them on a datetime axis and not worry about category index matching.