Related
I need to import a large table of data into my database in one go. I currently have it as an Excel file but I am happy to copy it to Google sheets etc.
So far I've added a few entries manually directly via cloud firestore.
Is there a solution already out there to achieve this?
I think the easiest way to export table data into Firestore is to use a Google Apps Script Library (for Google Sheets).
Step 1
Make a Copy of THIS example Google Spreadsheet I created as an example
Step 2
From the menu of YOUR copy of the Example Google Spreadsheet from step 1, click Tools > Script Editor. This should open up the example Google App Script associated with the example spreadsheet.
Step 3
Follow the Steps for installing this library and then update the script with the following:
email
key
projectID
These variables are generated by going to the Google Service Accounts page. This will require that you already have a Firebase or Google Cloud account setup. I won't repeat all the steps that are already iterated in in the aforementioned Github writeup. Just follow them carefully, and realize that the private_key is THE ENTIRE KEY starting with -----BEGIN PRIVATE KEY-----\n, EVERYTHING in between, and ending with \n-----END PRIVATE KEY-----\n
Step 4
Insert a page on your spreadsheet that contains your data, and EDIT the script to use your new sheet name and your data. I have HEAVILY commented the script so it's pretty clear what almost every line of code is doing. For those of you that just want to peek at the Google App Script that's behind this spreadsheet, here's the code:
// Note this Script uses an external library as per this page:
// https://github.com/grahamearley/FirestoreGoogleAppsScript
// This solution requires a Google Spreadhseet and a Firebase Account
// FOLLOW THE INSTRUCTIONS ON THAT GITHUB REPO TO SETUP NEEDED API KEYS!!!
//Global Variables
const ss = SpreadsheetApp.getActiveSpreadsheet(); // Gets the active "workbook"
const sheet = ss.getSheetByName('Restaurants'); // CHANGE TO YOUR SHEET NAME
const headerRowNumber = 1; // If you have more than one row for your header, then change this value to number of header rows
// If you want to mark modified cells, then set up a trigger for the following function:
// Edit > Current Project Triggers > (+ Add Trigger) > On Edit Spreadsheet etc
function onEdit(e) {
var cell = ss.getActiveCell(); //This will also effectively get our row
var dataRange = sheet.getDataRange(); //This checks for all rows/columns with data
var modifiedCol = dataRange.getLastColumn()-1; //Our "modified" column should be the second to last
if (cell.getColumn() < modifiedCol && cell.getRow() > headerRowNumber) { //If we edit any cells to the left of our modified column and below our header...
var celltoMark = sheet.getRange(cell.getRowIndex(),modifiedCol) //Get the R/C cordinates of cell to place modified time
celltoMark.setValue(new Date()); //write timestamp to that cell
}
};
// This will parse any comma separated lists you create in any of your fields (useful for search words, or attributes, etc)
function listToArray(list) {
var ogArray = list.split(","); //Input is a comma separated list
let trimmedArr = ogArray.map(string => string.trim()); //Let's strip out the leading/trailing whitespaces if any
return trimmedArr; //return the cleaned array
}
function writeToFireStore() {
const email = 'sheets#yourprojectid.iam.gserviceaccount.com'; // CHANGE THIS!!!
const key = '-----BEGIN PRIVATE KEY-----\nYOURPRIVATEKEY\n-----END PRIVATE KEY-----\n'; // CHANGE THIS!!!
const projectID = 'yourprojectid'; // CHANGE THIS!!!
var firestore = FirestoreApp.getFirestore(email, key, projectID);
const collection = "MySpreadsheetData"; // Name of your Firestore Database "Collection"
var dataRange = sheet.getDataRange().offset(headerRowNumber, 0, sheet.getLastRow() - headerRowNumber); //this is your data range
var data = dataRange.getValues(); // this is an array of your datarange's values
var lastCol = dataRange.getLastColumn(); // this is the last column with a header
var newDoc = {}; // Instantiate your data object. Each one will become the data for your firestore documents
// r = row number in this case
for (let r = 0; r <= dataRange.getLastRow(); r++) {
//Logger.log("R = ",r);
var cellMod = dataRange.getCell(r+1, lastCol-1);
var cellFS = dataRange.getCell(r+1, lastCol);
var cellModVal = cellMod.getValue();
var cellFSVal = cellFS.getValue();
//
// IMPORTANT READ THIS IMPORTANT READ THIS IMPORTANT READ THIS IMPORTANT READ THIS IMPORTANT READ THIS!!!
// Well, read the line below...
if (r > 2) break; //Comment Out this line after you're done testing otherwise you'll write all your rows to firestore after every run
newDoc[r] = {
name : data[r][1],
category : data[r][2],
cuisine : data[r][3],
address: {
add1: data[r][4],
add2: data[r][5],
city: data[r][6],
state: data[r][7],
zip: data[r][8]
},
tel: data[r][9],
searchterms: listToArray(data[r][10]) //Let's turn a csv list into an array
}
// For the sake of efficiency and to save $, we WON'T create documents that have already been created...
// ...and we won't update documents that have a fireStore Timestamp that's newer than a Modified Timestamp
// If there's not firestore timestamp in our spreadsheet, then let's create firestore document and update firestore stamp:
if (!cellFSVal) {
var now = new Date(); //Generate timestamp right now
try {
firestore.createDocument(collection + "/" + data[r][0], newDoc[r]); // To Use Your Own Document ID
//Now let's insert a timestamp in our Firestore TS column of the sheet so we know it's been added to Firestore
cellFS.setValue(now);
Logger.log("Row ",r,"(",data[r][1],") is NEW and was added to FireStore Successfully");
} catch (e) {
Logger.log("Error: ",e," : Document with same name already existed in Firestore.");
}
}
//var if FS Timestamp exists but, the modified time stamp is greater, let's update the Firstore Document
else if ((cellFSVal) && (cellModVal > cellFSVal)) {
try {
firestore.updateDocument(collection + "/" + data[r][0], newDoc[r]);
//Now let's insert a timestamp in our Firestore TS column of the sheet so we know it's been updated to Firestore
cellFS.setValue(now);
Logger.log("Row ",r,"(",data[r][1],") updated/edited.");
} catch (e) {
Logger.log("Error: ",e," : Document existed, we tried updating it, but jack shit happened.");
}
}
else {
Logger.log("Row ",r,"(",data[r][1],") Already in Firestore & hasn't been modified. Skipped.");
}
}
}
Step 5
Once your script is modified to your needs, it's time to run the script. Simply save it (File > Save), then choose the function "writeToFireStore" from the "Select function" dropdown selector in the menu bar (in between the icon of the bug, and the lightbulb), then hit the PLAY icon (to the left of the bug icon). At this point, you will likely be prompted to accept permissions to run the script, (which you need to accept if you want to run the script). Once you've accepted the permissions, then run the "writeToFireStore" function again if it hasn't already run, and voila!
NOTES:
I created a function that automatically writes a Modified Timestamp to the second to last column in the target sheet, and when you run the function, writes a Firestore Timestamp (so you know which rows have been successfully exported to Firestore). This way, if you run the firestore function again, and you haven't changed the data on your sheet, it won't bother updating the database with the same data (and will save you money and/or server resources). For this functionality to work, you must setup project Triggers (which is explained inside the script in the comments).
I am updating sort & filter models via api:
this.gridApi.setFilterModel(filterModels);
this.gridApi.setSortModel(sortModels);
The problem with this is I have a server request bound to the change even of both sort & filter so when user changes then the data is updated. This means when I change model on code like restoring a state or resetting the filters it causes multiple requests.
Is there a way to update the filter/sort model without triggering the event?
I see there is a ColumnEventType parameter but couldn't see how it works. Can I specify some variable that I can look for inside my event handlers to get them to ignore calls that are not generated from user?
I am trying to manage URL state so when url query params change my code sets the models in the grids but this ends up causing the page to reload multiple times because the onFilter and onSort events get called when the model is set and there is no way I can conceive to prevent this.
At the time, you are going to have to manage this yourself, ie, just before you call the setModel, somehow flag this in a shared part of your app (maybe a global variable)
Then when you react to these events, check the estate of this, to guess where it came from.
Note that at the moment, we have added source to the column events, but they are not yet for the model events, we are planning to add them though, but we have no ETA
Hope this helps
I had to solve similar issue. I found solution which working for my kind of situation. Maybe this help someone.
for (let j = 0; j < orders.length; j++) {
const sortModelEntry = orders[j];
if (typeof sortModelEntry.property === 'string') {
const column: Column = this.gridColumnApi.getColumn(sortModelEntry.property);
if (column && ! column.getColDef().suppressSorting) {
column.setSort(sortModelEntry.direction.toLowerCase());
column.setSortedAt(j);
}
}
this.gridApi.refreshHeader();
Where orders is array of key-value object where key is name of column and value is sorting directive (asc/desc).
Set filter without refresh was complicated
for (let j = 0; j < filters.length; j++) {
const filterModelEntry = filters[j];
if (typeof filterModelEntry.property === 'string') {
const column: Column = this.gridColumnApi.getColumn(filterModelEntry.property);
if (column && ! column.getColDef().suppressFilter) {
const filter: any = this.gridApi.getFilterApi(filterModelEntry.property);
filter['filter'] = filterModelEntry.command;
filter['defaultFilter'] = filterModelEntry.command;
filter['eTypeSelector'].value = filterModelEntry.command;
filter['filterValue'] = filterModelEntry.value;
filter['filterText'] = filterModelEntry.value;
filter['eFilterTextField'].value = filterModelEntry.value;
column.setFilterActive(true);
}
}
}
Attributes in filter:
property - name of column
command - filter action (contains, equals, ...)
value - value used in filter
For anyone else looking for a solution to this issue in Nov 2020, tapping into onFilterModified() might help. This gets called before onFilterChanged() so setting a value here (eg. hasUserManuallyChangedTheFilters = false, etc.) and checking the same in the filter changed event is a possible workaround. Although, I haven't found anything similar for onSortChanged() event, one that gets called before the sorting is applied to the grid.
I am not sure any clean way to achieve this but I noticed that FilterChangedEvent has "afterFloatingFilter = false" only if filterModel was updated from ui.
my workaround is as below
onFilterChanged = event:FilterChangedEvent) => {
if(event.afterFloatingFilter === undefined) return;
console.log("SaveFilterModel")
}
I'm new to stackexchange so my apologies if this question is too extensive or already answered somewhere I couldn't find. You can find the spreadsheet here, the script here and the dashboard (dev version) here.
I have been banging my head on handling dates in the google app script visualization for days.
My ultimate goal is to make a dashboard that includes an annotated timeline as well as other charts based on a data set in a spreadsheet. I have started this process using Mogsdad tutorial on creating a 3-tier google visualization dashboard, where the data is pulled from external spreadsheet and then pulled into the DataTable using arrayToDataTable. Everything worked great out of the box. However, my data contains dates, so I added a date column to the original data, but alas arrayToDataTable doesn't accept date type per this post. So when a Date column is added i get the following result:
ScriptError: The script completed but the returned value is not a
supported return type.
I have tried multiple approaches to ensure even date formatting: options includes putting the values in the date column through new Date(dateColumn[i]), dateColumn[i].toJSON() (renders the dash board, but dates aren't able to be processed), forced date formats in the spreadsheet (yyyy-MM-dd), using the DataView outlined in the post above (dashboards don't get past 'Loading'), and such.
So my question is what is the alternatives to arrayToDataTable that will accept date columns in this 3-tier approach? Or alternatively, what are the errors in the below methods?
For all the cases when I have attempted to add columns I have changed the code from var data = google.visualization.arrayToDataTable(response,false) to var data = google.visualization.DataTable()
I have tried the following:
Manually adding columns and manually adding data (not working)
//Add Columns
data.addColumn('string','Name');
data.addColumn('string','Gender');
data.addColumn('number','Age');
data.addColumn('number','Donuts eaten');
data.addColumn('date','Last Donut Eaten');
//Add Rows
data.addRows([
['Miranda','Female', 22,6,6],
['Jessica','Female',22,6,12],
['Aaron','Male',3,1,13]
]);
Automatically adding the rows without dates (The rows are added, but it only works if there are no date columns)
//Add Rows
for (var i=1; i<response.length; i++) {
data.addRow(response[i]);
}
Manually adding columns and automatically adding rows (not working, combination of 1 and 2)
Automatically adding the columns with loops (not working, neither if dates or not)
for (var i=0; i<response[0].length; i++) {
if (response[1][i] instanceof Date) { //Checks if first value is Date
data.addColumn('date',response[0][i]);
};
else if (response[1][i] instanceof Number) //Checks if first value is Number
data.addColum('number',response[0][i]);
else data.addColumn('string',response[0][i]; //Otherwise assume string
};
Thank you so much for your help!
you can use the Query (google.visualization.Query) class to pull the data from the spreadsheet,
this will convert the date column properly...
google.charts.load('current', {
packages:['table']
}).then(function () {
var queryURL = 'https://docs.google.com/spreadsheets/d/1aaxYNLCuPz3o3TA1jdryenUP01Qbkdaut4AR5eIhe9s/edit#gid=0';
var query = new google.visualization.Query(queryURL).send(function (response) {
var data = response.getDataTable();
// show column types
for (var i = 0; i < data.getNumberOfColumns(); i++) {
console.log(data.getColumnLabel(i), '=', data.getColumnType(i));
}
// draw table chart
var table = new google.visualization.Table(document.getElementById('chart-table'));
table.draw(data);
});
});
<script src="https://www.gstatic.com/charts/loader.js"></script>
<div id="chart-table"></div>
note: the example uses jsapi to load the library,
this library should no longer be used.
according to the release notes...
The version of Google Charts that remains available via the jsapi loader is no longer being updated consistently. Please use the new gstatic loader.js from now on.
this will only change the load statement, see above snippet...
I'm trying to test out Firebase to allow users to post comments using push. I want to display the data I retrieve with the following;
fbl.child('sell').limit(20).on("value", function(fbdata) {
// handle data display here
}
The problem is the data is returned in order of oldest to newest - I want it in reversed order. Can Firebase do this?
Since this answer was written, Firebase has added a feature that allows ordering by any child or by value. So there are now four ways to order data: by key, by value, by priority, or by the value of any named child. See this blog post that introduces the new ordering capabilities.
The basic approaches remain the same though:
1. Add a child property with the inverted timestamp and then order on that.
2. Read the children in ascending order and then invert them on the client.
Firebase supports retrieving child nodes of a collection in two ways:
by name
by priority
What you're getting now is by name, which happens to be chronological. That's no coincidence btw: when you push an item into a collection, the name is generated to ensure the children are ordered in this way. To quote the Firebase documentation for push:
The unique name generated by push() is prefixed with a client-generated timestamp so that the resulting list will be chronologically-sorted.
The Firebase guide on ordered data has this to say on the topic:
How Data is Ordered
By default, children at a Firebase node are sorted lexicographically by name. Using push() can generate child names that naturally sort chronologically, but many applications require their data to be sorted in other ways. Firebase lets developers specify the ordering of items in a list by specifying a custom priority for each item.
The simplest way to get the behavior you want is to also specify an always-decreasing priority when you add the item:
var ref = new Firebase('https://your.firebaseio.com/sell');
var item = ref.push();
item.setWithPriority(yourObject, 0 - Date.now());
Update
You'll also have to retrieve the children differently:
fbl.child('sell').startAt().limitToLast(20).on('child_added', function(fbdata) {
console.log(fbdata.exportVal());
})
In my test using on('child_added' ensures that the last few children added are returned in reverse chronological order. Using on('value' on the other hand, returns them in the order of their name.
Be sure to read the section "Reading ordered data", which explains the usage of the child_* events to retrieve (ordered) children.
A bin to demonstrate this: http://jsbin.com/nonawe/3/watch?js,console
Since firebase 2.0.x you can use limitLast() to achieve that:
fbl.child('sell').orderByValue().limitLast(20).on("value", function(fbdataSnapshot) {
// fbdataSnapshot is returned in the ascending order
// you will still need to order these 20 items in
// in a descending order
}
Here's a link to the announcement: More querying capabilities in Firebase
To augment Frank's answer, it's also possible to grab the most recent records--even if you haven't bothered to order them using priorities--by simply using endAt().limit(x) like this demo:
var fb = new Firebase(URL);
// listen for all changes and update
fb.endAt().limit(100).on('value', update);
// print the output of our array
function update(snap) {
var list = [];
snap.forEach(function(ss) {
var data = ss.val();
data['.priority'] = ss.getPriority();
data['.name'] = ss.name();
list.unshift(data);
});
// print/process the results...
}
Note that this is quite performant even up to perhaps a thousand records (assuming the payloads are small). For more robust usages, Frank's answer is authoritative and much more scalable.
This brute force can also be optimized to work with bigger data or more records by doing things like monitoring child_added/child_removed/child_moved events in lieu of value, and using a debounce to apply DOM updates in bulk instead of individually.
DOM updates, naturally, are a stinker regardless of the approach, once you get into the hundreds of elements, so the debounce approach (or a React.js solution, which is essentially an uber debounce) is a great tool to have.
There is really no way but seems we have the recyclerview we can have this
query=mCommentsReference.orderByChild("date_added");
query.keepSynced(true);
// Initialize Views
mRecyclerView = (RecyclerView) view.findViewById(R.id.recyclerView);
mManager = new LinearLayoutManager(getContext());
// mManager.setReverseLayout(false);
mManager.setReverseLayout(true);
mManager.setStackFromEnd(true);
mRecyclerView.setHasFixedSize(true);
mRecyclerView.setLayoutManager(mManager);
I have a date variable (long) and wanted to keep the newest items on top of the list. So what I did was:
Add a new long field 'dateInverse'
Add a new method called 'getDateInverse', which just returns: Long.MAX_VALUE - date;
Create my query with: .orderByChild("dateInverse")
Presto! :p
You are searching limitTolast(Int x) .This will give you the last "x" higher elements of your database (they are in ascending order) but they are the "x" higher elements
if you got in your database {10,300,150,240,2,24,220}
this method:
myFirebaseRef.orderByChild("highScore").limitToLast(4)
will retrive you : {150,220,240,300}
In Android there is a way to actually reverse the data in an Arraylist of objects through the Adapter. In my case I could not use the LayoutManager to reverse the results in descending order since I was using a horizontal Recyclerview to display the data. Setting the following parameters to the recyclerview messed up my UI experience:
llManager.setReverseLayout(true);
llManager.setStackFromEnd(true);
The only working way I found around this was through the BindViewHolder method of the RecyclerView adapter:
#Override
public void onBindViewHolder(final RecyclerView.ViewHolder holder, int position) {
final SuperPost superPost = superList.get(getItemCount() - position - 1);
}
Hope this answer will help all the devs out there who are struggling with this issue in Firebase.
Firebase: How to display a thread of items in reverse order with a limit for each request and an indicator for a "load more" button.
This will get the last 10 items of the list
FBRef.child("childName")
.limitToLast(loadMoreLimit) // loadMoreLimit = 10 for example
This will get the last 10 items. Grab the id of the last record in the list and save for the load more functionality. Next, convert the collection of objects into and an array and do a list.reverse().
LOAD MORE Functionality: The next call will do two things, it will get the next sequence of list items based on the reference id from the first request and give you an indicator if you need to display the "load more" button.
this.FBRef
.child("childName")
.endAt(null, lastThreadId) // Get this from the previous step
.limitToLast(loadMoreLimit+2)
You will need to strip the first and last item of this object collection. The first item is the reference to get this list. The last item is an indicator for the show more button.
I have a bunch of other logic that will keep everything clean. You will need to add this code only for the load more functionality.
list = snapObjectAsArray; // The list is an array from snapObject
lastItemId = key; // get the first key of the list
if (list.length < loadMoreLimit+1) {
lastItemId = false;
}
if (list.length > loadMoreLimit+1) {
list.pop();
}
if (list.length > loadMoreLimit) {
list.shift();
}
// Return the list.reverse() and lastItemId
// If lastItemId is an ID, it will be used for the next reference and a flag to show the "load more" button.
}
I'm using ReactFire for easy Firebase integration.
Basically, it helps me storing the datas into the component state, as an array. Then, all I have to use is the reverse() function (read more)
Here is how I achieve this :
import React, { Component, PropTypes } from 'react';
import ReactMixin from 'react-mixin';
import ReactFireMixin from 'reactfire';
import Firebase from '../../../utils/firebaseUtils'; // Firebase.initializeApp(config);
#ReactMixin.decorate(ReactFireMixin)
export default class Add extends Component {
constructor(args) {
super(args);
this.state = {
articles: []
};
}
componentWillMount() {
let ref = Firebase.database().ref('articles').orderByChild('insertDate').limitToLast(10);
this.bindAsArray(ref, 'articles'); // bind retrieved data to this.state.articles
}
render() {
return (
<div>
{
this.state.articles.reverse().map(function(article) {
return <div>{article.title}</div>
})
}
</div>
);
}
}
There is a better way. You should order by negative server timestamp. How to get negative server timestamp even offline? There is an hidden field which helps. Related snippet from documentation:
var offsetRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/.info/serverTimeOffset");
offsetRef.on("value", function(snap) {
var offset = snap.val();
var estimatedServerTimeMs = new Date().getTime() + offset;
});
To add to Dave Vávra's answer, I use a negative timestamp as my sort_key like so
Setting
const timestamp = new Date().getTime();
const data = {
name: 'John Doe',
city: 'New York',
sort_key: timestamp * -1 // Gets the negative value of the timestamp
}
Getting
const ref = firebase.database().ref('business-images').child(id);
const query = ref.orderByChild('sort_key');
return $firebaseArray(query); // AngularFire function
This fetches all objects from newest to oldest. You can also $indexOn the sortKey to make it run even faster
I had this problem too, I found a very simple solution to this that doesn't involved manipulating the data in anyway. If you are rending the result to the DOM, in a list of some sort. You can use flexbox and setup a class to reverse the elements in their container.
.reverse {
display: flex;
flex-direction: column-reverse;
}
myarray.reverse(); or this.myitems = items.map(item => item).reverse();
I did this by prepend.
query.orderByChild('sell').limitToLast(4).on("value", function(snapshot){
snapshot.forEach(function (childSnapshot) {
// PREPEND
});
});
Someone has pointed out that there are 2 ways to do this:
Manipulate the data client-side
Make a query that will order the data
The easiest way that I have found to do this is to use option 1, but through a LinkedList. I just append each of the objects to the front of the stack. It is flexible enough to still allow the list to be used in a ListView or RecyclerView. This way even though they come in order oldest to newest, you can still view, or retrieve, newest to oldest.
You can add a column named orderColumn where you save time as
Long refrenceTime = "large future time";
Long currentTime = "currentTime";
Long order = refrenceTime - currentTime;
now save Long order in column named orderColumn and when you retrieve data
as orderBy(orderColumn) you will get what you need.
just use reverse() on the array , suppose if you are storing the values to an array items[] then do a this.items.reverse()
ref.subscribe(snapshots => {
this.loading.dismiss();
this.items = [];
snapshots.forEach(snapshot => {
this.items.push(snapshot);
});
**this.items.reverse();**
},
For me it was limitToLast that worked. I also found out that limitLast is NOT a function:)
const query = messagesRef.orderBy('createdAt', 'asc').limitToLast(25);
The above is what worked for me.
PRINT in reverse order
Let's think outside the box... If your information will be printed directly into user's screen (without any content that needs to be modified in a consecutive order, like a sum or something), simply print from bottom to top.
So, instead of inserting each new block of content to the end of the print space (A += B), add that block to the beginning (A = B+A).
If you'll include the elements as a consecutive ordered list, the DOM can put the numbers for you if you insert each element as a List Item (<li>) inside an Ordered Lists (<ol>).
This way you save space from your database, avoiding unnecesary reversed data.
I have implemented a gwt incubator table following the example at http://zenoconsulting.wikidot.com/blog:17
this works great however, the order of rows is not same as it is passed using the list object.
the data in this example is set in the following part of the code:
public long showMessages(ArrayList<Message> list) {
long start = System.currentTimeMillis();
// update the count
countLabel.setText("There are "+ list.size() + " messages.");
// reset the table model data
tableModel.setData(list);
// reset the table model row count
tableModel.setRowCount(list.size());
// clear the cache
cachedTableModel.clearCache();
// reset the cached model row count
cachedTableModel.setRowCount(list.size());
// force to page zero with a reload
pagingScrollTable.gotoPage(0, true);
long end = System.currentTimeMillis();
return end - start;
}
Please provide some solution to fix this issue.
awesome! thank you! somehow i ignored this part and internally down the line code was using a hashMap to manage table data which obviously does not retain the sort order. i changed it to linkedHashMap and it worked