Testing Cloud Function - Cannot read property 'data' of undefined - raspberry-pi

Question

I've got a Raspberry Pi and setup a weather (and soil moisture) rig.
I found this guide: https://codelabs.developers.google.com/codelabs/iot-data-pipeline which I followed and got stuck around Step 6-7.
From what I understand - when I send data to PubSub - nothing happens. On the Raspberry End I sort of get the idea that data is being sent but it doesn't get passed into BigQuery. I did some print statements at various points to try and see where it got stuck.
As I was trying to find the error I slowly backtracked to Step 5 (Create a Cloud Function).
Step 5 along with associated code I copied can be seen here: https://codelabs.developers.google.com/codelabs/iot-data-pipeline/#4
In GCP - I click into Cloud Function -> function-weatherPubSubToBQ -> Testing (tab)
Under the heading - Trigger event - I filled out the JSON below:
{
"sensorID":"Raspberry",
"timecollected":"2020-09-11 06:45:19",
"zipcode":"00000",
"latitude":"0.0",
"longitude":"0.0",
"temperature":"-273",
"humidity":"-1",
"dewpoint":"-273",
"pressure":"0"
}
When I click on - Test the function - the output is as below
**Error: function execution failed. Details:
Cannot read property 'data' of undefined**
Screen capture of JSON and error message
I am guessing one of these two things are causing the problem.
event.data or PubSubMessage.data
I tried to make some changes to the code but I am just shooting in the dark.
I was wondering if:
I did something wrong which means there might be some other
issues somewhere else.
This guide is slightly old and there have
been some updates which make the older code in the guide not
function as desired. (not step/image in the guide matches with what
I saw online, as of Sep 2020)
If someone knows what is wrong in
the code and is able to let me know how to solve it that would be
much appreciated.
Thanks in advance.

Answer 1

TLDR: The tutorial is outdated, don't use it, unless you want to face multiple problems and want to learn in the hard way
I went through the tutorial and I was able to replicate the issue.. and many more. As you already mentioned it, the tutorial is outdated and many things have changed, you can infer that by looking at the images and noticing that the UI is even different, so I wouldn't recommend this tutorial to anyone who is new to GCP.
The first issue:
**Error: function execution failed. Details: Cannot read property 'data' of undefined**
Can be easily resolved by looking at the structure of what it's expected from a pub/sub message:
{
"data": string,
"attributes": {
string: string,
...
},
"messageId": string,
"publishTime": string,
"orderingKey": string
}
So easy right? However once you mimic the structure of the message with your own variables as I did:
{
"data": "",
"attributes": {
"sensorID":"Raspberry",
"timecollected":"2020-09-11 06:45:19",
"zipcode":"00000",
"latitude":"0.0",
"longitude":"0.0",
"temperature":"-273",
"humidity":"-1",
"dewpoint":"-273",
"pressure":"0"
},
"messageId": "id_1",
"publishTime": "2014-10-02T15:01:23Z",
"orderingKey": ""
}
You will get an error regarding the JSON:
SyntaxError: Unexpected token ' in JSON at position 1
This error is due to the use of ' on the construction of the JSON inside the variable incomingData so you have to change the first variable declaration, I did it by using template literals:
const incomingData = PubSubMessage.data ? Buffer.from(PubSubMessage.data, 'base64').toString() : `{"sensorID": "na","timecollected":"01/01/1970 00:00:00","zipcode":"00000","latitude":"0.0","longitude":"0.0","temperature":"-273","humidity":"-1","dewpoint":"-273","pressure":"0"}`;
But this is not the end of the issues, after doing some tests while trying to insert into BigQuery, I got an error regarding the insertion, but didn't get a clue of what was really happening, so I isolated the consult in an external script and found that the error handling was wrong, the first thing I recommend you to change is the BigQuery version in the package.json from:
"#google-cloud/bigquery": "^0.9.6"
Into
"#google-cloud/bigquery": "5.2.0"
Which is the last version at the time of writing this answer. The next part is to redefine the way you're using BigQuery constructor into:
const bigquery = new BigQuery({
projectId: projectId
});
Then after many tests I found that the catch wasn't doing it's job as expected, so have to rewrite that part into:
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows)
.then((foundErrors) => {
rows.forEach((row) => console.log('Inserted: ', row));
if (foundErrors && foundErrors.insertErrors != undefined) {
foundErrors.forEach((err) => {
console.log('Error: ', err);
})
}
})
.catch((err) => {
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows)
.then((foundErrors) => {
rows.forEach((row) => console.log('Inserted: ', row));
if (foundErrors && foundErrors.insertErrors != undefined) {
foundErrors.forEach((err) => {
console.log('Error: ', err);
})
}
})
.catch((err) => {
if(err.name=='PartialFailureError'){
if (err && err.response.insertErrors != undefined) {
err.response.insertErrors.forEach((errors) => {
console.log(errors);
})
}
}else{
console.log("GENERIC ERROR:",err)
}
});
});
After this you will finally notice that the error is due to (again) the incomingData variable:
Could not parse \'01/01/1970 00:00:00\' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]'
You have to change the date from 01/01/1970 00:00:00 into 1970-01-01 00:00:00.
Still here with me? Well there's another error coming from the usage of callback at the end of the CF:
This is due to the fact that Cloud Functions now require three parameters and callback is the last one, so change the function declaration into:
exports.subscribe = function (event, context, callback)
After all of this you have been able to insert data into BigQuery, however we're using the local variable and not the data coming from pub/sub, at this point I gave up, since I will need to practically rewrite the full function in order to make it work by using Attributes instead of the data.
So as mentioned it before don't follow this tutorial if you're beginning in the GCP world.