While trying to use the NodeJS Client Library (getFiles -
https://googleapis.dev/nodejs/storage/latest/Bucket.html#getFiles)
and the JSON API (objects/list - https://cloud.google.com/storage/docs/json_api/v1/objects/list) The response seems to not return certain folders in both cases.
While using gsutil ls it lists all the folders as expected and using the Google Cloud Console to browse the GCS bucket also lists all files as expected.
Has there been change in the Storage API that could be causing this to happen?
UPDATE:
The code to list GCS Folder paths, that uses the NodeJS Client SDK for Storage:
import * as functions from 'firebase-functions';
import { Storage } from '#google-cloud/storage';
import { globVars } from '../admin/admin';
const projectId = process.env.GCLOUD_PROJECT;
// shared global variables setup
const { keyFilename } = globVars;
// Storage set up
const storage = new Storage({
projectId,
keyFilename,
});
export const gcsListPath = functions
.region('europe-west2')
.runWith({ timeoutSeconds: 540, memory: '256MB' })
.https.onCall(async (data, context) => {
if (context.auth?.token.email_verified) {
const { bucketName, prefix, pathList = false, fileList = false } = data;
let list;
const options = {
autoPaginate: false,
delimiter: '',
prefix,
};
if (pathList) {
options.delimiter = '/';
const verboseResponse = await storage
.bucket(bucketName)
.getFiles(options);
list = verboseResponse[2].prefixes;
}
if (fileList) {
const [files] = await storage
.bucket(bucketName)
.getFiles(options);
list = files.map((file) => file.name);
}
return { list };
} else {
return {
error: { message: 'Bad Request', status: 'INVALID_ARGUMENT' },
};
}
});
Looks like the pagination feature was limiting the results. Implementing nextQuery
returns the next page of listings.
Related
I currently have the following code, which saves the temp file to public/files I have tried to understand the MongoDB GridFS documentation but with no success.
I am wondering how do I get the files to save inside MongoDB GridFS instead of my public/file directory
I am aware I am missing the part where I need to send the uploaded file to mongodb - this is the part I don't know how to do.
In mongodb example they say to do something like:
fs.createReadStream('./myFile').pipe(
bucket.openUploadStream('myFile', {
chunkSizeBytes: 1048576,
metadata: { field: 'myField', value: 'myValue' },
})
);
however I am not using FS or do I need to upload the file to the temp and then do the fs
import formidable from 'formidable';
import { MongoClient, ObjectId } from 'mongodb';
var Grid = require('gridfs-stream');
export const config = {
api: {
bodyParser: false,
},
};
export default async (req, res) => {
const uri = process.env.MONGODB_URI;
let client;
let clientPromise;
const options = {};
client = new MongoClient(uri, options);
clientPromise = client.connect();
const clients = await clientPromise;
const database = clients.db('AdStitchr');
var gfs = Grid(database, client);
gfs.collection('uploads');
const form = new formidable.IncomingForm();
form.uploadDir = 'public/files';
form.keepExtensions = true;
form.parse(req, (err, fields, files) => {
var file = files.file;
console.log(JSON.stringify(file));
try {
const newFile = File.create({
name: `files\${file.newFilename}.mp3`,
});
res.status(200).json({ status: 'success' });
} catch (error) {
res.send(error);
}
});
};
I have connected MongoDB to my discord.js code and have made a setwelcome command as per-server data so that each server can customize their own welcome message. Everything works great, I just want to know if there is any way that I can make the message appear as an embed? Here's the code:
//importing all the needed files and languages
const mongo = require('./mongo')
const command = require('./command')
const welcomeSchema = require('./schemas/welcome-schema')
const mongoose = require('mongoose')
const Discord = require('discord.js')
mongoose.set('useFindAndModify', false);
//my code is inside this export
module.exports = (client) => {
//this next line is for later
const cache = {}
command(client, 'setwelcome', async (message) => {
const { member, channel, content, guild } = message
//checking to see that only admins can do this
if (!member.hasPermissions === 'ADMINISTRATOR') {
channel.send('You do not have the permission to run this command')
return
}
//simplifying commands
let text = content
//this is to store just the command and not the prefix in mongo compass
const split = text.split(' ')
if (split.length < 2) {
channel.send('Please provide a welcome message!')
return
}
split.shift()
text = split.join(' ')
//this is to not fetch from the database after code ran once
cache[guild.id] = [channel.id, text]
//this is to store the code inside mongo compass
await mongo().then(async (mongoose) => {
try {
await welcomeSchema.findOneAndUpdate({
_id: guild.id
}, {
_id: guild.id,
channelId: channel.id,
text,
}, {
upsert: true
})
} finally {
mongoose.connection.close()
}
})
})
//this is to fetch from the database
const onJoin = async (member) => {
const { guild } = member
let data = cache[guild.id]
if (!data) {
console.log('FETCHING FROM DATABASE')
await mongo().then( async (mongoose) => {
try {
const result = await welcomeSchema.findOne({ _id: guild.id })
cache[guild.id] = data = [result.channelId, result.text]
} finally {
mongoose.connection.close()
}
})
}
//this is to simplify into variables
const channelId = data[0]
const text = data[1]
/*this is where the message sends on discord. the second of these 2 lines is what I want embedded
which is basically the welcome message itself*/
const channel = guild.channels.cache.get(channelId)
channel.send(text.replace(/<#>/g, `<#${member.id}>`))
}
//this is to test the command
command(client, 'simjoin', message => {
onJoin(message.member)
})
//this is so the command works when someone joins
client.on('guildMemberAdd', member => {
onJoin(member)
})
}
I know how to usually make an embed, but I'm just confused at the moment on what to put as .setDescription() for the embed.
Please advise.
If you just want to have the message be sent as an embed, create a MessageEmbed and use setDescription() with the description as the only argument. Then send it with channel.send(embed).
const embed = new Discord.MessageEmbed();
embed.setDescription(text.replace(/<#>/g, `<#${member.id}>`));
channel.send(embed);
By the way, if you are confused about how to use a specific method you can always search for the method name on the official discord.js documentation so you don’t have to wait for an answer here. Good luck creating your bot!
I discovered that by default the API responds with 30 repos.
May I know how should I use for loop to retrieve all repos?
const axios = require('axios');
const repoUrl = `https://api.github.com/users/USERNAME/repos`;
access_token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
// console.log
const config = {
headers: {Authorization: `Bearer ${access_token}`} # only 30
};
axios.get(repoUrl, null, config).then((responses) => {
const repos = responses.data.map(({name, language, html_url, created_at, description}) => {
return {name, language, html_url, created_at, description};
})
console.log("number of repo ", repos.length);
}).catch(error => {
console.log(`getrepos error: ${error}`)
});
by https://docs.github.com/en/rest/reference/repos#list-repositories-for-the-authenticated-user--parameters , there is a page parameter
but how should I write the for loop? should I loop the page with 1,2,3,4.... until server give me some error?
You can make a GraphQL query with Axios, as in this example, or in this article:
axios({
url: 'https://graphql.com/graphql',
method: 'post',
data: {
query: `
query {
viewer {
repositories(isFork: false) {
totalCount
}
}
}`
}
}).then((result) => {
console.log(result.data)
});
Once you have the total number of repositories, you start looping (only the exact amount of loop), as described here:
//Start fetching every page of repos.
const fetchPromises = [], pageCount = Math.ceil(repoCount /
MAX_PER_PAGE);
for (let pageI = 1; pageI <= pageCount; ++pageI) {
const fetchPagePromise = fetch(baseUrl + '&page=' + pageI);
fetchPromises.push(fetchPagePromise);
}
I am new to Next.js and using this example from Next.js https://github.com/zeit/next.js/tree/master/examples/api-routes-apollo-server-and-client.
However, the example is silent on MongoDB integration (also I could not find any other example for the same). I have been able to make database-connection but NOT able to use it in resolvers.
My Code
pages/api/graphql.js
import { ApolloServer } from 'apollo-server-micro'
import { schema } from '../../apollo/schema'
const MongoClient = require('mongodb').MongoClient;
let db
const apolloServer = new ApolloServer({
schema,
context: async () => {
if (!db) {
try {
const client = await MongoClient.connect(uri)
db = await client.db('dbName')
const post = await Posts.findOne()
console.log(post)
// It's working fine here
}
catch (e) {
// handle any errors
}
}
return { db }
},
})
export const config = {
api: {
bodyParser: false,
},
}
export default apolloServer.createHandler({ path: '/api/graphql' })
apollo/schema.js
import {makeExecutableSchema} from 'graphql-tools';
import {typeDefs} from './type-defs';
import {resolvers} from './resolvers';
export const schema = makeExecutableSchema({
typeDefs,
resolvers
});
apollo/resolvers.js
const Items = require('./connector').Items;
export const resolvers = {
Query: {
viewer(_parent, _args, _context, _info) {
//want to populate values here, using database connection
return { id: 1, name: 'John Smith', status: 'cached' }
},
...
}
}
I am stuck in the resolvers.js part. Don't know how to get the cached database connection inside resolvers.js. If I create a new database connection file, top-level await is not supported there, so how do I proceed?
If context is a function, whatever you return from the function will be available as the context parameter in your resolver. So if you're returning { db }, that's what your context parameter will be -- in other words, you can access it as context.db inside your resolver.
I have my REST API where I put my pdf file, now I want my angular app to download it on click via my web browser but I got HttpErrorResponse
"Unexpected token % in JSON at position 0"
"SyntaxError: Unexpected token % in JSON at position 0↵ at JSON.parse (
this is my endpoint
#GetMapping("/help/pdf2")
public ResponseEntity<InputStreamResource> getPdf2(){
Resource resource = new ClassPathResource("/pdf-sample.pdf");
long r = 0;
InputStream is=null;
try {
is = resource.getInputStream();
r = resource.contentLength();
} catch (IOException e) {
e.printStackTrace();
}
return ResponseEntity.ok().contentLength(r)
.contentType(MediaType.parseMediaType("application/pdf"))
.body(new InputStreamResource(is));
}
this is my service
getPdf() {
this.authKey = localStorage.getItem('jwt_token');
const httpOptions = {
headers: new HttpHeaders({
'Content-Type': 'application/pdf',
'Authorization' : this.authKey,
responseType : 'blob',
Accept : 'application/pdf',
observe : 'response'
})
};
return this.http
.get("http://localhost:9989/api/download/help/pdf2", httpOptions);
}
and invocation
this.downloadService.getPdf()
.subscribe((resultBlob: Blob) => {
var downloadURL = URL.createObjectURL(resultBlob);
window.open(downloadURL);});
I resolved it as follows:
// header.component.ts
this.downloadService.getPdf().subscribe((data) => {
this.blob = new Blob([data], {type: 'application/pdf'});
var downloadURL = window.URL.createObjectURL(data);
var link = document.createElement('a');
link.href = downloadURL;
link.download = "help.pdf";
link.click();
});
//download.service.ts
getPdf() {
const httpOptions = {
responseType: 'blob' as 'json')
};
return this.http.get(`${this.BASE_URL}/help/pdf`, httpOptions);
}
I solved the issue in this way (please note that I have merged multiple solutions found on stack overflow, but I cannot find the references. Feel free to add them in the comments).
In My service I have:
public getPDF(): Observable<Blob> {
//const options = { responseType: 'blob' }; there is no use of this
let uri = '/my/uri';
// this.http refers to HttpClient. Note here that you cannot use the generic get<Blob> as it does not compile: instead you "choose" the appropriate API in this way.
return this.http.get(uri, { responseType: 'blob' });
}
In the component, I have (this is the part merged from multiple answers):
public showPDF(fileName: string): void {
this.myService.getPDF()
.subscribe(x => {
// It is necessary to create a new blob object with mime-type explicitly set
// otherwise only Chrome works like it should
var newBlob = new Blob([x], { type: "application/pdf" });
// IE doesn't allow using a blob object directly as link href
// instead it is necessary to use msSaveOrOpenBlob
if (window.navigator && window.navigator.msSaveOrOpenBlob) {
window.navigator.msSaveOrOpenBlob(newBlob, fileName);
return;
}
// For other browsers:
// Create a link pointing to the ObjectURL containing the blob.
const data = window.URL.createObjectURL(newBlob);
var link = document.createElement('a');
link.href = data;
link.download = fileName;
// this is necessary as link.click() does not work on the latest firefox
link.dispatchEvent(new MouseEvent('click', { bubbles: true, cancelable: true, view: window }));
setTimeout(function () {
// For Firefox it is necessary to delay revoking the ObjectURL
window.URL.revokeObjectURL(data);
link.remove();
}, 100);
});
}
The code above works in IE, Edge, Chrome and Firefox. However, I don't really like it, as my component is pulluted with browser specific stuff which will surely change over time.
For Angular 12+, I came up with something like this:
this.ApiService
.getFileFromApi()
.pipe(take(1))
.subscribe((response) => {
const downloadLink = document.createElement('a');
downloadLink.href = URL.createObjectURL(new Blob([response.body], { type: response.body.type }));
const contentDisposition = response.headers.get('content-disposition');
const fileName = contentDisposition.split(';')[1].split('filename')[1].split('=')[1].trim();
downloadLink.download = fileName;
downloadLink.click();
});
The subscribe is on a simple get() with the Angular HttpClient.
// api-service.ts
getFileFromApi(url: string): Observable<HttpResponse<Blob>> {
return this.httpClient.get<Blob>(this.baseApiUrl + url, { observe: 'response', responseType: 'blob' as 'json'});
}
You can do it with angular directives:
#Directive({
selector: '[downloadInvoice]',
exportAs: 'downloadInvoice',
})
export class DownloadInvoiceDirective implements OnDestroy {
#Input() orderNumber: string;
private destroy$: Subject<void> = new Subject<void>();
_loading = false;
constructor(private ref: ElementRef, private api: Api) {}
#HostListener('click')
onClick(): void {
this._loading = true;
this.api.downloadInvoice(this.orderNumber)
.pipe(
takeUntil(this.destroy$),
map(response => new Blob([response], { type: 'application/pdf' })),
)
.subscribe((pdf: Blob) => {
this.ref.nativeElement.href = window.URL.createObjectURL(pdf);
this.ref.nativeElement.click();
});
}
// your loading custom class
#HostBinding('class.btn-loading') get loading() {
return this._loading;
}
ngOnDestroy(): void {
this.destroy$.next();
this.destroy$.complete();
}
}
In the template:
<a
downloadInvoice
[orderNumber]="order.number"
class="btn-show-invoice"
>
Show invoice
</a>
My answer is based on #Yennefer's, but I wanted to use the file name from the server since I didn't have it in my FE. I used the Content-Disposition header to transmit this, since that is what the browser uses for a direct download.
First, I needed access to the headers from the request (notice the get method options object):
public getFile(): Observable<HttpResponse<Blob>> {
let uri = '/my/uri';
return this.http.get(uri, { responseType: 'blob', observe: 'response' });
}
Next, I needed to extract the file name from the header.
public getFileName(res: HttpResponse<any>): string {
const disposition = res.headers.get('Content-Disposition');
if (!disposition) {
// either the disposition was not sent, or is not accessible
// (see CORS Access-Control-Expose-Headers)
return null;
}
const utf8FilenameRegex = /filename\*=UTF-8''([\w%\-\.]+)(?:; |$)/;
const asciiFilenameRegex = /filename=(["'])(.*?[^\\])\1(?:; |$)/;
let fileName: string = null;
if (utf8FilenameRegex.test(disposition)) {
fileName = decodeURIComponent(utf8FilenameRegex.exec(disposition)[1]);
} else {
const matches = asciiFilenameRegex.exec(disposition);
if (matches != null && matches[2]) {
fileName = matches[2];
}
}
return fileName;
}
This method checks for both ascii and utf-8 encoded file names, prefering utf-8.
Once I have the file name, I can update the download property of the link object (in #Yennifer's answer, that's the lines link.download = 'FileName.ext' and window.navigator.msSaveOrOpenBlob(newBlob, 'FileName.ext');)
A couple of notes on this code:
Content-Disposition is not in the default CORS whitelist, so it may not be accessible from the response object based on the your server's configuration. If this is the case, in the response server, set the header Access-Control-Expose-Headers to include Content-Disposition.
Some browsers will further clean up file names. My version of chrome seems to replace : and " with underscores. I'm sure there are others but that's out of scope.
//Step: 1
//Base Service
this.getPDF() {
return this.http.get(environment.baseUrl + apiUrl, {
responseType: 'blob',
headers: new HttpHeaders({
'Access-Control-Allow-Origin': '*',
'Authorization': localStorage.getItem('AccessToken') || ''
})
});
}
//Step: 2
//downloadService
getReceipt() {
return new Promise((resolve, reject) => {
try {
// {
const apiName = 'js/getReceipt/type/10/id/2';
this.getPDF(apiName).subscribe((data) => {
if (data !== null && data !== undefined) {
resolve(data);
} else {
reject();
}
}, (error) => {
console.log('ERROR STATUS', error.status);
reject(error);
});
} catch (error) {
reject(error);
}
});
}
//Step 3:
//Component
getReceipt().subscribe((respect: any) => {
var downloadURL = window.URL.createObjectURL(data);
var link = document.createElement(‘a’);
link.href = downloadURL;
link.download = “sample.pdf";
link.click();
});
This also works in IE and Chrome, almost the same answer only for other browsers the answer is a bit shorter.
getPdf(url: string): void {
this.invoiceService.getPdf(url).subscribe(response => {
// It is necessary to create a new blob object with mime-type explicitly set
// otherwise only Chrome works like it should
const newBlob = new Blob([(response)], { type: 'application/pdf' });
// IE doesn't allow using a blob object directly as link href
// instead it is necessary to use msSaveOrOpenBlob
if (window.navigator && window.navigator.msSaveOrOpenBlob) {
window.navigator.msSaveOrOpenBlob(newBlob);
return;
}
// For other browsers:
// Create a link pointing to the ObjectURL containing the blob.
const downloadURL = URL.createObjectURL(newBlob);
window.open(downloadURL);
});
}