No such file or directory (os error 2), stat for rendering public files in oak - ejs

I am trying out Deno (v1.16.4) with Oak (v10) and ejs for the first time. The tailwind file is just a regular CSS file, so no installs. I got stuck serving a static CSS file. I must be overlooking something.
When I go to localhost:8080/tailwind.css, it'll show the file contents, but when going to localhost:8080/admin, it won't show No such file or directory (os error 2), stat '/Users/XXXXXXXX/Documents/projectNameHere/public/admin'. I've tried adjusting root and removing "/public" but now the tailwind file or the html files aren't found.
However, it doesn't seem to be ejs files because removing the first app.use() will display the admin pages (without styles, of course). So those are working.
This is the app.js for app entry.
import { Application } from "https://deno.land/x/oak/mod.ts";
import { route_home } from "./routes/home.js";
import { route_login } from "./routes/login.js";
import { route_admin } from "./routes/admin.js";
import { route_project } from "./routes/projectSingle.js";
import { route_about } from "./routes/about.js";
import { route_404 } from "./routes/404.js";
const app = new Application();
const port = 8080;
app.use(async (ctx) => {
console.log(ctx.request.url.pathname);
await send(ctx, ctx.request.url.pathname, {
root: `${Deno.cwd()}/public`,
});
});
app.use(route_home.routes());
app.use(route_login.routes());
app.use(route_admin.routes());
app.use(route_project.routes());
app.use(route_about.routes());
app.use(route_404.routes());
await app.listen({ port: port });
This is the admin.js for Admin routes.
import { Router } from "https://deno.land/x/oak/mod.ts";
import { renderFile } from "https://deno.land/x/dejs#0.10.2/mod.ts";
const router = new Router();
export const route_admin = router
.get("/admin", async (ctx) => {
console.log(ctx);
const body = renderFile(Deno.cwd() + "/views/adminDashboard.ejs");
ctx.response.body = await body;
})
.get("/admin/project-list", async (ctx) => {
const body = renderFile(Deno.cwd() + "/views/adminProjectsList.ejs");
ctx.response.body = await body;
})
.get("/admin/project/:id", async (ctx) => {
const body = renderFile(Deno.cwd() + "/views/adminProjectSingle.ejs");
ctx.response.body = await body;
})
.get("/admin/edit-other-pages", async (ctx) => {
const body = renderFile(Deno.cwd() + "/views/adminOtherPages.ejs");
ctx.response.body = await body;
})
.get("/admin/user-account", async (ctx) => {
const body = renderFile(Deno.cwd() + "/views/adminUserAccount.ejs");
ctx.response.body = await body;
});
Any thoughts?
Updated Code
app.use(async (ctx, next) => {
try {
const dirPath = `${Deno.cwd()}/public`;
if (existsSync(dirPath)) {
console.log("yes, dir exists");
for await (const file of Deno.readDir(dirPath)) {
console.log("file: ", file);
}
await send(ctx, ctx.request.url.pathname, {
root: `${Deno.cwd()}/public`,
});
}
await next();
} catch (error) {
console.log("error: ", error);
}
});
file structure:
app.js
views
index.ejs
long list of other views continues...
routes
home.js
list of other routes continues...
public
tailwind.css
imagesForHomePage
googleLogo.png
models
pages.js
list of other models continues...
this now gives me the following error:
[uncaught application error]: Http - error from user's HttpBody stream: body write aborted
at async HttpConn.nextRequest (deno:ext/http/01_http.js:67:23)
at async serve (https://deno.land/x/oak#v10.1.0/http_server_native.ts:237:34)
[uncaught application error]: Http - error from user's HttpBody stream: body write aborted
and in the chrome console:
Failed to load resource: the server responded with a status of 404 (Not Found)
Failed to load resource: the server responded with a status of 404 (Not Found)

I'll give it a try!
I think your first middleware will try to serve static files on any request. You need a way to distinguish your static files from the rest of your app. For instance, you could try to put your tailwind find at /public/tailwind.css instead of tailwind.css. Then you can rewrite your middleware this way:
app.use(async (ctx, next) => {
if (ctx.request.url.pathname.startsWith('/public')) {
const filename = ctx.request.url.pathname.substring(7)
await send(ctx, filename, { root: 'public'})
} else {
await next()
}
})
This implementation checks that the file should be served as static, then serves it without the prefix (otherwise it would try to find /public/public/tailwind.css), but other wise you need to call next to pass to the next middleware, here the routers middlewares

Related

What is wrong with my findOne() implementation?

I have followed all basic tutorials in search to fix this. My implementation of collection.findOne({_id: ctx.params.id}) is just not working.
// import statements for version transparancy, they are not really used here
import { MongoClient } from "https://deno.land/x/mongo#v0.29.2/mod.ts";
import { Application, Router } from "https://deno.land/x/oak#v10.4.0/mod.ts";
// ...
const recipes = db.collection<Recipe>("recipes");
// ...
router.get("/rec/:id", async (ctx) => {
const id = ctx.params.id;
const recipe = await recipes.findOne({ _id: { $oid: id } });
// const recipe = await recipes.findOne({_id: id});
if (recipe) {
ctx.response.status = 200;
ctx.response.body = recipe;
} else {
ctx.response.status = 404;
ctx.response.body = { message: "No recipe found" };
}
});
Is there a current bug that I am not aware of?
Once I call that endpoint with http://localhost:5000/rec/622b6be81089abbc4b4144a1 (and yes, that ID is in my database), I get an internal server error (500) with the following message:
[uncaught application error]: Error - MongoError: {"ok":0,"errmsg":"unknown operator: $oid","code":2,"codeName":"BadValue"}
request: {
url: "http://localhost:5000/rec/622b6be81089abbc4b4144a1",
method: "GET",
hasBody: false
}
response: { status: 404, type: undefined, hasBody: false, writable: true }
at WireProtocol.commandSingle (https://deno.land/x/mongo#v0.29.2/src/protocol/protocol.ts:44:13)
at async FindCursor.executor (https://deno.land/x/mongo#v0.29.2/src/collection/commands/find.ts:17:24)
at async FindCursor.execute (https://deno.land/x/mongo#v0.29.2/src/protocol/cursor.ts:34:21)
at async FindCursor.next (https://deno.land/x/mongo#v0.29.2/src/protocol/cursor.ts:48:7)
at async file:///home/andy/dev/deno/denodb/routes.ts:42:20
at async dispatch (https://deno.land/x/oak#v10.4.0/middleware.ts:41:7)
at async dispatch (https://deno.land/x/oak#v10.4.0/middleware.ts:41:7)
at async dispatch (https://deno.land/x/oak#v10.4.0/middleware.ts:41:7)
at async Application.#handleRequest (https://deno.land/x/oak#v10.4.0/application.ts:376:9)
Downgrading the version did not help either.

Unit and integration test of Express REST API and multer single file update middleware

Introduction
Hello everybody,
I'm pretty new to unit and integration testing. The current REST API I'm working on involves file uploads and file system. If you want me to explain what's API this is, I can explain it to you using few sentences. Imagine a system like Microsoft Word. There are only users and users have documents. Users' documents are only JSON files and they are able to upload JSON file to add a document. My API currently has 3 routes, 2 middlewares.
Routes:
auth.js (authorization route)
documents.js (document centered CRUD operations)
users.js
Middlewares:
auth.js (To check if there is valid JSON web token to continue)
uploadFile.js (To upload single file using multer)
I have been able to unit/integration test auth.js, users.js routes and auth.js middleware. These routes and middlewares were only involving small packages of data I/O, so they were pretty easy for me. But documents.js router and uploadFile.js middleware is pretty hard for me to overcome.
Let me share my problems.
Source codes
documents.js Router
.
.
.
router.post('/mine', [auth, uploadFile], async (req, res) => {
const user = await User.findById(req.user._id);
user.leftDiskSpace(function(err, leftSpace) {
if(err) {
return res.status(400).send(createError(err.message, 400));
} else {
if(leftSpace < 0) {
fs.access(req.file.path, (err) => {
if(err) {
res.status(403).send(createError('Your plan\'s disk space is exceeded.', 403));
} else {
fs.unlink(req.file.path, (err) => {
if(err) res.status(500).send('Silinmek istenen doküman diskten silinemedi.');
else res.status(403).send(createError('Your plan\'s disk space is exceeded.', 403));
});
}
});
} else {
let document = new Document({
filename: req.file.filename,
path: `/uploads/${req.user.username}/${req.file.filename}`,
size: req.file.size
});
document.save()
.then((savedDocument) => {
user.documents.push(savedDocument._id);
user.save()
.then(() => res.send(savedDocument));
});
}
}
});
});
.
.
.
uploadFile.js Middleware
const fs = require('fs');
const path = require('path');
const createError = require('./../helpers/createError');
const jsonFileFilter = require('./../helpers/jsonFileFilter');
const multer = require('multer');
const storage = multer.diskStorage({
destination: function(req, file, cb) {
console.log('file: ', file);
if(!req.user.username) return cb(new Error('Dokümanın yükleneceği klasör için isim belirtilmemiş.'), null);
let uploadDestination = path.join(process.cwd(), 'uploads', req.user.username);
fs.access(uploadDestination, (err) => {
if(err) {
// Directory with username doesn't exist in uploads folder, so create one
fs.mkdir(uploadDestination, (err) => {
if(err) cb(err, null);
cb(null, uploadDestination);
});
} else {
// Directory with username exists
cb(null, uploadDestination);
}
});
},
filename: function(req, file, cb) {
cb(null, `${file.originalname.replace('.json', '')}--${Date.now()}.json`);
}
});
module.exports = function(req, res, next) {
multer({ storage: storage, fileFilter: jsonFileFilter }).single('document')(req, res, function(err) {
if(req.fileValidationError) return res.status(400).send(createError(req.fileValidationError.message, 400));
else if(!req.file) return res.status(400).send(createError('Herhangi bir doküman seçilmedi.', 400));
else if(err instanceof multer.MulterError) return res.status(500).send(createError(err.message, 500));
else if(err) return res.status(500).send(createError(err, 500));
else next();
});
}
Questions
1. How can I test user.leftDiskSpace(function(err, leftSpace) { ... }); function which has a callback and contains some Node.js fs methods which also has callbacks?
I want to reach branches and statements user.leftDiskSpace() function containing. I thought of using mock functions to mock out the function but I don't know how to do so.
2. How to change multer disk storage's upload destination for a specified testing folder?
Currently my API uploads the test documents to development/production uploads disk storage destination. What is the best way to change upload destination for testing? I thought to use NODE_ENV global variable to check if the API is being tested or not and change destination in uploadFile.js middleware but I'm not sure if it's a good solution of this problem. What should I do?
Current documents.test.js file
const request = require('supertest');
const { Document } = require('../../../models/document');
const { User } = require('../../../models/user');
const mongoose = require('mongoose');
const path = require('path');
let server;
describe('/api/documents', () => {
beforeEach(() => { server = require('../../../bin/www'); });
afterEach(async () => {
server.close();
await User.deleteMany({});
await Document.deleteMany({});
});
.
.
.
describe('POST /mine', () => {
let user;
let token;
let file;
const exec = async () => {
return await request(server)
.post('/api/documents/mine')
.set('x-auth-token', token)
.attach('document', file);
}
beforeEach(async () => {
user = new User({
username: 'user',
password: '1234'
});
await user.save();
token = user.generateAuthToken();
file = path.join(process.cwd(), 'tests', 'integration', 'files', 'test.json');
});
it('should return 400 if no documents attached', async () => {
file = undefined;
const res = await exec();
expect(res.status).toBe(400);
});
it('should return 400 if a non-JSON document attached', async () => {
file = path.join(process.cwd(), 'tests', 'integration', 'files', 'test.png');
const res = await exec();
expect(res.status).toBe(400);
});
});
});

Flutter Web multipart formdata file upload progress bar

I'm using Flutter web and strapi headless cms for backend. I'm able to send the files successfully, but would like its progress indication. Backend restrictions: File upload must be multipart form-data, being it a buffer or stream. Frontend restrictions: Flutter web doesn't have access to system file directories; files must be loaded in memory and sent using its bytes.
I'm able to upload the file using flutter's http package or the Dio package, but have the following problems when trying to somehow access upload progress:
Http example code:
http.StreamedResponse response;
final uri = Uri.parse(url);
final request = MultipartRequest(
'POST',
uri,
);
request.headers['authorization'] = 'Bearer $_token';
request.files.add(http.MultipartFile.fromBytes(
'files',
_fileToUpload.bytes,
filename: _fileToUpload.name,
));
response = await request.send();
var resStream = await response.stream.bytesToString();
var resData = json.decode(resStream);
What I tryed:
When acessing the response.stream for the onData, it only responds when the server sends the finished request (even though the methods states it's supposed to gets some indications of progress).
Dio package code
Response response = await dio.post(url,
data: formData,
options: Options(
headers: {
'authorization': 'Bearer $_token',
},
), onSendProgress: (int sent, int total) {
setState(() {
pm.progress = (sent / total) * 100;
});
The problems:
It seems the package is able to get some progress indication, but Dio package for flutter web has a bug which has not been fixed: requests block the ui and the app freezes until upload is finished.
Hi you can use the universal_html/html.dart package to do the progress bar, here are steps:
to import universal package
import 'package:universal_html/html.dart' as html;
Select files from html input element instead using file picker packages
_selectFile() {
html.FileUploadInputElement uploadInput = html.FileUploadInputElement();
uploadInput.multiple = false;
uploadInput.accept = '.png,.jpg,.glb';
uploadInput.click();
uploadInput.onChange.listen((e) {
_file = uploadInput.files.first;
});
}
Create upload_worker.js into web folder, my example is upload into S3 post presigned url
self.addEventListener('message', async (event) => {
var file = event.data.file;
var url = event.data.uri;
var postData = event.data.postData;
uploadFile(file, url, postData);
});
function uploadFile(file, url, presignedPostData) {
var xhr = new XMLHttpRequest();
var formData = new FormData();
// if you use postdata, you can open the comment
//Object.keys(presignedPostData).forEach((key) => {
// formData.append(key, presignedPostData[key]);
//});
formData.append('Content-Type', file.type);
// var uploadPercent;
formData.append('file', file);
xhr.upload.addEventListener("progress", function (e) {
if (e.lengthComputable) {
console.log(e.loaded + "/" + e.total);
// pass progress bar status to flutter widget
postMessage(e.loaded/e.total);
}
});
xhr.onreadystatechange = function () {
if (xhr.readyState == XMLHttpRequest.DONE) {
// postMessage("done");
}
}
xhr.onerror = function () {
console.log('Request failed');
// only triggers if the request couldn't be made at all
// postMessage("Request failed");
};
xhr.open('POST', url, true);
xhr.send(formData);
}
Flutter web call upload worker to upload and listener progress bar status
class Upload extends StatefulWidget {
#override
_UploadState createState() => _UploadState();
}
class _UploadState extends State<Upload> {
html.Worker myWorker;
html.File file;
_uploadFile() async {
String _uri = "/upload";
final postData = {};
myWorker.postMessage({"file": file, "uri": _uri, "postData": postData});
}
_selectFile() {
html.InputElement uploadInput = html.FileUploadInputElement();
uploadInput.multiple = false;
uploadInput.click();
uploadInput.onChange.listen((e) {
file = uploadInput.files.first;
});
}
#override
void initState() {
myWorker = new html.Worker('upload_worker.js');
myWorker.onMessage.listen((e) {
setState(() {
//progressbar,...
});
});
super.initState();
}
#override
Widget build(BuildContext context) {
return Column(
children: [
RaisedButton(
onPressed: _selectFile,
child: Text("Select File"),
),
RaisedButton(
onPressed: _uploadFile,
child: Text("Upload"),
),
],
);
}
}
that's it, I hope it can help you.

Using koa-jwt with koa-router

I am implementing the a Nextjs service with koa, koa-router and kow-jwt, but I'm confused with the routing setting with them.
My project have 2 pages, one is dashboard and the other is login. The dashboard need to pass the verification and the login not. If the auth failed, then redirect user to login page.
I've search on the Internet, and found some examples as following, none of them chain them together.
Nextjs custom server
kow-jwt
Please give me some advice to make them work well together.
const app = next({dev});
const handle = app.getRequestHandler();
app.prepare()
.then(() => {
const server = new koa();
const router = new koaRouter();
router.get('/login', async ctx => {
await app.render(ctx.req, ctx.res, '/login', ctx.query);
ctx.respond = false;
});
router.get('/dashboard',
jwt({
secret: config.graphqlSecret
}),
async ctx => {
await app.render(ctx.req, ctx.res, '/dashboard', ctx.query);
ctx.respond = false;
}
);
// what is the purpose of this route?
router.get('*', async ctx => {
await handle(ctx.req, ctx.res);
ctx.respond = false;
});
server.use(async (ctx, next) => {
try {
await next();
} catch (err) {
if (err.statusCode === 401) {
ctx.redirect('/login');
}
}
});
server.use(router.routes());
server.use(router.allowedMethods());
server.listen(3000);
});
with the code above, the behavior is
If I link to dashboard with and without jwt token, it always redirect to login page.
If I link to dashboard from menu (implement with <Link> in Nextjs), it shows the content of dashboard.
Thank you for your help.
You need to include the jwt part in your server.use, not within the router. Make two different routers, one with the open routes and one with the protected ones. Then set open routes, set jwt middleware and then set protected routes:
const app = next({ dev })
const handle = app.getRequestHandler()
app.prepare()
.then(() => {
const server = new Koa()
const router = new Router()
router.get('/login', async ctx => {
await app.render(ctx.req, ctx.res, '/login', ctx.query);
ctx.respond = false;
});
router.get('/dashboard', async ctx => {
await app.render(ctx.req, ctx.res, '/dashboard', ctx.query);
ctx.respond = false;
});
router.get('*', async ctx => {
await handle(ctx.req, ctx.res)
ctx.respond = false
})
// this will keep redirecting user to login until is logged in
// if you remove it, will get an auth error unless you go manually
// to the login path
server.use(async (ctx, next) => {
try {
await next();
} catch (err) {
if (err.statusCode === 401) {
ctx.redirect('/login');
}
}
});
// we need to do it this way because of the way nextjs works with '*' path
// Middleware below this line is only reached if JWT token is valid
server.use(jwt({ secret: 'shared-secret' }).unless({ path: [/^\/b/] }));
// specify in unless the unprotected path
server.use(jwt({secret: config.graphqlSecret}).unless({ path: [/^\/login/] })).use(router.allowedMethods());
// every route protected by default
server.use(router.routes())
server.listen(3000);
})

Puppeteer Generate PDF from multiple HTML strings

I am using Puppeteer to generate PDF files from HTML strings.
Reading the documentation, I found two ways of generating the PDF files:
First, passing an url and call the goto method as follows:
page.goto('https://example.com');
page.pdf({format: 'A4'});
The second one, which is my case, calling the method setContent as follows:
page.setContent('<p>Hello, world!</p>');
page.pdf({format: 'A4'});
The thing is that I have 3 different HTML strings that are sent from the client and I want to generate a single PDF file with 3 pages (in case I have 3 HTML strings).
I wonder if there exists a way of doing this with Puppeteer? I accept other suggestions, but I need to use chrome-headless.
I was able to do this by doing the following:
Generate 3 different PDFs with puppeteer. You have the option of saving the file locally or to store it in a variable.
I saved the files locally, because all the PDF Merge plugins that I found only accept URLs and they don't accept buffers for instance. After generating synchronously the PDFs locally, I merged them using PDF Easy Merge.
The code is like this:
const page1 = '<h1>HTML from page1</h1>';
const page2 = '<h1>HTML from page2</h1>';
const page3 = '<h1>HTML from page3</h1>';
const browser = await puppeteer.launch();
const tab = await browser.newPage();
await tab.setContent(page1);
await tab.pdf({ path: './page1.pdf' });
await tab.setContent(page2);
await tab.pdf({ path: './page2.pdf' });
await tab.setContent(page3);
await tab.pdf({ path: './page3.pdf' });
await browser.close();
pdfMerge([
'./page1.pdf',
'./page2.pdf',
'./page3.pdf',
],
path.join(__dirname, `./mergedFile.pdf`), async (err) => {
if (err) return console.log(err);
console.log('Successfully merged!');
})
I was able to generate multiple PDF from multiple URLs from below code:
package.json
{
............
............
"dependencies": {
"puppeteer": "^1.1.1",
"easy-pdf-merge": "0.1.3"
}
..............
..............
}
index.js
const puppeteer = require('puppeteer');
const merge = require('easy-pdf-merge');
var pdfUrls = ["http://www.google.com","http://www.yahoo.com"];
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
var pdfFiles=[];
for(var i=0; i<pdfUrls.length; i++){
await page.goto(pdfUrls[i], {waitUntil: 'networkidle2'});
var pdfFileName = 'sample'+(i+1)+'.pdf';
pdfFiles.push(pdfFileName);
await page.pdf({path: pdfFileName, format: 'A4'});
}
await browser.close();
await mergeMultiplePDF(pdfFiles);
})();
const mergeMultiplePDF = (pdfFiles) => {
return new Promise((resolve, reject) => {
merge(pdfFiles,'samplefinal.pdf',function(err){
if(err){
console.log(err);
reject(err)
}
console.log('Success');
resolve()
});
});
};
RUN Command: node index.js
pdf-merger-js is another option. page.setContent should work just the same as a drop-in replacement for page.goto below:
const PDFMerger = require("pdf-merger-js"); // 3.4.0
const puppeteer = require("puppeteer"); // 14.1.1
const urls = [
"https://news.ycombinator.com",
"https://en.wikipedia.org",
"https://www.example.com",
// ...
];
const filename = "merged.pdf";
let browser;
(async () => {
browser = await puppeteer.launch();
const [page] = await browser.pages();
const merger = new PDFMerger();
for (const url of urls) {
await page.goto(url);
merger.add(await page.pdf());
}
await merger.save(filename);
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;