I am looking for a way to use headless chrome similar to what chromeless does but instead of being implemented as a nodejs endpoint, allowing restful requests with the html content as a payload.
I want to run this service on aws lambda being triggered through API Gateway. Does anyone have experience with this usecase?
There's nothing keeping you from using Chromeless in your use-case. Chromeless can be used within an AWS Lambda function. You can take a (RESTful) request coming from AWS API Gateway and then do something with it and Chromeless. You can combine the #serverless-chrome/lambda package with Chromeless to get headless Chrome running within Lambda so that Chrome is available to Chromeless. The Chromeless Proxy works in a similar way. For example, your Lambda function's code might look like (this is untested code I just cobbled together, but should convey the idea):
const launchChrome = require('#serverless-chrome/lambda')
const Chromeless = require('chromeless').Chromeless
module.exports.handler = function handler (event, context, callback) {
const body = JSON.parse(event.body) // event.body coming from API Gateway
const url = body.url
const evaluateJs = body.evaluateJs
launchChrome({
flags: ['--window-size=1280x1696', '--hide-scrollbars'],
})
.then((chrome) => {
// Chrome is now running on localhost:9222
const chromeless = new Chromeless({
launchChrome: false,
})
chromeless
.goto(url)
.wait('body')
.evaluate(() => `
// this will be executed in headless chrome
${evaluateJs}
`)
.then((result) => {
chromeless
.end()
.then(chrome.kill) // https://github.com/adieuadieu/serverless-chrome/issues/41#issuecomment-317989508
.then(() => {
callback(null, {
statusCode: 200,
body: JSON.stringify({ result })
})
})
})
.catch(callback)
})
.catch((error) => {
// Chrome didn't launch correctly
callback(error)
})
}
You'll find a similar thread on the Chromeless Issue tracker here.
Disclosure: I'm a collaborator/author of these packages.
Related
Now we can start a react App with bun as a server
Can we use Bunjs as complete backend server?
For Example, Can bun run this code?
const express = require('express')
const app = express()
app.get('/', (req, res) => {
res.send('hello world')
})
app.listen(3000)
I guess Bun does not YET implement all node.js api's. I tried http and it seems currently missing. And as much I understand it currently has its own built-in HTTP server.
Check the "Getting started" section on -> https://bun.sh/
A sample server:
export default {
port: 3000,
fetch(request) {
return new Response("Welcome to Bun!");
},
};
(This example reminds me of serverless functions.)
As this is the case, it seems you can not rely on Node.js http, or most probably any server framework like express.
At least for now, bun's roadmap (https://github.com/oven-sh/bun/issues/159) shows a line, which I am not sure is talking about node's http server or sth. else about Bun's own server.
Once complete, the next step is integration with the HTTP server and
other Bun APIs
Bun api is really different from nodejs, I created a library called bunrest, a express like api, so new user does not need to learn much about bun.
Here is how to use it
Install the package from npm
npm i bunrest
To create a server
const App = require('bunrest');
const server = new App.BunServer();
After that, you can call it like on express
server.get('/test', (req, res) => {
res.status(200).json({ message: 'succeed' });
});
server.put('/test', (req, res) => {
res.status(200).json({ message: 'succeed' });
});
server.post('/test', (req, res) => {
res.status(200).json({ message: 'succeed' });
});
To start the server
server.listen(3000, () => {
console.log('App is listening on port 3000');
});
Other way is using Hono: https://honojs.dev/
There is a working demo: https://github.com/cachac/bun-api
Import package and set new instance
import { Hono } from 'hono'
const app = new Hono()
Create routes:
Instead of NodeJs res.send(), use c.json({})
app.get('/hello', c => c.json({ message: 'Hello World' }))
export and run api server:
export default {
fetch: app.fetch,
port: 3000
}
I have a PWA project where I send the data to server. During this process, if the user is offline then the data is stored in indexedDb and a sync tag is registered. So, then when the user comes online that data can sent to the server.
But In my case the sync event gets executed immediately when the we register a sync event tag, which means the data is tried to be sent to server while its offline, which is not going to work.
I think the sync event supposed to fire while its online only, what could be issue here ?
The service worker's sync event works accordingly when I tried to enable and disable the offline option of chrome devtools, and also works correctly in my android phone.
This is how I register my sync tag
function onFailure() {
var form = document.querySelector("form");
//Register the sync on post form error
if ('serviceWorker' in navigator && 'SyncManager' in window) {
navigator.serviceWorker.ready
.then(function (sw) {
var post = {
datetime1: form.datetime1.value,
datetime: form.datetime.value,
name: form.name.value,
image: form.url.value,
message: form.comment.value
};
writeData('sync-comments', post)
.then(function () {
return sw.sync.register('sync-new-comment');
})
.then(function () {
console.log("[Sync tag registered]");
})
.catch(function (err) {
console.log(err);
});
});
}
}
And this is how the sync event is called
self.addEventListener('sync', function (event) {
console.log("[Service worker] Sync new comment", event);
if (event.tag === 'sync-new-comment') {
event.waitUntil(
readAllData('sync-comments')
.then(function (data) {
setTimeout(() => {
data.forEach(async (dt) => {
const url = "/api/post_data/post_new_comment";
const parameters = {
method: 'POST',
headers: {
'Content-Type': "application/json",
'Accept': 'application/json'
},
body: JSON.stringify({
datetime: dt.datetime,
name: dt.name,
url: dt.image,
comment: dt.message,
datetime1: dt.datetime1,
})
};
fetch(url, parameters)
.then((res) => {
return res.json();
})
.then(response => {
if (response && response.datetimeid) deleteItemFromData('sync-comments', response.datetimeid);
}).catch((error) => {
console.log('[error post message]', error.message);
})
})
}, 5000);
})
);
}
});
you mention
The service worker's sync event works accordingly when I tried to enable and disable the offline option of chrome devtools, and also works correctly in my android phone.
So I'm not sure which case is the one failing.
You are right that the sync will be triggered when the browser thinks the user is online, if the browser detects that the user is online at the time of the sync registration it will trigger the sync:
In true extensible web style, this is a low level feature that gives you the freedom to do what you need. You ask for an event to be fired when the user has connectivity, which is immediate if the user already has connectivity. Then, you listen for that event and do whatever you need to do.
Also, from the workbox documentation
Browsers that support the BackgroundSync API will automatically replay failed requests on your behalf at an interval managed by the browser, likely using exponential backoff between replay attempts.
I'm struggling to perform a test with jest concerning an axios api call
here is my API call, that works perfectly within my program
import axios from 'axios';
import crypto from 'crypto';
import { prop } from 'ramda';
const baseUrl = 'http://gateway.marvel.com:80';
const uri = '/v1/public/characters';
const charactersUrl = baseUrl + uri;
const timestamp = [Math.round(+new Date() / 1000)];
const privateApi = 'XXX';
const publicApi = 'XXX';
const concatenatedString = timestamp.concat(privateApi, publicApi).join('');
const hash = crypto.createHash('md5').update(`${concatenatedString}`).digest('hex');
const charactersApi = () =>
axios
.get(charactersUrl, {
params: {
ts: timestamp,
apikey: publicApi,
hash,
},
})
.then(prop('data'));
export default charactersApi;
When I'm trying to test it, that way:
import axiosMock from 'axios';
import charactersApi from '../marvelApi';
jest.mock('axios', () => ({
get: jest.fn(),
}));
describe('tools | marvelApi', () => {
const piece = { name: '3D-MAN' };
axiosMock.get.mockResolvedValueOnce({ data: piece });
it('should get the character', () => {
return charactersApi().then(elem => {
expect(elem.name).toEqual('3D-MAN');
});
});
});
I get the following message from jest
TypeError: Cannot read property 'then' of undefined
16 |
17 | const charactersApi = () =>
> 18 | axios
| ^
19 | .get(charactersUrl, {
20 | params: {
21 | ts: timestamp,
at charactersApi (src/tools/marvelApi.js:18:3)
at Object.<anonymous> (src/tools/tests/marvelApi.test.js:13:12)
What I have tried
A common error is to forget the return statement within the function that contain the request API, in my case it's done correctly (first piece of code -> charactersApi()) source1, source2
I also tried to return a Promise from the mocked Axios as I have seen on another SO ticket
jest.mock('axios', () => ({
get: jest.fn(() => Promise.resolve()),
}));
I think my axios mock is not correct, because the struggle comes from the test while the production version work well
Any thoughts ?
You can spy on the "axios.get" calls and resolve them to a fixed (mocked) value:
/**
* #jest-environment jsdom
*/
const axios = require('axios')
beforeAll(() => {
jest.spyOn(axios, 'get').mockImplementation()
})
afterAll(() => {
jest.restoreAllMocks()
})
it('returns the mocked response', async () => {
axios.get.mockResolvedValue({ data: 'foo' })
const res = await axios.get('https://api.github.com')
expect(res).toEqual({ data: 'foo' })
})
You shouldn't use jest.mock because it mocks a module that your imported code may be using. As far as I know, it doesn't affect the current module's imports (and you import axios as a part of your test).
Recommended solution
I strongly discourage you from spying/mocking axios directly. See my argumentation below.
You're mocking implementation details of axios. In other words, you take the axios.get function and throw it away, alongside any internal logic it may have, and replace it with a hard mock. This means your test no longer uses axios, instead it uses an emptied mocked shell of axios. This makes your test different from your actual code, which, in turn, decreases the confidence such a test gives you.
You're coupling your mocks with a specific request client (axios). Such an approach is not a long-term investment, as you're writing axios-specific mocks. You can't reuse such mocks for requests made by other clients (i.e. window.fetch, Apollo, etc.), because they have their own implementation details (i.e. window.fetch has no .get() to spy on), which only encourages you to write more implementation-specific logic in tests.
You can learn more about the disadvantages of direct mocking of request clients in the Stop mocking fetch article by Kent C. Dodds. It uses window.fetch mocks as an example, but you may replace it with ANY_REQUEST_CLIENT when reading.
I highly recommend using tools like Mock Service Worker (MSW) that will encourage you to write abstracted mocks that don't rely on any request clients (you can use them no matter how your tested code makes a request) and can even be reused across different testing levels (the same mocks for Jest, Storybook, or Cypress).
Here's how your test would look like with MSW:
import { rest } from 'msw'
import { setupServer } from 'msw/node'
import charactersApi from '../marvelApi';
const server = setupServer(
rest.get('http://gateway.marvel.com:80/v1/public/characters', (req, res, ctx) => {
return res(ctx.json({
data: {
name: '3D-MAN'
}
}))
})
)
beforeAll(() => server.listen()
afterAll(() => server.close())
describe('tools | marvelApi', () => {
it('should get the character', () => {
return charactersApi().then(elem => {
expect(elem.name).toEqual('3D-MAN')
})
})
})
Notice how there are no details about how the request is made, only which request to intercept and mock its response.
You can follow a detailed tutorial on how to Get started with MSW. There's also a great video on API mocking and what problems MSW solves.
I created a toy mern app which can upload images to mongodb. It works fine in the development but after I deploy to heroku, I get a timeout error. After doing a fair amount of investigations I come to the conclusion that web workers will help sort this out. So I've done this:
This is a snippet from the component that takes care of sending this request, to upload an image:
const handleUploadPhoto = evt => {
evt.preventDefault();
const worker = new Worker('imageUpload.js');
worker.postMessage(selectedPhotos);
worker.onmessage = evt => console.log(evt.data);
closeModal();
};
This whole component is actually a modal, and after I submit this handleUploadPhoto function the modal closes. Notice the closeModal function;
In the same folder as this component I have the imageUpload.js worker file:
self.addEventListener('message', evt => {
const { selectedPhotos } = evt.data;
selectedPhotos.map(photo => {
const formData = new FormData();
formData.append('photo', photo);
fetch( `/api/photo/upload/${JSON.parse(localStorage.getItem('loggedUser'))._id}`, {
method: 'POST',
body: formData,
headers: ['Content-Type', 'multipart/form-data'],
}).then(() => console.log('OK'));
});
self.postMessage(evt.data);
});
The idea behind this functionality is that I select one or more images, then I map through them and send a post request to the server with each image as formData. So I thought to move the request to a worker so that it doesn't break my UI anymore. Needless to say that it doesn't work and it gives net::EMPTY_RESPONSE error after some seconds. What I also noticed is that the worker doesn't communicate with the file it was issued in at all. I tried to make a text message pop in the console:
const handleUploadPhoto = evt => {
evt.preventDefault();
const worker = new Worker('imageUpload.js');
worker.postMessage('Hello');
worker.onmessage = evt => console.log(evt.data);
closeModal();
};
In the same folder as this component I have the imageUpload.js worker file:
self.addEventListener('message', evt => {
self.postMessage(evt.data + ' world');
});
It doesn't work. I tried the same approach in a separate project and it worked.
How can I make an HTTP request do what it's supposed to do from a web worker?
Tests hang in the testcafe browser when you try to test a rest api url directly.
I am trying to run a test against my rest API endpoint using request hooks, but when I run the test from the command line, the browser opens the API endpoint and loads it and hangs. The test doesn't pass or fail and hangs. The rest API endpoint just returns a JSON response.
const logger = RequestLogger('https://example.com/search/suggestions?search=testkeyword');
fixture `test`
.page('https://example.com/search/suggestions?search=testkeyword');
test
.requestHooks(logger)
('test', async t => {
// Ensure that the response has been received and that its status code is 200.
await t.expect(logger.contains(record => record.response.statusCode === 200)).ok();
const logRecord = logger.requests[0];
console.log(logRecord.userAgent);
console.log(logRecord.request.url);
console.log(logRecord.request.method);
console.log(logRecord.response.statusCode);
});
I expect the test to pass checking for 200 status code, but the test hangs without showing pass/fail. Does testcafe support testing of rest API endpoints? I have checked this issue - https://github.com/DevExpress/testcafe/issues/1471 where it says testcafe doesn't support non-html pages. Please confirm.
You are right, TestCafe is intended to work with html pages, but it will use the "about:blank" page if you don't define any url. You can use the regular node.js HTTP API without request hooks for this case. Look at the following example:
import http from 'http';
const getResponseData = (url) => new Promise((resolve, reject) => {
http.get(url, res => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => resolve({ statusCode, contentType, rawData }));
}).on('error', e => reject(e));
});
fixture `Test REST API`;
test('Simple Test', async t => {
const response = await getResponseData('http://example.com/api/1');
await t
.expect(response.statusCode).eql(200);
});
TestCafe 1.20.0+ offers the t.request method. You can use it for REST API testing. In other words, you can incorporate API testing right into your existing functional TestCafe tests. You no longer need to use any third-party libraries.
You can read about the API testing feature in the corresponding guide.