ReferenceError: TextEncoder is not defined in Github Actions Jest Script - mongodb

I have an error that's only happening in my Github Actions workflow (when I run my Jest script locally, it's fine). I've only found this SO answer and this one but error still persists. Any thoughts on what to check next?
Here's the error:
> jest server/test/test --config=server/test/jest.config.js
FAIL server/test/test.js
● Test suite failed to run
ReferenceError: TextEncoder is not defined
at Object.<anonymous> (../../node_modules/mongodb-connection-string-url/node_modules/whatwg-url/dist/encoding.js:2:21)
at Object.<anonymous> (../../node_modules/mongodb-connection-string-url/node_modules/whatwg-url/dist/url-state-machine.js:5:34)
Here's my Jest config and script:
jest.config.js
module.exports = {
preset: '#shelf/jest-mongodb'
};
test.js
const dotenv = require('dotenv');
const path = require('path');
process.env = dotenv.config({path: path.resolve(__dirname, '.env')}).parsed;
const request = require('supertest');
const {app, server} = require('../server');
const { MongoClient } = require('mongodb');
const { TextEncoder } = require('util');
global.TextEncoder = TextEncoder;
describe('GET /', () => {
let mongoClient;
beforeAll(async () => {
app.locals.mongoClient = mongoClient = await MongoClient.connect(process.env.MONGODB_URI, { useNewUrlParser: true, useUnifiedTopology: true });;
});
afterAll(async () => {
await mongoClient.close();
server.close();
});
it('responds with path of /', (done) => {
request(app).get('/').expect(JSON.stringify({path: '/'}), done);
});
});
My Github Actions workflow is erroring at the npm run test-server step:
name: cicd
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [ 10.x, 12.x, 14.x, 15.x ]
fail-fast: false
steps:
- uses: actions/checkout#v2
- name: Create environment variables
run: |
touch server/test/.env
echo MONGODB_URI=${{ secrets.QA_MONGODB_URI }} >> server/test/.env
- name: Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v2
with:
node-version: ${{ matrix.node-version }}
- run: npm ci
- run: npm run build --if-present
- run: npm run test-server
deploy:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: akhileshns/heroku-deploy#v3.12.12
with:
heroku_api_key: ${{ secrets.HEROKU_API_KEY }}
heroku_app_name: "my-app-name"
heroku_email: "my-email"

I had the same problem and a just inserted the two lines at the top
global.TextEncoder = require("util").TextEncoder;
global.TextDecoder = require("util").TextDecoder;
reference: ReferenceError: TextEncoder is not defined with mongodb nodes
global.TextEncoder = require("util").TextEncoder;
global.TextDecoder = require("util").TextDecoder;
import { MongoMemoryServer } from 'mongodb-memory-server';
import mongoose from 'mongoose';
let mongod: MongoMemoryServer;
beforeAll(async () => {
const mongod = await MongoMemoryServer.create();
const mongoUri = mongod.getUri();
await mongoose.connect(mongoUri);
});
beforeEach(async () => {
const collections = await mongoose.connection.db.collections();
for (let collection of collections) {
await collection.deleteMany({});
}
});
afterAll(async () => {
await mongod.stop(true);
await mongoose.connection.close();
});

Related

Running sequilize migration with umzug through github ci/cd

im using sequlize with umzug - migrations work locally, when i create a job for it, it cannot find the neccessery modules.
I got a mirgrator.js file.
const { migrator } = require('./iumzug.js');
migrator.runAsCLI()
And an iumzug.ts file as well, which configured like this.
const { Sequelize } = require('sequelize');
const { envVar } = require('./src/utilities/env-var')
const { Umzug, SequelizeStorage } = require("umzug")
const sequelize = new Sequelize({
database: envVar.DB_DATABASE,
host: envVar.DB_HOST,
port: 5432,
schema: ["TEST"].includes(envVar.NODE_ENV) ? 'test' : 'public',
username: envVar.DB_USERNAME,
password: envVar.DB_PASSWORD,
dialect: 'postgres',
ssl: true,
dialectOptions: {
ssl: {
require: true,
},},});
const migrator = new Umzug({
migrations: {
glob: ["./src/database/*.ts", { cwd: __dirname }],
resolve: ({ name, path, context }) => {
// eslint-disable-next-line #typescript-eslint/no-var-requires
const migration = require(path);
return {
// adjust the parameters Umzug will
// pass to migration methods when called
name,
up: async () => migration.up(context, Sequelize),
down: async () => migration.down(context, Sequelize)
};
}
},
context: sequelize.getQueryInterface(),
storage: new SequelizeStorage({
sequelize,
modelName: "migration_meta"
}),
logger: console
});
module.exports = { migrator }
i created a migration job on my github yml file as followes:
migrations:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout#v3
- name: migrations step
run: |
node migrator.js up
when i run github action - i get this error
looking for alternatives / directions to fix it.
Let me know if i need to add anymore code / pictures of the process.

How do I create a Github Action that will convert my markdown files into a single .json file

I am trying to create a Github Action that will take multiple markdown files and convert them into a single .json file. I tried the following and it did not work.
I used this found script.
// filename: buildJson.js
const fs = require("fs");
const path = require("path");
const matter = require("gray-matter");
const getAll = dir => {
// Read files at _posts/{directory}
const directory = path.join(process.cwd(), `content/${dir}`);
const fileNames = fs.readdirSync(directory);
// Get the content of the files as JSON
const content = fileNames.map((fileName, index) => {
const slug = fileName.replace(/\.md$/, "");
const fullPath = path.join(directory, fileName);
const fileContents = fs.readFileSync(fullPath, "utf8");
const matterResult = matter(fileContents);
return {
id: index + 1, slug, ...matterResult
};
});
// Return a big array of JSON
return JSON.stringify(content);
};
const allPosts = getAll("blog");
const postFileContents = `${allPosts}`;
// Create the cache folder if it doesn't exist
try {
fs.readdirSync("public/cache");
} catch (e) {
fs.mkdirSync("public/cache");
}
// Create our cached posts JSON
fs.writeFile("public/cache/posts.json", postFileContents, err => {
if (err) return console.log(err);
console.log("Posts cached.");
});
and this yml file in .github/workflows
name: create-json
on: [push]
jobs:
create-json:
runs-on: macos-latest
steps:
- uses: actions/checkout#v3
- uses: actions/setup-node#v3
with:
node-version: '18'
- run: npm install
- run: node buildJson.js
The script ran in Github Actions, it outputted the conole.log message I left myself, it did not create the .json file.

GitHub Actions create or update a Git Tag

Each time I commit to a Pull Request branch I want to tag it like 0.0.0-pr1
The below action works on initial commit.
But not on subsequent commits I get Error: Unhandled error: HttpError: Reference already exists
Is there a create or update Ref API I could use?
env:
PR_NUMBER: ${{ github.event.number }}
steps:
- name: Create Git tag for PR
uses: actions/github-script#v4
with:
script: |
github.git.createRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: "refs/tags/0.0.0-pr${{env.PR_NUMBER}}",
sha: context.sha
})
You can simply use EndBug/latest-tag GitHub action from Marketplace that adds (or updates) your custom tag:
env:
PR_NUMBER: ${{ github.event.number }}
steps:
# make sure you checkout first
- name: "⏬ Check out repository code"
uses: actions/checkout#v3
- name: "🏷️ Push tag"
uses: EndBug/latest-tag#latest
with:
# You can change the name of the tag with this input.
# Default: 'latest'
tag-name: ${{env.PR_NUMBER}}
# If a description is provided, the action will use it to create an annotated tag. If none is given, the action will create a lightweight tag.
#description: Description for the tag
As you can see, I had an existent tag (2.0.0-41) which got overwritten
Here is a possible solution
jobs:
tag:
runs-on: ubuntu-latest
name: "Tag Releases"
steps:
- uses: actions/github-script#v2
env:
CLIENT_PAYLOAD: ${{ toJSON(github.event.client_payload) }}
with:
script: |
const { owner, repo } = context.repo;
const { sha, releases } = JSON.parse(process.env.CLIENT_PAYLOAD);
for (const release of releases) {
const tagName = `${release.module}/${release.newVersion}`;
let ref = `refs/tags/${tagName}`;
console.log('Tagging', tagName, 'as', sha);
await github.git.createRef({ owner, repo, ref, sha });
const releaseVersion = `${release.newVersion}`;
const module = `${release.module}`;
try {
ref = "tags/" + module + "/v" + releaseVersion.split(".")[0]
console.log('Updating existing tag', ref, 'as', sha);
await github.git.updateRef({ owner, repo, ref, sha });
}
catch {
ref = "refs/tags/" + module + "/v" + releaseVersion.split(".")[0]
console.log('Creating new tag', ref, 'as', sha);
await github.git.createRef({ owner, repo, ref, sha });
}
}
In this case I have a monorepo which contains several terraform modules and they are versioned as modulename/SemVer and also moduleName/MajorVersion
console output:
Tagging monorepo-release-template/1.0.1 as 42b.....
Updating existing tag tags/monorepo-release-template/v1 as 42b....
```

github actions can not find package.json

I'm trying to set-up some basic GitHub action to write comments on the PRs.
Action is published on github and looks like this:
action.yml file:
name: !!name!!
description: !!description!!
author: !!me!!
inputs:
token:
description: "Github token"
required: true
runs:
using: "node12"
main: "index.js"
index.js file:
const core = require("#actions/core");
const { execSync } = require("child_process");
const { GitHub, context } = require("#actions/github");
const main = async () => {
const repoName = context.repo.repo;
const repoOwner = context.repo.owner;
const token = core.getInput("token");
const testCommand = "yarn test --watchAll=false";
const prNumber = context.payload.number;
const githubClient = new GitHub(token);
const reportAsString = execSync(testCommand).toString();
const commentBody = `<div><h2>Test report</h2>
<p>
<pre>${reportAsString}</pre>
</p>
</div>`;
await githubClient.issues.createComment({
repo: repoName,
owner: repoOwner,
body: commentBody,
issue_number: prNumber,
});
};
main().catch((err) => core.setFailed(err.message));
In the project, I've added action through github, like this
.github/workflows/main.yml file:
on: [push]
jobs:
start_tests:
runs-on: ubuntu-latest
name: A test job
steps:
- name: !!name!!
uses: !!link to my action on github with version!!
with:
token: ${{ secrets.GITHUB_TOKEN }}
However, my PRs actions are failing and this is the reason:
error Couldn't find a package.json file in "/home/runner/work/!!project_name!!/!!project_name!!"
##[error]Command failed: yarn test --watchAll=false
error Couldn't find a package.json file in "/home/runner/work/!!project_name!!/!!project_name!!"
So, anyone has an idea what am I doing wrong here? I tried to google the solution but erm... unsuccessfully :(
Thanks for your time!
package.json file has to be present to run yarn test command. As we can see in error there is no such file just because there is no project folder at all. It means that job doesn't know anything about project to work with. So, you have to do one of the following changes:
in case your action is published to marketplace
on: [push]
jobs:
start_tests:
runs-on: ubuntu-latest
name: A test job
steps:
- uses: actions/checkout#v2.1.0
- name: !!name!!
uses: !!link to my action on github with version!!
with:
token: ${{ secrets.GITHUB_TOKEN }}
in case your action is not published yet or you just want to run it "locally"
on: [push]
jobs:
start_tests:
runs-on: ubuntu-latest
name: A test job
steps:
- name: !!name!!
uses: ./
with:
token: ${{ secrets.GITHUB_TOKEN }}
If you have a similar structure in your github repo, docker/build-push-action#v3 file and context properties should be set appropriately in your workflow file.

Getting mongo bson error using Docker on Mongoose/Express

I'm working on a small test project and everything was working fine until I added date-logger "service" which uses mongoose.
I get the following error upon running docker-compose up --build:
Error: Cannot find module 'mongodb/node_modules/bson'
Require stack:
- /app/node_modules/mongoose/lib/drivers/node-mongodb-native/objectid.js
...
Everything else gets started and runs correctly, except for this service which immediately fails as soon as require('mongoose') is reached.
Here are some of the important files:
docker-compose.yml:
version: '3'
services:
postgres:
image: 'postgres:latest'
redis:
image: 'redis:latest'
api:
build:
dockerfile: Dockerfile.dev
context: ./server
volumes:
- /app/node_modules
- ./server:/app
environment:
- REDIS_HOST=${REDIS_HOST}
- REDIS_PORT=${REDIS_PORT}
- PG_USER=${PG_USER}
- PG_HOST=${PG_HOST}
- PG_DB=${PG_DB}
- PG_PASSWORD=${PG_PASSWORD}
- PG_PORT=${PG_PORT}
date-logger:
build:
dockerfile: Dockerfile.dev
context: ./date-logger
volumes:
- /app/node_modules
- ./date-logger:/app
environment:
- EXPRESS_PORT=${EXPRESS_PORT}
depends_on:
- "mongo"
client:
build:
dockerfile: Dockerfile.dev
context: ./client
volumes:
- /app/node_modules
- ./client:/app
worker:
build:
dockerfile: Dockerfile.dev
context: ./worker
volumes:
- /app/node_modules
- ./worker:/app
environment:
- REDIS_HOST=${REDIS_HOST}
- REDIS_PORT=${REDIS_PORT}
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
ports:
- '3050:80'
mongo:
image: mongo
restart: always
container_name: mongo
Dockerfile.dev on date-logger
FROM node:alpine
WORKDIR '/app'
COPY ./package.json .
RUN npm install
COPY . .
CMD ["npm", "run", "dev"]
package.json on date-logger:
{
"dependencies": {
"express": "~4.17.0",
"mongoose": "~5.5.11",
"cors": "2.8.4",
"body-parser": "~1.19.0"
},
"devDependencies": {
"nodemon": "1.19.0"
},
"scripts": {
"start": "node index.js",
"dev": "nodemon"
}
}
index.js on date-logger:
let express = require('express');
let bodyParser = require('body-parser');
// This console.log works
console.log('date-logger hello ---');
// ****!!!! IT FAILS IMMEDIATELY ON THIS REQUIRE !!!****
let mongoose = require('mongoose');
let app = express();
app.use(
bodyParser.urlencoded({
extended: true
})
);
app.use(bodyParser.json());
mongoose.connect('mongodb://mongo/date-logger');
const ItemSchema = new mongoose.Schema({
value: {
type: String,
required: true
},
date: {
type: Date,
default: Date.now
}
});
Item = mongoose.model('item', ItemSchema);
var port = process.env.EXPRESS_PORT || 8080;
app.get('/', (req, res) => res.send('Hello World with Express'));
app.get('/date', (req, res) => res.send('data'));
app.get('/date/all', (req, res) => {
Item.find({}).exec().then(results => res.send(results))
});
app.post('/date/add', (req, res) => {
const newItem = new Item({
value: req.body.value
});
newItem.save().then(item => res.send('success'));
});
app.listen(port, function() {
console.log('Running RestHub on port ' + port);
});
I'm new to devops world so all/any help is super appreciated! Thanks!
I think your problem is related to mongoose and not docker,
check this issue
This is very old but I came across a similar problem recently (but using only docker, not docker-compose).
I solved the problem by removing package-lock.json from my .dockerignore file (it should have never been there at all).
And of course make sure you can run npm install in your local machine, try removing package-lock and node_modulues and running npm install again if you can't. And update any modules you can.