Golang database/sql does not connect when using env file - postgresql

This works perfectly and the database is connected, no problems. But I do not want to hardcode my values in the function and hence am using an env file. But using os.getenv doesn't connect to the database.
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
const (
host = "localhost"
port = 5432
user = "bond"
password = "password"
dbname = "bookstore"
)
func main() {
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
"password=%s dbname=%s sslmode=disable",
host, port, user, password, dbname)
db, err := sql.Open("postgres", psqlInfo)
if err != nil {
panic(err)
}
defer db.Close()
err = db.Ping()
if err != nil {
panic(err)
}
fmt.Println("You connected to your database.")
}
The following code gives the error- panic: pq: password authentication failed for user "bond"
var (
host = "localhost"
port = 5432
user = "bond"
password = os.Getenv("DATABASE_PWD")
dbname = "bookstore"
)
Why does this happen?

It appears that you are doing something like this (playground):
var (
env = make(map[string]string) // Simulated environoment
password = env["DATABASE_PWD"]
)
func init() {
// Simulate loading .env file (godotenv.Load())
env["DATABASE_PWD"] = "foo"
}
func main() {
fmt.Println(password) // Outputs a blank line
fmt.Println(env["DATABASE_PWD"]) // Outputs "foo"
}
This will not work because the variables are initialized before init is run; this is as per the spec:
A package with no imports is initialized by assigning initial values to all its package-level variables followed by calling all init functions in the order they appear in the source, possibly in multiple files, as presented to the compiler. If a package has imports, the imported packages are initialized before initializing the package itself...
So the value of password is set before godotenv.Load() is run (meaning the password will be whatever value was in the environment when you started the app - probably empty).
You can fix this by loading environmental variables after you have run godotenv.Load(). I would suggest doing this in main so that you can handle any errors (godotenv.Load() may fail!). See the example in the package repo.
Note: This is why it's important to include a minimal, reproducible, example. When creating such an example you would probably have noticed that password was empty and that the issue was not at all related to database/sql. The question could not be answered without more info than provided in the question.

Why your solution does not work as intended
I think you misunderstood what the os.GetEnv() function does, your operating system has got something that is called environment variables and that is what you are trying to get here.
Your code returns either an empty string because it is not set as an Environment variable.
You can look at the following places to learn a bit more about this function and how it works:
official documentation
simple tutorial
What you want to do
Now allow me to explain what you want to actually do, you want to get these variables from a file, so you need to:
tell the programs where it is
open it
parse it
read the values
You can do that in two ways.
First solution (Do it Yourself)
First you open the file and pass its content to a scanner.
You then iterate over each line and split the line in two using the '=' character (Env format), finally store its key/value in something like a map (or whatever you want/need it to be).
How to read a file in go
Scanner read line doc
Second solution (for the lazy dev or complex project)
The second, if you don't want to do all of this, just use a config library or parser do it for you.
Here is a link to do it using Viper:
Load config from file & environment variables in Golang with Viper

Related

500 Internal Server Error - Azure Function through Data Factory using Postgres DB

I am trying to edit someone else's azure function app. When I run theirs, it works fine and connects to their DB successfully. When I try and change the connection string to my DB it gives me the error
HTTP response code: 500 Internal Server Error
without any other information.
Even if I just change the one line of code which defines the DB connection, it doesn't work. I have tried it on my local machine and it works, it just doesn't work in Azure functions.
Their original code (which works):
postgresql = os.environ.get('POSTGRES_SQL')
cnxn = psycopg2.connect(postgresql)
vs. mine (which doesn't work):
postgresql = 'postgresql://sqladmin:{my-password}#{db-connection-string}?sslmode=require'
cnxn = psycopg2.connect(postgresql)
I am also not sure where their DB connection comes from, using ".get('POSTGRES_SQL')", as they don't pass that parameter in anywhere. Below is how they call the function in Azure Data Factory, note that no parameters are passed in (nor anywhere in the function):
Even I try just a bare bones block of code as seen below, it gives me the same error.
def main(req: func.HttpRequest) -> func.HttpResponse:
postgresql = 'postgresql://sqladmin:{my-password}#{db-connection-string}?sslmode=require'
cnxn = psycopg2.connect(postgresql)
cursor = cnxn.cursor()
cursor.execute("CREATE TABLE staging.test (some-column varchar(100) null)")
cnxn.commit
cursor.close()
return func.HttpResponse(f"This HTTP triggered function executed successfully.")
Please let me know what I'm missing, or if you need any other info. I have already tried all other responses to similar StackOverflow questions.

How can I use FastAPI Routers with FastAPI-Users and MongoDB?

I can use MongoDB with FastAPI either
with a global client: motor.motor_asyncio.AsyncIOMotorClient object, or else
by creating one during the startup event per this SO answer which refers to this "Real World Example".
However, I also want to use fastapi-users since it works nicely with MongoDB out of the box. The downside is it seems to only work with the first method of handling my DB client connection (ie global). The reason is that in order to configure fastapi-users, I have to have an active MongoDB client connection just so I can make the db object as shown below, and I need that db to then make the MongoDBUserDatabase object required by fastapi-users:
# main.py
app = FastAPI()
# Create global MongoDB connection
DATABASE_URL = "mongodb://user:paspsword#localhost/auth_db"
client = motor.motor_asyncio.AsyncIOMotorClient(DATABASE_URL, uuidRepresentation="standard")
db = client["my_db"]
# Set up fastapi_users
user_db = MongoDBUserDatabase(UserDB, db["users"])
cookie_authentication = CookieAuthentication(secret='lame secret' , lifetime_seconds=3600, name='cookiemonster')
fastapi_users = FastAPIUsers(
user_db,
[cookie_authentication],
User,
UserCreate,
UserUpdate,
UserDB,
)
After that point in the code, I can import the fastapi_users Routers. However, if I want to break up my project into FastAPI Routers of my own, I'm hosed because:
If I move the client creation to another module to be imported into both my app and my routers, then I have different clients in different event loops and get errors like RuntimeError: Task <Task pending name='Task-4' coro=<RequestResponseCycle.run_asgi() running at /usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py:389> cb=[set.discard()]> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.8/asyncio/futures.py:360]> attached to a different loop (touched on in this SO question)
If I user the solutions of the "Real World Example", then I get stuck on where to build my fastapi_users object in my code example: I can't do it in main.py because there's no db object yet.
I considered making the MongoDBUserDatabase object as part of the startup event code (ie within async def connect_to_mongo() from the Real World Example), but I'm not able to get that to work either since I can't see how to make it work.
How can I either
make a global MongoDB client and FastAPI-User object in a way that can be shared among my main app and several routers without "attached to a different loop" errors, or
create fancy wrapper classes and functions to set up FastAPI users with the startup trigger?
I don't think my solution is complete or correct, but I figured I'd post it in case it inspires any ideas, I'm stumped. I have run into the exact dilemma, almost seems like a design flaw..
I followed this MongoDB full example and named it main.py
At this point my app does not work. The server starts up but result results in the aforementioned "attached to a different loop" whenever trying to query the DB.
Looking for guidance, I stumbled upon the same "real world" example
In main.py added the startup and shudown event handlers
# Event handlers
app.add_event_handler("startup", create_start_app_handler(app=app))
app.add_event_handler("shutdown", create_stop_app_handler(app=app))
In dlw_api.db.events.py this:
import logging
from dlw_api.user import UserDB
from fastapi import FastAPI
from fastapi_users.db.mongodb import MongoDBUserDatabase
from motor.motor_asyncio import AsyncIOMotorClient
LOG = logging.getLogger(__name__)
DB_NAME = "dlwLocal"
USERS_COLLECTION = "users"
DATABASE_URI = "mongodb://dlw-mongodb:27017" # protocol://container_name:port
_client: AsyncIOMotorClient = None
_users_db: MongoDBUserDatabase = None
def get_users_db() -> MongoDBUserDatabase:
return _users_db
async def connect_to_db() -> None:
global _users_db
# logger.info("Connecting to {0}", repr(DATABASE_URL))
client = AsyncIOMotorClient(DATABASE_URI)
db = client[DB_NAME]
collection = db[USERS_COLLECTION]
_users_db = MongoDBUserDatabase(UserDB, collection)
LOG.info(f"Connected to {DATABASE_URI}")
async def close_db_connection(app: FastAPI) -> None:
_client.close()
LOG.info("Connection closed")
And dlw_api.events.py:
from typing import Callable
from fastapi import FastAPI
from dlw_api.db.events import close_db_connection, connect_to_db
from dlw_api.user import configure_user_auth_routes
from fastapi_users.authentication import CookieAuthentication
from dlw_api.db.events import get_users_db
COOKIE_SECRET = "THIS_NEEDS_TO_BE_SET_CORRECTLY" # TODO: <--|
COOKIE_LIFETIME_SECONDS: int = 3_600
COOKIE_NAME = "c-is-for-cookie"
# Auth stuff:
_cookie_authentication = CookieAuthentication(
secret=COOKIE_SECRET,
lifetime_seconds=COOKIE_LIFETIME_SECONDS,
name=COOKIE_NAME,
)
auth_backends = [
_cookie_authentication,
]
def create_start_app_handler(app: FastAPI) -> Callable:
async def start_app() -> None:
await connect_to_db(app)
configure_user_auth_routes(
app=app,
auth_backends=auth_backends,
user_db=get_users_db(),
secret=COOKIE_SECRET,
)
return start_app
def create_stop_app_handler(app: FastAPI) -> Callable:
async def stop_app() -> None:
await close_db_connection(app)
return stop_app
This doesn't feel correct to me, does this mean all routes that use Depends for user-auth have to be included on the server startup event handler??
The author (frankie567) of fastapi-users created a repl.it showing a solution of sorts. My discussion about this solution may provide more context but the key parts of the solution are:
Don't bother using FastAPI startup trigger along with Depends for your MongDB connectivity management. Instead, create a separate file (ie db.py) to create your DB connection and client object. Import this db object whenever needed, like your Routers, and then use it as a global.
Also create a separate users.py to do 2 things:
Create globally used fastapi_users = FastAPIUsers(...) object for use with other Routers to handle authorization.
Create a FastAPI.APIRouter() object and attach all the fastapi-user routers to it (router.include_router(...))
In all your other Routers, import both db and fastapi_users from the above as needed
Key: split your main code up into
a main.py which only import uvicorn and serves app:app.
an app.py which has your main FastAPI object (ie app) and which then attaches all our Routers, including the one from users.py with all the fastapi-users routers attached to it.
By splitting up code per 4 above, you avoid the "attached to different loop" error.
I faced similar issue, and all I have to do to get motor and fastapi run in the same loop is this:
client = AsyncIOMotorClient()
client.get_io_loop = asyncio.get_event_loop
I did not set on_startup or whatsoever.

How to create kubernetes objects by passing a yaml file like kubectl apply in golang

I am creating a tool that will create kubernetes object using a yaml file. I am writing it in golang. The tool would like to achieve same behaviour as kubectl apply -f test.yaml
Basically, the fastest is to use an existing library like the official go client.
In essence kubectl also uses client-go so you can also follow its source code as a guideline.
You can also use k8s which is another K8s go client.
kubectl source code is hard to understand, especially when it involved many non-fixed object type to create, how to parse yaml to go object ( I got this working, but needs manual case switch to convert to real go type, which is a single type).
here's the secret create code, is there something general create that I can pass a list of different object? or better, just plain yaml string?
secretClient := client.CoreV1().Secrets("default")
result, err := secretClient.Create(secret)
link: https://sourcegraph.com/github.com/kubernetes/kubectl#master/-/blob/pkg/cmd/apply/apply.go#L328:22&tab=def
func (o *ApplyOptions) GetObjects() ([]*resource.Info, error) {
var err error = nil
if !o.objectsCached {
// include the uninitialized objects by default if --prune is true
// unless explicitly set --include-uninitialized=false
r := o.Builder.
Unstructured().
Schema(o.Validator).
ContinueOnError().
NamespaceParam(o.Namespace).DefaultNamespace().
FilenameParam(o.EnforceNamespace, &o.DeleteOptions.FilenameOptions).
LabelSelectorParam(o.Selector).
Flatten().
Do()
o.objects, err = r.Infos()
o.objectsCached = true
}
return o.objects, err
}
Lost for r.Infos, not sure where it defined.

Warning: Expression following 'return' is treated as an argument of the 'return'

I get this warning in my code and I can't figure out what it means and what I need to do to fix it.
The warning says the following:
Expression following 'return' is treated as an argument of the 'return'
And it's shown on this line:
// Tell Realm to use this new configuration object for the default Realm
Can someone explain what I'm doing wrong? See code part below.
Some background info, the code is part of a database manager class which I use to migrate a unencrypted realm database to an encrypted one, if the encrypted db doesn't exist yet.
If the encrypted db already exists, it configures realm to use this one as default. If it cannot open it (e.g. because of wrong encryption key) it creates a new database.
let exists = self.encryptedDatabaseExists(config)
if exists {
//Try to open Realm with new config. If unsuccessful, it needs to be removed and a new one created
do {
_ = try RLMRealm(configuration: config)
// Tell Realm to use this new configuration object for the default Realm
RLMRealmConfiguration.setDefaultConfiguration(config)
RLMRealm.defaultRealm()
} catch let err {
Log.error("Encrypted DB could not be opened: \(err)")
self.createNewEncryptedDatabase(config)
}
} else {
self.migrateToEncryptedDatabase(config)
}
Swift 5
Easy way enjoy it
//MARK:- Use it like function its will work
return()
NB: Adding this here since I've encountered the issue before.
According to the swift docs, if an expression follows immediately after the return statement, it will be returned/run as well.
In this case there's a return statement followed by other code expressions that gets treated as an argument.
To prevent this just add a ; after the return and code execution will stop.
return;

What is the write constant [duplicate]

I'm installing an application and want to set values for an ini file. Unfortunately, our main application is still built on a platform that gets redirected to the virtual store. Is there a way to get Inno Setup to store the ini file in the virtual store directly?
I believe there's even no Windows API to retrieve the path to the virtual store, let alone possibility to retrieve it reliably using Inno Setup.
But you can guess it to be {localappdata}\VirtualStore\path.
[Files]
Source: "MyProg.ini"; DestDir: "{code:GetVirtualStore|{app}}"
[Code]
function GetVirtualStore(Path: string): string;
var
Drive: string;
begin
Result := Path;
Drive := ExtractFileDrive(Path);
if CompareText(Drive, Copy(Path, 1, Length(Drive))) = 0 then
begin
Result := Copy(Result, Length(Drive) + 1, Length(Result) - Length(Drive));
Result := ExpandConstant('{localappdata}\VirtualStore') + Result;
end;
end;
You should probably also check that the path is on a system drive.