I am attempting to use googleapiclient.discovery.build('compute', 'v1').instances() to generate a list of devices. The below works fine but I want to know if there is a way I can pass format "json(disks,networkInterfaces,machineType,scheduling)" through cmdCompute.instances().list or even csv format. The current output is really messy.
I have gone through the docs but cannot seem so see a way. I know I can do it with gcloud but I am not able to use gcloud in a gcloud function which is where this script is going to be executed. The file will be used by other cloud functions so I would like a cleaner output.
`
from datetime import datetime, timedelta
from google.cloud import storage
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
def fncListVMs(self):
strHostProjectID='project'
strZone='zone'
strBucketName = 'bucketname'
credentials = GoogleCredentials.get_application_default()
cmdCompute=build('compute', 'v1',credentials=credentials, static_discovery=False)
strOutput = cmdCompute.instances().list(project=strHostProjectID, zone=strZone,filter= ('status=RUNNING')).execute()
strFile = strHostProjectID+"_Instance.json"
stoClient = storage.Client()
strBucket=stoClient.get_bucket(strBucketName)
strBlob = strBucket.blob('scripts/output/'+strFile)
with strBlob.open(mode='w') as strTarget:
strTarget.write(str(strOutput))
return "Done"
`
Related
We keep getting the following error on a Foundry Code Repo transform. It works in preview mode, but fails in build mode.
No transforms discovered in the pipeline from the requested files.
Please add the transform to the pipeline definer.
If using the Build button in Authoring, please ensure you are running the build from the file where the transform is generated.
Also note generated transforms may not be discovered using the Authoring Build button, and can be triggered
instead through the Dataset Preview app.: {filesWithoutDatasets=[transforms-python/src/name_of_transform_file.py]}
The input dataset is the result of a Data Connector REST ingest. And there's a column which I will call jsonResponseColumn that contains the actual json response from said ingest.
The code roughly looks like
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, ArrayType, StringType, DecimalType
from transforms.api import transform_df, Input, Output
#transform_df(
Output("output_df_location"),
input_df=Input("input_df_location"),
)
def compute(input_df):
schema = create_schema()
parsed_df = input_df
parsed_df = parsed_df.withColumn('newField1', F.from_json(parsed_df.jsonResponseColumn, schema, {"mode": "FAILFAST"}))
parsed_df = parsed_df.withColumn('newField2', F.explode(parsed_df.newField1.fieldInJsonResponse))
parsed_df = parsed_df.withColumn('newField3', parsed_df.newField2.nestedFieldInJsonResponse)
parsed_df = parsed_df.withColumn(
'id', ...
).withColumn(
'key', ...
).withColumn(
.....
)
return parsed_df.select(
...
)
def create_schema():
//basically returns a StructType([...]) that matches the json response from the REST ingest
The sub-folder of this x-form file was missing in the pipeline.py. I added the missing x-form sub-folder and it worked.
And just so other devs don't have to update this file every time they add a new sub-folder, I moved all the sub folders to /transforms and changed pipeline.py to what I have below & it works too!
from transforms.api import Pipeline
import repo_name.transforms as transforms
my_pipeline = Pipeline()
my_pipeline.discover_transforms(transforms)
I can use MongoDB with FastAPI either
with a global client: motor.motor_asyncio.AsyncIOMotorClient object, or else
by creating one during the startup event per this SO answer which refers to this "Real World Example".
However, I also want to use fastapi-users since it works nicely with MongoDB out of the box. The downside is it seems to only work with the first method of handling my DB client connection (ie global). The reason is that in order to configure fastapi-users, I have to have an active MongoDB client connection just so I can make the db object as shown below, and I need that db to then make the MongoDBUserDatabase object required by fastapi-users:
# main.py
app = FastAPI()
# Create global MongoDB connection
DATABASE_URL = "mongodb://user:paspsword#localhost/auth_db"
client = motor.motor_asyncio.AsyncIOMotorClient(DATABASE_URL, uuidRepresentation="standard")
db = client["my_db"]
# Set up fastapi_users
user_db = MongoDBUserDatabase(UserDB, db["users"])
cookie_authentication = CookieAuthentication(secret='lame secret' , lifetime_seconds=3600, name='cookiemonster')
fastapi_users = FastAPIUsers(
user_db,
[cookie_authentication],
User,
UserCreate,
UserUpdate,
UserDB,
)
After that point in the code, I can import the fastapi_users Routers. However, if I want to break up my project into FastAPI Routers of my own, I'm hosed because:
If I move the client creation to another module to be imported into both my app and my routers, then I have different clients in different event loops and get errors like RuntimeError: Task <Task pending name='Task-4' coro=<RequestResponseCycle.run_asgi() running at /usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py:389> cb=[set.discard()]> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.8/asyncio/futures.py:360]> attached to a different loop (touched on in this SO question)
If I user the solutions of the "Real World Example", then I get stuck on where to build my fastapi_users object in my code example: I can't do it in main.py because there's no db object yet.
I considered making the MongoDBUserDatabase object as part of the startup event code (ie within async def connect_to_mongo() from the Real World Example), but I'm not able to get that to work either since I can't see how to make it work.
How can I either
make a global MongoDB client and FastAPI-User object in a way that can be shared among my main app and several routers without "attached to a different loop" errors, or
create fancy wrapper classes and functions to set up FastAPI users with the startup trigger?
I don't think my solution is complete or correct, but I figured I'd post it in case it inspires any ideas, I'm stumped. I have run into the exact dilemma, almost seems like a design flaw..
I followed this MongoDB full example and named it main.py
At this point my app does not work. The server starts up but result results in the aforementioned "attached to a different loop" whenever trying to query the DB.
Looking for guidance, I stumbled upon the same "real world" example
In main.py added the startup and shudown event handlers
# Event handlers
app.add_event_handler("startup", create_start_app_handler(app=app))
app.add_event_handler("shutdown", create_stop_app_handler(app=app))
In dlw_api.db.events.py this:
import logging
from dlw_api.user import UserDB
from fastapi import FastAPI
from fastapi_users.db.mongodb import MongoDBUserDatabase
from motor.motor_asyncio import AsyncIOMotorClient
LOG = logging.getLogger(__name__)
DB_NAME = "dlwLocal"
USERS_COLLECTION = "users"
DATABASE_URI = "mongodb://dlw-mongodb:27017" # protocol://container_name:port
_client: AsyncIOMotorClient = None
_users_db: MongoDBUserDatabase = None
def get_users_db() -> MongoDBUserDatabase:
return _users_db
async def connect_to_db() -> None:
global _users_db
# logger.info("Connecting to {0}", repr(DATABASE_URL))
client = AsyncIOMotorClient(DATABASE_URI)
db = client[DB_NAME]
collection = db[USERS_COLLECTION]
_users_db = MongoDBUserDatabase(UserDB, collection)
LOG.info(f"Connected to {DATABASE_URI}")
async def close_db_connection(app: FastAPI) -> None:
_client.close()
LOG.info("Connection closed")
And dlw_api.events.py:
from typing import Callable
from fastapi import FastAPI
from dlw_api.db.events import close_db_connection, connect_to_db
from dlw_api.user import configure_user_auth_routes
from fastapi_users.authentication import CookieAuthentication
from dlw_api.db.events import get_users_db
COOKIE_SECRET = "THIS_NEEDS_TO_BE_SET_CORRECTLY" # TODO: <--|
COOKIE_LIFETIME_SECONDS: int = 3_600
COOKIE_NAME = "c-is-for-cookie"
# Auth stuff:
_cookie_authentication = CookieAuthentication(
secret=COOKIE_SECRET,
lifetime_seconds=COOKIE_LIFETIME_SECONDS,
name=COOKIE_NAME,
)
auth_backends = [
_cookie_authentication,
]
def create_start_app_handler(app: FastAPI) -> Callable:
async def start_app() -> None:
await connect_to_db(app)
configure_user_auth_routes(
app=app,
auth_backends=auth_backends,
user_db=get_users_db(),
secret=COOKIE_SECRET,
)
return start_app
def create_stop_app_handler(app: FastAPI) -> Callable:
async def stop_app() -> None:
await close_db_connection(app)
return stop_app
This doesn't feel correct to me, does this mean all routes that use Depends for user-auth have to be included on the server startup event handler??
The author (frankie567) of fastapi-users created a repl.it showing a solution of sorts. My discussion about this solution may provide more context but the key parts of the solution are:
Don't bother using FastAPI startup trigger along with Depends for your MongDB connectivity management. Instead, create a separate file (ie db.py) to create your DB connection and client object. Import this db object whenever needed, like your Routers, and then use it as a global.
Also create a separate users.py to do 2 things:
Create globally used fastapi_users = FastAPIUsers(...) object for use with other Routers to handle authorization.
Create a FastAPI.APIRouter() object and attach all the fastapi-user routers to it (router.include_router(...))
In all your other Routers, import both db and fastapi_users from the above as needed
Key: split your main code up into
a main.py which only import uvicorn and serves app:app.
an app.py which has your main FastAPI object (ie app) and which then attaches all our Routers, including the one from users.py with all the fastapi-users routers attached to it.
By splitting up code per 4 above, you avoid the "attached to different loop" error.
I faced similar issue, and all I have to do to get motor and fastapi run in the same loop is this:
client = AsyncIOMotorClient()
client.get_io_loop = asyncio.get_event_loop
I did not set on_startup or whatsoever.
I want to save the results of different runs in a compare runs experiment in Anylogic in an excel sheet or to some variable or output so that I use these values for some further calculations. How can I do that
Another flexible way of doing this is by writing the results to CSV. This should work for any kind of simulation or experiment with AnyLogic.
You'd have to import standard external libraries into the environment and add code to the experiment to be executed at the end of a simulation run.
Under imports section you could have this:
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.OutputStream;
import java.io.PrintStream;
First create a variable for the filename called csvFileName (you can call it whatever obviously). Then create a function in root called updateMyCSV() or something like that. This function would have a general structure of:
FileWriter pw = new FileWriter(csvFileName, true);
StringBuilder sb;
sb = new StringBuilder();
sb.append(<variablename>); sb.append(',');
Then repeat the above for each variable you want exported
Then finish the function with:
sb.append('\n');
pw.append(sb);
pw.flush();
pw.close();
Best use the build-in database and let it write to Excel at the end of all your runs.
Create a dbase table with columns such as "run_number", "replication_number", "experiment_parameter_X", "my_result_y". Each of your params on Main should get a column (at least those you vary) and each of the output values you are interested in.
This way, you can easily write your model results after each model run using the insertInto command (link).
Finally, just tick the "write to Excel" tickbox in your dbase, select the file and it will write all your raw results.
Also check the example models using the dbase, several use this approach
I'm relatively confused here, and upon trying to research for an answer, I'm not seeming to find anything that makes any sense to me. I have created a discord bot with 5 cogs, and in each one I import discord, os, and from discord.ext import commands In various other cogs I import other modules such as random as the case may be, but those are the three common ones.
The problem is that in every module, import discord is grayed out (PyCharm IDE), suggesting that is never used. Despite this, my bot runs perfectly. I don't seem to be able to use things like the wait_for() command, I presume it is because it is in the discord module? Am I not setting things up correctly to use this?
I will post the initial startup module and a small snippet of another module, rather than list module. If you need more information, let me know.
initial startup:
import discord
import os
from discord.ext import commands
token = open("token.txt", "r").read()
client = commands.Bot(command_prefix = '!')
#client.command()
async def load(ctx, extension):
client.load_extension("cogs." + extension)
#client.command()
async def unload(ctx, extension):
client.unload_extension("cogs." + extension)
for filename in os.listdir("./cogs"):
if filename.endswith('.py'):
client.load_extension("cogs." + filename[:-3])
client.run(token)
another module:
import discord
from discord.ext import commands
import os
import json
from pathlib import Path
class Sheet(commands.Cog):
def __init__(self, client):
self.client = client
#commands.command()
#commands.dm_only()
async def viewchar(self, ctx):
#Snipped code here to make it shorter.
pass
#viewchar.error
async def stats_error(self, ctx, error):
if isinstance(error, commands.PrivateMessageOnly):
await ctx.send("You're an idiot, now everyone knows. Why would you want to display your character sheet "
"in a public room? PM me with the command.")
else:
raise error
def setup(client):
client.add_cog(Sheet(client))
That just means that your code doesn't directly reference the discord module anywhere. You're getting everything through the commands module.
You can remove the import discord from your code without breaking anything, because the code that relies on it will still import and use it behind the scenes.
I'm surprised to not find a previous question about this, but I did give an honest try before posting.
I've created a ui with Qt Creator which contains quite a few QtWidgets of type QLineEdit, QTextEdit, and QCheckbox. I've used pyuic5 to convert to a .py file for use in a small python app. I've successfully got the form connected and working, but this is my first time using python with forms.
I'm searching to see if there is a built-in function or object that would allow me to pull the ObjectNames and Values of all widgets contained within the GUI form and store them in a dictionary with associated keys:values, because I need to send off the information for post-processing.
I guess something like this would work manually:
...
dict = []
dict['checkboxName1'] = self.checkboxName1.isChecked()
dict['checkboxName2'] = self.checkboxName2.isChecked()
dict['checkboxName3'] = self.checkboxName3.isChecked()
dict['checkboxName4'] = self.checkboxName4.isChecked()
dict['lineEditName1'] = self.lineEditName1.text()
... and on and on
But is there a way to grab all the objects and loop through them, even if each different type (i.e. checkboxes, lineedits, etc) needs to be done separately?
I hope I've explained that clearly.
Thank you.
Finally got it working. Couldn't find a python specific example anywhere, so through trial and error this worked perfectly. I'm including the entire working code of a .py file that can generate a list of all QCheckBox objectNames on a properly referenced form.
I named my form main_form.ui from within Qt Creator. I then converted it into a .py file with pyuic5
pyuic5 main_form.ui -o main_form.py
This is the contents of a sandbox.py file:
from PyQt5 import QtCore, QtGui, QtWidgets
import sys
import main_form
# the name of my Qt Creator .ui form converted to main_form.py with pyuic5
# pyuic5 original_form_name_in_creator.ui -o main_form.py
class MainApp(QtWidgets.QMainWindow, main_form.Ui_MainWindow):
def __init__(self):
super(self.__class__, self).__init__()
self.setupUi(self)
# Push button object on main_form named btn_test
self.btn_test.clicked.connect(self.runTest)
def runTest(self):
# I believe this creates a List of all QCheckBox objects on entire UI page
c = self.findChildren(QtWidgets.QCheckBox)
# This is just to show how to access objectName property as an example
for box in c:
print(box.objectName())
def main():
app = QtWidgets.QApplication(sys.argv) # A new instance of QApplication
form = MainApp() # We set the form to be our ExampleApp (design)
form.show() # Show the form
app.exec_() # and execute the app
if __name__ == '__main__': # if we're running file directly and not importing it
main() # run the main function
See QObject::findChildren()
In C++ the template argument would allow one to specify which type of widget to retrieve, e.g. to just retrieve the QLineEdit objects, but I don't know if or how that is mapped into Python.
Might need to retrieve all types and then switch handling while iterating over the resulting list.