Private key from environment variable instead of file in Google Cloud Firestore - google-cloud-firestore

I need to connect to Google Cloud Firestore from my Next.js serverless function hosted in Vercel. I already have a service account set up, but all the docs out there rely on the credentials being a file, while I'd like to use environment variables (more natural in Vercel platform).
Example:
const Firestore = require('#google-cloud/firestore');
const firestore = new Firestore({
projectId: 'YOUR_PROJECT_ID',
keyFilename: '/path/to/keyfile.json',
});
I cannot use keyFilename, I'd rather pass the service account email and private key explicitly.

Working code:
const projectId = process.env.GOOGLE_PROJECT_ID;
const email = process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL;
const key = process.env.GOOGLE_PRIVATE_KEY.replace(/\\n/g, '\n');
const Firestore = require('#google-cloud/firestore');
const firestore = new Firestore({
projectId: projectId,
credentials: {
client_email: email,
private_key: key,
},
});
Please note that my GOOGLE_PRIVATE_KEY env var has literal \ns, exactly as Google Cloud's JSON comes, so I use .replace() on it to translate them to actual newlines. This is actually only needed in my local environment, where I use .env.local, since Vercel env vars can take actual newlines.
Source
The settings object (argument for the constructor Firestore()) is poorly documented, but I was able to figure it out by myself grepping through the source code, where I found:
node_modules/#google-cloud/firestore/types/firestore.d.ts
Line 217:
/**
* Settings used to directly configure a `Firestore` instance.
*/
export interface Settings {
/**
* The project ID from the Google Developer's Console, e.g.
* 'grape-spaceship-123'. We will also check the environment variable
* GCLOUD_PROJECT for your project ID. Can be omitted in environments that
* support {#link https://cloud.google.com/docs/authentication Application
* Default Credentials}
*/
projectId?: string;
/** The hostname to connect to. */
host?: string;
/** The port to connect to. */
port?: number;
/**
* Local file containing the Service Account credentials as downloaded from
* the Google Developers Console. Can be omitted in environments that
* support {#link https://cloud.google.com/docs/authentication Application
* Default Credentials}. To configure Firestore with custom credentials, use
* the `credentials` property to provide the `client_email` and
* `private_key` of your service account.
*/
keyFilename?: string;
/**
* The 'client_email' and 'private_key' properties of the service account
* to use with your Firestore project. Can be omitted in environments that
* support {#link https://cloud.google.com/docs/authentication Application
* Default Credentials}. If your credentials are stored in a JSON file, you
* can specify a `keyFilename` instead.
*/
credentials?: {client_email?: string; private_key?: string};
/** Whether to use SSL when connecting. */
ssl?: boolean;
/**
* The maximum number of idle GRPC channels to keep. A smaller number of idle
* channels reduces memory usage but increases request latency for clients
* with fluctuating request rates. If set to 0, shuts down all GRPC channels
* when the client becomes idle. Defaults to 1.
*/
maxIdleChannels?: number;
/**
* Whether to use `BigInt` for integer types when deserializing Firestore
* Documents. Regardless of magnitude, all integer values are returned as
* `BigInt` to match the precision of the Firestore backend. Floating point
* numbers continue to use JavaScript's `number` type.
*/
useBigInt?: boolean;
/**
* Whether to skip nested properties that are set to `undefined` during
* object serialization. If set to `true`, these properties are skipped
* and not written to Firestore. If set `false` or omitted, the SDK throws
* an exception when it encounters properties of type `undefined`.
*/
ignoreUndefinedProperties?: boolean;
[key: string]: any; // Accept other properties, such as GRPC settings.
}

Related

Spring Cloud Contract and Zuul

Using Spring Cloud Dalston, we have built a proxy service that, of course, uses Zuul. I am now trying to add Spring Cloud Contract tests to verify our proxy service works as expected from a contract adherence perspective. What is strange is that I can send a request and receive a 200 status code back, but the expected response body, content type header, etc. is empty, and my tests fail as a result.
Are there additional configurations not specified in the Spring Cloud Contract documentation necessary for testing services that use Zuul capabilities?
Here you have an example https://github.com/spring-cloud/spring-cloud-contract/issues/450
/**
* Abstraction of configuration to test contracts with external microservices which are using a zuul-gateway in between.
*
* When a provider of data only allows communication through a Zuul-Gateway, a special way to ensure valid contracts is needed.
* The Goal is to emulate the Zuul-Interception to be able to use the contracts stored within the providing service.
*
* F.e.: Consumer-Service consumes the Provider-Service, but the provider can not be called directly.
* The Consumer-Service calls an URL with a prefix consisting of the name of the gateway ("gateway") and name of the
* service (in this example "Provider-Service"). For example: http://gateway/provider/path/to/resource.
* The contract provided by Provider-Service holds only a part of the provided URL (here: "/path/to/resource").
* The resolution of "gateway" and "provider" is done within this class.
*/
#RunWith(SpringRunner.class)
#SpringBootTest(classes = EmbeddedZuulProxy.class,
properties = {"server.port: 8080",
"zuul.routes.provider.path: /provider/**",
"zuul.routes.provider.service-id: provider",
"zuul.routes.provider.url: http://localhost:9090/" //the url of the provider-stub is at port 9090
},
webEnvironment = WebEnvironment.DEFINED_PORT) //defined port is important! Ribbon points to zuul, which then points to the provider-stub
#AutoConfigureMockMvc
#AutoConfigureJsonTesters
//the stub runs on fixed port 9090, so that zuul can point to it
#AutoConfigureStubRunner(ids = "<group-id>:<artifact-id>:+:stubs:9090")
#DirtiesContext
public abstract class ZuulContractBase {
}
/**
* Configuration and Setup of an embedded Zuul-Gateway.
* This way it is possible to use contracts, stored in providing service
*/
#Configuration
#ComponentScan(basePackages = "<path.to.feign.client>") //autowiring feign client
#EnableAutoConfiguration
#EnableZuulProxy
#EnableFeignClients
#RibbonClient(name = "gateway", configuration = SimpleRibbonClientConfiguration.class)
class EmbeddedZuulProxy {
#Bean
RouteLocator routeLocator(DiscoveryClient discoveryClient,
ZuulProperties zuulProperties) {
return new DiscoveryClientRouteLocator("/", discoveryClient, zuulProperties);
}
}
/**
* Ribbon Load balancer with fixed server list for "simple" pointing to localhost,
* which holds the mocked zuul-gateway
*
* f.e. a call with feign would normally look like:
* http://gateway/provider/rest/path/to/your/{url}
* which is mapped to:
* http://localhost:{zuulport}/provider/rest/path/to/your/{url}
*/
#Configuration
class SimpleRibbonClientConfiguration {
#Value("${server.port}")
private Integer zuulPort;
#Bean
public ServerList<Server> ribbonServerList() {
return new StaticServerList<>(new Server("localhost", zuulPort));
}
}

In an extbase extension, how to access the persistence layer from a scheduler task?

What sounds a bit academic in the title is actually quite straightforward: I have set up a TYPO3 6.1 extbase extension that I've equipped with a scheduler task. The task is supposed to import a CSV file and save it into the extension's database fields.
But how do I tell the scheduler task to use the extension's model etc. and save the received data into the persistence layer?
I've seen this answer to a similar question: Execute repository functions in scheduler task and I think it points the right way, but I think need a full example to start understanding how the dependency injection works.
First you have to consider the aspect of performance:
If you want to insert a big amount of data, you should not use the Extbase persistence for such a task. Because if you do so, it will generate an object for each row you want to insert and persist it immediately. This is quite slow and has a big memory footprint.
If you don't have much data or you split the jobs (e.g. perform 100 import jobs per scheduler run), then use the Extbase persistence.
You can have both in CommandController context, and since CommandControllers are straight-forward to set up, you should go for them instead of an own Scheduler task.
Using Extbase persistence
In the CommandController, inject your repository:
/**
* myRepository
*
* #var \Venor\Myext\Domain\Repository\MyRepository
* #inject
*/
protected $myRepository
Then iterate through the rows you want to import (foreach) and create a new object for every row and add it to your repository:
$myObject = $this->objectManager->get('Vendor\Myext\Domain\Model\MyModel');
$myObject->setProperty('foo');
$myObject->setOtherProperty('bar');
$this->myRepository->add($myObject);
To actually save the objects to the database, you need to persist them. So you also inject the persistenceManager:
/**
* #var \TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager
* #inject
*/
protected $persistenceManager;
And use it:
$this->persistenceManager->persistAll();
You should not do that for every single object (for performance reasons); but for memory usage reasons you should not wait until after thousands of objects to persist, either. So you just insert an iterator to your foreach loop and persist every 20th, 40th, or whatever loop.
Please don't forget that the Scheduler works in Backend context, so the TypoScript must be available by module.tx_yourext. If you want to share the same settings/storagePid with the frontend part of your app, use
module.tx_yourext.persistence < plugin.tx_yourext.persistence
[...]
The TypoScript needs to be present in the root page of your website for backend modules/CommandControllers to use them. I suggest you add the stuff to myext/Configuration/TypoScript/setup.txt and add the static template of your extension to the root page.
Using DataHandler
The TYPO3 DataHandler (formerly TCEmain) is the engine the TYPO3 backend uses for inserting and modifying database records. It is very powerful.
Instead of an object, inside your loop you create an array containing all the data. The first array index is the table, the next level is the affected record, where NEW means that a new record is created. Then you can just set every field of a table with the desired value
$data = array();
$data['be_users']['NEW'] = array(
'pid' => 0,
'username' => $staffMember['IDPerson'],
'password' => md5(GeneralUtility::generateRandomBytes(40)), // random password
'usergroup' => '1,2',
'email' => $staffMember['Email'],
'realName' => $staffMember['Nachname'] . ' ' . $staffMember['Vorname'],
'lang' => 'de',
);
Now you can make an Instance of DataHandler and persist the changes:
/** #var $tce t3lib_TCEmain */
$tce = GeneralUtility::makeInstance('TYPO3\CMS\Core\DataHandling\DataHandler');
$tce->bypassAccessCheckForRecords = TRUE;
$tce->start($data, array());
$tce->admin = TRUE;
$tce->process_datamap();
$newRecordsUidArray = $tce->substNEWwithIDs['NEW'];
Please note the line $tce->admin = TRUE. This suggests to DataHandler that an admin is performing the action. This is convenient because you don't have to set allowed exclude fields for the Scheduler user and can also insert records to PID 0. But it is a possible security flaw, so carefully consider its usage.
Records inserted/updated by DataHandler logged correctly, can be reverted etc.. You can find some examples (such as adding pictures, resolving MM relations) here. In this case all DataHandler related functions were moved to an external repository class that is injected to the CommandController as described above (it's just named in Extbase convention).
A good overview of DataHandler functions can be found here.
In Addition to lorenz's answer: Beginner's Guide to set up a Command Controller Scheduler task:
My example is an import task. Change the Name part "Import" to your needs.
Create a new file EXT:Classes/Controller/ImportCommandController.php
<?php
namespace NAMESPACE\Myextension\Controller;
/***************************************************************
* Copyright notice
*
* (c) 2014
* All rights reserved
*
* This script is part of the TYPO3 project. The TYPO3 project is
* free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* The GNU General Public License can be found at
* http://www.gnu.org/copyleft/gpl.html.
*
* This script is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* This copyright notice MUST APPEAR in all copies of the script!
***************************************************************/
/**
*
*
* #package Myextension
* #license http://www.gnu.org/licenses/gpl.html GNU General Public License, version 3 or later
*
*/
class ImportCommandController extends \TYPO3\CMS\Extbase\Mvc\Controller\CommandController {
/**
* itemRepository
*
* #var \NAMESPACE\Myextension\Domain\Repository\ItemRepository
* #inject
*/
protected $itemRepository;
/**
* #var \TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager
* #inject
*/
protected $persistenceManager;
/**
*
* #param \integer $storagePid
* #param \string $url
*
* #return bool
*/
// very nice: parameters will be fields in the scheduler!
public function importCommand($storagePid = 0,$url = NULL) {
$source = utf8_encode(utf8_encode(file_get_contents($url)));
// set storage page ourselves
// not sure if really necessary
$querySettings = $this->itemRepository->createQuery()->getQuerySettings();
$querySettings->setRespectStoragePage(FALSE);
$this->itemRepository->setDefaultQuerySettings($querySettings);
// do your stuff here
$source = 'foo';
$rendered = 'bar';
// then store it
// this seems to be only necessary if we don't have an existing item yet
// but as we don't know that here, we have to do it
$item = $this->objectManager->get('STUBR\Therapiestellen\Domain\Model\Item');
// find all existing items
$all = $this->itemRepository->findAll();
// if we have an item already, take the first (and only one)
if(count($all) > 0){
$item = $all->getFirst();
}
// set / update properties
$item->setSource($source);
$item->setRendered($r);
$item->setPid($storagePid);
// either update or add it
if(count($all) > 0){
$this->itemRepository->update($item);
}
else {
$this->itemRepository->add($item);
}
// persist it
$this->persistenceManager->persistAll();
}
}
?>
In EXT:ext_localconf.php, add the command controller:
$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['extbase']['commandControllers'][] = 'NAMESPACE\\Myextension\\Controller\\ImportCommandController';
Configure in Scheduler:
That's basically it!

SphinxQL & Phalcon\Mvc\Model

I have a Sphinx search engine running on MySQL protocol and I use Phalcon\Db\Adapter\Pdo\Mysql to connect to it. Sphinx tables are implemented as models.
When I try to select (using SpinxQL) I, obviously, get an error when database adapter attempts to extract table metadata running queries against tables which are not supported and not present respectively in SpinxQL. There is a workaround in the documentation showing how to manually assign metadata... But being to lazy by nature I want to try to automate metadata generation.
I assume that metadata is produced by the database adapter, probably as a result of calling getColumnsList() on the instance following getColumnDefinition() or something else (???). Is this my assumption correct? I want is to extend Phalcon\Db\Adapter\Pdo\Mysql and override those methods to be compatible with Sphinx.
Thanks in advance for your suggestions!
Ok, you need to override at least two methods to make this work, the following class would work:
<?php
class SphinxQlAdapter extends Phalcon\Db\Adapter\Pdo\Mysql implements Phalcon\Db\AdapterInterface
{
/**
* This method checks if a table exists
*
* #param string $table
* #param string $schema
* #return boolean
*/
public function tableExists($table, $schema=null)
{
}
/**
* This method describe the table's columns returning an array of
* Phalcon\Db\Column
*
* #param string $table
* #param string $schema
* #return Phalcon\Db\ColumnInterface[]
*/
public function describeColumns($table, $schema=null)
{
}
}
Then in your connection, you use the new adapter:
$di->set('db', function(){
return new SphinxQlAdapter(
//...
);
});

How can I protect my methods bodies (not the attached JavaDoc and Signature) using Acceleo code-generator

I use Acceleo in order to generate code with a model I have made. I managed to protect my methods in order to protect them usinig "#generated NOT" in case I need to regenerate my code with Acceleo. The problem is that adding #generated NOT protect all the method content, that is to say the body, the signature and JavaDocs.
The thing is that I only need to keep the method body, or at least the method body and its signature, but I need the doc to be updated. How can I do this ?
Just for information here is an example of a potential generated class :
/*
* #generated
*/
public class ActeurRefEntrepriseServicesImpl implements ActeurRefEntrepriseServices {
#Autowired
HelloWorldService helloWorldService;
/**
* Service which say hello
*
* #param name
* user name
* #return print Hello username
*
* #generated NOT
*/
#Override
public void sayHello(final String name) {
helloWorldService.print(name);
}
}
Baptiste,
The #generated tags use the standard EMF protection rules : "#generated" means that the body of the block for which it is set will be generated, anything else means no re-generation. If you set something as "#generated" in any of your metamodels' generated code, you will see that there, too, the javadoc is preserved whatever the edits you do.
In short, you cannot tell EMF to re-generate anything other than the code itself.
If you need to have the body protected but not the javadoc, you have to shift from the "#generated" protection to Acceleo's [protected] blocks. i.e, change your template from :
[template generatedMethod(methodName : String)]
/**
* Some doc.
* #param param1
* param documentation.
* #generated
*/
[generateSignature(methodName)/] {
[generateBody()/]
}
[/template]
to something using a protected block :
[template generatedMethod(methodName : String)]
/**
* Some doc.
* #param param1
* param documentation.
*/
[protected (methodName)]
[generateSignature(methodName)/] {
[generateBody()/]
}
[/protected]
[/template]
With this paradigm, anything that is outside of the protected area will be regenerated, everything else will remain untouched by a regeneration.
See also the full documentation available from the Acceleo website.
If you absolutely need to use the "#generated" protection method for your model, you will need to tamper with the JMerger API from EMF and alter the launcher Acceleo generated for you in order to use your own merging strategy (see the getGenerationStrategy method from that launcher). Note that this is by no means an easy task.

Notes Plugin: Where are Custom User Settings Stored?

We've developed a custom plugin for Notes 8.5.2. It records a number of custom user preferences. The class that does so is shown below:
import java.util.prefs.Preferences;
/**
* Provides programmatic access to Windows Registry entries for this plug-in.
*/
public class Registry
{
Preferences prefs;
/**
* Initializes a new instance of the Registry class.
*/
public Registry()
{
prefs = Preferences.userNodeForPackage(Registry.class) ;
}
/**
* Gets the value of a registry key.
*
* #param keyName The name of the key to return.
*
* #return A string containing the value of the specified registry key. If a key with the specified name cannot be
* found, the return value is an empty string.
*/
public String GetValue(String keyName)
{
try
{
return prefs.get(keyName, "NA") ;
}
catch(Exception err)
{
return "" ;
}
}
/**
* Sets the value of a registry key.
*
* #param keyName The name of the registry key.
*
* #param keyValue The new value for the registry key.
*/
public void SetValue(String keyName, String keyValue)
{
try
{
prefs.put(keyName, keyValue);
prefs.flush();
}
catch(Exception err)
{
}
}
}
A sample of the code that uses it is as follows:
Registry wr = new Registry();
String setting1 = wr.GetValue("CustomSetting1");
wr.SetValue("CustomSetting1", newValue);
Now, I've scanned the Windows Registry, and these settings do not exist. I've indexed my entire hard disk, and I cannot find these entries in any file.
So, where the heck are these settings being stored?
On Windows the Java Preferences API uses the Registry as it's backing store for the Preferences class. The keys are rooted by package name under HKEY_CURRENT_USER\Software\JavaSoft\Prefs.
Your code doesn't specify a package, so it uses the following location by default (tested on Windows Vista and 7):
HKEY_CURRENT_USER\Software\JavaSoft\Prefs\<unnamed>
There is an articled titled "Sir, What is Your Preference?" by Ray Djajadinataz at the Sun Developer Network where you can get a little more background on this API with some screen shots showing registry locations.
I wonder if you were searching for the key name, eg CustomSetting1, and not finding it because it is saved as /Custom/Setting1 to note that the C and S are capitalized (see API docs.)