Integration tests with mongodb - mongodb

I'm using mongodb as my main database, so on my integration tests I try to test if my system handles the data as expected. The process is as follow:
Insert data on needed collections to prepare scenario
Perform call to the system
Validate data change or exist on database, depending of the scenario
I'm using behat as my BDD testing tool and I implement on my FeatureContext to insert and read data from collections, something like this:
Given I have the following data on collection Brand
[
{
"_id": "575df0b00419c905492d0461",
"name": "Adidas",
"image": "adidas.png"
}
]
Then I validate the following data exists on collection Brand
[
{
"_id": "575df0b00419c905492d0461",
"name": "Adidas",
"image": "adidas.png"
}
]
EDIT
/**
* #Given I have the following data on collection :collectionName
* #param string $collectionName
* #param PyStringNode $collectionData
* #throws Exception
*/
public function iHaveTheFollowingDataOnCollection($collectionName, PyStringNode $collectionData)
{
$data = json_decode($collectionData->getRaw(), true);
foreach ($data as $index => $element) {
foreach ($element as $key => $value) {
if ($key === '_id') {
$data[$index][$key] = new \MongoDB\BSON\ObjectID($value);
}
if (strpos($key, "#") !== false) {
$key = explode("#", $key)[1];
$data[$index][$key] = array(
"\$ref" => $value["\$ref"],
"\$id" => new \MongoDB\BSON\ObjectID($value["\$id"]),
"\$db" => $value["\$db"],
);
}
if ($value === "UNIX_TIME()") {
$data[$index][$key] = time();
}
}
}
$collection = $this->db->$collectionName;
foreach ($data as $document) {
$collection->insertOne($document);
}
}
END EDIT
The issue is that I have at least 1k different scenarios on different features and with that heavy usage test randomly fails becase it don't find expected data on collection.
My first thought is that exist some delay on insertion and availability of the data.
Any thoughts and how to fix it?

Related

How to use createUser in Facebook Ads (to replace deprecated addUsers)

Now that I've upgraded to "facebook/php-ads-sdk": "2.8.*" (https://github.com/facebook/facebook-php-ads-sdk), this function of mine doesn't work anymore:
public function addToCustomAudience($entriesArray, $audienceId, $inputType = CustomAudienceTypes::EMAIL) {
$audience = new CustomAudience($audienceId);
$result = $audience->addUsers($entriesArray, $inputType);
return $result;
}
Apparently addUsers is no longer available.
I see a createUser function, but it looks quite different, and there is no documentation online about how to migrate from addUsers to createUser.
What I want to do is simple.
Given an array of email addresses and an ID of an audience, how can I add all of those email addresses to that Facebook Custom Audience?
From what I can see in the code, addUsers is still there, and it's documented on the Developer site.
I just used the latest SDK along with the following code to update an audience:
use FacebookAds\Object\CustomAudience;
use FacebookAds\Object\Values\CustomAudienceTypes;
$emails = array(
'test1#example.com',
'test2#example.com',
'test3#example.com',
);
$audience = new CustomAudience(<CUSTOM_AUDIENCE_ID>);
$audience->addUsers($emails, CustomAudienceTypes::EMAIL);
This seems to work for my purposes.
I copied some code from the facebook-php-ads-sdk as a workaround.
$audience = new CustomAudience($audienceId);
$params = $this->formatParams($entriesArray, $inputType, [], false);
$audience->createUser([], $params, false);
/**
* Copied this from Facebook's https://github.com/facebook/facebook-php-ads-sdk/blob/d51193b19d730ae9274d45540986e1ac311b074d/src/FacebookAds/Object/CustomAudience.php#L363
* Take users and format them correctly for the request
*
* #param array $users
* #param string $type
* #param array $app_ids
* #param bool $is_hashed
* #return array
*/
protected function formatParams(array $users, $type, array $app_ids = array(), $is_hashed = false) {
if ($type == CustomAudienceTypes::EMAIL || $type == CustomAudienceTypes::PHONE) {
$normalizer = new EmailNormalizer();
$hash_normalizer = new HashNormalizer();
foreach ($users as &$user) {
if ($type == CustomAudienceTypes::EMAIL) {
$user = $normalizer->normalize(CustomAudienceTypes::EMAIL, $user);
}
if (!$is_hashed) {
$user = $hash_normalizer->normalize(
CustomAudienceTypes::EMAIL, $user);
}
}
}
$payload = array(
'schema' => $type,
'data' => $users,
);
if ($type === CustomAudienceTypes::ID) {
if (empty($app_ids)) {
throw new \InvalidArgumentException(
"Custom audiences with type " . CustomAudienceTypes::ID . " require"
. "at least one app_id");
}
$payload['app_ids'] = $app_ids;
}
return array('payload' => $payload);
}

TYPO3 Extbase - Paginate through a large table (100000 records)

I have a fairly large table with about 100000 records. If I don't set the limit in the repository
Repository:
public function paginateRequest() {
$query = $this->createQuery();
$result = $query->setLimit(1000)->execute();
//$result = $query->execute();
return $result;
}
/**
* action list
*
* #return void
*/
public function listAction() {
$this->view->assign('records', $this->leiRepository->paginateRequest());
//$this->view->assign('records', $this->leiRepository->findAll());
}
... the query and the page breaks although I'm using f:widget.paginate . As per the docs https://fluidtypo3.org/viewhelpers/fluid/master/Widget/PaginateViewHelper.html I was hoping that I can render only the itemsPerPage and 'parginate' through the records ...
List.hmtl
<f:if condition="{records}">
<f:widget.paginate objects="{records}" as="paginatedRecords" configuration="{itemsPerPage: 100, insertAbove: 0, insertBelow: 1, maximumNumberOfLinks: 10}">
<f:for each="{paginatedRecords}" as="record">
<tr>
<td><f:link.action action="show" pageUid="43" arguments="{record:record}"> {record.name}</f:link.action></td>
<td><f:link.action action="show" pageUid="43" arguments="{record:record}"> {record.lei}</f:link.action></td>
</tr>
</f:for>
</f:widget.paginate>
Model:
class Lei extends \TYPO3\CMS\Extbase\DomainObject\AbstractEntity {
...
/**
* abc
*
* #lazy
* #var string
*/
protected $abc = '';
...
I use in TYPO3 9.5. The next function in repository:
public function paginated($page = 1, $size = 9){
$query = $this->createQuery();
$begin = ($page-1) * $size;
$query->setOffset($begin);
$query->setLimit($size);
return $query->execute();
}
And in the controller I am using arguments as parameter to send the page to load in a Rest action.
public function listRestAction()
{
$arguments = $this->request->getArguments();
$totalElements = $this->repository->total();
$pages = ceil($totalElements/9);
$next_page = '';
$prev_page = '';
#GET Page to load
if($arguments['page'] AND $arguments['page'] != ''){
$page_to_load = $arguments['page'];
} else {
$page_to_load = 1;
}
#Configuration of pagination
if($page_to_load == $pages){
$prev = $page_to_load - 1;
$prev_page = "http://example.com/rest/news/page/$prev";
} elseif($page_to_load == 1){
$next = $page_to_load + 1;
$next_page = "http://example.com/rest/news/page/$next";
} else {
$prev = $page_to_load - 1;
$prev_page = "http://example.com/rest/news/page/$prev";
$next = $page_to_load + 1;
$next_page = "http://example.com/rest/news/page/$next";
}
$jsonPreparedElements = array();
$jsonPreparedElements['info']['count'] = $totalElements;
$jsonPreparedElements['info']['pages'] = $pages;
$jsonPreparedElements['info']['next'] = $next_page;
$jsonPreparedElements['info']['prev'] = $prev_page;
$result = $this->repository->paginated($page_to_load);
$collection_parsed_results = array();
foreach ($result as $news) {
array_push($collection_parsed_results, $news->parsedArray());
}
$jsonPreparedElements['results'] = $collection_parsed_results;
$this->view->assign('value', $jsonPreparedElements);
}
The result of this, is a JSON like this:
{
"info": {
"count": 25,
"pages": 3,
"next": "",
"prev": "http://example.com/rest/news/page/2"
},
"results": [
{ ....}
] }
How large / complex are the objects you want to paginate through? If they have subobjects that you dont need in the list view, add #lazy annotation to those relations inside the model.
Due to this large amount of records, you should keep them as simple as possible in the list view. You can try to only give the result as array to the list view using $this->leiRepository->findAll()->toArray() or return only the raw result from your repository by adding true to execute(true).
You can also create an array of list items yourself in a foreach in the controller and only add the properties you really need inside the list.
If your problem is the performance, just use the default findAll()-Method.
The built-in defaultQuerySettings in \TYPO3\CMS\Extbase\Persistence\Repository set their offset and limit based on the Pagination widget, if not set otherwise.
If the performance issue persists, you may have to consider writing a custom query for your database request, that only requests the data your view actually displays. The process is described in the documentation: https://docs.typo3.org/typo3cms/ExtbaseFluidBook/6-Persistence/3-implement-individual-database-queries.html

How to prevent SQL injection in PhalconPHP when using sql in model?

Let's say I am building a search that finds all the teacher and got an input where the user can put in the search term. I tried reading the phalcon documentation but I only see things like binding parameters. I read the other thread about needing prepare statements do I need that in Phalcon as well?
And my function in the model would be something like this:
public function findTeachers($q, $userId, $isUser, $page, $limit, $sort)
{
$sql = 'SELECT id FROM tags WHERE name LIKE "%' . $q . '%"';
$result = new Resultset(null, $this,
$this->getReadConnection()->query($sql, array()));
$tagResult = $result->toArray();
$tagList = array();
foreach ($tagResult as $key => $value) {
$tagList[] = $value['id'];
....
}
}
My question is for the Phalcon framework is there any settings or formats I should code for this line $sql = 'SELECT id FROM tags WHERE name LIKE "%' . $q . '%"';
And also any general recommendation for preventing SQL Injection in PhalconPHP controllers and index would be appreciated.
For reference:
My controller:
public function searchAction()
{
$this->view->disable();
$q = $this->request->get("q");
$sort = $this->request->get("sort");
$searchUserModel = new SearchUsers();
$loginUser = $this->component->user->getSessionUser();
if (!$loginUser) {
$loginUser = new stdClass;
$loginUser->id = '';
}
$page = $this->request->get("page");
$limit = 2;
if (!$page){
$page = 1;
}
$list = $searchUserModel->findTeachers($q, $loginUser->id, ($loginUser->id)?true:false, $page, $limit, $sort);
if ($list){
$list['status'] = true;
}
echo json_encode($list);
}
My Ajax:
function(cb){
$.ajax({
url: '/search/search?q=' + mapObject.q + '&sort=<?php echo $sort;?>' + '&page=' + mapObject.page,
data:{},
success: function(res) {
//console.log(res);
var result = JSON.parse(res);
if (!result.status){
return cb(null, result.list);
}else{
return cb(null, []);
}
},
error: function(xhr, ajaxOptions, thrownError) {
cb(null, []);
}
});
with q being the user's search term.
You should bind the query parameter to avoid an SQL injection. From what I can remember Phalcon can be a bit funny with putting the '%' wildcard in the conditions value so I put them in the bind.
This would be better than just filtering the query.
$tags = Tags::find([
'conditions' => 'name LIKE :name:',
'bind' => [
'name' => "%" . $q . "%"
]
])
Phalcon\Filter is helpful when interacting with the database.
In your controller you can say, remove everything except letters and numbers from $q.
$q = $this->request->get("q");
$q = $this->filter->sanitize($q, 'alphanum');
The shortest way for requests:
$q = $this->request->get('q', 'alphanum');

Update embedded subdocument in array with doctrine and mongodb

I want to update (replace) a subdocument within an array of all the documents, that have this specific embedded subdocument.
My sample content object is:
{
"_id" : ObjectId("51f289e5345f9d10090022ef"),
"title" : "This is a content",
"descriptors" : [
{
"_id" : ObjectId("51f289e5345f9d10090022f4"),
"name" : "This is a descriptor",
"type" : "This is a property"
},
{
"_id" : ObjectId("51f289e5345f9d10090022f0"),
"name" : "This is another descriptor",
"type" : "This is another property"
}
]
}
I want to find the object with a specific descriptor._id.
I want to replace the one subdocument with another one (respectivley with an updated version of the old object, but not necessarily with the same properties).
While this works well in RoboMongo / shell...
db.Content.update(
{
'descriptors._id': ObjectId("51f289e5345f9d10090022f4")
},
{
$set: {
'descriptors.$': {
"_id" : ObjectId("51f289e5345f9d10090022f4"),
"name" : "This is the updated descriptor",
"category" : "This is a new property"
}
}
},
{
'multi': true
})
...and with the plain php methods...
$descriptor = array();
$descriptor['_id'] = new \MongoID('51f289e5345f9d10090022f4');
$descriptor['name'] = 'This is the updated descriptor';
$descriptor['category'] = 'This is a new property';
$mongo = new \Mongo('mongodb://localhost:27017');
$database = $mongo->selectDB('MyDatabase');
$output = $database->selectCollection('Content')->update(
array('descriptors._id' => $descriptor['_id']),
array('$set' => array('descriptors.$' => $descriptor)),
array("multiple" => true)
);
...it doesn't work with Doctrine MongoDB ODM...
$descriptor = array();
$descriptor['_id'] = new \MongoID('51f289e5345f9d10090022f4');
$descriptor['name'] = 'This is the updated descriptor';
$descriptor['category'] = 'This is a new property';
$query = $dm->createQueryBuilder('Content')
->update()->multiple(true)
->field('descriptors._id')->equals($descriptor['_id'])
->field('descriptors.$')->set($descriptor)
->getQuery()->execute();
...because it fails with the following error:
Notice: Undefined offset: 2 in C:\MyProject\vendor\doctrine\mongodb-odm\lib\Doctrine\ODM\MongoDB\Persisters\DocumentPersister.php line 998
So I assume, that Doctrine MongoDB ODM needs three parts in the dot-notation. The not so nice solution would be to iterate over the properties of the subdocument and set them manually.
$descriptor = array();
$descriptor['_id'] = new \MongoID('51f289e5345f9d10090022f4');
$descriptor['name'] = 'This is the updated descriptor';
$descriptor['category'] = 'This is a new property';
$query = $dm->createQueryBuilder('Content')
->update()->multiple(true)
->field('descriptors._id')->equals($descriptor['_id']);
foreach ($descriptor as $key => $value) {
$query->field('descriptors.$.'.$key)->set($value);
}
$query->getQuery()->execute();
But this will only update existing and add new properties, but won't remove old/unnecessary properties from the subdocument.
Any ideas how to solve the problem:
with a simple query
while using Doctrine MongoDB ODM
without looping over the subdocument-array in php
I'm using:
Mongo-Server: 2.4.5
PHP: 5.4.16
PHP-Mongo-Driver: 1.4.1
Composer:
"php": ">=5.3.3",
"symfony/symfony": "2.3.*",
"doctrine/orm": ">=2.2.3,<2.4-dev",
"doctrine/doctrine-bundle": "1.2.*",
"doctrine/mongodb-odm": "1.0.*#dev",
"doctrine/mongodb-odm-bundle": "3.0.*#dev"
This was a bug in ODM and should be fixed in PR #661. Please take a look and ensure that the revised test satisfies your use case.
Until the next beta of ODM is tagged, this will only reside in the master branch; however, that should match your 1.0.*#dev version requirement.
You can do this thing but in a "not easy" way becuse when you use EmbedMany or ReferenceMany in Doctrine becames a Doctrine ArrayCollection objects that implemented the ArrayCollection contains() method with (as explained here) with the PHP in_array() method with the $strict parameter set to true... this way works with ReferenceMany but not with EmbedMany!
The concept is: ArrayCollection -> array -> ArrayCollection
In the definition of Document
...
/**
* #ODM\EmbedMany(targetDocument="EmbeddedElement")
*/
private $elements = array();
...
public function getElements() { return $this->elements; }
public function setElements($elements) { $this->elements = $elements; }
public function addElement(Element $element) { $this->elements[] = $element; }
public function setElement(Element $element, $index=0)
{
$elementsArray = $this->elements->toArray();
$elementsArray[$index] = $element;
$this->elements = new ArrayCollection($elementsArray);
}
public function removeElement(Element $element)
{
$index = array_search($element, $this->elements->toArray(), $strict=false);
if($index !== false)
$this->elements->remove($index);
}
public function removeElementByIndex($index) { $this->elements->remove($index); }

How to duplicate and EmbeddedDocument in Doctrine (with MongoDb)

In in Doctrine ODM 1.0beta3, I need to take an #EmbeddedDocument from a Document, and to put it in another Document.
However when I try to do this, I get strange results.
There is a way to duplicate and EmbeddedObject?
EDIT:
The "strange" result is that a property of this embedded document is inserted alongside the other embedded documents.
More details:
I have a Queue embedded document with several Job documents in an #EmbedMany relationship:
Queue: -> [ Job, Job, Job]
Also some Jobs have another Job in a #EmbedOne property called $callback:
class Job {
/** #EmbedOne(targetDocument="Cron\Model\Document\Job") */
private $callback;
/** #Hash */
private $result;
}
/*
* #EmbeddedDocument
*/
class Queue {
/** #EmbedMany(targetDocument="Cron\Model\Document\Job") */
private $jobs = array();
//Getters and setters...
}
What I'm trying to do is to get the $callback Job, add a value to the $result hash and to enqueue this Job in the $jobs array().
What I get is the Job enqueued, but also the $result hash in the $jobs array, as if it was a Job document.
jobs array from mongodb:
...
[2] => Array
(
[parameters] => Array
(
[field] => office
[result] => "foo"
)
)
[3] => Array
(
[_id] => 638
[parameters] => Array
(
[field] => office
[result] => "foo"
)
)
The code is this:
$job = $queue->getLastJob();
$callback = $job->getCallback();
$params = $callback->getParameters();
$params['result'] = 'foo';
$callback->setParameters($params);
$queue->addJobs($callback);
I think you would tackle this problem as you would in any other language.
Just copy the object:
$some_obj["embedded_document"] = $other_obj["embedded_document"];
My understanding is that php makes a copy of the object instead of a reference and I've confirmed it locally.
What strange results are you getting exactly? If you provide some more information I can try to help.
Cheers!