Import a huge among of data from Elasticsearch 2 to Mongodb fails on memory limit - mongodb

I need help with import about 25 millions items from Elasticsearch to Mongodb. I wrote php script to do it but when the script reaches 16 millions items it fails on memory limit and throws me an error: VirtualAlloc() failed: [0x000005af] The paging file is too small for this operation to complete. I changed the system settings - virtual memory (paging file) to 100 000 according this web, but it is still not enough. I dont understand why it allocates so much memory. To get data from Elasticsearch I use scroll api. Look at the script:
<?php
error_reporting( E_ALL );
ini_set( 'memory_limit', -1 );
ini_set( 'max_execution_time', -1 );
/** #var \Nette\DI\Container $container */
$container = require( __DIR__ . '/../app/bootstrap.php' );
echo "----------------------------------------------------------------\n";
echo "--------------------- EVENT INDEX IMPORT -----------------------\n";
echo "----------------------------------------------------------------\n";
echo 'memory_limit: ' . ini_get( 'memory_limit' ) . "\n";
/** #var MongoConnect $mongo */
$mongo = $container->getService( 'mongo' );
/** #var \MongoDB\Collection $eventsCollection */
$eventsCollection = $mongo->selectCollection( 'Events', 'events' );
/** #var Elastica\Client $elastic */
$elastic = new Elastica\Client();
/** #var Elastica\Index $elasticIndex */
$elasticScrollData = $elastic->getIndex( 'event' )->request( '_search?scroll=10s', 'GET', ['size' => 250, 'sort' => ['_doc']] )->getData();
$countAll = $elasticScrollData['hits']['total'];
echo 'ES ALL ITEMS COUNT ' . $countAll . "\n";
$offset = 0;
saveToMongo( $elasticScrollData, $countAll, $offset, $elastic, $eventsCollection );
function saveToMongo( $scrollData, $countAll, $offset, \Elastica\Client $elastic, \MongoDB\Collection $mongoCollection )
{
$documents = [];
foreach ( $scrollData['hits']['hits'] as $item )
{
$doc = [];
$doc['ico'] = (array)$item['_source']['ico'];
$doc['data'] = $item['_source'];
if( isset( $item['_type'] ) ) $doc['type'] = $item['_type'];
if( isset( $item['_source']['key'] ) ) $doc['key'] = $item['_source']['key'];
if( isset( $item['_source']['action'] ) ) $doc['action'] = $item['_source']['action'];
if( isset( $item['_source']['publishDate'] ) ) $doc['publishDate'] = stringToDate( $item['_source']['publishDate'] );
if( isset( $item['_source']['generateDate'] ) ) $doc['generateDate'] = stringToDate( $item['_source']['generateDate'] );
if( isset( $item['_source']['eventDate'] ) ) $doc['eventDate'] = stringToDate( $item['_source']['eventDate'] );
$documents[] = $doc;
$offset++;
}
try
{
$mongoCollection->insertMany( $documents, ['ordered' => FALSE] );
echo '--- offest ' . ( $offset ) . ' OK' . "\n";
}
catch( \Exception $e )
{
echo '+++ insert exception: ' . $e->getMessage() . "\n";
}
if( $offset < $countAll )
{
$scrollData = $elastic->request( '_search/scroll', 'GET', ['scroll' => '10s', 'scroll_id' => $scrollData['_scroll_id']] )->getData();
saveToMongo( $scrollData, $countAll, $offset, $elastic, $mongoCollection );
}
}
function stringToDate( $string )
{
if( preg_match( '/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+\+[\d:]+$/', $string ) ) $format = 'Y-m-d\TH:i:s.uT';
elseif( preg_match( '/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+$/', $string ) ) $format = 'Y-m-d\TH:i:s.u';
elseif ( preg_match( '/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+[\d:]+$/', $string ) ) $format = 'Y-m-d\TH:i:sT';
elseif ( preg_match( '/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}$/', $string ) ) $format = 'Y-m-d\TH:i:s';
elseif ( preg_match( '/^\d{4}-\d{2}-\d{2}\+[\d:]+$/', $string ) ) $format = 'Y-m-dT';
elseif ( preg_match( '/^\d{4}-\d{2}-\d{2}$/', $string ) ) $format = 'Y-m-d';
return DateTime::createFromFormat( $format, $string );
}
echo "------------------------------------------------------------------------\n";
echo "------------------------- EVERYTHING IS DONE ---------------------------\n";
echo "------------------------------------------------------------------------\n";

Related

Gravity Forms - Get a Date value and adjust it, make it into a merge tag

So what I want to be able to do is take a field value, a date field, and add a set period of time to it and then make that into a merge tag that I can then add back into that value or use else where.
I know how to make a new merge tag, that's not the issue. My question is how do I get a field value to use in that calculation?
add_filter( 'gform_replace_merge_tags', 'new_date_plus_30', 10, 7 );
function new_date_plus_30( $text, $form, $entry, $url_encode, $esc_html, $nl2br, $format ) {
$merge_tag = '{date_plus_30}';
$new_date = date('m/d/Y', strtotime('+30 days'));
return str_replace( $merge_tag, $new_date, $text );
}
So where I do the new date calculation, I need to be able to pull in a field from that post and use it.
I was also thinking of doing a If / Else script where I would do the date calculation based on what was set in a form. So if a user said repeat this every 15 days, I would have something like:
add_filter( 'gform_replace_merge_tags', 'new_date_plus_30', 10, 7 );
function new_date_plus_30( $text, $form, $entry, $url_encode, $esc_html, $nl2br, $format ) {
if ( $form_id == 34 && $field_id == 2 && $value == 'add 30 days') {
$merge_tag = '{date_plus_30}';
$new_date = date('m/d/Y', strtotime('+30 days'));
}
else if ( $form_id == 34 && $field_id == 2 && $value == 'first of month') {
$merge_tag = '{first_of_month}';
$new_date = date('m/d/Y', strtotime('first of next month'));
}
}
return str_replace( $merge_tag, $new_date, $text );
}
But my issue is still the same. How can I use two filters at the same time? I assume I need to use the gform_get_input_value. Kindly review my code and give feedback is there other way?
Or maybe something like this...
add_filter( 'gform_merge_tag_filter', function ( $value, $merge_tag, $options, $field, $raw_value ) {
if ( $field->id == '2' && $value == 'first of the month') {
$merge_tag = '{the_next_date}';
$thedatetochange = 'Not sure how to get the date value here...';
$value = date('m/d/Y', strtotime($thedatetochange . 'first of the next month'));
return $value;
}
else if ( $field->id == '2' && $value == 'the 15th') {
$merge_tag = '{the_next_date}';
$thedatetochange = 'Not sure how to get the date value here...';
$the_first_date = date('m/d/Y', strtotime($thedatetochange . 'first of the next month' ));
$value = date('m/d/Y', strtotime($the_first_date . '+15 days' ));
return $value;
}
}, 10, 5 );
So after doing more digging, would I be able to use something like this to get the value of the field?
$theDateToChange = rgar( $entry, ‘3’);
This assumes that the field 3 is a date value. Would this work for retrieving the current entry date?
The $entry is passed through the gform_replace_merge_tags filter. You can fetch any field value from the $entry by its field ID. For example, if your field ID was 1:
$value = $entry[1];
Alternately, if you're open to capturing this modified date as a secondary Date field in your form, we have a snippet that can handle the functionality for you.
https://gravitywiz.com/populate-dates-gravity-form-fields/
new GW_Populate_Date( array(
'form_id' => 1,
'target_field_id' => 2,
'modifier' => '+30 days'
) );
So here is my current working code...
add_action( 'gform_admin_pre_render', 'add_merge_tags' );
function add_merge_tags( $form ) {
?>
<script type="text/javascript">
gform.addFilter('gform_merge_tags', 'add_merge_tags');
function add_merge_tags(mergeTags, elementId, hideAllFields, excludeFieldTypes, isPrepop, option){
mergeTags["custom"].tags.push({ tag: '{the_next_date}', label: 'The Next Date' });
return mergeTags;
}
</script>
<?php
//return the form object from the php hook
return $form;
}
add_action('wp', 'add_merge_tags');
/** MY MERGE TAGS HERE */
add_filter( 'gform_replace_merge_tags', 'new_date', 10, 7 );
function new_date( $value, $merge_tag, $options, $field, $raw_value, $entry, $text, $form, $url_encode, $esc_html, $nl2br, $format ) {
$pmoptions = $entry[7];
if ( $pmoptions == 'Monthly') {
$merge_tag = '{the_next_date}';
$old_date = $entry[2];
$new_date = date('m/d/Y', strtotime( $old_date . '+1 month'));
return str_replace( $merge_tag, $new_date, $text );
}
else if ( $pmoptions == 'Quarterly') {
$merge_tag = '{the_next_date}';
$old_date = $entry[2];
$new_date = date('m/d/Y', strtotime( $old_date . '+3 month'));
return str_replace($merge_tag, $new_date, $text);
}
}
apply_filters( 'gform_replace_merge_tags', $value, $merge_tag, $options, $field, $raw_value, $entry, $text, $form, $url_encode, $esc_html, $nl2br, $format );

Showing course image on custom page in Moodle

I have created a custom page on which all available courses are displayed. I have also uploaded the image for the course and now want to show the name of the course along with the image. I am able to get the names of courses from the database but how to get the image.
Try something like this
// Create a course_in_list object to use the get_course_overviewfiles() method.
require_once($CFG->libdir . '/coursecatlib.php');
$course = new course_in_list($courseid);
$outputimage = '';
foreach ($course->get_course_overviewfiles() as $file) {
if ($file->is_valid_image()) {
$imagepath = '/' . $file->get_contextid() .
'/' . $file->get_component() .
'/' . $file->get_filearea() .
$file->get_filepath() .
$file->get_filename();
$imageurl = file_encode_url($CFG->wwwroot . '/pluginfile.php', $imagepath,
false);
$outputimage = html_writer::tag('div',
html_writer::empty_tag('img', array('src' => $imageurl)),
array('class' => 'courseimage'));
// Use the first image found.
break;
}
}
echo $outputimage;
You may get course image directly from 'overviewfiles' files area.
function get_course_image()
{
global $COURSE;
$url = '';
require_once( $CFG->libdir . '/filelib.php' );
$context = context_course::instance( $COURSE->id );
$fs = get_file_storage();
$files = $fs->get_area_files( $context->id, 'course', 'overviewfiles', 0 );
foreach ( $files as $f )
{
if ( $f->is_valid_image() )
{
$url = moodle_url::make_pluginfile_url( $f->get_contextid(), $f->get_component(), $f->get_filearea(), null, $f->get_filepath(), $f->get_filename(), false );
}
}
return $url;
}

Modify code to auto complete on taxonomies instead of title

I've been trying for hours to modify this plugin to work with custom taxonomies instead of post titles and/or post content
http://wordpress.org/extend/plugins/kau-boys-autocompleter/
Added the code / part of the plugin where I think the modification should be done. I could post some stuff I tried but I think it would just confuse people, I don't think I ever came close. Normally I can modify code perfectly but just can't seem to find it. I've tried this method posted here.
http://wordpress.org/support/topic/autocomplete-taxonomy-in-stead-of-title-or-content
But it doesn't work. I've tried searching for possible solutions around his suggestion but I'm starting to think his suggestion isn't even close to fixing it.
if(WP_DEBUG){
error_reporting(E_ALL);
} else {
error_reporting(0);
}
header('Content-Type: text/html; charset=utf-8');
// remove the filter functions from relevanssi
if(function_exists('relevanssi_kill')) remove_filter('posts_where', 'relevanssi_kill');
if(function_exists('relevanssi_query')) remove_filter('the_posts', 'relevanssi_query');
if(function_exists('relevanssi_kill')) remove_filter('post_limits', 'relevanssi_getLimit');
$choices = get_option('kau-boys_autocompleter_choices');
$framework = get_option('kau-boys_autocompleter_framework');
$encoding = get_option('kau-boys_autocompleter_encoding');
$searchfields = get_option('kau-boys_autocompleter_searchfields');
$resultfields = get_option('kau-boys_autocompleter_resultfields');
$titlelength = get_option('kau-boys_autocompleter_titlelength');
$contentlength = get_option('kau-boys_autocompleter_contentlength');
if(empty($choices)) $choices = 10;
if(empty($framework)) $framework = 'jQuery';
if(empty($encoding)) $encoding = 'UTF-8';
if(empty($searchfields)) $searchfields = 'both';
if(empty($resultfields)) $resultfields = 'both';
if(empty($titlelength)) $titlelength = 50;
if(empty($contentlength)) $contentlength = 120;
mb_internal_encoding($encoding);
mb_regex_encoding($encoding);
$words = '%'.$_REQUEST['q'].'%';
switch($searchfields){
case 'post_title' :
$where = 'post_title LIKE "'.$words.'"';
break;
case 'post_content' :
$where = 'post_content LIKE "'.$words.'"';
default :
$where = 'post_title LIKE "'.$words.'" OR post_content LIKE "'.$words.'"';
}
$wp_query = new WP_Query();
$wp_query->query(array(
's' => $_REQUEST['q'],
'showposts' => $choices,
'post_status' => 'publish'
));
$posts = $wp_query->posts;
$results = array();
foreach ($posts as $key => $post){
setup_postdata($post);
$title = strip_tags(html_entity_decode(get_the_title($post->ID), ENT_NOQUOTES, 'UTF-8'));
$content = strip_tags(strip_shortcodes(html_entity_decode(get_the_content($post->ID), ENT_NOQUOTES, 'UTF-8')));
if(mb_strpos(mb_strtolower(($searchfields == 'post_title')? $title : (($searchfields == 'post_content')? $content : $title.$content)), mb_strtolower($_REQUEST['q'])) !== false){
$results[] = array(
'url' => get_permalink($post->ID),
'title' => highlightSearchString(strtruncate($title, $titlelength, true), $_REQUEST['q']),
'content' => (($resultfields == 'both')? highlightSearchString(strtruncate($content, $contentlength, false, '[...]', $_REQUEST['q']), $_REQUEST['q']) : '')
);
}
}
printResults($results, $framework);
function highlightSearchString($value, $searchString){
if((version_compare(phpversion(), '5.0') < 0) && (strtolower(mb_internal_encoding()) == 'utf-8')){
$value = utf8_encode(html_entity_decode(utf8_decode($value)));
}
$regex_chars = '\.+?(){}[]^$';
for ($i=0; $i<mb_strlen($regex_chars); $i++) {
$char = mb_substr($regex_chars, $i, 1);
$searchString = str_replace($char, '\\'.$char, $searchString);
}
$searchString = '(.*)('.$searchString.')(.*)';
return mb_eregi_replace($searchString, '\1<span class="ac_match">\2</span>\3', $value);
}
function strtruncate($str, $length = 50, $cutWord = false, $suffix = '...', $needle = ''){
$str = trim($str);
if((version_compare(phpversion(), '5.0') < 0) && (strtolower(mb_internal_encoding()) == 'utf-8')){
$str = utf8_encode(html_entity_decode(utf8_decode($str)));
}else{
$str = html_entity_decode($str, ENT_NOQUOTES, mb_internal_encoding());
}
if(mb_strlen($str)>$length){
if(!empty($needle) && mb_strpos(mb_strtolower($str), mb_strtolower($needle)) > 0){
$pos = mb_strpos(mb_strtolower($str), mb_strtolower($needle)) + (mb_strlen($needle) / 2);
$startToShort = ($pos - ($length / 2)) < 0;
$endToShort = ($pos + ($length / 2)) > mb_strlen($str);
// build the prefix and suffix
$prefix = $suffix;
if($startToShort){
$prefix = '';
}
if($endToShort){
$suffix = '';
}
// set maximum length
$length = $length - mb_strlen($prefix) - mb_strlen($suffix);
// get the start
if($startToShort){
$start = 0;
} elseif($endToShort){
$start = mb_strlen($str) - $length;
} else {
$start = $pos - ($length / 2);
}
// shorten the string
$string = mb_substr($str, $start, $length);
if($cutWord){
return $prefix.$string.$suffix;
} else {
$firstWhitespace = ($startToShort)? 0 : mb_strpos($string, ' ');
$lastWhitespace =($endToShort)? mb_strlen($string) : mb_strrpos($string, ' ');
return $prefix.' '.(!empty($lastWhitespace)? mb_substr($string, $firstWhitespace, ($lastWhitespace - $firstWhitespace)) : $string).' '.$suffix;
}
} else {
$string = mb_substr($str, 0, $length - mb_strlen($suffix));
return (($cutWord) ? $string : mb_substr($string, 0, mb_strrpos($string, ' ')).' ').$suffix;
}
} else {
return $str;
}
}
function printResults($results, $framework){
if($framework == 'scriptaculous'){
echo '<ul>';
foreach($results as $result){
echo ' <li>
<a href="'.$result['url'].'">
<span class="title">'.$result['title'].'</span>
<span style="display: block;">'.$result['content'].'</span>
</a>
</li>';
}
echo '</ul>';
} else {
foreach($results as $result){
echo str_replace(array("\n", "\r", '|'), array(' ',' ', '|'), '<span class="title">'.$result['title'].'</span><p>'.$result['content'].'</p>')
.'|'
.str_replace(array("\n", "\r", '|'), array(' ',' ', '|'), $result['url'])
."\n";
}
}
}

Magento - Get list of all Manufacturers with product count

I am using the following code to list all the manufacturers and it works like a charm:
$attribute = Mage::getModel('eav/entity_attribute')
->loadByCode('catalog_product', 'manufacturer');
$valuesCollection = Mage::getResourceModel('eav/entity_attribute_option_collection')
->setAttributeFilter($attribute->getData('attribute_id'))
->setStoreFilter(0, false);
$preparedManufacturers = array();
foreach($valuesCollection as $value) {
$preparedManufacturers[$value->getOptionId()] = $value->getValue();
}
if (count($preparedManufacturers)) {
echo "<h2>Manufacturers</h2><ul>";
foreach($preparedManufacturers as $optionId => $value) {
echo "<li>" . $value . " - (" . $optionId . ")</li>";
}
echo "</ul>";
}
What I am looking for is a way to display the number of products associated with each of the manufacturers. Can someone please tell me the way of doing this?
Many thanks
Not all mine, but works for me in 1.6+?
<?php
include_once 'app/Mage.php';
Mage::app();
$attribute = Mage::getModel('eav/entity_attribute')
->loadByCode('catalog_product', 'manufacturer');
$valuesCollection = Mage::getResourceModel('eav/entity_attribute_option_collection')
->setAttributeFilter($attribute->getData('attribute_id'))
->setStoreFilter(0, false);
$preparedManufacturers = array();
foreach ($valuesCollection as $value) {
$preparedManufacturers[$value->getOptionId()] = $value->getValue();
}
if (count($preparedManufacturers)) {
echo "<h2>Manufacturers</h2><ul>";
foreach ($preparedManufacturers as $optionId => $value) {
$collection = Mage::getModel('catalog/product')->getCollection();
$collection->addFieldToFilter(array(array('attribute' => 'manufacturer', 'eq' => $optionId)));
$mumberOfProducrt = count($collection);
echo "<li>" . $value . " - (" . $mumberOfProducrt . ")</li>";
}
echo "</ul>";
}
?>
This would work but won't be the most efficient:
foreach($valuesCollection as $value) {
$preparedManufacturers[$value->getOptionId()] = $value->getValue();
$collection = Mage::getModel('catalog/product')->getCollection();
$collection
->addAttributeToSelect('*') // '*' not efficient though
->addAttributeToFilter('manufacturer', array('eq' => $value->getOptionId()))
//->addAttributeToFilter('manufacturer', $value->getOptionId())
;
$count = $collection->->getSize();
}
It's an extra query for each manufacturer so it's not great, if you have some caching etc it won't be too bad though.
Working code should do it
$collection = Mage::getModel('catalog/product')->getCollection()->groupByAttribute('manufacturer')
->addFieldToFilter('status',Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
->addExpressionAttributeToSelect("count",'COUNT({{entity_id}})', 'entity_id');
Mage::getSingleton('cataloginventory/stock')->addInStockFilterToCollection($collection);

insert_id mysqli

I'm trying to return the inserted id from a mysql INSERT query. Each time I run the function I get 0 as the result. Can someone please explain at what point I can retrieve the value because although the script below executes I cannot retireve the inserted id. Probably done something stupid.
<?php
public function execSQL($sql, $params, $close){
$mysqli = new mysqli(DB_HOST,DB_USER,DB_PASSWORD,DB_NAME);
$stmt = $mysqli->prepare($sql) or die ("Failed to prepared the statement!");
call_user_func_array(array($stmt, 'bind_param'), $this->refValues($params));
$this->insert_id($this->connection);
$stmt->execute();
if($close){
$result = $mysqli->affected_rows;
} else {
$meta = $stmt->result_metadata();
while ( $field = $meta->fetch_field() ) {
$parameters[] = &$row[$field->name];
}
call_user_func_array(array($stmt, 'bind_result'), $this->refValues($parameters));
while ( $stmt->fetch() ) {
$x = array();
foreach( $row as $key => $val ) {
$x[$key] = $val;
}
$results[] = $x;
}
$result = $results;
}
$stmt->close();
$mysqli->close();
return $result;
}
?>
Check $mysqli->insert_id after executing insert query.