Moving Wagtail pages in a migration - django-migrations

I'm restructuring my Wagtail app to remove an IndexPage that only has a single item in it, and moving that item to be a child of the current IndexPage's parent.
basically moving from this:
Page--|
|--IndexPage--|
|--ChildPages (there's only ever 1 of these)
to this:
Page--|
|--ChildPage
I've made the changes to the models so that this structure is used for creating new content and fixed the relevant views to point to the ChildPage directly. But now I want to migrate the current data to the new structure and I'm not sure how to go about it... Ideally this would be done in a migration so that we would not have to do any of this manipulation by hand.
Is there a way to move these ChildPage's up the tree programmatically during a migration?

Unfortunately there's a hard limitation that (probably) rules out the possibility of doing page tree adjustments within migrations: tree operations such as inserting, moving and deleting pages are implemented as methods on the Page model, and within a migration you only have access to a 'dummy' version of that model, which only gives you access to the database fields and basic ORM methods, not those custom methods.
(You might be able to work around this by putting from wagtail.wagtailcore.models import Page in your migration and using that instead of the standard Page = apps.get_model("wagtailcore", "Page") approach, but I wouldn't recommend that - it's liable to break if the migration is run at a point in the migration sequence where the Page model is still being built up and doesn't match the 'real' state of the model.)
Instead, I'd suggest writing a Django management command to do the tree manipulation - within a management command it is safe to import the Page model from wagtailcore, as well as your specific page models. Page provides a method move(target, pos) which works as per the Treebeard API - the code for moving your child pages might look something like:
from myapp.models import IndexPage
# ...
for index_page in IndexPage.objects.all():
for child_page in index_page.get_children():
child_page.move(index_page, 'right')
index_page.delete()

Theoretically it should be possible to build a move() using the same sort of manipulations that Daniele Miele demonstrates in Django-treebeard and Wagtail page creation. It'd look something like this Python pseudocode:
def move(page, target):
# assuming pos='last_child' but other cases follow similarly,
# just with more bookkeeping
# first, cut it out of its old tree
page.parent.numchild -= 1
for sib in page.right_siblings: # i.e. those with a greater path
old = sib.path
new = sib.path[:-4] + (int(sib.path[-4:])-1):04
sib.path = new
for nib in sib.descendants:
nib.path = nib.path.replace_prefix(old, new)
# now, update itself
old_path = page.path
new_path = target.path + (target.numchild+1):04
page.path = new_path
old_url_path = page.url_path
new_url_path = target.url_path + page.url_path.last
page.url_path = new_url_path
old_depth = page.depth
new_depth = target.depth + 1
page.depth = new_depth
# and its descendants
depth_change = new_depth - old_depth
for descendant in page.descendants:
descendant.path = descendant.path.replace_prefix(old_path, new_path)
descendant.url_path = descendant.url_path.replace_prefix(old_path, new_path)
descendant.depth += depth_change
# finally, update its new parent
target.numchild += 1
The core concept that makes this manipulation simpler than it looks is: when a node gets reordered or moved, all its descendants need to be updated, but the only update they need is the exact same update their ancestor got. It's applied as a prefix replacement (if str) or a difference (if int), neither of which requires knowing anything about the descendant's exact value.
That said, I haven't tested it; it's complex enough to be easy to mess up; and there's no way of knowing if I updated every invariant that Wagtail cares about. So there's something to be said for the management command way as well.

Related

CQ - Moving content from one page to another

I realize that this is a pretty specific question but I would imagine someone has run into this before. So I've got about fifty pages or so that were created about a year ago. We're trying to revamp the page with new components specifically in the header and the footer. Except the content in the main-content area will stay the same. So I'm trying to move over everything from the old pages to the new pages but just keep the main-content area. The problem is I can't just change the resource type on the old page to point to the new page components because the content is different and I'll have a bunch of nodes in the header and footer that I don't want. For example here is my current content structure:
Old Content
star-trek
jcr:content
header
nav
social
chat
main-content
column-one
column-two
footer
sign-up
mega-menu
New Content
star-wars
jcr:content
masthead
mega-menu
main-content
column-one
column-two
bottom-footer
left-links
right-links
Does anybody have any ideas on how to move just the content in the main-content node and somehow remove the other nodes. I'm trying to somehow do this programmatically cause I don't want to create 50 pages from scratch. Any help is appreciated!
You can use the JCR API to move things around at will, I would
Block users from accessing the content in question. Can be done with temporary ACLs, or by closing access on the front-end if you can.
Run a script or servlet that changes the content using JCR APIs
Check the results
Let users access the content again
For the content modification script I suggest a script that modifies a single page (i.e. you call it with an HTTP request that ends in /content/star-trek.modify.txt) so that you can run it either on a single page, for testing, or on a group of pages once it's good.
The script starts form the current node, recurses into it to find nodes that it knowns how to modify (based on their sling:resourceType), modifies them and reports what it did in the logs or on its output.
To modify nodes the script uses the JCR Node APIs to move things around (and maybe Worskpace.move).
It is indeed possible to write a code which does what you need :
package com.test;
import java.io.File;
import java.io.IOException;
import javax.jcr.ItemExistsException;
import javax.jcr.Repository;
import javax.jcr.RepositoryException;
import javax.jcr.Session;
import javax.jcr.SimpleCredentials;
import javax.jcr.Node;
import org.apache.jackrabbit.commons.JcrUtils;
import org.apache.jackrabbit.core.TransientRepository;
import org.xml.sax.SAXException;
public class test {
public void test(Document doc) throws RepositoryException {
try {
// Create a connection to the CQ repository running on local host
Repository repository = JcrUtils
.getRepository("http://localhost:4502/crx/server");
System.out.println("rep is created");
// Create a Session
javax.jcr.Session session = repository.login(new SimpleCredentials(
"admin", "admin".toCharArray()));
System.out.println("session is created");
String starTrekNodePath = "/content/path/";
String starWarsNodePath = "/content/anotherPath"
Node starTrekpageJcrNode = null;
Node starWarstext = null;
setProperty(java.lang.String name, Node value)
boolean starTrekNodeFlag = session.nodeExists(starTrekNodePath);
boolean starWarsNodeFlag = session.nodeExists(starWarsNodePath);
if (starTrekNodeFlag && starWarsNodeFlag) {
System.out.println("to infinity and beyond");
Node starTrekNode = session.getNode(starTrekNodePath);
Node starWarsNodeFlag = session.getNode(starWarsNodePath);
//apply nested looping logic here; to loop through all pages under this node
//assumption is that you have similar page titles or something
//on these lines to determine target and destination nodes
//2nd assumption is that destination pages exist with the component structures in q
//once we have the target nodes, the following segment should be all you need
Node starTrekChildNode = iterator.next();//assuming you use some form of iterator for looping logic
Node starWarsChildNode = iterator1.next();//assuming you use some form of iterator for looping logic
//next we get the jcr:content node of the target and child nodes
Node starTrekChildJcrNode = starTrekChildNode.getNode("jcr:content");
Node starWarsChildJcrNode = starWarsChildNode.getNode("jcr:content");
// now we fetch the main-component nodes.
Node starTrekChildMCNode = starTrekChildJcrNode.getNode("main-content");
Node starWarsChildMCNode = starWarsChildJcrNode.getNode("main-content");
//now we fetch each component node
Node starTrekChildC1Node = starTrekChildMCNode.getNode("column-one");
Node starTrekChildC2Node = starTrekChildMCNode.getNode("column-two");
Node starWarsChildC1Node = starWarsChildMCNode.getNode("column-one");
Node starWarsChildC2Node = starWarsChildMCNode.getNode("column-two");
// fetch the properties for each node of column-one and column-two from target
String prop1;
String prop2;
PropertyIterator iterator = starTrekChildC1Node.getProperties(propName);
while (iterator.hasNext()) {
Property prop = iterator.nextProperty();
prop1 = prop.getString();
}
PropertyIterator iterator = starTrekChildC2Node.getProperties(propName);
while (iterator.hasNext()) {
Property prop = iterator.nextProperty();
prop2 = prop.getString();
}
// and now we set the values
starWarsChildC1Node.setProperty("property-name",prop1);
starWarsChildC2Node.setProperty("property-name",prop2);
//end loops as appropriate
}
Hopefully this should set you on the right track. You'd have to figure out how you want to identify destination and target pages, based on your folder structure in /content, but the essential logic should be the same
The problem with the results I'm seeing here is that you are writing servlets that execute JCR operations to move things around. While technically that works, it's not really a scalable or reusable way to do this. You have to write some very specific code, deploy it, execute it, then delete it (or it lives out there forever). It's kind of impractical and not totally RESTful.
Here are two better options:
One of my colleagues wrote the CQ Groovy Console, which gives you the ability to use Groovy to script changes to the repository. We frequently use it for content transformations, like you've described. The advantage to using Groovy is that it's script (not compiled/deployed code). You still have access to the JCR API if you need it, but the console has a bunch of helper methods to make things even easier. I highly recommend this approach.
https://github.com/Citytechinc/cq-groovy-console
The other approach is to use the Bulk Editor tool in AEM. You can export a TSV of content, make changes, then reimport. You'll have to turn the import feature on using an administrative function, but I've used this with some success. Beware, it's a bit buggy though, with array value properties.

CQ5 / AEM5.6 Workflow: Access workflow instance properties from inside OR Split

TL;DR version:
In CQ workflows, is there a difference between what's available to the OR Split compared to the Process Step?
Is it possible to access the /history/ nodes of a workflow instance from within an OR Split?
How?!
The whole story:
I'm working on a workflow in CQ5 / AEM5.6.
In this workflow I have a custom dialog, which stores a couple of properties on the workflow instance.
The path to the property I'm having trouble with is: /workflow/instances/[this instance]/history/[workItem id]/workItem/metaData and I've called the property "reject-or-approve".
The dialog sets the property fine (via a dropdown that lets you set it to "reject" or "approve"), and I can access other properties on this node via a process step (in ecma script) using:
var actionReason;
var history = workflowSession.getHistory(workItem.getWorkflow());
// loop backwards through workItems
// and as soon as we find a Action Reason that is not empty
// store that as 'actionReason' and break.
for (var index = history.size() - 1; index >= 0; index--) {
var previous = history.get(index);
var tempActionReason = previous.getWorkItem().getMetaDataMap().get('action-message');
if ((tempActionReason != '')&&(tempActionReason != null)) {
actionReason = tempActionReason;
break;
}
}
The process step is not the problem though. Where I'm having trouble is when I try to do the same thing from inside an OR Split.
When I try the same workflowSession.getHistory(workItem.getWorkflow()) in an OR Split, it throws an error saying workItem is not defined.
I've tried storing this property on the payload instead (i.e. storing it under the page's jcr:content), and in that case the property does seem to be available to the OR Split, but my problems with that are:
This reject-or-approve property is only relevant to the current workflow instance, so storing it on the page's jcr:content doesn't really make sense. jcr:content properties will persist after the workflow is closed, and will be accessible to future workflow instances. I could work around this (i.e. don't let workflows do anything based on the property unless I'm sure this instance has written to the property already), but this doesn't feel right and is probably error-prone.
For some reason, when running through the custom dialog in my workflow, only the Admin user group seems to be able to write to the jcr:content property. When I use the dialog as any other user group (which I need to do for this workflow design), the dialog looks as though it's working, but never actually writes to the jcr:content property.
So for a couple of different reasons I'd rather keep this property local to the workflow instance instead of storing it on the page's jcr:content -- however, if anyone can think of a reason why my dialog isn't setting the property on the jcr:content when I use any group other than admin, that would give me a workaround even if it's not exactly the solution I'm looking for.
Thanks in advance if anyone can help! I know this is kind of obscure, but I've been stuck on it for ages.
a couple of days ago i ran into the same issue. The issue here is that you don't have the workItem object, because you don't really have an existing workItem. Imagine the following: you are going through the workflow, you got a couple of workItems, with means, either process step, either inbox item. When you are in an or split, you don't have existing workItems, you can ensure by visiting the /workItems node of the workflow instance. Your workaround seems to be the only way to go through this "issue".
I've solved it. It's not all that elegant looking, but it seems to be a pretty solid solution.
Here's some background:
Dialogs seem to reliably let you store properties either on:
the payload's jcr:content node (which wasn't practical for me, because the payload is locked during the workflow, and doesn't let non-admins write to its jcr:content)
the workItem/metaData for the current workflow step
However, Split steps don't have access to workItem. I found a fairly un-helpful confirmation of that here: http://blogs.adobe.com/dmcmahon/2013/03/26/cq5-failure-running-script-etcworkflowscriptscaworkitem-ecma-referenceerror-workitem-is-not-defined/
So basically the issue was, the Dialog step could store the property, but the OR Split couldn't access it.
My workaround was to add a Process step straight after the Dialog in my workflow. Process steps do have access to workItem, so they can read the property set by the Dialog. I never particularly wanted to store this data on the payload's jcr:content, so I looked for another location. It turns out the workflow metaData (at the top level of the workflow instance node, rather than workItem/metaData, which is inside the /history sub-node) is accessible to both the Process step and the OR Split. So, my Process step now reads the workItem's approveReject property (set by the Dialog), and then writes it to the workflow's metaData node. Then, the OR Split reads the property from its new location, and does its magic.
The way you access the workflow metaData from the Process step and the OR Split is not consistent, but you can get there from both.
Here's some code: (complete with comments. You're welcome)
In the dialog where you choose to approve or reject, the name of the field is set to rejectApprove. There's no ./ or anything before it. This tells it to store the property on the workItem/metaData node for the current workflow step under /history/.
Straight after the dialog, a Process step runs this:
var rejectApprove;
var history = workflowSession.getHistory(workItem.getWorkflow());
// loop backwards through workItems
// and as soon as we find a rejectApprove that is not empty
// store that as 'rejectApprove' and break.
for (var index = history.size() - 1; index >= 0; index--) {
var previous = history.get(index);
var tempRejectApprove = previous.getWorkItem().getMetaDataMap().get('rejectApprove');
if ((tempRejectApprove != '')&&(tempRejectApprove != null)) {
rejectApprove = tempRejectApprove;
break;
}
}
// steps up from the workflow step into the workflow metaData,
// and stores the rejectApprove property there
// (where it can be accessed by an OR Split)
workItem.getWorkflowData().getMetaData().put('rejectApprove', rejectApprove);
Then after the Process step, the OR Split has the following in its tabs:
function check() {
var match = 'approve';
if (workflowData.getMetaData().get('rejectApprove') == match) {
return true;
} else {
return false;
}
}
Note: use this for the tab for the "approve" path, then copy it and replace var match = 'approve' with var match = 'reject'
So the key here is that from a Process step:
workItem.getWorkflowData().getMetaData().put('rejectApprove', rejectApprove);
writes to the same property that:
workflowData.getMetaData().get('rejectApprove') reads from when you execute it in an OR Split.
To suit our business requirements, there's more to the workflow I've implemented than just this, but the method above seems to be a pretty reliable way to get values that are entered in a dialog, and access them from within an OR Split.
It seems pretty silly that the OR Split can't access the workItem directly, and I'd be interested to know if there's a less roundabout way of doing this, but for now this has solved my problem.
I really hope someone else has this same problem, and finds this useful, because it took me waaay to long to figure out, to only apply it once!

Django Tastypie, Remove Elements From ManyToMany Fields

I am using Tastypie, Django for my project.
To Update a many to many field I have used save_m2m hook.
def save_m2m(self, bundle):
for field_name, field_object in self.fields.items():
if not getattr(field_object, 'is_m2m', False):
continue
if not field_object.attribute:
continue
if field_object.readonly:
continue
related_mngr = getattr(bundle.obj, field_object.attribute)
related_objs = []
print bundle.data[field_name]
for related_bundle in bundle.data[field_name]:
try:
stock = Stock.objects.get(nse_symbol = related_bundle.obj.nse_symbol)
print stock.__dict__
except Stock.DoesNotExist as e:
dataa = {"error_message": e}
raise ImmediateHttpResponse(response=HttpBadRequest(content=json.dumps(dataa), content_type="application/json; charset=UTF-8"))
related_objs.append(stock)
related_mngr.add(*related_objs)
Now I want to remove elements from the same many to many field.
How should I achieve this. Do I have to send a patch request or delete request and how to handle this.
I am begineer in tastypie. I googled it some time and I couldn't find a proper way. Please guide me how to complete this.
Thanks.
I've thought a lot about handing m2m relationships, since most of our app depends on m2m links.
I've settled for the approach of an update method. Pass in the all the references of the relationships you want changed (add and remove), then update the db accordingly. We only pass in the changed values, since if you have a paginated list, you only want to update the items the user has identified. Generally I use a custom hook for this defined in override_urls.
I used to have a separate add and remove method, which worked well until we changed the gui and allowed users simply to change checkboxes. In that approach having an update method was much more useful. You'll have to decide on which method suits your application the best.

Zend Framework / Form Element is rendering as a text box rather than a dropdown box

I have the following in a config.ini file: (Zend_Form_Element)
site_status.name = "site_status"
site_status.type = "select"
site_status.label = "Status"
site_status.options.multiOptions.active.key = "Active"
site_status.options.multiOptions.active.value = "Active"
site_status.options.multiOptions.active.key = "Inactive"
site_status.options.multiOptions.active.value = "Inactive"
As you can see this is supposed to be a dropdown (select) box, however it is being rendered as a standard text box. What am I doing wrong?
--> Edit
Rather than tying the elements to a form, I am trying to tie them to a database: In my code it would look something like this:
[{tablename}] // the table name would represent a section in the ini
{column}.name = "{column_name/form_field_id}";
{column}.type = "{form_element_type}"
{column}.label = "{form_element_label}"
...
From there I would pull in the database table(s) that the form would represent data for (one or more tables as necessary). As far as the reasoning for this approach is that (down the road), I want to define (either by ini or some other storage method), a configuration file that would be a list of fields/elements that belong to a specific form (that a non-programmer type could easily edit), that the 'generic' form class would read, pull in the element info, and create the form on the fly.
I do realize however this poses another problem which I haven't yet figured out, and that is how to use table lookups for select elements (without coding the database retrieval of the lookup into the form, so that a non-user could easily just define it without any programming, purely configuration, but that is a whole other topic not part of my question here. (and I think I have viable ideas/solutions to that part of the problem anyhow) -- extra config entries and a generic routine pretty much.
I hope that clarifies my thought process and reason why I am doing it the way I am in the example above.
I have not yet played with using a Zend_Config to construct an instance of Zend_Form.
But a look at the code suggests that Zend_Form::addElement() doesn't directly take a Zend_Config instance as a param. Rather, it looks like you need pass your Zend_Config instance to the form constructor. It also seems that the config format needs to be a little deeper in order to map config keys to setXXX() calls.
In path/to/config/myForm.ini:
[myForm]
myForm.elements.site_status.name = "site_status"
myForm.elements.site_status.type = "select"
myForm.elements.site_status.label = "Status"
myForm.elements.site_status.options.multiOptions.active.key = "Active"
myForm.elements.site_status.options.multiOptions.active.value = "Active"
myForm.elements.site_status.options.multiOptions.inactive.key = "Inactive"
myForm.elements.site_status.options.multiOptions.inactive.value = "Inactive"
Then instantiating:
$formConfig = new Zend_Config_Ini('path/to/config/myForm.ini', 'myForm');
$form = new Zend_Form($formConfig);
Not tested, but looking at this example:
Using Zend_Form with Zend_Config - Andrew Vayanis
it feels like it should go something like the above.
Update
In view of the comments/feedback from #Aaron, two more approaches.
We could extend Zend_Form, implementing a method called something like addElementByConfig in which we would pass the shallow Zend_Config instance that describes the element itself. In fact, we could even just override addElement(), taking a recursive approach: if the first param is an instance of Zend_Config, then call addElement() using the component data.
If the atomicity and re-usability are the primary benefits we seek in using Zend_Config to describe an element, then perhaps we just make a custom element extending Zend_Form_Element. Then we could use these elements in any forms we wish.

Problem importing ZEXP files programmatically

I'm developing a Plone Product that needs to import objects programmatically previously exported to ZEXP files. It's working perfectly, except the navigation bar. When one object is imported, it does so correctly, but the navication bar is not updated. The object can be accessed through it's URL and it's parent container contents tab.
Bellow is the code I used to import the objects. It's based on zope's ObjectManager._importObjectFromFile implementation.
def importDocument( app, fileName, container ):
app._p_jar.sync()
owner = 1
connection = container._p_jar
ob = connection.importFile( config.REMOTE_DIR + fileName, customImporters={ magic: importXML, } )
id = ob.id
if hasattr(id, 'im_func'): id = id()
try:
container._setObject( id, ob, set_owner = owner, suppress_events=False )
except AttributeError:
print "AttributeError"
# Try to make ownership implicit if possible in the context
# that the object was imported into
ob = container._getOb( id )
ob.manage_changeOwnershipType( explicit = 0 )
transaction.commit()
return True
I've noticed that the _setObject implementation fires an ObjectAddedEvent event in it's code, and it's after that event that the menu gets updated when I use the ZMI interface to import an object, so I figure something is listening to this event and handling the menu, but oddly, it doesn't happen when using my code.
Generally speaking, importing zexp objects is not supported (in part due to cases like this where unexpected or unintended results may occur). If it works, great. If it doesn't, you are "on your own" and probably better off copying the Data.fs file to a new software stack.
That said, I'm not sure I understand why clear and rebuild the catalog (ZMI -> portal_catalog -> tab 'advance' -> 'clear & rebuild') is not the answer here. According to its description its job is to "walk the entire portal looking for content objects which should be indexed in the catalog and index them".
Unless I misunderstand, you've just described a situation where newly imported content "should be indexed" because it hasn't been indexed yet.
If you are worried about the length of time required to clear and rebuild, try running it from the command line with something like this:
http://svn.plone.org/svn/plone/plone.org/Products.PloneOrg/trunk/scripts/catalog_rebuild.py
If you are worried about crawling the whole site, then call indexObject() on each object (http://dev.plone.org/plone/browser/plone.org/Products.PloneOrg/trunk/scripts/catalog_rebuild.py#L109)
Maybe try manually rebuilding the whole catalog after the import is complete? It might give you more hints to what is wrong ...
ZMI -> portal_catalog -> tab 'advance' -> 'clear & rebuild'.
You may need to "publish" the object after import to make it visible.
Use the manage_importObject method instead.