How do you scraping sites with DOM manipulation when loading sites - dom

Usually I have code snippets on what I'm working on, but this time I'm having trouble finding anything technologies for this.
What is the best way to web scrape sites that have a form to submit and then results page using dynamic DOM manipulation like adding div containers, updating data values using technology like angularjs or react.
Good example sites are southwest airlines or Booking.com.
I tried nightmare and phantoms, but they both seem to not work well on the wait and then retrieve.
Anyone have any technology suggestions or sites that explain what to do well?

Related

On Google Sites, is it possible to directly provide my own pages, stylesheets etc without going thru the creating process?

Now I have to pick a template and jump thru bunch of loops to create a page that I don't like. So we wonder if it's possible to just drop a few web pages and the css that we created on our own as our web site on Google Apps, doable?
AFAIK, that is not possible. I made a google site for a client last year, and it takes a lot of hoops to go through for customization. You can do quite a bit, it might not be readily noticeable. I found googling the things I wanted to do tended to yield the quickest results.

Why use WebMatrix Facebook helpers verus just using facebook generator?

I am wondering why would you use the html helper for something like the like button
then using the generator from facebook
Why use the Twitter helper, or the Gravatar helper for that matter? Or any helper?
WebMatrix is targeted at an audience of learners. This group is barely assumed to know HTML (although I realise that description covers a lot of experienced Web Forms developers too), so providing a reasonably consistent set of APIs to help make use of social networking widgets makes sense to me. It means that the learner can have Twitter feeds and Like buttons on their first site in a few seconds, rather than having to trawl all over the Internet to find out how these things are done, and then potentially getting confused by the developer material offered by these sites. If they achieve a degree of success early on, they will more likely persist with the framework.

Can WordPress handle these functionalities?

I'm a front-end designer/developer whose weapon of choice for the back-end is WordPress. Up to this point all of my projects involving WordPress were fairly basic and it has handled everything beautifully. I just landed a new client that wants some extra functionality built into his next project and I'm hoping some of you WordPress wizards can give me some good advice while I'm putting together the quote.
I'm trying to limit the need for any subcontracting for the back-end functionality, so my question is whether or not WordPress can handle the following (via plugins or light custom manipulation):
The idea behind the site is to be a community calendar based on location that Health Care providers can log in and post their events to, as well as participate in discussions, blogs and all the other WordPress goodness. The specific functionalities that I'm unsure of the best way to accomplish are:
Full featured calendar that members with access can add their own events to - must be searchable by date/type of event/location etc
Event generator module for members that integrates with calendar - includes upload field for images and forms for details event info
Interactive map to filter both of the above by location (I'm assuming this will need to be flash, but I'd rather find another solution if possible)
I know there are other solutions out there that may be more suited to this than WordPress (Drupal, custom build, etc) but if it's at all possible to tackle this as a one man show then I'm going to charge it head-on!
Stack Overflowers and fellow WordPress fans...your insight would be much appreciated. Thanks in advance for your time.
This graph grants your experience with your weapon of choice, but the results are still clear. You can still tackle this as a 1 man show, it will just take a bit of a learning curve to conquer the fundamentals of a CMS more suited to the task at hand. I'm sure plenty of WordPress affecionados will come along and strangle my reputation, but I've worked with both and have found that in terms of flexibility, WordPress is not king, and for the custom coding you are going to have to do (hope you have some PHP?), I feel that you will find it easier to integrate with another platform. This task will be difficult if not impossible to accomplish without writing code, even if there is a set of plugins that appear on the face to match your needs perfectly.
But anyway, since you probably don't really care that much about my opinion, for WordPress, your plugin options look like..
Calendar - Events Calendar
http://wordpress.org/extend/plugins/events-calendar/
The screenshots don't look terribly promising though.
Most plugins I have found are geared toward being administered from the admin panel, it may be difficult to provide a user interface to such plugins, and it does not look like the event calendar is an exception. An experienced developer should be able to hook into the event publishing code with relative ease, but it could be a frustrating experience for the inexperienced.
For interactive maps, the Google Maps API is very feature rich, and you should be able to adapt it to your suit your mapping needs, regardless of platform.
If you want all of your providers to have their own blog, etc, what was once the WordPress MU plugin, but is now core-bundled WordPress MS (multisite) is what you need.
This again may also prove rigid, and you may encounter difficulty trying to bend the iron of WordPress enabling all your multisite users to be able to post to a common community site. I've only built 2 platforms with MU, so I'm not positive about this.
To unapologetically reiterate my first point, what would be light custom code may turn impossibly frustrating using WordPress.
I like WordPress, and choose it often for my clients. I have never extended it to suit a larger project.
If you do decide to use it, I look forward to hopefully helping you with any questions you may have along the way, feel free to ask.

Events Website CMS/Framework Suggestions

I'm planning to build a website that has the following in the first Phase
Events List
Events Management by admin
Register for events (buy tickets)
News List
Manage News
Support Multi-Language
In the next phase i would like the site to be a social networking site (considering elgg)
I want the website to be light and fast. I've tried Joomla/Drupal. They seem to be slow.
Any recommendations for a framework/CMS?
I've been using Textpattern for a CMS based website I am developing, and so far it seems like it can handle all of the CMS work I push at it while staying out of the way as far as code. It has a bit of a learning curve like most CMS programs, but is pretty easy to pick up. You'll still have to build out the functionality for events (look into the plugin ZemEvents), as Textpattern starts out with just the base install and you add on to it as needed. You may need to handle E-commerce differently though, not sure if there are any plugins for Textpattern that could handle that.
I, personally second LocalPCGuy’s recommendation since Textpattern is, in my eyes, the most underestimated CMS in the market. I especially love it for its simple XML-like templating tags.
Talking about easy templating you might also want to check out the Python based Django framework. This is, by far, the fastest framework/cms I ever came across.

How to integrate vBulletin features into an external site

I have a web site I'm building and the client wants to have features from vBulletin (blog, forums) integrated into the site. Its not enough to simply add the sites skin to vBulletin. Is there a way to do this?
I would expect there to be documentation on how, if it is possible, to do such a thing but haven't been able to find anything.
I'd rather not connect and query the vBulletin database directly.
There is no proper API for this yet, so you'd either have to rely on things like RSS, or query the database directly. RSS won't get you old data, nor any forum structures, etc. just basics of new data.
After much research (see: cursing) I've found that external.php and blog_external.php do what I want though not quite as elegantly as I would like.
So if you want to incorporate forum threads into your web page then external.php is what you need. It appears to be a bit more customizable in that you can have it output in JavaScript, XML, RSS, and RSS Enclosure (podcasting).
If you want to incorporate blog posts you appear to be limited to RSS only. Like I said, less than ideal but at least its something.
There is more information here: http://www.vbulletin.com/docs/html/vboptions_group_external