Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a serious question. Is it ever ethical to ignore the presence of a robots.txt file on a website? These are some of the considerations I've got in mind:
If someone puts a web site up they're expecting some visits. Granted, web crawlers are using bandwidth without clicking on ads that may support the site but the site owner is putting their site on the web, right, so how reasonable is it for them to expect that they'll never get visited by a bot?
Some sites apparently use a robots.txt exactly in order to keep their site from being crawled by Google or some other utility that might grab prices and therefore allow people to do price comparisons easily. They have private search engines on the site so they obviously want people to be able to search the site; apparently they just don't want people to be able to easily compare their information with other vendors.
As I said, I'm not trying to be argumentative; I would just like to know if anyone has ever come up with a case where it's ethically permissible to ignore the presence of a robots.txt file? I cannot think of a case where it's permissible to ignore the robots.txt mainly because people (or businesses) are paying money to put up their web sites so they should be able to tell the Googles/Yahoos/Other SE's of the world that they don't want to be on their indices.
To put this discussion in context, I'd like to create a price comparison website and one of the major vendors has a robots.txt that basically prevents anyone from grabbing their prices. I'd like to be able to get their information but, as I said, I can't justify simply ignoring the wishes of the site owner.
I have seen some very sharp discussion here and that's why I would like to hear the opinions of developers that follow Stack Overflow.
By the way, there is some discussion of this topic on a Hacker News question but they seem to mainly focus on the legal aspects of this.
Arguments:
A robots.txt file is an implied license, especially since you are aware of it. Thus, continuing to scrape their site could be seen as unauthorized access (i.e., hacking). Sucks, but arguments like this have been made in other legal cases recently (not directly related to robots.txt, but in relation to other "passive controls".)
Grabbing prices violates no copyright law, including DMCA, since copyright does not include factual information, only creative.
Ethically, you should not grab prices because the vendor should have the ability to change prices without worrying about being accused of a bait/switch by people coming from your site.
Have you taken the high road, explaining the site to them and saying you'd love to include them in your list of vendors? Maybe they will love the idea and actually expose the data in a way that is easy for you to consume and less resource-intensive for them to produce.
There are no laws written directly about robots.txt because netiquette is generally followed. Don't be one of the "bad guys."
Some people filter robots because they use URL links to perform "actions" like adding things to carts, and robots leave them with massive numbers of abandoned shopping carts in their database.
Some people filter robots because they have exclusive prices that they can't advertise openly based on agreements with their vendors. You could be putting them in a bad position by exposing those prices on your site.
In this economy, if a company doesn't want to do everything possible to advertise themselves, it's their own fault that you don't include them.
The other use of robots.txt is to help protect web spiders from themselves. It's relatively easy for a web spider to get mired in an infinitely deep forest of links, and a properly constructed robots.txt file will tell the spider that "you don't need to go here".
Many people have tried to build businesses off building "price comparison" engines that scraped major sites.
Once you start getting any sort of traffic/revenue to speak of, you will receive a cease and desist. It's happened to dozens, if not hundreds of projects. I even worked on a small project that received a C&D from Craigslist.
You know how they say "It's easier to ask forgiveness than it is to get permission"? It doesn't hold true with page scraping. Get permission, or you will be hearing from their lawyers.
If you're lucky, it'll be early on, when you've got nothing to lose. If it's late, you may lose your business and all your work overnight, with a single letter.
Getting permission shouldn't be hard. Unless you're doing something sneaky, you're likely going to drive them additional traffic. Hell, once your product takes off, sites may be begging you, or even paying you to add their data.
One reason we allow robots to dig through the web without complaint is that we have a way to stop them if we want to. Protects both sides.
Remember the uproar when Cuil's robots were accused of going over-the-top, apparently acting like a DoS attack in some cases and using up bandwidth allowances of some small sites?
If too many people violate robots.txt we might get something worse.
"No" means "no".
To answer the narrow question, for the price comparison website you're probably best grabbing the price in real time, rather then scrapping the database in advance. Hard to imagine that being a problem.
An interesting IRL version of story involving The Harvard Coop:
Coop Calls Cops On ISBN Copiers.
Short answer: No.
On the narrow issue: If a seller says that their prices are secret, I think you have to respect that. I'd contact them and ask if they really don't want price comparison engines like yours to include them, or if the "no trespassing" sign is for technical reasons. If the latter, perhaps they'll provide you with an alternative. If the former, then I'd say too bad, they don't get included, they lose some business, and it's their problem.
Tangential rant: Personally, I get pretty annoyed with companies that make me jump through hoops to find out the price of their products, places that make me call and talk to a salesman so he can give me a hard-sell pitch, or worse, make me give them my phone number so their salesman can call and harass me. I figure that if they're afraid to tell me the price, it probably means that it's too high.
In general: A robots.txt file is like a "No Trespassing" sign. It's the owner's right to say who is allowed on their property. If you think their reasons are dumb, you can politely suggest they take the sign down. But you don't have the right to disregard their wishes. If someone puts a No Trespassing sign on his yard, and I say, "Hey, I just want to take a quick short cut, what's the big deal?" -- Maybe I'm stepping on his prized Bulgarian violet bulbs and destroying a valuable investment. Maybe I'm crossing his people's sacred burial ground and offending their religious sensibilities. Or maybe he's just an ornery jerk. But it's still his property and his right. Oh, and if I fall into the dangerous sinkhole after ignoring the No Trespassing sign, who's to blame? (In America, I could probably still sue him for all he's worth despite the fact that he warned me, but is that right?)
I'm showing some ignorance here, but I always thought a bot was something only sent out by a search engine. Like Google or Yahoo.
Thus, if you wrote an application that searched content on the internet, I wouldn't consider that a search engine bot, which to my knowledge is what robots.txt is trying to block.
But this may just be selective ignorance, because I might do it until the webmaster of that site contacted me and asked me to stop :)
If people make it available to public access, they shouldn't try to put limits on it. Adding a robots.txt file to your site is the equivalent to putting a sign on your lawn that says "Please don't look at me."
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I've just picked up a contract to sort out a vipers-nest of e-commerce websites that a previous 'developer' left for one of my clients. There's about a couple of dozen of them using a custom shopping cart and CMS system that's too embedded to dump and works well enough, but desperately needs cleaning up, re-factoring, and bug fixing, so a reasonably substantial recoding job.
As part of this my client is desperate to ensure the best search engine placement he can get. Like many developers I've a nodding acquaintance with the idea, but no real knowledge, and it seems that it would be helpful to get up to speed on this so I can build appropriately into the code.
So can people advise on useful quality resources - books, websites, blogs etc? I do not wish to obsess over every last detail on this (he can use a specialist if he decides to pull every last ounce out - although I've always regarded such as little better than snake-oil peddlers), but I would like to build code and reconfigure templates in a manner that helps rather than hinders placement.
Look at Wikipedia with styles off. See how they order their content? See how they use correct tags to label the content? These are the keys to long term success.
The most important SEO advice is to create a semantic, logical site. The content comes first and is ordered by importance. Use the correct tags, don't do tables. Then apply styles. Then apply script to make it fancy. (Like Tomas said)
Know the difference between what is content (pictures of your products) and not (your logo tiled on the background). Basically do your best to present good content in a good way. You can't game the system for long term gains.
This will give you long term placement. Most SEO companies just do tricks or links farms or worry about keywords and meta tags, so they are temporary at best. I think it is good to assume that Google works just as hard to drop that crap from it's index as the SEO marketers do to include it.
According to Google's patent
Age of domain.
Important sites that link to you.
Content. Make it real and accurate.
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20050071741&OS=20050071741&RS=20050071741
A main concern when thinking SEO should always be usability. Make sure the mark-up is semantic, make sure the site works without javascript, css and images - in other words, make sure that 'low-level browsers' are able to read the contents of your page. Think of optimizing the site for a screen reader for the blind and visually impaired, and you will automatically cover the search engines, because that is exactly what they are - blind browsers.
A standard no-no is a table layout, mainly because it does not order the site content by relevance. To a human being reading your site, it is no problem looking two inches down instead of at the very top of your page, but the screenreader reads the code, not the visual view.
Thus, you should make sure that you have the important things - a high-level heading (preferrably <h1> or <h2>), the main menu (in a <ul> list) and the content in divs at the top of the source, and less important content (the logo, banners, quickmenus etc that aren't really crucially needed to be accessible) further down. You are always able to re-order where the stuff is actually shown in your css.
New tag to use called CANONICAL can now also be used, from Google, click HERE
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
A client of ours is a membership organization and they are looking for functionality that seems closely aligned with Google Sites capabilities.
They want a system where their members can have a content managed site of their own that one or more admins can create by submitting a simple form.
The member organization could then add/remove pages, add/edit/remove content, add their own users, modify their color scheme and layout.
They would like the ability to have a url structure like, "member_org_url_to_be_named/member_name" - but it could also be subdomains (i.e. "member_name.member_org_url_to_be_named").
So they need a security hierarchy to be able to have different levels of users:
Admin - can add/edit/remove sites, users, etc.
Member Admin - can add/edit content within their site, add users that are also able to add/edit content within their site.
Member user - can add/edit content within their site.
From what I've seen and read, Google Sites seems to be able to handle this functionality. It's a little difficult to get in touch with someone there who would be able to tell me this definitively, however. So I'm wondering if there are any other platforms that might be able to handle this workflow.
Obviously, I'd love to hear from anyone who has implemented a system like this before. I'd also love to hear from anyone who has actually used Google Sites.
(Disclaimer: I work for Google. I don't know much about Sites though.)
Have you actually tried to use Google Sites for this? It strikes me that it shouldn't take very long to give it a whirl. If you have any Sites-specific questions, the Google Sites help centre and user forum are probably good starting places.
This sounds like content management with roles. Drupal fits this purpose pretty much perfectly.
http://drupal.org/
I've used Google Sites (the free "standard edition") a very little bit, it was easy to setup + easy to reconfigure my DNS records via nearlyfreespeech.net to setup CNAME and MX records to a domain I own.
The mailing list stuff works nicely. The site editing is very easy for anyone to use but a bit slllllooooowwww and somewhat clumsy, and doesn't appear to "play nicely" with the concept of uploading/downloading via FTP/SFTP/etc. I don't like the idea of my group's users spending all this time developing a website, that I can't backup or transfer to someone other than Google if I run into an issue.
I don't know if these issues are addressed in the pay version of Google Sites. For the moment I'm definitely keeping the email-mailing-list features going, but looking around elsewhere for something similar that works better.
(If you find something please post!)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I have the task to build a web-site for a smaller non-profit organization. I have a bit experience with ASP.NET but because ASP.NET hosting is rather expensive here in germany (we will also need a lot of webspace and traffic) and aslo because there are quite a hughe list of features I think I should go with a PHP/MySQL based CMS (correct me if I am wrong). The question is wich one? There are are so many free CMSs out there. If I tell you what I need, can you tell me what would be a good choice?
Here are my requirement (sorted by priority):
Ease of use (installation, configuration, maintainance) for me who builds the site but also for the members of the organisation they must be able to easily change the the content of some main pages, add supages, add a new mailinglist upload a file to the repository etc.
A membership/role management system. Based on the role of a member access to certain subpages, subforums or folders in the file management system etc. must be restricted. Only certain roles must be allowed to add new mailinglists or moderate the forum.
A mailingsystem that allows me or the members of the organistation to add new adresses, maillinglists or newsletters.
A file management system. Members should be able to upload arbitary files on the server and browse them via a web frontent. Access to folder should be restrictable based on the member rights.
A pulbic forum with private subforums.
Localization. As much as possible of the final site (if not everything) that is presented to the visitor of the site as well as to the members who maintain the site should be presented in german.
Good control over design/look and feel of the overall site. I should have good control over how I want the site to look like.
A lively community with enough momentum to find ressources and help when I am lost.
Extensibility. In case there are some smaller features missing or f the behavior of an existing feature is not quite the way I want it, it would be cool to easily add it myself.
As mentioned by others, Joomla might be a good option for you, although Wordpress may also work (and it's easier). However, I highly recommend that you check out OpenSourceCMS.com, which hosts demos for many of the free and open source CMS's, blogs, forums, shopping carts, etc. For most of them, you can try out both the admin and the frontend. Spend some time looking them over and then when you find some that you like, investigate them further by going to their websites.
Concrete5 is a new Content management system that is excellent. Easily themable, user friendly, great little dashboard for management
and it's open source
My vote would be Joomla. It has most fot he features you discussed, if not in the core as an extension, component or theme. I've set several up and rarely have to use any coding during setup, plus there's a vibrant community if you need help. It also integrates nicely with other 'best of breed' apps like BB forums, Coppermine photo gallery's and others.
Joomla! would seem to fit your needs, and I'd also suggest Kunena for the forum (which integrates with Joomla) and DOCman for the file management solution.
Well, I'd suggest Drupal for any sort of advanced web site. While Joomla! and similar systems are a bit easier to get started with, Drupal's a lot more flexible and extensible.
You want full control over theming? Or user authentication? Access control? Database queries? User picture scaling? Tagclouds?
It's there when you need it :)
I would go with Joomla too, even though I'm a ASP.NET developer.
Joomla is very flexible and customizable, so it fills all your need, because of the big community.
As a complete CMS noob, I asked myself the same question. I started with Drupal. Sure you can do lots of things with it. Very extensible and customizable.
But after implementing my first site in Drupal, I decided to create the next one in Joomla, to have a frame of comparison. Conclusion: Joomla is a looooooooooooot easier to get started and takes a lot less time to create a functioning website from scratch. Sure, for very big projects, Drupal lets you customize more, but for the type of project you suggest: I would recommend Joomla...
I found this PDF report comparing WordPress, Joomla, Drupal and Plone quite helpful when I was asking myself the same question recently. Nice comments with a focus on non-profits - find the comparison summary/recommendation on pp 13-14.
Definitely read the report because it talks about the strengths/issues of each package - given your prioritized list of features, I'd recommend Joomla. I think you can do what you want with reasonable effort and good recent modules like DocMan. WordPress is awesome for a blogging site, but lacks a number (2,3,4,9) of your requirements, and though Drupal is powerful and flexible, it definitely has a steep learning curve. Plone even more so.
Good luck!
I'd vote for DotNetNuke, it's ASP.Net but that saves you the overhead of learning a complete new environment, language and tool just to implement a 'free' CMS, realistically how much is your time worth? It meets all your criteria, and there are currently 600,000 users - which means that you can just get on with it, somewhere someone has already worked on your problems for you.
As for expensive hosting, why not host in another country, I get US$5/month for DotNetNuke in the states, and I code from Australia.
Ive had some experience with Joomla. Highly customizable, plenty of plugins and one of the liveliest communities in development. It has a great admin panel also.
Well, as told by Tom Deleu, yes, though Drupal is very capable and strong CMS but it is tough and complicated to work on. Coming to another option of CMS that is Wordpress. Though it is very easy to develop a content based websites with wordpress but with very limited scope and flexibility.
As per your requirement my vote will also go to Joomla. It is very user friendly, optimized and a CMS you can rely on. Small applications as mentioned by you like "mailing system", "File system" etc. suits joomla more than others.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I'm currently trying to build a personal website to create a presence on the web for myself. My plan is to include content such as my resume, any projects that I have done on my own and links to open source projects that I have contributed to, and so on. However, I'm not sure which approach would be better from a perspective of "advertising" myself, since that what this site does, especially since I am a software developer.
Should I use an out-of-the-box system and extend it as needed, with available modules and custom modules where needed or should I custom build a site and all of its features as I need them? Does a custom site look better in the eyes of a potential employer who might visit my site?
I've toyed with this idea in the past but I don't think it's really a good idea for a number of reasons. Firstly, there are a number of places that can take care of most of this without you needing to do the work or maintenance. Just signing up for a linkedIn account for example will allow you to get most of your needs catered for in this regard. You can create your resume there and bio information etc and make it publicly viewable. The other issue with your "own site" is that if you don't update it often, the information gets stale, and worse yet, people have no reason to go back because "nothing has changed" - and that's not much of an advert for you is it?
Now that I've said all that, I'll make another recommendation. Why not start a blog instead?! If you've got decent experience, why not share that. I'd be willing to bet that this will be the best advert for your skills because:
It's always updated (if you post often)
It's not like you're looking for work doing it - but your (future) employer, or their developers will check it out anyway to get a better insight into your character.
Putting something on your resume doesn't mean you can do it. I'm not saying that you'd lie about your skills :-), but there's no argument about your ability when you're writing articles about the stuff, getting comments and feedback, and better yet, learning EVEN MORE about your passions.
Best of all - you can run your blog from your chosen domain and also point to your resume that is stored in linkedIn. Just an idea...
That's my two pennys worth on that - hope it helps you come to a decision!
If you are a web-specific developer I would go with a custom site, but if you focus more on desktop applications or backend technologies, I think an out of the box system would be fine.
A nice looking, default, off the shelf, complete website could be more impressive than a poorly done, broken, tacked together, incomplete website. Perhaps start with something "off the shelf" but nice looking, keep it simple, professional, and then eventually add more custom functionality, style and content. Potential employers may like to see that you are capable of reusing tried and trued solutions instead of trying to create everything from scratch without a good reason. Or you could spend time combining great components into something even better than the sum of the parts, as Jeff Atwood talks about extensively in the Stack Overflow podcasts. Stack Overflow is a good example of writing lots of custom code, but combining that with some of the best Web 2.0 technologies/widgets/etc. into something coherent, instead of trying to prove that they could implement x/y/z from scratch.
(On the other hand, it's really fun to build your own login system, blog, or photo gallery. If you really enjoy it and you want to learn a lot or create something new and different, then go for it!)
Here's what I did (or am currently doing). First, use an out of the box solution to begin with. In my case, I used BlogEngine.NET, which was open source and easy to set up. This allows me to put content on my site as fast as possible. Now, I can continue to use BlogEngine.NET, and skin my site to give it more personality or I can start rolling out my own solution. However, I haven't found a requirement yet that would give me a reason to waste time building my own solution. Odds are you probably won't either.
I don't think it matters if your site is blatantly using a framework or other "generic" solution. The real question is "is it done well, with taste?" If you are using an out of the box solution, you should take the time and pay attention to details when customizing it as if you were creating it from scratch.
Alternatively, if you're looking for a great learning experience and something to spend a lot of your free time on -- write it yourself. But know that you are re-inventing the wheel, and embrace it.
edit
A recent post from 37Signals, Gearheads don't get it, really sums up a good point about not focusing on the technical details, but "content and community".
Reinventing the wheel is not such a great idea when you are building a personal site. Building your own CMS is fun, and to some degree is something to brag about, but not so much features you won't have the time to build and all the security holes that you won't have the time to fix.
It's much better to pick a good, well-established engine, build a custom theme, and contribute a module or two to it: you'll be writing code that you can show off as a code sample and at the same time creating something useful.
Knowing your way around an open source CMS is a good skill in just about any job: when your boss says - hey, we need a three pager site for client/product/person X in 10 hours, you can say - no problem.
For a simpler portfolio site, Wordpress might meet your needs.
You can set up 'static' Wordpress pages for contact information, various portfolios, a resume, etc. This would also give you a blog if you want to do this.
Wordpress does give you the flexibility to "hide" the blogging part of it and use it basically as a simpler CMS. For example, your root URL of example.com could point to a WP static page, while example.com/blog would be the actual blog pages.
If you self-host Wordpress on your own domain (which I really would recommend instead of going through wordpress.com), it should be trivial to set up a few subdomains for extra content. For example, downloads.example.com could host the actual downloads for projects you've developed linked from the Wordpress portfolio pages. Similarly, if you're doing a lot of web work, a subdomain like lab.example.com or samples.example.com could then host various static (or dynamic) pages where you show off sandboxed pages that are not under the control of Wordpress.
Keep in mind though that you'll want to make your page look good. A sloppy looking site can scare away potential clients, even if you are not looking to do any web work for them.
Putting your resume up online somewhere helps, I get a lot of recruitment emails from people who happened on my resume via googling. However I agree with ColinYounger in that you'll probably get more bang for your buck from LinkedIn.
My advice is this - if you want to take the time out to LEARN a CMS or something, to better yourself, then why not make your first project in one be your homepage?
Maybe enlighten us as to the "features" you want to have on a personal homepage? Outside of a link to an HTML resume and perhaps some links to things you like, not sure exactly what the features of a homepage would be...
It really depends on:
a) what services you provide
b) what your skill level is when it comes to web design/development
If you are primarily a web applications developer then running an off the shelf product or using blatantly using DreamWeaver to develop it may not be so smart -- or maybe your clients aren't adept enough to notice?
Likewise if you're primarily a web designer then it is probably a good idea to design your own website.
Just as a side question and following up on my 'ego trip' comment: why would you take anything on the web to be 'true'? IME printed submissions, while not necessarily accurate, tend to be slightly less, erm... exaggerated than web submissions.
Do those responding\viewing ever hire? I wouldn't google for a candidate. I might ego surf for a respondent, but would ignore CVs.
Rounding back to the OP, I would suggest that you need to SHOW what you're good at - participate in Open Source projects and POST on their forums, link to projects you can post details of and generally try to show what a Good Employee you could be. Just telling me that you're good at [insert latest trend here] means diddly.
I have come to see that the best way to advertise yourself is to put quality content out there. If you write about the technology that you have experience in, maybe create a few tutorials, and if you do all that often enough, that shows some authority in your chosen field of work.
This alone is one of the best advertisements. However, you also want to show passion. And online, that can be shown through how meticulously your site is done (it doesn't have to be a super great UI or something), but it should be neat, clean, and professional. It doesn't matter if its out of the box, or custom designed.
Either way, you will have to work hard to make it look good.