First
No I am not asking you to teach me hacking, I am just curious about this file and its content.
My journey
When I dived into the new HTML5 Boilerplate I came accross the humans.txt. I googled for it and I came at this site http://humanstxt.org/.
Immediately my attention went to this picture:
Do I read this correctly? Hackers.txt?
So I resumed my journey in google and stopped at this articles
When I started reading this I had the feeling that its about the difference between Hackers and Crackers. Later I got the feeling that I'm might be wrong and that this place is that this hackers.txt file is a sort of guestbook for hackers?
Also other examples about hackers.txt files I found here
Some files contain code, others have just hurtfull information.
Now I'm realy confused, guestbook, hack tutorials or just history?
Question
What is the use of this hackers.txt file?
The way I see things:
robots.txt contains information and instructions for robots (so it should be read/used by web crawlers, spiders and other kind of bots)
humans.txt contains useful information to be consumed by humans, according to http://humanstxt.org/
hackers.txt should be targeted towards hackers, so it should contain any information the site owner might want to transmit to a hacker, as Ze'ev pointed out. I don't think this should be a place for hackers to write anything, but rather to get information from the site owner (perhaps on how to report vulnerabilities, as others suggested).
Commonly known as Eduardo Vela, Eduardo A. Vela Nava (or sirdarckcat on Github and Twitter) has been a Security Engineer at Google since 2010. (He currently has the role of Product Security Response Team Lead).
As other security experts before him, he pondered the issue of effectively communicating the details of a site's vulnerability reward program to white hat hackers/pen-testers.
One specific such person is Chema Alonso (also on Twitter).
He is well-known enough to warrant a Spanish Wikipedia entry
Between 2005 and 2011 Alonso was awarded the Microsoft Most Valuable Professional Award for Enterprise Security 6 years in a row. That should tell you something about his "skillz".
On February 3rd 2011 Alonso wrote about his frustrations regarding the topic of communication between the administrators and/or developers of a site and hackers.
He proposes a similar initiative as humans.txt but for hackers. As he mentions this hackers.txt initiative in his blog-post.
In April 2011 The humanstxt.org website got a new design which includes the image which mentions the hackers.txt file.
At this point, I must sadly submit to conjecture, but... consider:
The team behind humans.txt are all from Spain (mostly Barcelona)
At this point Alonso is already quite well known in the Spanish developer community
Would it be such a far stretch to imagine that they got to know of each other's efforts?
On May 14th 2014 Vela, already working at Google, commented on a blog-post by Alonso. It is most likely that they had further contact in a professional setting. Whether or not thay extively shared their idea's regarding anything related to hackers.txt is unknown.
On July 6th 2017 Vela posted a question to this extent on twitter:
How about we create a /hackers.txt that says whether something is in scope or not of a vulnerability reward program and where to report it?
Subsequently, an empty git repository was created for hackerstxt.org on github
and an email thread was opened at Google Groups to discuss this idea further.
On August 13 2017, Edwin Foudil (or EdOverflow on Github and Twitter) created a git repository for security.txt on Github and responded to the mailing list:
I have published a similar project to the one being discussed in this group (https://github.com/EdOverflow/security-txt) and would love to get some of your feedback and ideas.
The project is the equivalent of robots.txt, but for defining a security policy. Companies can add a security.txt to their website and define clear guidelines of what security researchers must do when they discover a security issue. security.txt also allows bug bounty programs to add their scope there. security.txt uses a similar syntax to robots.txt, which should make it easier for machines to parse.
He was, in part, inspired by an open-source project he was working on at the time called GratiPay. GratiPay had a SECURITY.txt file since 2013.
His inspiration also drew from the SECURITY.md files that more and more open-source projects were adding to their repositories.
On September 10th 2017, Foudil submitted a first draft for security.txt to the Internet Engineering Task Force.
On September 14th 2017 Alonso wrote a blog post with the title (translated from Spanish) "Security.TXT an IETF draft for my Hackers.TXT".
Beyond the title, Alonso does not allude to the fact that his 2011 idea was the origin of the draft but he does state his approval of the effort.
On February 3rd 2018, the mail group was informed to concede to security.txt and Vela tweeted that Google had already implemented one.
Further information
Details and a nifty tool to generate your own security.txt can be found at
https://securitytxt.org/
Adoptation
Even though the RFC is still in draft, the standard is already being adopted quite well by major players on the web.
Besides the security.txt at Google, there is also one on the website of:
1password
BBC
bit.ly - http://bit.ly/security.txt (can't be linked because StackOverflow blacklist the use of common link shorteners in posts)
CERT NZ
DailyMotion
Dropbox
Facebook
Github
haveibeenpwned
NodeJs
NPM
Open SSL
Shopify
(Feel free to add more from well-known sites, if you find 'm)
As with humans.txt, there also seems to be a hackers.txt site at http://www.hackerstxt.org/. I'm not sure if someone has set the site up as a joke or not, but it links to a blog post on someone's Blogger site.
The post rambles on a bit (I put it through Google Translate) about the poster's history as a 'hacker'. Anyway, towards the end the writer says:
therefore believe we should promote an initiative type hackers.txt , in which managers leave us a message to potential "aliens who are good" that makes it clear they will do managers receive a report of a vulnerability in your site. I've been circling this , the truth is that it is difficult to finish shaping, because perhaps some "alien who is not so good" , type Brainiac , take a free hand to brush a site, or the "good board administrator" , decide to change your mind and Liem, but I think we should be able to do something, I dunno, maybe having Jon Jonz , or perhaps thinking about how to write that file hackers.txt . How do you see it? Greetings Evil!
So I assume that the poster wants to start a sort of hackers.txt standard in the vein of humans.txt, but hasn't finished it off, or hasn't gotten it into the English speaking world.
Digging around, the Blogger site seems to be owned by a guy called Chema Alonso, who must be fairly reputable in the world of Spanish programmers as he has about 35k Twitter followers (https://twitter.com/chemaalonso). He seems to work for a company called ElevenPaths (http://elevenpaths.com/), which says that it's driving "radical innovation in security product development". A quick Whois check shows that the hackerstxt.org domain is registered by someone in Madrid, so I would assume it's Alonso.
The .txt file over at http://www.textfiles.com/news/hackers.txt, which has been refered to by some of the other answers in this thread, doesn't seem to have anything to do with the hackers.txt reference over at http://humanstxt.org/, and neither do most other search results for 'hackers.txt'.
It's possibly a joke, but If humans.txt is for humans to read then maybe hackers.txt is a warning for hackers.
Like the notice you get when you SSH into some more public terminals. "You are being watched... we will get you if you do anything bad..." That sort of thing.
If a hacker did compromise the site, the might notice the file, read it, realise you mean business and be scared away!
Interesting idea.
As this question is somewhat open, I think you are also expecting some assumptions, I write here (not in a comment) my opinion, but if it should be there, I'm sorry.
I think that the idea lying behind humans.txt (which I heard of before) is to make a new habit, new style or something like that. In fact, you can put a contact page, where all these data from humans.txt can be put. I think that hackers.txt could be also something like new style.
I suppose that hackers.txt was much earlier, maybe for 20 years, when www servers and popular web knowledge was poor, when using localhost Apache+PHP+MySQL was making you "a hacker", and if someone could access the file other than index.html (and linked pages from this), reading hackers.txt was some kind of prize, or maybe some kind of filter to show some information to "those who behold" (like this one perhaps).
I think hackers.txt should contain notes on how the site owner would like for data to be used... E.g. "I don't mind if you scrape the movie listings, but please don't hot link out images in your app"
Related
I'm newbie in back-end development, that's why I want to start learning not from pure coding but choosing framework. Want to understand, how it works and start writing unique code by myself.
I've tried to choose framework basing on my future project and couldn't do it without the qualified help. I've searched before to write here, but couldn't find all I need. I don't ask for deep pieces of advice, manuals or so here. Hope, that you as more experienced can show me few ways and I'll choose the one by myself.
What I need to do with framework capabilities:
Forum:
one (or crossing) users' database with site
one header and footer for forum and site
changeble design themes (full CSS support)
groups' rights and design
user's rights and design
moderators, admins, plain users
forum sections, that could be nested
themes
visual post formatting
images inserting
symbols counting in posts
symbols counting for each user in theme
Site:
same to forum users, their groups and rights
user profile with settings
guest book
education module: timetable by user group, courses with lessons, homework sending/discussing, homework statuses, marks/journal of all users and user groups, educational statistics for user groups
public user profile with text form data and all marks, forum activity, statuses and other information
autoposting some data from user profile to forum and updating it when changing the profile
inactivation/activation/ban for user accounts
IP viewing for admins: can see, what users/forum's messages have the same IP
messenger: private chats, private and public chat groups, auto-adding users to chats by users' group
private user notes, that are visible only for admins
bonus accrual
store, when users can change bonuses for virtual goods and session of playing game for random gifts
visual map, where everyone can see all forum sections and what users are on them
plain newspaper: issues with structured articles (a la usual blog posts), commenting for users, offering article feature
I know, that it's so much, but project is mine. And it can be developing for a long time. If I need to study a year or few, it's okay.
How do you think, what frameworks have the most of modules for described functional? Please, don't argufy. Write here if you had experience (or you know somebody who had) of realising features, that're described above.
Programming language doesn't matter (because I'm noob in all of them), but I think about Ruby and PHP (and PHPBB forum). Others are ok too, if them can afford what I need.
Sorry if I unknowingly said something wrong. Tell me and I'll fix it.
Your wanting to build a lot for someone that knows nothing about any languages.
First you need to pick a platform/OS.. Windows or Unix/Linux?
From there, you can pick a Web Server to run on based on that OS.
Then that filters out what kind of language your going to write with.
EDIT:
When someone wants to be a mechanic, do they go to each mechanic shop and ask what is the best car to work on? Each mechanic would have a different answer. No one here, will be able to give you a non-bias answer. So here is what I like....
I write code in 18 languages and been have developed software for 25 years. I've written code for DOS, Windows, and Linux/Unix. Each language has it's own limitation and perfections, but I'm not a fan of open source and most Linux/Unix preferred languages even though some can run on Windows. I have a tendency to lean towards windows and enjoy back end development. I've written Web UIs in Angular, JSP, Java Spring Boot, WebFOCUS and ASP Classic. Now that you know more about me, here are my favorites.
I prefer Windows. I prefer C#. If I have to build a web UI, I prefer HTML 5/CSS3/Javascript with an ASP.NET/C# backend. I don't like bootstrap or jquery as I like to write with a small footprint as do most any older developer. The internet is full of garbage that isn't being used by most every site you go to, just so the developer can get a few shortcuts. Bootstrap and JQuery are just javascript libraries of which most sites can do what they want with 1/10 the code the client has to download, if they really knew Javascript.
There ya go, my bias opinion, take it or leave it, but most likely the only honest answer your going to get here.
What is the best way to reach a live Microsoft developer on the phone who can answer technical questions about standardized OpenXML formats?
I have a paid MSDN support contract. I wanted to use one of my phone tickets but only production-related questions are eligible. For various reasons, I'm not interested in online support.
If this is the wrong place to ask this question, I'd appreciate a pointer in the right direction. I've been on the phone with Microsoft already and frankly I rely on StackOverflow more than Microsoft support.
Phone support is almost always paid for. One of the only ways you're going to get free phone support is find a Microsoft-employee-friend to give you a QuickAssist card.
Other than that, forums are the way to go.
The Office Open XML File Format Implementation forum is the place to ask your questions. Forewarning though - the responses can sometimes be incredibly slow as some of the folks manning the thread are not actually experts on Open XML but instead just try to read the ECMA spec to answer your questions. If you push hard enough, you may just have some luck though. (You'll see me on there as both "otaku" and "terlo").
The other place to ask questions that is "sponsored" by Microsoft is OpenXMLDeveloper.org. The forums you'd probably be interested in are at http://openxmldeveloper.org/discussions/formats/default.aspx. They don't seem to be regularly viewed by Microsoft though, but sometimes. There are other folks who try to answer questions though and do a decent job. (You'll see me on there as "tendoors").
Finally, Stackoverflow.com has some decent folks following OPENXML and OPENXML-SDK tags. You can try in all 3 locations if you like :)
I'd suggest you ask your question here, and in the Open XML Developer forum.
The msdn forums don't offer much knowledge of Open XML.
The Open XML Developer forums (http://openxmldeveloper.org/discussions/formats/default.aspx) employ people to answer your questions so you're somewhat guaranteed a response, however YMMV.
Here, you might just be surprised what people know.
Does Apple provide developers a set of standard terms in differing languages? The reason why I ask is that I'm having portions of my application localized and want standard terminology consistently applied throughout the app. I have utilized some tactics to do this with terms like 'Loading...' by changing the language on my device and observing how Apple has interpreted those terms in other languages. This has only gotten me so far however, and a resource that I can give a translator would go a long way in creating a seamless experience with the consistent application of terminology.
It has been two and a half years since posting my radar, but Apple has finally posted its iOS glossaries:
You can download them from developer.apple.com, or use this link to quickly find them:
https://developer.apple.com/downloads/index.action?name=Glossaries%20-%20iOS
EDIT 29 FEB 2020: This link is still valid and the glossaries have been updated on 15 JAN 2020 with everything updated for iOS 13.3.
At WWDC this year I went to the Localization Lab to get an answer to this question, since a bona fide answer from Apple was needed.
From one of their head cheeses in localization he told me that currently there are not any publicly available resources for download for iOS or Snow Leopard. He did tell me, though, that these resources were slated for release in the coming weeks after WWDC.
This answer will be updated when the information becomes available.
UPDATE 19 JUN 2012: Wow, it has been a whole year now! Obviously Apple didn't come through for us 'in a few weeks'. I did talk to them again this year and was given a contact to follow up with via email. I explained that they made a claim to have it last year and this was the response I got:
I did check with the documentation folks and found that they're still
planning on doing this but it's fallen behind other priorities. If you
haven't already done so, would you mind filing a bug report about
this? That's one of the best ways to convey the desire to the
appropriate people. While I've relayed this feedback to some people
it's always best to have a bug report directly from folks outside of
Apple. Feel free to forward me the bug number and I'll keep an eye on
it.
Our best bet at this point is to keep filling bug reports so that this gets more attention. Just for the record, I did file a bug report last year.
Apple provides a number of translation resources that you can download which may or may not be useful. One of these is AppleGlot, a tool for replacing strings in application resources. A number of XML-based glossaries for different languages are also available, but they're specific to AppleGlot. You may be able to make AppleGlot work for you, or you might just want to extract what you can from the language glossaries. AppleGlot and the glossaries were created to support translation of MacOS applications, so the terms are related to MacOS X and not iOS. Nevertheless, I think it's worth a look.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a serious question. Is it ever ethical to ignore the presence of a robots.txt file on a website? These are some of the considerations I've got in mind:
If someone puts a web site up they're expecting some visits. Granted, web crawlers are using bandwidth without clicking on ads that may support the site but the site owner is putting their site on the web, right, so how reasonable is it for them to expect that they'll never get visited by a bot?
Some sites apparently use a robots.txt exactly in order to keep their site from being crawled by Google or some other utility that might grab prices and therefore allow people to do price comparisons easily. They have private search engines on the site so they obviously want people to be able to search the site; apparently they just don't want people to be able to easily compare their information with other vendors.
As I said, I'm not trying to be argumentative; I would just like to know if anyone has ever come up with a case where it's ethically permissible to ignore the presence of a robots.txt file? I cannot think of a case where it's permissible to ignore the robots.txt mainly because people (or businesses) are paying money to put up their web sites so they should be able to tell the Googles/Yahoos/Other SE's of the world that they don't want to be on their indices.
To put this discussion in context, I'd like to create a price comparison website and one of the major vendors has a robots.txt that basically prevents anyone from grabbing their prices. I'd like to be able to get their information but, as I said, I can't justify simply ignoring the wishes of the site owner.
I have seen some very sharp discussion here and that's why I would like to hear the opinions of developers that follow Stack Overflow.
By the way, there is some discussion of this topic on a Hacker News question but they seem to mainly focus on the legal aspects of this.
Arguments:
A robots.txt file is an implied license, especially since you are aware of it. Thus, continuing to scrape their site could be seen as unauthorized access (i.e., hacking). Sucks, but arguments like this have been made in other legal cases recently (not directly related to robots.txt, but in relation to other "passive controls".)
Grabbing prices violates no copyright law, including DMCA, since copyright does not include factual information, only creative.
Ethically, you should not grab prices because the vendor should have the ability to change prices without worrying about being accused of a bait/switch by people coming from your site.
Have you taken the high road, explaining the site to them and saying you'd love to include them in your list of vendors? Maybe they will love the idea and actually expose the data in a way that is easy for you to consume and less resource-intensive for them to produce.
There are no laws written directly about robots.txt because netiquette is generally followed. Don't be one of the "bad guys."
Some people filter robots because they use URL links to perform "actions" like adding things to carts, and robots leave them with massive numbers of abandoned shopping carts in their database.
Some people filter robots because they have exclusive prices that they can't advertise openly based on agreements with their vendors. You could be putting them in a bad position by exposing those prices on your site.
In this economy, if a company doesn't want to do everything possible to advertise themselves, it's their own fault that you don't include them.
The other use of robots.txt is to help protect web spiders from themselves. It's relatively easy for a web spider to get mired in an infinitely deep forest of links, and a properly constructed robots.txt file will tell the spider that "you don't need to go here".
Many people have tried to build businesses off building "price comparison" engines that scraped major sites.
Once you start getting any sort of traffic/revenue to speak of, you will receive a cease and desist. It's happened to dozens, if not hundreds of projects. I even worked on a small project that received a C&D from Craigslist.
You know how they say "It's easier to ask forgiveness than it is to get permission"? It doesn't hold true with page scraping. Get permission, or you will be hearing from their lawyers.
If you're lucky, it'll be early on, when you've got nothing to lose. If it's late, you may lose your business and all your work overnight, with a single letter.
Getting permission shouldn't be hard. Unless you're doing something sneaky, you're likely going to drive them additional traffic. Hell, once your product takes off, sites may be begging you, or even paying you to add their data.
One reason we allow robots to dig through the web without complaint is that we have a way to stop them if we want to. Protects both sides.
Remember the uproar when Cuil's robots were accused of going over-the-top, apparently acting like a DoS attack in some cases and using up bandwidth allowances of some small sites?
If too many people violate robots.txt we might get something worse.
"No" means "no".
To answer the narrow question, for the price comparison website you're probably best grabbing the price in real time, rather then scrapping the database in advance. Hard to imagine that being a problem.
An interesting IRL version of story involving The Harvard Coop:
Coop Calls Cops On ISBN Copiers.
Short answer: No.
On the narrow issue: If a seller says that their prices are secret, I think you have to respect that. I'd contact them and ask if they really don't want price comparison engines like yours to include them, or if the "no trespassing" sign is for technical reasons. If the latter, perhaps they'll provide you with an alternative. If the former, then I'd say too bad, they don't get included, they lose some business, and it's their problem.
Tangential rant: Personally, I get pretty annoyed with companies that make me jump through hoops to find out the price of their products, places that make me call and talk to a salesman so he can give me a hard-sell pitch, or worse, make me give them my phone number so their salesman can call and harass me. I figure that if they're afraid to tell me the price, it probably means that it's too high.
In general: A robots.txt file is like a "No Trespassing" sign. It's the owner's right to say who is allowed on their property. If you think their reasons are dumb, you can politely suggest they take the sign down. But you don't have the right to disregard their wishes. If someone puts a No Trespassing sign on his yard, and I say, "Hey, I just want to take a quick short cut, what's the big deal?" -- Maybe I'm stepping on his prized Bulgarian violet bulbs and destroying a valuable investment. Maybe I'm crossing his people's sacred burial ground and offending their religious sensibilities. Or maybe he's just an ornery jerk. But it's still his property and his right. Oh, and if I fall into the dangerous sinkhole after ignoring the No Trespassing sign, who's to blame? (In America, I could probably still sue him for all he's worth despite the fact that he warned me, but is that right?)
I'm showing some ignorance here, but I always thought a bot was something only sent out by a search engine. Like Google or Yahoo.
Thus, if you wrote an application that searched content on the internet, I wouldn't consider that a search engine bot, which to my knowledge is what robots.txt is trying to block.
But this may just be selective ignorance, because I might do it until the webmaster of that site contacted me and asked me to stop :)
If people make it available to public access, they shouldn't try to put limits on it. Adding a robots.txt file to your site is the equivalent to putting a sign on your lawn that says "Please don't look at me."
I wonder if anyone knows about WAFL (Write Anywhere File Layout), or a link to the topic of interest (not wikipedia), or a good bibliography online because I am investigating about operating systems, thanks to all.
The wikipedia page has links to a PDF from Network Appliance on the system as well as the patent link. If that's not going to satisfy you then you need to be more specific as to what kind of information you want.
The NetApp website has an extensive library of papers about WAFL and their file servers.
If you're interested in technical aspects of what WAFL is and how it works, the technical report linked from the Wikipedia article is a very good starting point. This article was originally published at the 1994 USENIX Conference, so it's 15 years old. Some things have changed---and a lot of features have been added---but it still provides a good description of the key innovations in WAFL.
p.s. FWIW, they stopped calling themselves "Network Appliance" a couple years ago and officially changed their name to "NetApp."