I am working on a system that has worked for years and now they decided to add a recommendation engine. After doing the A/B testing, we will decide on a model. Now, is there a way to measure how good this model is performing?
Your target metric is a matter of experiment design and generally noone will be able to answer this question better than you. Typically, you need a way to measure engagement. Some of this will be immediate but some of the effects may take a longer term to be registered, e.g. when a user remembered your recommendation but returned to it after some period of time. The immediate impact, like a click on a recommended item, is easy to record because it happens in the timeframe of the current user experience. The long term impact is harder to record because you'll need a way to correlate a latent action to recommendations that happened some time ago.
Related
Currently I am automating a module that requires lot of pre-requisite data before performing the actual action to be tested, using Cucumber with Selenium-Java . I am using rest services to create this data, however it keeps creating bad data as the services keep changing as we are working in a agile project. So need some ideas regarding what should be the automation approach in this case.
** Adding data from UI is not possible as the scenarios require 50 - 100 days back date data.
Not sure what sort of answer you're after here.
So much of the answer is "it depends" and after nearly 15 years in test consultancy i can advise you this a detailed conversation not a simple knowledge drop and run.
Take just one thought line....
The API is changing, so load straight into the database.
The database is complex and cross-linked, so copy production
Production contains user data, so obfuscate it...
You get the idea - for anything i could suggest it could so easily be rebutted - And that's just one avenue to look at. There also your code structure, environments, execution approach, technology, people and teams, etc...
For a different take on things, have a read of this article (disclaimer - i'm the author)
The entire article is great but the section you want is "It stinks that maintenance of our automation takes so long and costs so much".
...Does that sound about right? :-)
Paraphrasing from the article - You won't ever get away from test maintenance something you need to get across to the rest of the team is the effort it's taking you. If it takes you a week to maintain the automation for a 10 minute dev change, that needs to be addressed.
Beyond that, a few other ideas that might be worth considering (some are in the article too)- and they're generic, non-technical ideas because "it depends":
if it's ALWAYS changing it's not suitable for automation. Tell people you'll pick it up when it's stable. If needs be, get all the cards that will change the api prioritised so it's done asap and no more immediate changes are needed.
Do you need all the tests you have? You're (probably) doing automated regression testing, not functional testing. The more tests you have, the more effort it is to carry them forwards. Consider decreasing scope to decrease maintenance.
Can more of your tests become unit tests? Review what you're testing with a dev and see what they thing.
You're the QA team but quality is everyone's responsibility. Get the dev team doing the maintenance on such changes. Makes sure working automation is part of the definition of done/complete so that 10 minute card is captured for the full effort it is.
Use your retrospective to raise it and capture what's going on. The entire point of agile is to feedback, learn and adapt. That's the audience you want to raise this question with :-)
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
We are starting to use Kanban and my boss just asked me a question, as one of two people with prior Kanban experience within the group, that I don't really know how to answer.
My previous experience and training with Kanban had developers pulling stories in from the backlog by priority, in our case that was the topmost card. However, my boss would like certain stories to go to the developers that have domain knowledge for particular areas. For example, let's say Joe has the most experience in working with Contracts and a contract story comes onto the board. He would like Joe to be the one to work on that particular story.
This, to me, feels a little "off" and could lead to some developers having significant extra work due to having worked in any given area of functionality. My previous experience with Kanban worked under the assumption that any developer should be able to pick up the next card and figure out what to do and that this practice would eventually eliminate any single functionality area experts and level out developer expertise over time. However, I can also see how using subject matter experts can help move a story through the process faster.
What is the most "Kanban" way of handling priority vs. expertise when it comes to pulling in the next story?
Every system I've ever worked with allows a little bit of developer-level of prioritization. If the next card has the absolute (top-down driven) priority, then you have to pick that card. Mostly, though, I tend to work in places where "these next 6 cards are up, pick the one you like". This gives the developer a little bit of room for type of work he or she prefers. Plus, it gives the developers a greater sense of ownership since they did get to pick (to some extent) the work they were doing.
Regarding your example, it's a little off base. In an ideal world any developer should be able to pick up any card. In reality, this isn't always true. If I give this project to Jim, it might take 2 days. If I give it to not-Jim, it make take all week. This is a sign! What information sharing is missing? How do you get the other developers to understand the Contracts component as well as Jim?
If the priority is a little bit gray, this stuff tends to work itself out. All the other developers know that Jim can handle the Contracts stuff. However, if Jim has no capacity, then someone else must take up the challenge. Kanban is supposed to alert you to blocked stories.
Kanban is great for visualizing work flow, limiting WIP, and exposing bottle necks.
Henrik Kniberg has a great book, 'Lean from the Trenches'. He talks through many techniques he has used in real world examples. He described one approach to having avatars (that represent developers) that can be placed over task to show who is working on what.
One idea for your situation, would be to use this avatar approach to pre-assigning who should work on a task in the buffer leading into development.
If those pre-assigned tasks are not causing bottlenecks and flow is natural, everything is good. If they are causing bottlenecks early in your flow, you have a problem, but now you have an easy way to visualize and see that it is the pre-assigning that is causing your bottle neck!
A Kanban system shall display the real process. If the manager is assigning stories to developer then the system should reflect that. This can be done in several ways, you could have a specific item for developer X or you can write the developers name on the card. Another option is to have one swim-lane for each developer.
However, all of this is probably not good from a "global" perspective. You should share your Kanban data with your boss. What are your lead-times, what is your throughput etc? Then you should invite the boss to the process improvment meeting. How are we going to improve our figures? Hopefully he sees that Joe probably will be a bottleneck if he is assigning tasks directly to him. Teach him Littles Law, teach him about bottlenecks and Lean in general.
Don't forget to make your policies explicit, that is, it should be written on the wall how your prioritzation policy looks like. Good luck!
I was recently asked by a Team Leader (not mine) if I would be willing to undertake a programming project. The members of his team are currently pre-occupied with other more important projects. I graduated college two years ago, and up until now programming has only been a hobby of mine. Recently I decided that I would like to pursue a career in software development. I accepted his offer so that I can gain some real-world experience and start building a portfolio.
In about an hour I'm scheduled to meet with the Team Leader to discuss the details of what he needs. From a short e-mail exchange with him, I know that the base project is to update an existing ASP.NET form—but I also think there's more to it than that.
Considering that I'd like to eventually put this project in a portfolio, what kinds of notes should I take at the meeting?
Take whatever notes you can that will best help you understand the use cases and the user requirements. Everything else is just technical details that can be figured out later.
I graduated college two years ago, and up until now programming has only been a hobby of mine.
In that case, my suggestion is:
revel in your ignorance.
Make the most of the fact that you know nothing and you're being given an opportunity to learn - abuse the chance to ask as many questions as possible of the Team Leader in question regarding what type of questions you should be asking and how you should be documenting what you learn.
You only get one chance to be ignorant, once you've wasted it you have to spend the rest of your life as a know-it-all; take the chance to enjoy the learning process.
Get a list of people who are the intended users. Talking with them will allow you to flesh out the overview that the Team Leader gives you. It is likely that the intended users have a very different understanding of what the app is supposed to do than the TL does. So you'll likely be going back and forth for a while. It's well worth the effort though because you'll do much less re-coding.
Try to understand that the Team Leader him/herself might not even have all the requirements available right at the beginning. Be prepared to be hunting down people and writing all these requirements down as they come in.
Things will change during development, new problems and new requirements will always be popping up.
Three things:
What: What is the software supposed to do, the more detailed you can manage to get the other person to be, the better.
How: Are there any known constraints? For example, if it has to ask for a telephone number, does it have to validate nationally/internationally/not at all. Does it have to run on Windows 2008/2003/all
Who: Two sides:
Who will answer any questions you'll have, will you setup weekly progress meetings?
Who will use the software, can you get their early input on your prototypes, can you ask them for opinion/requirements?
One thing I've found very helpful is carrying a hard-copy of any existing requirements (use cases, wireframes, whatever) or any other potentially useful information in a 3 ring binder to any project meetings I attend. If the meeting strays off topic or questions about previous discussions or documents come up it is very nice to have the information at your fingertips in a format you can make notes on, pass around the table etc.
As a bonus, I find most people don't carry any documents to meetings, so you'll also end up looking like you are a real go-getter who is always prepared, which is never a bad thing.
Main downside to this is that you'll waste paper if the documents are updated and changed frequently.
Find out the where as well, where are the files you need stored on the network, where is the source control repository for the project, etc.
Since this is your first taste of doing a real world project, please please please make sure you use source control even if you are the only dev on the project. Your co-workers will thank you and you will thank you the first time you need to back out a change that didn't work.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I'm working in a small company and weeks away from deploying a web-app that will be used a lot. Everyone at one location will have to learn to use it, and although I think it's pretty easy and intuitive I may be biased.
I've written a help guide with plenty of screenshots that's available on every page, but I'll still need to train everyone. What's the best way? How do you take a step back and explain code you've been working on for weeks?
First try to avoid the training:
Perform usability testing to ensure your web app is intuitive. Usability testing is a very important aspect of testing and it is often ignored. How you see your system will probably be very different as how a new user sees your system.
Also add contextual help as often as you can. For example when I hover over a tag in stack overflow, I know exactly what clicking it will do, because it tells me.
Also this may seem obvious, but make sure you link to your documentation from the site itself. People may not think of looking in your documentation unless its right in front of their eyes.
About training documentation:
Try to split up your material into how your users would use the system. I personally like the "trails" option that Sun created for their Java tutorials. In this tutorial you can do several things, and you can chose on which trail you'd like to go.
Support random reads in your help documentation. If they have a task to do in your web app, then they should be able to get help on that without reading a bunch of unrelated content.
Make sure your documentation is searchable.
About actual training sessions:
If you are actually performing training sessions, stay away from explaining anything related to your code at all. You don't need to know about the engine to drive a car.
Try to split up your training sessions into very focused aspects of your system. If you only have 1 training session available to you then just do one specialized use case of your system + the overall description of the system. Refer to the different parts of documentation where they can get help.
Letting the community help itself:
No matter how extensive your documentation is, you'll always have cases that you didn't cover. That's why it's a good idea to have a forum available to all users of the system. Allow them to ask each other questions.
You can review this forum and add content to your documentation as needed.
You could also open up a wiki for the documentation itself, but this is probably not desirable if your user base isn't very large.
Few ideas:
Do you have some canned walk-through scenarios? Don't know if it is applicable for your product, but I built a pretty substantial product a couple years ago and developed some training modules that they'd work through - nothing long, maybe 15 minutes tops for each one.
I put together a slide presentation that hit the highlights to talk about what it does. I would spend about 10 minutes going through the app's highlights to familiarize them with it before doing the hands-on stuff.
People don't tend to read stuff, unfortunately. You could put hours and hours into a help document, and still find that folks simply don't read it or skim over it. That can be frustrating. Expect that answers that are in your guide will be the topic of questions your users will have.
Break up any training you do into manageable chunks. I've been to a full-day training exercise before and the trainer broke it into short pieces and made it easy for me to get the training topic in my head. You don't want to data-dump on them because their eyes will gloss over and you'll lose them.
Ultimately, if your app is highly usable, it should be a piece of cake. If it isn't, you'll find out. You might want to have a few folks you know run through your training ahead of time and give you constructive criticism on it. Better to fix it before the big group is trained. You'll be more confident in the product and the training materials (whatever they are) and you'll likely have a better training experience.
If applicable, provide an online help/wiki/faq for them. Sometimes that is helpful.
Best of luck!
You should really have addressed this issue a lot earlier in the development cycle than you are doing.
In my view the ideal scenario for corporate software is one where the users design their own application and write their own documentation and I always try to strive for this. You should have identified key users early on and designed the system with them (I try to get my users to do basic screen designs and menu layouts in Excel or similar - then I implement that as static pages and review before writing a line of significant code, obviously they won't get the design right first time, but it's your job to guide them - and ideally in a way where they think they came up with the correct design decisions, not you :-) ).
These users should then write the user documentation from this design in parallel with you developing the system. I have never seen help documentation delivered by a IT department/software company used significantly in a corporate setting. Instead what happens is the users will create their own folder of notes and work-arounds and refer to this (in fact if you're ever doing system analysis to replace an existing system finding the 'user-bible' for the old system is a key strategy). Getting the users to write their own documentation up-front simply harnesses what will happen anyway - but this is vastly easier if the users feel they have ownership of the system because they designed it themselves in the first place.
Of course this approach needs commitment and time from your users, but generally it's not that hard a sell. It's trite, but working as a facilitator so the users can develop there own system rather than as a third party to give them a system pretty much guarantees user acceptance.
As you are where you are you're too late to implement all of this, but if you can identify a couple of keen, key, users and get time from them to write their own documentation then that would be a good move. If you can't get even that then you need to identify an evangelist who you can train to be the 'departmental' expert and give them 110% of your energy to support them.
The bottom line is that user acceptance is based on perception, and this does not necessarily correlate with how usable an system actually is. You have to focus on the group psychology of this as much as the reality of the system, which tends to be tricky for developers as we're much more factually based than most people.
I'll be looking into something like this too in the next few months.
In your case, hopefully the UI has already undergone user acceptance testing. You say you work in a small company. Is it possible to get the least tech-savvy person there to try it out? In fact, get them to try it out without any guidance from yourself except for questions they ask. Document the questions and make sure your user-guide answers them.
The main thing for me would be logic and consistency. If the app's workflow relates logically to the task it has been designed to accomplish and the UI is consistent you should be OK.
Create a wiki page to describe the use of your system. Giving edit rights to the users of your system lets the users:
update the documentation to correct any errors in the initial release of documentation,
share any tips on usage they may have found.
share unusual uses for the system that you may not have thought of.
request features.
provide any workarounds they've found while waiting for the new functionality to be implemented.
Try a few users first, one or two in a small company. Mostly watch, help as little as possible. This tells you what needs to be fixed, and it creates an experienced user base - so you are not the "training bottleneck" anymore.
Turn core requirements/use cases/storycards into HowTo / walkthroughs for your documentation.
For a public training, prepare a 10..15 minute presentation (just that, not more!) that covers key concepts that the users absolutely must understand, than show your core walkthroughs. Reserve extra time for questions about how to solve various tasks.
Think as a user, not as a techie: - noone cares if it's a SQL database and you spent a lot of time to get the locking mechanisms right. They do care about "does it slow me down" and "does something bad happen when two people do that at the same time". Our job is to make complicated things look easy.
It may help to put the documentation on the intranet in an editable form - page "comments", or wiki maybe. And/or put up a "error wiki" for error messages and blips - where you or your users can quickly add recomendations, workarounds and reasons for anything that does not go as expected.
Rather then train all those people I have chosen a few superusers (at least one person from each department) and trained them to teach the rest of the employees. It is of course vital that those super users are
well respected in their departments
able to teach
like the application
The easy way to ensure that they like the app is to have them to define the way it should work :-). Since they should work with this app each and every day they are the prime stakeholders, no matter what management states
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
Over the past few weeks it seems like I've been interrupted by maintenance tasks from old projects quite a bit in addition to a taking a training class. I feel like I've lost all forward momentum on my current project. It's difficult to even start coding because I'm not sure what I was doing and what I was thinking before the interruption.
What tips or techniques do you have to help make it easier to get restarted after an interruption that takes you away from your current project for a couple of hours or days?
Take a minute to write notes to yourself (on paper right in front of you) about the project you're putting down before you go and pick another up
Still messed up? Headphones on, music on, loud enough to drown out your surroundings and no lyrics
Really messed up? Go for lunch, take a walk, get out in the air and don't come back til the interruption has faded
Really really messed up? Don't accept interruptions any more. Be really firm about this with colleagues and managers. If you need to ringfence yourself to get stuff done they should be able to appreciate that, at the end of the day what's efficient for you is efficient for them.
Some of my techniques include
Documentation. This gets you thinking about your project. If it's not easy to describe, it's probably not elegant enough yet
Static Code Analysis - FxCop, Lint, Cyclomatic Complexity, Security Analyzers. Now is a good time to step back from your code and check on best practices
Unit Tests. This again gets you thinking about the code and how to improve it.
I get interrupted frequently with phone calls, quotes for sales on more technical items, project managers asking me about feasability and time constraints, and junior developers asking for assistance with problem solving issues.
I've found the following, while not as ideal as being able to shut the office door and be left alone for hours on end, to be the most effective:
Install the Firefox add-on "LeechBlock" and add your most commonly used websites there. You could set a time-limit (i.e. 5 minutes every two hours) so you don't completely shut yourself off.
Wear headphones if you work in a cubicle or are busy.
Put your phone on do not disturb.
Turn off your email client and only open it at certain times (i.e. 11am, 3pm).
Ask co-workers to only see you during certain times as a courtesy (i.e. 11am, 3pm) but say you will still be available for emergencies.
Group as many like-tasks as possible together (i.e. Do maintenance tasks from 4pm to 5pm each day) and leave the remainder of your day for project work.
It's not ideal, but this is the best solution I have found.
Essentially what you are trying to do is get back into a productive "state" after an interruption and the key is to find the things that, for you, trigger that productive state. Although I am still learning about the subject, you may find some helpful answers in books on NLP (neuro linguistic programming).
Find small task to implement it first (for 1-6 hours long to do if you doesn't have interruption). During formulating what to do you slightly recall a state of the project. You can also look through the requirements from the customer or project manager.
Implement this task, but be in no hurry - you can spend much more time than you planned. Look around the code to recall deeper.
As for me after this I recall most things.
If you have a problem to continue after mentioned steps this can signal about some problems in the project like ugly design, absence of documentation, weak understanding of the purpose of the project or something else. And this is good time to have a fresh glance on the project. You can notice the problems which are difficult to catch when you are deep inside the project.
Of course this is useful for several days interruption, but not for several hours long.
Yeah, good luck with that! (I don't think there are good ways for interruptions to not be complete interruptions.)
Honestly, IMHO just sit down and do it. Complaining now is just a distraction from getting started. If you want to investigate how to ameliorate this problem in the future, do it in a few days, after you have picked the baton back up again.