User:Stuart/Scratchpad 2

I spent hours going through all the negatives - all the things I don't like about the way that we work now... but on re-reading it, it just seems pointless. It's no better than the meeting we had where I just opened my mouth and ranted incoherently for 2 hours.

So I'm just going to skip all that; most of my grievances have been heard and largely dismissed over the years anyway, so it's just old coals that I have no particular desire to rake over.

Instead, I'm going to just try and come up with the suggestions which might go some way to mitigating those grievances. This is me being grown up. Normal service will resume later, don't worry.

I'm going to put these in descending order of importance, obviously in my opinion.

Tech Lead

The very first grievance I wrote was about the lack of technical oversight. I think it's really important, regardless of how 'non-agile' it might be.

You need someone who can make final technical decisions - a gatekeeper who:

• monitors all aspects of development at a medium level, keeping track of all changes to ensure that someone is always aware of all the things that are going on now and that are coming up so that conflict can be kept to a minimum
• decides which third-party products are allowed into the development environment, and which ones aren't
• works with devops to design and prioritise the technical base that developers work from
• helps to maintain any of the surrounding tools / systems that the devs use
• documents all aspects of the development process, including tools and software used
• is final arbiter on coding standards issues - setting them, enforcing them, updating them.
• designs other aspects of development outside of the production line: branching strategy, labelling strategy, release management.
• works directly with the product owners with respect to planning and scheduling of the larger projects
• has some say over scheduling and prioritisation of non-user-facing tasks; 'technical' tasks such as looking into alternatives to the INA or pools; considering how to scale aspects of the product.
• creates and maintains documentation for the development of the product; eg. how database upgrade script numbering works; branching strategy; network diagrams; medium level class diagrams. All designed for developer use - ie. not designed for consumption outside the development team.

We have discussed this already - I don't think this needs to be someone with deep product knowledge. That will come with time. It's more important to have someone who:

• is technically proficient
• is comfortable with all development aspects of the product
• has the respect of the developers - they need to be willing to accept his/her decisions once they are made.
• has the respect of the management - they need to let him/her do the job his/her way, even if it goes against how they would do it.
• knows when to lay down the law and when to be liberal - when to insist that something is done a certain way, and when to allow the developer the freedom to do as they see fit.
• understands that this is a large project and consistency is sometimes preferable to localised 'perfect' code design.

My current recommendation, if I had to make one from our current crop, would be Dan - he has strong opinions which are often counter to mine, but he's pragmatic and has the respect of the rest of the developers.

Even if I were planning to stay, I would not choose me - because I fall down at this point: "is comfortable with all development aspects of the product" - I'm not; I have issues with the direction that actual development has gone in the last 6-8 months, and I'd be fighting the urge to roll back to a time before that direction was taken. If I did that, it would upset a number of developers, and I would fail on the "respect of the developers" bit then. Basically, I'm just not the right man for it any more.

Project Lead

On a lower level, I think every reasonably sized piece of work needs a lead, nominated at the outset, who works closely with the tech lead and product owners with a view to making all technical decisions for that piece of work. So someone who:

• ensures that requirements are ironed out before development starts
• creates and maintains the design and planning of a large piece of work
• organises the distribution of stories and handles the work being brought together
• reviews and merges all changes for that piece of work into the development branch
  • This is important - I think "anyone can review anything" doesn't work for a larger project - you need some consistency over a piece of work; you want someone who knows what's coming so that they can suggest that something should be done a different way in order to better cope with something further down the road.

This isn't a job title - it's just like when I was effectively 'Project Lead' for groups, with Giles working on the 'resources' part of it. I answered any questions he had, reworked code to meet requirements he had which I hadn't considered; reviewed all his code; made sure it wasn't conflicting with my own work - that sort of thing.

The quasi-democratic thing we have at the moment, I think, is causing us far more problems than it is solving.

Design

3 months ago, I would have argued that there is no design... we've actually improved at that a bit recently, with a few 'spike'/'r&d'/'some other name' pieces of work which were basically designs. We're obsessing a bit with what we call them (something I think is entirely unimportant), but at least we're doing them now.

We still don't do anything smart like have a formal process for design; even where the design is documented (if it is documented) is entirely dependent on who's doing it (Yodiz / Word document floating around selected inboxes / Wiki).

So I think you should - like I mention above, I think this should be owned by the project lead, but even if you don't go for a project lead, I think having a formal design process for medium-large features would be sensible. Detailing:

• What a design should cover
• To what level of detail a design should go
• Where the design should be hosted
• How the peer review process should occur
• Naming conventions for the hosted design

Personally, I think a design should:

• be a document targeted at developers and testers, not managers and certainly not customers
• explain background, current state, desired state at a quite high level
• provide a medium level description of proposed technical aspects, including:
  • classes and their relationships
  • database tables and their relationships
  • interface descriptions / mockups - including command line if appropriate
• highlight potential obstacles and possible mitigations
• enumerate the 'vertical slices' of functionality, at least at a reasonably high level - ie. the full sub-projects which would be accepted as releasable by product owners, regardless of their complexity.
• identify any dependencies on the pieces of work in the project
• Provide a set of example use cases which demonstrate the feature, including some 'failure' cases which demonstrate how failures should be handled.

Modularisation

A big part of the problem for me is that there is one big project that everybody is a part of... trying to be developers of everything all at once.

We've spoken about it before, but some modularisation of the product is essential, and then actually having teams responsible for different aspects of the product. Completely off the top of my head, something like:

• Server (Server, Scheduler, Database, Data Models)
• Client/Resource (User Interface, Running Sessions, Logging)
• Automation (Application Manager, Spying, Querying)

Whatever modules we choose, I think we need to start figuring out how to separate them in code, what interfaces are required, so that we can develop these things separately. Then when a piece of work straddles two teams, members of those two teams work together for that project, each responsible for their specialist area.

I don't have all the answers for this, but I think our current way of just throwing more and more teams at the same huge pile of code is not sustainable. There's going to be more and more conflicts as we go on and without some clear delineation of responsibility with clear space in between those areas of responsibility, it's going to occupy more time that should be being spent on actually solving product problems, not logistical ones.

Business Analysis

I do touch on this elsewhere, but I think we've always been pretty appalling at requirements.

You really could do with someone whose job it is to analyse the English level requirements that we get and turn them into meaningful specific requirements; ie. generate some useful acceptance criteria that can actually be used to test the work and ensure it's complete. I actually think this is what a product owner should be doing, but since it's not being done at all, perhaps getting a specialist in to do it would be worthwhile.

If the requirements aren't there, we need to be willing to hold the work back until such time that they are. If the developer is expected to come up with requirements... well, frankly, that's not ideal - it's not typically what we're best at unless you're looking for purely technical requirements (eg. "it needs to be able to create sessions on 100 resources less than 10s after startup" or such like).

Meetings

I have problems with our current sprints, but even more so with the retro / planning meetings. The 'Happy'/'Sad' paradigm rarely seems to spit out things worth discussing - the most useful and relevant discussions are the ones that get put in the middle - ie. "I just want to discuss X and how we approach it".

We also have 'release retros' which discuss the good and bad aspects of a release; something clearly not designed for releases several months apart.

So my preference to that would be something like:

Fortnightly Progress Meetings

Not dissimilar to what we have now, but more focused on the work and where we are with it. A look at where we are within the current project, and what's left to do. What's changed and what we might want to change going forward.

No 'Happy'/'Sad' stuff, just an update on our progress over the last 2 weeks, a look at the things we're going to be doing in the next 2 weeks, and a list of topics that people want to discuss - whether it's positive, negative or just seeking clarification on something. These topics can be built up over the 2 weeks like the topics are for the development team meetings we have; so we can have some discussion about it before the meeting if necessary, but more so that you don't have to remember the things that you wanted to talk about in the 5 minutes at the beginning of the retro.

I think the order of the things is important too - I would personally go for:

• Update on progress; what we've achieved in the last fortnight
• Topics of discussion
• A look at the next fortnight

Because the update may drive out further topics, and the topics will help to form the way the next fortnight is approached.

Project Retros

Much more relevant to us than 'Release Retros' - what did we try that worked in this project, what did we try that bombed, what would have made things easier with retrospect, what's outstanding - the little stuff that got lost along the way, what have we learnt?

These would be more nebulously scheduled - just when a project is 'Done', for the purposes of the release it will go into.

How do other people do it?

I've been pointed at a few alternatives by friends. There's a lot of interesting things out there that we could adopt.

I don't believe in one-size-fits-all anything so I don't really think we should just take someone else's methodology and impose it on the department, but taking ideas that might fit our situation or fix some issue that we have and trying it out seems eminently sensible to me.

Basecamp

My current favourite 'how to do it right' example is basecamp's, detailed on their Signal v Noise company blog site - How we structure our work and teams at Basecamp.

The teams are focused on a project, rather than just being an arbitrary number of people. Each iteration brings a new set of projects, which means a new set of teams. Developers choose (where possible, at least) which projects they want to work on, and the team size is kept minimal - typically three people with one lead and two developers.

Each iteration is 6 weeks, instead of 2, meaning you can actually sink your teeth into it and it allows for mistakes or dead ends without such a feeling of disaster that we have now. It also means that what you have at the end would be actually releasable, rather than 'potentially releasable' which, to me, just means 'not releasable in its current state'.

There are no 'standups' - as a company they are dead against them, as they opine in Status meetings are the scourge.

They say that:

We landed on this process after a decade of refinement.

Incidentally, if you'll allow me a little whine in a document that I'm trying to keep positive, I think, fundamentally, this is where we went wrong - we didn't refine; we discarded and replaced wholesale. I'll leave that there cos I suspect it's a point I've belaboured enough already.

Crucially, from my perspective is this:

Once a six week cycle is over, we take one or two weeks off of scheduled projects so everyone can roam independently, fix stuff up, pick up some pet projects we’ve wanted to do, and generally wind down prior to starting the next six week cycle. Ample time for context switching. We also use this time to firm up ideas that we’ll be tackling next cycle.

The ability for developers to have a go at a pet project is essential, in my humble. There are loads of things I wanted to do when there was 3 or 4 of us, and I just couldn't do them because "we only have 3 developers", "we only have 4 developers"... well, we have plenty now but every waking minute is planned for me from above - even moreso than before - so I never get chance to look into, eg, a better more scalable way of implementing work queues, different types of tasks in scheduler, considering why customers are using schedules how they are and if that goal could be met without the heavy weight of the schedules (similar to how we came up with environment locks out of customers using queues in a way they weren't designed for).

Fog Creek Software

Well, this isn't really a development methodology, so much as a list of 'rules' about writing good software - The Joel Test: 12 Steps to Better Code

Blue Prism actually does pretty well on the Joel test, although the one I keep banging on about is #5 - "Do you fix bugs before writing new code?".

---

OK, so all of the above is "things that I think are worth changing".

David asked me to put this together, despite the fact that I was pretty much dead set on leaving, at least partly because he said he wanted to know what might keep me here. Realistically, none of the above, really - if everything I suggest was implemented by Friday, I would still be going, because of the point I mention when discussing the Tech Lead, ie. when I said that a good tech lead candidate:

"is comfortable with all development aspects of the product" - I'm not; I have issues with the direction that actual development has gone in the last 6-8 months

I don't think my working on the BP product is salvageable - every day or so, someone says or does something that makes me glad that I'm on the way out so I don't have to follow where it's leading. I'm not going to go into detail - the problem in these cases is me; no other developers seem to have any issues in these regards or, at least, if they do, they're not considered important enough to them to speak out about it.

So rather than bitch about that, I wanted to articulate what might keep me here.

The main problem for me is siloing. Specifically the lack of it.

'Siloing' is what I would call 'expertise' - deep knowledge of a specific area of the application; for example, back in the day, I would be largely responsible for scheduler, release manager, queues, groups, a few other bits and bobs; Giles would look after mainframes and web services; Glyn handled dashboards and dependencies; Ciaran handled SAP and network stuff; and everything else was a free-for-all.

Back when we were at that Agile training course at Malmaison Manchester, I remember David trying to convince Ciaran and me that 'siloing' was causing us pain, and both Ciaran and I trying to make it clear that it really really wasn't. What I perhaps didn't realise at that point was how important 'siloing' is to the way I work.

I am an idealist when it comes to the stuff I work on. I want it to be perfect; I want to improve it to the point where it can't get any better. I take great pride in some of the stuff I've done within BP - none of it is perfect, but I know its flaws inside out; I know all the things I want to change or improve, and I have ideas for how to do them that I would love to spend some time working through to see if they have legs.

Primarily, though, I consider it mine - I either wrote it originally (scheduler, relman, groups), or I've done so much on it over the years I know the code inside out (queues, charmatching) - and as mine, I want to keep on improving it and updating it and extending it, and I want it to remain elegant and well structured and well thought out (on the assumption that it mostly is that in the first place).

This is all ego, I'm sure, but that's what drives me; Crafting something and working on it over time until it gets closer and closer to what I originally evisaged... to perfection.

Once that was taken away, and all I had left was the little fragments of projects; fractions of smaller bits of larger projects, designed by several committees with which team works on what decided in a closed room somewhere above me, where I have no say over the whole, where what is mine today could go to a whole different team tomorrow that doesn't understand what I was trying to build and stomps all over it... well, it turns out I don't cope awfully well with that.

So, given all that, what would keep me in Blue Prism?

Basically, it's what I reached out for at the end of February in the Centrix foyer. A new project that's mine or, less solipsistically, one where developers have silos - where a developer or a group of developers is primarily responsible for an aspect of the project that is theirs - that they can craft and work on and improve and update and document; where the developers are experts rather than jacks-of-all-trades.

I think perhaps I'm just better suited to smaller projects, for exactly that reason. Being able to 'own' a project just doesn't scale.

So that's what I will be looking for elsewhere.