Wednesday, August 5. 2009
.....change 'the rules' and try again.
For well over five years we have been trying to find ways of giving Exetel customers the maximum 'free' download allowances, over the longest possible period, for the lowest possible monthly cost without sending ourselves broke. For well over five years we have spent more and more money buying more bandwidth than we needed to try and meet an ever higher end user, very brief, 'peak' demand. For well over five years we have put more management and engineering time into trying to achieve this than we have in to any other aspect of the business. For well over five years we have failed to achieve this pretty simple objective. We gave up on trying to do this last Friday at midnight.
I was bitterly disappointed that with all the 'thought' and 'knowledge' and sheer effort of persuasion we had put into this key aspect of our business we had never been able to sensibly achieve it - I don't think I've ever been more frustrated over such a long period of time in the whole of my life - I just could never find a way of doing something that was so basically simple but proved to be so ridiculously and impossibly difficult. So we gave up and solved all our problems by simply giving up on making a 12 hour period available. Instead of the beautifully symmetric, and very visually appealing, 12 hours we have had to settle for ten hours because we could never find a way of dealing with a relatively very small number of user's obduracy and senselessness. A galling failure over a huge period of time.
Unlike every one of our competitors (or at least I assume this is the case) we have never been concerned about making as much money as possible from providing our services - we have always been driven by the objective of providing the best possible value. One way of doing this is to make use of the 'dead' bandwidth that exists on virtually very commercial network (though I have known of at least two exceptions) in the early hours of the morning. Typically on an efficiently utilised network such as Exetel's this costs us (at our current size) well over $A150,000 a month and the bandwidth is completely wasted. Simple solution offer it to your user base at no cost to them - how, over a five year period could you fail to make that work?.....giving away $A150,000 a month worth of services.....not a problem in the world. We couldn't do it - rather than using the virtually unused 3 am to 7 am period to set a schedule of downloads our user base insisted on starting them at one second past midnight EVERY night and, for over 90% of those users their downloads were completed by 12.30 am EVERY morning. Could they be persuaded to start their downloads after 2 am?......nothing we could do for five plus years could make that happen.
It was, is, an humiliating failure of communication and a shocking indictment of my personal terrible inability to persuade people to do what is obviously in their own and every other user's best interests to do. So there now exists a significant loss of 'appeal' of Exetel's services caused by, by my estimate, something less than 1,000 users. For a business to lose a significant operating advantage and all of its other customers to lose a significant usage benefit due to the thoughtless and senseless actions of a relatively few users is a devastating condemnation of the management of any company that could allow this to happen - and I am solely to blame for this failure.
So, the current change to 10 hours has instantly fixed the problem caused by 1,000 users if the results of the first four nights of the new scenario is to continue. We will make some slight adjustments over the balance of August to attempt to get the maximum improvements from the new times and then I'm going on holidays for a few weeks. We will track the results throughout September and if the current results continue we will have, finally, solved the problem that has proved to be impossible for the past five or so years.
However we will have also accomplished something else. We will have developed the basis for returning to the 12 midnight to 12 midday 'free' download period by the start of November - assuming that the current usage patterns continue - but this time without the previous issue. It isn't a 'back flip' if in fact we can do that because to do it we would have had to go through this, to me unnecessary, period first. Of course, there'll have to be one minor change because I'm sure there will always be a tiny minority of people who will always try to screw things up for the majority because the ass**** factor is now a permanent presence in twenty first century life.
One of the unprovable 'business school myths' (similar to the "every unhappy customer etc" nonsense) is that EVERY service provider would be better off without 1% - 2% (depending how unlucky they are) of their customers.
Display comments as (Linear | Threaded)
I must admit I was one of those people who set there downloads to start at midnight, like you say it had a nice symmetry, I never realised that there was a problem with doing that until I saw your comments in these blogs, I then mended my ways, but I guess you can't assume that people will realise until it's pointed out to them that if everyone else starts their download at the same time will cause such a problem. I think the staggered start time was a good solution and I was going to use it to move my start time until 2am. Anyway I guess I had better adjust my start time to 3am so that I don't continue the problem.
Enjoy your time off John
I too am one who starts his off-peak downloads at 12am. However not once have I seen any communication from Exetel asking heavy users to defer their downloads to slightly later in the morning. If they'd have asked publicly then I'm sure some of the 1000 customers would have responded accordingly. Now that the off-peak time has changed, I'll just set my scheduler for 2am. However if it goes back to 12am to 12pm, then I'll leave my scheduler at 2am.
How do you plan on returning to the 12-12 schedule by November?
You say "We will have developed the basis for returning to the 12 midnight to 12 midday 'free' download period by the start of November."
"...assuming that the current usage patterns continue but this time without the previous issue."
What's the new formula? Staggered downloads perhaps?
It just goes to show how selfish some people are when the majority of people causing the high bandwidth problem don't really need a 12 hour window to download in, but choose to download at 12:01 because of their inability to think about the customers other than themselves.
I know there's been many suggestions from people to solve this issue about bandwidth hogs that start at midnight.
I think it would be not a bad idea to offer the default package to new customers / contracts as 2am to 2pm if you do decide to go back to 12 hours off peak in the future.
For those 1000 or so people that overload the network at midnight, if you can identify those customers (say 3 months in a row) you could send them an email advising them that they have been moved to a slower connection or advise them if they do not change there usage 'habits' then you will be moving there off peak allowance to 2am to 2pm due to the load they have put on the network.
That seems like a fair way to work with these customers.
I would also be interested in learning how you will be able to make these changes mentioned in your network.
One other option, would be to have 3 starts to the 'month'. Such as 1st of the month for a 1/3 of the users, 10th of the month for another 1/3 and so on.... That would spread the load of the network, so instead of everyone going crazy in the first 10 days of the month because they ran out of data allowance last month, you would be able to spread the load better through the month.
We will simply ban P2P traffic in the 12 midnight to 2 am period if the user elects to make that an 'off peak' period. If it remains as 'peak' then the customer can do as they like.
The penalty for selecting that period as 'free' and then using it for P2P downloads will be the removal of the free period completely for that customer or termination of the service.
As you mention 'We will simply ban P2P traffic...' how is this done? Through a filter type set up or monitoring bandwidth?
Not familiar with how these things work.
It can be done any number of ways but I was thinking of doing nothing other than checking that no P2P traffic was used by a customer using the period as part of their 'free' time.
I'm increasingly ass**** averse but I have come to understand that it will continue to be a problem.
The problem with that is, for example, a P2P download which starts well before 12am and may well be low bandwidth. What you should be looking for is consistently increased usage in (say) the 12.00-12.30 period. It is not an easy problem to solve, otherwise you would have solved it 5 years ago...
I am wondering would Steam Downloads be counted as Peer 2 Peer by Exetel. In other words if the plan returned to 12-12 would I be able to download steam game from 12:30??
I am currently an exetel customer
Games playing wouldn't be counted but it is premature to discuss the intriaies of something that is in the future.
First of all, I have to say that my connection is one of those 1000 (I say connection, and not me, because it's my brother with his torrents which start at midnight - I don't really have a need for P2P).
However, I was not aware that there was a problem at all. You could have sent an email to the 1000 users to alert them of the problem.
I am still a bit confused. What exactly is the problem? Is it the fact that everyone starts their downloads at exactly the same time? How would a staggered start help? If everyone staggered their downloads between midnight and 2am, there would still be the same traffic from 2am-12 midday.
Hi. My household is also one of the 1000 users, most likely. We were also unaware of the issues with starting our torrent schedulers at 12am and had never been made aware of it by Exetel.
It's a bit rich to call us selfish assholes, don't you think? All it would've taken was a simple email to the customers base to explain the issue.
Now that I've been made aware though (via being linked to the IT News article on Facebook randomly enough), I'll talk to my flatmates tonight about staggering our download traffic between 12am - 12pm.
See? All fixed. Most torrent downloaders aren't idiots and most of us really like using Exetel. Don't underestimate us, John.
Perhaps you would check your email contact address?
Several emails were sent asking customers to do exactly that.
And, yes, Exetel and I personally really value customers like you.
Just checked my email history and I've been on the Exetel Notifications list since November last year.
The only email I remember that had anything to do with this was the email about the possible change of the off peak time a few months ago. But it didn't mention anything about schedulers being an issue...
So yeah, if an email was sent in this time, the issue definitely wasn't made clear to me (and by a few others if the comments are anything to go by). Not being a nit-picker - all I'm trying to point out is that there may be other customers like me who simply aren't aware of the issue and are perfectly capable of helping Exetel out by adjusting their download schedulers.
There were three emails - it is irrelevant now - nothing can change the past.
Great to hear you have a holiday booked!!! And more than one week too!
I was getting a bit worried.
Hope you can leave your phones/laptops/anything else that communicates behind too.
This one didn't take long to hit the news...
I really have to learn to write Engish more clearly - but it is far too late in my life to learn that skill apparently.
Is it me or did they take JL's words and mix them around a bit to create a more scandalous article?
Well it's neither what I said nor is it true... perhaps it merely adheres to the stand by of all unprinciled hacks - if you can't get a real news worthy quote - invent one.
It must be a slow news day.
That journalist should probably read this; http://www.brisbanetimes.com.au/opinion/trust-like-respect-must-be-earned-and-the-media-take-that-job-seriously-20090804-e8ko.html?page=1
I remember getting the email from Exetel encouraging people to consider moving their download queue to start at 2am to 'spread the load' or somesuch.
It really should have been a 'demand' rather than 'request', given the critical impact on the network of 12am starts (which wasn't clear in the email). I recall thinking 'Well, if everyone moves to 2am won't that just shift the problem to 2am!' so I didn't change.
I have now been rewarded for that well-intentioned thinkng by having my off-peak time reduced to 10 hours, while everyone on autopilot who shifted to 2am is now continuing to enjoy 12 hours.
Oh well, doesn't really matter. I look forward to the return to 12 hours, and will leave my downloads starting at 2am this time
Well.....one way or another we may be able to ensure that every Exetel customer continues to benefit from Exetel's attempts to provide as much 'free time' as possible with as much 'free time as possible'.
...more than one way to skin a cat?
Why not just throttle P2P traffic very heavily in the 12-2am bracket for everyone? Put it in the small print for the terms and conditions and some other more obvious place where people can find out about it.
We could do that - but it isn't a preferred solution.
yeah they could do that. the problem is that exetel will have to configure another server or someone will have to manage it etc. still costs the business (not as much as the 150k but it still does) i am happy with my 2am to 2pm change (as i changed it before the mandatory change) i really dont care when my p2p things finish downloading as long as they have finished downloading that night. even if u kept it at 10 hours wouldnt worry me.
So, throttle P2P at midnight, down to something bearable, like 64k per user, or X % of the overall backbone. And then progressively unthrottle as the offpeak period progresses.
This manages your bandwidth, and also removes the value in anyone scheduling a torrent at 12:01.
A bit of technical embuggerance to change the throttle every 15 or 30 minutes, but you should only have to develop the solution once.
Just a thought, and maybe you've considered it already.
You're quite right - it would be very, very easy.
There is one major technical problem with it (at least for us), which is why we abandoned that option a while ago.
Care to elaborate? "Very very easy" != "major technical problem"
No - 5 years experience makes it a non starter.
There's no need for exetel to do it anyway. Any decent torrent client has scheduling built in with bandwidth throttling. e.g. utorrent
It allows me to set my peak speeds, off-peak speeds as well as total off periods while I'm playing my MMO's etc.
I'm only a new convert to Exetel so missed all the emails apart from the reduced off-peak period emails.
However, have you thought about letting the customer chose their peak/off-peak periods? e.g. I'd be happy with a 12 hour window that started anywhere between midnight and 6am. As long as my bulk downloads happen while I'm either asleep or at work I don't really care
If 3am - 7am still has a mass of unused bandwidth, would it be possible to offer higher offpeak quotas to those interested and reduce their offpeak hours to strictly 3-7am?
It does, and it would but what are people going to use mor than 60 gb for at the moment?
The same thing people use up their 60gb with. Considering the popularity of TPGs former 200gb plan, it seems people are finding use for high quotas.
In my opinion you have solved the problem already.
I see many worse problems with returning to 12 hours but voluntarily banning P2P in the first 2 hours or worse getting rid of customers who "abuse" the service in the first 2 hours.
I am a big stickler for simple rules, a 12 hours off-peak period with a fixed generous allowance to me means 12 hour of use. There should not be onerous conditions as many family members, especially children, are hard to control. What will happen next? You can’t start P2P at exactly 2am?
Another problem, what if you have paid for a generous peak allowance such as the INC2DSLE 54GB plan and hence use P2P at all sorts of times in peak. You have to remember to stop at 12 midnight everyday otherwise you may get asked to leave. A little bizarre.
You had the SAME problem with your previous unlimited* off peak period in the old days. It caused huge problems with some people downloading 300GB which stopped DEAD once you had a fixed quota with charges for going over. Simple is best. A philosophical attachment to an idea is sometimes not the way to go.
What is wrong with a 10 hour free period? It still practically the most generous one out there.
I'm a brand new Exetel User, been with you for one week. I just started my downloading a couple of days ago and automatically set it to 12am start based on what I read through when I signed up. As a business manager I can understand where your coming from however I didn't have the objectivity to realise that this would create a problem. I am more than happy to set my scheduler to start a little later, as I believe most of your new customers would be if there was something in your SFOA which explained the direction intended behind your "free/bonus" data periods and that you "may" have to review these timeframes if they are abused. At least then there would be something to refer back to now rather than us users unknowingly causing you heartache. Looking forward to many happy years of service from Exetel and I am about to have a best mate sign up in the next few weeks too.
I apologise if I have missed this somewhere in one of your posts but won't this just move the problem from 12am to 2am and potentially make it even worse because those people that have switched to scheduling their downloads at 2am now have a whole bunch of other users doing the same?
To get back to a simpler solution. Wouldn't it be easier to have some plans with off-peak and some plans without (which I believe you have in some limited fashion)? Those that have off-peak allowances just have to use the limited resources available whenever they decide to download data during their off-peak time (and deal with a degraded network) whereas plans without off-peak data never have to deal with a degraded network because they pay for everything they use?
If you look at the MRTG reports the move to 2 am has solved the problem completely over the past five nights.
But is this just because people haven't moved their downloads yet? Some people may not even realise that their downloads aren't working properly from 12am. Just saying it might be a bit soon to make a judgement.
Exetel has always been a leader, not a follower. How about:
scrapping peak and offpeak; OR
reducing offpeak and increasing peak; OR
reintroducing the separate "bucket" of bandwidth for some users eg allow 100Gb but limit their bandwidth so their 100Gb is slower than everyone else's 60Gb.
If you offer 60Gb offpeak you're encouraging people using all of that. How about reduce it back to say 40Gb and at the same time increasing peak by a similar amount.
How did the local time zone system work? It seemed to only be in for a week when you went with the new system.
Final option - keep the 2am start but have a 2pm finish. As a business customer, I'm happy with 30Gb no peak/offpeak. And I don't notice ANY speed difference regardless of the time of day.
For a slightly different approach consider the method used in the industrial electricity supply market where high peak loads add to the cost of infrastructure.
The customers bill has two costs, the total amount energy they consumed and a cost for the peak loading based on the highest load in a 15 (or 30) minute interval during the month.
Mapping this to the ISP industry could mean setting a surcharge for sustained high speed transfers. Low speed customers would not be affected since they could not physically achieve the speed and high speed customers could use the common software speed limit settings to avoid the extra cost and the resulting statistics would improve your utilisation factor.
This needs no extra equipment in Exetel and does not disrupt a customers ability to use their service by speed or protocol but matches demand with supplier cost.
Regards C Bumkin
While I don't come close to using the 60G offpeak quota, I, too have previously scheduled my downloads to start at midnight. I (like others posting here) have been unaware of the 3am-7am 'sweet spot' you describe. Sounds reasonable. But it was not communicated to me. You say emails were sent out - either I didn't get them or it was buried somewhere in a newsletter I skimmed through. If it's that important a matter, perhaps you need to look at how to most effectively communicate with your customers.
Btw I don't have any problem with reducing the hours of off-peak usage - I'm still extremely happy with the services and allowances you provide
What about randomly handing out off peak times? Each subscriber gets put into a pool of either 12am to 10am, 1am to 11am, or 2am to 12pm. Perhaps they could choose.
Then your 'off peak peaks' will be reduced by 2 thirds, spread out over a 3 hour period.
John, I do appreciate that through your blog we can have insight into the running of Exetel and how we can personally make a difference. We all benefit in this manner, and I certainly don't want to jeopardise the benefits we receive by behaving badly.
Still, until I read it here, I was unaware that starting at 12 was such an issue. For one, looking at the MRTG graphs, I think I'll push my scheduler start time till 3am anyway, just to avoid that 2am peak...
On a moderately unrelated issue; there is one 'killer app' for a lot of bandwidth that I'm not sure you're aware of, John. It's not really big in AU yet, but we've started renting HD movies and buying seasons of TV via the iTunes store. I'm pretty sure that was around 10-15 GB of our bandwidth last month.
Anyway, keep up the good work. I'll continue recommending people. I may be a high bandwidth user, but I assure you my mother-in-law isn't.
Perhaps our bulk email program has some sort of bug.
I will have it checked but as I am on the same list (ADSL user) I always get those emails so I doubt it has a problem.
Anyway - perhaps you could check your main email content address?
I've gone back through all my emails received from 'Exetel Notification'. The only relevant emails I can find are:
15 April: 'Possible Change to Off Peak Period', asks for comments but no explanation in the email.
24 April: 'Slight Modification to Off Peak Period', tells us we can select our off-peak period.
17 July: 'Change to Off Peak Period', tells us why the change has been made.
18 July: (Duplicate of above, but clarifying start date)
30 July: 'Important Change Notice (etc)', flagged important, reiterates the importance of changing schedulers.
Were there any more emails than these? I cannot find any email that explains the bandwidth situation clearly, and asks us as customers to avoid the (eg) 12-1 am period for downloading. At least for myself, personally, I quickly read through 'advisory' emails and if there isn't anything important I'll forget about it straight away.
A similar thing happened with the off-peak period choice; I received an email regarding the ability to select it. I went back later to the customer site and tried to find the option, but it was buried, and I just assumed I didn't have the option.
Possibly a suggestion for future changes, is to provide the messages via a notice on the member site as well - which would then have the relevant URLs to make any changes. Then the email could also have a link to the member site, the notice, etc.
I'm glad you've finally found a solution.
I would like to know your thoughts about this solution adding value to the 72/54 grandfathered plan.
I would like to know if you would like users of this plan to stop their downloads from midnight to 2am.
From what I have read, the majority of your customers are on significantly cheaper plans, hence I have no idea if customers who pay a premium (for a premium service) are included too.
I would also like to know a limit on uploads that is sustanable with this plan (peak/off peak or flat limit). I try to limit the house to a 1:1 download to upload ratio.
I am asking because I have read how small a profit you make off your customers. I want you to make money off me so I don’t lose you as a service provider.
There is plenty of time to download after 2 am so that would be sensible.
12 midnight to 2 am is no longer 'free' so it woud also be more economic.
As another new Exetel customer I too signed up before the reduction but after the option to change times. I read about the 12am bandwidth issue on whirlpool and although Im not a heavy user was going to change to 2am/2pm so I didnt contribute. I knew in the T&C's it could be changed at anytime but it was a drawcard for me and initially I thought I was hard done by. Then I had a bit of a think about it and realised it didnt make any difference what so ever to me. The way I see it there are two classes in this; Browsers- who for whatever reason are up at 12am doing what everyone does on the internet and Downloaders- those who choose to use their offpeak times and quota that they signed up for. I class myself as a downloader and on anything over 512kb/s could use my entire quota in less than 2 weeks and on 8mb/s in only a few days even with 10 hours off peak so unless on the rare chance (never) I really need whatever Im getting right at midnight I couldnt see how over the month it affected me so I have since calmed down and instead felt sorry for the browsers because although they can finally use the internet it now goes on their peak allowance, although they might say its a good trade off. So at the end of the day Im still very happy with what im getting and I know you and Exetel will do whatever it takes to keep giving us a great value and quality ISP.
Instead of an opt-in system where you swear not to P2P, is it technically possible to return the off-peak to 12 hours but count P2P traffic in the first 1-2 hours as peak?
That way people can have the best of both worlds, they can P2P before 2am (though it still counts as peak) yet still have the ability to download from other sources between 12am-2am and make it count as off-peak.
Just one further question, somewhat related: What's the situation with uploads?
As well as P2P transfers, I also upload a fair few HD slideshows and photos for my business. Is there a preferred time for uploads? Is there any sort of bandwidth issue or are uploads essentially 'free' from Exetel's perspective?
I must add, I appreciate that uploads with Exetel are unmetered. It galls me when I see the major providers counting both down and up towards the quota, although with the prevalence of P2P I can see the business case in it.
Upload capacity never approaches even 60% of available capacity so any time is as good as any other.
In todays world of 'status updates', 'tweets' etc people are getting more and more used to succint, instantaneous modes of communication. Couple this with email inboxes full of newsletters, junk and advertisements and it is little wonder that people either a) didn't read the email b)skimmed the email and did not digest the content or c) didn't get the email.
Have you considered going with the flow and moving with the times, and perhaps communicating messages that you see as so vital in a more accessable fashion?
I have been with Exetel for around 4 years total and in that time I have had 2 problems. I remember receiving an SMS update on the progress of my fault. Why not use that same technology?
I do not remember receving this email, and to tell you the truth, I hardly ever read the emails anyway.
I have no problem with the changes that you are implementing to combat this problem, and I will do my part to support your company and help alleviate any problem that I may be a part of, but I feel that perhaps some of your frustration could have been avoided.
Thank you for your support.
I've thought about my personal and Exetel's general failure to communicate for quite a while now.
The conclusion I've come to is that i no longer have,if indeed I ever did have, the ability to communicate to multiple 'levels' of what now represents a residential customer base of over 100,000 users.
Worse, I have reached an age and level of tiredness, I hesitate to use the word boredom but it might be more accurate, whereby I no longer really care.
I think the only real solution is to find a replacement who is both better attuned to the 'audience' and has sufficient energy to deal with the frustrations.
I don't any more.
Well I doubt that at the inception of your company you did not have the ability to communicate with all users.
In one way, it is a shame to make that transition from a small community where people talk, people listen and people cooperate with each other to ensure the best possible outcome... to a larger evironemt, where no one cares, everyone has a sense of entitlement and people only change their habits if they are forced to. No one feels privledged to be a part of Exetel anymore, and it is a shame. I know when I first joined, I was refered by a mate.
It was always our little secret that we had a way better ISP than everyone else.
It would be a damn shame to lose the person that started it all.
I do understand your frustrations though. Exetel is now a massive machine. There is very little control that you can excercise just by asking nicely.
It's simply a fact of both corporate and personal life.
We all get down in the dumps sometimes.
But it's really sad to see such a dynamic businessman have such a defeatest attitude.
I hope you rise out of the darkness soon John.
There is a diffeence between callow people who take set backs immaturely and old men who finally have to accept that decades of unhealthy living and extreme personal and business pressure have wrought more physical damage than they chose to notice.
Get over it John. It's the way of the business world. Gotta keep moving forward. Showing weakness with your defeatist attitude will help no one.
Only a fool expects his abilities to remain at their zenith for ever.
Why not just make a free period during whichever time the link is at the minimum usage? In a way retain the off-peak bonus(or cut down by abit) data usage while the same time giving free data usage during time where link is at the minimum.
I am sure leechers will do whatever they can to adapt
Great blog post and I agree, the 1-2% is spot on. Something that i do like is spreading the times for different plans like Netspace has done to try and minimise this problem.
55gb plan offpeak is 5am to midday, 160gb plan is 3am to 10am.
Another one of the 1000 here. Prior to the change I would have my utorrent scheduled for 12 midnight - 12 noon, meaning all my downloads would start at midnight and be done by about 1am.
As others have stated, you shouldn't assume the worst about your customer base - I simply has no idea that this was causing a problem, and would have been happy to change it had I know. All it would have taken was a brief, plainly-worded email saying "hey, if you're running schedulers and don't need them on all night, can you please set them to start at 2 or 3am instead of right on midnight".
I guarantee that myself and many others would be happy to oblige in order to let you keep this great service for us. I moved my scheduler back to 2am after the change, but knowing what I do now I'm quite happy to put it back another hour to 3am.
Thank you for your consideration in doing that.
We did actually send out two emails asking for that to be done.
We also opened a forum thread on the topic and included the request in the monthly news letter.
Hopefully we have now found a sensible solution.
External PHP Application