John Linton ...seems to be presenting more difficulties than expected.
Exetel was 'forced' to commission a second PoP last year by an archaic (it seems to us) Telstra Wholesale restriction that makes it mandatory to have separate physical locations if you want to terminate more than 1 gbps of capacity to deliver data via Telstra's ADSL1 network.
Exetel run all NSW ADSL1 customers in NSW via Telstra and we reached 900 mbps in the middle of last year and therefore needed to go to a second GigE link which, in turn meant that Telstra would only terminate a second link smewhere other than the single NSW PoP we then had.
We had a very small presence in the Verizon data centre in Ultimo where we terminated our VoIP traffic via Quintum switches with a small cross connect from the ADSL PoP in Clarence Street. So we decided we would 'build out' the PoP in the Verizon centre to duplicate the facilities we had already established in a second PoP over a two year time frame.
This, obviously, was a 'complication' rather than a 'simplification' but it was a mandatory requirement - in that we had no alternative if we wished to keep growing our NSW customer base. - which we did at that time; probably still do.
In any event we commissioned new cross connects and bought additional routers and servers and have begun the long, and difficult, process of splitting our major ADSL facilities across two separate locations. Everything has gone quite well to date until around 10.30 pm last night when we 'lost' half of our international bandwidth due to one of our cross connects that splits the 1.1 gbps Verizon sourced international bandwidth across the two PoPs and had some other, as yet undetermined router problem that managed that traffic.
It took almost five hours from late Friday night to early Saturday morning to diagnose the problem and eventually get the cross connect supplier to fix their problem so we could fix our problem and restore the services.
Later this year we will swap the Optus Sydney terminated bandwidth for Verizon bandwidth directly connected to the inter-State PoPs which wil provide true redundancy in that each of those PoPs will have its own Verizon feed plus a back haul connection to Sydney as a back up.
We will also truly split the Sydney Verizon feeds so that they don't rely on one cross connect but have dual paths to and from each of the Sydney PoPs.
So last nights problem will not occur in the future - however this adds complexity and cost - not the path to simplicity.
So the path to simpler has been obstructed by the need to accommodate an external requirement that has not been properly implemented by us; at least in terms of the redundancy that is required. This means that we will have to now further complicate the design by addng in a second/duplicate cross connect and then additional routers to terminate that cross connect at both ends and to 'manage the two routers in case either the cross connect or the routers terminating the cross connect fail.
So we end up with up to six extra routers/switch ports to provide redundancy for a part of the network that used not to be necessary.
We will start the replacement of Cisco 7301's with Cisco 10000s (on a 1 for 4 basis) next week which will cut down the number of LNS routers we deploy in Sydney from 12 physical boxes to 3 physical boxes that will be a simplification in both routing and switching if not in rack space (given the physical size of a Cisco 10000).
It's always with some hesitation that you move from a ''tried and true' network architecture (John Linton's non-engineering knowledge view has always been networks only break when network engineers try to 'improve' them) but it's now inevitable that devices that are limited to 1 gbps perormance are no longer sufficient for Exetel's customer's needs.
We will use six of the 'spare' Cisco 7301s in the planned Perth, Adelaide and Hobart PoPs over the next few months so financially we will derive a small benefit from this 'upscaling' in Sydney. However adding three more 'remote' PoPs is a further complication in a year that was meant to be concentrating on 'simplification'. In the space of a little over twelve moths we will have gone from one PoP located ten minutes from our offices to 7 PoPs located in every State capital in Australia.
We have yet to make a decision on adding PoPs in the ACT and the NT and will not look at that decision until we are 'forced' to do so and we hope that won't be for some months yet.
Hardly an obvious path to a simpler network deployment.