Blog Detail

The Power of Prayer!

You are probably wondering what prayer has to do with telecommunications, read on and you decide.

In 1998 I was seconded from Optus Communications to work for Telecom Vanuatu Limited (TVL) in the role of Technical Manager. I was responsible for a group of 7 line managers and 100 technicians and every facet of the country’s telecommunications infrastructure. The infrastructure included an Alcatel Standard “C” International Earth Station, an Alcatel E10B Telephone exchange with a remote switch, a TotalTel Pre-Paid IN node, an Internet network with dual-homed core routers and leased line and dial-up access routers, a JNA DDN node, a 7 hop 34 meg PDH Microwave backbone, an extensive IRT-2000 TDMA Radio telephone network, and patchy underground copper cable networks. The network provided telecommunications services to the population of around 200,000 people. The two main population centres were the capital of Port Vila on Efate and Luganville on Espirito Santo. I lived and worked in Port Vila where the International Earth Station and Telephone Exchange were located. Vanuatu was, and still is, a Tax Haven and the tiny town of Port Vila home to some 300 offshore banks and hundreds of investment companies.

I’d been working for TVL for 6 months or so and felt fairly comfortable with my role when a fault was escalated to me. It seemed fax calls had stopped working both incoming and outgoing from Vanuatu. Fax machines could not connect for some reason while voice calls using the same international trunks were fine. This fault proved to be the most challenging and stressful fault I’ve ever encountered in my 25 years as a technician.

After some initial investigations we identified the bit error counters on the telephone exchange’s international PCM links were incrementing at an alarming rate. The links were running errors at a rate where you could just discern an occasional clicking on voice calls but all fax calls failed to connect. We also determined the international leased data services were running errors at elevated rates. None of the technical staff had been able to determine the cause of this sudden degradation of the international communications links.

It appeared the clocking on the international links was “slipping” possibly due to a network synchronisation problem. I asked my managers to explain how the network synchronisation was configured and my question was met with blank stares and shrugs of shoulders. My days working on analogue leased line faults in the ISTC at Paddington came flooding back to me. I thought to myself “I know how clocking works, how hard could this be”. I was about to find out!
Every day the French Managing Director (MD) would ask if we had made any progress. Each day customers would ring and complain until eventually the MD was forced to release a press statement “TVL is aware of a network problem and are working on it. We expected a quick resolution”.

Over the next two weeks I worked day and night with the managers and technicians trying to solve the problem. We started by identifying all the network elements, checking the clocking configuration of each to produce a clocking diagram for the network. We identified a 2.048megbps Rubidium Oscillator as the clock source for the Vanuatu network. The output of the oscillator was connected to the external clock input of the Alcatel E10B Telephone exchange. The telephone exchange PCM links clocked the Digital Cross Connect (DACS) switch at the International Earth Station which in turn clocked the IDR modems carrying the international traffic.

I asked if the output frequency of the Rubidium Oscillator was correct and was told yes – probably! Measuring the output of the Ribidium clock source with an aging frequency counter was giving misleading results. I asked if there were any spares for this unit and was told maybe. We found several spares of unknown condition and tried inserting them but the fault persisted. I contacted the manufacturer of the unit in the USA who explained as long as there were no alarms the unit should be ok!

Next we investigated the Digital Cross Connect (DACS) and IDR satellite modem configurations. The problem with this investigation was that if someone had changed a setting and not told anyone then it was possible the settings were incorrect. So we tried all the various combination of clocking strategies to see if we could correct the problem. At one stage we had a situation where we set the Vanuatu network to clock off the received clock from the Telstra Australia IDR modem, in this mode the international leased data services ran error free but the voice trunks began to slip more often and fax calls still failed.

After two weeks of making no progress I decided to eat humble pie and contacted Bill Millar, my old boss in Optus, to seek the technical support of the Optus Engineering group. I’d never worked in an Earth Station and finally accepted I was out of my depth. I faxed the clocking diagram for the network to Bill and we scheduled a technical conference for the following day. That night I phoned the MD and my managers and told them we’d have to call on overseas support to fix this problem.

On the morning of the conference call I awoke feeling tired and very stressed, as I got out of bed I said a silent prayer to myself “Dear God please help us fix this today”. I’m not a regular church-goer so this shows the desperation of the situation.

On the way to my Office I thought I’d check in at the Earth Station. As I walked in the Manager said to me “it’s fixed!” I said what do you mean it’s fixed? He said one of the Technicians had been preparing a new IDR service and had plugged in a lead where he shouldn’t have and when he pulled it out the slippage stopped. I asked how long had the lead been plugged in; he replied two weeks!

Well you can imagine my mixed emotions. I felt like a heavy weight leapt from my shoulders. I couldn’t believe the problem was fixed so easily. Given the high profile investigations over two weeks I also couldn’t believe no-one had noticed that lead.

The conference was called off and I thanked Bill gratefully for his offer of support. I tried to investigate who had done what, and why, but the explanation lacked sufficient detail to determine the root cause of the problem.

At the end of the day, at home sitting on my back balcony overlooking the lagoon, I pondered if someone had set me up to prove a point. Regardless the problem was fixed, I had a cold beer in my hand, and I could return to a more relaxed lifestyle. Thank God!