The Limits of the Catalogue

I have a problem. I don't normally like to talk about it, but you may share the same problem and if we get it out in the open, then maybe we can all cope with it better. The problem with this problem, is it doesn't have an obvious name. So while no one likes to be labeled, without a label it is hard to communicate easily. How to describe it? Where to begin?

The problem stems from the nature of the product we supply and how it is handled by EDI (Electronic Data Interchange). These widgets come in different sizes. There are industry standard sizes. 10 different widths, 15 different heights, 5 different depths.

So that is 10 x 15 x 5 = 750 different sizes.

Then there are the colors. We bring out new ones each year, some colors are retired and some popular classics are always going to be available. Say about 25 colors are current with about 5 changing each year.

750 x 25 = 18,750

Don't think these are just boring old boxes. We have several 'styles' to choose from. Contemporary, Classic, Gothic, Art deco. The list goes on. Like colors, say 15 styles with about 3 changing each year.

18,750 x 15 = 281,250

As an available option we can supply a low power economy version, standard, high power or "Max Power" business version.

281,250 x 4 = 1,125,000

They can be with or without an extra adapter port. With or without the toughened, rubberized embedded protection. With or without an "easy-grip" handle.

1,125,000 x 2 x 2 x 2 = 90,000,000

That is 90 million different products! I will stop there but actually there is more. We don't just do widgets. We do widget fittings and widget accessories. We deal with products that are alternatives to widgets and complementary to widgets. We also do bespoke "made-to-measure" widgets.

Now here is the killer contradiction. By volume and by value, 80% of everything we supply is covered by 2000-3000 products, so we give them individual product codes. The catalogue is more than 50 pages. However this 80% of products only gives us 20% of our profits. The bulk of our earnings come from the 20% "non-standard" product.

Now I am ready to confront my problem, "My name is EDI Eddy and my catalogue has been living a lie!".

Most EDI assumes ordering is done by product code reference. This means that for a customer, with a computerised (and EDI enabled) purchasing system, to order from us, they must set up a database with millions of entries. Even if this was achievable, problems still remain.

• How does a purchaser find the correct product code in their database?
• If it is difficult to order electronically, will they order manually or order from some one else?
• When our range of products change regularly, how do thousands of database entries get added and deleted?

So the reality is our EDI is limited to the low margin, less valued end of our product range. Over the years we have been constantly expanding EDI, handling higher message volumes, supporting more message formats and more delivery methods. The reality is EDI has been getting less and less important :(

It is an Ill wind that blows nobody any good - from Berlin to Haiti

For as long as I can remember I have been reading the phrase "EDI has its beginnings in the Berlin Airlift in 1948". The next bit of history usually jumps to the 1960's American Transport industry. I read this again recently and for once it got me wondering. What electronic communication was there in 1948? What message was it and in what format, and how was it delivered? How was it developed and agreed? What was it about the Berlin Airlift that needed an EDI message?

The details took quite a bit of tracking down (I have included some links below), but the best source I could find for this bit of history is one Maj. Edward Guilbert who was a traffic manager in Berlin. On leaving the army he went to work... ...in the transport industry.

To answer the questions above,

What electronic communication was there in 1948? Teletype.

What message was it, in what format, and how was it delivered? Manifest or Advanced Ship Note. No examples are believed to survive. It was radioed by teletype to Berlin from the planes departing airport, as soon as possible after the planes wheels left the ground.

How was it developed and agreed? You're in the Army now. You do as you are told. I don't know if the supplying airports and plane companies adopted the same procedures for their other business, but I doubt it. But it made clear what was possible.

What was it about the Berlin Airlift that needed an EDI message? In a word "Bottleneck" Every last kilo of supplies was vital but it couldn’t just sit on the ground at the Berlin airports. At the rate of a plane every few minutes, parking space for planes would soon disappear. Even if planes unloaded and departed, hangers would soon fill up. The goods needed to be moved on which meant the right people needed to be ready to receive the right goods.

There is a wonderful story about how they tried to stop the incoming pilots disappearing for a hot drink and a snack, by supplying mobile refreshment manned by pretty young frauleins, all to shave valuable minutes off the turnaround time.

It struck me that what was game changing about this was not Electronics or Message Formats. It wasn’t about Cost Saving. When you are fighting a Cold War, cost is no object. It seems to me that what made all the difference was Standardization. Different parties, all did the same thing, in the same way, to speed up processes that would otherwise jam up and slow to a crawl.
The Berlin Airlift would make a good motion picture. The struggle to overcome adversity through perseverance and ingenuity. How the little guys succeeded by teamwork and coordination. I can see it now, “EDI – The Movie”. Well OK, maybe not.

When listening to news reports of the Haiti earthquake relief efforts, I was struck at the similarities in the problems they experienced at the airport. It seems there is nothing new under the sun.

Computerworld Article - 1910 Telegraphs to XML

ECommerce Google Books - The Maj. Edward Guilbert Story

Ecommerce Connexion Artitcle - A short history summary of the Berlin Airlift

When Lost, it helps to remember where you want to go

I have the occasional high school student pass through my office on work experience. They do the rounds going to each department. When they come to me they have already spent time in the Sales and Purchasing departments. I am expected to give them a 30 minute overview of EDI (I don't just do EDI, but no one else understands it, or wants to understand it, so that is what they ask me to do). Then they move on to the help desk where they occupy them with tasks, like prepping desktops. They always like that, but I can't claim any are enthusiastically interested in what I have to say. I find the exercise helps to keep my feet firmly on the ground and head out of the stars, as they sit there wondering (despite my best efforts) what I'm talking about.

Well, the other day it was the turn of the one of my IT colleagues offspring. Keen to make a good impression, I was spurred on to think up a new and better explanation than I had used before. I was pleased with the results so I thought I would reproduce a version of it here.

***

When you enter Sales Order Processing department, you will see people opening the mail and extracting customer purchase order forms. Some of them are hand written, some are computer printed.

There is also have a Fax machine dedicated to customer orders that spews out a steady stream of order forms. Again, some of them are hand written, some are copies of computer generated order forms.

We also have a team of people who take orders verbally over the phone.

Finally, some customer orders come in as emails. Usually as PDF or Microsoft Word attachments. All these orders are read and the details typed into our Computer system.

This raises 2 questions, and leads to a 3rd great big "What If" question.

Q1. Why aren't all orders emailed?

It is cheaper not to have labor deployed in the mail room, or manning the phones, or maintaining the fax line and fax machine. The cost of receiving emails is tiny in comparison. When you scale up to hundreds, thousands, tens of thousands a day - for emails, the cost graph raises far less steeply. The same is true for our customers. Most of them are companies and organisations. It is cheaper for them not to pay for postage, or phone calls.

So I repeat the question. Why aren't all orders emailed? The answer? I don't know, you should ask the customers. But the point is, whatever the real answer (and it might be different for different customers) take a note of how difficult it is to change our customers behaviour. We can't afford to turn away business by not accepting the orders and we don't want, or to make it hard to order from us.

Q2. Why do all orders have to be re-typed into our Computer system?

Every product is made up of many components and materials. All of which have to be purchased. Every product requires many different sorts of resources, people and machines. All have to be planned and instructed what to do. Controlling all these aspects takes a complicated Computer program (called an ERP system) and that is why they have to be entered.

Q3. What if, instead of having to re-type everything, you could click and drag the email attachment, drop it on the ERP icon on your desktop, and the software processed it for you?

I will tell you "what if". If you could do that you will have achieved what businesses and organisations have been trying to achieve for decades. This is the ultimate aim of EDI.

Why is this so difficult? To a computer, text, is text, is text. How does a computer tell one piece of text is a product description and another is an address, one is a quantity and another is a price, one a delivery date another is a created date? To humans this is easy. For computers this is hard.

Ever tried to open a Microsoft Word document with Notepad? That is what the file looks like to a programmer. What do all the non-text characters mean? Only those who have signed a non-disclosure contract with Microsoft know, and none of them have permission to tell you.

First thing you have to do is agree a document format that is open to everyone. Then everyone has to agree where to place each bit of data or how to label it. In short, globally speaking, we can't agree. Many have been expecting one of the many formats to emerge as the dominant one, like the way every one uses MP3 instead of WMA or WAV, but this hasn't happened.

The next thing is product identity. We distribute a catalogue describing all our products and product options along with product codes (you see these bar-coded on the packaging). We encourage our customers to quote these in the order so that there is no miss-understanding. How do our customers (and their computer systems) know this information? If they order the same product from different suppliers, each will have their own code.

At least sending the document is easy. I mean everyone uses email, right? How else would you send electronic data from A to B? Er.. hmm... well... No. We have been moving electronic data around since before email and the Internet. There are dedicated EDI networks. There are Extranet web sites. Third party hosted web app clearing houses. There are different ways of securing emails. There are different encryption solutions. Security Certificates. Web of trust. etc. etc. etc.

So what are we to do? We could pick one solution and tell all our customers to communicate in that way. But remember these are the same customers, some of whom are still using post and fax. Remember how difficult it is to change some customers behaviour? Some of the big customers might prefer a different solution. We won't turn them away, but very quickly we find ourselves supporting many different forms of EDI. The cost of implementing another one has to be weighed against the volume of business (and cost saving) we can expect in the future.

Now turn your point of view around. When we order from our suppliers, we become the customer. The complexity just doubled. I have only talked about orders, but there are many other documents that are exchanged in business.

***

Are you Lost? Can you remember where we were heading? Any bright ideas?

CSVML - Accept No Compromise

Can't decide between XML and CSV? I have the answer, and it is not JSON as I previously thought.

I have seen the light.

The answer is here. Hilarious! (well it is hilarious if you are a geek).

In a future post I shall show how the future of EDI (Electronic Data Interchange) is a merger of Edifact, X12 & Tradacom by using Facsimile technology...

Green Coffee XML

I am not kidding (pdf). Some might think this is great. Some might think is shows how wonderful XML is. I don't. To me it represents a lot of what is mixed up about EDI (Electronic Data Interchange). I want to make 2 points...



What is so special about Green Coffee that it needs it's own schema?

• Well reading the docs it seems coffee dealers are a bit fussy about defining when ownership of the product and ownership of the risk (associated with product delivery) is transferred. So they have 9 different order types.
• As well as the buyer and seller, they need to be precise about the Broker and the Shipper.
• The quality of the product is defined by a standard and is reflected in the product codes.
• Pricing can be by formula.
• Unit of measure is usually Kgs but when it comes to weighing coffee it seems to be important who weighs, when, and who pays for the weighing. I count 8 weighing types.
• The journey coffee makes can be long and the value of the coffee at different stages changes so it seems the "place of tender" is important. A simple "delivery date" is not precise enough and must be qualified.

Phew! Complicated. But excuse me. Is any one of these points unique to coffee? Maybe the combination is unique. Maybe it is more sophisticated than Acme retail EDI. But what does it gain us to reject all that has gone before in 60 years of EDI and create new EDI ghettos ?


I hope they didn't. I hope they just defined some extra tags and specified some extra attribute values, and added them on to some existing, already utilised and proven XML order standard. Which brings me to my next point.


How (for the love of coffee!) can I implement this?


I went in search of the technical details. The PDF document listed 4 XML Appendices on the contents page. They seem to be missing from the web. I went to root URL and clicked around. I couldn't even find my way back to the document. I used Google to search the site - zilch. I used Google to search the web for "Green Coffee XML", no luck.

How can you expect a schema to be used if you wont tell anybody the details? If you want it to succeed make it freely available! Have you not heard of Peer Review?

Is this what hm.gov.uk thinks is EDI? - Revisited

2 CDs with 7 million names addresses of children and parents, some with bank details, are put in the post and go missing. When I heard about this story I worried about the little guy. Well now the UK Independent Police Complaints Commission have investigated and released their report. It is well worth a read (pdf). All 61 pages and 282 paragraphs.

At the time the Minister was quick to publicly blame a "junior official" not following the "rules".

But what if there are no rules? Or too many rules? Or rules that constantly change? Or no one responsible? Or responsibility shared by too many senior people? Or if everyone is responsible it means no one is responsible?

It looks to me like the little guy was trying his best. See paragraph 130

He forwarded this email to Employee J in IMS and asked him to provide the 12 records as requested. Employee F included the following explanation in his email to Employee J:
…All we wanted was for NAO to realise exactly what they were
asking for, i.e. the scan data is live records of seven million Chb
customers when they only want to look at a dozen cases from
the scan. More importantly we needed to get the assurance of
how they would securely handle the discs containing the data
and how they would dispose of them once they had completed
the checking.
Obviously NAO should automatically realise this confidential
data has to be protected and no doubt they would do so.
However we needed something more than a verbal request to
ensure we had the paperwork to back up the request, things do
get mislaid and imagine the uproar if the discs containing the
ChB customer data went astray and turned up where they
shouldn’t – the long knives would be out. At least we would be
covering ourselves by getting the right assurance.

Is this what hm.gov.uk thinks is EDI?

Electronic Data transfer in the news. I think it is called pigeon protocol.

No, you can't ridicule it because it is too serious. Unfortunately I suspect this "behind the mailbox" infrastructure is all too common.

I hope the little I.T. guy archives his email.

EDI Assumptions

I was starting to sketch out my next post. It was going to be on EDI (Electronic Data Interchange) System design and I started with "Assuming you have an ERP system that processes sales orders, invoices, remittances, stock balances etc. how do add EDI capabilities?"

Then I thought wait!!! Important EDI life experience lesson - assume nothing!!!

EDI has a wide reach. It is used in all sorts of areas. While I am most familiar with EDI in ERP systems, it is also used in health care, logistics, military etc. So I thought I'd better at least provide a link to explain the term. This got me thinking about the different types of people who might be interested in these posts, what your situation is and how you came to EDI. And this brought to mind 2 experiences that challenged contrasting assumptions.

1) Assuming Everyone is Big

After many years of being familiar with ERP systems running businesses that employ many people, I had a shock. I came across a software package being used to run a small family service business. It was the sort of thing you can buy off the shelf in PCWorld. It was one of those friend-of-a-friend, help desk situations. I took the opportunity to familiarise myself with it, always looking to learn something new.

My shock came from the fact it had an invoicing function but no orders or quotes. It could purchase but had no stock control. Most of the functionality I thought was needed was missing. How could they run their company with this? The answer was, quite well thank you. You see they did have an order book but the number of orders you could count on your fingers. High value, but low volume. They didn't have a product line, no two jobs were the same. They would produce whatever the customer wanted. If they wanted to know what material stock they had, they looked in the shed.

It made me think. Could I configure my current, all singing, all dancing ERP system to run a similar company? Would it be as easy to use? If they experienced rapid growth, at what point would they need additional functionality?

What if they got an important contract from a big customer who wanted to send orders by EDI and receive invoices by EDI? How would they cope? How would they benefit?

2) Assuming Everyone is Small

Three years later I am summoned to a meeting with our then biggest customer, a major retailer. Lots of suppliers are there as they announce their Electronic Trading imitative. They have done their research and decided that past EDI initiatives have failed because they were too technically complicated and costly for all their small suppliers. They had a solution and from the audience response it went down well.

All suppliers needed was a browser and an Internet connection. They could log on to a website, print off PDF's of orders, confirm acceptance, record supplier order references, alter pricing, update delivery dates, confirm deliveries and even read the latest news from our important customer.
The small suppliers, who had feared warnings of "integrate or die", loved it. Suddenly I could see this was the solution for the small company I had experienced earlier.

The big suppliers sighed. How were we going to explain? You see we already did all this inputting on our computer systems. To repeat it on a web page would dramatically increase our workload. We organised hundreds of deliveries a week to hundreds of stores. Confirming a full order delivery was 1 click, detailing a partial delivery (or split delivery) was click, type, click, type, click, type... Unless we could persuade them to add another option, we were going to have to throw human resource at the problem.

Meeting after meeting followed.They eventually offered an FTP connection, but although most suppliers already outputted information in one of two EDI formats, the customer wouldn't accept them. They invented their own format. Every time we requested more time, we were told we could go live with the browser interface.

After we did go live, the customer provided a telephone help line. When we called we would get nonsense questions about what we had clicked on the web page.

Me: [Sigh, keep calm] We don't use the web page. We communicate server-to-server.

Operator: Eh... I'll just speak to my supervisor.

Lessons Learnt?

Trading partners have differing levels of ability.

Trading partners have many varied business styles and processes.

When volume splits 80/20 and problems split 20/80, 80 x 20 = 20 x 80 (if you see what I mean). Both half's are important. Don't design a solution to one half without consideration of the other.

Maybe Mashups are the future.

Story from the Trenches

I'm working at this company. No one wanted to be responsible for EDI (Electronic Data Interchange). It was seen as a problem (not a solution), so it gets dumped on the new guy - me.

With a lot of patience and time I slowly train myself and begin to understand. I had inherited a third party communication and translation software. It used 1 of only 2 modems in the entire company - it was along time ago. Users hated it. Apart from processing everything twice, it kept breaking down.

Time passes and I slowly improve the system and increase reliability. More customers start to use EDI. We had several customers connected by EDI. 2 different standards. 2 different networks. They were sending us Sales Orders and we were sending the Invoices. Hey! I was proud of what I achieved and everyone else (who didn't want to touch EDI in the first place) saw me as an expert.

Then one day a key supplier suggested we send our Purchase Orders to them by EDI. Hmm EDI with a supplier, not a customer. This was new. So I ring up my contact at our EDI software supplier. I know I need to get the trading partner/document relationship validated on the VAN. Oh, and the software will need some mapping tables for purchase orders. I know they are going to cost.

Me: Can I buy the translation tables for Purchase Orders please?

Contact: According to our records you purchased them some time ago. See your reference XXXXXXX

Me: Let me see (my files are organised). Ah yes, I have it in front of me. No, that was for Sales Orders. I want Purchase Orders.

Contact: What's the difference?

Me: Duh! I see what you mean. Never mind. Click