The Limits of the Catalogue

I have a problem. I don't normally like to talk about it, but you may share the same problem and if we get it out in the open, then maybe we can all cope with it better. The problem with this problem, is it doesn't have an obvious name. So while no one likes to be labeled, without a label it is hard to communicate easily. How to describe it? Where to begin?

The problem stems from the nature of the product we supply and how it is handled by EDI (Electronic Data Interchange). These widgets come in different sizes. There are industry standard sizes. 10 different widths, 15 different heights, 5 different depths.

So that is 10 x 15 x 5 = 750 different sizes.

Then there are the colors. We bring out new ones each year, some colors are retired and some popular classics are always going to be available. Say about 25 colors are current with about 5 changing each year.

750 x 25 = 18,750

Don't think these are just boring old boxes. We have several 'styles' to choose from. Contemporary, Classic, Gothic, Art deco. The list goes on. Like colors, say 15 styles with about 3 changing each year.

18,750 x 15 = 281,250

As an available option we can supply a low power economy version, standard, high power or "Max Power" business version.

281,250 x 4 = 1,125,000

They can be with or without an extra adapter port. With or without the toughened, rubberized embedded protection. With or without an "easy-grip" handle.

1,125,000 x 2 x 2 x 2 = 90,000,000

That is 90 million different products! I will stop there but actually there is more. We don't just do widgets. We do widget fittings and widget accessories. We deal with products that are alternatives to widgets and complementary to widgets. We also do bespoke "made-to-measure" widgets.

Now here is the killer contradiction. By volume and by value, 80% of everything we supply is covered by 2000-3000 products, so we give them individual product codes. The catalogue is more than 50 pages. However this 80% of products only gives us 20% of our profits. The bulk of our earnings come from the 20% "non-standard" product.

Now I am ready to confront my problem, "My name is EDI Eddy and my catalogue has been living a lie!".

Most EDI assumes ordering is done by product code reference. This means that for a customer, with a computerised (and EDI enabled) purchasing system, to order from us, they must set up a database with millions of entries. Even if this was achievable, problems still remain.

• How does a purchaser find the correct product code in their database?
• If it is difficult to order electronically, will they order manually or order from some one else?
• When our range of products change regularly, how do thousands of database entries get added and deleted?

So the reality is our EDI is limited to the low margin, less valued end of our product range. Over the years we have been constantly expanding EDI, handling higher message volumes, supporting more message formats and more delivery methods. The reality is EDI has been getting less and less important :(

When Lost, it helps to remember where you want to go

I have the occasional high school student pass through my office on work experience. They do the rounds going to each department. When they come to me they have already spent time in the Sales and Purchasing departments. I am expected to give them a 30 minute overview of EDI (I don't just do EDI, but no one else understands it, or wants to understand it, so that is what they ask me to do). Then they move on to the help desk where they occupy them with tasks, like prepping desktops. They always like that, but I can't claim any are enthusiastically interested in what I have to say. I find the exercise helps to keep my feet firmly on the ground and head out of the stars, as they sit there wondering (despite my best efforts) what I'm talking about.

Well, the other day it was the turn of the one of my IT colleagues offspring. Keen to make a good impression, I was spurred on to think up a new and better explanation than I had used before. I was pleased with the results so I thought I would reproduce a version of it here.

***

When you enter Sales Order Processing department, you will see people opening the mail and extracting customer purchase order forms. Some of them are hand written, some are computer printed.

There is also have a Fax machine dedicated to customer orders that spews out a steady stream of order forms. Again, some of them are hand written, some are copies of computer generated order forms.

We also have a team of people who take orders verbally over the phone.

Finally, some customer orders come in as emails. Usually as PDF or Microsoft Word attachments. All these orders are read and the details typed into our Computer system.

This raises 2 questions, and leads to a 3rd great big "What If" question.

Q1. Why aren't all orders emailed?

It is cheaper not to have labor deployed in the mail room, or manning the phones, or maintaining the fax line and fax machine. The cost of receiving emails is tiny in comparison. When you scale up to hundreds, thousands, tens of thousands a day - for emails, the cost graph raises far less steeply. The same is true for our customers. Most of them are companies and organisations. It is cheaper for them not to pay for postage, or phone calls.

So I repeat the question. Why aren't all orders emailed? The answer? I don't know, you should ask the customers. But the point is, whatever the real answer (and it might be different for different customers) take a note of how difficult it is to change our customers behaviour. We can't afford to turn away business by not accepting the orders and we don't want, or to make it hard to order from us.

Q2. Why do all orders have to be re-typed into our Computer system?

Every product is made up of many components and materials. All of which have to be purchased. Every product requires many different sorts of resources, people and machines. All have to be planned and instructed what to do. Controlling all these aspects takes a complicated Computer program (called an ERP system) and that is why they have to be entered.

Q3. What if, instead of having to re-type everything, you could click and drag the email attachment, drop it on the ERP icon on your desktop, and the software processed it for you?

I will tell you "what if". If you could do that you will have achieved what businesses and organisations have been trying to achieve for decades. This is the ultimate aim of EDI.

Why is this so difficult? To a computer, text, is text, is text. How does a computer tell one piece of text is a product description and another is an address, one is a quantity and another is a price, one a delivery date another is a created date? To humans this is easy. For computers this is hard.

Ever tried to open a Microsoft Word document with Notepad? That is what the file looks like to a programmer. What do all the non-text characters mean? Only those who have signed a non-disclosure contract with Microsoft know, and none of them have permission to tell you.

First thing you have to do is agree a document format that is open to everyone. Then everyone has to agree where to place each bit of data or how to label it. In short, globally speaking, we can't agree. Many have been expecting one of the many formats to emerge as the dominant one, like the way every one uses MP3 instead of WMA or WAV, but this hasn't happened.

The next thing is product identity. We distribute a catalogue describing all our products and product options along with product codes (you see these bar-coded on the packaging). We encourage our customers to quote these in the order so that there is no miss-understanding. How do our customers (and their computer systems) know this information? If they order the same product from different suppliers, each will have their own code.

At least sending the document is easy. I mean everyone uses email, right? How else would you send electronic data from A to B? Er.. hmm... well... No. We have been moving electronic data around since before email and the Internet. There are dedicated EDI networks. There are Extranet web sites. Third party hosted web app clearing houses. There are different ways of securing emails. There are different encryption solutions. Security Certificates. Web of trust. etc. etc. etc.

So what are we to do? We could pick one solution and tell all our customers to communicate in that way. But remember these are the same customers, some of whom are still using post and fax. Remember how difficult it is to change some customers behaviour? Some of the big customers might prefer a different solution. We won't turn them away, but very quickly we find ourselves supporting many different forms of EDI. The cost of implementing another one has to be weighed against the volume of business (and cost saving) we can expect in the future.

Now turn your point of view around. When we order from our suppliers, we become the customer. The complexity just doubled. I have only talked about orders, but there are many other documents that are exchanged in business.

***

Are you Lost? Can you remember where we were heading? Any bright ideas?

Green Coffee XML

I am not kidding (pdf). Some might think this is great. Some might think is shows how wonderful XML is. I don't. To me it represents a lot of what is mixed up about EDI (Electronic Data Interchange). I want to make 2 points...



What is so special about Green Coffee that it needs it's own schema?

• Well reading the docs it seems coffee dealers are a bit fussy about defining when ownership of the product and ownership of the risk (associated with product delivery) is transferred. So they have 9 different order types.
• As well as the buyer and seller, they need to be precise about the Broker and the Shipper.
• The quality of the product is defined by a standard and is reflected in the product codes.
• Pricing can be by formula.
• Unit of measure is usually Kgs but when it comes to weighing coffee it seems to be important who weighs, when, and who pays for the weighing. I count 8 weighing types.
• The journey coffee makes can be long and the value of the coffee at different stages changes so it seems the "place of tender" is important. A simple "delivery date" is not precise enough and must be qualified.

Phew! Complicated. But excuse me. Is any one of these points unique to coffee? Maybe the combination is unique. Maybe it is more sophisticated than Acme retail EDI. But what does it gain us to reject all that has gone before in 60 years of EDI and create new EDI ghettos ?


I hope they didn't. I hope they just defined some extra tags and specified some extra attribute values, and added them on to some existing, already utilised and proven XML order standard. Which brings me to my next point.


How (for the love of coffee!) can I implement this?


I went in search of the technical details. The PDF document listed 4 XML Appendices on the contents page. They seem to be missing from the web. I went to root URL and clicked around. I couldn't even find my way back to the document. I used Google to search the site - zilch. I used Google to search the web for "Green Coffee XML", no luck.

How can you expect a schema to be used if you wont tell anybody the details? If you want it to succeed make it freely available! Have you not heard of Peer Review?

Why isn't EDI easier? Part I - One Standard to Rule Them All

EDI (Electronic Data Interchange) really should be less difficult than it is. If I am a start-up company and I want to purchase timber, metal, paper, widgets or some other commodity item, EDI is just too hard.

The overhead on establishing a relationship, agreeing a standard, and testing communications is huge. EDI is supposed to reduce cost but if it results in a supplier tie-in then it will produce unwelcome influences.

If it takes a department of expensive-to-employ / prickerly-to-manage IT geeks to support, then this "cost saving" thing called EDI, just got too expensive.

The first thing that springs to most peoples mind is Standards. How may ways do we need to represent a Purchase Order? Why are there many Standards? Wouldn't it be easier to just all agree on one standard? The answer to the last question is, not necessarily.

To explain this, consider the classic example of Purchase Order Delivery date. On a multi-line purchase order, a customer will probably want all the items delivering together. But while some customers will send a delivery date in the order header section of the message, some will send delivery dates in the order line sections. Some will send both. Some will have differing dates in the detail line section. Some customers will omit dates altogether indicating the goods are required ASAP.

Don't stop me, I'm on a roll...

Sometimes it will be appropriate to send an Earliest and a Latest date range. This could be in either section. Given a free text field some will write 9/11/01, some will write 11/9/2001 and some computers will generate 20010911. By the way, not everyone in the world agrees what year it is or how many months there are.

At the other extreme, timing might be important. For example when ordering services like an aircraft flight time, or an insurance period start and finish. This brings in the question of time zones and daylight saving adjustments.

• So if a Standard is to be universal, it has to be large and complex.

• But if it is large and complex, it won't be easy to implement.

• And your next partner will use the Standard in a way that is at least slightly different to all your others.

When you read "X12 is popular in North America, EDIFACT in Europe, TRADACOM in Britain" remember, just agreeing one Standard doesn't solve anything. One Standard used in two different ways is like two Standards. So the incentive to converge isn't there.

I have been watching the ODF v OOXML Standards dust up with interest. The difference with EDI is that an EDI Standard has many thousands of individual implementations. If we accept Standards overlapping as much as ODF/OOXML damage each other. If interpretation of the specification will lead to differing implementations, which will lead to interchange problems between office applications. Then we shouldn't be surprised if EDI is in trouble.

I think this is the real source of the excitement over XML. It gives structure to data even without a standard. If we accept that even with a standard, there is a need for a "mapping" function, then maybe we should aim to make this as easy as possible.

EDI System Design - Overview

Part I - Message Control
Part II - Export Mapping
Part III - Import Mapping
Part IV - Transaction Processing

Story from the Trenches

I'm working at this company. No one wanted to be responsible for EDI (Electronic Data Interchange). It was seen as a problem (not a solution), so it gets dumped on the new guy - me.

With a lot of patience and time I slowly train myself and begin to understand. I had inherited a third party communication and translation software. It used 1 of only 2 modems in the entire company - it was along time ago. Users hated it. Apart from processing everything twice, it kept breaking down.

Time passes and I slowly improve the system and increase reliability. More customers start to use EDI. We had several customers connected by EDI. 2 different standards. 2 different networks. They were sending us Sales Orders and we were sending the Invoices. Hey! I was proud of what I achieved and everyone else (who didn't want to touch EDI in the first place) saw me as an expert.

Then one day a key supplier suggested we send our Purchase Orders to them by EDI. Hmm EDI with a supplier, not a customer. This was new. So I ring up my contact at our EDI software supplier. I know I need to get the trading partner/document relationship validated on the VAN. Oh, and the software will need some mapping tables for purchase orders. I know they are going to cost.

Me: Can I buy the translation tables for Purchase Orders please?

Contact: According to our records you purchased them some time ago. See your reference XXXXXXX

Me: Let me see (my files are organised). Ah yes, I have it in front of me. No, that was for Sales Orders. I want Purchase Orders.

Contact: What's the difference?

Me: Duh! I see what you mean. Never mind. Click