When do Standards become Standards ?

I want to promote good sense when I read it. Adam Bosworth has a very good article on standards. He is not talking about EDI standards, but technical and software standards in general. Yet everything he says I feel is applicable to EDI.

In summary, standards should be simple, readable, focused, precise, implemented, forgiving and free.

Adam's standards background is ODBC, AJAX and XML. These technologies are used so computer can talk to computer, servers and clients can talk to clients and servers, so Electronic Data can be... Interchanged.

Creating an FTP Script

In the last post I discussed using a web FTP service as an alternative EDI service, so I thourght I should also provide some help with FTP scripts. You can use a graphical tool like windows explorer for example by typing ftp://someserver.net in the address box. However to eliminate the work of the human operator as far as possible, scripts are needed.

That said the best first step to achieve this is to make sure you can do everything manually first. So drop to the command line. On a windows system click Start > Run, and type CMD. Now try and replicate something like the session below, the local machine prompts are in red, the ftp server responces are in blue, inputs in black.

C:\Documents and Settings\ACME>
C:\Documents and Settings\ACME>ftp ltoons.com
Connected to ltoons.com.
220 ready.
User: ACME
331 Password required for ACME.
Password:
230 User ACME logged in.

ftp> cd /outbox
250 CWD command successful.
ftp> put acme-bbunny-321.txt acme-bbunny-321.tmp
200 PORT command successful.
150 Opening ASCII mode data connection for put acme-bbunny-321.txt.
226 Transfer complete.
ftp> rename acme-bbunny-321.tmp acme-bbunny-321.txt
350 File exists, ready for destination name.
250 RNTO command successful.

ftp> cd /inbox
250 CWD command successful.
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
  bbunny-acme-123.txt
  dduck-acme-88.txt
226 Transfer complete.
ftp> get bbunny-acme-123.txt
200 PORT command successful.
150 Opening ASCII mode data connection for bbunny-acme-123.txt (999 bytes).
226 Transfer complete.
ftp> del test.tmp
250 DELE command successful.

ftp> quit
221 Goodbye.
C:\Documents and Settings\ACME>

This follows the procedure laid out in the previous post. Sending a file, listing files to be fetched and fetching them. If you can do this then you are well on your way. If not then any amount of program testing and debugging wont help you.

The next stage is some quick and dirty code just to get the job done. This example is in Python.

from ftplib import FTP

host = 'toons.com'
user = 'acme'
password = 'password'
filestosend =["acme-bbunny-321.txt"]

# Log on

ftp = FTP(host)
ftp.login(user,password)

# send

ftp.cwd('/outbox')
for filename in filestosend:
fileinput = open(filename, 'r')
ftp.storlines('STOR tempory.tmp',fileinput)
ftp.rename('tempory.tmp',filename)

# fetch

ftp.cwd('/inbox')
filestofetch = ftp.nlst()
for filename in filestofetch:
if filename[-4:] != '.tmp':
  fileoutput = open(filename, 'w').write
  ftp.retrlines('RETR ' + filename,fileoutput)
  ftp.delete(filename)

ftp.quit()

Does this work? If it does then well done, but your work isn't finished. This script has no error traps, no try statements. The 4 variable assignments at the beginning need to be parametrised in some way. (Exercise for the reader)

How to use an FTP Host as an Alternative EDI Network

Web site hosting services are very competative now. Easy to use and ever cheaper for both individuals and companies. These servers and systems can also easily provide FTP connections. Dozens of user logins and Giga bytes of storage can be had for an annual cost of less than the purchase price of a desktop computer. With the aid of some relatively simple scripts your computer system can reach out across the internet and use these as a stripped down, bargin basement, alternative EDI (Electronic Data Interchage) network. Here's how...

Create a folder for each partner. Within each folder create 2 sub-folders, inbox & outbox. See below

tom

tom\inbox

tom\outbox

dick

dick\inbox

dick\outbox

harry

harry\inbox

harry\outbox


You could call these sub folders something else, like upload & download, or simply in & out. However using the inbox/outbox terms helps to relate them to their function and models the familiar email arrangement.

Assign each partner a user id & password and set their "home" folder. When they log on they should only be able to access their own folder.

To send you a file, the partner will log on and "put" (in FTP terms) the file in the postbox folder. To receive any files from you they will list ("ls") the mailbox and "get" anything they find then "del".

To send a file to a partner, log on and "put" the file in the partner\mailbox folder for them to collect. remember as host you start from the root folder. To receive any files from your partners, list all outbox folders and "get" anything you find then "del".

This is a simple arrangement, but there are some suttle potential gotchas.

• All files sent and received should be have unique names. If you send a daily file (for example orders.csv) and the receiver doesn't pick it up before the next one is sent, it will get overwritten. To avoid this the sender keeps a counter updated on the generating system and the number is incorparated into the file name e.g. orders123.csv

• If both sender and receiver are logged on at the same time then it is possible to list a file that is only partly transmitted. To avoid receiving part files, the sender should "put" using a tempory name (typically with a .tmp extension), then "rename" it to its correct name. The receiver just egnores any listed .tmp file.

• The FTP protocol has 2 modes. ASCII & Binary. Most EDI files are text files. On Windows/Mac/Linux/Unix the end of a line (EOL) in a text file is marked by different characters. New lines, form feed etc. If sender and receiver have different EOL styles then this can cause problems. This is solved if the FTP is done in ASCII mode (which is usually the default). However for things like excel and jpeg's is would cause data curruption so binary mode must be used.

It doesn't really matter which party is the host. If you are at the client end of the relationship and have to deal with several different hosted sites, the overhead for each additional FTP connection and session is small. Address, username & password. No complicated configuration.

If Tom, Dick, Harry need to exchange files with each other, as well as yourself, then this can also be achieved. In traditional EDI the "to" & "from" envelope data is found by reading the beginning of the message. This method could be used but it would require each file to be moved to your system so it could be read and then imediately retured to the host server in the correct folder. A better way that does not restrict the files to formal EDI formats, is to require the file names to be in the form from-to-count.extn e.g. tom-dick-99.txt or harry-tom-321.xls

Embedded EDI

I have just come across the phrase Embedded EDI. I think it fits quite nicely as a label for what I have been trying to articulate.

Some highlights...

• "Are a CEO’s email or Text Messages more or less important than an X12 Shipping notice?"

• "Why is EDI not subject to universal addressing via DNS or URI name-spaces?"

• "CEO’s and children expect email to reach it’s destination, but Supply chain managers need private networks. Why?"

Good questions...

I am Eddy

Eddy (fluid dynamics): The swirling of a fluid and the reverse current created when the fluid flows past an obstacle.

I am not Eddy

I am not angry with every idiot.

When Lost, it helps to remember where you want to go

I have the occasional high school student pass through my office on work experience. They do the rounds going to each department. When they come to me they have already spent time in the Sales and Purchasing departments. I am expected to give them a 30 minute overview of EDI (I don't just do EDI, but no one else understands it, or wants to understand it, so that is what they ask me to do). Then they move on to the help desk where they occupy them with tasks, like prepping desktops. They always like that, but I can't claim any are enthusiastically interested in what I have to say. I find the exercise helps to keep my feet firmly on the ground and head out of the stars, as they sit there wondering (despite my best efforts) what I'm talking about.

Well, the other day it was the turn of the one of my IT colleagues offspring. Keen to make a good impression, I was spurred on to think up a new and better explanation than I had used before. I was pleased with the results so I thought I would reproduce a version of it here.

***

When you enter Sales Order Processing department, you will see people opening the mail and extracting customer purchase order forms. Some of them are hand written, some are computer printed.

There is also have a Fax machine dedicated to customer orders that spews out a steady stream of order forms. Again, some of them are hand written, some are copies of computer generated order forms.

We also have a team of people who take orders verbally over the phone.

Finally, some customer orders come in as emails. Usually as PDF or Microsoft Word attachments. All these orders are read and the details typed into our Computer system.

This raises 2 questions, and leads to a 3rd great big "What If" question.

Q1. Why aren't all orders emailed?

It is cheaper not to have labor deployed in the mail room, or manning the phones, or maintaining the fax line and fax machine. The cost of receiving emails is tiny in comparison. When you scale up to hundreds, thousands, tens of thousands a day - for emails, the cost graph raises far less steeply. The same is true for our customers. Most of them are companies and organisations. It is cheaper for them not to pay for postage, or phone calls.

So I repeat the question. Why aren't all orders emailed? The answer? I don't know, you should ask the customers. But the point is, whatever the real answer (and it might be different for different customers) take a note of how difficult it is to change our customers behaviour. We can't afford to turn away business by not accepting the orders and we don't want, or to make it hard to order from us.

Q2. Why do all orders have to be re-typed into our Computer system?

Every product is made up of many components and materials. All of which have to be purchased. Every product requires many different sorts of resources, people and machines. All have to be planned and instructed what to do. Controlling all these aspects takes a complicated Computer program (called an ERP system) and that is why they have to be entered.

Q3. What if, instead of having to re-type everything, you could click and drag the email attachment, drop it on the ERP icon on your desktop, and the software processed it for you?

I will tell you "what if". If you could do that you will have achieved what businesses and organisations have been trying to achieve for decades. This is the ultimate aim of EDI.

Why is this so difficult? To a computer, text, is text, is text. How does a computer tell one piece of text is a product description and another is an address, one is a quantity and another is a price, one a delivery date another is a created date? To humans this is easy. For computers this is hard.

Ever tried to open a Microsoft Word document with Notepad? That is what the file looks like to a programmer. What do all the non-text characters mean? Only those who have signed a non-disclosure contract with Microsoft know, and none of them have permission to tell you.

First thing you have to do is agree a document format that is open to everyone. Then everyone has to agree where to place each bit of data or how to label it. In short, globally speaking, we can't agree. Many have been expecting one of the many formats to emerge as the dominant one, like the way every one uses MP3 instead of WMA or WAV, but this hasn't happened.

The next thing is product identity. We distribute a catalogue describing all our products and product options along with product codes (you see these bar-coded on the packaging). We encourage our customers to quote these in the order so that there is no miss-understanding. How do our customers (and their computer systems) know this information? If they order the same product from different suppliers, each will have their own code.

At least sending the document is easy. I mean everyone uses email, right? How else would you send electronic data from A to B? Er.. hmm... well... No. We have been moving electronic data around since before email and the Internet. There are dedicated EDI networks. There are Extranet web sites. Third party hosted web app clearing houses. There are different ways of securing emails. There are different encryption solutions. Security Certificates. Web of trust. etc. etc. etc.

So what are we to do? We could pick one solution and tell all our customers to communicate in that way. But remember these are the same customers, some of whom are still using post and fax. Remember how difficult it is to change some customers behaviour? Some of the big customers might prefer a different solution. We won't turn them away, but very quickly we find ourselves supporting many different forms of EDI. The cost of implementing another one has to be weighed against the volume of business (and cost saving) we can expect in the future.

Now turn your point of view around. When we order from our suppliers, we become the customer. The complexity just doubled. I have only talked about orders, but there are many other documents that are exchanged in business.

***

Are you Lost? Can you remember where we were heading? Any bright ideas?

What if EAN/UCC numbers didn't exist?

I was in the process of setting up a mapping for a new EDI (Electronic Data Interchange) customer. One of the first things I had to do was to take the senders ID from the message envelope and create a look-up link to the customers account on our ERP system. Now just lately I have become obsessed by leaner, simpler EDI. So I looked at this "simple" process with new eyes.

The senders ID, and by extension any party ID, has to be agreed to in advance. This is because it is no good if Acme Inc uses an ID of ACM001 if it is also used by The Association of Comics & Mimes. The receiver wont know which party it is. Now a human will be able to deduce who it is from if the message includes a full address and this is displayed for manual processing. But EDI is not about manual processing.

One solution is for the sender to use the receivers’ allocated ID for the sender (themselves). But they won't know this unless they communicate first.

The most common solution is to use a neutral and authoritative third party’s' allocated ID. The most commonly used are EAN/UCC numbers. However this is not very ubiquitous. Ask any big company CEO or any small one person trader, what their company’s EAN/UCC number is, you will get a blank look. Ask them what their tax number is and they would be able to give you an answer quite quickly. Tax numbers are not a universal solution as there are occasions when they would be not applicable.

But as a receiver of unsolicited messages, a new, never before seen EAN/UCC number is not much use. They are not routinely stored on our ERP customer/supplier database. I can't do a Google search to tell me who is 555987654321 or what is Acme's EAN/UCC. I am stuck with asking and potential EDI partner what their EAN/UCC number is. We are back to prior communication again.

The EDIFACT format standard, NAD segment, lists 300+ organisations that can be used to supply ID numbers. Tax offices, Banking organisations and Standard bodies dominate (EAN/UCC is number 9). None of those listed stood out to me as a better solution so I guess EAN/UCC dominates as it is the least worst.

At this point I realised something. These messages from this sender are not being delivered by a traditional EDI method and as a result our system already knows who they are from. You see they are already linked with an Internet domain name. Now this strikes me as a very good pre-existing "identity" code. It is unique. It is well known. It is in common use. For me it passes the "Fit for purpose" test and the "Ubiquitous" test. I searched the EDIFACT code list but apart from the catch all ZZZ (= Mutually Defined), none seemed applicable.

But then where in all the EDIFACT standard does it refer to a URL, a URI, or an email address? I guess the relevant parts (as they are pretty fundamental) probably date from the early history of the standard, circa 1989. I think I sent my first email on JANET around about then. (That is much earlier than anyone else I know. Whoops! I guess that ages me.)

It is twenty years later now and I think EDI has some catching up to do.

Million Dollar Traders, Nickel & Dime EDI

They don't seem to do "Documentaries" any more. It is all "Reality TV" these days, and I have mostly had enough of them. I wish shows like "The Apprentice" would show more details of the problem solving, less of the problem personalities. But such is the rarity of Business related material I gave this new show a go.

Million Dollar Traders - the premise: take some ordinary people who show some promising aptitude, train them intensively for a short period, then let them loose with real money and see how then perform as a Hedge Fund.

Unfortunately I didn't learn much about the workings of The City, Finance and Hedge Funds. I learnt more than I wanted to about ordinary people under stress.

What caught my eye was the technology on display. Each trader had 2 or 3 computer monitors showing large amounts of numbers and graphs. Real time data was so important it was considered a major faux pas not to be at the desk the moment the markets opened. They were constantly crunching numbers with calculators as they balanced (or hedged) risk trying to find a profit margin. All this was happening as the Credit Crunch Tsunami hit, and banks crumbled into chaos. After 3 weeks I think the best trader made about 1% gain, compared with an industry average of 5% loss. My rough guess is that this represents £2,000.

Throughout this time we watched as the trainee traders weighed their investment decisions, the tension and drama was built up. We sat on the edge of our seats as they finally committed themselves to make a deal....

...and then they picked up the phone...

Uh?

No "Accept" button to click. No https protocol session. But a conversation with a real person. A request to buy or sell (go long or short in the jargon). A query on the price and a confirmation on the total cost. They then got out of their chair with a piece of paper (presumably some sort of record of the deal), walked across the room and got it stamped in some sort of machine that made a satisfying "kerching" sound.

At first I put this down to artistic license and the program makers wanting to make more interesting imagery. Like in the film Independence Day where, they bring down an alien war machine with a software virus. As unlikely as it is that the aliens have no anti-virus software, they leave out the incredibly quick development of the USB port to NBSA port adapter hardware (Never Before Seen Alien port).

Then in the last episode a strange thing happened. The boss was upset. He was a REAL trader. He knew how it was. The new guys were not coming up to scratch. He didn't like their attitude. It was all wrong. So they started discussing acceptable telephone manners when on the phone to a broker. At issue was the fact that some of the traders were spending too much time saying "hello", "please" and "thank you". He insisted that the Brokers will be laughing at them the minute the phone was put down. If the traders were not short and blunt they would get no respect. The brokers would be less likely to do their job as efficiently.

My jaw dropped. It was for real. The boss was prepared to openly admit a willingness to trample all in his way in the focused pursuit of money. I understood that. It was in the nature of the job. Yet he was prepared to rely on a fallible human, being able to interpret verbal instructions communicated by another fallible human. Indeed, in the first episode one guy accidentally got his "buy" and "sell" mixed up. In the 20 seconds it took to correct his mistake, he lost money.

The labour cost of those transactions was just silly. Yet the volume of trading, city wide, is huge. I know the term "EDI" is not a trendy buzzword, but where was the - Electronic - Data - Interchange?

I know the world of Finance has taken a heavy PR knock recently. I have heard the view expressed elsewhere that those in charge didn't really understand what was going on. But so long as the money kept rolling, questionable practices remained under the radar. If this TV program is representative then to me it would seem to confirm this view.

Here is an important life lesson I learnt as a small child watching The Wizard of Oz.

Despite what the Wizard says, don't ignore the man behind the curtain. His presence is telling you something!