Aggregated Fail States

The concept of a program failure, in the mind of a software developer, usually takes the form of a boolean or an exception. It’s either true or false, fail or no-fail.

But sometimes you come across more subtle error states – and this is especially true with web-based applications or applications that consume external services. Temporary errors, due to connectivity issues, pop up from time to time, and handling those correctly means the program has to have some sense of whether the error is temporary, localized, important-enough-to-abort, and handle multiple errors at the same time.

Collecting data from Facebook API’s, for instance, can result in most calls succeeding, but one or two failing. Maybe Facebook deprecated a particular metric. Maybe there’s packet loss on the way. Maybe there’s a slight temporary error… and sometimes there are just hard failures (like authorization failures where access tokens have expired).

I am moving to something I call a ResultState. It’s a simple class that looks something like this:

    public class ResultState
    {
        public bool Success { get; set; }
        public bool Finished { get; set; }
        public int TryCount { get; set; }
        public List<LogEntry> Messages { get; } = new List<LogEntry>();

        public bool Retry(int maxAttempts)
        {
            Finished = TryCount++ >= maxAttempts;
            Success = false;
            return !Finished;
        }

        public void Set(bool successful, LogEntry message = null)
        {
            Finished = true;
            Success = successful;
            if (LogEntry != null)
                Messages.Add(LogEntry);
        }
    }

It’s a simple class I’m using in a Batch Call function – every request inside a batch request is capable of being successful, not successful, or unfinished (retrying). The Batch Processor will keep submitting the request to the external API until all calls are complete – whether they ended in a failure or not.

    // Run batch request
    do {
        ...
    } while (!batchItems.All(x => x.Finished));

    // Dump messages to application log
    Logger.Write(batchItems.SelectMany(x => x.Messages).ToList());

The important thing is to constantly be able to feed upper-level methods with a “state” – the LogEntry object carries a descriptive message along with a severity code that allows the main process to evaluate the severity of the situation and take appropriate action. Indeed, from the Batch request it receives a List that describes all the various states and errors that were returned. It runs a Distinct() on these to feed the end result to the user.

For headless operations (where a process runs activities in the background), there are several consumers of ResultStates that might be interesting:

  • Upper-level methods and diagnostic software that must act on the information.
  • System operators, responsible for monitoring and debugging the software.
  • End users that are interested in the result of the operation, and need to know when or why things are failing.

Nothing is worse than running an automated job and receiving no response whatsoever whether it failed or ran successfully. And nothing is more frustrating than to receive a simple “failed” status back without any indication as to why. Aggregated fail states that can bubble up messages from individual methods are necessary in communicating just what happens, and why. Ideally, a system operator should be able to investigate the failure by just looking at easily accessible web-based logs.

Windows CMD reference

I spent a part of yesterday looking through all the commands in Cmd.Exe (the Windows command shell, inheritor of the old COMMAND.COM). It’s interesting how they’ve added switches and stuff to improve on it since the early days, but how almost nobody seems to use it. Admittingly, it’s a far cry from bash, the Unix command shell, but there’s a few old tricks you still can pull out of the hat. People may flame it and despise it, but I always thought you should be able to do more with the good ol’ shell.

Here’s a few of the things I found … in alphabetical order.

ATTRIB [/s] [/d]

Changes file attributes. /s makes it recursive, /d makes it operate on directories as well. I didn’t know about these switches before. Handy.

CALL :label arguments

Neither did I know you could call a label in a batch file. This should make it easier to write “gosub”-like routines. And the parameters can be expanded with new interesting features, see below. To exit from the subroutine, use the “goto :eof” statement.

CHKNTFS

Schedule a check-disk on next boot. Might come in handy sometime.

EXIT /b [errorlevel]

Exits the command shell. If you use the /b switch, exits the current batch file. You can also pass an errorlevel along.

F7

Pressing F7 brings up the history list. How come I never knew that?

FINDSTR [/r] [/c:]”search string” filespec

Find strings in files. I might still use Turbo Grep, but this is cool too. Normally it searches using an OR pattern on the search string (meaning “I love you” finds all instances of “I”, “love” or “you”), use the /c: switch to make it an AND search.

The /r parameter turns the search string into a regexp. Note, some of the fancier stuff might not work as usual, consult the FINDSTR /? or the online help for further information.

FOR %%v IN (set) DO …

The FOR command is one of the coolest features in batch programming. I had no idea you could do so much with it.

FOR %%f IN (dpr pas dfm res) DO COPY *.%%f \deploy

Copy all Delphi source files for a project to a specific directory.

FOR /d %%d IN (set) DO ...

Match directories in wildcards instead of files.

FOR /r [path] %%v IN (set) DO ...

Recursive operation on files found, optionally operating relative to “path” instead of the current directory. It might be used like “FOR /r c:\deploy %%f IN (*.*) DO ATTRIB -r %%f”, which will recursively remove the read-only attribute from all files in c:\deploy.

FOR /l %%v IN (start, step, end) DO ...

For-loop. “FOR /l %%v IN (1, 1, 5)” gives the sequence 1 2 3 4 5.

FOR /f ["options"] %%v IN (file-set | "string" | 'command') DO ...

The /f parameter is probably the most interesting feature I’ve found. It reads lines from an input file, string, or result from a shell command, tokenizes them and processes a command for each line. Normally, the token delimiters are space and tab, and it usually operates on the first token found, so without extra options you will always get the first word in each line. But it can be modified with the following options:

eol=c             Set the end-of-line character. One character only.
skip=n            Skip n lines in the beginning.
delims=xxx        Delimiter set, default is space and tab.
tokens=x,y,n-m    Which tokens to feed into the command. Variables start at the variable 
                  given, and allocates further as needed in alphabetical order. * means
                  "the rest of the line".
usebackq          Use backticks instead of apostrophe for the command evaluation. The 
                  format changes to ("file-set" | 'string' | `command`). Required if you 
                  use filenames with spaces.
FOR /f "eol=; tokens=2,3* delims=, " %%i IN (myfile.txt) DO @ECHO %%i %%j %%k

Parse each line in myfile.txt, ignoring lines that begin with semicolon, pass 2nd and 3rd tokens into the command, separating each token my either comma or space. Notice how the sequence goes: %i, %j, %k.

FOR /f "usebackq delims==" %%i IN (`set`) DO @ECHO %%i

Enumerate all variables found.

Expansion of variable parameters is also available, see below.

GOTO :eof

Jump to end of file. Handy way of exiting from a script.

IF comparisons

IF [NOT] ERRORLEVEL n ..
IF [NOT] EXIST filename ...
IF [NOT] string1==string2 ...
IF [/i] string1 EQU|NEQ|LSS|LEQ|GTR|GEQ string2 ...

The normal IF command is enhanced, too. It can check errorlevels as before, file existance, and compare strings. But is also has new operators. For instance, the “IF ERRORLEVEL 3” statement can now be written “IF %ERRORLEVEL% LEQ 3”.

/i means case-sensitive (or case-insensitive, I forgot which). Numeric strings evaluate as numbers, not strings.

IF now also supports multi-line statements and ELSE statements, see below.

MD \a\b\c\d

Will create new directories in sequence.

MORE /e

Extended more. These keys are available:

P n     - next n lines
S n     - skip n lines
F       - next file
Q       - quit
=       - show line number
?       - show help line
space   - next page
return  - next line

MORE +n

Start from line n.

PUSHD \\server\path

Create a temporary drive allocation, starting from Z:, for the particular UNC path. This will be cleared with POPD.

RD /s /q

Very dangerous command.

SET

SET [var[=[value]]]
SET /a [var=]expression
SET /p var=[prompt]

SET only will display all variables. SET P will display all values starting with P. SET P= will clear variable P.

SET /a will perform a calculation, for instance SET /a X=2*2 + 5.

SET /p will prompt for user input and store the result in a variable.

SETLOCAL / ENDLOCAL

Make local changes to the environment. Work all you want with it, then call endlocal to revert back to where you were. Also handy, especially with some of the advanced SET features.

SHIFT [/n]

Shift parameters. Optionally start at the nth position, preserving all elements %0 .. %(n-1).

SORT [/+n] [/o outfile]

Sort may start sorting at the nth position now. Could be good for unwanted stuff in the beginning (timestamp in logs, perhaps). /o is faster than piping.

Interesting ways of treating variable expansion

Some new ways of treating variables are available. Like, string substitution and substring matching.

%PATH:str1=str2%      Substitute all occurrences of str1 with str2.
%PATH:~10,5%          Substring, start at position 10 and extract 5 characters.
%PATH:~-10%           Only get the last 10 characters.
%PATH:0,-2%           Extract all but the last 2 characters.

%CD%                  The current path
%DATE%                Current date
%TIME%                Current time
%RANDOM%              A random number between 0..32767.
%ERRORLEVEL%          The current errorlevel.

%*                    All arguments
%0 .. %9              Arguments
%~1                   Remove quotes from parameter 1
%~f1                  Expand to fully qualified filename
%~d1                  Expand to drive letter only
%~p1                  Expand to path only
%~n1                  Expand to file name only
%~x1                  Expand to extension only
%~s1                  Expand to short file name only
%~a1                  Expand to file attributes
%~t1                  Expand to file date/time
%~z1                  Expand to file size
%~$PATH:1             Search through all directories specified in %PATH%. If the file is found,
                      return the fully qualified filename in that directory. If the file isn't 
                      found, return blank.
%~dp1                 Expand to drive and path. (Further elements may be combined: 
                      %~ftza1 gives a DIR-like output)

Interesting ways of doing IF and FOR statements

There’s a syntax I’ve never seen either with IF and FOR statements. You can use IF-ELSE-syntax in this way:

IF EXIST hello.txt (
    DEL hello.txt
) ELSE (
    ECHO hello.txt is missing!
)

…or even…

IF EXIST hello.txt (DEL hello.txt) ELSE (ECHO hello.txt is missing!)

And how about this?

FOR /l %%v IN (1 1 5) DO (
    ECHO This is line number %%v.
)

The crucial thing seems to be, in ELSE statements, that ELSE has to be written “on the same line” as the IF statement. This is why ELSE is written on the same line as the parantheses.

So there you are. A whole new way of writing batch files. No extra software needed, just plain old Windows XP.

Michaelangelo and Queen Arabella

Listen, I now will tell you a story fair,
Of queen Arabella in a distant land
Her wisdom as wonderful as the golden hair
That playfully flowed over shoulders bare –
Like sunrays glittered every strand.
Her knowledge had no discernful bound;
Every problem graceful answer found
And loving eyes of coolest blue
Like radiant sapphire in their hue
Shone from her head with emerald crowned.
And a multitude of roses gently sprung
From softest soil onto her castle walls
Which made of whitest stone so nobly swung
In rosy arches over ancient halls;
And through those halls her silvery laughter
Rippled like streams in early spring
And gave joyous voice to every living thing
That dancingly followed her thereafter.
Throughout all the kingdom spread her fame
And from far beyond the people came
To see the glory behind Arabella’s name.

…to be continued…

MySQL Function to Calculate Excel-style Dates

drop function if exists exceldate;

delimiter //

-- Function that returns an Excel-style or Delphi-style date value
-- from a MySQL date. A date value of 0 represents 1899-12-30.&lt;/em&gt;
create function exceldate(p_date date)
    returns int
    sql security invoker
begin
    return to_days(p_date) - 693959;
end //

delimiter ;

select '1899-12-31', exceldate('1899-12-31') -- should be 1
union
select '2011-10-18', exceldate('2011-10-18'); -- should be 40834

Luhn (mod 10) Check Digit Algorithm in MySQL

drop function if exists luhn;
drop function if exists luhn_check;

delimiter //

-- Function that calculates a Luhn (mod 10) check digit from a numeric string.
-- The behavior is undefined if the string contains anything else than digits.
-- Assumes that the string does not have a check digit added yet, so it starts
-- with a weight of 2 at the last digit.
create function luhn(p_number varchar(31))
    returns char(1)
    sql security invoker
begin
    declare i, mysum, r, weight int;

    set weight = 2;
    set mysum = 0;
    set i = length(p_number);

    while i > 0 do
        set r = substring(p_number, i, 1) * weight;
        set mysum = mysum + if(r > 9, r - 9, r);
        set i = i - 1;
        set weight = 3 - weight;
    end while;

    return (10 - mysum % 10) % 10;
end //

-- Check if a numeric string has a valid check digit. Does this by cutting off
-- the last digit, recalculating the Luhn check digit, and comparing the strings.
create function luhn_check(p_number varchar(32))
    returns boolean
    sql security invoker
begin
    declare luhn_number varchar(32);
    
    set luhn_number = substring(p_number, 1, length(p_number) - 1);
    set luhn_number = concat(luhn_number, luhn(luhn_number));
    
    return luhn_number = p_number;
end //

delimiter ;

Common ffmpeg parameters

Information

-codecsDisplay codecs
-formatsDisplay formats
-f fmtForce format “fmt”
-i filenameSet input file name
-yOverwrite output file
-t secsForce duration to specific length (hh:mm:ss[.xxx])
-fs limitSet file size limit
-ss secsSeek to given time position (hh:mm:ss[.xxx])
-target typeSpecify target type (“vcd”, “svcd”, “dvd”, “dv”, “dv50”, “pal-vcd” etc); all format options are set automatically
 

Video

-b bitrateVideo bitrate in bps
-r fpsSet frame rate (default 25)
-s WxHSet frame size (default same as source)
-vf scale=W:HRescale video (use <=1 to scale, e.g. “640:-1” means resize height to scale, -2 means same but maintain even /2 count)
-vf transpose=nRotate video (0 = 90° counterclockwise and vertical flip; 1 = 90° clockwise; 2 = 90° counterclockwise; 3 = 90° clockwise and vertical flip)
-aspect aspectSet aspect ratio (4:3, 16:9 etc)
-vnDisable video
-pass nMultipass rendering (1 or 2)
-vcodecForce video codec (e.g. “h264”, “copy”)
-crf nnConstant Rate Factor (0=lossless, 23=default, 17-28=acceptable)
 

Audio

-ar freqAudio frequency (default 44100 Hz)
-ab bitrateAudio bitrate in bps (default 64k)
-aq qualityAudio quality (codec-specific, VBR)
-ac channelsAudio channels (ac=2 to downmix 5.1 to stereo)
-anDisable audio
-acodec Force audio codec (“aac”, “mp3”, “copy”)
 

Subtitles

-scodecForce subtitle codec (“copy” to copy stream)
-snDisable subtitles

My Idea for a Grand Opera

STAGE: A rural village in feudal Japan. A small, traditional Japanese house is nearby. Some cherry trees can be seen in the distance.

ACT ONE:

A samurai warrior enters the stage, inspecting his lands and his village. He starts singing the first aria of the opera, “A Sacred Land, A Sacred Call”, extolling the virtues of the samurai and the honor that lies with his profession. “To die for the emperor”, he sings, “a duty; an honor – oh that I would be found worthy of doing so”. Two women nearby sit kneeling with their heads bowed in deep respect for this great warrior. After he is gone, they discuss the theme of the aria between them, and how they are, like him, honor-bound in their call to serve. “This is the Meaning of Life”, they sing together in wonderful, tear-jerking duet.

But events are afoot. The samurai’s son suddenly enters the stage, looking for his father after many years away in Kyoto. They meet; father is delighted to see him and wonders how he has been doing. Alas, it is soon revealed, that the son has not followed in his father’s footsteps; he has become a traveling salesman for Hershey’s Chocolate Kisses. The samurai, enraged, commands him to stop immediately and storms out. The end of Act One ends with the son, singing to a sad tune on the clarinet, “How I Love Japan; But I Love Hershey’s Kisses Too”.

ACT TWO:

The Japanese villagers are now talking among themselves and rumor quickly spreads that the samurai’s son is a salesman for Hershey’s. Some of the villagers argue that chocolate, in every form, is a good thing and Japan must embrace the influences of the new world; others argue against; when suddenly the samurai himself appears. Finding that his authority is weakening, he quickly summons his son and asks him “very well, have you changed your mind?” The son refuses to leave his new profession, and the samurai, dishonored and enraged, throws him into a bamboo prison cell. “There you will stay”, he bellows, “until you respect your honor!”

The drama develops when a team of Hershey Co. lawyers emerge on the scene, singing a transformation of the main theme, “What Ho, What Ho? What Transpires Here?” The samurai threatens to kill the lawyers on the spot, but they quickly produce a document signed by the Emperor himself, which no samurai can question, that Hershey’s Chocolates are legitimate all over Japan by royal decree. The second act ends with the samurai father falling on his knees in shame and dishonor, crying. The villagers look on terrified.

ACT THREE:

The samurai, unable to bear the shame of his son as a traveling Hershey’s salesman, has reached a decision: He will commit suicide. The villagers are mortified, and the team of Hershey’s lawyers are beaten by them until they repel them by threatening to sue them for libel. The samuari ends this quarrel by stepping onto the stage with his swords; a gray, somber figure with ashen face, prepared to do his duty. His son sings his final aria to him from the bamboo prison, “Will You Not See: a New Dawn for Old Japan”, but the father refuses to listen.

But by mistake, when the samurai reaches for his last sake glass, he accidentally grabs a Hershey’s Chocolate Kiss instead and puts it in his mouth! Apalled at first, his countenance soon changes as he realizes he has made a dreadful mistake, and everyone soon starts laughing. He lets out his son, forgiving him with tears in his face, and the opera ends with the grand finale, the duet between father and son, singing “Here is Tradition Too”, indicating that there are traditions in Hershey’s Company as well, as it is in feudal Japan, and that both can coexist together through honor and mutual respect. The villagers and lawyers combine in a final, grand chorus. The sun sets over the cherry trees which are now blossoming in full, and the curtain falls. The End.

I Had a Strange and Curious Dream

I had a strange and curious dream last night
I dreamt my heart grew wings and then took flight
Flew far and wide o’er hills and fields of green
Above those golden clouds, to things unseen –
Plucking roses sweet that never grew
And singing childish rhymes all while I flew.
I touched the sky, I think, with outstretched hand
Drew fine-art paintings in the ocean sand
And swiftly sailed across the deep blue sea
To search for wonders named in poetry.

In this dream I had, I know it’s strange
I dreamt I leaped across a mountain range!
The pinnacles where draped in snowy white
And gleaming in the shining sun so bright.
But linger long up there was not for me,
What worlds that lay beyond I had to see.
Quickly went I down the mountain side
T’was there I saw the meadows open wide.

I stood in flowery grass so tall and green
Such wonderful flowers; I had never seen!
The field rolled gently down towards a stream
Beyond which a city seemed to softly gleam.
It was a city made of purest gold
Of its glory every psalm and hymn had told.
At length I stood there, taking in the sight –
Until a voice spoke softly to my right.

The joy I felt I cannot ever tell
Suffice to say, on my knees I quickly fell.
And on my cheeks, round tears then softly broke
As He, my King of Kings, so sweetly spoke.
“This is the home”, He said, “for you I made;”
“Do not fear, the price has all been paid.”
And from His love I then began to weep –
T’was then, that I awoke out of my sleep.

I had a strange and curious dream last night
I dreamt my heart grew wings and then took flight.
The day will come again when I will fly
Beyond those golden clouds in sunset sky.
And nevermore will I this planet roam
For somewhere over there is home, sweet home.

The Old Mainframe Computer

In a dark and gloomy dungeon
Racked by storms and thunder
A valiant server stood on guard
As lightning ripped the night asunder.
It stood there lonesome and forlorn
Working quietly in the freezing cold
But not a tear was seen
On its faithful color screen
This server had a heart of gold.

In its core a dignified processor worked
Although old, still bravely faithful
And though its software wasn’t new
Its owners still were very grateful.
It flawlessly performed, both day and night
It ticked so quietly, so gracefully and bright
A thousand users everywhere
Put the server’s idle time on hold
But no one knew the server’s heart
Yes, this server had a heart of gold.

Deep down in this server’s brittle chips
A stable, very stable kernel ran.
And through thick cables proceeding forth
It quietly communicated on the WAN.
Hundreds of transactions were processed
On the company’s intra-network site.
But the only evidence of work there was
Was a little light that gently flickered in the night.

And although many years had passed
And newer systems came and went
And elaborate technologies passed by
ODBC, DCOM, Windows and Lucent
Still they just failed and failed and failed
And came nowhere near this server’s unique mold
Of faithful trust and tender hope
Buried in this server’s gentle heart of gold.
Yes, this server had been working quite a while
And if it could be said it had a mind
Anyone who looked inside would find
That splendid willingness to “walk the extra mile”.

Then, one day, a strategic decision was made.
“Windows is the future. Old IBMs are not.”
So this server was sadly carried away
And replaced with something that was “hot”.
And the damp computer place was changed
Into nice, air-conditioned halls
With carpets, supervising systems
And thin fiber cables lined the walls.
And in the old computer’s place now there sat
An NT 5.0, purring like a cat.

But soon enough, something happened
Users called and gleefully complained
Connections lost, memory faults all over
And a general confidence that waned.
Computers crashed, and deals weren’t met
And executives and computer guys began to fret.
Previously trusting users collapsed and cried
As their screens shone brightly blue
With exception faults and many, many GPFs…
It was a horrid nightmare now come true.

In the meantime, this old server sat
Now abandoned, so silent and forlorn.
Its kernel didn’t work. No lights were on.
It couldn’t even mourn.
The letters IBM, once bright and shining
And the cover, once so neat, with silver lining
From all its work so badly torn.
It now simply occupied the tiny space
Between a dead screen and an old computer case.

But then, as its tiny little chips gave up all hope
It finally was remembered.
An angry group of software engineers
Brought the NT system, now dismembered.
They threw the stupid thing against the wall
And took the old computer through the hall.
And accompanied by the staff’s wild cheer
On the old screen there appeared a little tear.
And as it was plugged in again,
Its systems flared to life once more
The BIOS booted, and the kernel started up
The old processes spread through its trusty core.

Then everything was back to normal.
Everything ran well again.
And all throughout the place, executives agreed
That new technology is just in vain.
That trusted systems should not be replaced,
Once it worked, it continually kept administrators amazed.
And deep down, in that old dungeon, now so bright
That used to be forgotten and so cold
There worked a valiant server,
And that server had a heart of gold.