Aggregated Fail States

The concept of a program failure, in the mind of a software developer, usually takes the form of a boolean or an exception. It’s either true or false, fail or no-fail.

But sometimes you come across more subtle error states – and this is especially true with web-based applications or applications that consume external services. Temporary errors, due to connectivity issues, pop up from time to time, and handling those correctly means the program has to have some sense of whether the error is temporary, localized, important-enough-to-abort, and handle multiple errors at the same time.

Collecting data from Facebook API’s, for instance, can result in most calls succeeding, but one or two failing. Maybe Facebook deprecated a particular metric. Maybe there’s packet loss on the way. Maybe there’s a slight temporary error… and sometimes there are just hard failures (like authorization failures where access tokens have expired).

I am moving to something I call a ResultState. It’s a simple class that looks something like this:

    public class ResultState
    {
        public bool Success { get; set; }
        public bool Finished { get; set; }
        public int TryCount { get; set; }
        public List<LogEntry> Messages { get; } = new List<LogEntry>();

        public bool Retry(int maxAttempts)
        {
            Finished = TryCount++ >= maxAttempts;
            Success = false;
            return !Finished;
        }

        public void Set(bool successful, LogEntry message = null)
        {
            Finished = true;
            Success = successful;
            if (LogEntry != null)
                Messages.Add(LogEntry);
        }
    }

It’s a simple class I’m using in a Batch Call function – every request inside a batch request is capable of being successful, not successful, or unfinished (retrying). The Batch Processor will keep submitting the request to the external API until all calls are complete – whether they ended in a failure or not.

    // Run batch request
    do {
        ...
    } while (!batchItems.All(x => x.Finished));

    // Dump messages to application log
    Logger.Write(batchItems.SelectMany(x => x.Messages).ToList());

The important thing is to constantly be able to feed upper-level methods with a “state” – the LogEntry object carries a descriptive message along with a severity code that allows the main process to evaluate the severity of the situation and take appropriate action. Indeed, from the Batch request it receives a List that describes all the various states and errors that were returned. It runs a Distinct() on these to feed the end result to the user.

For headless operations (where a process runs activities in the background), there are several consumers of ResultStates that might be interesting:

  • Upper-level methods and diagnostic software that must act on the information.
  • System operators, responsible for monitoring and debugging the software.
  • End users that are interested in the result of the operation, and need to know when or why things are failing.

Nothing is worse than running an automated job and receiving no response whatsoever whether it failed or ran successfully. And nothing is more frustrating than to receive a simple “failed” status back without any indication as to why. Aggregated fail states that can bubble up messages from individual methods are necessary in communicating just what happens, and why. Ideally, a system operator should be able to investigate the failure by just looking at easily accessible web-based logs.

Windows CMD reference

I spent a part of yesterday looking through all the commands in Cmd.Exe (the Windows command shell, inheritor of the old COMMAND.COM). It’s interesting how they’ve added switches and stuff to improve on it since the early days, but how almost nobody seems to use it. Admittingly, it’s a far cry from bash, the Unix command shell, but there’s a few old tricks you still can pull out of the hat. People may flame it and despise it, but I always thought you should be able to do more with the good ol’ shell.

Here’s a few of the things I found … in alphabetical order.

ATTRIB [/s] [/d]

Changes file attributes. /s makes it recursive, /d makes it operate on directories as well. I didn’t know about these switches before. Handy.

CALL :label arguments

Neither did I know you could call a label in a batch file. This should make it easier to write “gosub”-like routines. And the parameters can be expanded with new interesting features, see below. To exit from the subroutine, use the “goto :eof” statement.

CHKNTFS

Schedule a check-disk on next boot. Might come in handy sometime.

EXIT /b [errorlevel]

Exits the command shell. If you use the /b switch, exits the current batch file. You can also pass an errorlevel along.

F7

Pressing F7 brings up the history list. How come I never knew that?

FINDSTR [/r] [/c:]”search string” filespec

Find strings in files. I might still use Turbo Grep, but this is cool too. Normally it searches using an OR pattern on the search string (meaning “I love you” finds all instances of “I”, “love” or “you”), use the /c: switch to make it an AND search.

The /r parameter turns the search string into a regexp. Note, some of the fancier stuff might not work as usual, consult the FINDSTR /? or the online help for further information.

FOR %%v IN (set) DO …

The FOR command is one of the coolest features in batch programming. I had no idea you could do so much with it.

FOR %%f IN (dpr pas dfm res) DO COPY *.%%f \deploy

Copy all Delphi source files for a project to a specific directory.

FOR /d %%d IN (set) DO ...

Match directories in wildcards instead of files.

FOR /r [path] %%v IN (set) DO ...

Recursive operation on files found, optionally operating relative to “path” instead of the current directory. It might be used like “FOR /r c:\deploy %%f IN (*.*) DO ATTRIB -r %%f”, which will recursively remove the read-only attribute from all files in c:\deploy.

FOR /l %%v IN (start, step, end) DO ...

For-loop. “FOR /l %%v IN (1, 1, 5)” gives the sequence 1 2 3 4 5.

FOR /f ["options"] %%v IN (file-set | "string" | 'command') DO ...

The /f parameter is probably the most interesting feature I’ve found. It reads lines from an input file, string, or result from a shell command, tokenizes them and processes a command for each line. Normally, the token delimiters are space and tab, and it usually operates on the first token found, so without extra options you will always get the first word in each line. But it can be modified with the following options:

eol=c             Set the end-of-line character. One character only.
skip=n            Skip n lines in the beginning.
delims=xxx        Delimiter set, default is space and tab.
tokens=x,y,n-m    Which tokens to feed into the command. Variables start at the variable 
                  given, and allocates further as needed in alphabetical order. * means
                  "the rest of the line".
usebackq          Use backticks instead of apostrophe for the command evaluation. The 
                  format changes to ("file-set" | 'string' | `command`). Required if you 
                  use filenames with spaces.
FOR /f "eol=; tokens=2,3* delims=, " %%i IN (myfile.txt) DO @ECHO %%i %%j %%k

Parse each line in myfile.txt, ignoring lines that begin with semicolon, pass 2nd and 3rd tokens into the command, separating each token my either comma or space. Notice how the sequence goes: %i, %j, %k.

FOR /f "usebackq delims==" %%i IN (`set`) DO @ECHO %%i

Enumerate all variables found.

Expansion of variable parameters is also available, see below.

GOTO :eof

Jump to end of file. Handy way of exiting from a script.

IF comparisons

IF [NOT] ERRORLEVEL n ..
IF [NOT] EXIST filename ...
IF [NOT] string1==string2 ...
IF [/i] string1 EQU|NEQ|LSS|LEQ|GTR|GEQ string2 ...

The normal IF command is enhanced, too. It can check errorlevels as before, file existance, and compare strings. But is also has new operators. For instance, the “IF ERRORLEVEL 3” statement can now be written “IF %ERRORLEVEL% LEQ 3”.

/i means case-sensitive (or case-insensitive, I forgot which). Numeric strings evaluate as numbers, not strings.

IF now also supports multi-line statements and ELSE statements, see below.

MD \a\b\c\d

Will create new directories in sequence.

MORE /e

Extended more. These keys are available:

P n     - next n lines
S n     - skip n lines
F       - next file
Q       - quit
=       - show line number
?       - show help line
space   - next page
return  - next line

MORE +n

Start from line n.

PUSHD \\server\path

Create a temporary drive allocation, starting from Z:, for the particular UNC path. This will be cleared with POPD.

RD /s /q

Very dangerous command.

SET

SET [var[=[value]]]
SET /a [var=]expression
SET /p var=[prompt]

SET only will display all variables. SET P will display all values starting with P. SET P= will clear variable P.

SET /a will perform a calculation, for instance SET /a X=2*2 + 5.

SET /p will prompt for user input and store the result in a variable.

SETLOCAL / ENDLOCAL

Make local changes to the environment. Work all you want with it, then call endlocal to revert back to where you were. Also handy, especially with some of the advanced SET features.

SHIFT [/n]

Shift parameters. Optionally start at the nth position, preserving all elements %0 .. %(n-1).

SORT [/+n] [/o outfile]

Sort may start sorting at the nth position now. Could be good for unwanted stuff in the beginning (timestamp in logs, perhaps). /o is faster than piping.

Interesting ways of treating variable expansion

Some new ways of treating variables are available. Like, string substitution and substring matching.

%PATH:str1=str2%      Substitute all occurrences of str1 with str2.
%PATH:~10,5%          Substring, start at position 10 and extract 5 characters.
%PATH:~-10%           Only get the last 10 characters.
%PATH:0,-2%           Extract all but the last 2 characters.

%CD%                  The current path
%DATE%                Current date
%TIME%                Current time
%RANDOM%              A random number between 0..32767.
%ERRORLEVEL%          The current errorlevel.

%*                    All arguments
%0 .. %9              Arguments
%~1                   Remove quotes from parameter 1
%~f1                  Expand to fully qualified filename
%~d1                  Expand to drive letter only
%~p1                  Expand to path only
%~n1                  Expand to file name only
%~x1                  Expand to extension only
%~s1                  Expand to short file name only
%~a1                  Expand to file attributes
%~t1                  Expand to file date/time
%~z1                  Expand to file size
%~$PATH:1             Search through all directories specified in %PATH%. If the file is found,
                      return the fully qualified filename in that directory. If the file isn't 
                      found, return blank.
%~dp1                 Expand to drive and path. (Further elements may be combined: 
                      %~ftza1 gives a DIR-like output)

Interesting ways of doing IF and FOR statements

There’s a syntax I’ve never seen either with IF and FOR statements. You can use IF-ELSE-syntax in this way:

IF EXIST hello.txt (
    DEL hello.txt
) ELSE (
    ECHO hello.txt is missing!
)

…or even…

IF EXIST hello.txt (DEL hello.txt) ELSE (ECHO hello.txt is missing!)

And how about this?

FOR /l %%v IN (1 1 5) DO (
    ECHO This is line number %%v.
)

The crucial thing seems to be, in ELSE statements, that ELSE has to be written “on the same line” as the IF statement. This is why ELSE is written on the same line as the parantheses.

So there you are. A whole new way of writing batch files. No extra software needed, just plain old Windows XP.