unit testing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

unit testing

Graydon Hoare
Hi,

Rust presently has no "standard" mechanism for unit testing. I'd like to
adopt one, or at least move towards having one. We're getting enough
library written-in-rust code now that it makes sense.

I'm interested in pursuing a slightly-structured technique, and have
been considering a mechanism similar to the D 'version' and 'unittest'
system(s).

The idea has three parts:

1. Add a command-line flag to specify a set of configuration flags to
enable during a particular build. say -cfg win32 or such. These are not
structured values, just opaque strings.

2. Add a syntax extension also called cfg that marks an item as
to-be-included only if one of its arguments appears in the set of
configuration flags.

With 1 and 2, we'd be at the point where a module marked as, say,
"#cfg(test) mod test { ... }" would wind up conditionally compiled only
if you ran the compiler with -cfg test. (This would also double as a
mechanism for including modules on a per-platform basis, including debug
code, that sort of thing.) But programmers are lazy. So let's go one
step further:

3. Add a *further* command-line flag called '-test' that does two things:

   - Adds #cfg(test) to the set of config flags, so all test-conditional
     code gets included.

   - Changes the target the compiler's building. Instead of groveling
     for a main function, or just building a library (as in -shared),
     the -test target is a synthetic target that runs every function
     marked with #test, passing in a value of type std.test.harness,
     a basic xUnit-ish test harness.

     (should also provide runtime subset-selection of tests in the
      harness, etc, etc., but you get the idea)

If this works out nicely, we can expand the interface a bit in the
future to include specifying a test harness rather than hard-wiring it
to the one in std. For the time being I'd be happy to have any sort of
structured system, and I'd prefer to err on the side of "discoverable"
rather than "configurable" for testing.

(Also, this way we increase the odds of the standard-library one
maturing along with community needs, rather than rotting away. Nothing
sadder than a standard library component nobody uses!)

Comments? Preferences?

-Graydon

Reply | Threaded
Open this post in threaded view
|

unit testing

Brian Anderson-2
Hi Graydon,

This is in response to an old thread about unit testing.

I'm not familiar with D. What's the advantage of integrating unit testing so
tightly with the compiler? What does this provide over xunit-style
frameworks that just use reflection to find and run tests?

Googling doesn't turn up much. The only interesting thing I see is that
compiling in unit tests causes the test suite to be run prior to main(), and
that really doesn't seem very compelling since no user is going to have an
interest in doing that.

It seems like if you went this route, where you're defining the canonical
test framework for the language, that it should be done in a very neutral,
minimal and extensible way so that other test frameworks can piggyback on
it.

Regards,
Brian

On Thu, Oct 21, 2010 at 5:40 PM, Graydon Hoare <graydon at mozilla.com> wrote:

> Hi,
>
> Rust presently has no "standard" mechanism for unit testing. I'd like to
> adopt one, or at least move towards having one. We're getting enough library
> written-in-rust code now that it makes sense.
>
> I'm interested in pursuing a slightly-structured technique, and have been
> considering a mechanism similar to the D 'version' and 'unittest' system(s).
>
> The idea has three parts:
>
> 1. Add a command-line flag to specify a set of configuration flags to
> enable during a particular build. say -cfg win32 or such. These are not
> structured values, just opaque strings.
>
> 2. Add a syntax extension also called cfg that marks an item as
> to-be-included only if one of its arguments appears in the set of
> configuration flags.
>
> With 1 and 2, we'd be at the point where a module marked as, say,
> "#cfg(test) mod test { ... }" would wind up conditionally compiled only if
> you ran the compiler with -cfg test. (This would also double as a mechanism
> for including modules on a per-platform basis, including debug code, that
> sort of thing.) But programmers are lazy. So let's go one step further:
>
> 3. Add a *further* command-line flag called '-test' that does two things:
>
>  - Adds #cfg(test) to the set of config flags, so all test-conditional
>    code gets included.
>
>  - Changes the target the compiler's building. Instead of groveling
>    for a main function, or just building a library (as in -shared),
>    the -test target is a synthetic target that runs every function
>    marked with #test, passing in a value of type std.test.harness,
>    a basic xUnit-ish test harness.
>
>    (should also provide runtime subset-selection of tests in the
>     harness, etc, etc., but you get the idea)
>
> If this works out nicely, we can expand the interface a bit in the future
> to include specifying a test harness rather than hard-wiring it to the one
> in std. For the time being I'd be happy to have any sort of structured
> system, and I'd prefer to err on the side of "discoverable" rather than
> "configurable" for testing.
>
> (Also, this way we increase the odds of the standard-library one maturing
> along with community needs, rather than rotting away. Nothing sadder than a
> standard library component nobody uses!)
>
> Comments? Preferences?
>
> -Graydon
> _______________________________________________
> Rust-dev mailing list
> Rust-dev at mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20101226/f7121072/attachment.html>

Reply | Threaded
Open this post in threaded view
|

unit testing

Graydon Hoare
On 10-12-26 07:58 PM, Brian Anderson wrote:

> I'm not familiar with D. What's the advantage of integrating unit testing so
> tightly with the compiler? What does this provide over xunit-style
> frameworks that just use reflection to find and run tests?

What I said in the email: I'd like to err on the side of "discoverable"
over "configurable". Configuring a unit testing framework is nice; but
I've seen far too many projects fall down the slope of "didn't even
agree on how to do unit testing, decided someone else would get around
to it, resulting system has no tests".

I want our library and ecosystem of programs to have a
falling-down-easy, standard, discoverable-as-stdio kind of mechanism for
doing unit tests, so it always gets done. Likewise logging. It's more
important to have a system that gets used than to have a maximally
flexible one.

(At least *some* amount of integration with the compiler or build
process is necessary in a systems language like this, because ultimately
the output file has to say where the OS drops a PC when the image is
loaded into memory. Someone has to tell the compiler not to jump to
'main'. This is why we have a -shared flag currently in rustboot as
well, for libraries.)

> It seems like if you went this route, where you're defining the canonical
> test framework for the language, that it should be done in a very neutral,
> minimal and extensible way so that other test frameworks can piggyback on
> it.

Well, a bit. How about "defining *a* canonical test framework". Maybe
lightly extensible (an obj interface you can provide your own impl for),
but I mean, basic test-reporting interfaces shouldn't be that hard. If a
standard library can provide things like containers, PRNGs, IO systems,
file formats, OS interfaces, i18n and debug interfaces ... it seems like
it can provide a unit test framework. Many languages provide such now.

-Graydon

Reply | Threaded
Open this post in threaded view
|

unit testing

Brian Anderson-2
On Wed, Dec 29, 2010 at 4:53 PM, Graydon Hoare <graydon at mozilla.com> wrote:

> On 10-12-26 07:58 PM, Brian Anderson wrote:
>
>  I'm not familiar with D. What's the advantage of integrating unit testing
>> so
>> tightly with the compiler? What does this provide over xunit-style
>> frameworks that just use reflection to find and run tests?
>>
>
> What I said in the email: I'd like to err on the side of "discoverable"
> over "configurable". Configuring a unit testing framework is nice; but I've
> seen far too many projects fall down the slope of "didn't even agree on how
> to do unit testing, decided someone else would get around to it, resulting
> system has no tests".
>
> I want our library and ecosystem of programs to have a falling-down-easy,
> standard, discoverable-as-stdio kind of mechanism for doing unit tests, so
> it always gets done. Likewise logging. It's more important to have a system
> that gets used than to have a maximally flexible one.
>
> (At least *some* amount of integration with the compiler or build process
> is necessary in a systems language like this, because ultimately the output
> file has to say where the OS drops a PC when the image is loaded into
> memory. Someone has to tell the compiler not to jump to 'main'. This is why
> we have a -shared flag currently in rustboot as well, for libraries.)


That's what I was missing. I guess I'm pretty used to the JVM and CLR where
there's not really a distinction between executables and libraries.
I think I understand now.



>
>  It seems like if you went this route, where you're defining the canonical
>> test framework for the language, that it should be done in a very neutral,
>> minimal and extensible way so that other test frameworks can piggyback on
>> it.
>>
>
> Well, a bit. How about "defining *a* canonical test framework". Maybe
> lightly extensible (an obj interface you can provide your own impl for), but
> I mean, basic test-reporting interfaces shouldn't be that hard. If a
> standard library can provide things like containers, PRNGs, IO systems, file
> formats, OS interfaces, i18n and debug interfaces ... it seems like it can
> provide a unit test framework. Many languages provide such now.


I agree that providing a unit testing framework in the standard library is a
good thing, and leaving the interface for defining tests open to other
implementations is possibly all that's needed as an extension mechanism. I
was thinking about the diversity of test frameworks in Java and how many of
them provide some kind of JUnit adapter to ease adoption. It's good to leave
people the option of building sprawling BDD frameworks and such on top of
the standard one.

-Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20101230/332c3ee6/attachment.html>