From ff8af7a8958e412a03445e71340aedab2dc5fb05 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:25:17 +0100 Subject: open_issues/unit_testing: Add 2011-03-05 IRC discussion. --- open_issues/unit_testing.mdwn | 163 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index d7821fd3..8749e650 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -70,3 +70,166 @@ abandoned). * [LaBrea](https://github.com/dustin/labrea/wiki), or similar tools can be used for modelling certain aspects of system behavior (long response times, for example). + + +# Discussion + +freenode, #hurd channel, 2011-03-05: + + what about testing though? + like sort of "what's missing? lets write tests for it so that + when someone gets to implementing it, he knows what to do. Repeat" + project + you mean creating an automated testing framework? + this is actually a task I want to add for this year, if I get + around to it :-) + yeah I'd very much want to apply for that one + cuz I've never done Kernel hacking before, but I know that with + the right tasks like "test VM functionality", I would be able to write up + the automated tests and hopefully learn more about what breaks/makes the + kernel + (and it would make implementing the feature much less hand-wavy + and more correct) + antrik, I believe the framework(CUnit right?) exists, but we just + need to write the tests. + do you have prior experience implementing automated tests? + lots of tests! + yes, in Java mostly, but I've played around with CUnit + ah, great :-) + here's what I know from experience: 1) write basic tests. 2) + write ones that test multiple features 3) stress test [option 4) + benchmark and record to DB] + well, what we'd rather need is to fix the issues we already know + from the existing testsuites :) + +[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. + + youpi, true, and I think I should check what's available in way + of tests, but if the tests are "all or nothing" then it becomes really + hard to fix your mistakes + they're not all or nothing + youpi: the existing testsuites don't test specific system features + libc ones do + we could also check posixtestsuite which does too + +[[open_issues/open_posix_test_suite]]. + + AFAIK libc has very few failing tests + +[[open_issues/glibc_testsuite]]. + + err, like twenty? + € grep -v '^#' expected-results-i486-gnu-libc | wc -l + 67 + nope, even more + oh, sorry, I confused it with coreutils + plus the binutils ones, i guess + yes + +[[open_issues/binutils#weak]]. + + anyways, I don't think relying on external testsuites for + regression testing is a good plan + also, that doesn't cover unit testing at all + why ? + sure there can be unit testing at the translator etc. level + if we want to implement test-driven development, and/or do serious + refactoring without too much risk, we need a test harness where we can + add specific tests as needed + but more than often, the issues are at the libc / etc. level + because of a combination o fthings at the translator level, which unit + testing won't find out + * nixness yewzzz! + sure unit testing can find them out. if they're good "unit" tests + the problem is that you don't necessarily know what "good" means + e.g. for posix correctness + since it's not posix + but if they're composite clever tests, then you lose that + granularity + youpi, is that a blackbox test intended to be run at the very end + to check if you're posix compliant? + also, if we have our own test harness, we can run tests + automatically as part of the build process, which is a great plus IMHO + nixness: "that" = ? + oh nvm, I thought there was a test stuie called "posix + correctness" + there's the posixtestsuite yes + it's an external one however + antrik: being external doesn't mean we can't invoke it + automatically as part of the build process when it's available + youpi, but being internal means it can test the inner workings of + certain modules that you are unsure of, and not just the interface + sure, that's why I said it'd be useful too + but as I said too, most bugs I've seen were not easy to find out at + the unit level + but rather at the libc level + of course we can integrate external tests if they exist and are + suitable. but that that doesn't preclude adding our own ones too. in + either case, that integration work has to be done too + again, I've never said I was against internal testsuite + also, the major purpose of a test suite is checking known + behaviour. a low-level test won't directly point to a POSIX violation; + but it will catch any changes in behaviour that would lead to one + what I said is that it will be hard to write them tight enough to + find bugs + again, the problem is knowing what will lead to a POSIX violation + it's long work + while libc / posixtestsuite / etc. already do that + *any* unexpected change in behaviour is likely to cause bugs + somewher + but WHAT is "expected" ? + THAT is the issue + and libc/posixtessuite do know that + at the translator level we don't really + see the recent post about link() + +[link(dir,name) should fail with +EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) + + in my memory jkoenig pointed it out for a lot of such calls + and that issue is clearly not known at the translator level + so you're saying that the tests have to be really really + low-level, and work our way up? + I'm saying that the translator level tests will be difficult to + write + why isn't it known at the translator level? if it's a translator + (not libc) bug, than obviously the translator must return something wrong + at some point, and that's something we can check + because you'll have to know all the details of the combinations + used in libc, to know whether they'll lead to posix issues + antrik: sure, but how did we detect that that was unexpected + behavior? + because of a glib test + at the translator level we didn't know it was an unexpected + behavior + gnulib actually + and if you had asked me, I wouldn't have known + again, we do *not* write a test suite to find existing bugs + right, took one for the other + doesn't really matter actually + antrik: ok, I don't care then + we write a test suite to prevent future bugs, or track status of + known bugs + (don't care *enough* for now, I mean) + hmm, so write something up antrik for GSoC :) and I'll be sure to + apply + now that we know some translators return a wrong error status in a + particular situation, we can write a test that checks exactly this error + status. that way we know when it is fixed, and also when it breaks again + nixness: great :-) + sweet. that kind of thing would also need a db backend + nixness: BTW, if you have a good idea, you can send an application + for it even if it's not listed among the proposed tasks... + so you don't strictly need a writeup from me to apply for this :-) + antrik, I'll keep that in mind, but I'll also be checking your + draft page + oh cool :) + (and it's a well known fact that those projects which students + proposed themselfs tend to be the most successful ones :-) ) + * nixness draft initiated + youpi: BTW, I'm surprised that you didn't mention libc testsuite + before... working up from there is probably a more effective plan than + starting with high-level test suites like Python etc... + wasn't it already in the gsoc proposal? + bummer + nope -- cgit v1.2.3