[federicof]

Ok but here the point is not about technology, it's very easy for us to give you access to the output of your program (in fact if you use a compiled language, and the compilation phase breaks, we show you the log). The point is that any kind of feedback that I give you access to that is generated from the input we send you to you program, will give you the chance to produce a solution that i not generic, because you know the input in advance, you can precompute the solution and just "print" it. You can object that we should generate the inputs randomly and on the fly. That can be definitely done for the very trivial problems (like reverse a string or a sequence) but will be impractical for more difficult problems that have very few input data sets that lead to a "meaningful" result.

Anyway, we're open to suggestions on how to improve the process. :)

[tetha]

Well you have a correct solver to the problem. Couldn't you just randomly generate an input, push it through the certified solver and supply me with the input-output-pair?

For one, even if this is the actual test input you use, I will not know it.

For two, it might not contain the trigger for my errors, but if I can get enough such samples, I should eventually get a sample which contains the problematic input.

For three, from what I see, the problems should be possible to generate randomly, especially if the certified solver rejects malformed input. E.g. for the missle problem, throw together N numbers and look what happens until the certified solver outputs a proper solution.

[federicof]

In theory it can be done, but it would be very expensive in terms of computational power. Anyway, you raised a very good point that we will keep in mind for the near future. Thanks indeed for your feedback and the nice discussion.