What do you think of Cody, new service for MATLAB Central

What do you think of Cody? I'm interested in your comments

[Walter Roberson]

I think most people probably have not heard of Cody yet.

[Image Analyst]

I haven't, unless he means the kid in my Boy Scout troop, which I doubt. Maybe someone can post a link, if it's worth knowing about.

[Walter Roberson]

http://www.mathworks.com/matlabcentral/cody

[Matt Tearle]

Well you guys are just behind the curve, then!

http://www.mathworks.com/matlabcentral/trendy/plots/677

(Haha. "Behind the curve". See what I did there?)

Answer 1

[Jan]

As Oleg has mentioned already, the measurement of the code leads to bad programming practize. Therefore the Cody-users learn to use Matlab inefficiently.

The "code-size" system for "measuring" the submitted solution do not reward:

• check number and type inputs
• consider NaN's
• handle arrays of all size, even empty arrays
• pre-allocation
• local constructions instead of calling external functions, e.g. "y = x(end:-1:1)" is worse than "y = fliplr(x)"
• optional parenthesis added to improve the readibility.

While some hard-working contributors in Answers try to provide as much good programming practice as possible to improve the total quality of the user-land code, Cody encourages to be accustomed to a compact but for larger projects horrible programming style. In consequence I think that Answers and Cody are antagonists.

Imagine I try to employ some programmers for a serious Matlab project. Will I ask the top ten of Answers, FileExchange or Cody?

It is not really clear to me, in which order I should solve the problems. Sometimes the solution of one problem let me look on the solutions of other problems I've solved some days before. This has a limited pedagogical quality only. I can see that there is a very compact solution for a certain problem. To learn how it was implemented, I have to submit any valid solution for another problem. This encourages to submit even bad solutions get the the wanted information.

Currently the method to measure the code-size can easily be confused. What happens, if the actual calculations are embedded as string in the comment section and EVALed? Can we create M-files dynamically, which shadow the functions required for testing the success? How long will it take until somebody breaks out of the sandbox and injects some SQL code?

I see a growing number of droll cheating already. But I see an invitation to (for?) fraud also.

[EDITED] See also: Blogs: Scoring in Cody

[Oleg Komarov]

I agree that CODY is antagonist of ANSWERS, therefore I would propose to change to a scoring system based on speed of execution and memory consumption, but I am afraid it is unfeasible.

[Jan]

Speed and memory depend on the size of the input. A method suitable for random 1x1000 vectors might be very slow for sorted 1000x1000 matrices. Therefore the problem must contain the targetted input data, if speed is a considered.

[Sean de Wolski]

Jan, you can write problems for CODY that meet those requirements :)

There are definitely some problems on there already that do use empty sets/nans etc and expect the solution to be correct.

[Sean de Wolski]

Perhaps you could make one that is strictly - Does this data get to move forward?

[Jan]

Sean, I see a fast evolution in Cody. That's fine.

I still do not think, that the magic code-size measurement it a good method. Are there better methods to measure the complexitiy of the code?

Answer 2

[Oleg Komarov]

I think it's:

• fun
• it teaches you may things apart matlab

But:

• Encourages bad programming practice with it's rating system
• Some problems are buggy, badly stated or the checks allow cheating

EDIT Now it's getting too tempting to brute force the solution.

EDIT2

The voting system is bugged. Once you get it right with any high score, you can fail with a low score and see the solutions of those with lower score than you.

[Jan]

+1 for mentioning, that the "short code size" measurement provokes bad programmint practize.

Answer 3

[samil]

It's good for people who want to improve themselves in a casual manner (e.g. me), but might be bad for people who are obsessed with collecting points.

Answer 4

[Bjorn Gustavsson]

Once uppon a time there were irregularly golfing contest problems on CSSM. I found those fun, inspirational and occasionally educational. There nothing was hidden behind any fancy locks and organized scoring, but worked just the same or more efficiently if learning is concerned.

Cody seems a total waste.

Answer 5

[Aurelien Queffurust]

Cody is my new drug

[Jan]

Is this a positive or negative opinion about Cody?

[Aurelien Queffurust]

positive , I have learnt a lot of tricks from others solutions

drawback : sometimes the problem is unclear

[Jan]

What kind of tricks did you learn? Are they useful only inside Cody, or also inside real-world Matlab programs?

Answer 6

[Albert Yam]

I found that a lot of the Cody (official) problems after a while turned into a lot of 'regexp' stuff and was a bit too esoteric for me.

At first, looking at other people's solutions was good. Seeing other ways to solve to problems was interesting, but after a while, the 'best' solution just gets saturated with the same thing. An explanation of what is going on in people's code would be helpful.

Have the 'best' solution be voted by players. Perhaps only allow a player's first successful submission be shown (or branch further optimizations), so that people don't just copy the 'best' and resubmit.

Answer 7

[Kevin Holst]

I like Cody as a set of brain teasers that may force you to think about fairly unique problem that you may not have encountered before. I would echo what has been said about the rating system. On the surface, it seems very good to be able to write just a few lines of code to do a task, but like Jan has said, it produces code that may be less efficient and that is usually not very readable.

Cody also allows the user to see what tests their codes will encounter if they fail the first time. I feel like this does not make the programmer write general enough functions, and allows them to just write a function that solves that SPECIFIC problem. My solution to this would be to have many different tests and only use 2 or 3 for any given check.

[Sean de Wolski]

That's an interesting idea for tests. What if the author could check the tests (s)he wants visible and hide the others? Hidden tests would require a verbose error message but you wouldn't be able to see the test.

E.g:

A test that uses nans and is hidden could say, "failed with nans".

[Kevin Holst]

That would be good, too. Still allows all tests to be conducted, but some more generic ones are hidden.

[Jan]

Some problems cannot be solved without seeing the tests, because the description does not contain enough information. Therefore I think that hidden tests are not useful in general, but the opposite: Show the tests initially also, to define unequivocally what must be done to solve the tests.

Of course this allows to create poor solutions which solve only the defined tests by using a lookup table. This can be controlled either by letting an editor delete the nonsense-solutions, or by adding further tests and re-evaluating all submitted solutions.

Answer 8

[Matt Tearle]

OK, so you know that I'm biased, but let me be clear that I'm stating my personal opinion here, not that of my corporate overlords.

I don't see Cody as antithetical or antagonistic to Answers. I see them as complementary, serving different purposes. I think it's important to see Cody for what it is: a game. A fun way to flex your MATLAB muscles.

For me in my professional life, I see Cody as a great resource for those learning MATLAB. A few minutes a day on Cody would, I think, help a new user develop and solidify their skills. Once they are writing serious code for a specific purpose, I'd expect them to use Answers as a resource for getting help with specific difficulties.

I understand, and share, the concern about teaching bad practices. However, I'm not worried about things like checking input types and dimensions, dealing with NaNs, and so on. Again, I don't see Cody as a way to develop serious MATLAB programming or application development skills. I see it as a way to develop MATLAB language skills. It's a game, and part of the rules of the game is defining what the input to this function might be. What does concern me with the scoring is that inefficient or inelegant approaches may score the best. My canonical example for that is the "pyramid number" problem, where sum(1:n) is "better" than n*(n+1)/2. FWIW, the Cody team are aware of these issues.

One specific quibble: as others have noted, there's a lot of regexp, which, to me, is not really why MATLAB is so awesome.

Occasionally I've seen some neat tricks that I may or may not tuck away in some spare neurons for future reference.

Bottom line(s):

• I like Cody as a fun way to get you to play around in MATLAB.
• I like Answers as an efficient way to get help on particular MATLAB issues.
• I like the File Exchange as a way to get code that achieves a common task, but isn't part of MATLAB.
• I like CSSM as a way to have pointless arguments about 0- or 1-based indexing ;)

Answer 9

[James Tursa]

My first experience with CODY:

Tried the "sum integers from 1 to 2^n" problem. Example used only a scalar input, but I coded up a vectorized version anyway and submitted it. It wound up in 139th place according to size. Hmmm...

So I looked at the solutions ... WHOOPS! It wouldn't let me. I had to solve another problem just to gain the right to look at the other solutions to the first problem I already solved. OK, so a bit of wasted time but I finally got permission to look at the solutions. Here is what I found, of the 184 correct solutions all ranked on size:

Solutions #1-#93 all use slight variations of sum(1:2^n). Nice, compact, short, ... and horrible programming. Forming the explicit integer array up to 2^n is a bad use of time and resources to solve this problem.

Solution #94 on the list is the first one that goes directly after the solution based on a closed form expression instead of forming the integer array explicitly. It is vectorized, but it suffers from calling the exponentation twice. Well, at least we are getting significantly better than the 1:2^n approach, but we had to wade through several pages of the 1:2^n stuff to get here ... not good.

Solution #106 on the list is the first one to use the pow2 function, but it still calls it twice.

Solution #121 on the list is the first one that calls the exponentiation ^ only once, but it is not vectorized.

My solution, #139 on the list, is the first one that gets the answer via the closed form expression, does the exponentiation only once, and is vectorized. In fact, after scanning all of the solutions mine was the only one that had all three of these features. (One would have gotten to it quicker by scanning the list of 184 correct solutions in reverse order.) I'm pretty sure that it would have wound up in dead last place had I included any argument checking.

Well, so what? CODY is being touted as just a game by TMW, not a serious programming aid. On that level, fine, I suppose. And maybe as a teaching aid in MATLAB syntax or as a general introduction to functions you didn't know about, it is OK as well. But as an aid to good programming practice I think it fails, unless there are comments to go along with the solutions.

Q: Does TMW want comments on bad programming practice etc to appear on the leading solutions? I.e., is this really just a game or does TMW want CODY to be used as a legitimate programming aid as well?

[Jan]

As far as I understand, Cody is just a game: http://blogs.mathworks.com/desktop/2012/02/06/scoring-in-cody

The sum of 1:2^n is an important example: The smart method Gauss has found in the school is a basement of the education of efficient algorithms. It hurts to see that "sum(1:2^n)" is called the "leading solution".

Answer 10

[Daniel Shub]

I tried Cody when it first came out. At that time it appeared to be about two things: solving problems and writing "good" code. After I submit, I can see if I solved a problem and if not, where I failed. I like that, so the solving problems part is fine. For the "good" code part Cody gives me a seemingly random number. I don't like playing games in which I do not know what I am being scored on.

Then I saw http://blogs.mathworks.com/desktop/2012/02/06/scoring-in-cody/#comment-8624 and read in the about cody section: http://www.mathworks.co.uk/matlabcentral/about/cody/ that I could see the scoring code. Okay, this sound fun, I may disagree with there definition of "good", but at least I can read the rules.

The rules are basically:

x = length(mtree(answer))

sounds good, except that most of the work in mtree appears to depend on the compiled closed source built-in mtreemex. So the rules apparently are written down, but are not allowed to be read. That to me is the definition of a stupid game.

[Jan]

Matlab 2009a and 2011b contain the source code of mtree, see "edit mtree" and "depfun mtree". It has 2600 lines of code and a lean documentation only. It reveals some interesting information, but it is not easy to get behind the meaning of the internal comments.

[Daniel Shub]

Sorry about that, my comment/answer was confusing. I edit it to try and clarify. In 2011a the source of mtree is also available (3200 lines of code). The issue is that it appears that most of the 3200 lines of code for mtree depend on what is returned by mtreemex (a built-in function).

[Jan]

Thanks, now I've found the mtreemex call also.

'a+b' has a shorter mtree than 'a+(b)'. At least this detail is a stupid rule for a coding contest.

Answer 11

[bym]

I find myself in the past few days exploring Cody, rather than providing answers on "answers". Probably not a great drawback to answers, but I do feel a little guilty for not providing help where I can.

I like seeing Cody solutions to problems that are elegant, that I would not have thought of. Kind of a mini contest primer.

Now if I can just hunker down and improve my regexp() skills :(

[Jan]

I prefer to get back Matt Fig and his humpday puzzlers.

Matt, where are you?

Answer 12

[David Young]

I just wrote a longish answer but lost it when the site went down briefly. The gist was that I think Cody is entertaining, a bit educational, but not to be taken too seriously.

However, I've noticed that solutions are now appearing that use evalin to hack the answer. This is so boring! I think it's vital that eval, evalin and assignin are banned. 'Solutions' that use these spoil the fun.

[Sean de Wolski]

I lost a long answer too. We are aware of the server issues.

[Walter Roberson]

I recommend "Lazarus Forms Recovery" to anyone using a Mozilla derivative (e.g., Firefox)

[David Young]

Thanks Walter!

[Helen Chen]

The server hiccups from yesterday should be fixed, so if you see any more of this behavior, please let us know.

We are looking at ways to address the eval etc concerns. Stay tuned for an update on this!

[Helen Chen]

We have updated Cody to block use of assignin, builtin, eval, evalc and evalin. We have rescored all submissions that used any of these functions. Thanks to everyone for the feedback! :-)

[David Young]

Helen, that's great, thank you!

[Jan]

@Helen: How does STR2NUM work with a blocked EVAL?

[Ned Gulley]

We have blocked some of the obvious ways to evaluate code, but there are others. Cody is just a game, and at some point people will have to decide whether they'd rather spend their time cheating or playing. We'll try to remove obvious cases of cheating, and over time we'll add features that help mitigate cheating. All online games have to deal with this.

[Jan]

Thanks, Ned. Now it is getting clearer to me. I'm interested in blocking EVAL securely after I tried to create a bullet-proof P-file for an access limitation and copy protection.

It is very hard to define accurately, what a "game" is. Without doubt it needs well defined rules. Why is EVAL() cheating, while TRUE+'0' is not?

I think the reason is, that we tell the users in Matlab Answers every day, the EVAL is cheating in real programs also.

[David Young]

Jan, I think also that if someone uses eval() to reduce the apparent size of their program, or evalin() to poof the answer in rather than compute it, that's boring for everyone else and a waste of time for them. They will not learn new techniques or solve new problems, but they will add noise to the set of solutions.

Other slightly strange methods, like '011'-'0' to reduce the node count, may only make sense in the context of the game, but to exploit them you do have to think about the problem and how to solve it.

Those are the sort of reasons that make me feel it's sensible to ban eval etc.: they reduce the need to solve the problems, while tricks with different classes and so on don't have that effect.

Oh, and I do agree with you about avoiding eval() in real programs, in general.

[Wesley Ratnakumar]

Using cheats like,

!echo "function assert(~,~)" > assert.m

should also be blocked

Answer 13

[David Young]

It would be helpful to know what toolboxes are available to solution code. I'm sure I saw a solution that used a function from the stats toolbox (which I don't have) but functions from the Image Processing Toolbox aren't found. Should be all or none, or perhaps a definite list with some rationale behind it.

[David Young]

... and it seems that imfilter (in the IPT) used to be available, but isn't any more.

[Jan]

There was a nice solution using IMDILATE to create a special pattern in a matrix. But as David found out already, the IPT is not available.

Answer 14

[K E]

Would love it, but keep getting errors that 'License checkout failed' on some of the testing of my solutions which requires resubmitting. Why?

[Sean de Wolski]

Please contact [e-mail]

[Lindsay Coutinho]

Hi K E, was just looking over your problem. Sorry we didn't get back to you sooner about this but the problem seems like you are using a flavor of 'diff' function from symbolic toolbox. We only allow MATLAB now. Make sure you are not using any toolboxes other than MATLAB. I hope this helps.

[K E]

Thanks!