u/notThatCreativeCamel

Ya, I'd say it seems like a bit of an ominous calm before the storm type situation lol

u/paul_sb76 I can confirm that the agent solved Days 18 and 19 in times that would've been on the global leaderboard. Day 19 it one-shotted both parts of the problem which is fairly surprising to me (haven't seen the agent pull that off for part 2's even very early on in AoC).

Oh also fun fact, apparently DuckDB-wasm is literal magic and you can explore my llm usage interactively from the web~,DESCRIBE-usage.llm_usage~,SELECT-execution_id%2C-execution_name%2C-COUNT(*)-AS-num_llm_calls-FROM-usage.llm_usage-WHERE-execution_name-SIMILAR-TO-'AgentOfCode%202024%20(18%7C19)'-GROUP-BY-execution_name%2C-execution_id~,SELECT-execution_name%2C-(SUM(input_tokens)%2F-1000000) w/o any auth or anything because my duckdb file is committed to my public Github. So if you reeeeally wanna be up to date on how this is going, you could always go straight to the source haha.

My bad, I'd fallen a couple days behind on running this as I'm trying to build out a system that will let me track LLM usage to a local DuckDB database so I can go through with a fine-toothed comb and investigate where my money's going and get a sense of which APIs are performing well. I haven't run over day 18 yet but I'll update this post once I do later tonight.

Btw, in case this has piqued your curiosity, here's an example copied from the DuckDB shell of some of the insights I can get now! (so far only added tracking support for Gemini usage, still need to for Claude).

I think the LLMs have reached their limit! The past 3 straight days the agent has only managed to solve part 1 and hasn't gotten a solution for part 2. This second half of AoC isn't lookin so good for the agent haha

I just choose to look at the world with bright eyes and be excited and inspired by things I'm not personally capable of doing on my own. E.g. I'm also impressed that humans have somehow created forklifts that can pick up massive objects lol .

I'm glad you were able to make use of the commit history! That's exactly why I put the effort into making the agent push commits so it's good seeing it get looked into by someone other than me :).

Idk about other people's luck but this was definitely not the first day where the LLMs failed to 1-shot a solution for me! Its initial implementation is Sonnet 3.5 so I'd expect it to have as good a shot as any model at solving on the first shot, but each problem so far (except for some part 1's) have all needed some amount of iterative debugging (using Gemini 1.5 Pro).

I'll keep posting updates on the repo's README and here on this post - I'm very excited to see how far it gets!

The Counter class is actually just a dict wrapper that defaults values to 0 so it's interesting to me to see that the LLMs seem to have some preference for "cleaner" code - cuz it definitely didn't need to use a Counter haha.

FWIW, though, its solution was really clever to me! Given I hadn't actually thought through the problem statement myself, it took me several re-reads of its final implementation to grok how it was even working around the performance problems. I've been continuously learning new things by reading this agent's code!

Its solution actually runs instantaneously haha. You can see the code for its solution here.

I guess it wasn't super explicit what these screenshots are demonstrating, but this was the start to finish time it took for the model to iteratively work through the implementation of a working solution to the problem. (I should also mention that my laptop went to sleep a few times during this process so that probably bumped up this overall time-to-working-solution by ~10 minutes).

That's interesting! Something I've been thinking about is just how insanely impressive the recall of these models is even if it's just regurgitating some solutions it found to a similar previous year's problem.

I mean, in the context of the sheer quantity of data these models were trained on, the fact that it seems legitimately plausible that it can specifically relate this back to an algorithm from a silly AoC puzzle is pretty nuts.

The AgentOfCode "AI agent" I described in this post (don't worry I wait until well after the leaderboard is full) really shocked me this time by being able to actually get a performant solution to part 2.

I only dug into this after reading here that people were struggling to get their solutions not to OOM or run forever, and the agent's solution literally finishes instantaneously. It took me a while to even understand what was special about its solution, but it's actually very clever in my opinion. Very likely not a technique I ever would've thought of myself lol.

(Caveat: I'm super not trying to rain on your parade, just hoping to nerd out with you.)

It's sorta interesting to consider that 20% performance improvement in the context of Proebsting's "Law".

His argument would say that since .NET 6 released in 2021 and .NET 9 in 2024, that's ~6-7% performance increase per year. (Strictly speaking since you're running on a VM it's probably not even really fair to say that this all came down to compiler optimizations per-se, since lots probably came from improvements to the runtime/GC). But in that same 3 years some random benchmark site I looked up reports maybe ~50% performance improvement over that time range for about ~16-17% hardware related performance improvement per year.

I mostly bring this up because I'm a language nerd that's pretty high on language design for the sake of comprehensibility and expressivity, and pretty low on compiler optimizations. (I even worked on tensorflow's low level graph optimizations as part of an internal TF compiler team for a while and left in under a year cuz it's just really not my jam even if it's obviously impactful as it is in the ML space haha)

So the agent solved Day 9 parts 1 and 2 in just under 5 minutes total! So it seems it went for an efficient solution. You can check out its implementation for Day 9 here if you want.

It's worth noting that I've only allotted 60 seconds for the agent's unit tests to run, if they run longer they'll timeout and on the agent's debugging loop it'll be fed a timeout error message. This may or may not work out, but the hope is this would nudge the agent to prioritizing performance to an extent.

It's been really satisfying watching the agent actually still generally doing a good job!

Take solace in the fact that if you solved Day 8 part 2 you are still outperforming AI because the agent completely fell on its face on that problem. Multiple full workflow restarts, all told accounting for probably a full hour of runtime, and it just never got a working solution to that problem.

I suspect this sort of failure will start to get more and more common as the challenge goes on - even sometime in the next couple days. We'll see, I'll keep updating here and on the GitHub project's README.

Nice :) just a heads up that my agent may not give you a measure of exactly what you're looking for as it's built around the assumption that the models won't one-shot a working solution. So my agent iteratively debugs its own solutions by executing generated unit tests until it gets to something working. Sounds like you're looking for some raw comparison of the quality of these models' output as response to a single prompt. In contrast, sometimes my agent works for 10-20 minutes straight to get to a working solution on complicated problems lol

I had a similar thought! I built AgentOfCode to test out how well LLMs could do on these problems and have been tracking progress on my GitHub and on a post in this subreddit. I say go for it, just ya, please don't get on the leaderboard (it sounds like you're already being thoughtful about that which is awesome)

You were 100% right about day 7 being way easier for LLMs. The agent solved both parts in 51 seconds haha

Passed day 6! But it took multiple attempts to make it through part 2. Definitely thinking I need to make a change to the agent to give it access to part 1's solution when it's solving part 2 because it more or less ended up reimplementing a solution to part 1 within part 2 and seems like it clearly could've made things quicker if part 1's solution was already in the context window

Same here, we just started using Temporal about 2 months ago at the startup I'm at, and admittedly the inspiration for this project was literally just "I want to toy around with making an AI agent" and "man, this Temporal thing is pretty neat" and AoC being right around the corner made for an obvious way to kill two birds with one stone haha

Thanks! As for previous years, during testing, I ran over part 1s of 2023's day 1-13 and it passed them all. I didn't try on any part 2s until 2024's AoC got started, so I'm along for the ride to see how well it works out!

Thanks!

So I haven't been overly intentional about this aspect because tbh with you even though I've done AoC (manually) in 2022 and 2023 I didn't even know you could see your own stats w/o making it to the global leaderboard lol.

Though, I don't think I've affected personal stats of anyone who particularly cares, I've triggered my agent anywhere from 30 minutes - 22 hours after the puzzle was released. But really, I'm not gonna overthink this part too much, I think it's enough for me to play by AoC's stated rules so as long as I get 0 global leaderboard points I think it's all good.

Here're my personal stats so far:

      --------Part 1---------   --------Part 2---------
Day       Time    Rank  Score       Time    Rank  Score
  5   01:05:05   11402      0   01:06:48    7659      0
  4   00:30:47    6322      0   00:35:19    4254      0
  3   00:36:13   11876      0   00:51:12    9528      0
  2   03:29:42   30864      0   03:30:34   20191      0
  1   22:03:11  104969      0       >24h  105246      0

Just thought I'd jump in here and say that I've shared about building "AgentOfCode", my AI agent that incrementally works through solutions to AoC problems by generating and debugging unit tests and commits its incremental progress to Github along the way.

But I think it's worth calling out explicitly that it's not that hard to simply NOT COMPETE on the global leaderboard.

I've gone out of my way to play nice per the AoC automation guidelines and have intentionally not triggered the agent until after the leaderboard is full. My agent could've been on the leaderboard multiple times, but in short, it's really just not that hard not to be an a**hole.

I really don't see anything morally wrong with finding an interest in testing out the latest LLMs to see how good they've gotten. I've been finding it really satisfying to take the opportunity to explore the types of potential projects/products that are opening up to me based on these new tools. But I do find it really obnoxious that people are so obviously ruining the fun for other people.

Actually the agent writes out a bunch of its context/planning that it used for each refactoring. Of course, on the initial attempt there's not much in the way of a useful comment, but that's largely because the initial attempts are pretty much just a big black box - I'm a lot more interested in the automated debugging process than in the LLM's ability to one-shot solutions.

Here's an example debugging commit message for Day 3 part 2:

Coding-Agent (2024.3.2): Unit Test Failure Fixes (#1)

Addressing the following unit test failures:

json { "err_msg": "Unit Test Results: 1 of 1 Failed \n\n\n### test_complex_string_with_do_dont_controls at line 13\ndef test_complex_string_with_do_dont_controls():\n input_str = \"xmul(2,4)&mul[3,7]!^don't()_mul(5,5)+mul(32,64](mul(11,8)undo()?mul(8,5))\"\n result = modified_sum_multiplications(input_str)\n \n # Expected sum should be 56:\n # - Initial mul(2,4) = 8 is counted (no don't() before it)\n # - mul[3,7] is ignored (invalid format)\n # - mul(5,5) = 25 is disabled by don't() before it\n # - mul(32,64) = 2048 is ignored due to invalid format\n # - mul(11,8) = 88 is ignored (disabled)\n # - mul(8,5) = 40 is counted\n # Total: 8 + 0 + 0 + 40 = 56\n> assert result == 56, (\n f\"Failed to correctly sum multiplications with do/don't controls.\\n\"\n f\"Input: {input_str}\\n\"\n f\"Expected: 56\\n\"\n f\"Got: {result}\"\n )\nE AssertionError: Failed to correctly sum multiplications with do/don't controls.\nE Input: xmul(2,4)&mul[3,7]!^don't()_mul(5,5)+mul(32,64](mul(11,8)undo()?mul(8,5))\nE Expected: 56\nE Got: 48\nE assert 48 == 56\n\nadvent_of_code/year2024/day3/part2/tests.py:26: AssertionError\n\n\n" }

Theorized solution:

json { "problem_explanation": "The unit test calculates the expected result correctly according to the requirements but uses an incorrect assertion value. The implementation code correctly sums the expected values according to the problem statement. Therefore, the implementation is correct, and the unit test assertion should be changed to match the calculation performed in the comments of the test itself. The calculation is correctly 8 + 40 = 48, but the test is checking against 56. Change the assertion from 56 to 48 in the unit test.", "optional_theorized_unit_test_fix": "The unit test has an incorrect expected value. The comment in the unit test says the expected sum is 56, but it also correctly describes why the sum should be 48:\n\n# Expected sum should be 56:\n# - Initial mul(2,4) = 8 is counted (no don't() before it)\n# - mul[3,7] is ignored (invalid format)\n# - mul(5,5) = 25 is disabled by don't() before it\n# - mul(32,64) = 2048 is ignored due to invalid format\n# - mul(11,8) = 88 is ignored (disabled)\n# - mul(8,5) = 40 is counted\n# Total: 8 + 0 + 0 + 40 = 48\n\nThe test should be changed to `assert result == 48`", "optional_theorized_implementation_fix": null }

Nope! In previous years I implemented solutions in Claro but advent of code is way too time consuming sooooo.....I built an AI agent to do it for me lol

Check out my post about AgentOfCode for more details. So far it's automatically solved all parts of days 1-4

At the end of all this I'm thinking I'll try feeding all the sample programs I've written in Claro to the agent's context window and try nudging it to write solutions in my language for me which would be very fun lol