Potpie'S METRICS
Potpie closing the Evaluation Gap
Tested against real-world queries across five large open-source codebases, Potpie sets a new benchmark in AI-driven code analysis.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 607 63" xmlns="http://www.w3.org/2000/svg"><g d="M 0 67 L 0 0 L 801 0 L 801 67 Z M 5.846 65.761 L 71.607 0 L 73.022 1.414 L 7.261 67.175 Z M 17.022 65.761 L 82.783 0 L 84.197 1.414 L 18.436 67.175 Z M 28.198 65.761 L 93.959 0 L 95.373 1.414 L 29.612 67.175 Z M 39.372 65.761 L 105.133 0 L 106.547 1.414 L 40.786 67.175 Z M 50.548 65.761 L 116.309 0 L 117.723 1.414 L 51.962 67.175 Z M 61.723 65.761 L 127.484 0 L 128.899 1.414 L 63.138 67.175 Z M 72.899 65.761 L 138.66 0 L 140.074 1.414 L 74.313 67.175 Z M 84.073 65.761 L 149.834 0 L 151.248 1.414 L 85.487 67.175 Z M 95.249 65.761 L 161.01 0 L 162.424 1.414 L 96.663 67.175 Z M 106.425 65.761 L 172.186 0 L 173.6 1.414 L 107.839 67.175 Z M 117.598 65.761 L 183.359 0 L 184.773 1.414 L 119.012 67.175 Z M 128.774 65.761 L 194.535 0 L 195.949 1.414 L 130.188 67.175 Z M 139.95 65.761 L 205.711 0 L 207.125 1.414 L 141.364 67.175 Z M 151.124 65.761 L 216.885 0 L 218.299 1.414 L 152.538 67.175 Z M 162.3 65.761 L 228.061 0 L 229.475 1.414 L 163.714 67.175 Z M 173.475 65.761 L 239.236 0 L 240.65 1.414 L 174.889 67.175 Z M 184.649 65.761 L 250.41 0 L 251.824 1.414 L 186.063 67.175 Z M 195.825 65.761 L 261.586 0 L 263 1.414 L 197.239 67.175 Z M 207.001 65.761 L 272.762 0 L 274.176 1.414 L 208.415 67.175 Z M 218.175 65.761 L 283.936 0 L 285.35 1.414 L 219.589 67.175 Z M 229.35 65.761 L 295.111 0 L 296.525 1.414 L 230.764 67.175 Z M 240.526 65.761 L 306.287 0 L 307.701 1.414 L 241.94 67.175 Z M 251.7 65.761 L 317.461 0 L 318.875 1.414 L 253.114 67.175 Z M 262.876 65.761 L 328.637 0 L 330.051 1.414 L 264.29 67.175 Z M 274.051 65.761 L 339.812 0 L 341.226 1.414 L 275.465 67.175 Z M 285.225 65.761 L 350.986 0 L 352.4 1.414 L 286.639 67.175 Z M 296.401 65.761 L 362.162 0 L 363.576 1.414 L 297.815 67.175 Z M 452.401 65.761 L 518.162 0 L 519.576 1.414 L 453.815 67.175 Z M 608.401 65.761 L 674.162 0 L 675.576 1.414 L 609.815 67.175 Z M 307.577 65.761 L 373.338 0 L 374.752 1.414 L 308.991 67.175 Z M 463.577 65.761 L 529.338 0 L 530.752 1.414 L 464.991 67.175 Z M 619.577 65.761 L 685.338 0 L 686.752 1.414 L 620.991 67.175 Z M 318.751 65.761 L 384.512 0 L 385.926 1.414 L 320.165 67.175 Z M 474.751 65.761 L 540.512 0 L 541.926 1.414 L 476.165 67.175 Z M 630.751 65.761 L 696.512 0 L 697.926 1.414 L 632.165 67.175 Z M 329.927 65.761 L 395.688 0 L 397.102 1.414 L 331.341 67.175 Z M 485.927 65.761 L 551.688 0 L 553.102 1.414 L 487.341 67.175 Z M 641.927 65.761 L 707.688 0 L 709.102 1.414 L 643.341 67.175 Z M 341.102 65.761 L 406.863 0 L 408.277 1.414 L 342.516 67.175 Z M 497.102 65.761 L 562.863 0 L 564.277 1.414 L 498.516 67.175 Z M 653.102 65.761 L 718.863 0 L 720.277 1.414 L 654.516 67.175 Z M 352.276 65.761 L 418.037 0 L 419.451 1.414 L 353.69 67.175 Z M 508.276 65.761 L 574.037 0 L 575.451 1.414 L 509.69 67.175 Z M 664.276 65.761 L 730.037 0 L 731.451 1.414 L 665.69 67.175 Z M 363.452 65.761 L 429.213 0 L 430.627 1.414 L 364.866 67.175 Z M 519.452 65.761 L 585.213 0 L 586.627 1.414 L 520.866 67.175 Z M 675.452 65.761 L 741.213 0 L 742.627 1.414 L 676.866 67.175 Z M 374.628 65.761 L 440.389 0 L 441.803 1.414 L 376.042 67.175 Z M 530.628 65.761 L 596.389 0 L 597.803 1.414 L 532.042 67.175 Z M 686.628 65.761 L 752.389 0 L 753.803 1.414 L 688.042 67.175 Z M 385.801 65.761 L 451.562 0 L 452.976 1.414 L 387.215 67.175 Z M 541.801 65.761 L 607.562 0 L 608.976 1.414 L 543.215 67.175 Z M 697.801 65.761 L 763.562 0 L 764.976 1.414 L 699.215 67.175 Z M 396.977 65.761 L 462.738 0 L 464.152 1.414 L 398.391 67.175 Z M 552.977 65.761 L 618.738 0 L 620.152 1.414 L 554.391 67.175 Z M 708.977 65.761 L 774.738 0 L 776.152 1.414 L 710.391 67.175 Z M 407.977 65.761 L 473.738 0 L 475.152 1.414 L 409.391 67.175 Z M 563.977 65.761 L 629.738 0 L 631.152 1.414 L 565.391 67.175 Z M 719.977 65.761 L 785.738 0 L 787.152 1.414 L 721.391 67.175 Z M 418.977 65.761 L 484.738 0 L 486.152 1.414 L 420.391 67.175 Z M 574.977 65.761 L 640.738 0 L 642.152 1.414 L 576.391 67.175 Z M 730.977 65.761 L 796.738 0 L 798.152 1.414 L 732.391 67.175 Z M 429.977 65.761 L 495.738 0 L 497.152 1.414 L 431.391 67.175 Z M 585.977 65.761 L 651.738 0 L 653.152 1.414 L 587.391 67.175 Z M 741.977 65.761 L 807.738 0 L 809.152 1.414 L 743.391 67.175 Z M 440.977 65.761 L 506.738 0 L 508.152 1.414 L 442.391 67.175 Z M 596.977 65.761 L 662.738 0 L 664.152 1.414 L 598.391 67.175 Z M 752.977 65.761 L 818.738 0 L 820.152 1.414 L 754.391 67.175 Z" fill="transparent" height="67.17514421272202px" id="MkHHY6sQn" transform="translate(-68.5 -2.089)" width="820.152194993972px"><path d="M 0 67 L 0 0 L 801 0 L 801 67 Z" fill="rgba(239, 230, 223, 0.2)" height="67px" id="xwmz2wcqx" width="801px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="JJhZqQE3u" transform="translate(5.846 0)" width="67.175144212722px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="l8jPtNcC5" transform="translate(17.022 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="dkY_ZEQtX" transform="translate(28.198 0)" width="67.175144212722px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="hN7CtK8tS" transform="translate(39.372 0)" width="67.175144212722px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="L7ZdzrBaK" transform="translate(50.548 0)" width="67.175144212722px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="tUV46HNl5" transform="translate(61.723 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="oAwPrUHeR" transform="translate(72.899 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="jzlFUJsIh" transform="translate(84.073 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="QnIN09SnJ" transform="translate(95.249 0)" width="67.175144212722px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="Ivs4gSTae" transform="translate(106.425 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="MoiPDdIjK" transform="translate(117.598 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="O6WDO62gp" transform="translate(128.774 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="MEWa1wZZd" transform="translate(139.95 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="aavaai5Mh" transform="translate(151.124 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="Aiwr4YMKA" transform="translate(162.3 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="qNLrUJi9W" transform="translate(173.475 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="erJra6U_7" transform="translate(184.649 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="oaqSLhkAK" transform="translate(195.825 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="gNFFTPHCh" transform="translate(207.001 0)" width="67.17514421272196px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="M6Cs0QwIm" transform="translate(218.175 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="U2qJ6wldl" transform="translate(229.35 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="cR3UPdl3q" transform="translate(240.526 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="GVPfEKQIt" transform="translate(251.7 0)" width="67.17514421272202px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="EiCcpEe28" transform="translate(262.876 0)" width="67.17514421272205px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="v3EZlqWdv" transform="translate(274.051 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="H9mJOSLCd" transform="translate(285.225 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="vkLaZOu0b" transform="translate(296.401 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="bs9ESjn4d" transform="translate(452.401 0)" width="67.17514421272205px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="SQjUytR2g" transform="translate(608.401 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="cLsUBpiPa" transform="translate(307.577 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="Mf4kYCaXp" transform="translate(463.577 0)" width="67.17514421272193px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="x7ITV2Os5" transform="translate(619.577 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="TWi9rYJpU" transform="translate(318.751 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="rSXsBx_qf" transform="translate(474.751 0)" width="67.17514421272193px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="FbJbDb03g" transform="translate(630.751 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="oYpIUDXsL" transform="translate(329.927 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="D_89hTk88" transform="translate(485.927 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="yn1JlRbjA" transform="translate(641.927 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="vx0kwxPJq" transform="translate(341.102 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="y3A4MU_Za" transform="translate(497.102 0)" width="67.17514421272205px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="dV_xHdeQF" transform="translate(653.102 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="Nl5rVkVDJ" transform="translate(352.276 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="K_jwYWAz6" transform="translate(508.276 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="ovZfGln0K" transform="translate(664.276 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="cLU2gq3qM" transform="translate(363.452 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="T89O6CyE6" transform="translate(519.452 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="JpTwn5R7u" transform="translate(675.452 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="SoZSZA55j" transform="translate(374.628 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="bZPMK70P3" transform="translate(530.628 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="W31V65xv7" transform="translate(686.628 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="rAMYsSr7x" transform="translate(385.801 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="qya8p2RAX" transform="translate(541.801 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="nlTMfplST" transform="translate(697.801 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="asgda7734" transform="translate(396.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="TQtBtdcWN" transform="translate(552.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="EsGNrD5O5" transform="translate(708.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="H0nLMlh0H" transform="translate(407.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="hiH4IGcK8" transform="translate(563.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="HD2gW45l_" transform="translate(719.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="AxKyFKwCz" transform="translate(418.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="xsKMh0MXw" transform="translate(574.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="EU999VdIj" transform="translate(730.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="urnBvmW2l" transform="translate(429.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="XB4y0xTkN" transform="translate(585.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="shxphg_w0" transform="translate(741.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="HVK6IxzVc" transform="translate(440.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="vryp6o9y3" transform="translate(596.977 0)" width="67.17514421272199px"/><path d="M 0 65.761 L 65.761 0 L 67.175 1.414 L 1.414 67.175 Z" fill="rgba(239, 230, 223, 0.2)" height="67.17514421272202px" id="tPwz5K_k5" transform="translate(752.977 0)" width="67.17514421272199px"/></g></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 178" xmlns="http://www.w3.org/2000/svg"><path d="M 0 85.833 L 148.668 0 L 297.335 85.833 L 148.668 171.667 Z" fill="rgb(0, 49, 48)" height="171.66666666666666px" id="gMI0SlmfZ" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.72" stroke="rgb(104, 154, 16)" transform="translate(5.723 3.167)" width="297.33524999999673px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 103 178" xmlns="http://www.w3.org/2000/svg"><path d="M 0 113.617 L 0 0 L 98.395 56.809 L 98.395 170.426 Z" fill="rgb(0, 49, 48)" height="170.4255319148936px" id="kA3lkAnGm" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="3.79" stroke="rgb(104, 154, 16)" transform="translate(2.267 3.787)" width="98.39518085106621px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 103 178" xmlns="http://www.w3.org/2000/svg"><path d="M 0 170.426 L 0 56.809 L 98.395 0 L 98.395 113.617 Z" fill="rgb(0, 69, 66)" height="170.4255319148936px" id="iftfqTrI5" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="3.79" stroke="rgb(104, 154, 16)" transform="translate(2.267 3.787)" width="98.39518085106275px"/></svg>)
[PR READY]
Overview
We evaluate multiple AI code understanding agents across a set of curated questions drawn from real-world repositories to highlight practical differences in how they retrieve context, reason across files, and explain code behavior. The goal is to surface how well each agent handles tasks that resemble everyday developer queries.
Each agent is evaluated using default configurations with no additional tuning or repository-specific customization. We measure answer accuracy, reasoning quality, context retrieval, and consistency across queries to reflect how these systems perform in realistic development workflows.
[METHODOLOGY]
Five diverse, Production-grade open-source repositories
For QA benchmark, we selected repositories that represented real-world software complexity and workflows. The chosen repos are very popular open source projects with active development history, meaningful dependency structures, and diverse code patterns so agents must reason across files, modules, and context. We evaluated our codegen against the standard swe-bench lite dataset which comprises of 300 curated tasks taken from real GitHub issue–pull request pairs from repos like django, astropy etc.
Describe your feature idea..... (e.g., Integrate payment with stripe)
Select repo
Attach
Build a feature
Describe a feature idea and let the agent build it end-to-end with comprehensive planning, design and implementation
Full stack implementation
Build a workflow
Create automated workflows and integrations that streamline your development process and seamlessly connect your services
Automation and integration
Debug an error
Share an error message, stack trace, or failing behavior and let the agent analyze, diagnose, and guide you to a fix.
Error analysis and troubleshooting
apache/airflow
Distributed workflow orchestration system with complex state management
8 queries
mattermost/mattermost
Real-time collaboration platform with WebSocket architecture
10 queries
microsoft/vscode
Large-scale, well-documented editor platform
6 queries
supabase/supabase
Backend-as-a-service platform with TypeScript/React patterns
10 queries
tinygrad/tinygrad
Deep learning framework focusing on tensor operations
10 queries
Analysis
We further analyze results by categorizing questions into architecture guarantees and implicit behavior, data flow, and error handling. Although QA evaluations can rely on a range of metrics, our analysis prioritizes correctness and completeness. In our benchmarks, Claude and Potpie produced comparable results in terms of correctness, but Potpie scored slightly higher on completeness by consistently covering more relevant details and context in its answers. We used opus 4.6 for all agents in this analysis.
Ready to experience engineering without the grind?
Start using Potpie and let agents handle the complexity while your team focuses on impact.
© 2026 Potpie. All rights reserved.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 178" xmlns="http://www.w3.org/2000/svg"><path d="M 153.903 0.03 C 154.026 -0.018 154.169 -0.009 154.285 0.058 L 306.831 88.147 L 306.833 88.148 C 306.967 88.226 307.05 88.368 307.05 88.521 L 307.036 88.632 C 307.006 88.739 306.934 88.833 306.833 88.892 L 154.23 176.429 C 154.128 176.486 154.008 176.502 153.896 176.472 L 153.789 176.429 L 144.581 171.157 L 144.578 171.154 L 0.218 88.339 C 0.081 88.261 0 88.117 0 87.967 L 0.015 87.857 C 0.044 87.749 0.115 87.654 0.218 87.595 L 0.217 87.594 L 153.824 0.069 Z" fill="rgb(0, 53, 51)" height="176.48689063250688px" id="t2zO9Quc7" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.55" stroke="rgb(180, 209, 63)" transform="translate(0.773 0.758)" width="307.0499654602649px"/></svg>)
[CODEBASE Q&A AGENT]
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 235" xmlns="http://www.w3.org/2000/svg"><path d="M 0 0 L 0 49.251 L 136.104 127.832 L 136.104 78.58 L 44.424 25.66 L 35.485 20.488 L 17.558 10.146 Z" fill="rgb(0, 53, 51)" height="127.83155915492955px" id="F7uNl9dyd" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(8.021 100.549)" width="136.10429429577454px"/><path d="M 120.738 0 L 166.541 2.512 L 302.646 81.116 L 259.995 105.741 L 140.463 174.742 L 48.783 121.822 L 39.844 116.65 L 21.917 106.308 L 4.359 96.162 L 0 69.714 Z M 84.883 90.646 L 93.847 95.818 L 111.75 106.16 L 140.734 122.881 L 155.534 97.468 L 158.883 91.73 L 212.812 81.264 L 161.197 51.467 L 141.177 39.918 L 130.096 46.32 L 124.653 49.473 L 69.099 81.535 Z M 27.753 84.712 L 130.096 25.635 L 146.718 16.031 L 130.096 15.292 L 26.472 75.132 Z" fill="rgb(0, 53, 51)" height="174.74215475179432px" id="XL4eRTHOZ" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(3.647 4.386)" width="302.6461113816823px"/><path d="M 0 0 L 0 10.343 L 16.622 0.739 Z" fill="rgb(0, 53, 51)" height="10.34275225352113px" id="cBwRaCSIs" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(133.76 19.669)" width="16.622023943661958px"/><path d="M 0 59.84 L 1.28 69.419 L 103.623 10.343 L 103.623 0 Z" fill="rgb(0, 53, 51)" height="69.41898253521126px" id="yI1DYW3PI" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(30.132 19.669)" width="103.62314999999992px"/><path d="M 0 0 L 0 49.251 L 14.357 57.55 L 17.706 51.812 L 71.636 41.346 L 20.021 11.549 Z" fill="rgb(0, 53, 51)" height="57.54972676056339px" id="swIgz6ZQs" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(144.827 44.294)" width="71.63555704225351px"/><path d="M 0 41.617 L 15.785 50.728 L 24.748 55.9 L 42.651 66.242 L 72.078 49.251 L 72.078 0 L 60.997 6.403 L 55.555 9.555 Z" fill="rgb(0, 53, 51)" height="66.2424190140845px" id="Ou0foMTnv" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(72.761 44.294)" width="72.07816690140838px"/><path d="M 0 93.626 L 0 142.877 L 162.183 49.251 L 162.183 0 L 119.532 24.626 Z" fill="rgb(0, 53, 51)" height="142.87681267605637px" id="F6Glizjhy" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(144.11 85.495)" width="162.1830021126761px"/><path d="M 0 0 L 0 49.251 L 4.359 75.699 L 4.359 26.448 Z" fill="rgb(0, 53, 51)" height="75.69868943661974px" id="CEJbzDHVA" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.46" stroke="rgb(180, 209, 63)" transform="translate(3.661 74.105)" width="4.358684366197165px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 310 279" xmlns="http://www.w3.org/2000/svg"><path d="M 306.082 274.211 C 306.082 274.577 305.887 274.915 305.57 275.099 C 305.253 275.281 304.864 275.279 304.547 275.097 L 0.512 99.11 C 0.196 98.927 0 98.589 0 98.224 L 0 1.022 C 0 0.657 0.195 0.32 0.512 0.137 C 0.828 -0.046 1.218 -0.046 1.535 0.137 C 103.711 59.13 203.347 117.428 305.57 176.322 C 305.887 176.504 306.082 176.843 306.082 177.209 Z" fill="rgba(180, 209, 63, 0.2)" height="275.2348071827544px" id="tOCVqXAMo" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.05" stroke="rgb(180, 209, 63)" transform="translate(2.023 1.876)" width="306.08189403973506px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 310 276" xmlns="http://www.w3.org/2000/svg"><path d="M 304.914 0.132 C 305.229 -0.046 305.613 -0.044 305.926 0.138 C 306.239 0.32 306.432 0.654 306.432 1.016 L 306.432 97.578 C 306.432 97.943 306.235 98.281 305.918 98.462 L 1.521 272.088 C 1.206 272.267 0.819 272.265 0.506 272.084 C 0.193 271.901 0 271.566 0 271.204 L 0 175.241 C 0 174.877 0.195 174.541 0.51 174.36 L 219.833 48.435 L 219.837 48.431 Z" fill="rgba(180, 209, 63, 0.2)" height="272.22086421478906px" id="PPsuL8gM9" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="2.04" stroke="rgb(180, 209, 63)" transform="translate(2.016 1.798)" width="306.4324539473682px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 178" xmlns="http://www.w3.org/2000/svg"><path d="M 153.903 0.03 C 154.026 -0.018 154.169 -0.009 154.285 0.058 L 306.831 88.147 L 306.833 88.148 C 306.967 88.226 307.05 88.368 307.05 88.521 L 307.036 88.632 C 307.006 88.739 306.934 88.833 306.833 88.892 L 154.23 176.429 C 154.128 176.486 154.008 176.502 153.896 176.472 L 153.789 176.429 L 144.581 171.157 L 144.578 171.154 L 0.218 88.339 C 0.081 88.261 0 88.117 0 87.967 L 0.015 87.857 C 0.044 87.749 0.115 87.654 0.218 87.595 L 0.217 87.594 L 153.824 0.069 Z" fill="rgb(0, 53, 51)" height="176.48689063250686px" id="YZ_qP17EP" stroke-dasharray="" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.55" stroke="rgb(180, 209, 63)" transform="translate(0.773 0.758)" width="307.04996546026496px"/><g d="M 62.549 28.787 C 62.536 28.761 62.526 28.736 62.519 28.709 C 62.535 28.768 62.56 28.824 62.6 28.878 C 62.578 28.849 62.562 28.818 62.549 28.787 Z M 62.549 28.787 L 62.6 28.878 C 62.56 28.824 62.535 28.768 62.519 28.709 Z M 54.656 27.136 L 53.678 29.273 C 52.829 31.129 53.725 33.105 56.041 34.482 L 67.491 41.285 L 70.928 43.27 C 73.406 44.7 77.049 45.237 80.421 44.654 L 121.141 37.611 C 120.159 39.696 120.291 41.929 121.582 44.003 C 125.001 49.499 120.112 55.555 110.627 57.574 C 105.569 58.649 101.585 60.95 99.722 63.869 C 96.226 69.347 85.734 72.169 76.215 70.194 C 71.142 69.142 65.525 69.424 60.847 70.967 C 52.07 73.864 40.885 72.132 35.787 67.087 C 33.07 64.398 28.45 62.529 23.146 61.975 C 13.195 60.934 6.552 55.458 8.262 49.704 C 9.172 46.637 7.711 43.493 4.263 41.1 C -2.205 36.611 -1.202 29.951 6.515 26.18 C 10.627 24.17 13.008 21.221 13.03 18.11 C 13.07 12.271 21.254 7.547 31.367 7.524 C 36.757 7.511 41.866 6.135 45.347 3.761 C 50.795 0.044 59.728 -0.975 67.055 0.959 Z M 121.101 29.063 C 121.295 29.225 121.319 29.421 121.365 29.811 C 121.437 30.422 121.416 31.043 121.303 31.658 C 121.245 31.973 121.216 32.132 121.07 32.303 C 120.952 32.442 120.731 32.596 120.516 32.689 C 120.25 32.804 119.941 32.858 119.324 32.965 L 78.041 40.112 C 77.569 40.193 77.056 40.118 76.707 39.917 L 73.268 37.93 C 72.552 37.516 72.859 36.812 73.828 36.645 L 118.972 28.911 C 119.692 28.787 120.052 28.726 120.384 28.774 C 120.655 28.814 120.941 28.93 121.101 29.063 Z M 77.422 5.057 L 85.829 15.345 L 85.834 15.352 C 85.944 15.485 85.999 15.553 86.076 15.611 C 86.144 15.664 86.226 15.711 86.317 15.751 C 86.42 15.796 86.54 15.83 86.779 15.895 L 107.021 21.407 L 107.041 21.413 C 107.775 21.613 108.144 21.713 108.236 21.838 C 108.316 21.947 108.297 22.073 108.187 22.173 C 108.059 22.288 107.655 22.352 106.863 22.48 L 64.254 29.318 C 63.803 29.4 63.362 29.338 63.037 29.193 L 62.989 29.169 C 62.826 29.09 62.694 28.987 62.61 28.87 L 62.559 28.779 L 62.529 28.701 C 62.501 28.597 62.509 28.489 62.561 28.376 L 73.706 5.239 L 73.713 5.226 C 74.151 4.315 74.371 3.857 74.769 3.71 C 75.115 3.582 75.548 3.562 75.927 3.654 C 76.364 3.761 76.717 4.193 77.422 5.057 Z" fill="transparent" height="72.36999344346039px" id="PThBnjLXo" transform="translate(95.525 49.205)" width="122.67217843649964px"><path d="M 0.03 0.077 C 0.017 0.052 0.007 0.027 0 0 C 0.016 0.059 0.041 0.115 0.081 0.169 C 0.059 0.139 0.043 0.109 0.03 0.077 Z" fill="rgb(180, 209, 63)" height="1px" id="FZhYXbO6L" transform="translate(62.519 28.709)" width="1px"/><path d="M 62.549 28.787 L 62.6 28.878 C 62.56 28.824 62.535 28.768 62.519 28.709 Z M 54.656 27.136 L 53.678 29.273 C 52.829 31.129 53.725 33.105 56.041 34.482 L 67.491 41.285 L 70.928 43.27 C 73.406 44.7 77.049 45.237 80.421 44.654 L 121.141 37.611 C 120.159 39.696 120.291 41.929 121.582 44.003 C 125.001 49.499 120.112 55.555 110.627 57.574 C 105.569 58.649 101.585 60.95 99.722 63.869 C 96.226 69.347 85.734 72.169 76.215 70.194 C 71.142 69.142 65.525 69.424 60.847 70.967 C 52.07 73.864 40.885 72.132 35.787 67.087 C 33.07 64.398 28.45 62.529 23.146 61.975 C 13.195 60.934 6.552 55.458 8.262 49.704 C 9.172 46.637 7.711 43.493 4.263 41.1 C -2.205 36.611 -1.202 29.951 6.515 26.18 C 10.627 24.17 13.008 21.221 13.03 18.11 C 13.07 12.271 21.254 7.547 31.367 7.524 C 36.757 7.511 41.866 6.135 45.347 3.761 C 50.795 0.044 59.728 -0.975 67.055 0.959 Z M 121.101 29.063 C 121.295 29.225 121.319 29.421 121.365 29.811 C 121.437 30.422 121.416 31.043 121.303 31.658 C 121.245 31.973 121.216 32.132 121.07 32.303 C 120.952 32.442 120.731 32.596 120.516 32.689 C 120.25 32.804 119.941 32.858 119.324 32.965 L 78.041 40.112 C 77.569 40.193 77.056 40.118 76.707 39.917 L 73.268 37.93 C 72.552 37.516 72.859 36.812 73.828 36.645 L 118.972 28.911 C 119.692 28.787 120.052 28.726 120.384 28.774 C 120.655 28.814 120.941 28.93 121.101 29.063 Z M 77.422 5.057 L 85.829 15.345 L 85.834 15.352 C 85.944 15.485 85.999 15.553 86.076 15.611 C 86.144 15.664 86.226 15.711 86.317 15.751 C 86.42 15.796 86.54 15.83 86.779 15.895 L 107.021 21.407 L 107.041 21.413 C 107.775 21.613 108.144 21.713 108.236 21.838 C 108.316 21.947 108.297 22.073 108.187 22.173 C 108.059 22.288 107.655 22.352 106.863 22.48 L 64.254 29.318 C 63.803 29.4 63.362 29.338 63.037 29.193 L 62.989 29.169 C 62.826 29.09 62.694 28.987 62.61 28.87 L 62.559 28.779 L 62.529 28.701 C 62.501 28.597 62.509 28.489 62.561 28.376 L 73.706 5.239 L 73.713 5.226 C 74.151 4.315 74.371 3.857 74.769 3.71 C 75.115 3.582 75.548 3.562 75.927 3.654 C 76.364 3.761 76.717 4.193 77.422 5.057 Z" fill="rgb(180, 209, 63)" height="72.36999344346039px" id="AQyCIziS7" transform="translate(0 0)" width="122.67217843649964px"/></g></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 251" xmlns="http://www.w3.org/2000/svg"><path d="M 165.698 0 L 261.375 55.239 L 226.342 130.688 L 95.677 150.906 L 0 95.666 L 35.018 20.227 Z" fill="rgb(0, 53, 51)" height="150.90560294117645px" id="jxii70vzV" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(24.37 4.791)" width="261.37546588235216px"/><path d="M 29.623 10.224 L 36.309 7.429 L 43.611 4.988 L 51.344 3.009 L 59.322 1.509 L 67.439 0.515 L 75.693 0.009 L 84.164 0 L 92.557 0.515 L 100.919 1.544 L 108.728 3.018 L 116.384 4.988 L 123.793 7.455 L 130.373 10.224 L 136.629 13.481 L 142.286 17.103 L 147.097 20.91 L 151.356 25.179 L 154.784 29.643 L 157.336 34.117 L 159.104 38.936 L 159.995 43.657 L 159.981 48.672 L 159.104 53.438 L 157.336 58.222 L 154.784 62.73 L 151.356 67.194 L 147.128 71.41 L 142.286 75.271 L 136.629 78.892 L 130.373 82.149 L 123.686 84.945 L 116.384 87.386 L 108.728 89.356 L 100.843 90.838 L 92.557 91.859 L 84.164 92.374 L 75.632 92.365 L 67.439 91.859 L 59.322 90.865 L 51.344 89.365 L 43.611 87.386 L 36.233 84.918 L 29.623 82.149 L 23.351 78.883 L 17.709 75.271 L 12.928 71.464 L 8.639 67.194 L 5.227 62.775 L 2.613 58.124 L 0.892 53.438 L 0 48.592 L 0 43.782 L 0.892 38.936 L 2.613 34.25 L 5.227 29.599 L 8.639 25.179 L 12.867 20.963 L 17.709 17.103 L 23.351 13.49 Z" fill="rgb(0, 53, 51)" height="92.37355058823529px" id="LN6Mj2wjX" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(75.05 69.207)" width="159.99511058823543px"/><path d="M 29.623 10.224 L 36.309 7.429 L 43.611 4.988 L 51.344 3.009 L 59.322 1.509 L 67.439 0.515 L 75.693 0.009 L 84.164 0 L 92.557 0.515 L 100.919 1.544 L 108.729 3.018 L 116.383 4.988 L 123.794 7.455 L 130.374 10.224 L 136.63 13.481 L 142.287 17.103 L 147.098 20.91 L 151.357 25.179 L 154.785 29.643 L 157.337 34.117 L 159.103 38.936 L 159.996 43.657 L 159.979 48.672 L 159.103 53.438 L 157.337 58.222 L 154.785 62.73 L 151.357 67.194 L 147.129 71.41 L 142.287 75.271 L 136.63 78.892 L 130.374 82.149 L 123.687 84.946 L 116.383 87.385 L 108.729 89.355 L 100.843 90.838 L 92.557 91.858 L 84.164 92.374 L 75.632 92.365 L 67.439 91.858 L 59.322 90.864 L 51.344 89.364 L 43.611 87.385 L 36.233 84.918 L 29.623 82.149 L 23.351 78.883 L 17.709 75.271 L 12.928 71.463 L 8.639 67.194 L 5.227 62.775 L 2.613 58.124 L 0.892 53.438 L 0 48.592 L 0 43.781 L 0.892 38.936 L 2.613 34.25 L 5.227 29.599 L 8.639 25.179 L 12.867 20.963 L 17.709 17.103 L 23.351 13.49 Z" fill="rgb(0, 53, 51)" height="92.37427764705882px" id="jDPtsCydt" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(75.76 83.833)" width="159.99583764705716px"/><path d="M 29.623 10.224 L 36.309 7.429 L 43.611 4.988 L 51.344 3.009 L 59.322 1.509 L 67.439 0.515 L 75.693 0.009 L 84.164 0 L 92.557 0.515 L 100.919 1.544 L 108.728 3.018 L 116.384 4.988 L 123.793 7.455 L 130.373 10.224 L 136.629 13.481 L 142.286 17.103 L 147.097 20.91 L 151.356 25.179 L 154.784 29.643 L 157.336 34.117 L 159.104 38.936 L 159.995 43.657 L 159.981 48.672 L 159.104 53.438 L 157.336 58.222 L 154.784 62.73 L 151.356 67.194 L 147.128 71.41 L 142.286 75.271 L 136.629 78.892 L 130.373 82.149 L 123.686 84.945 L 116.384 87.386 L 108.728 89.356 L 100.843 90.838 L 92.557 91.859 L 84.164 92.374 L 75.632 92.365 L 67.439 91.859 L 59.322 90.865 L 51.344 89.365 L 43.611 87.386 L 36.233 84.918 L 29.623 82.149 L 23.351 78.883 L 17.709 75.271 L 12.928 71.464 L 8.639 67.194 L 5.227 62.775 L 2.613 58.124 L 0.892 53.438 L 0 48.592 L 0 43.782 L 0.892 38.936 L 2.613 34.25 L 5.227 29.599 L 8.639 25.179 L 12.867 20.963 L 17.709 17.103 L 23.351 13.49 Z" fill="rgb(0, 53, 51)" height="92.37355058823529px" id="UKafYhxUp" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(75.05 69.207)" width="159.99511058823543px"/><path d="M 29.623 10.224 L 36.309 7.429 L 43.611 4.988 L 51.344 3.009 L 59.322 1.509 L 67.439 0.515 L 75.693 0.009 L 84.164 0 L 92.557 0.515 L 100.919 1.544 L 108.728 3.018 L 116.384 4.988 L 123.793 7.455 L 130.373 10.224 L 136.629 13.481 L 142.286 17.103 L 147.097 20.91 L 151.356 25.179 L 154.784 29.643 L 157.336 34.117 L 159.104 38.936 L 159.995 43.657 L 159.981 48.672 L 159.104 53.438 L 157.336 58.222 L 154.784 62.73 L 151.356 67.194 L 147.128 71.41 L 142.286 75.271 L 136.629 78.892 L 130.373 82.149 L 123.686 84.945 L 116.384 87.386 L 108.728 89.356 L 100.843 90.838 L 92.557 91.859 L 84.164 92.373 L 75.632 92.365 L 67.439 91.859 L 59.322 90.865 L 51.344 89.365 L 43.611 87.386 L 36.233 84.918 L 29.623 82.149 L 23.351 78.883 L 17.709 75.271 L 12.928 71.463 L 8.639 67.194 L 5.227 62.775 L 2.613 58.124 L 0.892 53.438 L 0 48.592 L 0 43.781 L 0.892 38.936 L 2.613 34.25 L 5.227 29.599 L 8.639 25.179 L 12.867 20.963 L 17.709 17.103 L 23.351 13.49 Z" fill="rgb(0, 53, 51)" height="92.3733688235294px" id="BTS7eFR5p" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(75.05 48.404)" width="159.99511058823543px"/><path d="M 128.886 85.255 L 0 105.189 L 0 19.934 L 128.886 0 Z" fill="transparent" height="105.18923470588237px" id="XpYvdAzW3" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(57.724 4.311)" width="128.88608117646865px"/><path d="M 92.835 53.589 L 92.835 138.028 L 0 84.439 L 0 0 Z" fill="transparent" height="138.02757352941174px" id="K8lS4_UO3" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(192.202 5.927)" width="92.8345058823547px"/><path d="M 30.371 77.481 L 0 142.926 L 0 65.41 L 30.371 0 Z" fill="transparent" height="142.92631411764705px" id="ckGqTr2MB" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(21.656 34.705)" width="30.37088294117827px"/><path d="M 30.371 77.481 L 0 142.928 L 0 65.411 L 30.371 0 Z" fill="transparent" height="142.9275864705882px" id="r3233RfUB" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(254.683 70.769)" width="30.371064705883697px"/><path d="M 92.835 53.589 L 92.835 138.027 L 0 84.439 L 0 0 Z" fill="transparent" height="138.02721000000003px" id="LSrhVSYKp" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(21.656 104.548)" width="92.83486941176557px"/><path d="M 128.886 85.255 L 0 105.19 L 0 19.934 L 128.886 0 Z" fill="transparent" height="105.18959823529411px" id="x2Xopuli2" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="round" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(120.064 138.931)" width="128.88626294117825px"/><path d="M 29.623 10.224 L 36.309 7.429 L 43.611 4.988 L 51.344 3.009 L 59.322 1.509 L 67.439 0.515 L 75.693 0.009 L 84.164 0 L 92.557 0.515 L 100.919 1.544 L 108.728 3.017 L 116.384 4.988 L 123.793 7.455 L 130.373 10.224 L 136.629 13.481 L 142.286 17.103 L 147.097 20.91 L 151.356 25.179 L 154.784 29.643 L 157.336 34.116 L 159.104 38.936 L 159.995 43.657 L 159.981 48.672 L 159.104 53.438 L 157.336 58.221 L 154.784 62.73 L 151.356 67.194 L 147.128 71.41 L 142.286 75.271 L 136.629 78.892 L 130.373 82.149 L 123.686 84.945 L 116.384 87.385 L 108.728 89.356 L 100.843 90.838 L 92.557 91.859 L 84.164 92.373 L 75.632 92.364 L 67.439 91.859 L 59.322 90.865 L 51.344 89.365 L 43.611 87.385 L 36.233 84.918 L 29.623 82.149 L 23.351 78.883 L 17.709 75.271 L 12.928 71.463 L 8.639 67.194 L 5.227 62.774 L 2.613 58.124 L 0.892 53.438 L 0 48.592 L 0 43.781 L 0.892 38.936 L 2.613 34.249 L 5.227 29.599 L 8.639 25.179 L 12.867 20.963 L 17.709 17.103 L 23.351 13.49 Z" fill="rgb(0, 53, 51)" height="92.3733688235294px" id="dtqTIoHPq" stroke-dasharray="" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-width="5.69" stroke="rgb(180, 209, 63)" transform="translate(75.05 30.156)" width="159.99511058823543px"/></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 178" xmlns="http://www.w3.org/2000/svg"><g d="M 0 0 C 62.424 52.356 97.323 76.699 124.122 133.612 C 125.774 137.121 129.275 139.414 133.153 139.414 L 178.332 139.414 C 182.27 139.414 185.824 137.027 187.434 133.433 C 220.347 59.939 255.786 56.508 309 0 C 223.902 11.24 92.218 13.79 0 0 Z" fill="transparent" height="139.41414141414143px" id="Tnd2Ny6Ly" transform="translate(0 19.295)" width="309px"><path d="M 0 0 C 62.424 52.356 97.323 76.699 124.122 133.612 C 125.774 137.121 129.275 139.414 133.153 139.414 L 178.332 139.414 C 182.27 139.414 185.824 137.027 187.434 133.433 C 220.347 59.939 255.786 56.508 309 0 C 223.902 11.24 92.218 13.79 0 0 Z" fill="rgb(244, 255, 200)" height="139.41414141414143px" id="YPPdTHgjp" width="309px"/></g></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 211" xmlns="http://www.w3.org/2000/svg"><g d="M 2.159 40.336 L 81.816 193.109 C 85.087 199.382 91.575 203.316 98.649 203.316 L 199.157 203.316 C 206.445 203.316 213.09 199.143 216.257 192.578 L 289.654 40.384 C 294.95 29.401 288.66 16.33 276.702 13.944 C 186.518 -4.05 102.962 -5.096 14.535 13.537 C 2.503 16.072 -3.526 29.434 2.159 40.336 Z" fill="transparent" height="203.31595618278868px" id="zl0pIq6pj" transform="translate(9.001 4.34)" width="291.547144895145px"><path d="M 2.159 40.336 L 81.816 193.109 C 85.087 199.382 91.575 203.316 98.649 203.316 L 199.157 203.316 C 206.445 203.316 213.09 199.143 216.257 192.578 L 289.654 40.384 C 294.95 29.401 288.66 16.33 276.702 13.944 C 186.518 -4.05 102.962 -5.096 14.535 13.537 C 2.503 16.072 -3.526 29.434 2.159 40.336 Z" fill="rgb(174, 218, 0)" height="203.31595618278868px" id="beF0XCFO6" transform="translate(0 0)" width="291.54714489452647px"/></g></svg>)
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 309 214" xmlns="http://www.w3.org/2000/svg"><g d="M 1.851 35.23 L 64.467 192.416 C 68.34 202.139 77.749 208.518 88.214 208.518 L 194.612 208.518 C 205.187 208.518 214.67 202.007 218.469 192.138 L 278.208 36.927 C 285.378 18.296 269.348 -1.055 249.578 1.717 C 168.185 13.131 109.4 10.303 29.167 0.211 C 9.712 -2.236 -5.406 17.013 1.851 35.23 Z" fill="transparent" height="208.5180952217884px" id="zmobpQvfk" transform="translate(14.419 2.548)" width="279.949712674067px"><path d="M 1.851 35.23 L 64.467 192.416 C 68.34 202.139 77.749 208.518 88.214 208.518 L 194.612 208.518 C 205.187 208.518 214.67 202.007 218.469 192.138 L 278.208 36.927 C 285.378 18.296 269.348 -1.055 249.578 1.717 C 168.185 13.131 109.4 10.303 29.167 0.211 C 9.712 -2.236 -5.406 17.013 1.851 35.23 Z" fill="rgb(145, 181, 0)" height="208.51809526570761px" id="tGoxWLGVd" transform="translate(0 0)" width="279.94971259877605px"/></g></svg>)
Potpie'S METRICS
Potpie closing the Evaluation Gap
Tested against real-world queries across five large open-source codebases, Potpie sets a new benchmark in AI-driven code analysis.
[PR READY]
[PR READY]
Overview
We evaluate multiple AI code understanding agents across a set of curated questions drawn from real-world repositories to highlight practical differences in how they retrieve context, reason across files, and explain code behavior. The goal is to surface how well each agent handles tasks that resemble everyday developer queries.
Each agent is evaluated using default configurations with no additional tuning or repository-specific customization. We measure answer accuracy, reasoning quality, context retrieval, and consistency across queries to reflect how these systems perform in realistic development workflows.
[METHODOLOGY]
Five diverse, Production-grade open-source repositories
For QA benchmark, we selected repositories that represented real-world software complexity and workflows. The chosen repos are very popular open source projects with active development history, meaningful dependency structures, and diverse code patterns so agents must reason across files, modules, and context. We evaluated our codegen against the standard swe-bench lite dataset which comprises of 300 curated tasks taken from real GitHub issue–pull request pairs from repos like django, astropy etc.
Describe your feature idea..... (e.g., Integrate payment with stripe)
Select repo
Attach
Build a feature
Describe a feature idea and let the agent build it end-to-end with comprehensive planning, design and implementation
Full stack implementation
Build a workflow
Create automated workflows and integrations that streamline your development process and seamlessly connect your services
Automation and integration
Debug an error
Share an error message, stack trace, or failing behavior and let the agent analyze, diagnose, and guide you to a fix.
Error analysis and troubleshooting
Slack
Communication
Connect your Slack workspace to get AI-powered assistance directly in your team channels.
Github
Development
Connect your GitHub repositories and manage pull requests, issues, and commits.
Notion
Documentation
Sync with Notion for knowledge management and project documentation.
Describe your feature idea..... (e.g., Integrate payment with stripe)
Select repo
Attach
Build a feature
Describe a feature idea and let the agent build it end-to-end with comprehensive planning, design and implementation
Full stack implementation
Build a workflow
Create automated workflows and integrations that streamline your development process and seamlessly connect your services
Automation and integration
Debug an error
Share an error message, stack trace, or failing behavior and let the agent analyze, diagnose, and guide you to a fix.
Error analysis and troubleshooting
Slack
Communication
Connect your Slack workspace to get AI-powered assistance directly in your team channels.
Github
Development
Connect your GitHub repositories and manage pull requests, issues, and commits.
Notion
Documentation
Sync with Notion for knowledge management and project documentation.
We further analyze results by categorizing questions into architecture guarantees and implicit behavior, data flow, and error handling. Although QA evaluations can rely on a range of metrics, our analysis prioritizes correctness and completeness. In our benchmarks, Claude and Potpie produced comparable results in terms of correctness, but Potpie scored slightly higher on completeness by consistently covering more relevant details and context in its answers. We used opus 4.6 for all agents.
Analysis
Ready to experience engineering without the grind?
Start using Potpie and let agents handle the complexity while your team focuses on impact.
© 2026 Potpie. All rights reserved.
[CODEBASE Q&A AGENT]
© 2026 Potpie. All rights reserved.
[CODEBASE Q&A AGENT]
Potpie Specialists
Potpie closing the Evaluation Gap
Tested against real-world queries across five large open-source codebases, Potpie sets a new benchmark in AI-driven code analysis.
[PR READY]
Overview
We evaluate multiple AI code understanding agents across a set of curated questions drawn from real-world repositories to highlight practical differences in how they retrieve context, reason across files, and explain code behavior. The goal is to surface how well each agent handles tasks that resemble everyday developer queries.
Each agent is evaluated using default configurations with no additional tuning or repository-specific customization. We measure answer accuracy, reasoning quality, context retrieval, and consistency across queries to reflect how these systems perform in realistic development workflows.
[custom agents]
Build agents for any workflow in your codebase
From migrations to refactors to architecture reviews create specialized agents for whatever your team needs.
Describe your feature idea..... (e.g., Integrate payment with stripe)
Select repo
Attach
Build a feature
Describe a feature idea and let the agent build it end-to-end with comprehensive planning, design and implementation
Full stack implementation
Build a workflow
Create automated workflows and integrations that streamline your development process and seamlessly connect your services
Automation and integration
Debug an error
Share an error message, stack trace, or failing behavior and let the agent analyze, diagnose, and guide you to a fix.
Error analysis and troubleshooting
apache/airflow
Distributed workflow orchestration system with complex state management
8 queries
mattermost/mattermost
Real-time collaboration platform with WebSocket architecture
10 queries
microsoft/vscode
Large-scale, well-documented editor platform
6 queries
supabase/supabase
Backend-as-a-service platform with TypeScript/React patterns
10 queries
tinygrad/tinygrad
Deep learning framework focusing on tensor operations
10 queries
Describe your feature idea..... (e.g., Integrate payment with stripe)
Select repo
Attach
Build a feature
Describe a feature idea and let the agent build it end-to-end with comprehensive planning, design and implementation
Full stack implementation
Build a workflow
Create automated workflows and integrations that streamline your development process and seamlessly connect your services
Automation and integration
Debug an error
Share an error message, stack trace, or failing behavior and let the agent analyze, diagnose, and guide you to a fix.
Error analysis and troubleshooting
apache/airflow
Distributed workflow orchestration system with complex state management
8 queries
mattermost/mattermost
Real-time collaboration platform with WebSocket architecture
10 queries
microsoft/vscode
Large-scale, well-documented editor platform
6 queries
supabase/supabase
Backend-as-a-service platform with TypeScript/React patterns
10 queries
tinygrad/tinygrad
Deep learning framework focusing on tensor operations
10 queries
Analysis
We further analyze results by categorizing questions into architecture guarantees and implicit behavior, data flow, and error handling. Although QA evaluations can rely on a range of metrics, our analysis prioritizes correctness and completeness. In our benchmarks, Claude and Potpie produced comparable results in terms of correctness, but Potpie scored slightly higher on completeness by consistently covering more relevant details and context in its answers. We used opus 4.6 for all agents.
Ready to experience engineering without the grind?
Start using Potpie and let agents handle the complexity while your team focuses on impact.
© 2026 Potpie. All rights reserved.
[CODEBASE Q&A AGENT]
© 2026 Potpie. All rights reserved.
[CODEBASE Q&A AGENT]