NEW YORK—A Federal Reserve tool designed to set a lower boundary on short-term rates didn’t work that way Tuesday, reinforcing concerns that new limits could thwart its effectiveness.

The Fed plans to use the tool, known as overnight reverse repurchase agreements, when it starts raising rates, likely some time next year. Through these trades, the central bank takes in cash from money-market mutual funds and other nonbank financial institutions in exchange for one-day loans of Treasury securities and pays them interest in return. Before Tuesday, the Fed paid a rate of 0.05%.

Fed officials say they expect the so-called reverse repo rate will serve as a floor under short-term interest rates when they start lifting borrowing costs. But on Tuesday, when heavy demand for the facility exceeded new caps on its use, the repo rate fell to zero.

Another short-term rate in the private market fell below zero Tuesday. Called the general collateral repo rate, it is the return firms receive for lending cash overnight in exchange for bonds. A negative rate means firms were paying to make the loans. In recent days, some short-term Treasury yields had also turned negative.

Analysts had warned this could happen after the Fed, wary of putting too much weight on the reverse repo facility, announced in September it would limit its use. The analysts said this could cause short-term rates to periodically drop below the intended floor, perhaps substantially.

The new limits cap the Fed’s total reverse repo activity at $300 billion daily, well above the average of about $120 billion this year. But demand for the repo trades typically surges at the end of a month and end of a quarter.

On Tuesday, the final day of September and the third quarter, participants in the repo trades wanted to park more than $407 billion at the central bank, exceeding the cap, the Federal Reserve Bank of New York said. That invoked new rules that led to the Fed’s repo rate being lowered to zero. On days with less than $300 billion in overall demand, the rate would go back to 0.05%, or wherever the Fed sets it.

The rate on Tuesday’s reverse repo operation was “even lower than what we envisioned,” said Stone & McCarthy Research Associates’ Ray Stone, one of those who had warned this could happen. The program’s overall aim to enforce a lower limit on rates “was not effective,” he said, adding that “this will probably be the situation at most future quarter-end offerings.”

Scott Skyrm, a former repo-market participant and author, said in a commentary that the general collateral repo rate fell to an average of negative 0.015%. “This is the first time” it appears to have happened at quarter end, he said.

Mr. Skyrm also attributed the recent negative Treasury yields to the cap on Fed repos. “With a limit on the facility, hundreds of billions of dollars began looking for a regulatory risk-free home and U.S. Treasury bills yields dipped into the negatives.”

The Fed declined to comment on the outcome of Tuesday’s operation. But Chairwoman Janet Yellen, at a news conference in September, didn’t appear worried about the risk of short-term rates falling below a desired level. She said the Fed expected its benchmark short-term rate could fall below the lower bound of its target range, adding “such movements should have no material effect on financial conditions or the broader economy.”

Fed officials have altered the terms of the reverse repo program before as they experimented with it over the past year, and could adjust it further if they are unhappy with how it works. They announced the new limits amid mounting concerns that if unlimited in size, the trades could distort markets and amplify financial turmoil in times of stress.

Officials have observed that short-term market rates are frequently volatile at the ends of quarters and years, and indicated they are willing to look past these disruptions, even if they result from the reverse repo program. They mostly likely would become concerned if short-term interest rates consistently fell short of their goals.

Mr. Stone said it is likely the money-market funds that are major program participants would like to see a higher overall cap on the reverse repo program, which would increase their odds of getting a better return on cash they park at the Fed. But he said those who got to use the repo facility Tuesday still got a better return than they would have in other parts of the short-term market, where rates for some investments turned negative.

Tuesday’s drop in short-term interest rates is expected to reverse quickly as demand for the repo facility eases. Demand for the Fed’s repo trades peaks at the ends of months and quarters as market participants deal with a variety of issues, including a shortage of places to park cash very short-term and regulatory factors. The Fed’s reverse repo operation on June 30, the end of the second quarter, recorded a record $339 billion in activity.

The reverse repos are one of several tools the Fed will use to raise short-term rates from near zero when the time comes. Many investors expect that to happen around the middle of next year.

The Fed has said it would continue to use a target range for the fed-funds rate, an overnight rate on interbank lending, as its key method of communicating where it wants short-term rates. The range is currently zero to 0.25%. The fed-funds rate influences other borrowing costs throughout the economy, such as on mortgages, credit cards and business loans.

The Fed plans to use the reverse repo rate to set the lower boundary on that range. Its chief tool for managing the fed-funds rate will be the interest rate it pays banks on the reserves they park at the central bank.