As drag decreases as AR increases & lift also decreases as AR increases, so the Aerodynamic Performance (L/D) decreases.
Why should lift also decrease? You should keep wing area constant, so lift should also stay constant. This translates into increased L/D.
Generally, an aspect ratio increase has these consequences:
More wingspan, so induced drag will become lower.
Higher wing root bending moment, so structural weight goes up. This has two contributing factors: A higher lever arm of the lift forces and a reduced root thickness because of reduced wing chord. For airliners, this puts the current optimum aspect ratio somewhere around ten, for gliders the optimum is somewhere around 30.
Reduced wing chord which reduces the wing's Reynolds number, increasing viscous drag slightly.
Reduced wing volume, so less fuel can be carried. The fuel requirement drove early jets to lower aspect ratios, which also had the benefit of less complex flaps since wing area was rather large.
The lower chord also reduces the wing's pitching moment, so a smaller tailplane can be employed, or the tail lever arm of the horizontal tail can be shortened.
Higher adverse yaw, which requires a larger vertical tail surface for coordinated flight or the tail's lever arm needs to be increased.
The larger aspect ratio means the span of the wing increases and hence so does the weight of the structure. So, to carry the same load you need a heavier wing, which will decrease the benefit of the lower drag, or you need to choose different materials (metal -> composite) or construction technique (lightening holes in ribs, for example) for the wing to decrease the total weight. It's a tradeoff that you have to balance.
Frontal drag increase, which is not very important on low speeds, but increases dramatically on higher speeds.
Construction weight and resillience - the longer wing, the lower strenghts it can carry. So the long wing is good choice for relatively slow glider (low induced drag - low strenghts and little resillience required), but for higher speeds the cost overcome the benefits.