The prize-winners this year, as so often in this uneven series, are no big deal: Leo Litwak's autobiographical ""The Eleventh Edition""; a pretentious fog of a story by Peter Matthiessen (""Lumumba Lives""); and an unusually heavy-handed story by Lore Segal (""The Reverse Bug""). Walking away with the honors instead are one story by a well-known writer and three by relative newcomers. T. Coraghessan Boyle's ""The Ape Lady in Retirement"" has Boyle's emblematic humor at full wattage. Bruce Fleming's ""The Autobiography of Gertrude Stein"" is a humane piece of lit crit folded into the pathos of a scaled-down life. Jane Brown Gillette's ""Sins Against Animals"" may be the best thing in the book: funny, spiky, long-viewed, and consistently surprising, the sad-funny tale of two women-friends' unprepared progress into disappointing life. And Devon Jersild's ""In Which John Imagines His Mind As a Pond"" has all the domestic-erotic attention of early Updike without the piety. Other worthy contributions are by Mice Adams, Janice Eidus, Marilyn Sides--all in all, making for a richer meal than this annual smorgasbord generally provides.