It refers to hunting or the chase. Let us leave unexplored the other meaning, which the Oxford English Dictionary defines as “The practice or pursuit of sexual pleasure”. At times it may seem to its participants to resemble a hunt, but that sense isn’t connected etymologically: it comes from Venus, the Roman goddess of beauty and love.
Our sense is from Old French venerie, which comes from Latin venari, to hunt. The monk in Chaucer’s Canterbury Tales describes himself as “An outrider, that loved venery”, by which he meant he loved hunting, though Chaucer’s tales are so bawdy that incautious readers might assume the other sense. Venery refers to hunting game animals such as wild boar, hares, wolves, bears and — especially — deer. A close relative to venery is venison, nowadays always meat from a deer but in earlier times the flesh of any hunted animal.
Venery dates from medieval England and is closely connected with the royal forests established by the Norman kings and nobles after the Conquest of 1066 (despite the name, the royal forests included fields and open ground as well as woods). The beasts of venery were those that were considered most noble to hunt. There were animals of a lower order, too, the beasts of the chase, which included the fox, as well as beasts of the warren, such as the rabbit and the pheasant.
This led to the expression terms of venery, collective nouns for types of game and other wild animals, such as cete of badgers, murder of crows, parliament of rooks and paddling of ducks.